AI Conversation Evaluator (Contract Role)

Gramian Consultancy is looking for experienced professionals in psychology, social work, counseling, communication, qualitative research, HCI, conflict resolution, or related advisory disciplines to support an AI conversation-evaluation project.

The role involves reviewing conversations between people and AI systems, assessing whether the AI gathered enough context before offering recommendations, identifying important missing information, and proposing relevant follow-up questions. Reviewers will distinguish between critical context and non-essential details and provide structured feedback to improve the quality of AI-assisted conversations.

Key Responsibilities

• Review AI conversations involving interpersonal, advisory, and everyday decision-making scenarios.
• Assess whether the AI gathered sufficient context before making recommendations.
• Identify important missing information and suggest relevant follow-up questions.
• Determine whether additional context could materially change the recommendation.
• Distinguish between load-bearing information and non-essential details.
• Conduct structured qualitative reviews using defined evaluation guidelines.
• Document findings clearly in spreadsheets, annotation tools, or labeling platforms.
• Apply consistent judgment across cases and flag ambiguous examples.
• Collaborate with research and quality teams to improve evaluation standards.
• Participate in calibration and quality-review activities.

Requirements

• One of the following academic or professional backgrounds:
• MSW with LCSW or an equivalent social-work license.
• MA/MS with LMFT, LPC, LCPC, or an equivalent counseling license.
• PhD or PsyD in Clinical, Counseling, Social, or Industrial-Organizational Psychology.
• MA or PhD in HCI, Communication, Anthropology, Sociology, Conflict Resolution, Qualitative Research, or a closely related field.
• Relevant professional credential combined with at least 5 years of direct advisory, coaching, mediation, consulting, or client-facing experience.
• At least 3 years of relevant professional practice, with 5+ years preferred for credential-based applicants.
• Experience gathering missing context through interviews, assessments, consultations, or advisory conversations.
• Ability to distinguish critical information from non-essential context.
• Strong written English and qualitative analytical skills.
• Strong attention to detail and ability to follow detailed labeling guidelines.