AI Quality & Evaluation Specialist

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

The Role

We are seeking an AI Quality & Evaluation Specialist to validate task quality, review AI agent performance, and audit grading logic. You will analyze execution traces, tool calls, reference solutions, and evaluation criteria to identify task defects, grading errors, and unjustified model failures. The ideal candidate combines technical fluency with strong analytical judgment and the ability to provide clear, evidence-based feedback.

Responsibilities

• Validate task instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
• Review AI agent execution traces, tool calls, and generated deliverables to assess whether outcomes are justified.
• Audit grading logic to identify brittle checks, incorrect expected answers, and unsupported rubric criteria.
• Identify cases where valid alternative solutions are unfairly penalized.
• Investigate discrepancies between model performance, task defects, grader errors, and environment or tool failures.
• Independently assess automated QC findings rather than accepting them without verification.
• Document concise, evidence-backed findings and actionable recommendations.
• Clearly communicate uncertainty and distinguish confirmed issues from potential problems.
• Verify that task revisions resolve previously identified defects.
• Collaborate with relevant teams to improve evaluation quality and reliability.

Requirements

• Experience in technical QA, AI evaluation, data analysis, software testing, or a related technical field.
• Comfortable reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.
• Strong analytical skills, including the ability to validate calculations and reconcile conflicting evidence.
• Experience assessing the correctness and completeness of technical or professional deliverables.
• Strong written English and the ability to provide specific, reproducible feedback.
• High attention to detail when reviewing task requirements, grading behavior, and model outputs.
• Ability to independently investigate issues and make evidence-based decisions.