Software Engineering Director, Agentic Evaluations
Job Description
Key Skills Required
Master these to land this role
Want to know if you're a match for this job?
As a Software Engineering Director, Agentic Evaluations, your responsibilities will center around delivering useful and credible evaluations of agents offered by software vendors. You’ll enhance the system that performs validation of agent performance and lead the core team behind it. You will leverage “agent-first” ways of working, applying agentic engineering techniques to deliver your work to production.
Detailed Responsibilities:
Build and enhance agent evaluations that run against live agents [80%]:
- Help scope out the feasibility and effort involved in evaluating an agent offered by a software vendor, identifying the most practical path to reaching a credible evaluation
- Own the full stack of the core evaluation system, from admin & API surface to eval workflow dispatch
- Design evaluation system primitives that generalize across software verticals, so onboarding a new category benefits from reuse
- Distill the appropriate and repeatable processes in onboarding and maintaining integrations into repeatable AI skills or agents to increase evaluation velocity
- Keep tabs on emerging frameworks and techniques for agent evaluation which offer improvements to evaluations, and provide industry-accepted means to distribute eval results
- Collaborate with data science peers in their pursuit to develop rich and proprietary benchmarks
Lead a group of engineers and be a source of wider influence [20%]:
- Facilitate the growth and mentorship of a core team of engineers, and be a source of technical leadership and subject matter expertise in evals for them
- Help socialize the use of evals in agent-oriented products built by the wider organization
How would you rate this job post?
See what other professionals think about this role.
Similar Opportunities
More Openings at G2
Explore Top Companies in this Space
Cube Asia
E-commerce Data Infrastructure & Intelligence / Market Measurement & Retail Analytics / Enterprise B2B SaaS / Consumer Insights Platforms
NewtonX
B2B Market Research / Data Intelligence / SaaS
Neko Health
Healthcare Technology / Digital Health / Preventive Medicine / AI-Driven Diagnostics
Local Infusion
Healthcare / Medical Services / Ambulatory Care / Proprietary Software
G2 is a premier, enterprise-grade B2B software and services review platform engineered to orchestrate massive-scale market intelligence ecosystems and intelligent frictionless buyer-decisioning workflows. Operating as the world’s largest and most trusted marketplace for software discovery, the company eliminates the operational friction of traditional, biased software procurement by seamlessly deploying advanced community-vetted review telemetry, rigorous algorithmic vendor-scoring architectures, and cohesive data-driven research frameworks. Moving beyond rigid legacy analyst-firm reports, G2 empowers over 5 million monthly buyers, high-growth startup marketers, and global enterprise procurement teams to dynamically synchronize their technology-acquisition pipelines with elite social proof—leveraging over 3.4 million authentic, human-verified user reviews. Under the hood, their sophisticated proprietary data infrastructure natively manages complex cross-category ingestion (spanning AI, CRM, ERP, and project management), instantaneous vendor-authority routing via their G2 Crowd scoring methodology, and strategic marketing-solutions telemetry that helps software vendors reach in-market buyers at the precise moment of intent. What sets G2 apart is its uncompromising dedication to frictionless market orchestration; by bridging the gap between volatile, high-velocity software innovation cycles and accessible, human-validated technical truth, the platform empowers the global B2B economy to radically accelerate their digital transformation velocity, optimize tech-stack investment economics, and build an unassailable foundation for continuous commercial and competitive dominance in the modern digital procurement landscape.
Safety First
- Never pay for a job application.
- Do not share sensitive bank info.
- Verify the client before starting work.