AI Benchmark Engineer | Native Language Specialist - Hindi

About the Role: Lilt is building a rigorous, verifiable evaluation suite of Terminal-Bench tasks designed to test the limits of large language models on multilingual software challenges. We are seeking an experienced native Hindi-speaking software engineer to design, build, and validate these benchmarks as an independent contractor.

What You'll Do

• Build realistic task environments using datasets and files in Hindi to genuinely measure multilingual model handling.
• Find failure points where AI does not work in your native language through prompting and translation.
• Support the development of robust solutions and write highly reliable, deterministic verifier scripts.
• Analyze execution logs and calibrate task difficulty against various model tiers (Haiku, Sonnet, Opus).
• Participate in a rigorous, 4-layer human quality control process alongside automated LLM-based checks.

What You Bring

• 5+ years of industry experience in software engineering at leading technology companies.
• Native or near-native fluency in Hindi and high English proficiency.
• Strong proficiency in Python, standard shell scripting, and data processing.
• Extensive experience with Terminal/CLI-based development workflows and coding agents.
• Deep technical understanding of multilingual text processing pitfalls (Unicode, locale conventions, text I/O).

Benefits

• Flexible schedule as an independent contractor—work when you want, as much or as little as you want.
• Competitive rates and prompt payments with no micromanaging.
• Contribute to cutting-edge AI and language technology shaping global communication.
• Fully remote flexibility from India.