Principal AI Architect

As a Principal AI Architect at FourKites, you will own the architecture underlying every FourKites AI agent: the harness they run in, the context they see, the models they call, and the evaluations determining their effectiveness. Our digital workers—such as Tracy for track and trace, Sam for supplier operations, Alan for appointment management, Cassie for customer service, and Polly for document management—perform real supply chain tasks on a shared platform, and you will define its direction. This is a hands-on individual contributor role based in Chennai (hybrid), where you will write code weekly, lead design reviews across agent teams, and make critical decisions about model choice, context strategy, and evaluation frameworks.

What you’ll be doing:

• Define the reference architecture for FourKites agents, including the agent loop, tool execution, state and checkpointing, retries and recovery, guardrails, permissions, and human-in-the-loop handoffs.
• Design systems for long-running, multi-step workflows that span multiple context windows and recover cleanly from failures (e.g., tool, carrier API, or model call failures).
• Determine optimal solutions for each problem—whether frontier LLM, open-weight model, fine-tuned encoder, or plain code—and manage model routing, cost-per-task, and latency budgets across the fleet.
• Integrate agents with FourKites’ event-driven platform (Kafka, microservices, Postgres) and customer systems via APIs and MCP servers.
• Set standards for agent context windows, including system prompts, tool definitions, retrieved documents, memory, and prior turns, within explicit token budgets.
• Develop shared components for retrieval, compaction, summarization, sub-agent isolation, prompt caching, and persistent memory to prevent redundant reinvention.
• Diagnose context failures (e.g., lost instructions, tool confusion, quality degradation) and implement reusable harness patterns for fixes.
• Build the offline evaluation framework, including golden datasets from production traces, code-based checks, LLM-as-judge graders calibrated against human labels, and CI regression suites that gate every prompt, model, or tool change.
• Establish online evaluation systems, such as scoring sampled production traffic, trace- and task-level quality metrics, drift detection, shadow and A/B rollouts, user feedback signals, and quality alerts.
• Connect evaluation metrics to business outcomes customers pay for, such as tracking accuracy, exceptions resolved without human intervention, and appointment fill rate.
• Review designs and pull requests for agent teams, author architecture documentation and playbooks, and mentor staff and senior engineers.
• Track the evolving model and tooling landscape, conduct critical experiments, and provide leadership with build, buy, or wait recommendations.
• Collaborate with Product, Solutions, and Customer Success to address field failures and implement platform-wide fixes.

About the team:

Our product and engineering teams are dedicated to delivering the industry’s best-in-class end-to-end supply chain visibility platform. The agent platform underpins every FourKites digital worker, enabling agents to track freight, manage supplier relationships, book dock appointments, answer customer inquiries, and process shipping documents in production. You will serve as the senior technical authority for how FourKites builds, deploys, and measures agents, working closely with agent teams in Chennai and colleagues in the US and Europe.

Who you are:

• Agents in production: You have architected and operated at least one LLM agent system with real users and real traffic, including tool calling, multi-step workflows, failure handling, and human escalation, and have owned it post-launch through incidents, cost, and latency management. Framework demos and POCs alone are insufficient.
• Deep ML knowledge of encoders and transformers: You understand attention mechanisms, positional encodings, encoder-only vs. decoder-only vs. encoder-decoder architectures, tokenization, and the KV cache. You have trained or fine-tuned encoder models (e.g., BERT-family, sentence embeddings, bi-encoders, cross-encoder re-rankers) for classification, extraction, or retrieval, and grasp how contrastive training, distillation, and quantization impact cost and quality.
• Context engineering: You have managed token budgets, built retrieval and memory systems for agents, used compaction and sub-agents to maintain coherence in long tasks, and measured how quality degrades as context windows fill.
• Harness engineering: You have built the execution layer around models, including agent loops, tool dispatch, state and checkpoints, interruption and recovery, sandboxing, permissions, guardrails, and model routing. You are fluent in MCP, structured outputs, and streaming.
• Hands-on online and offline evaluations: You have built evaluation suites, including datasets, graders, LLM-as-judge calibration, and CI regression gates offline; production scoring, trace sampling, drift monitoring, shadow deployments, and feedback loops online. You have used evaluations to block releases or justify model switches.
• 10+ years of experience building software or ML systems, with 3+ years focused on LLMs.
• Strong, hands-on experience with Python. Comfortable reading and reviewing Java or TypeScript services.
• Production experience with AWS or GCP, containers, Kubernetes, and event-driven systems.
• Excellent oral and written communication skills to explain technical trade-offs to engineers and business impacts to executives.

Nice to have:

• Experience in supply chain, logistics, or transportation.
• Document AI, including OCR, layout models, and extraction from bills of lading, invoices, and proof of delivery (POD) documents.
• Voice or email agents that act on behalf of businesses.
• Agent security, such as prompt injection defense, tool permissioning, and tenant data isolation.
• Hands-on experience with multiple model families (e.g., Claude, GPT, Gemini, open-weight models) and serving stacks.
• Published work, conference talks, or open-source contributions related to agents, retrieval, or evaluation.