Principal Backend Engineer

We are seeking a Principal Backend Engineer to own the scalability, reliability, and architectural evolution of an AI platform as it transitions from a prototype to a production-grade platform. This is a high-impact, high-ownership role at the intersection of distributed systems engineering and applied AI.

KEY RESPONSIBILITIES

• Platform Architecture & Scalability: Own end-to-end backend architecture for the AI platform; design for multi-tenant, high-throughput extraction workloads
• Extraction Pipeline Engineering: Architect and evolve AI pipelines, define pipeline abstractions, registry patterns, and execution strategies that balance accuracy, latency, and LLM API cost.
• Multi-Provider LLM Abstraction: Enhance and harden the unified LLM provider layer ; implement multi-key round-robin pooling, structured output schemas, tool-calling protocols, and provider failover logic.
• Async Worker & Queue Architecture: Design the distributed worker system for heavy pipeline execution observable queue with dead-letter handling, retry policies, and backpressure management.
• AWS Production Deployment: Lead the AWS deployment architecture (ECS/EC2, RDS PostgreSQL, S3, Bedrock, ALB, Secrets Manager, CloudWatch); define IaC, blue-green deployment strategies, and ensure the platform meets security, compliance, and data residency requirements.
• API Design & Governance: Define and enforce API design standards (OpenAPI 3.1, spec-first, versioning, deprecation); own the 25+ FastAPI endpoints, request/response schema evolution, and backward compatibility guarantees.
• Data Architecture: Own the PostgreSQL schema for extraction metadata (holes, dimensions, GD&T, title blocks, notes, extraction traces, batches); design indexing strategies, multi-tenant isolation, and efficient querying patterns for the document review and dataset export workflows.
• Security & Compliance: Implement secure service-to-service communication, secrets management via AWS Secrets Manager, IAM role-based access for Bedrock and S3, and CORS/auth policies
• Observability & Reliability: Instrument the platform with structured logging, distributed tracing (AWS X-Ray / OpenTelemetry), and CloudWatch alarms; define SLOs for pipeline throughput, LLM call latency, and extraction accuracy.
• Engineering Leadership: Mentor senior engineers, conduct architecture and design reviews, set coding standards, and drive the technical roadmap for the enterprise.

REQUIRED SKILLS & EXPERIENCE

• 12+ years of backend engineering experience with a strong focus on distributed systems and production-grade platform design.
• Expert-level Python proficiency; deep hands-on experience with FastAPI, SQLAlchemy, Pydantic v2, and async/concurrent programming (asyncio, ThreadPoolExecutor).
• Proven experience designing and operating microservices at scale — including service decomposition, inter-service communication, and failure isolation strategies.
• Strong PostgreSQL expertise: schema design, indexing, query optimization, multi-tenant isolation (RLS or schema-per-tenant), and migration management.
• Hands-on AWS production experience: EC2/ECS, RDS, S3, ALB, IAM, Secrets Manager, CloudWatch, and ideally Amazon Bedrock.