Senior AI Training Platform Engineer (GKE, GPU, PyTorch, Ray Train)

What You Will Do:

Training Platform

• Collaborate with the client’s infrastructure team to optimize the GKE training platform, including GPU node pools, job queueing, scheduling, quota, and capacity management.
• Diagnose and resolve platform failures at the root cause (e.g., memory issues in controllers) rather than simply restarting them.
• Tune multi-node GPU networking (RDMA/GPUDirect, NCCL) to maximize training throughput.

Distributed Training

• Execute, scale, and debug multi-node distributed training using PyTorch (FSDP, DeepSpeed, Megatron, or NeMo-style) and Ray Train.
• Profile jobs to identify and eliminate bottlenecks across compute, memory, input pipelines, and interconnects.

Benchmarking and Evaluation

• Design and conduct performance benchmarks for the cluster and training stack, focusing on throughput, scaling efficiency, and cost per run.
• Develop and maintain model evaluation harnesses to measure changes in the platform or training recipes.
• Instrument training and the platform with telemetry and monitoring, ensuring end-to-end pipeline tests align with user workloads.

• Treat the platform as your own: proactively identify issues, address gaps, and propose solutions before they become critical. Keep the team informed of status, risks, and support the scientists who rely on the platform.
• Justify and defend design decisions to client engineers and scientists in detail.
• Stay effective as the client’s priorities and team structure evolve.

Your Technical Toolkit:

• Core Languages: Mastery of Python and shell scripting.
• AI/LLM Ecosystem: Extensive experience with Google Gemini, GPT-4, or LLaMA; deep knowledge of Prompt Engineering and Fine-tuning.
• Data & Search: Expertise in Vector Databases (Vertex AI Vector Search, pgvector) and implementing Semantic Search.
• Infrastructure: Hands-on experience with Google Cloud (Vertex AI) and scalable software architectures.
• Frameworks: Proficiency in LangChain, LlamaIndex, or similar orchestration layers.
• Mindset: Strong software engineering foundation—write clean, maintainable code and understand the full Software Development Life Cycle (SDLC).

Basic Qualifications:

• 5+ years of engineering experience, with recent hands-on work on GPU training infrastructure and large-scale training.

• Strong Python software engineering: Production-quality code, tests, and code reviews. Coding is central to this role, not an afterthought.

• GKE and GPU scheduling: Kubernetes on Google Cloud with GPU node pools, job queueing (Kueue or similar), and capacity/quota management.

• Distributed PyTorch and Ray Train: Has personally run and debugged multi-node training jobs.

• RDMA and GPU networking: Experience with GPUDirect/RDMA, NCCL tuning, and multi-node GB200 (or H200) topology.

• Benchmarking and evals: Has designed performance benchmarks and model evaluations, using them to make informed decisions.

• Deep enough understanding to explain how and why a system works without relying on AI coding tools.

• A track record of visible ownership: present, communicative, and accountable on a demanding team.

Personal Attributes:

Nice to Have

• Experience with healthcare or life-sciences data (genomics, clinical, biomedical) and working under HIPAA compliance.

• Observability and telemetry for training platforms: metrics, tracing, alerting, and end-to-end pipeline testing.