Backend Engineer (LLM Observability) at Honeycomb

About the Role

The LLM Observability (LLM O11y) team at Honeycomb builds the foundational layer that makes AI workloads observable. You’ll design how AI systems are represented in telemetry (spans, traces, token usage, cost, eval signals, etc.) and ensure data from diverse ecosystems (e.g., LangChain, OpenLLMetry, Braintrust) is normalized into a consistent OpenTelemetry-based model.

This team owns the infrastructure and data model that all downstream AI observability experiences rely on. Your work will power everything from cost analysis to anomaly detection. If you’re excited about defining and building the “source of truth” for how AI systems behave in production, this is the team for you.

Key Responsibilities

• Design, build, and deliver backend systems and APIs. Architect and implement APIs and backend services in Go, emphasizing clarity, maintainability, and backward compatibility. Proactively address cost, scale, and risk—anticipating technical issues and exploring tradeoffs.
• Build and maintain full-stack product features. Contribute frontend (React/TypeScript) and backend (Go) code to deliver robust product features. Ensure impactful features add value for customers, the team, and Honeycomb.
• Provide technical leadership. Define patterns and operational guidelines that raise the bar for the team. Plan, lead, and deliver complex, multi-team projects—including those with external dependencies—ensuring alignment and mentoring teammates.
• Collaborate across disciplines. Work closely with design and product partners in a product triad model, iterating quickly based on user feedback. Contribute to the team’s technical roadmap and shape quarterly product roadmaps.
• Support production software. Participate in on-call rotations to ensure reliable operations and address user pain points. Balance forward-looking feature work with production needs.
• Communicate clearly and multiply impact. Regular written and verbal communication, collaboration, and feedback are foundational. Mentor engineers, share knowledge, and delegate to help others grow.

Required Qualifications

• Deep experience designing, building, and maintaining backend systems. Substantial expertise in Go (or similar languages) with strong opinions about API design, error handling, and developer experience. Experience evaluating technical challenges.
• Frontend proficiency with React and TypeScript. Professional experience delivering frontend features and contributing to full-stack software, though backend systems are the primary focus.
• Experience leading complex technical projects. Planned and delivered projects spanning multiple teams or systems, navigating external dependencies and uncertainty. Balanced technical investment with business goals.
• Ownership and autonomy. Proactively identify problems, propose solutions, and own projects end-to-end—from planning through deployment and maintenance. Proactively tackle tech debt.
• A track record of making others better. Mentored engineers, led technical design reviews, and raised the quality bar for teams or domains. People seek guidance from you.
• A collaborative and flexible mindset. Comfortable working cross-functionally, pivoting priorities, and maintaining strong teamwork. Iterate based on customer interaction.
• Clear, open communication. Regular, thoughtful written and verbal communication, including feedback. Essential for engineering impact.
• Interest in observability and production culture. Supported software in production and has opinions on best practices. Bonus if you want to integrate experience into Honeycomb’s product.

Nice-to-Have

Experience with AI-assisted development. Proficiency with AI coding tools (e.g., Copilot, Claude Code, Cursor) or excitement to incorporate them into workflows.