Observability Engineer

ClickHouse is looking for an experienced engineer to join our Observability organization. We are hiring across two closely aligned teams: Observability Platform and Internal Observability.

Together, these teams build and operate the systems behind ClickHouse’s internal observability and ClickStack Cloud. Our platforms process trillions of events per day, sustaining throughput in the hundreds of millions of events per second. The experience we gain operating observability at ClickHouse scale feeds directly back into the platform and product we provide to customers.

The Observability Platform team builds shared systems for telemetry ingestion, durable buffering, processing, storage, autoscaling, and service provisioning. The Internal Observability team operates ClickHouse’s company-wide observability platform and works with engineering teams to improve reliability, debugging, and operational efficiency.

This is a software engineering role at the intersection of distributed systems, cloud infrastructure, and production operations. You will build systems, operate what you build, respond to incidents, and turn recurring operational problems into durable software and automation.

Your background may be in backend engineering, infrastructure, SRE, or systems engineering. What matters most is your ability to solve ambiguous production problems, make sound engineering tradeoffs, and take ownership from design through operation.

What you’ll do

• Design, build, and operate distributed systems that ingest, process, and store telemetry at very high scale.
• Own the reliability, performance, capacity, and cost-efficiency of telemetry pipelines and storage systems.
• Participate in the on-call rotation, help resolve production incidents, and drive root-cause fixes through to completion.
• Build software and automation that eliminate repetitive operational work and make the platform easier to operate.
• Identify architectural bottlenecks and help shape the roadmap for the next stage of scale.
• Work closely with product, infrastructure, and service teams across ClickHouse.
• Contribute to architecture and design reviews and help raise engineering quality across the team.

How you work

• You take ownership and proactively improve the systems around you.
• You are comfortable debugging unfamiliar distributed systems in production.
• You make pragmatic tradeoffs between reliability, performance, delivery speed, and cost.
• You communicate clearly in a remote, async-friendly environment.
• You prefer incremental delivery: ship a focused solution, validate it in production, and improve it based on evidence.
• You address the underlying cause of operational problems rather than repeatedly treating the symptoms.

Technical experience

• 5+ years of experience building and operating production systems at scale.
• Strong proficiency in Go.
• Experience building and operating services on Kubernetes.
• Experience with infrastructure-as-code and GitOps tooling such as Terraform, Helm, and Argo CD.
• Production experience with at least one major cloud provider: AWS, GCP, or Azure.
• Hands-on experience with telemetry systems such as OpenTelemetry, Prometheus, Grafana, or comparable technologies.

Bonus points

• Experience with ClickHouse.
• Experience with high-throughput ingestion, streaming, queueing, or storage systems.
• Experience building multi-tenant cloud services.
• Experience optimizing infrastructure for both performance and cost.
• Experience with TypeScript.