Senior Analytics Engineer

Imagine Worldwide believes that every child should be empowered with the basic right to literacy and numeracy. They provide an innovative education technology solution to address the global literacy/numeracy learning crisis using the onebillion application and various toolkits.

The Senior Analytics Engineer builds, runs, and improves Imagine’s analytics platform, including infrastructure, pipelines, and core data models. The role involves scaling the platform as data volumes grow, managing costs, and ensuring performance without sacrificing reliability.

The platform processes data from over 1,500 schools across Malawi, Tanzania, and Sierra Leone, generating billions of records from tablet-based learning activities. These records are enriched with program context, such as school enrollment, device allocations, and operational data.

The role reports to the Director of Analytics, who sets the platform’s architecture and technical standards. The engineer must take initiative, translate architectural direction into working systems, and propose evidence-based improvements. As the engineer gains context, they will take increasing ownership of the platform’s day-to-day operations and technical decisions.

Key Responsibilities

The responsibilities are divided into four pillars: infrastructure, pipelines, orchestration, and reliability.

Data Platform and Infrastructure

The analytics platform runs on AWS and ClickHouse Cloud, with infrastructure managed as code.

• Build and maintain the platform’s AWS infrastructure, including S3, EC2, Glue Data Catalog, Lake Formation, IAM, and Secrets Manager.
• Manage infrastructure as code using Terraform, ensuring environments are reproducible, version-controlled, and reviewable.
• Administer the ClickHouse Cloud warehouse, including service configuration, user and role management, and monitoring of performance and capacity.
• Host and maintain self-hosted tools, including Apache Superset for BI and Apache Airflow for orchestration.
• Contribute to architectural decisions and trade-offs for platform evolution.

Large-Scale Data Processing and Pipelines

Data is ingested from application databases, operational systems, and field data tools, landing in an Apache Iceberg data lake on S3. Spark-based processing on AWS Glue handles heavy workloads before data moves into ClickHouse.

• Build and maintain ingestion pipelines using AWS Glue, Airbyte, and AWS Lambda.
• Implement incremental loading patterns for efficient and recoverable data processing.
• Develop and optimize Spark jobs for processing billions of device events into learning sessions and aggregates.
• Model program context such as schools, terms, device allocations, and enrollment.
• Design and maintain Iceberg tables for performance and cost optimization.
• Validate data at ingestion and surface quality issues early.
• Evaluate new processing approaches, including moving work upstream and assessing streaming.

Orchestration and Transformation

Apache Airflow orchestrates pipelines, and dbt transforms data in ClickHouse into models for reporting and analysis.

• Build and maintain Airflow workflows for ingestion, processing, and transformation.
• Develop and maintain dbt staging and core models in ClickHouse.
• Implement incremental models for high-volume tables.
• Write and maintain dbt tests and model contracts for core datasets.
• Document core models and their lineage.
• Collaborate with the Senior BI & Data Analyst and Data Scientist on datasets.

Reliability, Cost, and Engineering Practice

Ensure the platform runs predictably and is understood by the team.

• Build and maintain CI/CD pipelines in GitHub Actions for infrastructure, pipelines, and dbt code.
• Implement monitoring and alerting across pipelines, warehouse, and hosted tools.
• Respond to platform incidents and data issues, conduct root cause analysis.
• Apply and maintain access controls and security practices.
• Monitor platform costs and optimize workloads.
• Maintain backup and recovery procedures.
• Participate in code review and write clear technical documentation.