Lead Data Engineer

You will work closely with architects, engineers, and client stakeholders to deliver modern cloud-based data solutions that support operational, analytical, and AI-driven use cases. The role combines technical leadership with hands-on engineering, helping clients modernise their data estates while ensuring solutions are secure, reliable, and scalable.

Working across multiple projects, you will lead the development of batch and real-time data pipelines, support architectural decision-making, and help establish engineering best practices across delivery teams.

Strong experience with real-time streaming technologies is essential.

Key Responsibilities

• Lead the design, development, and deployment of secure, scalable data pipelines and data platform components.
• Design and implement batch and real-time streaming solutions using modern event-driven architectures.
• Own technical design decisions across projects, balancing scalability, security, performance, and delivery requirements.
• Lead technical workshops, discovery sessions, and architecture discussions with client stakeholders.
• Assess existing client data platforms and design practical approaches for modernisation, migration, and integration.
• Build and maintain data ingestion, transformation, and integration capabilities across a variety of data sources.
• Develop cloud-native data platforms using AWS services and open-source technologies.
• Ensure data quality, governance, lineage, and security requirements are embedded throughout the data lifecycle.
• Drive engineering best practices including code reviews, automated testing, CI/CD, and Infrastructure as Code.
• Support multiple client engagements, helping delivery teams maintain consistent engineering quality and standards.
• Mentor and support engineers across project teams, providing technical guidance and leadership.
• Perform data modelling, performance tuning, and root cause analysis to resolve complex data and platform issues.

Essential Skills & Experience

Real-Time Streaming & Data Engineering

• Strong commercial experience designing and implementing real-time streaming data platforms.
• Hands-on experience with technologies such as Apache Kafka, AWS Kinesis, Apache Flink, Spark Streaming, Apache Pulsar, or similar event-driven technologies.
• Experience building high-volume, low-latency, resilient data pipelines in production environments.
• Strong understanding of modern data engineering patterns, including batch, streaming, and hybrid architectures.

Data Engineering & Databases

• Extensive experience developing and operating enterprise-scale data platforms.
• Strong SQL skills and experience designing data models for analytical and operational workloads.
• Proficiency in Python or other modern programming languages used for data engineering.
• Experience configuring, tuning, and optimising relational and NoSQL databases.

Cloud & Software Engineering

• Strong experience working with AWS cloud services, including EC2, RDS, Lambda, Athena, Glue, S3, and IAM.
• Experience working within Agile delivery environments.
• Strong understanding of Test-Driven Development (TDD) and automated testing practices.
• Experience implementing CI/CD pipelines and engineering automation.
• Strong understanding of Infrastructure as Code principles.
• Experience using version control systems such as Git.

Regulated Environments

• Experience delivering technology solutions within UK Public Sector, Utilities, Energy, Water, Financial Services, or other highly regulated industries.
• Understanding of security, governance, and compliance considerations within regulated environments.

Desirable Skills

• Experience with Terraform.
• Experience with Docker and Kubernetes.
• Knowledge of Ansible and Packer.
• Linux and Windows administration experience.
• Experience supporting AI, machine learning, or advanced analytics platforms.
• Experience working on programmes requiring UK Government security clearance.

What We're Looking For

• Strong technical leadership and problem-solving skills.
• Ability to communicate effectively with both technical and non-technical stakeholders.
• Experience working in complex, regulated environments.
• A collaborative approach and willingness to support and mentor others.
• A focus on delivering practical solutions that create value for clients.
• A passion for building reliable, scalable, and secure data platforms.

Education

A Bachelor’s degree in computer science, Software Engineering, Data Engineering, Mathematics, or a related technical discipline, or equivalent practical experience.