.png?1722368836){kind=logo}
Senior Data Engineer
Job Description
Key Skills Required
Master these to land this role
Want to know if you're a match for this job?
540 is seeking a Senior Data Engineer to support a mission-critical federal health data modernization initiative. The team is bringing together information from diverse defense and federal systems to build scalable data pipelines, APIs, and data products that improve how critical health data is integrated, accessed, and used.
In this role, you’ll serve as a senior technical contributor, delivering scalable, production-ready data capabilities while establishing reusable patterns for data ingestion, integration, and delivery. You’ll solve complex data challenges, guide engineering teams, and help shape the technical foundation of a growing data platform ecosystem supporting federal health missions.
HOW YOU’LL DRIVE IMPACT
- Design, develop, and maintain scalable, production-ready data pipelines and data products using Spark (Python/SQL) in a Databricks environment
- Lead the integration and transformation of complex data from diverse DoW and federal health systems and sources into reliable, reusable data products
- Design scalable approaches for data ingestion, integration, and exchange, including API-based integrations and services
- Establish and promote reusable data engineering patterns, standards, and best practices that improve consistency, scalability, and maintainability across data products
- Provide technical guidance on data architecture, pipeline design, data modeling, integration approaches, and engineering practices
- Monitor, troubleshoot, and optimize production workflows and data pipelines, identifying performance, reliability, and scalability improvements
- Define and implement data validation, quality, and governance practices that improve the reliability and usability of data products
- Troubleshoot complex technical and data integration challenges, identify root causes, and drive sustainable solutions
- Collaborate with engineers, architects, analysts, and customer stakeholders to translate complex data needs into scalable technical solutions
- Provide technical guidance and mentorship to other engineers, helping teams navigate complex or unfamiliar technical challenges
- Proactively identify opportunities to improve engineering tools, processes, and patterns and help drive their adoption across the team
- Take ownership of complex technical areas and help maintain engineering quality, consistency, and cohesion as the platform and portfolio of data products grow
REQUIRED SKILLS & EXPERIENCE
- 10+ years of data engineering, software engineering, or related technical experience
- Extensive hands-on experience designing, building, and operating production data pipelines and data products
- Advanced proficiency with Python and SQL
- Strong experience with Apache Spark and distributed data processing
- Experience working with Databricks or similar modern data platforms
- Experience designing and maintaining ETL/ELT processes for complex, large-scale datasets
- Experience integrating data across disparate systems and consuming or developing API-based data integrations
- Strong understanding of data modeling, data architecture, data quality, and data governance principles
- Experience troubleshooting and optimizing complex production data pipelines for performance, reliability, and scalability
- Experience working with Git-based development workflows and modern software engineering practices
- Experience working in terminal / command-line environments
- Demonstrated experience providing technical guidance, mentoring engineers, and influencing engineering practices
- Strong client and stakeholder communication skills, with the ability to translate technical concepts and recommendations for both technical and non-technical audiences
- Ability to independently navigate ambiguity, identify technical risks, and drive complex engineering challenges toward resolution
- Experience spotting security, privacy and compliance issues and working with security/compliance/legal stakeholders
NICE TO HAVE SKILLS & EXPERIENCE
- AWS cloud experience
- Experience working with very large datasets, including datasets with billions of records
- Experience with Palantir Foundry
- GitLab experience
- Experience working with Advana or similar DoW data environments
- Experience working with federal health, financial, or other regulated and sensitive data
- Experience using AI/ML to accelerate work, including automating routine tasks, accelerating development and debugging
How would you rate this job post?
See what other professionals think about this role.
Similar Opportunities
More Openings at 540
540 is a forward-thinking company that embodies innovation and progress. With a name that is both simple and profound, 540 is poised to make a significant impact in the industry. As a company that values creativity and ingenuity, 540 is constantly pushing the boundaries of what is possible. With a strong foundation in technology and a passion for excellence, 540 is well-equipped to tackle the complex challenges of the modern world. The company's commitment to innovation and customer satisfaction has earned it a reputation as a leader in its field. Whether through cutting-edge products or exceptional services, 540 is dedicated to making a positive difference in the lives of its customers and the wider community. With a talented team of experts and a vision for the future, 540 is an exciting and dynamic company that is sure to continue growing and evolving in the years to come.
Safety First
- Never pay for a job application.
- Do not share sensitive bank info.
- Verify the client before starting work.