Staff Software Engineer (Platform Engineering), National Environment Agency
Core
Lead the build-out of NEA's Weather Data Lake, a Databricks-based analytics platform serving as the single source of truth for meteorological and climate data.
Role type
Senior Data & Platform Engineer
Builds
Weather Data Lake on Databricks (Unity Catalog, Delta Lake, Structured Streaming) running on ADLSv2 within GCC 2.0
Domain
Meteorology, Climate Data, Public Sector Digital Infrastructure
Required skills
Databricks, PySpark, Terraform, Azure (ADLSv2, GCC 2.0), Data Pipeline Architecture, Data Governance, CI/CD, Mentorship
Preferred skills
Unity Catalog, Delta Lake, CDC patterns, Serverless compute, Lakebase, API distribution
Technologies
Databricks, Azure, Terraform, PySpark, Unity Catalog, Delta Lake, ADLSv2, GCC 2.0
Responsibilities
Design and build the Weather Data Lake; Architect production data pipelines ingesting from GTS, ADES, AirViro, IWXXM, and WIS 2.0 sources; Own the data quality framework; Own the infrastructure as code layer; Define integration contracts between Azure Landing Zone and Weather Data Lake; Mentor junior engineers; Partner with MSS to translate domain requirements into technical designs; Implement data governance using Unity Catalog; Evaluate platform evolution opportunities.
Seniority
Senior, hands-on IC with mentorship responsibilities