Senior Data Engineer, Observability Engineering
Core
Design, build, and operate scalable Databricks-based data pipelines and lakehouse solutions to ingest, transform, and govern high-volume telemetry, infrastructure, and security data for enterprise observability and operational intelligence.
Role type
Senior individual contributor data engineer (observability & security data)
Builds
Enterprise observability lakehouse, resilient ingestion frameworks, and trusted data products for security analytics
Domain
Healthcare technology, observability, infrastructure security, and data engineering
Required skills
Python, PySpark, SQL, distributed data processing, Databricks, Delta Lake, Unity Catalog, Structured Streaming, data governance, CI/CD, cloud security principles
Preferred skills
OCSF framework familiarity, Databricks Asset Bundles, GitHub Actions, Azure DevOps, near real-time ingestion, observability platform support
Technologies
Databricks, PySpark, Delta Lake, Unity Catalog, Structured Streaming, Azure DevOps, GitHub Actions
Responsibilities
Design and deploy scalable Databricks pipelines for observability and security data; optimize PySpark workloads and streaming jobs; partner with security and infrastructure teams to onboard data sources and establish governance; contribute to engineering standards, code reviews, and platform optimization; promote best practices for data governance and cost-efficient operations
Seniority
Senior, hands-on IC