Data Engineer Consultant II
Core
Design, build, and operate scalable batch and streaming data pipelines to transform raw data into trusted, analytics-ready datasets for enterprise reporting and advanced data use cases.
Role type
Senior IC Data Engineer
Builds
Robust batch and streaming data solutions on modern cloud data platforms (Lakehouse architectures)
Domain
Insurance / Data Engineering
Deliverable
production ML models | product features
Required skills
Apache Spark, Python, SQL, Cloud data platforms (Data Lakes/Lakehouse), ETL/ELT, Data modeling, CI/CD, Distributed data processing
Preferred skills
Microsoft Fabric / One Lake, Real-time/event-driven processing, Data governance, Orchestration tools
Technologies
Apache Spark, Python, SQL, Microsoft Fabric, One Lake
Responsibilities
Design and maintain scalable batch and streaming data pipelines; Develop and optimize ETL/ELT workflows; Implement data modeling and transformation logic; Build and manage data processing workloads in Lakehouse platforms; Ensure data quality via validation checks and monitoring; Optimize Spark jobs for performance and cost; Manage data schemas and handle drift; Develop reusable frameworks and libraries; Implement CI/CD pipelines for data workloads; Monitor and troubleshoot pipeline failures; Partner with analytics engineers and data scientists to deliver curated datasets.
Seniority
Senior, hands-on IC
