Senior Data Generalist
Core
Build and maintain batch and streaming data pipelines, optimize data processing jobs, and design validations for data quality and freshness.
Role type
Senior IC data engineer (batch/streaming)
Builds
Production data pipelines, reusable datasets, and optimized data processing jobs
Domain
Data engineering, distributed data systems, data lakes
Required skills
Python, SQL, PySpark, Spark SQL, Spark Streaming, Parquet, Iceberg, batch processing, stream processing, query optimization, data lake architecture
Preferred skills
Flink, ML training/inference pipelines, ranking/recommendations/embeddings, privacy-sensitive data systems, Rust
Technologies
PySpark, Spark SQL, Spark Streaming, Parquet, Iceberg, Flink, Rust
Responsibilities
Build and maintain batch and streaming data pipelines; Move production data into archival/data lake storage; Improve quality, reliability, and observability of derived datasets; Design validations for freshness, completeness, schema correctness, and data quality; Optimize queries and data processing jobs for correctness, latency, and cost; Prototype and integrate production experiments with systems
Seniority
Senior, hands-on IC