Data Engineer with using Apache Spark, Python, Scala.
Core
Design, develop, and maintain robust data infrastructure and pipelines, optimizing data workflows and ensuring system reliability.
Role type
Data Engineer (Infrastructure & Pipelines)
Builds
Scalable data processing pipelines, ingestion/extraction processes, and data infrastructure.
Domain
Data Engineering / Big Data
Required skills
Apache Spark, Python, Scala, Kafka, Docker, Kubernetes, SQL, ETL, Data Warehousing, Performance Tuning, Unit Testing, BDD
Technologies
Apache Spark, Kafka, Docker, Kubernetes, NetApp S3, pytest, junit, Cucumber, Lettuce
Responsibilities
Design and optimize data processing pipelines; Develop and maintain data ingestion and streaming workflows; Implement performance tuning for Spark jobs and SQL queries; Collaborate with DevOps to deploy infrastructure on Kubernetes; Containerize applications using Docker; Write and execute test scenarios using BDD tools.
Seniority
Mid-Senior, hands-on IC