Senior Data Engineer :Scala, Spark, Python, Hadoop, Databricks
Core
Design, build, and maintain scalable data pipelines and distributed data processing systems to quantify and reduce external total exposure in adverse market conditions for a global securities firm.
Role type
Senior Data Engineer (Financial Risk)
Builds
Scalable batch and streaming data pipelines, real-time data processing solutions, and risk computation platforms.
Domain
Financial Services / Capital Markets / Credit Risk
Deliverable
production ML models
Required skills
Scala, Python, Apache Spark, Databricks Unity Catalog, ETL, Data Warehousing, SQL, NoSQL, Cloud Platforms (Azure/GCP), Containerization (Docker/Kubernetes), Akka/Pekko, Kafka, REST, gRPC
Preferred skills
C#, CI/CD, Infrastructure as Code, Data Governance, Machine Learning workflows, Real-time analytics, Derivatives pricing
Technologies
Scala, Python, Apache Spark, Databricks, Azure ADLS, Dremio, Hadoop, Kafka, Hive, HBase, Akka/Pekko, Docker, Kubernetes, Azure, GCP
Responsibilities
Design and develop distributed data processing systems; Build and optimize large-scale data pipelines for batch and streaming data; Maintain real-time data processing solutions; Develop and maintain ETL processes; Ensure data quality, reliability, and integrity through automated testing and monitoring; Work with DevOps teams to deploy and manage CI/CD and data pipelines in a cloud environment.
Seniority
Senior, hands-on IC
