Pre-Training Text Data
Core
Designing novel data collection strategies and building scalable pipelines to create, curate, and analyze high-quality text datasets for training and evaluating AI models.
Role type
Senior Data Engineer (AI/ML Data Infrastructure)
Builds
Scalable data ingestion, preprocessing, filtering, and annotation pipelines for text data.
Domain
Artificial Intelligence / Machine Learning / Data Engineering
Deliverable
production ML models
Required skills
Python, Pandas, NumPy, statistics, exploratory data analysis, Spark, Ray, Apache Beam, data pipeline architecture, dataset auditing, versioning, privacy compliance
Preferred skills
N/A
Technologies
Python, Pandas, NumPy, Spark, Ray, Apache Beam
Responsibilities
Develop novel data collection strategies; Improve dataset quality and integrity; Create high-quality datasets for training and evaluation; Run experiments on new datasets (data ablations) to assess their impact; Develop and maintain scalable data pipelines for text data ingestion, preprocessing, filtering, and annotation; Analyze real-world text datasets to assess quality, diversity, relevance, and identify areas for improvement; Build lightweight tools and workflows for dataset auditing, visualization, and versioning; Collaborate with Safety, Ethics, and Governance teams to ensure datasets meet standards for quality, privacy, and responsible AI practices.
Seniority
Senior, hands-on IC
