AI/ML Evaluation Specialist, Human Data
Core
Own operational strategy and continuous improvement of large-scale, multilingual human data programs for AI/ML features, focusing on data collection, curation, annotation, and human evaluation.
Role type
Senior IC ML Evaluation Specialist (Human Data Operations)
Builds
Scalable human-in-the-loop data pipelines, quality frameworks, and feedback systems for generative AI features.
Domain
Generative AI / Human-Centered AI / Data Operations
Deliverable
production ML models
Required skills
Program execution, statistical process controls, data pipeline design, cross-functional alignment, privacy compliance, workflow optimization, measurement framework design, vendor management, human alignment principles, quality audit design
Preferred skills
Data strategy for frontier AI, agentic GenAI systems, external workforce management, AI Safety principles
Technologies
Python, R, SQL
Responsibilities
Lead end-to-end execution of human data collection programs for multilingual, multimodal AI features; Design and own measurement frameworks for data collection initiatives including spend, speed, and inter-rater reliability; Build and implement human data quality frameworks with statistical process controls; Systematically analyze workflows to identify inefficiencies and design innovative agent- and machine-in-the-loop solutions; Partner with Legal, Privacy, and Security to ensure compliant data collection programs; Design data collection lifecycle with onboarding, calibration, and performance frameworks.
Seniority
Senior, hands-on IC
