Software Engineer
Core
Execute structured evaluation assignments simulating developer activities (version control, code review, CI/CD) to train next-generation AI systems.
Role type
Contractor Software Engineer (AI Evaluation)
Builds
AI training datasets and evaluation frameworks
Domain
Artificial Intelligence / Software Development
Deliverable
production ML models
Required skills
Git/GitHub workflows, CI/CD systems, Python or Bash scripting, REST APIs, OAuth, webhook integrations, enterprise SaaS platforms
Preferred skills
AI assistants/agent tools, Model Context Protocol (MCP), technical QA, data annotation
Technologies
GitHub Actions, Claude, ChatGPT, Slack, Google Workspace, Microsoft 365
Responsibilities
Execute structured evaluation assignments simulating authentic developer activities; Assess and grade the technical correctness of outputs; Design, seed, and maintain reproducible test environments; Configure, document, and troubleshoot integrations and authentication flows; Investigate and document undocumented product behaviors; Participate in calibration sessions to ensure consistent evaluation methods
Seniority
Mid-level (3+ years experience)
