代码AI效果评测产品经理-抖音AI
Core
Build an evaluation system for code LLMs and Agents in real-world software engineering scenarios for Douyin's frontend, backend, and client-side development.
Role type
Product Manager for AI Evaluation Systems
Builds
Evaluation frameworks, metrics, and attribution analysis for code generation and agent tasks
Domain
Software Engineering + AI/LLM
Deliverable
production ML models
Required skills
Structured analysis, complex problem decomposition, data sensitivity, experimental design, log analysis, AI/LLM fundamentals
Preferred skills
Frontend/Backend/Client-side development experience, AI evaluation product experience, knowledge of latest research methodologies
Technologies
Code LLMs, Agents, A/B testing, Log analysis
Responsibilities
Define evaluation datasets, standards, and metrics; Collaborate with R&D and data science to identify defects and optimization opportunities; Iterate evaluation schemes based on industry/academic research and real developer needs
Seniority
Mid-Senior, hands-on IC
