Principal AI Architect - M365 IC3 Team (Intelligent Conversation and Communications Cloud)
Core
Define technical vision and architecture for AI, LLM, RAG, and agent evaluation systems; design evaluation frameworks to assess product quality across the lifecycle from pre-implementation to post-ship.
Role type
Principal AI Architect (Evaluation & Architecture)
Builds
Evaluation frameworks, dashboards, scorecards, quality reports, product-health views, and systems connecting telemetry, offline evals, human judgment, and automated evals.
Domain
Enterprise AI, LLMs, RAG, Agentic Systems, Product Quality
Required skills
LLM evaluation design, RAG evaluation, agentic system behavior analysis, architecture of complex multi-component systems, CI/CD integration for evals, statistical analysis of nondeterministic systems, technical strategy definition, cross-functional leadership, coding and technical design, product lifecycle management
Preferred skills
Experience with enterprise search, grounding, indexing, ranking, permissions, and relevance systems; experience with red teaming, synthetic data, model monitoring, and responsible AI practices
Technologies
LLMs, RAG, agentic systems, ML pipelines, experimentation systems, CI/CD pipelines, data visualization systems
Responsibilities
Design evaluation frameworks for pre-implementation, development, launch, and post-ship phases; identify behavioral gaps, failure modes, and model limitations; partner with engineering and data science teams to integrate ML/DS methods into product codebases; integrate evals into build pipelines and release gates; mentor senior engineers on building reliable evaluation infrastructure; drive architecture decisions across components, services, and data pipelines; work with product leaders to prioritize evaluation investments.
Seniority
Principal, strategy & mentorship