Principal Backend QA Engineer (AI)
Core
Design automated test frameworks and validate non-deterministic AI systems (agents, chatbots) using LLM-as-a-Judge techniques and API automation.
Role type
Senior IC backend QA engineer (AI validation)
Builds
Automated test suites for AI agents and backend microservices
Domain
AI/ML engineering and backend infrastructure
Deliverable
production ML models
Required skills
API test automation, Karate Framework, Postman, LLM evaluation (LLM as a Judge), non-deterministic system testing, source code analysis, microservices architecture validation, shift-left testing, OpenAPI/Swagger specification analysis
Preferred skills
Agent Development Kit (ADK), MCP applications, contract testing
Technologies
Karate, Postman, OpenAPI, Swagger, ADK, MCP
Responsibilities
Analyze OpenAPI/Swagger specs to design comprehensive test plans; Test and evaluate non-deterministic AI outputs and internal AI agents; Inspect source code and upstream services to identify critical dependencies; Configure local development environments for early-stage testing; Design scalable test automation strategies integrating AI capabilities.
