AI Automation Engineer OR LLM Automation Engineer

AI/LLM Test Automation Engineer (SDET)Location: Bangalore (Hybrid preferred)Experience: 5-8 YearsJob SummaryWe are seeking a Senior Test Automation Engineer (SDET) with expertise in AI/LLM testing and Azure DevOps CI/CD to build robust automation frameworks for cutting-edge AI applications. The role combines deep programming skills (Java/Python), modern DevOps practices, and specialized LLM testing to ensure high-quality AI product delivery.Key ResponsibilitiesDesign, develop, and maintain automation frameworks using Java/Python for web, mobile, API, and backend testing.Create and manage YAML-based CI/CD pipelines in Azure DevOps for end-to-end testing workflows.Perform AI/LLM testing including prompt validation, content generation evaluation, model behavior analysis, and bias detection.Write and maintain BDD Cucumber feature files integrated with automation suites.Execute manual + automated testing across diverse application layers with focus on edge cases.Implement Git branching strategies, code reviews, and repository best practices.Track defects and manage test lifecycle using ServiceNow or similar tools.Conduct root-cause analysis, troubleshoot complex issues, and drive continuous quality improvements.Mandatory Skills & Experience✅ 5+ years SDET/Automation experience✅ Java/Python scripting for test frameworks (Selenium, REST Assured, Playwright)✅ Azure DevOps YAML pipelines (CI/CD end-to-end)✅ AI/LLM testing (prompt engineering, model validation, RAG testing)✅ Cucumber BDD (Gherkin feature files + step definitions)✅ Git (branching, PRs, GitFlow)✅ ServiceNow/Jira defect tracking✅ Manual + Automation testing (web/mobile/API/backend)Technical StackProgramming: Java, Python, JavaScriptCI/CD: Azure DevOps, YAML PipelinesTesting: Selenium, Playwright, REST Assured, PostmanBDD: Cucumber (Gherkin), JBehaveAI/ML: Prompt validation, LLM APIs (OpenAI, LangChain)Version Control: Git, GitHub/GitLabDefectTracking: ServiceNow, Jira, Azure BoardsPreferred QualificationsExposure to AI testing frameworks (LangSmith, Promptfoo)Experience with containerization (Docker) and KubernetesKnowledge of performance testing for AI workloadsAWS/GCP cloud testing experienceISTQB or relevant QA certificationsWhat We OfferWork on next-gen AI/LLM products with global impactModern tech stack with Azure-native DevOpsFlexible hybrid/remote work modelContinuous learning opportunities in AI testing