QA Consultant

EDV Werke is looking for a QA Consultant

Working Model: Remote

Form of cooperation: B2B Contract

Tasks:

• Generative AI: Deploy and maintain solutions using RAG architecture (Vector Search + LLM).

• Azure Cloud: Implement, test, and manage models in Azure AI Studio or Azure OpenAI Service.

• Python Development: Write complex Python scripts for data processing, testing, and automation.

• QA & Automation: Define and execute evaluation metrics for LLMs, including “LLM-as-a-Judge.”

• DataOps / CI/CD: Build Azure DevOps pipelines to automate AI model evaluation and deployment.

• Model Evaluation: Assess LLM performance and troubleshoot unexpected outputs.

• Collaboration: Work with data engineers, AI researchers, and business stakeholders to define requirements.

• Documentation & Training: Prepare technical documentation, test plans, and end-user guides.

• Performance Optimization: Monitor AI system performance and optimize scripts and evaluation pipelines.

• Governance & Security: Apply access controls and ensure compliance with AI governance standards.

• Ad-Hoc Analysis: Deliver on-demand analyses of AI outputs to support decision-making.

• Innovation & Research: Explore emerging AI frameworks, multi-agent systems, and potential integrations.

Must-Have Skills:

• Generative AI: 1–2 years of hands-on experience; practical use of RAG architecture.

• Azure Cloud: 5+ years of experience; deployed/tested models in Azure AI Studio or Azure OpenAI Service.

• Python Coding: 7+ years; able to write complex scripts (Pandas, PyTest), not basic automation.

• QA & Automation: 8+ years; deep understanding of LLM evaluation metrics.

• DataOps / CI/CD: 5+ years; Azure DevOps pipelines for automated AI evaluations.

• Education: Technical degree (CS, Engineering, Math) or equivalent experience.

Nice-to-Have Skills:

• Generative AI: Experience with Agentic AI or multi-agent systems.

• Azure Cloud: Azure AI Search experience, including indexing, vector profiles, hybrid search.

• Python Coding: Experience with LangChain or Semantic Kernel for test harnesses.

• QA & Automation: Experience using Prompt Flow for evaluation pipelines.

• DataOps / CI/CD: Creating Golden Datasets for regression testing.

• Education: Master’s degree in Data Science or AI-related field.

Benefits:

• Competitive salary with performance-based bonuses.
• Opportunities for professional development and advancement.
• Dynamic and collaborative work environment.