General Information

Position
Agentic AI Technical Test Lead | Poland
Work arrangement
Plný pracovní úvazek
City
Gdaňsk, Poznaň, Varšava, Vratislav
Country
Polsko
Department
Consulting
Team
Engineering, AI & Data
Area of interest
Consulting, IT - Services
Way of work
Hybridní model

Description & Requirements

Who we are looking for

We are looking for an experienced Agentic AI Technical Test Lead to define, build and execute testing approaches for AI agent solutions in a consulting environment.
The ideal candidate combines strong Quality Engineering expertise with hands-on understanding of LLMs, RAG, AI agents, evaluation frameworks and enterprise delivery. This person should be proactive, client-facing, passionate about AI, and able to set standards while also supporting execution, presentations and business development.

🎯Required skills and experience

  • Strong background in Quality Engineering, test automation, AI/ML testing, software testing or technical test leadership
  • Practical understanding of generative AI, LLM-based applications, RAG and Agentic AI solutions
  • Ability to define test strategies, quality gates and evaluation approaches for AI solutions
  • Experience with AI quality dimensions such as accuracy, pecision / recall, relevance, groundedness, hallucination risk, robustness, consistency, safety and compliance
  • Ability to design evaluations for prompts, AI-generated responses, agent workflows, tool-calling, multi-step reasoning, RAG pipelines and human-in-the-loop review
  • Good technical understanding of APIs, integrations, cloud platforms, CI/CD and solution architecture
  • Hands-on experience with Python or similar scripting/programming language
  • Excellent English, both written and spoken
  • Strong presentation and stakeholder management skills
  • Consulting mindset, ownership attitude and ability to support sales / pre-sales activities
  • Passion for AI and willingness to actively share knowledge across teams
  • Residence in the European Union

🎯Preferred experience

  • Hands-on experience testing LLM, RAG or Agentic AI solutions
  • Familiarity with modern AI evaluation and testing tools such as DeepEval, RAGAS, promptfoo, OpenAI Evals, Braintrust, LangSmith, TruLens, Giskard, MLflow Evaluation or Arize Phoenix
  • Experience with AI observability and monitoring platforms such as LangSmith, Langfuse, Arize Phoenix, MLflow, Weights & Biases Weave, Comet Opik, AgentOps, Datadog LLM Observability or Helicone
  • Familiarity with AI agent / orchestration frameworks such as LangGraph, LangChain, Microsoft Semantic Kernel, AutoGen, CrewAI or similar
  • Experience with RAG components, including vector databases, embeddings, document ingestion and retrieval evaluation
  • Understanding of Responsible AI, AI governance, privacy, security and regulatory expectations
  • Experience with DevOps, MLOps or LLMOps practices
  • Previous consulting, client-facing delivery or business development experience


Your future role

As an Agentic AI Technical Test Lead, you will be responsible for defining and leading the quality assurance approach for AI agent solutions delivered to enterprise clients.
You will create testing standards, prepare technical solutions, support proposals and presentations, and lead execution across AI testing workstreams.

Key responsibilities


🚀AI Testing Strategy & Standards

  • Define reusable testing standards for Agentic AI solutions
  • Create evaluation frameworks for prompts, RAG, agent workflows and tool-calling
  • Define quality gates, acceptance criteria, test metrics and reporting templates
  • Establish approaches for accuracy, precision, hallucination, robustness and safety evaluation
  • Build reusable assets, accelerators and best practices for AI testing

🚀AI Evaluation & Execution

  • Lead test design and execution for AI agents and LLM-based applications
  • Create golden datasets, evaluation scenarios and adversarial test cases
  • Validate agent reasoning, tool usage, API interactions and workflow orchestration
  • Support automated and human-in-the-loop evaluation processes
  • Analyze results and provide clear recommendations to technical and business stakeholders

🚀Technical Solution & Delivery Support

  • Design technical testing solutions for client engagements
  • Embed evaluation into CI/CD, DevOps, MLOps or LLMOps pipelines
  • Support observability, monitoring, feedback loops and production quality controls
  • Work with architects, AI engineers, data engineers and business teams
  • Identify quality risks and propose practical mitigation actions


🚀Consulting & Business Development

  • Support pre-sales, proposals, estimations and client presentations
  • Prepare technical materials explaining AI testing approaches
  • Facilitate workshops and discovery sessions
  • Act as a trusted advisor on AI quality, reliability and risk
  • Promote AI testing knowledge internally and externally


What we offer

  • Opportunity to shape testing standards for Agentic AI solutions
  • Work on innovative AI engagements for enterprise clients
  • Collaboration with AI, data, cloud, cybersecurity and consulting experts
  • Career path toward AI Quality Engineering Lead, AI Assurance Architect or AI Testing Offering Lead
  • Space to build thought leadership, accelerators and reusable AI testing assets
  • A collaborative environment focused on innovation, ownership and knowledge sharing


About Deloitte
https://www.deloitte.com/pl/pl/careers/deloitte-life/culture-book-en.html

In Deloitte we support leading global organizations in optimizing and transforming their organizations. This also relates to their finance function. Our consultants offer insights, capabilities, and solutions that change the way these organizations operate and make people-related decisions.
We are currently building in Poland a dedicated team which will focus on advisory in the area of Finance & Performance. The Team will advise our Clients in complex issues relating to i.a. Reporting, Budgeting, Controlling processes. We are looking for high-energy, committed and ambitious people to join our advisory practice. Next to advising on most sophisticated and challenging issues relating to the above mentioned areas, our Consulting professionals will also advise on potential solutions leading to optimisation of the above mentioned processes. This will also include applying the newest technologies.
 Are you ready to join our team and help our clients imagine, deliver, and run their future?


#LI-EJ1

< Zpět na vyhledávání pracovních pozic