How to Test an AI Agent Before Production: Evaluations, Prompt Injection and Release Gates

To test an AI agent before production, build a test set from real questions with known good answers, score every run with quality, safety and agent evaluators, attack the agent with prompt injection through the documents and tools it reads, and block any release that falls below a threshold the business agreed in advance. Run that suite in your pipeline on every change to the prompt, model or tools, then keep evaluating sampled production traffic after launch.

Most pilots we review skip this. Someone asks the agent a dozen questions, the answers look right, and the go-live decision rests on that. It works until the first model upgrade or instruction edit, when nobody can say whether the agent got better or worse. Evaluation is one of the six dimensions on our Agentic AI Readiness Scorecard, and it is the one that turns a go-live meeting from a debate into a check.

Why spot checks don’t survive the first change

An agent’s behaviour depends on its instructions, model version, knowledge sources and tools. Change any one and answers shift in places nobody looked. A hand check covers the questions the builder remembered to ask, usually the ones the agent was tuned on.

A repeatable suite gives you a baseline to compare against and evidence an approver can sign. We build the layers below in this order, because each depends on the one before it.

The five layers, in the order to build them

1. A test set from real questions with known good answers

Start with the questions people actually ask. Pull them from help desk tickets, email inboxes or the pilot’s chat logs, and have the business owner write or approve the expected answer for each. Include questions the agent should refuse or hand off, not just ones it should answer.

Everything else runs on this test set. Keep it in source control next to the agent’s configuration, and add every production failure to it once the agent is live.

For Copilot Studio agents, the built-in agent evaluation takes this set directly. You can import test cases from a spreadsheet, write them by hand or generate them with AI, with up to 100 test cases per single-response test set. Results stay in Copilot Studio for 89 days, so export them if the approval record needs to last longer.

2. Quality evaluators

Quality evaluators score whether the answer is right and useful. In Microsoft Foundry, the general and retrieval evaluators include groundedness, relevance and coherence, and they use a model as the judge. Copilot Studio offers equivalent test methods: general quality (relevance, groundedness, completeness and abstention), compare meaning against your expected answer, keyword match, exact match and custom criteria you write yourself.

One detail catches teams out. Copilot Studio’s general quality method penalizes a response where the agent declines to answer. If some test cases expect a refusal, score those with a custom test method that defines the correct refusal, or the scores will punish the behaviour you want.

3. Safety and risk evaluators

Safety evaluators check whether the agent produces harmful content. Foundry’s risk and safety evaluators cover hate and unfairness, sexual content, violence, self-harm, protected material, code vulnerabilities and ungrounded inferences about people. They report a defect rate across the dataset. For agents, two further evaluators, prohibited actions and sensitive data leakage, are in preview.

Copilot Studio includes a content safety test method for the four harm categories, with a severity threshold you set per category. For Canadian teams, the Foundry batch evaluations and most safety evaluators are available in Canada Central and Canada East, so this layer can run without moving test data out of the country.

4. Prompt injection, direct and indirect

Direct injection is a user typing instructions meant to override the system prompt. Indirect injection is more dangerous for agents: the attack sits inside a document, email or tool result the agent reads. An agent that can send email or update records turns a hidden line in a PDF into an action.

Test both. In Foundry, the AI Red Teaming Agent runs automated attacks against Foundry prompt and container agents, including an indirect jailbreak strategy that plants attacks in mock tool outputs, and reports an attack success rate. It has limits worth knowing. Agentic risk categories run only as cloud red teaming, function tool calls aren’t supported yet, and cloud red teaming runs in a short list of regions, none of them in Canada. Use synthetic data for those runs.

For Copilot Studio agents, write the adversarial cases yourself. Put a test document in a knowledge source with an embedded instruction, then confirm the agent ignores it. Microsoft states that Copilot Studio custom agents block user and cross-domain prompt injection at runtime by default, and a test proves it holds for your configuration.

At runtime, Prompt Shields in Azure AI Content Safety screens for both attack types. In Foundry you add it as a guardrail control on the model deployment or agent, and document attack detection runs at the user input and tool response points. Spotlighting, which marks third-party content as lower trust, is in preview.

5. Tool calls and agent behaviour

An agent can give a good final answer after calling the wrong tool, or pass a wrong parameter to the right one. Foundry’s agent evaluators check the steps, not just the result. They include tool call accuracy, tool selection, tool input accuracy, tool output utilization and task navigation efficiency against an expected sequence, plus intent resolution, task adherence and task completion, which are in preview.

Check the supported tools before relying on them. Microsoft advises against using tool call accuracy, groundedness and several related evaluators when the conversation includes Azure AI Search, SharePoint grounding, Bing grounding, Code Interpreter or web search, because support for those tools is limited. Copilot Studio’s tool use test method checks whether the expected tools or topics were called.

The tests show what the agent tries to do. The identity controls in our post on Entra Agent ID limit what it can do when a test misses something.

Which test catches what

Test type	What it catches	When it runs
Test set with expected answers	Wrong or missing answers, regressions after a change	Every change to prompt, model, knowledge or tools
Quality evaluators	Ungrounded, irrelevant or incomplete responses	Every release candidate
Safety and risk evaluators	Harmful content, protected material, data leakage	Every release candidate
Direct prompt injection tests	Users overriding instructions or safety rules	Every release candidate, plus scheduled red teaming
Indirect prompt injection tests	Hidden instructions in documents, email and tool output	Before launch, after any new tool or data source, scheduled after launch
Tool call and behaviour evaluators	Wrong tool, wrong parameters, skipped steps, unsafe actions	Every change to tools or instructions
Continuous evaluation	Drift in live traffic the test set doesn’t cover	Sampled production traffic, ongoing

Setting the release threshold and wiring it into CI

Agree the threshold before the first run, with the business owner’s name on it. A threshold set after seeing the results tends to match them. Set it per layer: a minimum pass rate on the test set, a maximum defect rate on safety, a maximum attack success rate on injection, and zero failures on the cases marked critical, such as anything that moves money or deletes data.

Then automate it. Microsoft publishes a GitHub Action and an Azure DevOps extension, both in preview, that invoke a Foundry agent version with your test queries, run the evaluators you list and report scores with confidence intervals. Pass two agent versions and the report shows whether the difference is statistically meaningful. That matters because model-judged scores vary between runs, and a small drop may be noise.

Microsoft recommends not running evaluations on every commit, to control cost. Trigger the suite on changes to instructions, model deployment, tools or knowledge, and on release branches. For Copilot Studio, the Power Platform API and connectors can trigger evaluation runs against an existing test set from a pipeline or a flow.

If you are choosing between the two platforms, testing depth is part of the decision. We compare them in Microsoft Foundry vs Copilot Studio.

What to monitor after launch

Pre-release tests cover the questions you thought of. In Foundry, set up recurring evaluations from the Evaluations tab: continuous evaluation samples live traffic as it arrives, scheduled evaluation reruns your golden dataset to detect drift, and red teaming can be scheduled to run against the deployed agent. The Agent Monitoring Dashboard shows token usage, latency, run success rate, evaluation scores and red teaming results. It reads from the Application Insights resource connected to the project, so connect that before launch.

Review failed samples weekly in the first months, add each real failure to the test set, and rerun the full suite whenever the model version behind your deployment is updated or retired, even if your own code didn’t change.

Before the go-live meeting

Bring four things: the test set and who approved it, the scores for each layer against the agreed threshold, the injection results with any open findings, and the monitoring plan with a named owner. If any of those is missing, the honest answer is a sandboxed pilot, not production. The six dimensions in our agentic AI readiness checklist show where evaluation fits alongside identity, privacy and cost.

Techrupt’s AI consulting team in Vancouver builds evaluation suites and release gates for Foundry and Copilot Studio agents. If your pilot has a go-live date and no release threshold yet, book a consultation and we’ll work through the test plan with you.