Your recommended path
For a result of Ready for a production pilot.
- 1
Weeks 1 to 2
Harden the release
Close any remaining items in your plan, set up monitoring, and agree the evaluation score a release must reach.
- 2
Weeks 3 to 8
Run the production pilot
Give real users a limited scope, and review evaluation results, cost and incidents every week.
- 3
Weeks 8 to 10
Decide
Compare results with the baseline, then widen the scope, add tools, or stop.
Plan the production pilotTimings are typical for a single, well-scoped use case and depend on how quickly privacy and security review can meet.
Timings are typical for a single, well-scoped use case and depend on how quickly privacy and security review can meet.
For a result of Ready for a sandboxed pilot.
- 1
Weeks 1 to 4
Build in a sandbox
Use synthetic or de-identified data in a separate environment, with the agent on its own identity from day one.
- 2
In parallel
Close the gaps
Work through the first column of your action plan, usually the privacy assessment, data access and evaluations.
- 3
Weeks 5 to 8
Move to real data
Once identity and privacy pass, rerun your evaluations on real data with a small group of users.
Get help with the pilotTimings are typical for a single, well-scoped use case and depend on how quickly privacy and security review can meet.
Timings are typical for a single, well-scoped use case and depend on how quickly privacy and security review can meet.
For a result of Fix the foundations first.
- 1
Weeks 1 to 2
Decide the basics
Agree the use case, its business owner and a baseline, and brief your privacy officer.
- 2
Weeks 2 to 4
Set up identity and data access
Give the agent its own identity and a sponsor, limit it to the sources it needs, and estimate the running cost.
- 3
Week 4
Rescore
Retake the scorecard. When identity and privacy pass, start a sandboxed pilot.
Talk through your use caseTimings are typical for a single, well-scoped use case and depend on how quickly privacy and security review can meet.
Timings are typical for a single, well-scoped use case and depend on how quickly privacy and security review can meet.
Use case
Pick one task that happens often, has an owner who wants it fixed, and can be measured, such as time per ticket or documents processed per day. Without a baseline, a pilot can't prove anything.
Record the baseline before you build. Measure the task as it runs today for a few weeks so the pilot has something to beat, and agree what result would justify production.
Your use case is well defined. Keep the scope narrow for the first release and resist adding tasks until the first one has proven its value.
Data access
An agent inherits every permission its connector has. List the minimum sources the task needs, and don't connect anything else until permissions on those sources have been reviewed.
Review permissions on each listed source before the pilot, remove broad sharing, and apply sensitivity labels in Microsoft Purview so confidential content is handled correctly.
Data access is scoped well. Re-check it whenever a new source or connector is added, because that's where access quietly widens.
Identity
This blocks production. Give the agent its own identity instead of borrowed credentials, name a human sponsor who is accountable for it, and grant roles on specific resources rather than whole subscriptions.
Narrow the roles to the individual resources the agent needs, and grant access with an expiry date through access packages rather than permanent assignments.
Identity is in good shape. Treat publishing the agent as a security gate, because a published agent can get a new identity whose permissions need reviewing again.
Privacy
This blocks production. If the agent touches personal information, BC's privacy laws apply as they would to any system. Bring your privacy officer in now, and confirm where prompts and responses are processed and stored.
Complete the privacy impact assessment before the pilot uses real personal information. Confirm that the model deployment type keeps processing in Canada, since some deployment types route requests to other regions.
Privacy is covered. Re-run the assessment if you change the model, the deployment type or the data the agent reads.
Evaluation
Build a test set of real questions with known good answers, and run it before every release. Hand-checking a few answers won't catch regressions when a prompt or model changes.
Add adversarial tests, especially prompt injection through the documents and tools the agent uses, and agree the score a release must reach before it ships.
Your evaluation practice is solid. Keep the test set growing with real failures from production so it reflects how people actually use the agent.
Cost
Estimate the cost per request from the model, the size of each request and the expected volume, then multiply by realistic usage. A smaller model is often accurate enough and far cheaper.
Turn your estimate into a per-transaction model and set Azure budgets and alerts before the pilot, so growth in usage doesn't become a surprise on the bill.
Cost is under control. Track token usage per agent after launch and revisit the model choice if volume grows.
