Agent tests and the legacy Performance workspace

Status: Current | Last reviewed: 2026-10-06

Agent Test dialog

Select Test in the editor header or Run tests in Preview. Both open the same Overview, Run Tests and Results dialog.

Generated scenarios can be edited. Refresh preserves authored changes and does not recreate generated scenarios you removed. Existing advanced test settings are retained when you use the simpler scenario editor. Coverage means identified behaviors have linked scenarios, not that every possible behavior has passed. Generation requires an active OpenAI connection; running and scoring use the agent's existing evaluation connections and services.

Legacy Performance workspace — direct URL only

The original page is retained for advanced setup and investigation, with no link from the normal Test UI. In Vanguard, open:

https://ai-manager.vanguard.dev.trueserve.deloitte.com/agents/<agent-id>/performance

Replace <agent-id> with the ID from the agent's editor URL. This page keeps Snapshot / Goldens / Goals / Reviews / Simulations / Continuous review and the original saved-test editor. Use it for tool fixtures, supervisor responses, copilot replay, detailed scoring, prior attempts and retrying failed checks. The remainder of this page describes that legacy workspace.

Its assistant is the same conversation used in the editor. Use New chat to start a clean thread. These controls and the new Test dialog share saved tests and evaluation records.

Advanced saved tests

Saved tests can also exercise a Copilot that listens to a customer and representative. In Copilot conversation replay, enable replay, add each speaker’s words and choose how long to wait for cards. Add checks for a component being present or absent after startup or a numbered turn. Optional text and exact field values make checks more specific. A late card does not satisfy an earlier check; a worker still running at the checkpoint also fails that check. Replay checks the cards the service publishes, excluding speech recognition, browser layout and manual dismissal.

Saved cases prepared through the API can include startup host events, such as Salesforce record context. These arrive before the startup card checks and remain separate from the speakers' words. Takeda's Takeda native cases check an empty greeting, one patient and multiple patients with different names.

Under Responses during the test → Saved tool responses, select a variant already saved in the tool’s Testing tab by its operation and variant name. This changes sandbox responses for that case. Connected runs use real tools.

Drafting goals

The legacy page's floating assistant helps write the Goals the automated judge scores conversations against. Ask it to draft them from the agent's instructions, then accept the ones you like.

Don't see the assistant button? The Copilot uses the same switches as Generate with AI and the editor Assistant: the deployment feature flag, the tenant AI Generation setting, and a configured AI engine integration. If any is off, the launcher doesn't appear.

Ask in plain language:

The Copilot reads your agent's behaviour prompt (and, when helpful, real past conversations), then proposes a short list of goals — each a single, checkable expectation with a severity:

Accept, edit, or dismiss

Each proposed goal appears as a card with its text, a severity dropdown, and a one-line reason. Nothing is saved until you say so:

The Copilot only ever proposes. You are always the one who adds a goal, so you stay in control of the rubric.

What the goals do

Goals are the rubric the LLM judge uses when it reviews conversations (on the Reviews and Continuous review tabs). Once you have a few, review a real conversation and the judge scores each goal as satisfied or violated, with its reasoning — so you can see exactly where the agent is falling short.

Drafting goldens

A golden is a full example conversation that shows what a good interaction with your agent looks like, start to finish. The judge compares real conversations against it, turn by turn. You can ask the Copilot to build one:

The Copilot tags each turn essential (must match), flexible (the gist must match), or ignore (skipped, e.g. pleasantries). Just like goals, it only ever proposes — a card appears showing the conversation with a tag dropdown on each turn. Adjust the tags if you like, then Accept to add it to the Goldens tab (or Dismiss to drop it). Nothing is saved until you accept.

A golden turn can also pin which tool the agent should call, not just what it should say — the judge then flags a review when the real conversation called a different tool (or none). When you draft or curate a golden the Copilot names the tool on the relevant turn, choosing only from the tools your agent is actually configured to use. When you edit a golden by hand on the Goldens tab, each turn has a Tool called dropdown listing your agent's configured tools; pick one to require it on that turn, or leave it as No tool. (Tool results aren't something you author — they're filled in only when a golden is curated from a real conversation.)

One thing to expect if you connect a Knowledge or MCP tool (a connection that bundles several operations, e.g. an "OTP Verification" tool that can both send and check a code): the dropdown lists it as one entry per operation — send_otp, verify_otp — under the operation's own name, not the connection's display name. Configured operation filters also apply to this list. Prompt actions retain the selected operation's input fields when you use the same tool in the editor. That is what the agent actually calls at runtime, so it is what a golden must name. If you pinned such a tool by its connection name in an older golden, that turn will show as not in config until you re-pick it against the specific operation.

Groundwork that writes itself when you edit the prompt

You don't have to ask. When you edit an agent's behaviour prompt in the editor and close the edit panel, the Copilot notices the change and does the groundwork for you: it drafts fresh goals from your new instructions and adds them straight to the Goals tab, and it drafts a fresh golden — an ideal conversation — and saves it to the Goldens tab. Both are deduped, so editing the prompt again won't pile up near-identical goals or goldens.

(This is the one place the Copilot saves without asking — asking in chat always proposes first, so you accept. On a prompt change it writes and tells you, and the undo is simply deleting it on the tab.)

You'll know it happened because the assistant pops open with a short message naming what it added — e.g. "I added 4 goals from your prompt change" and "I drafted a golden…". New messages do not link to the legacy workspace. To edit or remove these records, open its direct URL above and select Goals or Goldens. The messages stay in the assistant's history, so you can find them again even if you dismissed the popup.

This only fires when the prompt text actually changed — closing the panel without editing it, or a whitespace-only tweak, does nothing. (If AI generation isn't configured for your tenant, the automatic drafting simply doesn't run.)

Clearing out goals and goldens in bulk

Both the Goals and Goldens tabs let you remove more than one entry at a time. Each row (or golden card) has a checkbox; tick the ones you want, or use the Select all checkbox at the top of the list, then click Delete selected. You'll be asked to confirm, and the count is removed in one go. This is the quick way to tidy up after the Copilot has drafted several goals or goldens you don't want to keep — deleting is still the undo, now for many at once rather than one at a time. (Deleting goals or goldens never touches the conversations themselves or any reviews you've already run.)

Run saved tests

Click ▶ Run tests in the agent's Preview header or under Performance → Simulations. The Tests tab lists saved cases with their latest result. Search by name, select individual tests or Select all, then click ▶ Play N tests.

Preview uses the agent version you are viewing, including draft and full-screen preview. Run options lets you change the version. Pending draft edits are saved before starting; if saving fails, the window stays open so you can retry.

Run options → Runs per test repeats each selected case against the same saved agent configuration and test revision. Use ten runs to check whether each case passes at least nine times. Inspect individual failures and attempt history; an average review score is not a pass rate.

Voice profiles with separate transcription and speech generation are tested with synthetic speech, including their turn detection and playback. Copilot replay uses the saved transcript.

Tool responses defaults to Saved test responses. Choose Connected tools for an integration test against your configured connections. Those tools can perform real actions, including record updates; use a test or demo connection. Connected-tool failures do not automatically redial.

In the test editor, More caller details includes optional Incoming caller number and Dialed number, both in international format such as +12025550147. These reach startup lookups as call context. Leave the caller number blank to test missing caller identification. Use a fresh Amazon Connect contact identity for each attempt gives each run its own contact identity so retries cannot reuse another call’s records.

For human assistance, Responses during the test → How the decision returns selects either a waiting workflow or Supervisor chat after escalation. The chat option waits for the configured escalation workflow to finish and uses the actual supervisor agent to relay the simulated human’s question or decision. It does not treat the escalation itself as approval.

The window switches to live progress as conversations start. Each test moves through Waiting, Starting, In conversation, and Checking result. You can see the agent's latest reply while it runs. Results show Passed or Failed, with View result explaining the verdict. Couldn't run and Couldn't check identify execution or checking problems separately. An incomplete review is retried once using the same conversation. If it still cannot be checked, use Retry checks; this does not start another chat.

If the model service interrupts or blocks a sandbox conversation, the test can automatically retry once. Retrying means a fresh conversation is being started with the same test settings. Attempt history preserves the earlier error, including when the retry succeeds. A second execution failure shows Couldn't run. A completed conversation that fails its expected outcome is not automatically retried.

Click View conversation on a test to watch its chat as messages arrive, with the same message styling as a normal chat. For voice agents, the Multimodal tab shows visual cards. This view is read only: you cannot send messages, click through a form or interrupt the caller. View supervisor opens the supervisor's chat when the test uses one.

Use Back to test progress to return to the run. Reset view clears your selection and search and returns to the test list; your runs and results stay saved. Reload chat reconnects the viewer without restarting the test. You can also open earlier conversations from Attempt history. If a retry starts while you are watching, View current conversation takes you to it.

Close the window and keep editing, or leave the page. Tests continue in the background. Clicking Run tests again restores the active run. Once it finishes, the test list shows a Latest results snapshot; the Runs tab keeps recent runs and their results. Use Select these tests to prepare another run, Stop run to stop unfinished conversations, or Retry checks if checking failed. Large runs have Previous/Next controls.

Add or edit a test

Choose Add test, give it a name, describe What does the caller need? and What should happen?, then save. Optional caller details let you add an opening line, background and facts the caller can provide. Use the pencil beside a saved test to edit it. If you change a test after running it, its old result is labelled Changed since run. Existing runs keep the definition they started with.

Write observable outcomes: for example, “Confirm the appointment, charge the approved copay once, and end the session after goodbye.” Passing requires the case's expected outcome and the agent's essential goals to be met. Goals and golden examples help judge the conversation; a saved test also supplies the caller situation to execute.

Test supervisor responses

When adding or editing a test, open Responses during the test. Choose a reusable capability scenario, or set Supervisor to approve, decline, ask a question then approve, or not respond. You can provide an approved amount, clarification question and any conditions the supervisor should include.

Send the configured response tests how the agent and its workflow handle that decision. Use the configured supervisor agent also runs the supervisor chat and its live-call relay. The agent must already have Human Supervision configured, including its request workflow and supervisor agent.

Timing and repeated responses lets you delay the reply and check duplicate or late responses. You can also choose whether identity verification and payment succeed. Select Require the agent to end the session when a proper goodbye is part of the outcome.

Progress shows when the supervisor is being awaited, asking a question or has responded. View result includes checked outcomes such as the supervisor decision and session closure. A missed required outcome fails the test even if the rest of the conversation went well.

Appointment tests also check that each completed payment is recorded once, with the same transaction, amount and currency. A successful payment alone does not make the whole test pass.

What runs in the background

The real agent handles the simulated caller. The interaction mode is chosen automatically; GPT Live uses synthetic caller speech. These tests check conversation behavior. Text interaction does not measure voice quality or interruptions.

If the agent gives an update while the simulated caller is preparing a reply, the caller reads that update before answering. This helps it respond to the current question when a photo upload or payment finishes in the background.

The picker uses reusable test responses from each external tool's Testing section. Missing responses or unsupported workflow steps produce a setup message before any conversations start. See Tool testing. The simulated records should agree throughout the conversation: a patient lookup, appointment confirmation and receipt must refer to the same records, and a reduced copay must remain reduced on the receipt. Invalid tool inputs from the agent are conversation behavior to review; broken test responses are setup errors. A lookup blocked because verification or an earlier required step was skipped is a real test finding. Starting tests uses model tokens and, where needed, synthetic speech. Shared capacity may keep some selected tests waiting.

An administrator must configure an active OpenAI integration for the simulated caller, judge and synthetic speech, and run the conversation/scoring workers. An agent can use a different provider for its own responses.

Reviewing conversations on demand

On the Reviews tab, click Review conversations to pick finished conversations to score right now — you don't have to wait for continuous review to sample them. The picker lets you:

Each conversation is scored in the background against your current goals and goldens. The list updates itself as reviews move from scoring to done — an Updating… indicator shows while anything is in flight, so there's no need to reload the page (a Refresh button is there if you want it). Open a review and it keeps refreshing on its own too, so a re-score lands in place. Re-reviewing a conversation updates its existing row rather than adding a duplicate.

To clear reviews you no longer want, tick their checkboxes (or Select all shown) and click Delete selected. You can also use a row's trash icon or Delete inside an open review. This removes only the reviews — the conversations remain, and reviewing them again recreates the review rows.

Reading a review

Click any row to open the review. At the top is the verdict — the overall score, a one-word summary (Strong / Mixed / Needs work), and the judge's rationale in plain language. Below it:

Anywhere you see a Turn N chip (in the evidence, deviations, or flagged moments), click it to jump straight to that moment in the transcript and highlight it — so you can go from "it missed this goal" to the exact exchange in one click. Use Rescore to grade the conversation again (for example after editing your goals); it updates in place. The trash icon beside Rescore deletes the review.

Tips

A conversation that reaches its turn limit is still checked, so its result can explain why the caller's request was not completed.