Test an agent¶

Status: Current | Last reviewed: 2026-10-20

Select Test in the agent header or preview to open the testing workspace. You can run tests at any time: to explore the agent, check a change, or check that familiar scenarios still work. The workspace has Overview, Tests, and Results tabs. All tabs use the available width up to a common 1,440px maximum, with matching margins on smaller screens.

For the Vanguard review, use agent 4982, Testing Workspace (Athens). Its published v1 is already saved as a comparison point. Vanguard agent 4210 remains separate.

Run tests¶

On Overview, confirm Version to check, then choose:

• Quick run: one calm customer per selected scenario.
• Full run: four customer styles per scenario: calm, frustrated, confused, and in a hurry.

All scenarios are selected by default. Expand Scenarios and repetitions to select a smaller set or repeat conversations to check consistency. Choose Run tests to start. Progress opens in Results.

For the imported Athens plan, Quick runs all 20 scenarios with 20 conversations; Full runs 80 conversations. Services are simulated, while customer, agent and assessment models consume usage. Testing a subset only checks those scenarios.

Set up scenarios and success criteria¶

Open Tests → Scenarios. Each test describes a customer situation, such as asking about an order while the account service is unavailable. Expand its success criteria to see what the agent is expected to do. Edit the test to change the customer instructions, tool responses, optional group, or assigned criteria.

To add an ad-hoc expectation, open the test and choose Add criterion for this test under Success criteria. Write what the agent should do, choose Required to pass or Feedback only, and save the test. You can add more than one criterion. They save with the test and apply only to that test, including when you create the agent's first test. Future tests do not inherit them.

These criteria use the same assessment and result evidence as other criteria. They also appear in the criteria library, where you can edit or explicitly reuse them later. Cancelling the test editor leaves new criteria unsaved.

Tests → Shared success criteria manages expectations reused across scenarios. For example, “Verify the customer before sharing account details” can apply to all tests. Other criteria apply only to selected tests. The number assigned tells you which criteria will be checked, not whether they passed.

A criterion can be Required to pass or Feedback only. Feedback-only findings do not change pass/fail or the score. Some saved criteria have specific tool or event checks; when changing their wording, review those checks before saving.

Customer facts and service responses are saved test conditions. Editing an agent does not change them. Use Test plan and preparation on Overview when you want to prepare or update scenarios explicitly.

Choose tool responses for a test¶

Edit a test and open Tool responses. Each external tool shows the full simulated responses saved for that scenario and the conditions under which they are returned. The agent still decides which tools to call and supplies its own inputs.

• Keep saved response preserves the current response.
• Use example copies an existing response from this agent's saved tests or the tool's checked examples.
• Customize for this test opens typed fields. When a tool declares different response types, choose an outcome such as found, not found, or error. You can then edit its fields without writing JSON.

Values marked Uses the tool input continue to use the agent's actual input. Choose Use a fixed value only when this scenario needs a literal value. Existing matching conditions, response sequences and repeat behavior are preserved. A newly added tool response also has Example inputs for checking this response; these validate the response and do not dictate the agent's calls.

Choose Save to apply your changes to this test only. Cancel discards them. Other tests, shared tool settings, past runs and saved comparison points retain their original responses.

Success criteria → Expected actions shows any saved checks for tool-call order, inputs and expected response values. If you change a response in a way that conflicts with these checks, saving explains the conflict. Review the scenario and use Edit expected values when the expected result should also change. Checks are never rewritten automatically to match a new response.

Saved tests simulate external service responses. They still run the agent and the customer/assessment models; native actions and workflow logic retain their test-runtime behavior. This workspace does not provide a live-service mode. Checking a real service connection is a separate activity.

Organize test groups¶

Groups are optional labels for organizing scenarios and selecting several tests together. They do not affect scoring, criteria, or service responses. A single Order status group containing every test adds no useful distinction, so the list defaults to Display → All tests without group headings. Existing group names are retained.

To create a useful group:

1. Select the tests you want together. Use Clear all first if needed.
2. Choose Group selected, enter a name such as Verification or Service errors, review the selected tests, and choose Create group.
3. Use Display → By group to browse or select those groups together.

You can also edit a test's Group (optional) field to choose an existing group or type a new name. Leave it blank for Ungrouped. In the grouped view, Rename group renames all its tests, including those hidden by search.

Creating or renaming a group preserves test criteria and previous runs. Renaming to an existing name is blocked; move individual tests to combine groups. Groups exist while active tests use their names; they are not separate empty folders.

Review results¶

Results shows a Run history picker across the top. Select a run to see its outcomes, filter conversations by status, and open a conversation for its checks and transcript. Use Load earlier runs for older history. On smaller screens, checks and transcript stack vertically.

What needs attention on Overview links to failed or unresolved checks from the latest run. Saved results always describe the version and scenarios checked at that time. The run view's Attempt history retains previous attempts and their evidence.

Compare versions when needed¶

Enable Compare with a saved version on Overview. Choose a comparison point or save another one with a name and agent version. Saving records the agent and its test plan without starting conversations or changing earlier comparison points.

Confirm the selected version and choose Run comparison. Both versions use the same saved scenarios and criteria. Quick and Full still control customer styles; the displayed count includes both versions. For the imported plan, that is 40 conversations for Quick or 160 for Full.

If the current test plan differs from the saved plan, the workspace explains that separately. Save a new comparison point to adopt the updated plan for both versions. A changed expectation is not evidence that the agent improved.

Open Previous comparisons on Overview to revisit a paired result:

Passed first time is separate from success after recovery. Open either version's result to read the criteria, actual behavior, and supporting evidence.

Voice tests state the conversation mode they exercise. A passing dialogue check does not establish audio quality, interruption handling, or phone routing.