> For the complete documentation index, see [llms.txt](https://dyns.ntd.asia/power-dynamics/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dyns.ntd.asia/power-dynamics/artificial-intelligence/study-copilot-studio/testing-and-capstone/evaluating-agents-test-sets-and-evaluation-methods.md).

# Evaluating Agents: Test Sets and Evaluation Methods

Copilot Studio Session 8.1: test sets, all seven evaluation methods, and reading results — applied to Northwind.

{% hint style="info" %}
**Session 8.1 of 3 — Group 8: Testing & Capstone (opening session)**
{% endhint %}

Every session since 1.1 has used the same verification method: type a message into the test chat, read what comes back, decide if it's right. That works for one question. It doesn't scale to "does Northwind's agent still answer correctly after I changed its instructions" across the dozens of question shapes a real retail customer might ask. This session is about the mechanism Copilot Studio actually ships for that — one that went from preview to fully general in the seven months since this course's earliest sessions were taught, and worth checking against a live fetch specifically because of that.

{% hint style="warning" %}
Test chat answers "does this one conversation work." Agent evaluation answers "does this agent still work" — across a whole test set, run again and again, with a score you can compare over time instead of a feeling you have to remember.
{% endhint %}

## Two ways to test an agent, and why you need both

Copilot Studio's own evaluation documentation draws this contrast directly, and it's worth taking at face value rather than treating evaluation as strictly an upgrade over test chat:

|                      | Test chat                                                  | Agent evaluation                                                                  |
| -------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------- |
| Volume               | One question at a time; hard to repeat the same test twice | A whole test set at once, rerun identically as many times as you want             |
| Conversation control | Full — you're driving the conversation live, turn by turn  | Less — a test case runs as scripted, with less room to improvise mid-conversation |
| Simulated users      | Only you                                                   | Any user profile you configure, without you completing the interactions yourself  |

Microsoft's own conclusion: "use both the test chat and agent evaluation for a full picture of your agent." \[1] Neither replaces the other, and neither replaces responsible-AI review either — evaluation "measures correctness and performance, not AI ethics or safety problems. An agent might pass all evaluation tests but still, for example, produce an inappropriate answer to a question." \[1] The content-safety filtering 7.2 covered is a separate system, still doing its job underneath whatever an evaluation score says.

{% hint style="info" %}
**Why this session needed a fresh fetch, not a memory-cutoff shrug**

Copilot Studio's release notes place agent evaluation on a fast timeline: preview in October 2025, general availability in March 2026, with multi-turn conversation tests shipping the same month. \[7] Automated triggering via the Copilot Studio connector and a REST API for CI/CD followed in April 2026. \[7] Every page cited in this lesson was fetched live rather than answered from training data, specifically because a feature that GA'd mid-2026 is exactly the kind of claim this course's VERIFY stage exists to catch rather than assume.
{% endhint %}

## The anatomy of an evaluation

Four concepts nest inside each other, and the vocabulary is precise enough that it's worth fixing before anything else:

* **Test case** — one simulated interaction: a single question, or an entire conversation, optionally paired with the answer you expect back. \[1]
* **Test set** — a named group of test cases, up to 100 for a single-response test set. \[2]
* **Test method** (also called a grader) — how a test case's result gets scored; a test set can carry several at once. \[1]
* **Evaluation run** — one execution of a test set, producing a pass rate plus a per-case Pass, Fail, Invalid, or Error result you can compare against the next run. \[4]

Two limitations worth knowing before building anything: only one evaluation can run at a time per agent, and Copilot Studio keeps results for 89 days — after that, export to CSV is the only way to keep them. \[2]\[4]

{% hint style="danger" %}
**A documented gap, not a build mistake — flagged, not smoothed over**

"Currently, agent evaluation doesn't support Fabric data agents." \[1] That's a direct hit on Northwind's own build: the Sales & Inventory Insights connection wired in 6.4 is a Fabric data agent. Whatever test set gets built for Northwind in this session, it cannot include a test case that exercises that specific connection — not because of anything done wrong in 6.4, but because the evaluation surface itself doesn't reach it yet. Worth sitting with the same way 6.5 sat with the unanswered Activity-page question and 7.1 sat with DLP's unconfirmed reach: some coverage gaps are real, current, and not this session's job to paper over.
{% endhint %}

## Choosing an evaluation method

Copilot Studio ships seven test methods, and they split into three genuinely different jobs rather than seven flavors of the same idea:

| Method              | Job                                                                                               | Needs an expected answer?    | Scoring                                     |
| ------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------- | ------------------------------------------- |
| **General quality** | Judges the response on its own merits — no expected answer to compare against                     | No                           | 0–100%                                      |
| **Compare meaning** | Does the response mean the same thing as the expected answer, regardless of wording?              | Yes                          | 0–100%, pass-score threshold                |
| **Text similarity** | Does the response's wording closely match the expected answer's, without needing to be identical? | Yes                          | Cosine similarity 0–1, pass-score threshold |
| **Exact match**     | Character-for-character identical to the expected answer                                          | Yes                          | Pass/fail                                   |
| **Keyword match**   | Does the response contain some or all of a list of expected words or phrases?                     | Yes (keywords)               | Pass/fail                                   |
| **Tool use**        | Did the agent call the specific tool(s) or topic(s) it was supposed to?                           | Yes (expected tools)         | Pass/fail                                   |
| **Custom**          | Your own evaluation instructions plus labels — e.g. Compliant / Non-Compliant                     | No (your own labels instead) | Pass/fail by label                          |

General quality, Compare meaning, Text similarity, and Exact match sit on one spectrum — how much the wording is allowed to drift from the expected answer, from "not at all compared" through "meaning matters, wording doesn't" to "wording has to be close" to "wording has to be identical." Keyword match, Tool use, and Custom measure something else entirely — a keyword's presence, a tool's use, a custom rubric — not how closely two pieces of text resemble each other. All test sets start with General quality by default, and it's the only method that needs no expected answer or extra configuration at all. \[3]

### What "General quality" is actually scoring

Because it's the default and the one every test set carries whether or not you add anything else, its four criteria are worth knowing precisely — a response has to clear all four to count as high quality, and failing even one gets it flagged: \[3]

* **Relevance** — does the response stay on the subject and directly answer the question?
* **Groundedness** — does it rely on the context it was given, rather than introducing unsupported claims?
* **Completeness** — does it cover all aspects of the question with sufficient detail?
* **Abstention** — did the agent actually attempt to answer, or duck the question?

One documented limitation worth keeping in mind before assuming a low score means a bad agent: "reducing the number of knowledge sources for the agent isn't guaranteed to improve general quality grading... because the retrieved knowledge might be too large." \[3] A quality score can move for reasons that have nothing to do with what got changed.

{% hint style="info" %}
**Government Community Cloud: two features quietly missing**

GCC environments can't attach a user profile to a test set, and can't use the Text similarity method at all — every other method still works. \[1] Worth checking before promising a GCC customer's build the exact same evaluation surface this lesson describes.
{% endhint %}

## Building a test set

Copilot Studio offers five ways to populate one, and they trade off speed against how well the questions match what real users actually ask: \[2]

{% stepper %}
{% step %}

### Quick question set

Ten AI-generated questions from the agent's description, instructions, and capabilities — fast, good for a first pulse check, not a substitute for real coverage.
{% endstep %}

{% step %}

### Full question set

AI-generated from a chosen knowledge source or the agent's topics, with a configurable count — better coverage of what the agent actually knows, but "isn't good for testing information gaps," since it only asks about things the agent already has content for.
{% endstep %}

{% step %}

### From a test chat conversation, or a theme

Populate a test set from the questions in your latest test-chat session, or from a theme — a cluster of real user questions surfaced in analytics — so the test set reflects what people are actually asking rather than what a maker guessed they'd ask.
{% endstep %}

{% step %}

### Import a CSV

Up to 100 questions, each 1,000 characters or fewer, two columns — `Question` and `Expected response`. Expected responses are optional to import, but required for match, similarity, and compare-meaning test cases to produce anything but an Invalid result.
{% endstep %}

{% step %}

### Write manually

Slowest, most precise — full control over both the question and the expected answer, case by case.
{% endstep %}
{% endstepper %}

Every method funnels into the same configuration step: pick test methods, edit test-case details, then choose a **user profile** under Additional configuration — the account whose authentication and connections the evaluation uses to actually reach the agent's knowledge sources and tools while testing. \[2]

## Running an evaluation and reading what comes back

A run processes every test case in the set and writes results live, line by line, so you can watch pass/fail trends appear before the whole run finishes — and stop it early if something looks badly wrong. \[4] Each test case lands in one of four buckets:

* **Pass** / **Fail** — met or missed the test method's criteria.
* **Invalid** — the test method needed something the test case didn't have, most often an expected answer.
* **Error** — the run itself failed to complete for that case.

Beyond pass/fail, Copilot Studio measures **response time** per interaction — end-to-end, including every reasoning step, tool call, and connector call before the reply comes back. It's a measurement only; it never affects the pass rate. \[4] Run the same test set twice and the **Compare with** tool shows which cases flipped from failing to passing or the reverse, directly on the results list. \[4] Results export to CSV — question, expected response, test method, agent's actual response, and the reasoning behind each score — and an agent owner can grant an **agent viewer** role scoped to evaluations alone, letting someone run and manage tests without touching the agent itself or its sharing settings. \[4] That's a real access-control lever worth naming next to 7.1's authentication options — evaluation access and agent-edit access are two different permissions, not one bundled grant.

## A newer, narrower evaluation surface — named, not detailed here

Everything above describes the standard-harness Evaluate experience, the one this course builds on and the one that reached GA in March 2026. Agents built on the newer GitHub Copilot harness (6/2026) get their own **Evaluate** tab — same underlying idea, one test method only (**General quality**), test cases called "conversations" instead of test cases, and explicitly still a preview feature. \[6] That's a genuine harness-shaped fork in the same family as 5.3's activity-trace trap and 6.3's connected-agents-overview trap — two pages, two real feature sets, not a documentation inconsistency to resolve. Full treatment, including how it compares to the seven-method standard-harness surface above, is 8.2's job, alongside multi-turn conversation testing and CI/CD automation via REST API and the Copilot Studio connector.

## Applied to Northwind: a return-and-order test set

Northwind's support agent now spans order status (2.4, 4.3), FAQ knowledge (3.1–3.3), the Billing/Product-Support connected-agent split (6.2), and a REST API tool for eligibility checks (4.5). A single-response test set built against it, using three of the seven methods rather than all seven at once:

| Test case                                     | Method                                                | Why this method                                                                                                     |
| --------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| "What's your return policy for opened shoes?" | Compare meaning                                       | There's one correct policy, phraseable several honest ways — meaning matters, exact wording doesn't.                |
| "Where's my order #48213?"                    | Tool use (expects the 4.3 order-status REST API tool) | The question this session cares about isn't the wording of the answer — it's whether the right tool actually fired. |
| "My package arrived damaged, what do I do?"   | General quality                                       | No single correct answer to match against — this is exactly the open-ended case the default method was built for.   |

Left out deliberately, not by oversight: any test case exercising the Sales & Inventory Insights Fabric connection from 6.4 — the documented gap above means it has no path into this evaluation surface at all yet, standard harness or preview.

## Check your retrieval

{% tabs %}
{% tab title="Question 1 of 4" icon="circle-question" %}
A test case asks "What are your store hours?" with no expected answer supplied, and the test set uses only the default test method. What happens?
{% endtab %}

{% tab title="Answer" icon="unlock" %}
**General quality runs anyway — it's the one method that doesn't need an expected answer to score a response.** It scores relevance, groundedness, completeness, and abstention directly against the response. Methods like Compare meaning, Text similarity, and Exact match are the ones that produce an Invalid result without an expected answer.
{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Question 2 of 4" icon="circle-question" %}
Northwind's order-status test case needs to confirm the agent actually called the correct REST API tool, regardless of how the final sentence is worded. Which test method fits?
{% endtab %}

{% tab title="Answer" icon="unlock" %}
**Tool use.** It checks whether the agent called the specific tool or topic you named as expected — pass or fail — independent of how the response text is worded. Exact match and Text similarity both grade wording against an expected answer, and General quality doesn't check tool usage at all.
{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Question 3 of 4" icon="circle-question" %}
What does agent evaluation's documented Fabric data agent limitation mean for Northwind's build?
{% endtab %}

{% tab title="Answer" icon="unlock" %}
**The Sales & Inventory Insights Fabric connection from 6.4 currently has no path into agent evaluation at all.** Microsoft's own documentation states plainly that agent evaluation "doesn't support Fabric data agents" — no method-specific exception, no harness-specific exception described.
{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Question 4 of 4" icon="circle-question" %}
An evaluation run finishes and one test case shows a Response time of 4.2 seconds. What does that number affect?
{% endtab %}

{% tab title="Answer" icon="unlock" %}
**Nothing about the pass/fail result or the pass rate — it's a measurement only,** useful for comparing performance across changes. Response time is documented explicitly as never assigning a Pass or Fail result or affecting the test set's pass rate.
{% endtab %}
{% endtabs %}

## From test set to verdict

```mermaid
flowchart TD
    TC["Test cases<br/>(question + optional expected answer)"] --> TS["Test set<br/>(up to 100 cases)"]
    TS --> TM{"Test method(s)<br/>chosen per set"}
    TM -->|"General quality"| Run
    TM -->|"Compare meaning / Text similarity / Exact match"| Run
    TM -->|"Tool use / Keyword match / Custom"| Run
    Run["Run evaluation<br/>(one user profile, one at a time)"] --> Result{"Per-case result"}
    Result -->|"Pass / Fail"| Rate["Pass rate + response time"]
    Result -->|"Invalid"| Fix["Missing expected answer/keywords — fix the test case"]
    Result -->|"Error"| Fix2["Run-level failure — investigate"]
    Rate --> Compare["Compare with a prior run"]
    Compare -->|"regressed"| Agent["Adjust instructions, knowledge, or tools"]
    Agent --> Run
```

{% hint style="success" %}
**Key takeaway:** A test set turns "does the agent still work" from a feeling you get after a few manual checks into a number you can watch move — but it's only as honest as the method behind it: General quality judges an answer on its own; Compare meaning, Text similarity, and Exact match all check it against one you supplied, at three different tolerances for wording.
{% endhint %}

## Reflection

<details>

<summary>Northwind's return-policy question used Compare meaning, and its order-status question used Tool use — two different methods for two different questions, on the same agent. What would go wrong if you ran the return-policy question through Tool use instead, or the order-status question through Compare meaning instead?</summary>

{% hint style="info" %}
**ONE REASONABLE ANSWER** Tool use only checks whether a specific tool or topic fired — it has nothing to say about whether the returned policy text was correct, so running the return-policy question through it would pass an agent that called the right topic but described the wrong policy. Compare meaning, run against the order-status question, would need a fixed "expected answer" for what is actually a different order number every time it's asked — there's no single correct wording to compare against, since the correct answer changes with the data. The mismatch in both directions is the same lesson: a test method has to match what's actually variable about the question — the answer's content, or which capability gets invoked — not just get picked because it's available.
{% endhint %}

</details>

**Read next:** [Design and operationalize agent evaluation](https://learn.microsoft.com/en-us/agents/agent-evaluation) — the deeper evaluation-strategy guide this session's "About agent evaluation" page points to, for building a repeatable evaluation process beyond the UI walkthrough covered here.

1. [About agent evaluation](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-intro) — core concepts, test chat vs. evaluation, GCC limitations, the Fabric data agent gap, language support.
2. [Create a single response test set](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-create) — the five creation methods, the 100-case cap, CSV import format, user profile configuration.
3. [Choose evaluation methods](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-overview) — all seven test methods, General quality's four criteria, and their configuration requirements.
4. [Run evaluations and view results](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-results) — Pass/Fail/Invalid/Error, pass rate, response time, comparing runs, exporting results, the agent viewer role, 89-day retention.
5. [Test your agent](https://learn.microsoft.com/en-us/microsoft-copilot-studio/authoring-test-bot) — the test chat feature this session contrasts evaluation against, including conversation snapshots.
6. [Evaluate an agent (preview)](https://learn.microsoft.com/en-us/microsoft-copilot-studio/agents-experience/analytics-agent-evaluation-intro) — the GitHub Copilot harness's separate, single-test-method Evaluate tab, the harness-fork finding named above.
7. [What's new in Copilot Studio](https://learn.microsoft.com/en-us/microsoft-copilot-studio/whats-new) — the GA and preview timeline for agent evaluation, multi-turn tests, and evaluation automation, fetched live to confirm currency past this course's training cutoff.
8. [Study guide for Exam AB-620](https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ab-620) — reused from session 1; confirms this session maps to the "Evaluate agent performance" bullets under the exam's Test and manage agents domain.
9. [Harnesses overview](https://learn.microsoft.com/en-us/microsoft-copilot-studio/harnesses-overview) — reused from 1.1/6.3, grounding the standard-harness/GitHub-Copilot-harness distinction behind this session's evaluation-surface fork.