> For the complete documentation index, see [llms.txt](https://dyns.ntd.asia/power-dynamics/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dyns.ntd.asia/power-dynamics/artificial-intelligence/study-copilot-studio/testing-and-capstone/conversational-evaluation-automation-and-the-github-copilot-harness.md).

# Conversational Evaluation, Automation & the GitHub Copilot Harness

Session 8.2: multi-turn test sets, REST API/connector automation, the GitHub Copilot harness's Evaluate preview, and what the Copilot Agent Kit's pipeline test gate actually is.

{% hint style="info" %}
**Group 8 — Testing & Capstone, Session 8.2 of 3.** Builds on 8.1's single-response test sets and seven test methods. 8.3 (capstone) is next and closes the group.
{% endhint %}

8.1 built a three-case test set and ran it by hand — one question, one answer, per case. Real Northwind customers don't ask one question. They ask about a return, get an answer, then ask a follow-up that only makes sense if the agent remembered the first message. This session covers the three things 8.1 deliberately left for later: testing a whole conversation instead of a single exchange, running an evaluation from outside the Copilot Studio UI entirely, and a second Evaluate surface — on a different harness — that looks similar but isn't built the same way underneath.

{% hint style="success" %}
A single-response test set asks one question and grades one answer. A conversation test set asks a whole conversation and grades whether the agent held up across it — and it can only be scored by four of the seven methods 8.1 introduced, not all seven.
{% endhint %}

## From one exchange to a whole conversation

Conversational test sets live in the same Evaluation page as 8.1's single-response sets — select **New evaluation**, then **Conversation** instead of the default \[1]. Three ways to populate one, narrower than the five 8.1 covered for single-response sets:

{% stepper %}
{% step %}

### Quick conversation set

Ten short conversations, generated automatically from the agent's description, instructions, and capabilities — the multi-turn equivalent of 8.1's Quick question set, and just as much a first pulse check rather than real coverage.
{% endstep %}

{% step %}

### Full conversation set

Generated from the agent's knowledge or defined topics, with a choice between short and long conversations — better coverage of what the agent is actually built to handle.
{% endstep %}

{% step %}

### Use your test chat

Convert the most recent test-chat session — the one you drove by hand — directly into a test case. The fastest way to turn a conversation you already know is interesting into something you can rerun automatically.
{% endstep %}
{% endstepper %}

A conversation test set holds up to **20 test cases**, a fifth of the 100-case ceiling 8.1 named for single-response sets. Each test case caps out at **12 total messages — six question-and-answer pairs** \[1]. Both numbers are real ceilings, not soft defaults, and CSV import and manual entry — two of 8.1's five single-response creation paths — aren't offered for conversations at all; the documentation lists exactly the three methods above and no more.

## Which methods actually grade a conversation

8.1 covered all seven test methods as one family \[13]. For a conversation test set, only four of them are on the menu \[1]:

| Method               | Works on a conversation?  | Why, or why not                                                                                                                     |
| -------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| General quality      | Yes                       | Judges the whole exchange on its own merits — nothing to compare against, so conversation length doesn't matter.                    |
| Keyword match        | Yes                       | Checks for expected words or phrases anywhere across the conversation, not tied to any single message.                              |
| Tool use (Tools use) | Yes                       | Checks that the right tool or tools fired somewhere in the exchange — still a yes/no question regardless of how many turns it took. |
| Custom               | Yes                       | Your own rubric, applied to the conversation as a whole.                                                                            |
| Compare meaning      | No — single response only | Needs one expected answer to compare meaning against. A six-message conversation doesn't reduce to one answer.                      |
| Text similarity      | No — single response only | Built to score how closely one response's wording matches one expected string.                                                      |
| Exact match          | No — single response only | Character-for-character comparison doesn't extend to a multi-turn exchange at all.                                                  |

The pattern holds together once you look at what the three excluded methods have in common: all three need exactly one expected answer to compare against. A conversation doesn't have one expected answer — it has a shape, a sequence of turns that either goes right or goes wrong somewhere along the way. That's a structural reason, not an arbitrary product limitation, and it's also why General quality, Keyword match, Tool use, and Custom all made the cut: none of them require a single fixed string to grade against.

## Two ways to trigger an evaluation without opening Copilot Studio

Both automation paths shipped in April 2026, and both wrap the same underlying operations — but they're not the same maturity \[4]. The **Copilot Studio connector** reached general availability; the **Power Platform REST API** for evaluations is still in preview \[4].

|                       | Power Platform REST API                                                                    | Copilot Studio connector                                                       |
| --------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ |
| Maturity (April 2026) | Preview                                                                                    | General availability                                                           |
| What you write        | Direct HTTP calls, plus your own Entra ID OAuth 2.0 token handling via an app registration | No code — five prebuilt actions you drop into a flow or agent                  |
| Where it runs         | Any custom automation or CI/CD pipeline that can make an authenticated HTTP call           | Power Automate flows, Copilot Studio agent flows, or as a tool inside an agent |
| Prerequisites         | Bot ID, Environment ID, an existing test set, an Entra ID access token                     | The Microsoft Copilot Studio connection — no separate token handling           |

Underneath, they're the same four operations wearing two different interfaces. The REST API exposes them as endpoints under `/api/makerevaluation/` — list test sets, get one test set's details, start a run against a test set ID, and get a run's status and results by run ID \[2]. The connector wraps those same four jobs as **Evaluate Agent**, **Get Agent Test Set Details**, **Get Agent Test Sets**, and **Get Agent Test Run Details** (plus a fifth, **Get Agent Test Runs**, for listing every past run) \[3]. Starting a run is asynchronous either way — you get a run ID back immediately, then poll for status until it reads `Completed` \[2] \[3]. Both paths accept an optional connection ID as the user profile that authenticates the run's tool and knowledge-source calls — without one, the evaluation runs unauthenticated \[2].

{% hint style="warning" %}
**A scheduled evaluation, in Northwind's own terms.** Microsoft's own documented example is a Recurrence trigger — in Power Automate or, notably, in a Copilot Studio event trigger of the kind 5.1 designed and 5.2 built — that calls Evaluate Agent on a schedule, polls Get Agent Test Run Details until the run completes, then routes the results to a database, an email alert, or a Power BI dashboard \[3]. That's the same trigger type Northwind's return-follow-up automation already uses, aimed at a different job: not watching Dataverse for a status change, but watching the agent's own quality over time.
{% endhint %}

## Closing 7.4's loop: what the Copilot Agent Kit's test gate actually tests

7.4 named a Power Platform pipeline stage that pauses a deployment to run an automated test gate built on the **Copilot Agent Kit**, and left the mechanism itself unexplained. It's not the REST API or connector above wearing a pipeline costume. It's a separate system.

The Copilot Agent Kit is built and maintained by the Power Customer Advisory Team (Power CAT) — a Microsoft team, but not the Copilot Studio product team, and not shipped as part of the core platform \[5]. Its test engine runs against the **Copilot Studio Direct Line API**, a different surface entirely from the `makerevaluation` REST namespace this session's automation section covers \[7]. Its own records — Agent Test Set, Agent Configuration, Test Run — live in Dataverse, configured through a model-driven app rather than the Evaluation page \[6].

The Kit's test types don't line up with the seven this course has used since 8.1, either \[7]:

* **Response match**, **Attachments** (Adaptive Cards), and **Topic match** — the last one needs Dataverse.
* **Generative answers** — needs AI Builder and Application Insights, since it's comparing a nondeterministic answer against a sample using its own prompt-based scoring, not the platform's General quality criteria.
* **Multi-turn** — a set of ordinary test cases run in a specified order inside one conversation, aimed specifically at testing generative-orchestration agents end to end.
* **Plan validation** — checks that generative orchestration's dynamic plan includes the expected tools, to a set threshold. It never looks at what the agent said — only which tools it decided to call.

{% hint style="danger" %}
Two systems, both from Microsoft, both using the words "test set" and "multi-turn," both wired into automated workflows — and, as far as either one's own documentation says, they don't talk to each other. The Kit doesn't call the native evaluation REST API, and the pipeline stage 7.4 flagged runs entirely on the Kit's own Dataverse records and Direct Line calls. That's not a contradiction to resolve. It's the same shape of finding as 6.4's Fabric data agent fork — two real, independently built mechanisms that happen to share a name, confirmed by comparing what each one's documentation actually says it does, not assumed from either one's silence about the other.
{% endhint %}

## A second Evaluate tab, named in 8.1 and covered in full here

Agents built on the GitHub Copilot harness carry their own **Evaluate** tab, alongside Build, Preview, and Monitor \[8] \[14]. Every page describing it still opens with the same line: "This article is prerelease documentation and is subject to change." \[8] \[9] The vocabulary shifts along with the harness — what the standard harness calls a *test case*, this surface calls a **conversation**; what the standard harness calls a *test set*, this surface calls an **evaluation** \[8].

The biggest difference isn't the vocabulary. It's the count. Where the standard harness offers seven test methods (or four, for a conversation set), the GitHub Copilot harness's Evaluate tab offers exactly **one: General quality** — and the documentation is explicit that it "doesn't compare responses to expected answers" at all \[8]. There's no conversation-vs-single-response distinction on this harness either, because every evaluation here is already a conversation; the single-response concept 8.1 spent a whole session on doesn't exist on this surface.

Building a test set follows a familiar shape — upload a CSV, generate 10 with AI (then 25 or 50 more once you've run the first batch), or write conversations by hand — configured with a name, the one available test method, and a user profile \[9]. Running one processes every conversation and produces a score; re-running after a change is the explicit intended workflow, with each run kept separately so you can compare them \[10]. Results show, per conversation, the message count and a general-quality score, plus an evaluation summary — overall score, duration, test case count, data type (conversation is the only type this surface has), and user profile \[11]. Opening one conversation shows the user messages, the agent's actual responses, and a pass or fail \[11].

One thing this surface doesn't have, as far as every page fetched for this session says: no REST API, no connector, nothing documented for triggering a GitHub Copilot harness evaluation from outside the UI. Every automation path this session covers — the REST API, the connector, the Copilot Agent Kit's pipeline gate — is scoped explicitly to the standard harness \[2] \[3]. That's a genuine, currently-documented gap, the same kind 8.1 flagged for Fabric data agents: not confirmed to be permanent, just not there yet, and worth naming rather than assuming either way.

## Three testing surfaces, not two

Which test methods an agent can even use depends on its harness \[12] \[14] — not a soft guideline, a hard split confirmed across every source this course has fetched since 5.3's first harness-shaped finding.

|                 | Standard-harness evaluation          | GitHub Copilot harness Evaluate (preview) | Copilot Agent Kit                                                                             |
| --------------- | ------------------------------------ | ----------------------------------------- | --------------------------------------------------------------------------------------------- |
| Built by        | Copilot Studio product team          | Copilot Studio product team               | Power CAT (separate team)                                                                     |
| Talks to        | `makerevaluation` REST namespace     | (no documented external API)              | Direct Line API                                                                               |
| Test methods    | 7 (4 for conversations)              | 1 (General quality only)                  | 6 (response match, attachments, topic match, generative answers, multi-turn, plan validation) |
| Automatable how | REST API (preview) or connector (GA) | Not documented                            | Power Platform pipeline stage                                                                 |
| Status          | GA (March 2026)                      | Preview                                   | GA (Power CAT toolkit)                                                                        |

## Applied to Northwind: a conversation that crosses the 6.2 handoff

The most useful multi-turn test case for Northwind isn't a longer version of 8.1's single-response ones — it's a conversation that exercises something a single question never could: the Billing/Product-Support handoff 6.2 built.

| Turn | User says                                                        | What's actually being tested                                                                                             |
| ---- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| 1    | "My last order arrived with a cracked case, can I get a refund?" | Routes to Product-Support, not Billing — a wrong first route breaks everything downstream.                               |
| 2    | "How long will the refund take once it's approved?"              | A billing question inside a conversation Product-Support opened — does context carry, or does the agent lose the thread? |
| 3    | "Can you also check if my order #48213 shipped yet?"             | The 4.3 order-status REST tool, fired mid-conversation without derailing the refund thread already in progress.          |

Test methods: **Tools use**, expecting the order-status tool to fire by turn 3; **Keyword match**, expecting "refund" and a policy-specific term to appear somewhere in the response; **General quality**, as a catch-all for whether the whole exchange actually held together. Compare meaning, Text similarity, and Exact match are off the table here for the same structural reason named above — there's no single expected answer for a three-turn exchange that touches two subagents and one tool.

Left out on purpose, still: any test case that would exercise the Sales & Inventory Insights Fabric connection from 6.4. 8.1 named that gap for single-response test sets, and nothing in conversational evaluation changes it — the limitation lives at the level of what agent evaluation can reach, not at the level of test-set type.

## Check your retrieval

{% tabs %}
{% tab title="Question 1" %}
A conversation test case in Northwind's build needs to check whether the agent invoked the order-status tool at some point across a three-turn exchange. Which test method is actually available for a conversation test set?
{% endtab %}

{% tab title="Answer" %}
**Tools use** — one of the four methods available for conversation test sets. Compare meaning, Text similarity, and Exact match are single-response only, since each needs one fixed expected answer to compare against — something a multi-turn conversation doesn't reduce to.
{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Question 2" %}
As of April 2026, what's the actual maturity difference between the two documented automation paths for standard-harness agent evaluation?
{% endtab %}

{% tab title="Answer" %}
The **connector reached general availability**; the **REST API is still in preview**. Copilot Studio's release notes list the Copilot Studio connector without a preview tag in April 2026, while the REST API for automated evaluations is explicitly marked (Preview) in the same month's notes — same month, two different maturity levels.
{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Question 3" %}
7.4 flagged a Power Platform pipeline stage that runs an automated test gate using the Copilot Agent Kit, without explaining the mechanism. What does this session's research show that gate actually calls?
{% endtab %}

{% tab title="Answer" %}
The **Copilot Agent Kit's own test engine**, which runs against the Direct Line API and uses its own Dataverse-based Test Run records — a separate system from native evaluation, confirmed by its own documentation, not the makerevaluation REST namespace or connector this session's automation section covers.
{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Question 4" %}
Based on every page fetched for this session, what's true about automating an evaluation on the GitHub Copilot harness's Evaluate tab?
{% endtab %}

{% tab title="Answer" %}
**No REST API or connector automation is documented for it** — every automation path this session covers is scoped explicitly to the standard harness. It's a currently-real, openly-flagged gap, not a confirmed permanent limitation.
{% endtab %}
{% endtabs %}

## Triggering an evaluation from outside Copilot Studio

```mermaid
flowchart TD
    Trigger["Recurrence trigger<br/>(Power Automate or Copilot Studio)"] --> Start
    Start{"Start a run"} -->|"REST API (preview)"| API["POST .../testsets/{id}/run"]
    Start -->|"Connector (GA)"| Conn["Evaluate Agent action"]
    API --> RunId["runId returned immediately"]
    Conn --> RunId
    RunId --> Poll["Poll: Get Agent Test Run Details"]
    Poll -->|"state: Running"| Poll
    Poll -->|"state: Completed"| Route["Route results:<br/>database, email alert, or Power BI"]
    Route --> Decide{"Regression found?"}
    Decide -->|"yes"| Fix["Adjust instructions, knowledge, or tools"]
    Decide -->|"no"| Done["Deployment / schedule proceeds"]
    Fix --> Trigger
```

{% hint style="success" %}
**Key takeaway.** A conversation test set trades three of the seven test methods for the ability to test whether an agent holds context across turns — and running it from a pipeline instead of by hand means picking between a GA connector and a still-preview REST API, not assuming automation is one settled thing.
{% endhint %}

## Reflection

<details>

<summary>The Copilot Agent Kit's test engine and the native evaluation REST API both let you automate agent testing from outside the Copilot Studio UI, and both can sit inside a CI/CD pipeline. Given everything this session found about how differently they're built, when would picking the Kit over the native REST API actually be the right call — not just the available one?</summary>

**One reasonable answer:** The Kit earns its place when a test needs something the native seven methods can't express at all — plan validation's check of which tools a generative-orchestration plan included, or a generative-answer test that needs a custom rubric scored through AI Builder rather than the platform's fixed General quality criteria. It also earns its place when the deployment pipeline itself is the point: Copilot Agent Kit's pipeline stage is built specifically to gate a Power Platform pipeline promotion, with its own three-environment model, which the native REST API doesn't attempt to orchestrate on its own. The native REST API earns its place everywhere else: anywhere the seven documented test methods already say what's needed, since it stays inside the same test-set data the Evaluation page uses, with no separate Dataverse app, Direct Line wiring, or Power CAT dependency to install and maintain.

</details>

## Read next

[Automate agent evaluations with Power Platform API](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-rest-api) — the full endpoint reference for the automation path covered in this session, including the exact request and response shape for each operation.

### Sources

1. [Create a conversational test set](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-multi-turn) — the three creation methods, the 20-case/12-message limits, and which test methods apply to conversations.
2. [Automate agent evaluations with Power Platform API](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-rest-api) — the makerevaluation REST endpoints, prerequisites, and response fields.
3. [Trigger agent evaluations with connectors](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-automate-tools) — the five connector actions and the scheduled-evaluation example.
4. [What's new in Copilot Studio](https://learn.microsoft.com/en-us/microsoft-copilot-studio/whats-new) — reused from 8.1; confirms the March 2026 GA of evaluation and multi-turn tests, and the April 2026 GA/preview split between the connector and REST API.
5. [Copilot Agent Kit overview](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/kit-overview) — Power CAT ownership, and the kit's full capability list beyond testing.
6. [Automate testing and deployment of agents using pipelines in Power Platform](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/kit-automate-test-deploy) — the three-environment pipeline model 7.4 named, resolved here.
7. [Enhance agent testing with Copilot Agent Kit](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/kit-test-capabilities) — the Direct Line API basis and the six Kit test types, the evidence for this session's architecture-fork finding.
8. [Evaluate an agent (preview)](https://learn.microsoft.com/en-us/microsoft-copilot-studio/agents-experience/analytics-agent-evaluation-intro) — the GitHub Copilot harness's Evaluate tab: conversations, evaluations, the single General quality method.
9. [Create a test set for an agent (preview)](https://learn.microsoft.com/en-us/microsoft-copilot-studio/agents-experience/analytics-agent-evaluation-create) — the three ways to populate a GitHub Copilot harness evaluation.
10. [Run an evaluation for an agent (preview)](https://learn.microsoft.com/en-us/microsoft-copilot-studio/agents-experience/analytics-agent-evaluation-results) — the run and re-run workflow on this harness.
11. [View evaluation results for an agent (preview)](https://learn.microsoft.com/en-us/microsoft-copilot-studio/agents-experience/analytics-agent-evaluation-view) — what the results view actually shows, conversation by conversation.
12. [Run and validate an AI model migration for Copilot Studio agents](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/manage-agent-model-migration) — confirms test methods are harness-dependent, and frames automated evaluation as a migration-safety tool.
13. [Choose evaluation methods](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-overview) — reused from 8.1, grounding the seven-method definitions this session narrows down from.
14. [Harnesses overview](https://learn.microsoft.com/en-us/microsoft-copilot-studio/harnesses-overview) — reused from 1.1/6.3/8.1, grounding the standard-harness/GitHub-Copilot-harness distinction this session's evaluation-surface fork depends on.