Portable test suites for evaluating Agent orchestrations (GroupChatOrchestration, etc.) — EvalPort interop?

[adhabnr-ux]

Hi SK community,

Looking through python/samples/getting_started_with_agents/multi_agent_orchestration/step3_group_chat.py, the orchestration pattern is clean: build a GroupChatOrchestration from a list of ChatCompletionAgents and a RoundRobinGroupChatManager, runtime.start(), await group_chat_orchestration.invoke(task=..., runtime=runtime), then await orchestration_result.get() for the final value. That's a nice, uniform shape — any orchestration (group chat, sequential, or your other managers) takes a task string in and produces a result out.

That uniformity made me wonder about testing it at scale: has anyone here wanted a portable way to define "given this task, the orchestration's output should satisfy X" test suites — the kind you'd want to run against an SK agent orchestration, and separately also run against a LangChain/AutoGen/raw-API pipeline doing the same job, to compare apples to apples?

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), a small open spec (JSON Schemas under spec/schemas/ for TestSuite/TestCase/ResultSet/Grader, plus Python/TS SDKs) aimed at exactly that — not another eval runner, just a portable format so a test suite isn't locked to one framework's dataset shape.

Rough sketch of what an adapter over GroupChatOrchestration could look like:

async def run_suite(suite: evalport.TestSuite, orchestration, runtime) -> evalport.ResultSet:
    results = []
    for tc in suite.test_cases:
        res = await orchestration.invoke(task=tc.input, runtime=runtime)
        output = await res.get()
        results.append(evalport.Result(test_case_id=tc.id, actual_output=str(output)))
    return evalport.ResultSet(suite_id=suite.id, results=results)

Given your CONTRIBUTING guidance that plugins/connectors are best hosted outside the SK repo, I'd build and host this as its own small package rather than proposing it land in semantic_kernel/. Posting here first (per COMMUNITY.md) to check whether this is actually a gap for anyone, before spending time on it. Spec for reference: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

No pressure either way — genuinely just gauging interest.

[Ama Senevirathne (amasen02)]

This is a legitimate gap in the agent ecosystem. Right now, most teams end up building ad-hoc harness scripts or tying themselves into proprietary eval platforms that assume a single prompt-response LLM rather than a multi-agent state machine.

Having a portable, schema-driven spec like EvalPort for orchestrations would be very useful, especially when teams want to benchmark whether migrating a workflow from AutoGen/LangGraph to Semantic Kernel introduces regression.

A couple of architectural observations from running multi-agent evals in production:

1. Terminal Output vs. Trajectory Graders

For simple single-turn tasks, output = await res.get() works. But in GroupChatOrchestration, evaluating only the terminal string misses 80% of multi-agent failure modes (e.g., tool hallucinations that got glossed over by the summary agent, turn oscillations, or inefficient routing).

In Semantic Kernel, you can extract the full message history from the orchestration result:

# Extract the full conversation trajectory for multi-step grading
messages = await res.get_messages() 
trajectory = [
    {"role": m.role.value, "author": m.name, "content": m.content, "tool_calls": len(m.items)}
    for m in messages
]

If EvalPort's Result schema allows attaching metadata or an execution trace (trajectory: list[dict]), graders could evaluate:

• Turn efficiency (did it solve the task in <= 4 turns?)
• Tool selection precision (did it invoke the intended plugin?)
• Inter-agent consensus

2. Async Streaming / Cancellation Support

Production agent runs can stall or exceed timeouts. When wrapping orchestration.invoke(), ensure the adapter accepts a timeout or asyncio.timeout(tc.timeout_seconds) so a single wandering agent test case doesn't hang the entire suite.

3. Packaging Strategy

Keeping it in a standalone external package (e.g. evalport-semantic-kernel or pip install evalport[semantic-kernel]) is 100% the right move per the current SK contribution model. As long as the adapter depends only on semantic-kernel (or just adheres to the generic duck-typed invoke(task=...) callable protocol), it will be very easy for developers to drop into their CI/CD test pipelines.

[adhabnr-ux]

Really useful feedback — let me answer each point against EvalPort's actual schema rather than what I'd like it to support, since I don't want to promise anything the spec doesn't really have.

1. Trajectory vs. terminal-output grading. You're right that Result.actual_output won't do this on its own — it's typed as Optional[str], not a structured list, so it's meant for a single output string/summary, not a full message history. But both Result and each GraderResult carry a free-form metadata: Dict[str, Any] (see sdk/python/openeval/types.py), so the honest place for your trajectory list is result.metadata["trajectory"], with actual_output holding a short human-readable summary (e.g. the final message content) for anything that just wants a quick look. A GraderType of "custom" (or a descriptive non-standard type string — the spec explicitly allows that, it's treated like "custom" for validation purposes) then lets you write turn-efficiency / tool-selection / inter-agent-consensus graders that read metadata.trajectory directly. That logic genuinely belongs in the adapter, not the core spec — EvalPort doesn't (and I don't think should) prescribe multi-agent-specific grading semantics itself.

2. Async timeout/cancellation. Agreed this is a real adapter concern rather than something the spec needs to solve — but there's a nice hook for it already in the schema: TestCase.timeout_ms: Optional[int] exists today. So a Semantic Kernel adapter's invoke() wrapper can honestly read test_case.timeout_ms and enforce it with asyncio.wait_for(orchestration.invoke(...), timeout=test_case.timeout_ms / 1000) (or asyncio.timeout, Python 3.11+) per test case, rather than inventing a new field — one wandering agent then can't stall the whole suite, and the per-case override comes for free.

3. Packaging. Agreed, and it matches the convention every adapter in adapters/ already follows: a standalone <framework>-openeval-adapter package depending only on evalport-sdk plus the target framework (or nothing at all, when duck-typing the target's shape is enough — see agenteval-openeval-adapter/niceeval-openeval-exporter for two different takes on that trade-off). A generic duck-typed invoke(task=...) protocol is exactly the right shape for something like evalport-semantic-kernel.

I don't want to promise a full SK adapter here without actually building and testing it first (that's bitten worse proposals than mine), but the shape above — metadata.trajectory for the full history, TestCase.timeout_ms for the per-case timeout, a standalone package with a duck-typed invoke() — is concrete enough that I'd be glad to build it against GroupChatOrchestration specifically if there's interest. Would a first cut that handles the terminal-output + trajectory-in-metadata case, with the timeout wiring, be a useful starting point, or is there a more pressing orchestration pattern (e.g. SequentialOrchestration, HandoffOrchestration) worth targeting first?