feat(eval): add batch-evaluation evaluate by jariy17 · Pull Request #1965 · aws/agentcore-cli

jariy17 · 2026-08-10T19:13:38Z

Summary

Adds agentcore eval batch-evaluation evaluate — the write path that starts an evaluation over existing sessions. batch-evaluation previously shipped read-only (get + list); this makes it runnable.

The command submits an async StartBatchEvaluation job (the service gathers the sessions) and returns a durable job id. Retrieve results with the existing get.

Usage

agentcore eval batch-evaluation evaluate \
  --agent <harness-id|runtime-id> \
  --evaluator Builtin.Helpfulness \
  --name my-eval

Session source — exactly one

Flag	Source
`--agent <harness-id\|runtime-id>`	resolved to the runtime's CloudWatch log group (reuses `resolveAgentToRuntime`)
`--online-eval <id>`	sessions an online-eval config already sampled
`--data-source-config <json>`	raw SDK `DataSourceConfig` escape hatch — inline / `file://` / stdin

Filters & options

--start-time / --end-time (ISO-8601, together) — optional time window.
--session-ids — specific sessions (agent arm only).
--ground-truth <json> — inline / file:// / stdin → evaluationMetadata.
--name (required), --description, --kms-key-arn.

JSON-valued flags resolve via SourceResolver + parseJsonFlag, matching online-eval/create.

Design notes

Core: startBatchEvaluation on EvalClient maps the resolved source to the dataSourceConfig union (cloudWatchLogs / onlineEvaluationConfigSource / raw passthrough) and sends via the data-plane client. Reuses the existing agent→runtime→log-group resolution.
Source resolution is inlined in the handler (one consumer today). It will be extracted to a shared util when ondemand evaluate lands as a follow-up and becomes the second consumer.
Long flags only — this framework's flag() has no short aliases; matches every existing handler.

Scope

batch-evaluation evaluate only. ondemand evaluate (sync, client-side) and simulate (dataset replay) are deferred to follow-up phases.
--lookback-days is intentionally not included yet; only explicit --start-time/--end-time windows for now.
--insights and --tags (supported by the SDK request) are deferred.

Verification

Verified end-to-end against a live account (us-west-2, real runtime + Builtin.Helpfulness):

Submitted (HTTP 202, PENDING) → get polled to COMPLETED, 5/5 sessions, averageScore: 0.9, with the resolved dataSourceConfig echoed back.

Tests

Handler command-flow tests (real root handler + TestCoreClient): required --name/--evaluator, source-arm selection (agent + online-eval resolved shapes), zero/multiple arms rejected, --session-ids rejected on the online-eval arm.

bun run typecheck, bun test src/ (1001 pass), oxlint — all clean.

codecov-commenter · 2026-08-10T21:17:03Z

Codecov Report

❌ Patch coverage is 82.05128% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.63%. Comparing base (7537b98) to head (fdbffd7).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
.../handlers/eval/batch-evaluation/evaluate/index.tsx	76.02%	35 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #1965      +/-   ##
============================================
- Coverage     96.79%   96.63%   -0.17%     
============================================
  Files           306      307       +1     
  Lines         17051    17243     +192     
============================================
+ Hits          16505    16662     +157     
- Misses          546      581      +35

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

aidandaly24 · 2026-08-11T14:51:25Z

@@ -0,0 +1,20 @@
+{
+  "batchEvaluationId": "golden_batch_evaluate-0c95e0cf8b",
+  "batchEvaluationArn": "arn:aws:bedrock-agentcore:us-west-2:725476964917:batch-evaluate/golden_batch_evaluate-0c95e0cf8b",


these tests are using the wrong account 685197708687

I'll use 685197708687 to record these.

Hweinstock

mostly some questions about how we're using comments in the code. I am of the opinion we should reserve them for details the code can't express, and make the code as readable as possible.

I found most of the comments distracting/confusing, and wonder if it would have the same effect on agents (especially if the information becomes stale).

Hweinstock · 2026-08-11T17:54:33Z


 jobs:
  call:
+    # Fork PRs never receive repo secrets/OIDC from GitHub, so the reusable's


is this intentionally part of this PR? feels unrelated to batch evals?

No this was a artifact for another PR

Hweinstock · 2026-08-11T17:59:45Z

+  // startBatchEvaluation submits the async, service-side job. Core translates the
+  // resolved SessionSourceValue into the dataSourceConfig union: the agent arm
+  // resolves the harness/runtime id to a log group (reusing agentDataSource), the
+  // online-eval arm points at a config ARN, and the raw arm passes JSON through.


I'm not sure I follow this comment, is it describing the function below? If so, should we localize that info to avoid drift?

I'm just going to remove it because the code explains itself.

Hweinstock · 2026-08-11T18:05:01Z

+  ): Promise<DataPlaneDataSourceConfig> {
+    if (source.origin === "raw") return source.dataSourceConfig;
+
+    // SessionWindow is already { startTime, endTime } — the SDK's SessionFilterConfig shape.


I feel like the code expresses this more concisely than the comment by the fact that we didn't need to transform it before passing to SDK.

I tend to think comments should be reserved for pieces of information that the code cannot or fails to express.

I'll remove this commnet

Hweinstock · 2026-08-11T18:13:15Z

+    }
+
+    const qualifier = source.endpoint ?? DEFAULT_ENDPOINT_QUALIFIER;
+    const { runtimeId, runtimeName } = await resolveAgentToRuntime(


couldn't this be a runtime or a harness? I thought we could pass harnessId to eval apis?

async function resolveAgentToRuntime( agent: string, clients: AwsClients, options: CoreOptions, ): Promise<{ runtimeId: string; runtimeName: string }> { const control = clients.control(toClientConfig(options)); try { const runtime = await control.send(new GetAgentRuntimeCommand({ agentRuntimeId: agent })); if (runtime.agentRuntimeName) { return { runtimeId: agent, runtimeName: runtime.agentRuntimeName }; } } catch (error) { if ((error as Error).name !== "ResourceNotFoundException") throw error; } const harness = await control.send(new GetHarnessCommand({ harnessId: agent })); const environment = harness.harness?.environment; const runtimeEnv = environment && "agentCoreRuntimeEnvironment" in environment ? environment.agentCoreRuntimeEnvironment : undefined; if (!runtimeEnv?.agentRuntimeId || !runtimeEnv?.agentRuntimeName) { throw new InputValidationError(`"${agent}" does not exist as a runtime or a harness`, { meta: { agent }, }); } return { runtimeId: runtimeEnv.agentRuntimeId, runtimeName: runtimeEnv.agentRuntimeName }; }

As shown above, the function can return both an harness id/name or a runtime id/name. I'll rename this reflect this.

Hweinstock · 2026-08-11T18:15:07Z

+import { coreOptsFromCtx, parseJsonFlag } from "../../../utils";
+import type { SessionSourceValue, SessionWindow } from "../../types";
+
+// createEvaluateBatchEvaluationHandler wires `batch-evaluation evaluate`: an


Do you feel that this comment expresses something that the code is unable to?

Removing this code comment too.

Hweinstock · 2026-08-11T18:21:59Z

+    },
+  });
+
+type SourceFlags = {


is there a nice way we can derive this from the flags to avoid drift?

Discussed in person this is actually meant to match this API shape, not the flag shape: https://docs.aws.amazon.com/boto3/latest/reference/services/bedrock-agentcore/client/start_batch_evaluation.html#.

We might want to pull out this functionality for insights later, but thats TBD.

Hweinstock · 2026-08-11T18:23:30Z

+// it. `rawDataSourceConfig` is the already-parsed --data-source-config (JSON
+// resolution is async and happens in the handler; this stays sync/pure). Kept
+// local for now; extract to a shared util when on-demand evaluate reuses it.
+function resolveSource(


what exactly is the source of? Source of data for the evaluations? resolveSource feels somewhat ambiguous.

I'll rename it to resolveDataSource

Hweinstock · 2026-08-11T18:27:01Z

+
+// resolveSource picks exactly one source arm and validates the filters legal for
+// it. `rawDataSourceConfig` is the already-parsed --data-source-config (JSON
+// resolution is async and happens in the handler; this stays sync/pure). Kept


Isn't this already communicated by the function definition?

I'll remove this comment

Adds the write path for evaluating existing sessions service-side: `agentcore eval batch-evaluation evaluate` submits an async StartBatchEvaluation job and returns a durable job id (poll with the existing `get`). The session source is exactly one of: - --agent <harness-id|runtime-id> (resolved to a CloudWatch data source) - --online-eval <id> (an existing online-eval config's sessions) - --data-source-config <json> (raw DataSourceConfig escape hatch; inline, file://, or stdin) narrowed by --lookback-days or --start-time/--end-time, plus --session-ids on the agent arm. Ground truth (--ground-truth) maps to evaluationMetadata. Verified end-to-end against a live account (job COMPLETED, 5/5 sessions, averageScore 0.9). Tests cover the evaluate command flow: required flags, source arm selection, and filter mutual-exclusion. Note: on-demand evaluate is deferred to a later phase; the shared session-source resolver will be extracted to a util when that lands.

Hoist `hasIds` above the raw-source guard and define it as `!!flags["session-ids"]?.length` — undefined/empty/present all collapse correctly — replacing the two `(… ?? 0) > 0` expressions.

The golden/fixture suite (batch-evaluation.fixture.test.tsx) is the convention for exercising the command → SDK seam; the TestCoreClient flow tests for evaluate were redundant scaffolding. Keeps the hierarchy + get/list edge tests.

Records the --agent source path end to end: GetAgentRuntime (agent → runtime resolution) + StartBatchEvaluation (the submitted job), with evaluate.golden.json pinning the rendered output. Matches the fixture convention used by the get/list suite.

…lveDataSource Remove comments that restated the code (per review): the startBatchEvaluation and createEvaluateBatchEvaluationHandler doc blocks and the timeRange line note. Rename resolveSource -> resolveDataSource so the name says what it produces, and trim its comment to the one thing the code can't say (why it's still local).

…d account Record the evaluate fixtures against the shared fixture account (685197708687) using its asdf_MyAgent runtime, instead of a personal account, per review.

Name it for what it returns (runtime name + id), per review. Also drop the remaining local-helper note above resolveDataSource.

…atic class Rename for clarity and add a deferral note: when insights lands as a second consumer, promote the type + resolveDataSource into a static SessionSource class.

jariy17 had a problem deploying to e2e-testing August 10, 2026 19:14 — with GitHub Actions Failure

github-actions Bot added size/l PR size: L agentcore-harness-reviewing AgentCore Harness review in progress and removed agentcore-harness-reviewing AgentCore Harness review in progress labels Aug 10, 2026

jariy17 force-pushed the feat/eval-evaluate-clean branch from 1b4b5a0 to 2252051 Compare August 10, 2026 19:33

jariy17 changed the base branch from feat/eval-batch-evaluation-tui to refactor August 10, 2026 19:33

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 10, 2026

jariy17 had a problem deploying to e2e-testing August 10, 2026 19:33 — with GitHub Actions Failure

jariy17 force-pushed the feat/eval-evaluate-clean branch from 2252051 to b3ff3a2 Compare August 10, 2026 21:15

jariy17 force-pushed the feat/eval-evaluate-clean branch from b3ff3a2 to dc0bcf6 Compare August 10, 2026 21:32

jariy17 changed the title ~~feat(eval): add batch-evaluation evaluate + ondemand evaluate~~ feat(eval): add batch-evaluation evaluate Aug 10, 2026

jariy17 force-pushed the feat/eval-evaluate-clean branch from dc0bcf6 to 0dba42a Compare August 10, 2026 21:45

jariy17 requested review from aidandaly24 and nborges-aws August 10, 2026 22:46

jariy17 force-pushed the feat/eval-evaluate-clean branch from 5f08b10 to 1390aa5 Compare August 11, 2026 14:13

aidandaly24 reviewed Aug 11, 2026

View reviewed changes

aidandaly24 previously approved these changes Aug 11, 2026

View reviewed changes

Hweinstock reviewed Aug 11, 2026

View reviewed changes

jariy17 added 4 commits August 11, 2026 19:44

refactor(eval): simplify session-ids presence check

dfdf8f0

Hoist `hasIds` above the raw-source guard and define it as `!!flags["session-ids"]?.length` — undefined/empty/present all collapse correctly — replacing the two `(… ?? 0) > 0` expressions.

jariy17 dismissed aidandaly24’s stale review via b78318a August 11, 2026 19:44

jariy17 force-pushed the feat/eval-evaluate-clean branch from 7b521d7 to b78318a Compare August 11, 2026 19:44

jariy17 added 4 commits August 11, 2026 21:21

test(eval): re-record batch-evaluation evaluate fixtures on the share…

acd06e4

…d account Record the evaluate fixtures against the shared fixture account (685197708687) using its asdf_MyAgent runtime, instead of a personal account, per review.

refactor(eval): rename resolveAgentToRuntime -> resolveAgentToNameAndId

2e08c4e

Name it for what it returns (runtime name + id), per review. Also drop the remaining local-helper note above resolveDataSource.

refactor(eval): rename SourceFlags -> DataSourceFlags, note future st…

fdbffd7

…atic class Rename for clarity and add a deferral note: when insights lands as a second consumer, promote the type + resolveDataSource into a static SessionSource class.

Hweinstock approved these changes Aug 11, 2026

View reviewed changes

aidandaly24 approved these changes Aug 11, 2026

View reviewed changes

jariy17 merged commit 531d068 into aws:refactor Aug 11, 2026
8 of 11 checks passed

jariy17 deleted the feat/eval-evaluate-clean branch August 11, 2026 21:58

Uh oh!

Conversation

jariy17 commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Usage

Session source — exactly one

Filters & options

Design notes

Scope

Verification

Tests

Uh oh!

codecov-commenter commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hweinstock left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jariy17 commented Aug 10, 2026 •

edited

Loading

codecov-commenter commented Aug 10, 2026 •

edited

Loading

jariy17 Aug 11, 2026 •

edited

Loading