Skip to content

fix: resolve target-based AB test target name mismatch#1188

Merged
notgitika merged 2 commits intomainfrom
fix/ab-test-target-name-resolution
May 9, 2026
Merged

fix: resolve target-based AB test target name mismatch#1188
notgitika merged 2 commits intomainfrom
fix/ab-test-target-name-resolution

Conversation

@notgitika
Copy link
Copy Markdown
Contributor

@notgitika notgitika commented May 8, 2026

Summary

  • Fixes resolveTargetName() in post-deploy-ab-tests.ts which had a false-positive startsWith check that prevented the project prefix from being applied to target names
  • The target-based AB test was referencing "ProjectName-prod" while the actual AWS gateway target was created as "ProjectName-ProjectName-prod" by post-deploy-http-gateways.ts, causing the AB test to stay in NOT_STARTED forever
  • This has been causing the e2e-tests-full.yml workflow to fail on every run since the feature was introduced

Closes #1179
Closes #1091

Root Cause

post-deploy-http-gateways.ts (lines 104, 146, 315) unconditionally writes targets as ${projectName}-${tgt.name}.

ABTestPrimitive.createTargetBasedABTest() (line 634-635) stores the variant targetName as ${runtime}-${endpoint} — i.e., always the unprefixed logical name.

The old resolveTargetName() had a startsWith guard that misfired when runtime === projectName (the common default): "MyAgent-prod".startsWith("MyAgent-") → true, so it skipped prefixing. The AB test then referenced "MyAgent-prod" while the actual AWS target was "MyAgent-MyAgent-prod".

Unconditional prefixing is the right call since the config always stores the unprefixed logical name.

Changes

  1. post-deploy-ab-tests.ts: Remove the startsWith heuristic, always apply the project prefix
  2. post-deploy-ab-tests.test.ts: Add two unit tests covering target-based variant resolution:
    • runtime === projectName (the case the old check broke)
    • Different project/runtime names (standard prefixing)

Test plan

  • Unit tests pass (25/25 in post-deploy-ab-tests.test.ts, including 2 new)
  • Lint, prettier, typecheck all pass
  • Integration tests unaffected (only test local add/remove, not deploy)
  • Full e2e suite should now pass shard 5/6 (ab-test-target-based.test.ts)

The resolveTargetName() function had a false-positive in its startsWith
check that prevented it from applying the project prefix. When the
runtime name equals the project name (the default case), target names
like "Foo-prod" already start with "Foo-" so the guard incorrectly
assumed they were already prefixed and returned them unchanged.

However, post-deploy-http-gateways.ts unconditionally creates targets
as `${projectName}-${tgt.name}` (e.g., "Foo-Foo-prod"), so the AB test
referenced a non-existent target and stayed in NOT_STARTED forever.

The fix removes the startsWith heuristic and always applies the prefix,
matching what post-deploy-http-gateways.ts does.
@notgitika notgitika requested a review from a team May 8, 2026 23:49
@github-actions github-actions Bot added size/xs PR size: XS agentcore-harness-reviewing AgentCore Harness review in progress labels May 8, 2026
Copy link
Copy Markdown

@agentcore-cli-automation agentcore-cli-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix itself looks correct and the root-cause analysis matches what I see in the code: post-deploy-http-gateways.ts (lines 104, 146, 315) unconditionally writes targets as ${projectName}-${tgt.name}, and createTargetBasedABTest (ABTestPrimitive.ts line 634-635) stores the variant targetName as ${runtime}-${endpoint} — i.e., always the unprefixed logical name. So unconditional prefixing in resolveTargetName is the right call. The old startsWith check only misfired when runtime === projectName, which is the common default and is exactly what the e2e test was hitting.

One thing worth addressing before merge — see the inline comment on the test file.

Comment thread src/cli/operations/deploy/post-deploy-ab-tests.ts
@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 8, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 8, 2026

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 43.31% 9076 / 20954
🔵 Statements 42.59% 9639 / 22627
🔵 Functions 40.16% 1567 / 3901
🔵 Branches 40.15% 5856 / 14583
Generated in workflow #2688 for commit 9cfed0d by the Vitest Coverage Report Action

Covers the resolveTargetName fix with two cases:
- runtime === projectName (the false-positive case that caused the bug)
- different project/runtime names (standard prefixing)
@github-actions github-actions Bot added size/s PR size: S and removed size/xs PR size: XS labels May 8, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 8, 2026

Package Tarball

aws-agentcore-0.13.1.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-1188-tarball/aws-agentcore-0.13.1.tgz

@notgitika
Copy link
Copy Markdown
Contributor Author

notgitika commented May 9, 2026

Empirical verification

Reproduced the e2e test flow manually against my account:

  1. Created project TestABFix (runtime name == project name, matching the e2e test)
  2. Added runtime endpoints prod/staging, evaluator, online eval configs, target-based AB test with --enable
  3. Deployed

Result with the fix:

{
  "executionStatus": "RUNNING",
  "variants": [
    {"name": "C", "variantConfiguration": {"target": {"name": "TestABFix-TestABFix-prod"}}, "weight": 90},
    {"name": "T1", "variantConfiguration": {"target": {"name": "TestABFix-TestABFix-staging"}}, "weight": 10}
  ]
}

The AB test transitions to RUNNING immediately after deploy. Target names are correctly double-prefixed (TestABFix-TestABFix-prod), matching what post-deploy-http-gateways.ts creates on the gateway.

Without the fix, resolveTargetName would have returned TestABFix-prod (no prefix added due to the false-positive startsWith check), the targets wouldn't match any gateway target, and the test would stay NOT_STARTED — exactly what the e2e suite has been seeing.

@notgitika notgitika merged commit eb2e147 into main May 9, 2026
27 checks passed
@notgitika notgitika deleted the fix/ab-test-target-name-resolution branch May 9, 2026 01:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI Failure: E2E Tests (Full Suite) E2E test failures in shards 3/6 and 5/6 — regression since evo preview features commit

3 participants