Skip to content

feat(testing-agentforce): NGT (Agentforce Studio) authoring + result-shape support#28

Open
BunnyBlit wants to merge 7 commits into
SalesforceAIResearch:mainfrom
BunnyBlit:feat/ngt-cli-driver
Open

feat(testing-agentforce): NGT (Agentforce Studio) authoring + result-shape support#28
BunnyBlit wants to merge 7 commits into
SalesforceAIResearch:mainfrom
BunnyBlit:feat/ngt-cli-driver

Conversation

@BunnyBlit

Copy link
Copy Markdown

Summary

Adds Mode B support for NGT (AiTestingDefinition) to /testing-agentforce while keeping the legacy (AiEvaluationDefinition) path fully working. The skill now detects which runner the org/project supports, picks the right path automatically, and surfaces docs for both shapes. Verified end-to-end against a real org.

What's in this PR

  • Detection probe — reads sourceApiVersion from sfdx-project.json; selects NGT when >= 66.0, legacy below. Falls back gracefully if the org rejects NGT metadata even on a 66.0 project.
  • Pre-Run Probe — queries BotVersion.Status = 'Active' before running tests. Without this, sf agent test run silently returns testCases: [] for NGT or all-RETRY for legacy. Documented + scripted in SKILL.md.
  • NGT authoring reference (references/ngt-batch-testing.md, new file, ~520 lines) — 11-scorer catalog, multi-input cases, task_resolution conversation-history rules, multi-agent subjects, --from-definition round-trip caveats, four hard authoring rules with deploy-time error signatures.
  • Result-shape reference (references/test-report-format.md) — new Legacy Result JSON Shape section with the XML→JSON assertion-name mapping (topic_sequence_match → topic_assertion, etc.) and the no-active-version failure shape. NGT result shape with testScorerResults[] container, JSON-encoded subjectResponse/scorerResponse, per-grade field presence.
  • Troubleshooting — entries for the four NGT authoring deploy errors (DeveloperName, name/--api-name mismatch, 100-char description cap, silent empty cases), plus the legacy RETRY-mode failure analog.
  • SKILL.md — Mode B forks early on the detection-probe outcome; CRITICAL callouts for the four hard authoring rules; Pre-Run Probe section before deploy-and-run.

Integration testing

Validated end-to-end against NGA_ASA_Orchestrator (NGT) and OrderHelp_Legacy (legacy) on a live OrgFarm org. Both paths:

  • ran the detection probe correctly
  • authored a valid YAML spec
  • compiled YAML → metadata XML
  • deployed via sf project deploy start
  • ran the suite via sf agent test run
  • returned results matching the documented JSON envelope

Test plan

  • Detection probe picks NGT on a 66.0 project, legacy on a 64.0 project
  • All four CRITICAL authoring rules in references/ngt-batch-testing.md produce the documented error when violated
  • Pre-Run Probe halts the run when the subject bot has no Active BotVersion
  • Legacy result envelope verified: testCases[].testResults[] with name ∈ {topic_assertion, actions_assertion,
    output_validation}
  • NGT result envelope verified: testCases[].testScorerResults[] with JSON-encoded scorerResponse containing
    status/score/reasoning/actualValue/expectedValue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant