feat(testing-agentforce): NGT (Agentforce Studio) authoring + result-shape support#28
Open
BunnyBlit wants to merge 7 commits into
Open
feat(testing-agentforce): NGT (Agentforce Studio) authoring + result-shape support#28BunnyBlit wants to merge 7 commits into
BunnyBlit wants to merge 7 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Mode B support for NGT (
AiTestingDefinition) to/testing-agentforcewhile keeping the legacy (AiEvaluationDefinition) path fully working. The skill now detects which runner the org/project supports, picks the right path automatically, and surfaces docs for both shapes. Verified end-to-end against a real org.What's in this PR
sourceApiVersionfromsfdx-project.json; selects NGT when>= 66.0, legacy below. Falls back gracefully if the org rejects NGT metadata even on a 66.0 project.BotVersion.Status = 'Active'before running tests. Without this,sf agent test runsilently returnstestCases: []for NGT or all-RETRY for legacy. Documented + scripted in SKILL.md.references/ngt-batch-testing.md, new file, ~520 lines) — 11-scorer catalog, multi-input cases,task_resolutionconversation-history rules, multi-agent subjects,--from-definitionround-trip caveats, four hard authoring rules with deploy-time error signatures.references/test-report-format.md) — new Legacy Result JSON Shape section with the XML→JSON assertion-name mapping (topic_sequence_match → topic_assertion, etc.) and the no-active-version failure shape. NGT result shape withtestScorerResults[]container, JSON-encodedsubjectResponse/scorerResponse, per-grade field presence.Integration testing
Validated end-to-end against
NGA_ASA_Orchestrator(NGT) andOrderHelp_Legacy(legacy) on a live OrgFarm org. Both paths:sf project deploy startsf agent test runTest plan
references/ngt-batch-testing.mdproduce the documented error when violatedtestCases[].testResults[]withname∈ {topic_assertion,actions_assertion,output_validation}testCases[].testScorerResults[]with JSON-encodedscorerResponsecontainingstatus/score/reasoning/actualValue/expectedValue