Skip to content

feat(eval): add read-only Eval TUI - #1920

Merged
jariy17 merged 2 commits into
refactorfrom
feat/eval-readonly-tui
Aug 5, 2026
Merged

feat(eval): add read-only Eval TUI#1920
jariy17 merged 2 commits into
refactorfrom
feat/eval-readonly-tui

Conversation

@jariy17

@jariy17 jariy17 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What

Adds the interactive TUI over the existing eval command tree, scoped to read-only screens. It opens a working menu → list → detail → JSON flow, matching runtime/memory.

Screens

Path Screen
eval, eval evaluator, eval online-eval RouterScreen menus
eval evaluator list / online-eval list PaginatedTablePicker
eval evaluator get [/json] ResourceDetailScreen hub → JsonDetail
eval online-eval get [/json] ResourceDetailScreen hub → JsonDetail

Notes for reviewers

  • No Core changes, no new AWS calls, no new deps. Every screen reuses landed components (ResourceDetailScreen from refactor(tui): share resource detail screen #1890, PaginatedTablePicker, JsonDetail, RouterScreen). Only genuinely new code is two pickers + four screen files + three thin menu screens.
  • RouterScreen gains an omit?: string[] prop. The eval command tree still contains the mutation commands, and RouterScreen builds its menu from that tree. Without filtering, a mutation item in the menu would route to nothing and fall through to the *HelpScreen catch-all, which calls exit() and tears down the app. omit hides them from the menu; they remain usable from the CLI. Backwards-compatible (defaults undefined; existing menus unchanged). Read-only, llm-as-a-judge/code-based have only create/update children, so those submenus are omitted too — evaluator shows just get/list.
  • Evaluator kind (LLM-as-a-Judge vs code-based) is derived from which arm of the evaluatorConfig union is populated — the GetEvaluator response has no type field of its own (unlike the list summary's evaluatorType).
  • Online-eval sampling reads rule.samplingConfig.samplingPercentage; sampling/evaluator-count are only on the get response, not the list summary, so they live on the detail screen.

Behaviour change

Mounting withTuiOnEmptyFlagsAndArgs means a bare read command with no flags (e.g. eval evaluator get) now opens the TUI instead of erroring required option '--id' not specified. This matches runtime/memory. Existing leaf tests were updated to assert the headless path under --json (same convention as the memory read-only TUI), plus new tests assert bare read leaves open the TUI.

Testing

  • New: evaluator.screen.test.tsx, online-eval.screen.test.tsx — menus omit mutations, lists render/paginate/select, detail hubs render + route to JSON, retry, empty states.
  • Updated eval leaf tests for the --json headless contract.
  • Full suite green (720 tests), typecheck + lint clean.
  • Verified live against real us-west-2 data through the tui-harness (menu → list → detail hub → JSON, both resources).

Plan docs: read-only spec and Phase 2 (mutations) spec are tracked separately.

eval-tui-demo

Wire the interactive TUI over the existing eval command tree for the
read-only surface: evaluator and online-eval menus, paginated lists, and
resource detail hubs (with a raw JSON view). Mutating commands
(create/update/delete/pause/resume) stay CLI-only for now.

- New pickers: EvaluatorPicker, OnlineEvalPicker (over the existing
  listEvaluators / listOnlineEvaluationConfigs Core calls).
- New screens: eval / evaluator / online-eval menus, list + get(+json).
  get hubs reuse the shared ResourceDetailScreen; evaluator kind is
  derived from the evaluatorConfig union (no type field on the get
  response), online-eval sampling reads rule.samplingConfig.
- RouterScreen gains an `omit` prop so the eval menus hide the mutating
  subcommands. Without it those items would fall through to the
  HelpScreen catch-all, which exits the app.
- eval routers swap createHelpDefault for withTuiOnEmptyFlagsAndArgs +
  renderTui, matching harness/runtime/memory.

Bare read leaves now open the TUI instead of erroring on a missing
required flag; existing leaf tests assert the headless path under --json,
matching the memory read-only TUI convention.
@github-actions github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 5, 2026
@codecov-commenter

codecov-commenter commented Aug 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.28%. Comparing base (206540c) to head (4c0712c).

Additional details and impacted files
@@             Coverage Diff              @@
##           refactor    #1920      +/-   ##
============================================
+ Coverage     96.17%   96.28%   +0.10%     
============================================
  Files           236      245       +9     
  Lines         11778    12105     +327     
============================================
+ Hits          11328    11655     +327     
  Misses          450      450              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 5, 2026
@jariy17
jariy17 marked this pull request as ready for review August 5, 2026 17:38
AlexanderRichey
AlexanderRichey previously approved these changes Aug 5, 2026
aidandaly24
aidandaly24 previously approved these changes Aug 5, 2026
…y-tui

# Conflicts:
#	src/components/Root.tsx
@jariy17
jariy17 dismissed stale reviews from aidandaly24 and AlexanderRichey via 4c0712c August 5, 2026 19:38
@jariy17
jariy17 merged commit ae6e810 into refactor Aug 5, 2026
8 checks passed
@jariy17
jariy17 deleted the feat/eval-readonly-tui branch August 5, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants