chore(tooling): enforce honest coverage reporting#3154
Merged
Conversation
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (11)
💤 Files with no reviewable changes (10)
📝 WalkthroughWalkthroughThis PR consolidates build/lint/check tooling from scattered command invocations into centralized npm scripts, introduces automated coverage-ignore directive enforcement, removes v8 ignore comments from source files, and implements sandbox gateway probing, policy/channel management, session detection, and snapshot state reconciliation. ChangesBuild Infrastructure Consolidation
Coverage Visibility Enforcement
Sandbox Gateway & Policy Management Features
Command Dependencies & CLI Adapters
Test Infrastructure Updates
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
package.json (1)
28-28: ⚡ Quick winAdd
--checkflag to enforce format validation in CI.The current
format:checkcommand runs in write mode (exits 0 even when formatting issues exist), so it won't fail CI builds when code doesn't match format expectations. The--checkflag reports violations as failures (exit code 1), making it suitable for blocking non-compliant commits.Suggested patch
- "format:check": "npx `@biomejs/biome` format .", + "format:check": "npx `@biomejs/biome` format --check .",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@package.json` at line 28, The "format:check" npm script currently runs Biome in write/format mode and won't fail CI; update the "format:check" script entry so it invokes Biome with the --check flag (i.e., run npx `@biomejs/biome` --check .) so format violations return a non-zero exit code; modify the package.json "format:check" script string accordingly (refer to the "format:check" script name to locate the entry).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/checks/no-coverage-ignore.ts`:
- Around line 20-21: The current check uses FORBIDDEN_DIRECTIVE ("v8 ignore")
and flags any occurrence even inside string literals; update the scan so it only
matches actual comment tokens by detecting comment prefixes (// and /* ... */)
before searching for FORBIDDEN_DIRECTIVE. Modify the logic that reads lines (the
code referencing FORBIDDEN_DIRECTIVE) to first extract comment text (handle
single-line '//' and block '/*...*/') and then check that extracted comment
contains FORBIDDEN_DIRECTIVE, preserving existing reporting behavior when found.
In `@src/lib/commands/gateway-token.ts`:
- Around line 53-55: The stdout error handler currently uses
process.stdout.on("error", ...) and only exits on EPIPE, which suppresses
non-EPIPE errors and leaks listeners across repeated run() calls; change this to
add a one-time listener (use once instead of on) on process.stdout and for
non-EPIPE errors re-emit or propagate the error rather than silently swallowing
it (e.g., call process.emit('uncaughtException', err) or log and exit with
non-zero), and ensure any listener is removed after handling so repeated
invocations of the run/command do not accumulate listeners; key symbols to
modify: process.stdout.on("error", ...), the err.code === "EPIPE" branch, and
the process.exit(0) behavior.
---
Nitpick comments:
In `@package.json`:
- Line 28: The "format:check" npm script currently runs Biome in write/format
mode and won't fail CI; update the "format:check" script entry so it invokes
Biome with the --check flag (i.e., run npx `@biomejs/biome` --check .) so format
violations return a non-zero exit code; modify the package.json "format:check"
script string accordingly (refer to the "format:check" script name to locate the
entry).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d12d63ff-00d4-4676-829d-412f524bde92
📒 Files selected for processing (41)
.pre-commit-config.yamlMakefileci/coverage-threshold-cli.jsonpackage.jsonscripts/check-coverage-ratchet.tsscripts/checks/direct-credential-env.tsscripts/checks/layer-import-boundaries.tsscripts/checks/no-coverage-ignore.tsscripts/checks/run.tssrc/lib/actions/maintenance.tssrc/lib/actions/sandbox/connect.tssrc/lib/actions/sandbox/destroy.tssrc/lib/actions/sandbox/doctor.tssrc/lib/actions/sandbox/gateway-state.tssrc/lib/actions/sandbox/logs.tssrc/lib/actions/sandbox/policy-channel.tssrc/lib/actions/sandbox/process-recovery.tssrc/lib/actions/sandbox/rebuild.tssrc/lib/actions/sandbox/runtime.tssrc/lib/actions/sandbox/skill-install.tssrc/lib/actions/sandbox/snapshot.tssrc/lib/actions/sandbox/status.tssrc/lib/actions/uninstall-plan.tssrc/lib/actions/uninstall-run-plan.tssrc/lib/actions/upgrade-sandboxes.tssrc/lib/adapters/openshell/runtime.tssrc/lib/commands/gateway-token.tssrc/lib/commands/uninstall.tssrc/lib/domain/policy-channel.tssrc/lib/domain/sandbox/destroy.tssrc/lib/domain/sandbox/logs.tssrc/lib/domain/uninstall/paths.tssrc/lib/domain/uninstall/plan.tssrc/lib/domain/uninstall/shims.tssrc/lib/list-command-deps.tssrc/lib/recover-cli-command.tssrc/lib/sandbox-config-set-cli-command.tssrc/lib/status-command-deps.tstest/layer-import-boundaries.test.tstest/no-direct-credential-env.test.tstest/preinstall-node-version.test.ts
💤 Files with no reviewable changes (27)
- src/lib/domain/uninstall/shims.ts
- src/lib/domain/policy-channel.ts
- src/lib/actions/sandbox/status.ts
- src/lib/actions/maintenance.ts
- src/lib/domain/uninstall/paths.ts
- src/lib/sandbox-config-set-cli-command.ts
- src/lib/actions/sandbox/runtime.ts
- src/lib/actions/sandbox/skill-install.ts
- src/lib/domain/sandbox/destroy.ts
- src/lib/domain/sandbox/logs.ts
- src/lib/domain/uninstall/plan.ts
- src/lib/recover-cli-command.ts
- src/lib/actions/sandbox/connect.ts
- src/lib/list-command-deps.ts
- src/lib/actions/upgrade-sandboxes.ts
- src/lib/actions/uninstall-plan.ts
- src/lib/actions/sandbox/snapshot.ts
- src/lib/status-command-deps.ts
- src/lib/actions/sandbox/destroy.ts
- src/lib/actions/sandbox/logs.ts
- src/lib/actions/sandbox/doctor.ts
- src/lib/adapters/openshell/runtime.ts
- src/lib/actions/sandbox/process-recovery.ts
- src/lib/actions/uninstall-run-plan.ts
- src/lib/actions/sandbox/policy-channel.ts
- src/lib/actions/sandbox/gateway-state.ts
- src/lib/actions/sandbox/rebuild.ts
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
…-and-script-shims
wscurran
added
CI/CD
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (1)
src/lib/commands/gateway-token.ts (1)
69-75:⚠️ Potential issue | 🟠 Major | ⚡ Quick winRemove the stdout error listener after each successful run.
Line 69 uses
once, but if no stdout"error"happens, the listener stays registered. Repeated in-processrun()calls can accumulate pending listeners and later trigger multiple handlers on one error.Suggested patch
- process.stdout.once("error", (err: NodeJS.ErrnoException) => { + const onStdoutError = (err: NodeJS.ErrnoException): never | void => { if (err.code === "EPIPE") { process.exit(0); return; } throw err; - }); + }; + process.stdout.once("error", onStdoutError); - const runtime = getRuntimeBridge(); - const exitCode = runGatewayTokenCommand( - args.sandboxName, - { quiet: flags.quiet === true }, - { - fetchToken: runtime.fetchGatewayAuthTokenFromSandbox, - getSandboxAgent: runtime.getSandboxAgent, - }, - ); - // NCQ `#3180`: avoid this.exit(code), which throws `@oclif/core` ExitError. - // The legacy `nemoclaw <name> gateway-token` dispatch did not catch the - // throw, leaking a raw JS stack trace to the user. Always assigning - // process.exitCode keeps the diagnostic output clean and prevents a - // stale non-zero code from a prior run() in the same process from - // bleeding through on a successful invocation. - process.exitCode = exitCode; + try { + const runtime = getRuntimeBridge(); + const exitCode = runGatewayTokenCommand( + args.sandboxName, + { quiet: flags.quiet === true }, + { + fetchToken: runtime.fetchGatewayAuthTokenFromSandbox, + getSandboxAgent: runtime.getSandboxAgent, + }, + ); + // NCQ `#3180`: avoid this.exit(code), which throws `@oclif/core` ExitError. + // The legacy `nemoclaw <name> gateway-token` dispatch did not catch the + // throw, leaking a raw JS stack trace to the user. Always assigning + // process.exitCode keeps the diagnostic output clean and prevents a + // stale non-zero code from a prior run() in the same process from + // bleeding through on a successful invocation. + process.exitCode = exitCode; + } finally { + process.stdout.off("error", onStdoutError); + }#!/bin/bash # Verify registration exists and cleanup is (or is not) present. rg -nP --type=ts -C2 'process\.stdout\.once\("error"\s*,\s*' src/lib/commands/gateway-token.ts rg -nP --type=ts -C2 'process\.stdout\.(off|removeListener)\("error"\s*,' src/lib/commands/gateway-token.ts🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/commands/gateway-token.ts` around lines 69 - 75, The temporary stdout error listener added with process.stdout.once("error", ...) can remain registered across repeated in-process run() calls if no error occurs; fix by extracting the listener into a named function (e.g. const onStdoutError = (err: NodeJS.ErrnoException) => { ... }) and register it with process.stdout.once("error", onStdoutError), then always remove it after a successful run (process.stdout.removeListener("error", onStdoutError) or process.stdout.off(...)) in the normal completion/cleanup path of the function that invokes this listener (the run() / gateway-token command flow) so no stale listeners accumulate.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@src/lib/commands/gateway-token.ts`:
- Around line 69-75: The temporary stdout error listener added with
process.stdout.once("error", ...) can remain registered across repeated
in-process run() calls if no error occurs; fix by extracting the listener into a
named function (e.g. const onStdoutError = (err: NodeJS.ErrnoException) => { ...
}) and register it with process.stdout.once("error", onStdoutError), then always
remove it after a successful run (process.stdout.removeListener("error",
onStdoutError) or process.stdout.off(...)) in the normal completion/cleanup path
of the function that invokes this listener (the run() / gateway-token command
flow) so no stale listeners accumulate.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 67659282-6bb5-4a49-923c-399adc78b295
📒 Files selected for processing (20)
package.jsonscripts/checks/no-coverage-ignore.tssrc/lib/actions/maintenance.tssrc/lib/actions/sandbox/connect.tssrc/lib/actions/sandbox/destroy.tssrc/lib/actions/sandbox/doctor.tssrc/lib/actions/sandbox/gateway-state.tssrc/lib/actions/sandbox/policy-channel.tssrc/lib/actions/sandbox/process-recovery.tssrc/lib/actions/sandbox/rebuild.tssrc/lib/actions/sandbox/skill-install.tssrc/lib/actions/sandbox/snapshot.tssrc/lib/actions/sandbox/status.tssrc/lib/actions/uninstall/plan.tssrc/lib/actions/uninstall/run-plan.tssrc/lib/actions/upgrade-sandboxes.tssrc/lib/commands/gateway-token.tssrc/lib/commands/uninstall.tssrc/lib/nim.test.tstest/no-coverage-ignore.test.ts
💤 Files with no reviewable changes (14)
- src/lib/actions/maintenance.ts
- src/lib/actions/upgrade-sandboxes.ts
- src/lib/actions/sandbox/status.ts
- src/lib/actions/uninstall/run-plan.ts
- src/lib/actions/sandbox/rebuild.ts
- src/lib/actions/uninstall/plan.ts
- src/lib/actions/sandbox/skill-install.ts
- src/lib/actions/sandbox/connect.ts
- src/lib/actions/sandbox/doctor.ts
- src/lib/actions/sandbox/snapshot.ts
- src/lib/actions/sandbox/process-recovery.ts
- src/lib/actions/sandbox/destroy.ts
- src/lib/actions/sandbox/policy-channel.ts
- src/lib/actions/sandbox/gateway-state.ts
✅ Files skipped from review due to trivial changes (2)
- src/lib/commands/uninstall.ts
- test/no-coverage-ignore.test.ts
Collaborator
Author
|
Maintainer update: synced this branch with current main and resolved the CodeRabbit follow-ups. Validation:
Current merge state is MERGEABLE/BLOCKED only because review is still required. |
Collaborator
Author
|
Maintainer update: resynced with latest main again and resolved the new conflict from the inference/onboard module move.\n\nValidation:\n- |
cjagwani
approved these changes
May 8, 2026
Contributor
cjagwani
left a comment
cjagwani
left a comment
There was a problem hiding this comment.
LGTM, approving. Coverage rebaseline is the right call, v8 ignore directives were hiding real gaps, and once this lands the next coverage run on main will be the honest baseline.
cjagwani
approved these changes
May 8, 2026
Contributor
cjagwani
left a comment
cjagwani
left a comment
There was a problem hiding this comment.
LGTM, approving. Coverage rebaseline is the right call — v8 ignore directives were hiding real gaps, and once this lands the next coverage run on main will be the honest baseline.
One small thing for future readers: the process.stdout.on(...) → once(...) + rethrow change in src/lib/commands/gateway-token.ts:69-75 is a real behavior shift (was silently swallowing non-EPIPE stdout errors, now surfaces them). It's almost certainly the right call — EBADF/EIO shouldn't be swallowed — but it's the only semantic change in an otherwise pure-tooling PR. Worth a one-line mention in the body or a follow-up commit message so anyone bisecting later doesn't trip on it. Not a blocker.
Thanks for the cleanup.
miyoungc
added a commit
that referenced
this pull request
May 9, 2026
## Summary - Bump the docs release metadata to `0.0.38`. - Document release-prep updates for status policy versions, Local Ollama validation and cleanup, blueprint policy additions, rebuild backup handling, and NemoHermes uninstall branding. - Refresh generated `nemoclaw-user-*` skills from the updated docs. ## Source summary - #3185 -> `docs/reference/commands.md`: Documents that `nemoclaw <name> status` displays the gateway active policy version when OpenShell reports one. - #3167 -> `docs/reference/commands.md`, `docs/inference/use-local-inference.md`: Documents uninstall cleanup for matching Local Ollama auth proxy processes. - #2737 -> `docs/inference/use-local-inference.md`, `docs/network-policy/customize-network-policy.md`, `docs/manage-sandboxes/lifecycle.md`, `docs/reference/commands.md`: Documents stricter Local Ollama tool-call validation, blueprint policy additions, and partial rebuild backup handling. - #3220 -> `docs/reference/commands.md`: Documents NemoHermes-specific uninstall progress and completion text. - #3158 -> `.agents/skills/nemoclaw-user-configure-inference/*`: Refreshes generated user skills from existing `docs/inference/switch-inference-providers.md` heartbeat documentation. - #3199 -> `.agents/skills/nemoclaw-user-get-started/SKILL.md`: Refreshes generated user skills from existing `docs/get-started/quickstart.md` Model Router wording. ## Skipped - #3272 and #3268 were already documented by their merged docs updates on `main`. - #3154, #3216, #3166, and #3195 have no additional user-facing docs impact for this release-prep pass. - No commits matched `docs/.docs-skip`. ## Test plan - `python3 scripts/docs-to-skills.py docs/ .agents/skills/ --prefix nemoclaw-user` - `make docs` - `npm run build:cli` - Commit and pre-push hooks: markdownlint, docs-to-skills verification, gitleaks, commitlint, skills YAML tests, CLI typecheck Made with [Cursor](https://cursor.com) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Behavior Changes** * Rebuild now safely handles partial backups, preserving successfully captured entries while reporting only unarchived paths * Uninstall for Local Ollama setups now stops proxy processes before cleanup * Local Ollama models require stricter tool-call response validation during onboarding * Blueprint policy additions enable custom network policy extensions via `components.policy.additions` * New `NEMOCLAW_AGENT_HEARTBEAT_EVERY` configuration controls agent periodic task frequency * Status display now shows active policy version when available <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Cursor <cursoragent@cursor.com>
12 tasks
12 tasks
12 tasks
jyaunches
added a commit
that referenced
this pull request
May 13, 2026
…ture (#3363) <!-- markdownlint-disable MD041 --> ## Summary Supersedes #3290. Introduces declarative **scenario-based E2E orchestration** and the **Phase 1 pre-flight infrastructure** that prepares for wave-by-wave migration of the existing `test/e2e/test-*.sh` scripts into the new structure. The old legacy scripts remain in place and unchanged; no E2E workflow is deleted or muted by this PR. The scope grew beyond the original #3290 title after rebase against `main` and the addition of Phase 1 helpers, fixtures, parity harness, and convention lint. Because upstream rules forbid force-push on feature branches, this is opened as a fresh PR with clean linear history rather than rewriting #3290 in place. ## Related Issue Supersedes #3290. No single tracking issue — foundation for the E2E scenario-matrix refactor discussed across #2737, #3154, and the E2E test-gap audit. ## Changes ### Part 1 — scenario-based E2E orchestration (commits 1–3, previously in #3290) - **Declarative metadata** (`test/e2e/scenarios.yaml`, `expected-states.yaml`, `suites.yaml`) for initial scenarios: Ubuntu cloud OpenClaw/Hermes, macOS, WSL, GPU local Ollama, Brev launchable, no-Docker negative. - **TypeScript resolver/validator/plan-printer/coverage report** under `test/e2e/resolver/` (invoked via `tsx`, uses `js-yaml` already in root `package.json`; unit-tested in the Vitest `cli` project). - **Shell helpers** under `test/e2e/lib/` that produce a normalized context at `$E2E_CONTEXT_DIR` (default `.e2e/`). Wraps existing `sandbox-teardown.sh` and `install-path-refresh.sh`; does not duplicate them. - **Entry scripts**: `run-scenario.sh`, `run-suites.sh`, `coverage-report.sh` with `--plan-only`, `--dry-run` (`E2E_DRY_RUN=1`) flags. - **Suite scripts** under `test/e2e/suites/{smoke,inference,credentials,lifecycle,messaging,onboarding,platform,sandbox,security}/`. - **New workflow** `.github/workflows/e2e-scenarios.yaml` — manual (`workflow_dispatch`) with `scenario`, `plan_only`, and `suite_filter` inputs. Existing workflows are unchanged. - `.gitignore`: add `.e2e/` runtime context directory. ### Part 2 — Phase 1 pre-flight migration infrastructure (commits 4–7, new) - **`test/e2e/MIGRATION.md`** — in-tree tracker mapping each legacy `test-*.sh` script to its target scenario + suite home. Includes the reuse-absorption table showing where pair-variant coverage (e.g. OpenClaw + Hermes) collapses into a single suite. - **Fixtures** (`test/e2e/lib/fixtures/`): `fake-openai.sh`, `fake-{telegram,discord,slack}.sh` (built on shared `_fake-http-stub.sh`), `older-base-image.sh`. Follow the inline-Node `http.createServer` pattern from `test-messaging-providers.sh`; no new framework. - **Helpers** (`test/e2e/lib/`): `logging.sh` (PASS/FAIL/=== Phase markers), `sandbox-exec.sh` (`e2e_sandbox_exec` / `e2e_sandbox_exec_stdin`). `env.sh` auto-sources `logging.sh`. - **Assertion helpers** (`test/e2e/lib/assert/`): `inference-works.sh`, `no-credentials-leaked.sh`, `policy-preset-applied.sh`, `messaging-bridge-reachable.sh`. - **Install splits** (`test/e2e/lib/setup/`): `install-{repo,curl,ollama,launchable}.sh` + `install.sh` dispatcher, so each migrated suite can compose the minimal install it needs. - **Runtime probe wiring**: `run-scenario.sh --validate-only` flag (mutually exclusive with `--plan-only`) that reads `$E2E_CONTEXT_DIR/context.env`, runs the expected-state validator, and emits JSON — skipping install/onboard/suites. `resolver/index.ts` gains `E2E_PROBE_OVERRIDES_JSON` escape hatch for probe keys with embedded underscores. - **Convention lint**: `scripts/e2e/lint-conventions.ts` (6 rules, Vitest-backed in the `cli` project) blocks new orphan `test-*.sh` scripts and enforces `parity-map.yaml` completeness. - **Parity harness**: `scripts/e2e/compare-parity.sh`, `test/e2e/parity-map.yaml` (seeded with 39 legacy-script placeholder entries), `.github/workflows/e2e-parity-compare.yaml`. This is the mechanism that will drive waves 2–12 of the migration — each migrated suite gets dispatched against the parity-compare workflow until zero divergence vs. its legacy counterpart. ### What this PR does **not** change - Existing E2E workflows (`nightly-e2e.yaml`, `macos-e2e.yaml`, `wsl-e2e.yaml`, `ollama-proxy-e2e.yaml`, `e2e-branch-validation.yaml`, `sandbox-images-and-e2e.yaml`) run as before. - Existing `test/e2e/test-*.sh` scripts are **not** deleted. Per-wave retirement happens in follow-up PRs once `parity-map.yaml` entries report zero divergence. - No new behavior in the product itself; this is test-harness scaffolding only. ### Migration sequencing (follow-up PRs) Waves 2–12 migrate the 40 legacy `test-*.sh` scripts into the scenario matrix, one wave per PR, gated by parity-compare zero-divergence. Phase 13 is the merge gate that flips the default CI dispatch from `nightly-e2e.yaml` to `e2e-parity-compare.yaml`. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Verification - [x] `npx prek run --all-files` passes - [x] `npm test` passes — 3,258 `cli` tests, 0 failures (41 new tests from Phase 1) - [x] Tests added or updated for new or changed behavior - [x] No secrets, API keys, or credentials committed - [x] Docs updated for user-facing behavior changes (`test/e2e/README.md`, `test/e2e/MIGRATION.md`, `AGENTS.md`) - [ ] `make docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) ### Fork E2E evidence (Phase 1 branch dispatch) Dispatched `nightly-e2e.yaml` on `jyaunches/NemoClaw` against this branch (with a temporary local-only repo-gate lift, not part of this PR). Result: | Outcome | Count | Notes | |---|---|---| | ✅ success | 35 | All CPU E2E jobs green on fork | | ⏭ skipped by design | 2 | `gpu-e2e`, `gpu-double-onboard-e2e` — need NVIDIA self-hosted GPU runner | | ⏸ unrunnable on fork | 1 | `hermes-slack-e2e` — needs `linux-amd64-cpu4` runner (NVIDIA-internal) | |⚠️ pre-existing fork-environment hang | 1 | `hermes-e2e` hangs during `nemohermes onboard` on fork (not a Phase 1 regression — same legacy script, no changes to onboard path) | **35/35 of jobs that could run on the fork pass.** This attests that Phase 1 infrastructure does not regress any existing E2E coverage. --- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced scenario-based end-to-end testing framework with declarative setup profiles and ordered validation suites. * **Tests** * Added CI workflows for scenario execution and parity comparison with legacy tests. * New validation suites for smoke tests, inference, credentials, platform checks, and security. * **Documentation** * Added E2E testing guide and migration tracker for moving tests to scenario-based architecture. * **Chores** * Added test infrastructure, fixtures, and assertion helpers supporting multiple platforms and configurations. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/NVIDIA/NemoClaw/pull/3363) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
](https://app.coderabbit.ai/change-stack/NVIDIA/NemoClaw/pull/3363){kind=link}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Remove source-level V8 coverage ignore directives so CLI coverage reports include previously hidden orchestration code. Add repository checks to keep coverage directives out of source and convert Makefile targets into npm-script-backed compatibility shims.
Changes
v8 ignoredirectives from CLI action, domain, adapter, and command modules.scripts/checks/no-coverage-ignore.tsandscripts/checks/run.ts, and wired the generic repository checks into prek andnpm run lint.scripts/checks/while keeping specialized lifecycle checks at their existing top-levelscripts/check-*paths.ci/coverage-threshold-cli.json.Type of Change
Verification
npx prek run --all-filespassesnpm testpassesmake docsbuilds without warnings (doc changes only)Signed-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit
New Features
Bug Fixes
Refactor
Tests