feat: add systematic local Luna eval suites - #325
Merged
Conversation
Record the merged PR, merge commit, CI evidence, and current deployment state before evaluating the next increment.
Detail the Luna-only local suite and paired-comparison increment, including per-tool call counts and the current workload inventory.
Derive deterministic per-tool logical call counts from persisted metrics sequences and expose status-aware run comparison deltas without changing the metrics schema.
Keep per-workload calls-by-tool comparison state explicitly unknown whenever either side lacks logical telemetry, including empty workloads.
Validate the curated workload inventory and expose deterministic suite selection for later local execution.
Keep the measurement harness root responsible for workloads and reporting while allowing local eval launches, skills, guidance, and Git metadata to come from a target checkout.
Run descriptor and full profile shards concurrently with validated suite artifacts, content identities, partial failure accounting, and aggregate metrics.
Give each suite artifact a unique UUID, enforce the fixed profile and shard matrix, and surface child workload failures in shard status.
Run baseline and candidate suites through the shared Luna harness and produce validated offline and live comparison artifacts with containment and matched-cohort metrics.
Suppress ordered tool-sequence deltas when comparison cells are ineligible and reuse the report telemetry consistency check for suite and comparison tool aggregates.
Expose local Luna suite, pair, and offline comparison commands with strict argument validation, documented artifact semantics, and no public package impact.
Keep workload content identities path-sorted, require explicit shard outcomes, and expose complete comparison diagnostics for local regression analysis.
Contain report reconstruction, reject inconsistent suite identities, preserve clean attribution semantics, and normalize aggregate tool summaries before comparison.
Treat suite execution mode as a compatibility dimension, keep path tests platform-aware, and align the release fragment with repository format.
Derive target-root parsing expectations with node path helpers so the harness ownership regression remains valid on Windows as well as POSIX.
Document the measured Luna suite and paired-comparison evidence, including exact tool and workload metrics, and mark the local Phase 2 plan complete while retaining later CI and service decisions.
Keep CLI artifact assertions platform-aware and avoid materializing deliberately unsafe manifest paths before validation, including Windows drive paths.
Create disposable per-workload homes while preserving only caller-supplied Codex auth. Remove the full MCP CLI shim, enforce neutral result reports, and fail closed on external guidance reads or MCP CLI fallback with redacted evidence.
Keep the workload focused on its factual questions and source evidence while removing legacy product-specific and self-reporting instructions.
Document the neutral eval artifacts and defer usefulness or quality claims to a later grading stage. Export the disposable isolation contract used by eval tests.
Allow managed CODEX_HOME state while rejecting root global instructions, and disable Codex external app/plugin surfaces for every eval command. Update isolation docs with the partial canary evidence and add regression coverage for managed homes and all command profiles.
Align the validator name with its responsibility: a dedicated eval CODEX_HOME contains authentication and Codex-managed runtime state, not only authentication material.
Keep the agent in disposable home and config roots while passing only caller home values to the trusted MCP child for keychain lookup. Redact those runtime values after config consumption and document the partial authentication canary evidence.
Keep host-home values available to the trusted MCP child while limiting their persisted redaction to runtime configuration copies and command metadata. Preserve target attribution and agent evidence, with credential secrets still redacted across artifacts.
Compare existing guidance paths through canonical filesystem locations so macOS aliases do not reject workspace-installed skills. Keep lexical containment for missing or Windows-shaped paths and record the v3 canary evidence.
Close the Phase 2 isolation correction with the clean authenticated Luna descriptor/full canary. Keep the contaminated 42-cell behavior comparison diagnostic only and leave Phase 3 gated on product decisions.
Require a dedicated validated CODEX_HOME for live Codex skills as well as MCP evals, while keeping dry runs available without authentication. Extend the prelaunch and injected-command coverage and document that the v4 live canary covered MCP only.
Pass host XDG_CONFIG_HOME and APPDATA only to the trusted GitHits MCP child and redact them from persisted runtime configs. Also accept Codex home directory symlinks and normalize bare guidance references without weakening root guidance rejection.
Give the trusted GitHits MCP child platform-effective config roots when callers omit optional variables, while keeping agent homes disposable and redacting derived paths from artifacts. Cover the default roots in tests and documentation.
Allow native Windows absolute guidance paths through canonical containment while keeping Windows-shaped paths external on non-Windows hosts. Redact JSON-escaped host config roots in persisted runtime configs and cover both cases with focused tests.
Expand shared runtime redaction values with JSON-escaped forms so Codex command metadata receives the same host-path protection as persisted runtime configs. Extend the Windows-shaped dry-run regression to both workload and run command artifacts.
Add the neutral and GitHits intent profiles with closed MCP scenario validation, exact prompt identity hashing, and schema-v1 metrics normalization into the v2 record shape. Preserve legacy usage and tool semantics while exposing scenario identity in one-off reports.
Replace the fixed descriptor/full suite matrix with explicit discovery, intent, and full scenario cells. Normalize legacy suite artifacts and preserve matched comparison cohorts across scenario-aware identities.
Reuse the workload runner's dedicated Codex home validation and disposable environment for interactive sessions. Suppress ambient Codex configuration and preserve trusted host auth roots for the local MCP child.
Document the closed discovery, intent, and full contracts, schema-v2 identity and compatibility, scenario-aware suite defaults, and Codex-only interactive isolation boundaries.
Require the descriptor/full v1 matrix and shard contract, and reject null or missing profiles instead of inferring an intent scenario.
Keep interactive Codex compatible with the current CLI while preserving causal isolation through a strict dedicated home contract. Reject ambient skills and unsupported config keys before workspace setup, without changing non-interactive eval behavior.
Mark the Phase 2 scenario and Codex interactive-isolation correction complete, document the verified Luna artifacts and manual session boundary, and keep Phase 3 product decisions explicit. Correct historical schema and interactive CLI guidance without changing runtime behavior.
Propagate the selected agent CLI version into one-off report JSON and terminal identity, and warn on same-agent version drift without changing report schema compatibility.
Share direct Codex skills preflight across noninteractive and interactive paths, keep session isolation metadata truthful, and remove stale suite and metrics interfaces. Update maintainer documentation and regression coverage for the corrected contracts.
Document the shared direct-skills preflight across measured noninteractive and interactive Codex paths, close the Opus F1 contamination gap, and update final focused and full test totals.
Share the direct-skills preflight across Codex paths, document supplied dry-run home validation, and remove the unreachable runtime intent default while preserving parser-owned neutral defaults.
Make the workload preflight sentence unambiguous about which CODEX_HOME and instruction-file states are rejected.
Resolve the consumed eval-metrics release fragment under a fresh scenario-metrics name and refresh merged-tree verification evidence.
Compare isolated environment roots with node:path dirname so the interactive Codex assertion works on POSIX and Windows.
jlitola
marked this pull request as ready for review
August 31, 2026 07:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
discovery,intent, andfulleval scenarios with stable identity in run, metrics, suite, comparison, and human reportsCODEX_HOME; reject root global instructions and every direct skill except.systembefore measured or interactive use--ignore-user-configfor config suppression while explicitly validating direct skills, disabling external apps/plugins, and rejecting GitHits CLI fallbackScenario contract
Use GitHits for this task.nudgeHistorical schema-v1 descriptor cells normalize only to discovery and historical full cells only to full. Missing, null, or other profiles are rejected rather than inferred as intent.
Corrected live evidence
The earlier descriptor/full paid comparison remains capacity/cost evidence only because global skills contaminated behavior.
Accepted corrected Luna evidence:
githitsMCP connected with 18 toolscode_grep; 798,452 ms wall timeThe bounded intent package probe made three MCP calls but produced an inconclusive/low-confidence final answer. Current evidence proves harness isolation and tool execution, not answer quality.
Validation
bun test: 3,605 passed, 0 failed, 12,040 expectations across 186 filesbun run typecheckbun run format:check— 442 filesbun run lint— 442 filesbun run buildgit diff --checknode:pathsemanticsCODEX_HOME/skillsrejection for measured and interactive paths,.system-only acceptance, no-home dry-run acceptance, and truthful session metadataScope
This PR establishes reliable local Luna evaluation and metrics. It does not add scheduled paid CI, third-party service export/history, Haiku, or answer-quality grading. Those remain Phase 3 product decisions.