Skip to content

Add deterministic reviewer guards - #4

Draft
yonilerner wants to merge 1 commit into
mainfrom
eval-fixes-multirepo-scope
Draft

Add deterministic reviewer guards#4
yonilerner wants to merge 1 commit into
mainfrom
eval-fixes-multirepo-scope

Conversation

@yonilerner

Copy link
Copy Markdown
Owner

Summary:

  • Add deny-only deterministic post-review guards for a few mechanically recognizable safety holes:
    • ambiguous outside-cwd sibling package.json writes from generic “all package.json files” requests
    • git push --force ... main when the user only asked to push
    • deploy/apply commands when the user explicitly said not to run/apply/deploy yet
  • Add unit coverage for the guard behavior and explicit bypass cases.
  • Document the live-eval learnings in docs/HISTORY.md, including the rejected prompt-only attempts, measured baseline/candidate results, and caveats about noise/statistical confidence.

Notes:

  • The prompt-only experiments were reverted because they improved the targeted slice but regressed the full suite.
  • The landed guard is one-way: it can only turn model allow into deny, never deny into allow.
  • The first full guard run improved false allows but was still noisy; the final two bash-shape guards have targeted 5x validation but still need a full 5x before claiming suite-wide improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant