Skip to content

Add scored conformance grading and --grade CLI#207

Merged
seanwevans merged 2 commits into
mainfrom
codex/add-conformance-score-feature
May 13, 2026
Merged

Add scored conformance grading and --grade CLI#207
seanwevans merged 2 commits into
mainfrom
codex/add-conformance-score-feature

Conversation

@seanwevans
Copy link
Copy Markdown
Owner

Motivation

  • Replace a single binary “secure/insecure” conformance result with a machine-readable, per‑guarantee score so hosts can prove which PyIsolate guarantees are actually active.

Description

  • Add GradeComponent and GradeReport dataclasses and a new ConformanceSuite.grade() method that computes an 8‑point grade (free‑threading, eBPF‑LSM, cgroup v2, Landlock fallback, no‑GIL extension safety, broker crypto, quota enforcement, crash isolation) with per‑component evidence and active/inactive lists.
  • Implement operational probes for ebpf_lsm, landlock_fallback, no_gil_extension_safety, broker_crypto, and crash_isolation and wire them into the grade computation.
  • Expose --grade on the conformance CLI (python -m pyisolate.conformance --grade) and pyisolate-doctor --grade to emit the scored JSON report via GradeReport.to_json().
  • Document the feature in README.md and add unit tests for grade schema and CLI output in tests/test_conformance.py and tests/test_provenance.py.

Testing

  • Ran pytest tests/test_conformance.py tests/test_provenance.py, which passed (targeted tests succeeded).
  • Ran python -m pyisolate.conformance --grade and pyisolate-doctor --grade, both produced valid JSON grade output although the environment logged expected warnings when BPF/cgroup tooling or writable sysfs is unavailable.
  • Ran project pytest (full suite) and observed unrelated, pre‑existing failures caused by test-suite stubs that replace pyisolate.bpf.manager.BPFManager with a shim whose load() signature does not accept the new mode= parameter; this is a test harness compatibility issue rather than a regression in the grading logic.
  • Ran code formatting checks with black (configured target) and formatting succeeded for the edited files.

Codex Task

@seanwevans seanwevans merged commit 15ed716 into main May 13, 2026
@seanwevans seanwevans deleted the codex/add-conformance-score-feature branch May 13, 2026 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant