ReproJudge

Evidence-first evaluation infrastructure for autonomous scientific reproducibility agents.

ReproJudge evaluates any external agent command without trusting the agent's own claim that it succeeded. A versioned task contract defines the job, deterministic evaluator checks decide the verdict, and every run leaves an auditable bundle of artifacts, hashes, logs, timing, and provenance.

Status: 0.3 public-beta release candidate. This repository is the canonical standalone source. The core runtime has zero third-party Python dependencies. The first scientific shard contains 15 revision-pinned real-paper repository-discovery tasks. Those tasks are not 15 paper reproductions, and ReproJudge process mode is not a security sandbox.

Why ReproJudge

Need	ReproJudge surface
Evaluate different agent frameworks fairly	any executable after --; no agent SDK required
Keep benchmark truth outside the agent	evaluator checks are omitted from the normal agent-visible request
Make PASS auditable	deterministic checks + artifact SHA-256 + bounded logs + provenance
Compare repeated runs	summarize and agent-identity leaderboard outputs
Keep the evaluator lightweight	Python 3.11+ standard-library core, zero runtime dependencies
Preserve claim boundaries	versioned task/result contracts, explicit failure taxonomy, release evidence

ReproJudge is the evaluation layer, not the scientific agent itself. It is intended for agent benchmarks, reproducibility-agent regression suites, deterministic challenge sets, and evidence-preserving research evaluations.

60-second start

Install the current source and generate a complete starter benchmark:

git clone https://github.com/XiantingWu/ReproJudge.git
cd ReproJudge
python3 -m pip install .

reprojudge init my-benchmark
cd my-benchmark
reprojudge run tasks/hello-reprojudge.json --output runs -- python3 agent.py
reprojudge summarize runs

On Windows, use py -3 -m pip install . and py -3 agent.py in place of the python3 commands above.

reprojudge init creates a minimal valid task, a deterministic agent adapter, and a short local README. It refuses to overwrite a non-empty directory.

The distribution, Python package, and CLI intentionally share the single public namespace reprojudge, so install, import, and command identity stay consistent.

Prefer to inspect the repository demo directly?

reprojudge validate examples/tasks/demo-task.json
reprojudge run \
  examples/tasks/demo-task.json \
  --output .reprojudge/runs \
  -- python3 examples/demo_agent.py
reprojudge leaderboard .reprojudge/runs

A run preserves evidence instead of only printing a verdict:

request.json
result.json
stdout.log
stderr.log
agent-telemetry.json       # only when emitted by the agent
artifacts/
  ...

result.json records evaluator/task and agent-visible request fingerprints, exact argv, timing/platform/version, bounded-log truncation state, deterministic check outcomes, failure taxonomy, artifact sizes/hashes, and optional bounded telemetry.

Mental model

Scientific-agent demos often blur four roles:

task author -> agent under test -> evaluator -> public claim

ReproJudge keeps them explicit:

versioned evaluator task
        |
        +--> agent-visible request (private checks omitted)
        |
        v
 external agent process
        |
        +--> bounded stdout/stderr
        +--> declared artifacts
        +--> optional telemetry
        |
        v
 deterministic scorers
        |
        v
 result.json + hashes + provenance
        |
        +--> summary / leaderboard
        +--> release evidence

A process exit code of zero does not automatically become scientific success. Only declared evaluator checks determine the ReproJudge passed result.

Author your own benchmark

Start from the generated scaffold, then replace the synthetic task with a bounded claim you can justify:

reprojudge init my-benchmark
reprojudge validate my-benchmark/tasks/hello-reprojudge.json
reprojudge fingerprint my-benchmark/tasks/hello-reprojudge.json

The v1 task contract remains compatible with the canonical v1 seed contract published before 0.3:

{
  "schema_version": 1,
  "task_id": "governed-individuation-mechanism-v1",
  "domain": "agent-governance",
  "paper": "2607.04613v1",
  "expected_artifacts": []
}

0.3 optionally adds bounded title, instructions, tags, timeout_seconds, metadata, and deterministic checks.

Supported scorers:

• artifact_exists
• json_equals
• json_numeric with absolute/relative tolerances
• text_contains
• text_regex
• file_sha256

Evaluator checks are not copied into the generated agent-visible request.json. This prevents the normal request contract from handing expected numeric values, JSON values, or hashes directly to the agent. The full evaluator task is still fingerprinted in result.json.

See the benchmark authoring guide, task specification, and machine-readable contracts in schemas/.

Bring your own agent

ReproJudge integrates at the process boundary, so an agent can be local, closed-source, a research prototype, or a thin adapter around a remote service.

reprojudge run task.json -- python3 my_agent_adapter.py

The adapter reads the generated request from REPROJUDGE_TASK_MANIFEST, writes declared outputs under REPROJUDGE_OUTPUT_DIR, and may write bounded telemetry to REPROJUDGE_TELEMETRY_PATH.

By default ReproJudge passes a minimal environment rather than inheriting the caller's full environment. --inherit-env is an explicit authority increase for trusted integrations.

See Integrating agents for the complete adapter contract and acceptance checklist.

Real scientific shard

benchmarks/scientific-seed/ contains 15 revision-pinned papers across vision, NLP, generative modeling, scientific ML, dynamical systems, differentiable physics, and quantum computing.

The 0.3 shard measures one bounded stage only:

paper -> canonical public repository discovery

Each task asks an agent to emit:

{"repository_url":"https://github.com/owner/repository"}

Validate the complete shard:

python3 scripts/validate_scientific_seed.py

The validator requires exactly 15 unique task/paper/repository identities, explicit paper revisions, revision-matched evidence URLs, exact manifest/filesystem equality, deterministic task hashes, and a deterministic aggregate shard hash. Canonical GitHub repository spellings are curated release inputs; if an upstream repository is renamed or its canonical owner/name casing changes, the affected gold must be refreshed and the release evidence remeasured rather than silently accepting a stale oracle. A scheduled canonical-drift workflow watches those upstream identities.

This is not a 15-paper reproduction claim. The shard does not test environment reconstruction, experiment execution, numerical agreement, or scientific correctness. Those are separate benchmark stages.

Reference and compatibility evidence

The four-case deterministic reference suite tests evaluator mechanics including JSON equality/numeric tolerance, text/regex checks, exact SHA-256, and the zero-artifact compatibility contract:

python3 scripts/run_reference_suite.py --output .reprojudge/reference

External cross-repository compatibility attestation (the Repo1/VeriRepro 0.8 compatibility layer of earlier releases) is deferred to a future release: it is not re-issued under the current repository identity without a fresh, formally re-established measurement. No stale provenance is relabeled.

See Compatibility and the evidence model.

Safety model

ReproJudge is not a security sandbox.

The evaluator uses argv execution rather than a shell, refuses escaping/symlinked evaluator paths, bounds manifests/results/telemetry/logs/scorer/hash work, evaluates regex checks in a time-bounded child interpreter, cleans ordinary same-process-group descendants on POSIX, keeps private checks out of the normal agent request, and avoids inheriting caller credentials by default.

An evaluated agent is still arbitrary code with the authority of its OS process. A hostile process may inspect other readable host files or deliberately escape application-level process-group assumptions. Hidden-gold or hostile-agent evaluation belongs in an appropriate VM/container/worker boundary.

See SECURITY.md and the trust model.

CLI

reprojudge init [DIRECTORY]
reprojudge validate TASK.json
reprojudge registry TASK_DIRECTORY
reprojudge run TASK.json [--output DIR] [--cwd DIR] [--inherit-env] -- AGENT ...
reprojudge suite TASK_DIRECTORY [--output DIR] [--cwd DIR] [--inherit-env] -- AGENT ...
reprojudge summarize RUN_DIRECTORY [--format markdown|json]
reprojudge leaderboard RUN_DIRECTORY [--format markdown|json|csv]
reprojudge fingerprint TASK.json
reprojudge fingerprint PROJECT_ROOT --source-tree
reprojudge doctor [--strict] [--require-docker]

doctor --strict verifies the core process boundary and writable temporary storage. doctor --strict --require-docker additionally requires a usable Docker daemon for trusted container-backed integration work. Ordinary ReproJudge process-mode evaluation does not require Docker.

Development and CI

python3 -m pip install -e '.[dev]'
pytest -q
python3 scripts/check_public_identity_hygiene.py
python3 scripts/launch_surface_check.py
python3 scripts/validate_scientific_seed.py
python3 scripts/release_check.py
python3 scripts/release_source_check.py

Repository CI runs exclusively on disposable GitHub-hosted runners. SHA-pinned actions/setup-python selects exact CPython patch versions for the full 3.11 / 3.12 / 3.13 / 3.14 matrix on Linux, macOS (ARM64), and Windows lanes. Read-only repository permissions, non-persistent checkout credentials, and per-job isolated environments apply throughout. Every job runs the full test suite, public identity hygiene, launch-surface, scientific-seed, release, reference-suite, standalone-export, sdist-validation, and wheel/sdist clean-install gates. The Python 3.11 lane adds coverage (>= 90% statements, >= 85% branches), ruff, and mypy gates.

The 0.3 release-authority measurement is performed on the GitHub-hosted macOS ARM64 lane with pinned CPython 3.11.9; the other operating-system lanes independently exercise the same suite but only the authority lane produces formal release evidence. Docker capability (doctor --strict --require-docker) is verified on a separate GitHub-hosted Linux lane.

All third-party Actions are pinned to full commit SHAs. Fork and Bot-authored pull requests run on disposable GitHub-hosted runners with a read-only GITHUB_TOKEN; no persistent or personal runner infrastructure is used anywhere in the public repository.

Release evidence

0.3 release evidence keeps three statements separate:

1. 15 real-paper discovery tasks define the first scientific benchmark shard;
2. 4 deterministic reference cases test evaluator mechanics on the exact candidate source;
3. the trusted GitHub-hosted release audit proves the exact current standalone source, package, and evidence binding.

Formal ReproJudge source measurement uses:

repository: XiantingWu/ReproJudge
workflow: ReproJudge trusted release audit
runner_environment: github-hosted
platform: macOS ARM64 (GitHub-hosted)
python: 3.11.9

Candidate evidence is generated on the exact source head, verified by digest, promoted byte-exact as benchmarks/release-evidence-0.3.0.json, then revalidated on the exact committed-evidence head. Only that strictly versioned promotion file is excluded from the release-source fingerprint to avoid self-reference; tests, schemas, workflows, scripts, examples, benchmark inputs, and public documentation remain fingerprinted.

The PyPI release workflow is stricter than ancestry membership: the release tag must resolve to the exact current main commit in both the verification and publishing jobs. If main advances after a candidate was validated, publication fails closed and the release must be revalidated at the new authority head.

See the evidence model, release process, and public release checklist.

Deliberate non-claims

A ReproJudge 0.3 PASS does not mean:

• the 15 discovery papers were fully reproduced;
• an arbitrary scientific paper is reproducible;
• scientific correctness has been established;
• four reference tasks provide broad scientific benchmark coverage;
• external cross-repository compatibility was freshly re-established in this release;
• process success is scientific equivalence;
• regex/JSON/hash checks replace domain-expert metrics;
• process mode safely sandboxes hostile code;
• hosted-runner evidence is independent third-party certification.

Design principles

1. Agent-neutral — the runtime imports no sibling project.
2. Evidence-first — verdicts point to concrete artifacts, checks, and hashes.
3. Fail closed — malformed input, unsafe paths, invalid telemetry, missing artifacts, launch failures, non-zero exits, and timeouts remain explicit.
4. Bounded host work — untrusted manifests, telemetry, results, logs, scorer inputs, and artifact hashing have explicit limits.
5. Versioned contracts — historical task/result meaning does not silently change.
6. Honest claims — evidence says exactly what was measured and what was not.
7. Portable core — the evaluator runtime uses the Python standard library only.

Governance and support

• Governance — maintainer model, release authority, and decision process.
• Support — how to route bugs, features, benchmark proposals, usage, security, and conduct questions.
• Benchmark corpus policy — corpus acceptance, provenance, pinning, drift, and claim boundaries.

Documentation

Start here

• Getting started
• Authoring benchmarks
• Integrating agents
• FAQ

Contracts and evidence

• Task specification
• Result specification
• Evidence model
• Telemetry
• Leaderboard semantics
• Compatibility

Architecture and operations

• Architecture
• Trust model
• 0.3 launch readiness
• Release process
• Public release checklist
• Benchmark corpus policy
• Roadmap
• Contributing
• Security
• Code of Conduct
• Citation
• Changelog

License

MIT. See LICENSE. For academic use, see CITATION.cff.