Verify disk-cached values with non-local keys in per-kind batches

[xmakro]

Selects disk-cached values for incremental verification so that the selection is bit-identical between two different compiler builds: values keyed by a local DefPathHash keep per-key fingerprint sampling, and all other values rotate in per-kind batches slotted by a hash of the kind's label. This document is written as a complete handoff; it records the investigation that led here, every design iteration with measurements, and the methodology to reproduce them.

Origin and problem statement

Context: rustc-perf PR 2510 stops passing -Zincremental-verify-ich, relying on rust#160130 which changed the 1-in-32 verification sampling to select by key fingerprint and session count instead of by value fingerprint. Kobzol asked whether key fingerprints can depend on values from the compilation session. Investigating that question produced the following verified facts:

• Selection since rust#160130 is to_smaller_hash(key_fingerprint) % 32 == session_count % 32 in should_verify_loaded_value (rustc_query_impl/src/execution.rs), called from the green-load path and the promote path (plumbing.rs). Verification itself still re-hashes the value and compares against the stored value fingerprint.
• Key fingerprints cannot depend on session-transient state: the dep graph matches previous-session nodes by (kind, key_fingerprint) (SerializedDepGraph::node_to_index_opt), so any instability there would break green marking itself. Confirmed empirically: identical sessions re-run produce byte-identical selection logs.
• Two different compiler builds produce bit-identical key fingerprints for the same source under RUSTC_FORCE_RUSTC_VERSION (rustc-fake sets rustc-perf): in a cross-compiler experiment one compiler's loaded fingerprint set was an exact subset of the other's with zero selection disagreements. Changing the forced version reshuffles 17868 of 19821 load events (all local DefPathHash keys shift with the local StableCrateId).
• The one leak: sysroot crates (core/std) bake their StableCrateId at artifact build time with the artifact's real version string; bootstrap sets RUSTC_FORCE_RUSTC_VERSION only for compiletest, and the benchmark-time pin cannot apply retroactively. Master and try artifacts have different version strings, so any key naming a sysroot def has a different fingerprint on the two sides of a perf comparison. Measured by rebuilding std with a different forced stamp: 409 of 2963 distinct loaded keys shift (19 foreign-def keys across core and std, 390 opaque keys out of 1291), moving the verified subset (overlap 82 of ~93) and verified events 564 to 574.
• No sub-fingerprint escape exists: DefPathHash low halves chain from the crate root whose seed is Hash64::new(stable_crate_id.as_u64()) (rustc_hir/src/definitions.rs, root hash and compute_stable_hash), confirmed empirically (0 of 19 foreign low halves survive a std restamp). So sampling foreign keys by the low half re-introduces the noise. Opaque fingerprints are blended hashes and cannot reveal whether foreign defs contributed; in the toy experiment 390 of 1291 opaque keys were std-dependent, the rest local-only, but the selection function cannot tell them apart without walking the actual key.

Design iterations and measurements

All numbers are instructions:u of an incremental-unchanged rebuild (session 2), measured with an instrumented stage1 that selects the policy via a VERIFY_POLICY env var, so all policies run on one binary. Baseline is the current upstream sampling.

1. Always verify non-local keys (foreign-def + opaque), sample the rest: diesel-2.2.10 +0.07% (1042 of 344376 loads non-local), 13-benchmark sweep worst case deeply-nested-multi +0.59% (45% of loads non-local), bitmaps-3.2.1 +0.14%, all other real crates at or below noise. The non-local class is dominated by items_of_instance/symbol_name/size_estimate (instance-keyed, small values). Const-eval and large-allocation stress (ctfe-stress-5, include-blob, tuple-stress) measure ~0: large values do not flow through this load path.
2. Whole-class lump at session_count % 32 == 0: steady state ~0, but a fixed slot is a lose-lose: an early slot makes every fresh cache lineage pay at the start, a late slot is never reached by short-lived caches so the class is never covered.
3. Per-kind slots with naive classification: locality via the stable_crate_id query per load costs +0.12..0.14% on bitmaps (61240 loads, the most load-dense benchmark found); via key_fingerprint_style vtable read about the same; computing the kind-label hash for every load is worst, +0.35%.
4. Final: classify by comparing the fingerprint's high half against the local StableCrateId, cached in a new DepGraphData::local_stable_crate_id: OnceLock<u64>. Local keys sample per-key as today; everything else batches per kind at FxHash(dep_kind_label) % 32 == session_count % 32. Cost: bitmaps +0.060%, deeply-nested-multi +0.017%, everything else noise.

Per-kind slots resolve the fixed-slot dilemma because they reproduce the coverage profile of today's per-key rotation exactly: slots spread over all 32 sessions, so a lineage of length L covers about L/32 of the class, same as today, with the rotation input being a build-independent string instead of an unpinned hash. Unit-keyed kinds hold a single value, so per-kind equals per-key for them. HirId-keyed values take the batch path too (their high half is the owner's local hash, which does not match the local crate id); no HirId-keyed disk-cached query was observed in any experiment. Failure reproduction on retry is preserved: a failed session does not commit a graph, so session_count is unchanged on retry.

What the change touches

• rustc_middle/src/dep_graph/dep_node.rs: adds dep_kind_label to define_dep_nodes! (kind label as &'static str; unlike the discriminant, stable across builds).
• rustc_middle/src/dep_graph/mod.rs: re-exports it.
• rustc_middle/src/dep_graph/graph.rs: DepGraphData gains the lazily cached local StableCrateId and an accessor.
• rustc_query_impl/src/execution.rs: should_verify_loaded_value takes &DepNode, early-returns on -Zincremental-verify-ich (unchanged semantics; the flag still forces verification of every loaded value, verified at 445.7M vs 396.4M on the toy crate), then branches on the high-half comparison.
• rustc_query_impl/src/plumbing.rs: call-site adjustment.

Reproduction methodology

• Toy crate: 1200-line generated lib (150 structs/impls/generic fns), compiled directly with rustc -Cincremental under RUSTC_FORCE_RUSTC_VERSION=rustc-perf and RUSTC_FORCE_INCR_COMP_ARTIFACT_HEADER=rustc-perf; snapshot the incr dir after session 1 and restore it before each measured session-2 run so every measurement is the same session. Selection was logged with an env-gated eprintln of (fingerprint, kind, session, decision); sets compared across runs/compilers/std-stamps.
• Cargo benchmarks: copies of rustc-perf collector/compile-benchmarks/*; full build to populate, snapshot target/, restore + touch src/*.rs + perf stat -e instructions:u cargo build per policy, so every measured run is session 2 at the same slot. Caveats: absolute counts are only comparable within one compiler binary (rebuilding rustc shifts baselines); the snapshot/restore protocol inflates some baselines with policy-independent codegen work (diesel), which cancels in policy deltas; bitmaps is the extreme for classification overhead because of its load density.
• Simulating the perf-CI artifact split locally: rebuild std with RUSTC_FORCE_RUSTC_VERSION=<other> (touch library/std,core lib.rs to force), which restamps sysroot StableCrateIds the way two CI artifacts differ.

Alternatives considered and rejected

• Splitting DefPathHash seeding from the version-carrying StableCrateId (version-independent hashing id, version kept only for v0 symbol mangling): the principled complete fix, but a cross-cutting rustc change (second crate id, def_path_hash_to_def_id mapping, TypeId/debuginfo equality across toolchains needs review).
• Pinning the try-build sysroot version to the parent master's version string in CI: small, but only fixes try-vs-parent, and master history comparisons stay noisy.
• -Zincremental-verify-ich=never for rustc-perf: trivially identical but stops measuring verification work users pay, contradicting the measure-what-users-run goal of rustc-perf PR 2510.

State and next steps

• Branch perf/verify-ich-local-sampling (this PR), base branch perf/verify-sampling-base at 84b36a7 (2026-08-06 rollup, contains rust#160130). Single title-only commit.
• Related: on top of the mono partition replay branch (Cache and replay the mono item partitioning on green incremental rebuilds #109) the non-local class mostly disappears from the load path (the instance-keyed trio is exactly what partition replay stops loading on green rebuilds), shrinking even the batch cost.
• Open item: reply to Kobzol on rustc-perf PR 2510 with the investigation summary; this PR is the compiler-side fix that makes the selection bit-identical between compared artifacts.