Order the validation page as an evaluation workflow - #1023
Merged
Conversation
The page predated RATE, the DR/plug-in validation losses, the AUUC/Qini bootstrap CIs (#994) and the ground-truth metrics (#1015), so it described a subset of the package's validation tools in no particular order. This reorders it as the sequence an analysis uses -- check overlap, rank on held-out data with AUUC/Qini, test heterogeneity with RATE, select models with validation losses, measure error directly where ground truth exists, then stress unconfoundedness with sensitivity analysis -- and adds short sections for the tools that were missing, each deferring the math to its Methodology section. Two caveats are now stated: ranking metrics belong on held-out data, and comparing subgroup ATEs where the subgroups came from the model's own in-sample scores validates nothing. Existing content (synthetic mechanisms, multiple estimates, sensitivity analysis) is unchanged, only reordered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Validation page predated RATE, the DR/plug-in validation losses, the AUUC/Qini bootstrap CIs (#994) and the ground-truth metrics (#1015). This reorders it as the sequence an analysis uses and adds short sections for the missing tools, each deferring the math to its Methodology section:
return_ci=Truereports bootstrap uncertainty.autocvsqiniweighting.dr_score()/plug_in_t_score()for model selection; lower is better.pehe/ate_error/policy_risk, linked to the datasets page and the leaderboard notebook.6–8. Synthetic mechanisms, multiple estimates, sensitivity analysis — unchanged, only reordered.
Docs-only;
sphinx -Wclean on html, epub and latex.Third of the three P1 items from the docs roadmap.
🤖 Generated with Claude Code