Add StrategyValidator: schema checks + look-ahead bias detection (#9) - #26
Open
Dhiyaahaq33 wants to merge 1 commit into
Open
Add StrategyValidator: schema checks + look-ahead bias detection (#9)#26Dhiyaahaq33 wants to merge 1 commit into
Dhiyaahaq33 wants to merge 1 commit into
Conversation
…anshu2406#9) Adds strategy/strategy_validator.py with StrategyValidator, addressing the first part of himanshu2406#9 (ensuring no data leaks into the future): - validate_output_schema(): checks entries/exits/close_data/open_data share the same index/columns, entries/exits are boolean with no NaNs, and no bar has both an entry and exit signal for the same symbol. - detect_lookahead_bias(): runs a strategy on the full OHLCV history and on truncated prefixes of it, then verifies signals over the overlapping period are identical. Any mismatch proves the strategy's decision at time T depended on data from after T. - validate(): convenience method combining both, returning a ValidationReport with a readable summary of any issues found. Includes tests/test_strategy_validator.py (9 tests) using a clean EMA-crossover strategy (should pass), a deliberately leaky strategy that peeks at shift(-1) future close prices (should be caught by the look-ahead check), and a strategy with conflicting entry/exit signals (should be caught by the schema check). Also sanity-checked against the repo's real strategy.public.EmaStrat.EMAStrategy, which passes cleanly with no false positives. Adds a short usage section to docs/strategy_development.md. Scope note: this covers the 'no data leaks into the future' and 'trades are structurally valid' parts of himanshu2406#9. The third part (an indicator registry mirroring strategy_registry.py) is a separate, larger feature and intentionally left out of this PR -- happy to tackle it separately if useful.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the core, most concrete part of #9 — an abstract-style verification harness that checks a strategy for structural validity and, most importantly, look-ahead bias (data leaking from the future into past signals).
strategy/strategy_validator.py— newStrategyValidatorclass:validate_output_schema(): checksentries/exits/close_data/open_datashare the same index/columns,entries/exitsare boolean with no NaNs, index is chronologically sorted, and no bar has both an entry and exit signal for the same symbol.detect_lookahead_bias(): runs the strategy once on the full OHLCV history, then again on truncated prefixes (e.g. 50%/75%/90% of the data). A correct strategy's signal at time T can only depend on data up to T, so the truncated run's signals over the overlapping period must exactly match the full run's. Any mismatch proves the strategy read future data.validate(): convenience method running both checks, returning aValidationReport(passed: bool, plus readable issue descriptions).tests/test_strategy_validator.py(9 tests, all passing) using:PASSEDclose.shift(-1)(tomorrow's price) → caught bydetect_lookahead_biasvalidate_output_schemastrategy.public.EmaStrat.EMAStrategy— passes cleanly, no false positivesdocs/strategy_development.md.Scope note
#9 mentions three things: (1) no data leaks into the future, (2) trades are structurally valid, (3) an indicator registry similar to
strategy_registry.py. This PR covers (1) and (2) — they share the same validator and are directly testable/provable. (3) is a materially different, larger feature (a whole new registry + discovery mechanism for indicators) and felt out of scope for one PR — happy to take that on separately if you'd like it split out as its own issue/PR.Test plan
python -m pytest tests/test_strategy_validator.py -v— 9/9 passingStrategyValidatoragainst the realEMAStrategyinstrategy/public/EmaStrat.pywith synthetic OHLCV data — reportsPASSED, confirming no false positives on production code