Skip to content

Add StrategyValidator: schema checks + look-ahead bias detection (#9) - #26

Open
Dhiyaahaq33 wants to merge 1 commit into
himanshu2406:mainfrom
Dhiyaahaq33:feat/strategy-validator
Open

Add StrategyValidator: schema checks + look-ahead bias detection (#9)#26
Dhiyaahaq33 wants to merge 1 commit into
himanshu2406:mainfrom
Dhiyaahaq33:feat/strategy-validator

Conversation

@Dhiyaahaq33

Copy link
Copy Markdown

Summary

Implements the core, most concrete part of #9 — an abstract-style verification harness that checks a strategy for structural validity and, most importantly, look-ahead bias (data leaking from the future into past signals).

  • strategy/strategy_validator.py — new StrategyValidator class:
    • validate_output_schema(): checks entries/exits/close_data/open_data share the same index/columns, entries/exits are boolean with no NaNs, index is chronologically sorted, and no bar has both an entry and exit signal for the same symbol.
    • detect_lookahead_bias(): runs the strategy once on the full OHLCV history, then again on truncated prefixes (e.g. 50%/75%/90% of the data). A correct strategy's signal at time T can only depend on data up to T, so the truncated run's signals over the overlapping period must exactly match the full run's. Any mismatch proves the strategy read future data.
    • validate(): convenience method running both checks, returning a ValidationReport (passed: bool, plus readable issue descriptions).
  • tests/test_strategy_validator.py (9 tests, all passing) using:
    • a clean EMA-crossover strategy → validator reports PASSED
    • a deliberately broken strategy that peeks at close.shift(-1) (tomorrow's price) → caught by detect_lookahead_bias
    • a strategy with conflicting entry+exit signals on the same bar → caught by validate_output_schema
    • sanity-checked against the repo's real strategy.public.EmaStrat.EMAStrategy — passes cleanly, no false positives
  • Short usage section added to docs/strategy_development.md.

Scope note

#9 mentions three things: (1) no data leaks into the future, (2) trades are structurally valid, (3) an indicator registry similar to strategy_registry.py. This PR covers (1) and (2) — they share the same validator and are directly testable/provable. (3) is a materially different, larger feature (a whole new registry + discovery mechanism for indicators) and felt out of scope for one PR — happy to take that on separately if you'd like it split out as its own issue/PR.

Test plan

  • python -m pytest tests/test_strategy_validator.py -v — 9/9 passing
  • Ran StrategyValidator against the real EMAStrategy in strategy/public/EmaStrat.py with synthetic OHLCV data — reports PASSED, confirming no false positives on production code
  • Confirmed issue Add Strategy Verification Testing #9 is still open with no other PR addressing it

…anshu2406#9)

Adds strategy/strategy_validator.py with StrategyValidator, addressing
the first part of himanshu2406#9 (ensuring no data leaks into the future):

- validate_output_schema(): checks entries/exits/close_data/open_data
  share the same index/columns, entries/exits are boolean with no NaNs,
  and no bar has both an entry and exit signal for the same symbol.
- detect_lookahead_bias(): runs a strategy on the full OHLCV history and
  on truncated prefixes of it, then verifies signals over the
  overlapping period are identical. Any mismatch proves the strategy's
  decision at time T depended on data from after T.
- validate(): convenience method combining both, returning a
  ValidationReport with a readable summary of any issues found.

Includes tests/test_strategy_validator.py (9 tests) using a clean
EMA-crossover strategy (should pass), a deliberately leaky strategy
that peeks at shift(-1) future close prices (should be caught by the
look-ahead check), and a strategy with conflicting entry/exit signals
(should be caught by the schema check). Also sanity-checked against
the repo's real strategy.public.EmaStrat.EMAStrategy, which passes
cleanly with no false positives.

Adds a short usage section to docs/strategy_development.md.

Scope note: this covers the 'no data leaks into the future' and
'trades are structurally valid' parts of himanshu2406#9. The third part (an
indicator registry mirroring strategy_registry.py) is a separate,
larger feature and intentionally left out of this PR -- happy to
tackle it separately if useful.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant