Gate 2

Walk-forward validation

A strategy that only shines in-sample has not passed. Gate 2 attacks overfitting from three angles: out-of-sample consistency across rolling windows, resistance to parameter perturbation, and a Sharpe deflated by how many configurations were tried.

In plain terms

It's easy to find a strategy that fit the past perfectly — and is worthless. So Cerebro repeatedly tunes on an early slice and tests on the next, unseen slice, walking forward through history.

It also nudges the settings ±20% (a real edge shouldn't collapse) and penalizes results for every variation that was tried, because trying enough things guarantees a lucky winner.

Technically

walk_forward.py generates rolling train/test windows (504/126, step 126) and runs each via a subprocess. param_sensitivity.py perturbs key params ±20% and writes a trial sidecar; dsr.py computes the Bailey & López de Prado deflated Sharpe from N trials.

Rolling walk-forward

Each window trains on 504 bars and tests on the next 126, then the whole frame steps forward 126 bars. Out-of-sample performance is judged on the test segments only.

gantt dateFormat X axisFormat %s title Rolling walk-forward — train 504, test 126, step 126 (bars) section Window 1 train :0, 504 test :504, 126 section Window 2 train :126, 504 test :630, 126 section Window 3 train :252, 504 test :756, 126
Overlapping train/test frames stepping through history. OOS metrics are the median across countable windows; a window that correctly sat in cash (0 trades, risk-off) is excluded, not penalized.
Gate 2 criterionThreshold
OOS Sharpe vs in-sample (median)within 30%
Catastrophic drawdown in any OOS window≤ 30%
Parameter perturbation ±20%no >50% Sharpe drop
Window stability (coefficient of variation)≤ 1.5
Countable windows (≥ 8 trades/yr, scaled)≥ 5
Deflated Sharpe ratio≥ 0.95

Robustness & deflation

The evaluation hub, promote_check.py, runs in-sample and out-of-sample backtests in a subprocess (Nautilus initializes its logger once per process), then folds together the gate checks, auto-rejections, the deflated Sharpe, and regime-coverage warnings.

flowchart TD IN["backtest IS + OOS<br/>(run in subprocess)"] --> G1["check_gate1<br/>Sharpe / Sortino / DD / trades / valid"] IN --> AR["check_auto_rejections<br/>IS-only? Sharpe collapse? too few trades?"] TRIALS["param_trials sidecar<br/>(n_trials, sharpes)"] --> DSR["check_dsr<br/>deflated Sharpe at least 0.95"] REG["regime coverage<br/>GFC 2008 / COVID 2020 / 2022"] --> WARN["warnings"] G1 --> REP["PromotionReport<br/>overall_passed"] AR --> REP DSR --> REP WARN --> REP
Evaluation flow. The param-sensitivity run writes the trial count that the deflated-Sharpe check reads back — so "we tried 40 variants" is priced into the significance of the winner.

What "deflated Sharpe" means

If you try 100 random strategies, the best one looks great by luck alone. The deflated Sharpe asks: "given how many we tried, is this one really better than noise?" — and demands 95% confidence.

How it's computed

dsr.deflated_sharpe_ratio(sharpe, t_obs, n_trials, trial_sharpes) estimates the expected maximum Sharpe of n_trials noise strategies and returns the probability the observed Sharpe exceeds it. Threshold 0.95, capped at MAX_EFFECTIVE_TRIALS = 100.

zsh — run Gate 2
$ python -m src.runners.walk_forward      --strategy cross_sectional_momentum_etf
$ python -m src.runners.param_sensitivity --strategy cross_sectional_momentum_etf
# param_sensitivity writes data/sessions/param_trials_<strategy>.json for the DSR check
Automatic rejection

In-sample-only results are rejected outright. So is a Sharpe that collapses >50% from in-sample to out-of-sample, or a strategy with too few countable windows. Out-of-sample validation is mandatory — an in-sample number has not passed.

References