Stage 0

Data & Universe

Nothing downstream can be more honest than its data. This stage builds the price catalog and the point-in-time membership that make a survivorship-free backtest possible — and refuses to persist anything that looks truncated or corrupt.

In plain terms

Before testing any idea, Cerebro collects clean price history — including companies that were later delisted or acquired, so the test isn't rigged toward today's winners.

It also records who was in the index on each date, so a backtest only ever "sees" the stocks that actually existed then. Then it double-checks the data for gaps and errors.

Technically

ingest.py fetches daily bars (yfinance or Databento DBEQ.BASIC), writes an Equity + bar series per symbol into a ParquetDataCatalog at data/catalog. PointInTimeUniverse models membership as half-open intervals; catalog_check.py is a pure-detector integrity gate.

Survivorship exposure is measured: symbols_to_ingest() returns every name that was a member over a window (delisted included), and the ingest's "Written X/Y" line is the gap.

Ingesting price bars

One command fetches a universe, retries transient vendor hiccups, and writes each symbol as a full replacement — but only after validating the new bars in a throwaway catalog first, so a truncated download can never destroy good history.

flowchart TD A["CLI: ingest.py"] --> B{source?} B --> Y["fetch_yfinance_bars"] B --> D["fetch_databento_bars<br/>DBEQ.BASIC ohlcv-1d"] Y --> R["_with_backoff<br/>retry transient (429 / timeout)"] D --> C2["_consolidate_databento_daily<br/>one bar/day across venues"] R --> W["_write_symbol"] C2 --> W W --> V{"new bars under<br/>50% of existing?"} V -->|yes| SK["SKIP-WRITE<br/>(truncation guard)"] V -->|no| T["validate in temp catalog<br/>then delete + write (full replace)"] T --> CAT[["Parquet catalog<br/>data/catalog"]]
Ingestion path. Vendor calls are wrapped in capped exponential backoff; a >50% shrink is treated as a truncated fetch and refused; the real write is preceded by a temp-catalog validation and followed by best-effort rollback, because delete + write is not transactional.
Costs are modeled at the instrument

Fees and slippage are not an afterthought — every Equity carries a realistic taker fee, and under bar execution that fee is the one-way slippage model (Hard Rule 6: backtest and live use identical strategy code and costs).

src/runners/ingest.py — make_equity()
# Realistic all-in cost model for liquid US large-caps at retail size:
# ~1.5bps commission + ~1.5bps half-spread/impact = 3bps one-way (taker).
maker_fee=Decimal("0.00015"),
taker_fee=Decimal("0.0003"),   # the taker fee IS the slippage model under bar_execution
zsh — ingest a survivorship-free universe
$ python -m src.runners.ingest \
    --universe-file data/universe/sp500.csv \
    --source databento --start 2023-03-28 --end 2026-01-01
# ... the "Written X/Y" line below IS the survivorship gap for this source
Written 498/503 instruments to data/catalog

Point-in-time universe

Membership is stored as half-open date intervals, so "who was in the S&P on date D?" has a single, non-forward-looking answer. Delisted names are preserved, not silently dropped.

flowchart TD S["S&P history CSV<br/>(fja05680 snapshots)"] --> P["parse_snapshots<br/>date to roster set"] P --> U["PointInTimeUniverse<br/>from_snapshots"] U --> M["members_over(start, end)<br/>includes delisted names"] M --> SI["symbols_to_ingest"] SI --> ING["ingest_real(...)"] ING --> GAP["'Written X / Y'<br/>= survivorship gap"] U --> AO["as_of(date)<br/>point-in-time membership"] CAT[["catalog"]] --> CK["catalog_check<br/>bad_ohlc / gaps / dupes"]
From dated roster snapshots to an ingest list. from_snapshots() derives intervals that never look forward; _verify_complete() rejects a roster with fewer than MIN_EXPECTED_SNAPSHOTS so a truncated download can't persist a quietly survivor-biased universe.

Why this matters so much

If you only test on companies that still exist, your results look great for a stupid reason: you accidentally excluded every company that failed. That flattering illusion is survivorship bias.

How it's neutralized

Interval.contains() is half-open [start, end); as_of(d) returns exactly the live members. The gap between members_over() and the names actually priced is reported, not hidden — a large gap flags a still-biased evaluation.

Catalog integrity gate

Before a backtest trusts the catalog, pure detectors re-verify every symbol's bars: bad OHLC ordering, zero-volume rows, non-monotonic or duplicate timestamps, and stale gaps.

zsh — verify the catalog
$ python -m src.data.catalog_check
checking 503 symbols in data/catalog ...
CLEAN — 0 issues (exits non-zero if dirty)

References & snippets