boost / docs / evaluation system

Quality · shipped in PR #112 · #142 · #145 · #147

boost grades its own retrieval quality.

A package manager for AI skills is only as good as the skill it hands back for "the best thing for <task>." So boost measures that — with a required, deterministic golden-set gate that fails the build on a retrieval regression, plus opt-in tiers for statistical significance and LLM-judged faithfulness. Everything degrades gracefully; the required gate has zero dependencies.

5Evaluation tiers
0.919BM25 recall@10 · full catalog
43Golden judgments
0Deps in the required gate
The problem

Search quality is a number, not a vibe.

boost has several ways to find a skill — a frontmatter heuristic, a full-content BM25 index, an opt-in dense-vector backend, and an LLM reranker. Which one actually returns the right skill for a real question? The eval system answers that empirically, so a change that quietly degrades retrieval can't ship.

The core idea is the classic information-retrieval one: a golden set of query → the skill(s) that should come back judgments, scored with standard IR metrics (recall@k, hit@1, MRR, nDCG). It's the same shape as a TREC qrels file — hand-labelled ground truth that turns "seems good" into "recall went from 0.92 to 0.71, block the merge."

Architecture

One golden set, four engines, five verdicts

Every tier reuses the same golden judgments and the same retrieval stack. Data flows left to right; the golden set feeds the scoring stage from above.

Golden sets · ground truth golden.jsonl · recommend.jsonl · explain.jsonl 43 queries · 8 stacks · 7 skills — labelled by hand 1 · Corpus pinned taps.txt → BM25 index 2 · Engines catalog · BM25 dense · rerank 3 · Scoring recall@k · hit@1 MRR · nDCG · grounding 4 · Verdict floor gate (required) + 4 opt-in verdicts labels grade the ranked results Verdicts → Tier 1 recall floor 0.85 (required in CI) Tier 1b ranx significance · Tier 2a rerank lift Tier 2b recommend grounding · Tier 2c explain faithfulness

The required gate (Tier 1) is pure-stdlib and offline. Everything green-lit beyond it is opt-in.

The contract

Five tiers, one rule: never block on a maybe

Only Tier 1 gates the build. It is deterministic and dependency-free precisely so it can be required without ever flaking. Everything that needs a network, a key, or a heavy library is opt-in and skips cleanly when its prerequisites are absent.

TierWhat it gradesMetric / gateNeeds
1 · retrieval Does the right skill come back for a real query? recall@k ≥ 0.85 floor vs the pinned corpus nothing — pure stdlib, offline required
1b · significance Is engine A really better than B, or just lucky? ranx paired Student's t-test (p<0.05) [eval] extra (ranx) opt-in
2a · rerank How much does the LLM reranker improve ordering? hit@1 / MRR / nDCG lift over raw BM25 claude CLI or ANTHROPIC_API_KEY opt-in
2b · recommend Are boost recommend's AI picks relevant and grounded? precision@k + hard grounding gate (0 hallucinated picks) claude CLI or ANTHROPIC_API_KEY opt-in
2c · explain Does boost explain stay true to the SKILL.md? ragas faithfulness (claims verified vs source) [eval] extra + judge key opt-in
Sequence · the required gate

How Tier 1 runs on every PR

The golden set grades items by name, so the repos that hold them must be present. CI taps a pinned corpus first, builds the BM25 index, scores the golden set, and floors recall — all offline after the tap, all deterministic.

CI · lint job ensure_corpus + boost tap rag · BM25 golden set + floor gate run gate → tap pinned repos (network, once) catalog.all_entries() → build index (~9.5k chunks) retrieve(query) for each of 43 golden cases ranked names → compare to labelled relevant set recall@10 = 1.000 ≥ 0.85 → PASS ✓ fail-under 0.85 regression relaxed (drift-safe)

On the pinned corpus BM25 recall is 1.000, so the 0.85 floor leaves wide margin — a drift-resistant gate that still catches a broken index or a scoring regression.

Explained twice

Simple on the left, technical on the right

Same concept, two depths. Open the left card for the plain-English version, the right card for what actually happens in the code.

◍ SIMPLE
◍ TECHNICAL
# The golden set
What is a "golden set"?

A hand-written answer key. Each row is a question a user might type and the skill that should come back. To grade any search engine, you run every question through it and check whether the right answer showed up near the top.

qrels + IR metrics

tests/eval/golden.jsonl holds {query, relevant[], kind} rows — a TREC-style qrels file. The harness runs each ranker, then computes recall@k, hit@1, MRR and nDCG@k with binary relevance, sliced per item kind (skill / rule / workflow).

# Why BM25 beats the heuristic
Why not just match keywords?

The simple approach only reads a skill's short description. BM25 reads the whole skill file, so it finds matches the description never mentions. The eval proves it: BM25 finds the right skill far more often.

0.756 → 0.919 recall@10

The frontmatter heuristic (catalog.search) scores recall@10 = 0.756; full-content BM25 (rag.retrieve) scores 0.919 on the same 43 queries. That measured +0.163 is the evidence that justifies the RAG stack's complexity — not a hunch.

# Is the win real? (Tier 1b)
Could a better score be luck?

On only a few dozen questions, one engine can look better just by chance. A significance test asks: if you re-ran on fresh questions, would the winner keep winning? It reports "real difference" or "too close to call."

ranx paired t-test

--stats feeds the per-query scores to ranx, which runs a paired Student's t-test between engines and reports a p-value plus win/tie/loss. Deterministic, offline. On the small pinned corpus the gap is not yet significant (p≈0.11–0.26) — an honest signal to widen the set.

# Faithful explanations (Tier 2c)
Can explain make things up?

boost explain writes a plain-English summary of a skill. The risk is it invents abilities the skill doesn't have. This tier checks every sentence against the real skill file and scores how much of the summary is actually supported.

ragas faithfulness

scripts/eval_explain.py generates the explanation with boost's own ai.ask, then scores it with ragas faithfulness: the answer is decomposed into claims and each is verified against the SKILL.md context. LLM-judged, so it's key-gated and never part of make check.

# Why a pinned corpus
Where do the skills come from?

To grade "did the right skill come back," the skills have to exist. CI starts empty, so the gate first downloads a fixed, hand-picked list of skill repos — always the same list, so the score means the same thing every run.

taps.txt + ensure_eval_corpus.sh

tests/eval/taps.txt is the minimal repo set covering all 44 golden targets (boost tap --defaults omits every rule/workflow repo). ensure_eval_corpus.sh taps it into $BOOST_HOME — idempotent via a sentinel. Regression-vs-baseline is relaxed (--regression-eps 1) so upstream drift can't flake the required gate.

Run it yourself

One command per tier

make · boost evaluation tiers
# Tier 1 — required, deterministic, part of `make check`
$ make eval
  GATE recall@10 = 1.000  (min 0.850)  -> PASS

# Tier 1b — is the difference statistically significant? (needs [eval])
$ make eval-stats
  best engine: BM25 full-content
  vs catalog.search  ndcg@10  p=0.1127  not significant  W/T/L=7/34/2

# Tier 2a / 2b — LLM rerank lift & recommendation grounding (key-gated)
$ make eval-ai   # rerank: hit@1 +0.163 over raw BM25
$ make eval-rec  # recommend: grounded 1.000, 0 hallucinated picks

# Tier 2c — explain faithfulness via ragas (needs [eval] + a judge key)
$ make eval-explain
  skill                         faithfulness
  pdf                                  0.94
  MEAN                                 0.91   (n=7)

Illustrative output. Opt-in tiers print a one-line hint and exit 0 when a dependency or key is missing — they never block.