Benchmarks and Receipts

• This page owns the numbers. Every other wiki page that quotes a figure links back here for the measurement it came from, the arm it was measured against, and the caveat it carries.
• A default only changes behind a measurement. Each of the encoder default-off flips in 0.9.0, the RRF fusion default of 2026-07-06, and the wave-1 ranking flip in 0.9.1 is gated on a receipt with a named arm, a pinned bed, and order-reversed replication.
• The disclosures that cost something get published too. Turning dense off bought recall and lost p50 latency, and that regression is in the CHANGELOG, the README, and this page — not in a footnote nobody reads.
• A receipt without a ledger row is not comparable to anything. The rules are in Comparability rules below and enforced in docs/benchmarks/BASELINES.md.
• Some things are measured, some are estimated, and the two are labelled differently. Where a figure is an unverified design estimate, this page says so on the same line as the figure.

Token economics

Measured with the compressor (legacy: ribosome) disabled — that is, the default LLM-free configuration. N=15 query shapes, May 2026.

• The denominator is a model, not a competitor run. "Standard RAG" here is top-5 × 1,500 + 500 overhead = 8,000 tokens. It is a configurable modeled baseline — RAG_TOP_K, RAG_CHUNK_TOKENS and RAG_OVERHEAD are env-tunable in the reproducer — not a measured run of another system.
• Reproducer: benchmarks/bench_rag_vs_sike_tokens.py, run against your own knowledge store.
• The multi-turn figures are unverified design estimates. The session delivery register elides already-delivered documents; ad-hoc session traces showed a 37× reduction on repeated retrievals within a conversation and ~40% token savings on typical multi-turn work. Neither is a receipted benchmark. The counter that would measure them — cymatix_session_tokens_saved_total — is instrumented but has no published run; see Observability. Treat both figures as design estimates until it does.
• The 37× is elision, not compression. It counts bytes the register declined to re-send, not new content squeezed smaller.
• What is not published: a same-harness baseline/ablation frontier — BM25, BGE-M3, BM25+dense RRF, Cymatix full, Cymatix minus-cymatics — paired with correctness. That is the comparison that would settle the decision-useful claim ("equal-or-better task completion at fewer input tokens"), and it is future work.

Shipped-defaults operating point

What an untouched 0.9.0 install does, measured on the 829K-fragment EnterpriseRAG-Bench bed at full needle power. The default retrieval path is fully algorithmic here — dense, SPLADE and PKI all off, the cross-encoder rerank off as it always has been.

• Receipt: benchmarks/dogfood/erb/receipts/sema_readgate_829k_n469.json, ledger row 2026-08-19-sema-readgate-decider.
• This is a retrieval-layer measurement, not an end-to-end grade. It says how often the gold document reached the delivered window. It does not say whether the answer was right — see The ERB pair-quote rule.
• The abstain set on this run was 16 of 469, all score_below_floor, and byte-identical to the baseline's zero-delivery set.

0.9.1 measurements

Four of these are open pull requests at the time of writing; each is scoped with its PR number.

Wave-1 ranking flip (#407)

rrf_k 60 → 20, and rerank_combinator_by_class extended to all five classifier classes mapped to eps_band. Full-470 confirm at 829k scale:

• The post-flip repository cymatix.toml was verified field-identical to the measured arm config through load_config equality, and both confirm receipts carry post-run config_sha256 annotations.
• One-step revert: git revert 7a42576 restores the previous defaults. The flip ships as a single named commit for exactly that reason.
• The cross-shard merge constant deliberately stays at 60 — that surface was never measured, and is annotated as unmeasured in the code rather than flipped on faith.
• Ledger row 2026-08-27-ranking-under-width-wave1 records the bed identity, ingest_c = 6, the cold-bed house rule, and the warm-bed access-rate hazard. That ingest_c differs from the shipped-defaults row above, so the ledger's own rules do not license reading 0.630 as a successor to the 0.565 in the previous section — they are different rows.

Delivered-seat floor (#409) — knob shipped, flip not taken

[budget] min_delivered_docs, default 0 = byte-identical legacy behavior. At floor 12, on the same 470-needle sequence as the wave-1 confirm:

• Zero losses anywhere, and the map and final-rank bases are byte-identical per-needle — this is a pure delivery change, not a ranking change.
• The default flip was not taken, deliberately. The receipts justify the knob, but the benchmark measures delivery, not answer quality under a widened seat count: twelve full-length documents per window costs real tokens, and ERB does not grade that trade. The answer-quality lane has to run before the default moves.
• The plan doc for this arm also records a mid-flight diagnosis correction: the original token-budget attribution was refuted by an effective-lever probe (budget tier broad, hard floor uncrossed, zero evictions), and the classifier's per-rule assembly cap was pinned as the real lever before any bench compute was spent on the wrong fix.

Entity auto-link hub cutoff (#412) — knob shipped, default 0

[ingestion] entity_autolink_hub_cutoff, default 0 = off = legacy behavior, pinned byte-identical by test. Read-only replay on the 289k-document enronqa_padded bed, seed 53416, 500 linkable documents:

• Why the default stays 0: the edge delta is substantial. On the sample, the uncut arm formed 4,842 relation=5 COVER edges; at cutoff 200 it forms 2,867 — 3,143 lost, 1,168 gained, 1,699 kept. That is a real change to the graph, so the knob ships opt-in with the receipt attached.
• The mitigating context, not a justification: COVER edges are default-inert at query time — the wave-2 COVER-walk arm was killed by its own receipt — and they are already order-nondeterministic under parallel ingest.
• Flip conditions are specified on #411.

Tagger v2 (#413) — a behavior change with a version bump

The CPU ingest tagger now rejects entities containing newlines or tabs, plus email/MIME header field names, x-* extension headers, and MIME transport artifacts. Fresh receipt on a 2,000-file deterministic EnronQA sample, seed 20260830:

• This is a comparability break by design. Tags are part of the bed-content digest, so TAGGER_VERSION = 2 is declared in the tagger module, recorded in every bed manifest by scripts/build_fixture_matrix.py, and added to the BASELINES rules. Every bed built before 2026-08-30 is tagger_version = 1 and stays internally valid but is not cross-comparable with a v2 bed.
• There is no backfill script on purpose — stripping entities in place would create hybrid v1/v2 beds. v2 beds are fresh builds.

The ERB pair-quote rule

The externally-scored numbers below were measured under the 0.8.x all-encoders-on configuration — additive fusion + dense + SPLADE ON — which no longer ships by default. Every layer remains available opt-in.

Quote the delivery and correctness numbers as a pair. Under that same all-encoders-on config:

• Gold-document delivery was 55% at 829K-fragment scale and 82% at 50K.
• Delivery is not a graded pass. End-to-end correctness is the 41.6% above.
• When the gold document was delivered, the answer was correct 79% of the time — so retrieval breadth at extreme scale, not answer synthesis, is the current ceiling.
• The end-to-end judge protocol has not been re-run on shipped defaults. The measurement of the shipped world is the delivered-basis table above, and it is a retrieval-layer metric. Do not blend the two.

The claim being made is the operating point — the corpus was ingested as 829,131 fragments on a single consumer desktop with zero LLM calls on the retrieval path — not a leaderboard win. Full methodology and reproduction: docs/benchmarks/2026-07-10-erb-blob-829k-reproduction.md.

Why the encoders are off

Four flips took the default retrieval path fully neural-free, each on its own receipt.

• The lexical floor is most of the product. All-off scored 0.767–0.867 final recall@12 across the four scales; no isolated layer beat it on final ordering anywhere except SPLADE's single needle at 250k — and on the delivered basis even that inverts (250k is SPLADE's worst cell, −0.133).
• A basis correction is part of this receipt's history (#375). Earlier revisions labelled fr@12 "final (delivered) recall, the metric that matters." That was wrong: fr@12 is final-order recall; the post-budget-trim delivery slice is delivered_gold_rate. The correction addendum republishes the tables on the delivered basis. Dense's harm and the lexical-floor finding hold; SPLADE's one apparent win inverts.
• Opting back in is one config line per store. Existing beds keep their vectors and expansion tables — unused until you set the flag. If a bed was built while the ingest-side knobs were off, re-enabling retrieval also needs scripts/backfill_bgem3_v2.py (dense) or scripts/backfill_sema.py (SEMA).
• Full receipt: docs/benchmarks/2026-08-14-encoder-isolation-scale-curve.md.

The latency disclosure (#374)

• Turning dense off costs p50 latency. Dense was load-bearing as a latency device: its ANN gate capped the candidate list feeding splice, and dense-off degrades the median query to an uncapped lexical pool.

• The retrieval-layer ledger's own precondition — "cap the non-ANN branch's candidate list before dense-skip is viable" — has not landed. The lex-branch candidate cap is still open, alongside the related #336 splice char-target dynamics.

Other standing receipts and gaps

• Fusion. Reciprocal Rank Fusion has been the default ranker since 2026-07-06 — +12pp gold-document delivery over the legacy additive accumulator on the hardest internal bed (0.74 vs 0.62). Mechanism and the "additive" deprecation status: Retrieval Dimensions.
• Cross-encoder rerank (#341) ships default-off deliberately: it pays only on populations with near-cutoff rerank headroom, and costs ~270–560 ms/query on a GPU daemon at 829k documents (~285–315 ms at 100k).
• Control-tag neutralization was flipped on a null receipt, which is the point: the 141-needle 100k-carve A/B measured 141/141 assembled windows byte-identical across arms, recall and delivered rate unchanged to four decimals. A security default that changes retrieval would not have shipped.
• The know surface is coverage-dead at shipped defaults. 0 of 141 know emissions versus 23 of 141 with the pre-flip encoders on, because lexical_dense_agree is structurally dead on the neural-free path. It fails in the safe direction (0% false-KNOW) and is tracked as #287. Detail: Agent Contract.
• The sharded gap (#275): every headline number on this page runs the unsharded engine. The sharded path currently trails unsharded by ~31pp recall@10 / ~30pp MRR on the xl bed — dense recall and co-activation are not yet at parity across shards. "Local-first at scale" is a demonstrated research operating point on the unsharded engine, not a turnkey substrate for sharded corpora.
• The entity-graph layer ships ON and has never been ablated on the delivered basis — the retrieval-layer ledger classes it REMOVE-CANDIDATE and the arm sits in the #377 0.9.x backlog.

Two reading rules the receipts forced

• Check delivered_count before crediting a delivered-gold gain. A small class of queries discards a rank-1 gold document with an empty delivered window. Any perturbation that flips one back on looks like a retrieval win and is not one.
• Rank-based recall cannot see delivery. Signals that pull documents forward after ranking, and gates that trim the window before assembly, are both invisible to recall@k by construction. That is why the 0.9.x work reports the delivered basis, and why the #375 correction mattered.

Comparability rules

From the 2026-08-09 receipt invalidations onward, the ledger enforces:

• Compare arms only within their own row. Cross-row comparison is not licensed by default, even when the metric names match.
• Measure against a frozen tag, never a moving branch.
• A receipt without a row in the ledger is not comparable to anything.
• ingest_c — ingest concurrency — is part of bed identity. Parallel ingest is not bed-equivalent: pool=4 LLM tagging versus pool=1 on a pinned corpus changed document content on 105 of 150 shared documents and tag multisets on 86 of 150, at temperature 0 (LLM-batching drift; the controls held). Every bed records the concurrency it was built at; unknown is a legal value meaning not-comparable-across-beds. Comparable rebuilds ingest sequentially unless the comparator was itself built parallel.
• tagger_version is part of bed identity too (added by #413). Cross-bed comparisons require matching versions; every bed built before 2026-08-30 is v1.
• Bed-state note (2026-08-20 ops). The two 829k beds were rebuilt after the †-marked ledger rows ran — path_key_index dropped, FTS5 migrated to external-content, vacuum. Retrieval-signal impact is nil by design (a byte-identity receipt covers it, and PKI is read-gated off at shipped defaults), but bed-bytes and storage numbers from †-rows are not comparable to post-ops measurements, and reproducing a PKI-on arm against the post-ops beds requires scripts/backfill_path_key_index.py first.
• Release receipts run shipped defaults by construction — daemon off, workers 2 — because they certify what an untouched install does. The standard bench profile (encoder daemon on, per-box worker counts) is for comparative work and is a documented exception, not the release path.

Why the benchmarks look like this

Standard needle-in-a-haystack was built for a different class of system, and running it against a multi-axis index measures the wrong thing:

• NIAH assumes exactly one correct answer. On a corpus where thousands of documents legitimately answer "what is the value of url?", grading a hit only when the retriever returns the one document the bench randomly picked scores a perfect system near zero. Telepathy is not a retrieval property.
• NIAH assumes recall@1 is the right metric. A query that specifies one axis leaves the other signals with nothing to work with; recall@1 grades that as breakage.
• NIAH assumes the query maps to one dense vector. Real queries carry several narrowing signals that compose multiplicatively — project, then component, then target attribute.

The replacements, both maintained:

The multi-valid-gold rule is what makes ERB gradable at all; it is documented at docs/benchmarks/MULTI_VALID_GOLD.md.

Go deeper

• docs/benchmarks/BASELINES.md — the ledger: one row per campaign, the comparability rules, the standard bench profile, and the code pins
• docs/benchmarks/BENCHMARK_RATIONALE.md — the full why-NIAH-doesn't-fit discovery doc and the dimensional-lock design
• docs/benchmarks/2026-08-14-encoder-isolation-scale-curve.md — the four-scale isolation receipts, the latency table, and the #375 basis correction
• docs/benchmarks/2026-08-06-retrieval-layer-ledger.md — the dated cost-versus-impact audit of every retrieval layer, with its scope limits
• docs/benchmarks/2026-07-10-erb-blob-829k-reproduction.md — the 829K-fragment ERB methodology and repro
• docs/benchmarks/2026-07-06-sike-run2-fts-depth-fusion.md — the RRF-versus-additive receipt
• docs/benchmarks/2026-08-07-rerank-wiring-receipts.md — the cross-encoder rerank cost and its population caveat
• docs/benchmarks/BENCHMARKS.md — the practical harness reference
• CHANGELOG.md — every flip entry with its receipt path and its re-opt-in story
• Next: Retrieval Dimensions · Configuration · Roadmap and Releases · Observability