Skip to content

๐Ÿ“ Numbers worth memorising

When: you want to be able to produce a number cold, without a laptop.

Not trivia. Each of these settles an argument, and each one is either an arithmetic you can do in your head or a measured result you can cite with its condition attached.

Arithmetic you should be able to do out loud

Quantity The calculation Worked
Multi-hop recall floor r^hops under independence 0.86ยณ โ‰ˆ 0.64 โ€” and it is a lower bound, because hops correlate
Eval-set size for a delta n ร— (half_width / point_estimate)ยฒ, then double it 207 ร— (0.046/0.034)ยฒ โ‰ˆ 383 โ†’ budget ~750
Chance agreement pโ‚pโ‚‚ + (1โˆ’pโ‚)(1โˆ’pโ‚‚) 90/10 split: pโ‚‘ = 0.82, so 94% raw agreement is ฮบ โ‰ˆ 0.67
JL bound d โ‰ˆ 8 ln n / ฮตยฒ n=10โถ, ฮต=0.1 โ†’ โ‰ˆ11,000 dimensions, which is the surprise
Binary metric variance p(1โˆ’p), maximal at p=0.5 0.25 โ€” worst exactly where interesting systems live
Noise band, rough โ‰ˆ 2 / โˆšn for a binary metric n=200 โ†’ about ยฑ0.14 at worst; ยฑ0.06 typical

Measured here โ€” cite with the condition

Number What it is The condition that makes it true
31% Share of a monthly bill that is generation tokens Mid-size deployment, self-hosted encoder
0.469 โ†’ 0.531 Full-chain recall from a per-entity packing constraint k=8, +1.2% tokens
0.469 โ†’ 0.548 Same metric from raising k to 16 +93% tokens, context_precision 0.52 โ†’ 0.31
84 vs 27 Found-then-dropped vs never-retrieved The bottleneck was 3:1 on the packing stage
0.849 โ†’ 0.752 Evidence recall after adding a lexical-only reranker A reranker over the same signals is worse than none
0.38 Best abstention F1 by retrieval-score threshold Full eval set at its real 15% null base rate
+0.0338 โ†’ +0.0116 The same delta re-measured at n=812 The winner's curse was most of the effect

Orders of magnitude

Vector storage 4 ร— dim bytes per chunk, float32. 1M chunks ร— 768 dim โ‰ˆ 3 GB
BM25 index Roughly the size of the text, often less
Rerank cost Linear in candidates, nothing precomputable โ€” this is why depth matters
Cross-encoder latency ~10โ€“50 ms per pair. At depth 50 that is the latency budget
Embedding throughput Plan in GPU-hours per million chunks, not seconds per chunk

The three constants not to memorise

  • ฮฑ = 0.2 โ€” fitted to this corpus and this encoder. Quoting it elsewhere is the error the README warns about
  • k = 8 โ€” a budget decision, not a fact
  • chunk size = 512 โ€” there is no right answer; it is a bake-off

Knowing which numbers do not transfer is worth more than knowing the ones that do, and saying so unprompted is a strong signal.

The one to lead with

"On a binary metric, per-question variance is p(1โˆ’p) โ€” about 0.25, and it peaks at the 0.5 accuracy where interesting systems sit. So the same true effect will clear the band on a continuous metric and not on a binary one. That is one effect at two levels of statistical power, not two contradictory results."

It is short, it is exactly right, and it demonstrates the thing most candidates cannot: that you know why your own metrics disagree.