Discussion map¶

Every thread, grouped by purpose. Generated by scripts/discussion_map.py and refreshed weekly — edit the thread, not this file.

88 threads · 142 comments · 20 with a marked answer.

The depth column is comments plus nested replies. A thread with — is open and waiting for someone; that is an invitation, not a defect.

New here? The guide says where things go. If you want to contribute in fifteen minutes, the fastest useful thing is to reproduce a number in the docs and open an [errata] thread if it does not match.

Announcements¶

Thread	Depth
Welcome — start here, and how this place works	● 2
Three measurements in this repo contradict the material. We kept all three.	●● 4
[office hours · 2026-09-08] Eval sets, noise bands, and why your delta is not real	● 2
The L.A.B. simulator is live — 12 auto-graded labs on a prerequisite DAG	— 0
v0.1.0 — the first release whose numbers you can check	— 0

Q&A¶

Exercise clinic¶

One long-running thread per exercise. Ask here rather than opening a duplicate.

Thread	Depth
[clinic · EX-01] Attribute ten failures	— 0
[clinic · EX-02] Find the N where the ceiling stops moving	— 0
[clinic · EX-03] Break the funnel on purpose	— 0
[clinic · EX-04] Manufacture an eval set for a new domain	— 0
[clinic · EX-05] Make starvation reproduce	— 0
[clinic · EX-06] Null questions for a real corpus	— 0
[clinic · EX-07] Chunking bake-off on your own corpus	— 0
[clinic · EX-08] Implement a new chunking strategy	— 0
[clinic · EX-09] Survive an encoder upgrade	— 0
[clinic · EX-10] Prove permission isolation	— 0
[clinic · EX-11] BM25 by hand	— 0
[clinic · EX-12] The analyzer audit	— 0
[clinic · EX-13] Route α by query class	— 0
[clinic · EX-14] Beat the reranker	— 0
[clinic · EX-15] Make the dense leg earn its keep	— 0
[clinic · EX-16] Price your own k	— 0
[clinic · EX-17] Measure position sensitivity for real	— 0
[clinic · EX-18] Solve abstention	— 0
[clinic · EX-19] Calibrate a judge against humans	— 0
[clinic · EX-20] Build a third bias probe	— 0
[clinic · EX-21] Cost model for a real client shape	— 0
[clinic · EX-22] Fix evidence retention	— 0

L.A.B. track threads¶

One per track of the L.A.B. simulator.

Thread	Depth
[lab · T1] Corpus & Chunking — track thread	— 0
[lab · T2] Indexing & Retrieval — track thread	— 0
[lab · T3] Ranking & Packing — track thread	— 0
[lab · T4] Measurement — track thread	— 0
[lab · T5] Judgement — track thread	— 0
[lab · T6] Economics — track thread	— 0
[lab · T7] Agents & Traces — track thread	— 0
[lab · T8] Shipping — track thread	— 0

Derivations¶

Pairs with the derivations in 01-mathematical-foundations.

Thread	Depth
[maths] How small can I make my embeddings before retrieval degrades? ✅	●● 3
[maths] My judge agrees with me 94% of the time. Why is that not good enough? ✅	●● 5
[maths] Should I normalise my embeddings? What actually breaks if I do not? ✅	●● 3

Errata¶

The material is wrong, or more confident than it should be. Confirmed errata become issues.

Thread	Depth
[errata] The κ bands in the reading list are quoted as if they were a standard	● 1

Everything else¶

Thread	Depth
Why does Recall@N go up but full-chain recall stay flat? ✅	●●● 8
My reranker improved evidence recall but full-chain recall is 'inside the noise band'. Do I ship it? ✅	●●●● 10
Should I use RRF or weighted fusion? The notebook says RRF is the default but then measures it losing. ✅	●●● 7
Why is `answer_correct` so low on temporal questions when retrieval looks fine? ✅	●●● 7
Can I use these numbers in a client conversation? ✅	●● 4
The notebook gives different numbers than the README. Which is right? ✅	●● 3

Show and tell¶

Thread	Depth
[capstone] Two of my four improvements were inside the noise band	●● 5
[negative result] Contextual chunking cost 2.4× storage and did not clear the band	●●● 7
[negative result] HyDE cost 2 model calls per query and did not clear the band	●● 4
[solution · EX-12] The analyzer audit — I found three more separators past the one in #1	●● 5
[negative result] Query decomposition changed 47 of 60 candidate pools and moved nothing	●● 4
[solution · EX-19] I hand-labelled 200 answers and my judge scored κ 0.31	●● 3

Design Reviews¶

Thread	Depth
Retrieval for a regulated insurer — 40M docs, per-clause ACLs, 12 weeks ✅	●●● 7
Sufficiency check: a cheap model call, or a trained classifier? ✅	●●● 6
A support desk where a document edited five minutes ago must be findable ✅	●●● 6
Multi-tenant retrieval where two tenants want different encoders ✅	●●● 6

Interview Prep¶

Thread	Depth
Critique my answer: 'How would you separate a retrieval failure from a generation failure?' ✅	●●● 6
How do I talk about a synthetic-corpus project without it sounding like a toy? ✅	●● 4
[round · deployment-engineer shape] 45 minutes: retrieval over classified documents ✅	●● 3
Critique my answer: 'How would you evaluate a RAG system?' ✅	●● 3
[round · platform shape] Ten billion documents, fifty milliseconds ✅	● 1
[round · research-lab shape] Design the experiment that would falsify this paper ✅	● 1

Reading Club¶

Thread	Depth
Lost in the Middle (Liu et al., 2023) — is the U-curve still true, and does it matter? ✅	●● 3
Robertson & Zaragoza (2009) — the probabilistic relevance framework, and why nobody reads it	— 0
Malkov & Yashunin (2018) — HNSW, and the part that breaks in production	— 0
Cormack, Clarke & Buettcher (2009) — reciprocal rank fusion, and the unexplained 60	— 0
Khattab & Zaharia (2020) — ColBERT, and what late interaction actually buys	— 0
Asai et al. (2023) — Self-RAG, and whether a model can grade its own retrieval	— 0
Anthropic (2024) — contextual retrieval, and a precondition nobody states	— 0

Ideas¶

Thread	Depth
A measurement-smell linter — fail a PR that changes retrieval with no interval	●● 4
Replay real cohort questions as an eval slice — for distribution, not volume	●● 4
One command that turns a client corpus into an eval set	— 0
A public leaderboard for the L.A.B. simulator — and why it might be a bad idea	— 0
A retrieval postmortem template, and a public incident log	— 0
Paste a trace, get a verdict — a tiny diagnostic page on Pages	— 0
Generate a lab from a notebook section	— 0

Polls¶

Thread	Depth
[poll] Your change clears the band on one metric and not the other. Do you ship it?	● 1
[poll] How much of a RAG monthly bill is generation tokens?	— 0
[poll] Chunk size: what do you actually use, and did you measure it?	— 0
[poll] Your reranker is 40ms slower and 1.5 points better. Ship it?	— 0
[poll] Which of the four verdicts is most common in your system?	— 0
[poll] Would you let an LLM judge gate a release?	— 0
[poll] Post-filter or pre-filter? And have you measured what it costs you?	— 0

General¶

Thread	Depth
Introduce yourself — and say what you are actually trying to retrieve	— 0
The retrieval opinion you hold that most people here would disagree with	— 0
What did you get wrong that took longest to find?	— 0
A tooling census — what do you actually use to evaluate?	— 0
FDE, ML engineer, research engineer, platform — what actually differs day to day?	— 0
The one number you wish you had measured six months earlier	— 0

✅ marks a thread with an accepted answer. Those are harvested weekly into the FAQ, which links rather than copies — the thread carries the argument that got there, and that is usually the more useful half.