EX-19 · Calibrate a judge against humans

Difficulty: 🟡 · label difficulty: 2 Stuck? Ask in the clinic thread for EX-19 — one long-running thread per exercise, so the answers accumulate where the next person will look. Submit: post it in Show and Tell titled [solution · EX-19] ..., with your numbers and your interval.

Notebook 06 · ~2 h (needs two people) · Skill: interview Q6

Two people label the same 100 examples against the faithfulness rubric. Compute human–human agreement, then judge–human agreement, then compare.

Acceptance criteria

• Both κ values
• The disagreements read and categorised: judge bug vs ambiguous rubric
• At least one rubric revision, with the κ before and after
• The rubric versioned and fingerprinted

---