A demo passes because somebody asked it the questions it answers. The trouble starts the next morning, when the prompt changes or the model is swapped, and nobody can say what that did to the hundred answers no one re-read.
Thirty cases at ninety per cent carry a standard error of sqrt(0.9 × 0.1 / 30) = 0.0548, so the ninety-five per cent interval runs about eleven points either way. Eighty per cent sits inside it. A set that small cannot gate a release. It can only tell you the prompt still parses.
Why temperature zero does not make LLM output repeatable
Temperature zero stops the sampling, not the variation: the kernels pick a floating-point reduction order by batch size, so what else was batched alongside you changes the arithmetic. Batch-invariant kernels remove it, and a hosted endpoint doesn’t give you them. Thinking Machines’ Defeating Nondeterminism in LLM Inference counts 80 unique completions across 1,000 identical temperature-zero requests to Qwen3-235B, falling to 1 once the kernels are batch-invariant.
Pin the model too. Some providers resolve an unversioned name to whatever snapshot is current, others now pin every id — Anthropic’s models overview says every Claude model ID is a pinned snapshot, dateless ones included. Record the exact id each run used, for the system under test as much as the judge.
And a prompt edit touches every answer the system will ever give. Five complaints arrive, you rewrite, they pass, you ship. Five of a hundred and twenty-five question shapes is 5 / 125 = 4 per cent, and they are the five with the most traffic: the best case, not a sample.
Where the evaluation set comes from: questions people actually asked
Fifty to two hundred real questions, pulled from support tickets, the search log, the channel where people ask what this system answers, and last quarter’s complaints. Don’t write them yourself. A team writes the questions its own system answers.
Each row wants the question, the answer a knowledgeable colleague would give, the chunk ids it should come from, and whether the right behaviour is to decline. Take the decline share from the traffic you sampled: one ticket in three with no answer in the corpus means one row in three. Label at the granularity you retrieve at, because a labelled document scores a hit on any chunk of it. That is looser than it sounds, and it hides the failure I come back to below, where a table reaches the model without the heading that says what its columns mean.
Measure retrieval recall separately from answer accuracy
One aggregate at 0.58 sends a fortnight into prompt rewrites that cannot work, because it never says whether the passage holding the answer reached the context. Ask that instead. In a hundred-case set with twenty decline rows there are 100 − 20 = 80 answerable questions; say the right passage reaches the top ten for 62. Accuracy then has a ceiling of 62 + 20 = 82, plus whatever the model answers from memory, which your grounding rubric should fail rather than reward.
No prompt lifts the 62.
Retrieve fifty, rerank to ten, keep three numbers: recall@10 from the retriever, recall@10 after the reranker, recall@50. The last is a ceiling no reranker beats, since reordering cannot add a document nobody fetched. Ragas names the retrieval side differently — context precision, context recall, context entities recall — but not one of them is recall@k, and most are model calls themselves. Compute recall@k yourself. It needs no model. If recall@50 is low, the reranker isn’t the problem.
The chunker is.
How chunk size splits a table from its heading
A table longer than the overlap gets separated from its caption with probability (table length − overlap) ÷ (chunk size − overlap), which at the common defaults is worse than one table in three. The corpus below is invented and so is Lamplighter, a document-retrieval system over a contracts archive. Nobody’s archive is being described, and the counts were picked to make the arithmetic come out in round-ish numbers. The one thing not invented is the default chunk size, in the splitter’s own source: LangChain’s TextSplitter declares chunk_size: int = 4000, chunk_overlap: int = 200 and length_function defaulting to len. The unit is characters, not tokens. “Overlap in characters between chunks”, says the docstring.
Four variables stay live: chunk size, overlap, table length, plus retrieval depth k fixed at 10. Converting at four characters to a token throughout — a convention, not a measurement — the defaults give 4,000 ÷ 4 = 1,000 tokens of chunk and 200 ÷ 4 = 50 of overlap, so chunk starts advance by 1,000 − 50 = 950 tokens. A table of length T survives only when it lands wholly inside one window, which happens for (1,000 − T) of every 950 start positions, so the split probability is (T − 50) ÷ 950. At T = 400 that is 350 ÷ 950 = 0.368. At T = 1,800 the table outruns the chunk and the probability is 1. Check the first another way: boundaries fall every 950 tokens, so a 400-token table spans 400 ÷ 950 = 42 per cent of a stride and expects a boundary that often, the overlap rescues 50 ÷ 950 = 5 per cent of positions, and 42 − 5 = 37.
Call it the caption-detachment rate: the share of tables reaching the model with their header row in a different chunk from some of their numbers. Across 400 documents at three tables each, 400 × 3 = 1,200 tables, of which 1,200 × 0.368 = 442 arrive detached.
The honest case for the obvious fix, chunk_size at 2,000 tokens, is that it is one parameter and no code. Costed, it works: (400 − 50) ÷ (2,000 − 50) = 350 ÷ 1,950 = 0.179, so 1,200 × 0.179 = 215 broken tables instead of 442. The price is dilution. A 30-token answer sentence is 30 ÷ 1,000 = 3 per cent of a default chunk and 30 ÷ 2,000 = 1.5 per cent of the doubled one, mean-pooled against twice as much unrelated contract text. I can tell you the signal share halves. I can’t tell you what that does to recall@10 without measuring it on your corpus, and I won’t invent a figure.
The crossover is T = chunk size. Below it a bigger chunk genuinely wins; at or above it the probability is 1 until the chunk outgrows the table. Lamplighter’s longest tables run 1,800 tokens, so guaranteeing those intact needs 2,000 and more once the prose around them is counted, which leaves prepending the caption and header row to each chunk that continues a table. That isn’t free either: 40 tokens of header on the far side of every split, and 442 × 40 = 17,680 duplicated tokens in the index. Halve the mean table to 200 tokens and the rate only falls to (200 − 50) ÷ 950 = 0.158, still 190 broken tables, so the decision doesn’t flip. The approach stops working when documents have no stable heading structure, and then the fix is upstream of the splitter.
When an LLM-as-judge score is worth trusting
A judge is worth trusting on tone and grounding, and worth nothing on anything with a canonical value: a figure, a date, a document id, a status or a schema belongs in the deterministic tier. promptfoo evaluates every assertion in the list, so the tiers rank trust rather than execution order; split them across two CI stages if the cheap one must gate the judged one.
Judges favour their own outputs by an amount that varies with model and task, which is reason enough to judge with a different model pinned to a snapshot id. Unanchored one-to-five scores collapse onto a single digit, typically three, and the integers that remain produce ties rather than a ranking; G-Eval is where that was measured. Ask one binary question per named criterion, and check refusals deterministically, because a refusal is not a judgement.
Calibrate first, and not on raw agreement. Grade fifty by hand. Mark forty-five correct and a judge that answers “correct” unconditionally agrees on 45 of 50, ninety per cent, and catches none of the five failures you wanted it for; having no variance at all, its Cohen’s kappa on that table is exactly 0. Read the kappa, not the percentage. McHugh’s review of the statistic says “any kappa below 0.60 indicates inadequate agreement among the raters”. Below that line, fix the rubric, not the system.
Gold labels belong in metadata rather than vars, where they could reach the prompt — an invented case:
yaml
tests:
- vars: { question: "Refund window on an annual plan?" }
metadata: { doc: doc-0001, days: 14 }
assert:
- { type: is-json, value: { type: object, required: [answer, citations, refused, days] } }
- { type: javascript, value: "JSON.parse(output).refused === false" }
- { type: javascript, value: "JSON.parse(output).days === context.test.metadata.days" }
- { type: javascript, value: "JSON.parse(output).citations.includes(context.test.metadata.doc)" }
- { type: llm-rubric, provider: "<different model, pinned>", value: "Is every claim supported by the retrieved context?" }Two traps there, both checkable in a minute. The is-json schema needs type: object, because JSON Schema ignores required for anything that isn’t an object, so without it a bare string or an array sails through the assertion written to catch it. And the gold label sits at context.test.metadata; the shorter context.metadata is documented as a shortcut to the provider response’s own metadata, so it reads undefined, and against a model that omitted days the comparison becomes undefined === undefined and passes.
Run the evaluation set in CI on every prompt change
Keep the prompt in the repository beside the set and the runner, then run the set on every change to any of the three. Model upgrades and chunking edits count: that is where a change nobody labelled a prompt change does its damage. promptfoo’s GitHub Action posts the comparison on the pull request, DeepEval suits a team already in pytest, and LangSmith holds the dataset and compares experiments. Name the gate in writing: no case that passed may fail, and an aggregate drop larger than your noise floor blocks the merge.
Get the noise floor by running the set five times unchanged and taking the spread, which covers generation variance but not set size. At ninety per cent on a hundred cases the standard error is sqrt(0.9 × 0.1 / 100) = 0.03, three points, so the ninety-five per cent interval is about six points either way, which swallows a move from 87 to 89. Read the flipped cases instead: the same questions scored twice is a paired comparison, far more sensitive than the aggregate. Two hundred cases at three calls each is 200 × 3 = 600 calls a pull request, so cache by prompt hash and keep the judged set off every push.
What to log so you can reproduce a complaint
“It gave a bad answer on Tuesday” has to become one lookup, which means storing the prompt version, the retrieved chunk ids with their text as retrieved, the model plus its snapshot id, the tool calls, the output, the token counts, the latency. Langfuse says a trace captures “the exact prompt sent, the model’s response, token usage, latency, and any tools or retrieval steps in between”. Prompt version is the one item tracing alone won’t give you; it comes from linking prompt management to the trace. Decide what gets redacted and how long traces are kept — Langfuse masks fields and sets per-project retention down to three days, though retention itself is a paid tier — and self-host when the retrieved text cannot leave your network.
Every reproduced complaint becomes a new row. Do only that and the set fills with bugs already fixed while the prompt quietly tunes itself to them, so hold back a slice the authors never see and resample live traffic quarterly. Quarterly is a guess, mind. I have no defensible rule for the interval.
What pass rate should gate the launch?
Ninety per cent of the answerable cases correct, and at most one in twenty of the decline cases answered confidently rather than declined, agreed in writing with whoever is accountable when it’s wrong. The decline number is the stricter of the two because a confident wrong answer gets acted on. Ninety per cent across two hundred of your own tickets means something. Ninety per cent across thirty cases written by the people who wrote the prompt tells you they write good questions, and a standard error of sqrt(0.9 × 0.1 / 30) = 0.0548 puts the interval eleven points either way, so eighty per cent and ninety per cent are the same measurement.
But if you have one prompt and one surface over a corpus you could read in an afternoon, don’t build any of this yet. Read fifty outputs by hand and write down what was wrong. Come back when there is a second change to make. The set earns its keep from the second change onward: a cheaper answer than the one we’d be paid for, and still the right one. When it is time, the arguments that decide the shape of the thing are whether retrieval is the real problem and whether the open loop needs to be an agent at all; how we build these is on our AI and automation page.