Fine-tuning changes how a model answers. Retrieval changes what it can see. Almost every request that arrives as “train it on our documents” is a complaint about facts: wrong product names, an invented policy, last year’s pricing. Facts are a retrieval problem, and training on the documents will not fix it.
The two stack rather than compete: retrieval for the facts, a schema over the output, a tune only if a shape keeps slipping after the prompt has had its turn. Most teams never reach the third step, and saying so costs us the engagement.
Retrieval against fine-tuning: what each one changes
Retrieval changes what the model can see at question time; fine-tuning changes the shape of what it says. With retrieval you put the answering paragraphs into the prompt, source and date attached, so facts arrive dated and attributable. Edit the document and the next answer changes after the next reindex, so whatever cadence you pick becomes your staleness floor: reindex nightly and you have promised a day.
Fine-tuning works the other way round. You show the model input-output pairs and it moves towards that shape: format, tone, length, a decision boundary. Facts get absorbed lossily and without provenance. Fifty to a hundred pairs move format and tone — OpenAI’s supervised fine-tuning guide recommends starting at 50 and Google’s supervised tuning page at 100. Shifting accuracy needs far more.
So “it does not know our refund policy” is retrieval. “It writes four paragraphs where two sentences would do” is a tuning candidate, and even then, try the prompt first.
How to tell a retrieval problem from a fine-tuning problem
Paste the answering passage into the prompt by hand and see whether the model gets the question right. Collect twenty questions the system got wrong, real ones from real users. For each, paste the passage that answers it alongside four wrong-but-plausible ones, in the order a retriever would have handed them over. One perfect passage measures the ceiling, not the system.
If most now come right, the retriever is your problem and no training run reaches it. If they are still wrong with the answer in front of the model, split the failure: a wrong format or a missed instruction is the prompt, and a wrong conclusion from a correct passage is the model or its reasoning budget. A prompt rewrite costs one rerun of your eval set.
And if you could not find the passage at all, neither technique invents an answer nobody wrote down. If it took ten minutes to find, that is a finding about the documents.
When is fine-tuning worth the training run?
Fine-tuning earns its place in three cases: a fixed output shape, a voice, and a narrow high-volume task that lets you drop to a smaller model. Take the output shape first, and lead with the schema. Constrained decoding makes a schema violation impossible while the model is generating, which no tune can promise. It does not survive truncation: hit the token ceiling and you get a valid prefix that does not parse, so keep the length check. OpenAI’s structured outputs guide documents the guarantee and the refusal path next to each other. A tune is for the other half: valid shape, wrong enum value.
Voice is the second case. Tone is style, and style is what tuning transfers — the one place on this list where a tune genuinely beats a well-built prompt. Six examples in the prompt, scored against the tune on the same replies, and the tune earns its place when the examples stop holding at length.
Volume is the third, and two levers cut that bill before any tune does. The instruction block is a fixed prefix, so prompt caching bills those reads at a fraction of the input rate. Classification is rarely interactive, so batch processing takes another cut on top.
Now the arithmetic. Three hundred thousand calls a month at 1,000 tokens in and 30 out is 300 million input tokens and 9 million output, billed at different rates: (300 × the input price per million) + (9 × the output price). Output is 3% of the token volume, but it is priced four to five times higher, so it lands nearer a seventh of the bill than a thirtieth. At a representative small-model rate of $0.10 in and $0.50 out, that is 300 × $0.10 = $30 against 9 × $0.50 = $4.50, a total of $34.50 of which output is 13%. Input is still the number to attack. Then add the three lines your invoice does not have: training tokens, the tuned inference rate above the base rate, and any hourly charge for keeping the thing served. Against those, set the training set, the eval set and the engineer-weeks. Two quarters is our cut-off. If the saving does not clear the lot in that time, the tune is for the write-up rather than the budget.
What a fine-tune costs after it ships
A fine-tune is a snapshot of a process on the day it shipped, and it goes stale silently. A policy changes, a threshold moves, and the model keeps answering in the old shape, fluently, because fluency is exactly what you trained. Stale retrieval hands you a dated document and you can read the date. A stale tune hands you yesterday’s answer with nothing on it.
You have also taken on a training-data pipeline, an eval set and a retraining cadence. Base models get retired, usually on somebody else’s timetable. When yours goes you redo the work on a newer base that may beat the old tune outright, which is a good outcome and an unplanned sprint.
Use the document’s own units to chunk, not a fixed character window
Chunk on the unit people quote: the clause, the table row, the message. Retrieval disappoints for a narrower reason than most people think. Somebody split every document at 1,000 characters with 200 of overlap, embedded the lot, took the top five by cosine, and shipped. That window came from a tutorial, and a tutorial is sized for convenience rather than for your corpus. Worth a minute of your time: LangChain’s TextSplitter reference documents chunk_overlap at 200, which is where the 200 comes from, and chunk_size at 4,000, which is nothing like the 1,000 everybody copies.
What the chunk should be, by document shape, and what a fixed window does to each:
| Document shape | Chunk on | What a fixed window does |
|---|---|---|
| Clauses, policies, specs | the clause, with its number and heading path | splits a clause, loses the number people quote |
| Tables | the row with its column names, for lookups only | header in one chunk, figures in another, meaning nothing |
| Threads, tickets, minutes | the message, or the decision | welds two speakers or two decisions together |
| Long prose | a window prefixed with title and nearest heading | a paragraph about “the limit” with no clue which limit |
Clauses punish a fixed window hardest, because a refund rule and its exception are usually one sentence split by a comma.
Hybrid search: BM25, embeddings and reciprocal rank fusion
Run a lexical index and a vector index over the same corpus and fuse the two ranked lists. Embeddings are good at paraphrase and weak at exact tokens: part numbers, error codes, surnames. BM25 is the opposite, and it is already the default similarity in OpenSearch.
Negation belongs to neither. “Is refundable” and “is not refundable” differ by one low-IDF token, and under a standard English analyser the stop filter drops “not” altogether, so the two reduce to the same token stream. That wants a metadata filter or a query rewrite, not a better embedding model.
Fuse with reciprocal rank fusion: a document scores the sum of 1/(k + rank) across the lists. Everybody uses k = 60, and it is worth reading why. Cormack, Clarke and Büttcher’s SIGIR 2009 paper says 60 “was fixed during a pilot investigation and not altered during subsequent validation” — near-optimal on their data, and explicitly not a critical choice. Elasticsearch’s RRF retriever still defaults rank_constant to it. Rank fusion also throws away each retriever’s confidence: only the position enters the sum, so a document that barely led its list counts the same as one that led by a mile. Once you can measure, either weight the lists or move to score normalisation, which keeps the gaps. OpenSearch ships a normalization-processor in its hybrid search plugin for that.
When is a lexical index worth adding to a vector store?
Add the lexical half when more than roughly a quarter of your questions carry an identifier; below that, the lift does not pay for a second index to keep in sync. The threshold below is argued on Lamplighter, an invented document-retrieval system over a contracts archive. Lamplighter is not a client and never was. No past employer either. Every number was picked so the arithmetic is checkable, and none of it was measured on anyone’s system.
Lamplighter’s question log holds 200 questions. Of those, 140 quote an identifier — a clause number, a contract reference, a party name — and 60 are pure paraphrase. Vector-only retrieval gets recall@8 of 0.80 on the paraphrase questions (48 of 60) and 0.45 on the identifier ones (63 of 140). Overall: 111 of 200, or 0.56. Add the lexical index, fuse on rank, and the identifier half rises to 0.85 (119 of 140) while paraphrase holds at 0.80. Overall: 167 of 200, or 0.84. Count the misses instead and you land in the same place: 21 identifier questions and 12 paraphrase questions still fail, 33 of 200, and 1 − 0.165 = 0.835.
The lift is therefore 0.40 on the identifier share alone, and 0.40 × 0.70 = 0.28 is the entire headline. Halve that share to 35% and the lift is 0.40 × 0.35 = 0.14, taking 0.68 to 0.82, which is still worth building. At 15% it is six points for a whole second index, and we would not bother.
The obvious reading of 0.56 to 0.84 is that hybrid search fixed retrieval. It did not. Perfect lexical recall across all 140 identifier questions would reach 188 of 200, or 0.94, and the 12 that remain are paraphrase failures no fusion touches. The lexical index is not free either: it is a second thing to reindex, and a stale one will serve up a clause that was deleted last month with every appearance of being the current version of it.
How to measure retrieval quality before blaming the model
Measure the retriever on its own, before you touch the prompt or the model. Use a couple of hundred question-and-passage pairs. Retrieve wide, then rerank: pull fifty to a hundred candidates, score each with a cross-encoder, keep the best five to eight. That is one round trip but N scored pairs, so 100 candidates instead of 50 doubles the cost of that stage and nothing else.
Then watch two numbers. Recall at 100, the depth you retrieve, says whether the right paragraph is reachable at all. Recall at 8, the depth you pass to the model, says whether reranking kept it. A gap between them is a reranker problem, a low ceiling at 100 is a chunking problem, and the share of your questions that carry an identifier tells you which half of the retriever to suspect first.
I cannot tell you in advance which half a given corpus needs, and I have only ever found out by measuring. But that number, rather than the model, is usually what people mean when they say the AI is not good enough.