The crossover is 77,000 tokens of corpus, and which model you are on moves it by a factor of four.
Corpus size decides it. On Claude Sonnet 5, below about 77,000 tokens — roughly 300 kB of text — putting the whole corpus in the prompt on a cache hit costs less per query than retrieving eight passages from it. Above that, retrieval wins and keeps winning. Fine-tuning is not on this axis: it changes behaviour, not facts.
The ratio behind that number is published. Hany et al., “Inject or Navigate? Token-Efficient Retrieval for LLM Analysis of Transactional Legal Documents” (arXiv:2607.05764, 7 July 2026), derive what their abstract calls a “closed-form caching-crossover rule” and state it as a ratio: “cached injection is cheaper in dollars only while the corpus stays below roughly ten times the retrieval payload”. What nobody prints is the absolute number for a stated payload, the multiplier that moves it between two rows of one price table, or the reason most businesses never get to use it.
Retrieval, long context and fine-tuning are not three ways to do one job
Retrieval and long context are one technique at two settings of a single dial, and fine-tuning is not on that dial. Retrieval changes what sits in front of the model at question time. A long context window is retrieval with k set to everything, billed per token on every query. A fine-tune changes the model. Posing the choice as three options for one job is what stops people doing the division.
The fine-tuning overview in the Microsoft Foundry documentation opens with the definition:
Fine-tuning adapts a pretrained model to your task through additional training on task-specific data. It adjusts the model’s weights rather than adding examples to each prompt.
Weights, then. Not an index.
And the list of what that page says a tune is good for carries a bullet reading “Use retrieved context effectively”, which gives the game away: the vendor ships retrieval and tuning as things that stack. Comparing them honestly needs a fixed set of questions, and building that set is the first job rather than the last.
Use cost per query to choose between RAG and a long context window
Retrieval’s prompt is a fixed size whatever the corpus does, and that one fact settles most of the comparison. The arithmetic runs on Lamplighter, which is invented — a document-retrieval system over a contracts archive, no client’s and no past employer’s, with every figure chosen so the divisions come out checkable rather than measured on anything real.
Lamplighter’s archive holds 48 MB of extracted contract text; one counterparty folder in it holds 41 contracts and 1.6 MB. Chunks are 800 tokens, the retriever returns k = 8 passages, the instruction block is 1,200 tokens, a question is 100 and an answer 400. Traffic is 3,000 queries a month over 22 working days of 480 minutes.
Bytes become tokens by the rule the Claude pricing page prints beside its web fetch tool costs: a “Large documentation page (100 kB)” is “~25,000 tokens”, so 250 tokens per kB and four bytes a token. Prices come from the Model pricing table on the same page, read 29 October 2026.
text
corpus to tokens, at 250 tokens per kB
archive 48 MB -> 48,000 kB -> 48,000 x 250 = 12,000,000 tok 12x a 1M window
folder 1.6 MB -> 1,600 kB -> 1,600 x 250 = 400,000 tok fits
cost per query, Claude Sonnet 5: input $2.00/M, cache hit $0.20/M, output $10.00/M
retrieve 8 x 800 + 1,200 + 100 = 7,700 tok x $2.00/M = $0.0154
stuff 400,000 tok x $2.00/M = $0.80 = 51.9x retrieval
stuff + hit 400,000 tok x $0.20/M = $0.08 = 5.2x retrievalThe archive has no stuffing option at all, so the question only gets interesting one level down, at the folder. Both routes write the same 400-token answer at the same $10, so output cancels on Sonnet 5. That cancellation stops being true two sections from here.
Uncached, both sides pay $2 a million, so the crossover is just the token count: past 7,700 tokens of corpus (30.8 kB) stuffing costs more. Cached, set the two equal. C × $0.20 = 7,700 × $2.00, so C = 15,400 ÷ 0.20 = 77,000 tokens, and back down the ladder 77,000 ÷ 250 = 308 kB.
Derive it again from the monthly bill rather than the per-query rate. Retrieval at 3,000 queries a month costs 3,000 × $0.0154 = $46.20. At $0.20 a million for a cache hit, that $46.20 buys $46.20 ÷ $0.20 = 231 million cached tokens a month, and 231,000,000 ÷ 3,000 = 77,000 tokens of prefix per query — a budget divided, rather than two rates set equal. One line is missing from both routes and I won’t pretend otherwise: the index costs money to store and refresh, and no published price was to hand.
Whether 77,000 is your number turns on one cell. The prompt caching table gives cache read (hit) as “0.1x base input price (0.025x on Claude Fable 5.1 and Claude Mythos 5.1; 0.05x on Claude Opus 5.5 and Claude Sonnet 5.5)”. The crossover is the retrieval prompt divided by that multiplier, so the dollar prices cancel out and the model picks the answer.
text
crossover C = retrieval prompt / cache-read multiplier, at 7,700 tok retrieved
0.1x Sonnet 5, Opus 5, Haiku 5.5 7,700 / 0.1 = 77,000 tok = 308 kB
0.05x Opus 5.5, Sonnet 5.5 7,700 / 0.05 = 154,000 tok = 616 kB
0.025x Fable 5.1, Mythos 5.1 7,700 / 0.025 = 308,000 tok = 1,232 kBSonnet 5.5 sits one row above Sonnet 5 at the same $2 a million base input, with a cache hit at $0.10 rather than $0.20 — twice the corpus before retrieval starts paying. Raising k moves the ladder too: eight passages to sixteen takes the prompt from 7,700 to 14,100 and the Sonnet 5 row to 141,000. Quote 77,000 without the model and the k and you are quoting one cell of a two-way table.
Does prompt caching make long context cheaper than retrieval?
Only when the same corpus is read twice inside five minutes, and most document work is not shaped like that. The Claude prompt caching documentation is blunt: “By default, the cache has a 5-minute lifetime. The cache is refreshed for no additional cost each time the cached content is used.” The clock starts when the request begins, not when its response ends, so generation time comes out of the five minutes.
Price the window. A 5-minute write bills at 1.25× base input, so Lamplighter’s 400,000-token folder costs 400,000 × $2.50 ÷ 1,000,000 = $1.00 to write, against $0.80 for the same request uncached. Two requests inside the window: $1.00 + $0.08 = $1.08 against $1.60. The second read is where it turns over, which is what the page says in words — caching “pays off after one cache read”. The one-hour cache needs two of them, because its write is 2× base input: $1.60 + $0.08 = $1.68 still loses to two plain requests at $1.60, and only at $1.76 against $2.40 does it win. For document work that is the wrong default, and we don’t set it.
The 100,000-token price cliff, and the cache reads that count towards it
On Claude Haiku 5.5 the whole price row steps at 100,000 tokens of prompt, and cache reads count towards the length that gets you there. The step is not the news; what counts towards it is. The rule sits in the Long context pricing section of the Claude pricing page, the fifth subsection under Feature-specific pricing and four below the prompt-caching table this piece has already spent:
Claude Haiku 5.5 is priced by prompt length: a request whose prompt is over 100,000 tokens pays higher prices.
The same paragraph says the length takes in “all of its input tokens”, cache reads and writes included, and that “a request over the threshold pays the higher prices even when part of its prompt is a cache hit”. The cached tokens are themselves what push the request over the line, so caching cannot hold a stuffed corpus on the cheap tier. The Claude context windows page makes the matching point about space rather than money, and adds that “as token count grows, accuracy and recall degrade, a phenomenon known as context rot”. No figure attached, so I won’t attach one.
And output stops cancelling here. It cancelled on Sonnet 5 because the answer bills at $10 a million whatever the prompt length; on Haiku 5.5 the output column steps with everything else. The Model pricing table prints the model twice: $0.10 base input, $0.01 on a cache hit and $0.50 output under the line, then $0.50, $0.05 and $2.50 over it.
text
per query on Claude Haiku 5.5, 400-token answer, prompt length sets the tier
retrieved 7,700 tok in x $0.10/M = $0.00077
400 tok out x $0.50/M = $0.00020 -> $0.00097
stuffed 400,000 tok in x $0.50/M = $0.20000
400 tok out x $2.50/M = $0.00100 -> $0.20100 207x
stuffed, hit 400,000 tok in x $0.05/M = $0.02000
400 tok out x $2.50/M = $0.00100 -> $0.02100 21.6xPrice the input alone and you get 260 and 26. Carry the answer and it is 207 and 21.6, because the cheap side writes its 400 tokens at $0.50 a million while the expensive side writes the same 400 at $2.50.
Now collect that four-bytes-a-token rule from two sections back, because a threshold in tokens is not a threshold in bytes. The pricing page’s FAQ says “Claude 4.7 and later models” use “a newer tokenizer that produces approximately 30 percent more tokens for the same text than earlier models”. Sonnet 5 and Haiku 5.5 are both past 4.7, so the obvious estimate is the wrong one here. Lamplighter’s folder is 400,000 tokens at four bytes a token but nearer 400,000 × 1.3 = 520,000 on the tokeniser these two models use, which puts it well over the Haiku line where the rule of thumb had it under. Retrieval’s 7,700 is counted in tokens already and doesn’t move. So every stuffing figure here is about 30% optimistic, and the 308 kB crossover shrinks the same way, to 308 ÷ 1.3 = 237 kB of real text. I can’t tell which tokeniser the four-bytes rule assumes, so treat any byte-derived count as plus or minus thirty per cent and count the real thing with the token-counting endpoint.
What fine-tuning actually changes, in the provider’s own words
A tune changes weights. Behaviour and output shape are what move, and the vendor selling the tune says in its own documentation that current facts are not what weights are for. The Microsoft Foundry fine-tuning page lists six uses: less prompt overhead; style and structure; tool use; better accuracy and relevance; better use of retrieved context; dropping to a smaller model. Read that list for what it does not contain. No bullet says “hold your corpus”. The paragraph directly under it goes further:
Benefits aren’t guaranteed; held-out evaluation helps you compare the fine-tuned model with your baseline. Fine-tuning adds training and hosting costs and doesn’t replace retrieval for current information or application-level safety controls.
Two of the six can cut a bill, the shorter prompt and the smaller model; the other four change behaviour, which is the thing a tune moves reliably. I won’t put a percentage on how far, because the page attaches none and I have run no evaluation I could publish.
Lewis et al. make the point from the other end, with a magnitude. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, Section 4.5, under the paragraph headed Index hot-swapping, puts a 2016 Wikipedia index and a 2018 one under an unchanged model and asks about world leaders: 70% right with the 2016 index on 2016 leaders, 68% with the 2018 index on 2018 leaders, then 12% and 4% once index and question year are crossed. The 2018 case falls 68 − 4 = 64 points. Not a single weight moved. For a non-parametric memory “knowledge can be easily updated at test time”, the paper says, where “Parametric-only models like T5 or BART need further training to update their behavior as the world changes.”
At what query volume does a fine-tune pay for itself?
About 110,000 queries a month on the one tuned model that saves anything — and both that model and the platform selling it are being switched off. These prices are OpenAI’s published list, read 29 October 2026, because Anthropic publishes no first-party fine-tuning price. The fine-tuning section of that page opens not with a number but with “OpenAI is winding down the fine-tuning platform. The platform is no longer accessible to new users”, and the deprecations page puts 6 January 2027 on the end of it, after which “Active existing customers will no longer be able to create new fine-tuning jobs”. If you haven’t already fine-tuned something there, this is not a decision you get to make.
The shape of it outlives the vendor, though. Base gpt-4.1-mini is $0.40 a million input tokens, $0.10 cached and $1.60 output; fine-tuned gpt-4.1-mini-2025-04-14 is $0.80, $0.20 and $3.20. Divide across the row: 0.80 ÷ 0.40 = 2, 0.20 ÷ 0.10 = 2, 3.20 ÷ 1.60 = 2. A tuned model is exactly twice its base in every column, nano included, so a tune whose only saving is a shorter prompt breaks even at 2P′ = P, or P′ = P/2. Lamplighter’s prompt doesn’t get near that, even with 1,000 tokens of instruction block folded into the weights.
text
per query, 7,700 tok in / 400 tok out unless shown, OpenAI list prices
base mini 7,700 x $0.40/M + 400 x $1.60/M = $0.00308 + $0.00064 = $0.00372
tuned mini 6,700 x $0.80/M + 400 x $3.20/M = $0.00536 + $0.00128 = $0.00664 1.78x
tuned nano 7,700 x $0.20/M + 400 x $0.80/M = $0.00154 + $0.00032 = $0.00186 0.50xThe tuned mini costs more per query than the model it replaced, so no volume amortises it. Only the model drop saves anything: $0.00186 a query, exactly half, because tuned nano is half base mini in both columns. Lamplighter’s training set is 900 examples averaging 700 tokens over 3 epochs, so 900 × 700 × 3 = 1,890,000 tokens — tokens-in-file times epochs being my assumption, not something either page states. At nano’s published $1.50 a million that is $2.835. Read the cell rather than the heading, by the way: the section heads “Finetuning Prices per 1M tokens”, but o4-mini-2025-04-16 prices training at “$100.00 / hour” in that same column. Break-even on the training bill alone is $2.835 ÷ $0.00186 = 1,524 queries, fifteen days at 3,000 a month. That is the number this comparison prints. It is the wrong one.
Now the line the invoice does not have. One engineer-week at our published applied AI rate of $5,000 a month for 160 hours, so $31.25 an hour, is 40 × $31.25 = $1,250. That puts the tune at $1,252.835, with the compute at 0.23% of it ($2.835 ÷ $1,252.835). Break-even becomes $1,252.835 ÷ $0.00186 = 673,567 queries, 224.5 months at 3,000 a month.
Eighteen years and eight months.
And the model it is computed on is already gone. The deprecations page gives gpt-4.1-nano-2025-04-14 a shutdown date of 23 October 2026, six days before I read those prices, and adds that “Fine-tuned versions of these models follow the same shutdown schedule as their underlying base models”. The snapshot is dated 14 April 2025, so it lived about eighteen months against a payback of 224.5 — twelve times the life of the thing the money was spent on. Its sister gpt-4.1-mini-2025-04-14 goes on 14 January 2027, so read the whole exercise as a worked method on published list prices rather than a buy recommendation. The Foundry page’s Maintenance row says it in the vendor’s voice: “Reassess quality as task data changes or a base model approaches retirement. Retraining and redeployment can be necessary.”
Derate the saving 3× to $0.00062 and it is 2,020,702 queries, 56 years. Call the whole tune one engineer-day instead, $252.835: 135,933 queries, 45 months. The decision survives a 5× move either way. Double the traffic to 6,000 a month and payback is 112 months; halve it to 1,500 and it is 449 months.
Now the threshold, derived rather than asserted. The two-quarter payback cut-off is ours, and the Foundry page corroborates that horizon rather than setting it, since Maintenance is a standing challenge there and not a date. Six months means 673,567 ÷ 6 = 112,261 queries a month. Below roughly 110,000 a month we don’t fine-tune to cut inference cost, and I wouldn’t quote for it. Lamplighter runs 37 times under that line.
Long context wins when the question ranges over the whole set
Long context wins when no value of k is complete, which is a structural condition rather than an economic one. “Which of these 41 contracts has the shortest notice period” has no top-k answer, because no k guarantees completeness, and raising k until it does is stuffing the context with extra machinery in front of it.
There is a third answer, and it isn’t long context. Pull the 41 notice periods into a table once, then aggregate over rows: the model never sees a contract and the answer is computed rather than recalled. An engineer-day at the same $31.25 an hour is 8 × $31.25 = $250, so $250 ÷ $0.80 = 312 stuffed queries buys the extraction outright. Ask that question more than about 310 times and the table is already cheaper.
Below 310 it isn’t, and that regime is where I’d still stuff 400,000 tokens. The extraction doesn’t exist yet, the aggregation changes shape between askings, and each new shape is another engineer-day against another $0.80. The alternative is a person reading 41 contracts, and I’d sign that invoice without arguing. One caveat, though: a 1M-token window is not a 1M-token request everywhere, because the first-party Claude API caps a Messages request at 128 MB and returns 413 request_too_large, while Bedrock caps it at 20 MB.
Row-level permissions are why most businesses cannot stuff the context
A context window has nowhere to put a filter, and a retriever does. Of the four document-level access approaches the Azure AI Search document-level access overview lists, security filters is the one not marked (preview), and that page’s Choose an approach table gives the reason in six words: “API-agnostic, generally available, and based on simple string matching.” The security filter pattern page describes the mechanism plainly: “The principal is just a string, used in a filter expression, to include or exclude a document from the search results.” That filter rides on every query, and its shape matters: an equality expression over hundreds or thousands of values “slows down query response time by many seconds” where search.in gives “subsecond response times”.
A cached prefix has no equivalent. One prefix is one object. If six of Lamplighter’s 41 contracts are closed to a reader, that prefix is wrong for them, so you need one cache per permission set. Lamplighter has 60 users across 14 of them.
text
3,000 queries / 14 permission sets = 214 per set per month
214 / 22 working days = 9.7 per set per day
480 min / 9.7 = one every 49 min, against a 5-minute cacheA second request inside that window almost never arrives, so every request pays the write: $1.00 instead of $0.08, or $1.00 ÷ $0.0154 = 65 times retrieval. Derive the volume where it would work instead, and watch the halving. Two arrivals inside one five-minute window need a mean gap of 2.5 minutes, not five, so 480 ÷ 2.5 = 192 queries per set per working day. That is 192 × 22 = 4,224 a month per set, and 4,224 × 14 = 59,136 across all fourteen. Below roughly 60,000 queries a month at this fan-out the cache is decoration. Cut the fan-out to 5 sets and the bar is 192 × 22 × 5 = 21,120 a month, still seven times Lamplighter’s traffic. All of that assumes arrivals spread evenly, which they don’t: real traffic clusters, and clustering is the one thing that helps a five-minute cache. So 60,000 is the bar for traffic with no shape to it, and I have no distribution to put behind a better one.
The retriever side is not free either. Setting retrievable to false on the security field “isn’t a content-obfuscation or field-level security mechanism”: it stops the field coming back in a response, nothing more. A flag name that reads like a guarantee and isn’t one. Revocation lags as well, and the overview page scopes that carefully to the three preview approaches rather than to security filters — a permission change reaches search results only once the index has it, “through the source-specific mechanism, for example, a subsequent indexer run, a push-API update, or a Purview-driven refresh”. A security filter is a copy too, a group_ids field you push yourself, so the lag is real there and the refresh is yours rather than the service’s. Build on the approach that isn’t in preview. Then own the refresh, because nobody else is going to.