A semantic cache breaks even at one repeat in 10,000 — then the threshold eats it

Embedding one query with text-embedding-3-small costs $0.00000084. Generating the answer that query would otherwise get costs $0.0084. Both divide out of OpenAI’s pricing page, read on 29 October 2026, against a 42-token question and a 6,400-in, 400-out answer I have chosen below.

Embedding every query costs about a ten-thousandth of what generating one answer costs, so a semantic cache pays for itself at a repeat rate of 0.01 per cent — one query in 10,000. Cost is never what decides whether to build it. The false-hit budget is.

Lamplighter is an invented document-retrieval system over a contracts archive, not a client and not a past employer. It answers 180,000 contract questions a month. Each question runs about 42 tokens, each prompt about 6,400 input tokens (a 1,200-token system prompt plus eight retrieved chunks) and each answer about 400 output. The parameters are chosen, one is deliberately odd, and they are this article’s own: the same invented system is sized differently elsewhere in the library.

Park one thing: in pgvector the <=> operator returns a distance, not a similarity, so closer means smaller and every threshold below is written that way round. It comes back in the threshold section.

What repeat rate does a semantic cache need to break even?

0.01 per cent, one repeat in 10,000, and the division takes two lines. With a cache you pay to embed every query and to generate the ones that miss: V × C_emb + V × (1 − r) × C_gen. Without one you pay V × C_gen. Set the two equal, cancel V, cancel the generation term that survives on both sides, and what is left is C_emb = r × C_gen, so r* = C_emb ÷ C_gen.

OpenAI’s pricing page prints its unit above the table: “Prices per 1M tokens.” It lists gpt-6.1-terra short context at $1.00 input and $5.00 output, and text-embedding-3-small at $0.02 with no output price at all. Walk each rate down to one token: $1.00 per 1M is $0.001 per 1,000 and $0.000001 a token; $5.00 per 1M is $0.005 per 1,000 and $0.000005 a token; $0.02 per 1M is $0.00002 per 1,000 and $0.00000002 a token. Lamplighter bills at the short-context rate, because the page sets the tiers at “Short context: ≤272K input tokens.”

One answer costs 6,400 × $0.000001 = $0.0064 of input plus 400 × $0.000005 = $0.0020 of output, so C_gen = $0.0084. One embedded question costs 42 × $0.00000002 = $0.00000084, so C_emb = $0.00000084. Divide: $0.00000084 ÷ $0.0084 = 0.0001. One repeat in 10,000.

Guess before you divide. Thirty per cent of traffic is the number I would have reached for, and nothing is behind it but habit: 0.30 ÷ 0.0001 = 3,000 times what the two prices ask for.

Now the term that division leaves out. r* = C_emb ÷ C_gen prices tokens and nothing else — no vector column, no index, no probe. Give the lookup a price L a query and r* becomes (C_emb + L) ÷ C_gen. For the break-even to reach one per cent, a hundred times higher, L would have to be 0.01 × $0.0084 − $0.00000084 = $0.00008316 a query, or $83.16 a million. Nobody invoices that. An HNSW probe is RAM in a Postgres box you already run, and $0.00008316 × 180,000 = $14.97 a month is about what a share of one costs. The missing term is fixed rather than per-query, and a fixed cost has no break-even rate; it has a break-even volume, three sections down. Every published version of this division I have read sits at least an order of magnitude above 0.01 per cent, and the lookup is where the gap lives.

The same break-even rate, derived without dollars

Price one answer in embedding tokens rather than money and the same 10,000 falls out, with no currency in the working. An input token costs 50 times what an embedding token costs, and an output token 250 times, both ratios read off the two rows quoted above; one answer is 6,400 input tokens and 400 output: 6,400 × 50 + 400 × 250 = 320,000 + 100,000 = 420,000 embedding tokens’ worth of model time. One cache key is 42 embedding tokens. 420,000 ÷ 42 = 10,000.

Two routes share no intermediate figure except the token counts, and they land on the same number. The second is the first with its currency cancelled rather than carried, so read it as a check on the arithmetic; the genuinely independent route is the top-down one below.

Now look at what is absent from r* = C_emb ÷ C_gen. There is no V in it. Both sides grow linearly with traffic, so the break-even is identical at 18,000 queries a month and at 18,000,000. Volume moves the prize. But it never moves the break-even, which is why the cost question turns out to be the boring one.

Pick the similarity threshold from a false-hit budget, not from a round number

Start from the wrong-answer rate the business can live with, read the threshold off the distance distribution, and for Lamplighter that lands on a cosine distance of 0.02. The tolerance is commercial, not technical: for a contracts archive I would set one wrong answer per 10,000 questions asked, so ε = 0.0001.

A cached answer goes wrong in two ways, and only one of them is the threshold’s fault. Staleness is the other, and the pgvector section collects it. The threshold’s own failure is a stored question that sits inside t but asks something else, so wrong answers per query from that cause is f(t) × h(t): the false-hit share inside t times the hit rate at t. Call that product the false-hit budget. A threshold is admissible when it lands at or under ε. The product is the move, and the threshold tables I have read print the two rates side by side without ever multiplying them.

Chosen false-hit shares and hit rates for Lamplighter at four cosine-distance thresholds, with the wrong answers per query served that each pair implies. The first three columns are chosen, not measured; the fourth is their product.

Multiply the two middle columns and only the bottom two rows clear ε: (1 ÷ 900) × 0.07 = 0.00007778 at t = 0.02, one wrong answer in 12,857, and (1 ÷ 4,000) × 0.04 = 0.00001 at t = 0.01, one in 100,000, which buys only a lower hit rate. Take the loosest admissible threshold: 0.02, at a 7 per cent hit rate.

Now collect the operator. A cosine distance of 0.02 is a cosine similarity of 0.98, and the pgvector README gives the conversion in the Distances block of its Querying section: “For cosine similarity, use 1 - cosine distance”. Write the predicate the way people talk about it, WHERE embedding <=> $1 >= 0.98, and it still parses — except that what comes back is every row further away than 0.98 rather than closer. The cache then serves its worst match at full confidence and logs nothing. Sign errors in a threshold aren’t caught by types. They are caught by one labelled pair that should hit and doesn’t.

I don’t know the distance distribution for anyone’s corpus, and neither does anybody quoting a threshold at you. The shape is the part I would bet on: false hits fall faster than hits as the cut tightens, because near-duplicate phrasings cluster much closer than merely related questions. The numbers want a labelled pair set first, built the way an evaluation set gets built.

Why a 0.98 cosine threshold leaves you $60 a month

Because the cache hits 7 per cent of queries at that threshold and a free string hash was already catching 3 per cent, so the semantic part of the system is worth four percentage points and nothing more. Lamplighter’s generation bill is 180,000 × $0.0084 = $1,512 a month; its embedding bill is 180,000 × 42 = 7,560,000 tokens, and 7.56M × $0.02 per 1M = $0.1512. Fifteen cents. Divide that by one answer and the month’s whole embedding spend buys back $0.1512 ÷ $0.0084 = 18 answers, or 18 ÷ 180,000 = 0.01 per cent of traffic. The same break-even, top down, and nothing in this route touches the algebra. A 7 per cent hit rate saves 0.07 × $1,512 = $105.84 a month.

Before costing the alternative, the honest case for it. A lowercased, whitespace-normalised hash of the question string has no false hits by construction and needs nothing new in the database: no embedding call, no vector column, no index, no threshold to tune. It catches the repeats that matter most, the same person asking the same question twice. Its hit rate is 3 per cent, chosen like the rest: 0.03 × $1,512 = $45.36 a month.

The semantic premium is the money a semantic cache saves beyond what a normalised-string hash would have saved anyway, at the tightest threshold the false-hit budget allows. Per query it is (h − 0.03) × $0.0084 − $0.00000084, net of the embedding call every query now pays for. At h = 0.07 that is $0.00033516, so $60.33 a month and $723.95 a year. The two rates aren’t independent, which is where the number is softest: a cache’s hits contain the hash’s, so move the hash to 5 per cent and the same formula gives $30.09.

Sixty dollars a month isn’t a system.

A new component has to earn $2,000 a month before I would own it — chosen, and roughly where a thing starts paying for the attention it takes. V* = $2,000 ÷ $0.00033516 = 5,967,299 queries a month. One rung at a time: 5,967,299 ÷ 30 days = 198,910 a day, 198,910 ÷ 86,400 seconds = 2.30 a second, sustained. Lamplighter sits 5,967,299 ÷ 180,000 = 33.2 times under that, so I would not build the semantic cache for it, and a firm selling you one here would be selling the third-best thing on the table.

Running on the hash alone has a price. It never catches a paraphrase, so two people asking the same renewal question in different words both pay twice: $60.33 a month left on the floor on purpose.

pgvector cosine distance, HNSW defaults, and the filter that returns four rows

pgvector’s cosine distance operator is <=>, its HNSW build options default to m = 16 and ef_construction = 64, and hnsw.ef_search defaults to 40. The README defines the operator in four words, in its Querying section: “<=> - cosine distance”. Index Options prints the build pair as “(16 by default)” and “(64 by default)”. The query-time one I had wrong from memory: I would have said hnsw.ef_search is 64 too, by symmetry, and had to open the README to find 40.

The third default is the expensive one. A third of the way down the file, at line 450 of 1,361 and under the eighth of its top-level sections, headed Filtering, the README says:

With approximate indexes, filtering is applied after the index is scanned. If a condition matches 10% of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows will match on average.

A semantic cache needs a freshness predicate, because staleness is the second way a cached answer goes wrong and an answer about a contract since amended is worse than no cache at all. Add the predicate and it runs after the graph scan rather than during it. Hit rate falls, nothing throws, and the dashboard shows a cache that isn’t hitting.

-- 0.02 is the threshold derived above, and it is a DISTANCE, not a similarity
BEGIN;
SET LOCAL hnsw.iterative_scan = strict_order;   -- pgvector 0.8.0 and later
SELECT answer
FROM   answer_cache
WHERE  created_at > now() - interval '7 days'
  AND  (question_embedding <=> $1) <= 0.02
ORDER  BY question_embedding <=> $1
LIMIT  1;
COMMIT;

The BEGIN is load-bearing. PostgreSQL’s SET reference page says of LOCAL, down in its parameter list: “Issuing this outside of a transaction block emits a warning and otherwise has no effect.” Drop it and the setting reverts before the SELECT sees it.

Iterative scans arrived in pgvector 0.8.0 and keep going until enough rows survive the filter, bounded by hnsw.max_scan_tuples, which defaults to 20,000. The current release is 0.8.7, dated 1 October 2026 in the CHANGELOG, where 0.9.0 is marked unreleased and drops Postgres 13 and 14. A dimension trap sits beside it, and the trap is a default rather than a model. OpenAI’s embeddings guide says “By default, the length of the embedding vector is 1536 for text-embedding-3-small or 3072 for text-embedding-3-large”, and pgvector’s HNSW index takes the vector type only “up to 2,000 dimensions”. Keep the larger model’s default and CREATE INDEX stops at line 707 of src/hnswbuild.c with column cannot have more than 2000 dimensions for hnsw index. Storage runs to 16,000; the index does not. The README’s FAQ offers halfvec and 4,000 dimensions, and asking the endpoint for 1,536 is cheaper.

Model choice and prompt size both save more than the cache does

Model choice saves $13,608 a month and a shorter prompt $468, against the cache’s $60.33 on exactly these numbers, and neither of them can return a wrong answer. Start with the model. The same pricing page lists gpt-6-astra at $10.00 input and $50.00 output per 1M short-context tokens, which is $0.00001 and $0.00005 a token, so one answer costs 6,400 × $0.00001 = $0.064 plus 400 × $0.00005 = $0.020, or $0.084. Moving from astra to terra saves $0.084 − $0.0084 = $0.0756 an answer and 180,000 × $0.0756 = $13,608 a month, with no threshold to tune and no wrong answers to review.

And the prompt is cheaper still to attack. Eight chunks are 6,400 − 1,200 = 5,200 tokens, so four are 2,600 and the prompt is 3,800: 3,800 × $0.000001 + 400 × $0.000005 = $0.0058 an answer, a $0.0026 gap and $468 a month, conditional on recall holding. Prompt caching is the lever people reach for before either. In a retrieval prompt it discounts only the stable prefix, 1,200 of Lamplighter’s 6,400 input tokens, and the argument in the KV cache is the bill prices that share against a 40 per cent floor this prompt sits well under. At the page’s $0.05 cached rate the answer costs 1,200 × $0.00000005 + 5,200 × $0.000001 + 400 × $0.000005 = $0.00726, and the break-even moves to $0.00000084 ÷ $0.00726 = one repeat in 8,643. We would decline to build a semantic cache for anyone who had skipped all three.

Double the traffic to 360,000 queries and the generation bill goes to $3,024, the embedding bill to $0.3024, the premium to $120.66, and the break-even stays at 0.01 per cent, because volume cancels. Derate C_gen fourfold instead, to a 1,600-token prompt and a 100-token answer: 1,600 × $0.000001 + 100 × $0.000005 = $0.0021, and the break-even becomes $0.00000084 ÷ $0.0021 = 0.04 per cent, one repeat in 2,500. The conclusion doesn’t flip. The premium falls to (0.04 × $0.0021 − $0.00000084) × 180,000 = $14.97 a month.

When a semantic cache is worth building, and when it is not

A semantic cache is worth building when the tolerance for a wrong answer is loose enough to admit a threshold that actually hits, and that tolerance comes from what the answer is for, not from how much traffic arrives. Point the same table at a product-docs assistant, where one wrong answer in 100 is fine because the reader has the docs open anyway. ε = 0.01 admits t = 0.05, whose budget is 0.0036, one in 278, at an 18 per cent hit rate: 0.18 × $1,512 = $272.16 a month, and a premium over the hash of ((0.18 − 0.03) × $0.0084 − $0.00000084) × 180,000 = $226.65. Same prices, same volume, same model. The hit rate is two and a half times what it was, and the only thing that moved was the tolerance.

So the first line of a cache design document is the acceptable wrong-answer rate, and a team that can’t write that number down hasn’t earned a cache yet. Everything after it is arithmetic: the threshold, then the hit rate, then the premium. Above roughly six million queries a month the premium clears $2,000 and I would build it; at Lamplighter’s 180,000 I would not.

The component is small, and small isn’t free — a vector column, an index, an embedding call on the write path, a threshold and a review queue for the hits that turned out wrong. More of our applied AI work is taking a component out than putting one in. Prices here were read on 29 October 2026, the pgvector defaults against release 0.8.7.