Why a semantic cache breaks even at a 0.01 per cent repeat rate, how to set the threshold from a false-hit budget, and what the semantic half is worth.
Tokenisation to the KV cache, then the arithmetic: 128 KiB a token on Llama 3.1 8B, 16 GiB of cache for one full context window, and why prompt caching buys 22 per cent of the bill and 7 per cent of the clock.
Twenty minutes, no deck. We’ll tell you straight whether this is work we should be doing, and if it isn’t, we usually know who should.
We come back within one business day.
Analytics
We’d like to use Google Analytics to see which pages people find useful. It sets cookies, and only if you say yes. Your choice can be changed at any time under Cookie settings. Privacy