Semantic Caching Is A Practical Way To Cut The Cost And Latency Of Large Language Model Applications. Instead

The trouble is that a single threshold is a blunt instrument. Consider a customer who writes "I want to cancel my subscription" and another who writes "I want to pause my subscription." Those are near-miss queries. They share most of their words and structure, so their embeddings will be close, but the correct answers are entirely different. A semantic cache that treats them as identical will return a false cache hit, serving an answer that is plausible but wrong. The same problem appears with small wording changes that flip meaning, https://zenodo.org/records/23667256 such as "not available" versus "available" or "monthly" versus "annual.

A 2026 preprint by Guerin Green of Novel Cognition examined exactly this failure mode. The author built a labelled set of 243 English customer-support question pairs across eight domains. The set included 101 heavily reworded paraphrases that genuinely needed the same answer, 102 near-miss pairs that differed by a word or by word order but needed different answers, and 40 lightly reworded paraphrases reported separately. Scored by cosine similarity with the open embedding model Qwen3-Embedding-0.6B, the results were striking: the near-miss pairs scored higher (median 0.801) than the genuine paraphrases (median 0.691). The area under the ROC curve was 0.144, with a 95% bootstrap interval from 0.097 to 0.195. That means the two distributions were inverted, not merely overlapping. The model was systematically more confident about pairs that should not be cached than about pairs that should.

At a threshold of 0.85, which is the default of one commercial service, that model matched none of the 101 paraphrases and 29 of the 102 near-misses, a false cache hit rate of 28.4%. Raising the threshold did not help in the usual way. False matches reached zero only above a threshold of 0.985, and at that level 2 of the 40 near-verbatim repeats were still missed. In other words, there was no single threshold on this model that both caught the true paraphrases and rejected the near-misses. The overlap was not a matter of tuning; it was structural.

The preprint also found that similarity correlated strongly with word-set Jaccard overlap, with a Pearson r of 0.695, including within each label. That suggests the small embedding model was leaning heavily on lexical surface similarity rather than deeper semantic meaning. Interestingly, adding a query-instruction prefix halved false matches at 0.85, bringing the rate down to 12.7%, without changing the ordering of the pairs. That is a useful trick, but it does not fix the core issue.

The broader argument from the preprint is that a threshold is a property of one embedding model on one traffic distribution, not a transferable constant. A threshold that works for one model, one domain, and one set of query phrasings will not necessarily work elsewhere. Larger embedding models may behave differently, but they need their own measurement; no claim from this study applies to them. Practitioners should treat the default threshold in their caching system as a starting point, not a guarantee. The preprint describes a labelled near-miss protocol for setting a threshold on your own data, and it releases the pairs, the scores, and an analysis script under CC BY 4.0 so others can recompute every statistic.

For engineers building semantic caching, the lesson is to test against near-miss queries explicitly. A threshold that looks fine on a handful of obvious paraphrases can fail badly when customers rephrase with small, meaning-changing edits. The small open embedding model in this study is not reliable for such decisions at default thresholds. Larger models might do better, but they will not necessarily solve the problem either, because the difficulty is inherent to the task of distinguishing paraphrase from near-miss. Until embedding models become much better at that distinction, a semantic cache should err on the side of caution, or it will serve confident but wrong answers at scale.