Semantic Caching And The Trouble With A Single Similarity Threshold

Semantic caching is a practical way to cut the cost and latency of large-language-model applications. The idea is simple: when a user asks a question, zenodo.org/records/23667256 you compute an embedding of that query, compare it against embeddings of previously answered queries, and if the cosine similarity is high enough, you return the stored answer instead of calling the model again. In deployed products, that decision is governed by a single similarity threshold, typically shipped as a default between 0.80 and 0.95. The threshold sounds like a neutral dial, but it is not. It is a fragile assumption about what an embedding model considers similar, and that assumption breaks in a specific, measurable way when you add a query-instruction prefix.

To see why, consider a customer who writes "I want to cancel my subscription" and another who writes "I want to pause my subscription." A human sees two different intents. An embedding model, however, sees two sentences that share almost every word. Their cosine similarity will be high, perhaps above 0.85, so a semantic cache would return the cancellation answer for the pause request. That is a false cache hit. The problem is not that the embedding model is bad; it is that the similarity threshold is a single number applied to a single vector, and that vector encodes surface form, not meaning.

A 2026 preprint by Guerin Green of Novel Cognition investigated this exact failure mode. The author built a labelled set of 243 English customer-support question pairs across eight domains: 101 heavily reworded paraphrases, 102 near-miss pairs that differ by a word or by word order but need different answers, and 40 lightly reworded paraphrases reported separately. Using the open embedding model Qwen3-Embedding-0.6B, the results were striking. The near-miss pairs scored higher by cosine similarity, with a median of 0.801, than the genuine paraphrases, which had a median of 0.691. The area under the ROC curve was 0.144, meaning the distributions were inverted. In other words, the embedding model was systematically more confident about pairs that should not share an answer than about pairs that should.

At a similarity threshold of 0.85, which is the default of one commercial service, that model matched none of the 101 paraphrases and 29 of the 102 near-misses, a 28.4 percent false hit rate. To get false matches down to zero, you would need to raise the threshold to 0.985, and even then, 2 of the 40 near-verbatim repeats were still missed. So the threshold is not a dial you can tune once; it is a knife’s edge between missing valid cache hits and returning wrong answers.

The deeper issue is what the embedding model compares. When you add a query-instruction prefix, such as "Answer the following customer question:" before the actual query, you change the embedding of every query in the same way. The preprint found that adding such a prefix halved the false matches at 0.85, down to 12.7 percent, without changing the ordering of the pairs. That is counterintuitive but logical: the prefix shifts all embeddings into a region of the vector space where the model’s sensitivity to word order or minor word changes is slightly different. But it does not fix the core inversion. The similarity threshold is a property of one embedding model on one traffic distribution, not a transferable constant.

The preprint also reported that similarity correlated strongly with word-set Jaccard overlap, at a Pearson r of 0.695, even within each label. That means the embedding model is largely measuring lexical overlap, not semantic intent. So the threshold you set for one set of queries, or one domain, will not hold for another. The author argues for a labelled near-miss protocol to set the threshold per model and per traffic mix, and has released the pairs, scores, and analysis script under a permissive license. Other embedding models need their own measurement; the numbers here apply only to Qwen3-Embedding-0.6B.

For practitioners, the lesson is practical. A single similarity threshold is a convenient default, but it is not a safety guarantee. Before you rely on a semantic cache, you need to know how your embedding model treats near-miss queries that differ by one word or by word order. Otherwise, you are not caching answers; you are caching mistakes.