Semantic Caching And The Problem Of Small Edits

2026-10-28T23:33:26Z

GeraldPascal657: Utworzono nową stronę " Semantic caching is a popular way to reduce the cost and latency of large-language-model applications. Instead of calling the model for every new user question, the system stores the answers to previous queries. When a new query arrives, an embedding model converts it into a vector, and the system compares that vector against the vectors of stored queries using cosine similarity. If the similarity exceeds a threshold, the cache returns the stored answer without…"

Semantic caching is a popular way to reduce the cost and latency of large-language-model applications. Instead of calling the model for every new user question, the system stores the answers to previous queries. When a new query arrives, an embedding model converts it into a vector, and the system compares that vector against the vectors of stored queries using cosine similarity. If the similarity exceeds a threshold, the cache returns the stored answer without invoking the model again. In deployed products, this decision is governed by a single similarity threshold, typically shipped as a default between 0.80 and 0.95. The appeal is obvious: a high cache hit rate means fewer model calls, faster responses, and lower bills. The problem is that a single threshold assumes similarity of wording tracks similarity of meaning. That assumption breaks in predictable places, especially with word order, negation, and numbers. Consider a customer who asks, "How do I cancel my subscription?" and later asks, "How do I pause my subscription?" The words are close, the embedding vectors will be close, and a threshold set to catch genuine rewording will likely treat the second query as a hit on the first answer. But the correct answers differ entirely. A similar failure occurs with "Can I get a refund after 30 days?" versus "Can I get a refund before 30 days?" One word flips the policy. The same issue appears with "turn off notifications for all devices" versus "turn off notifications for this device." Small edits change the correct answer, yet the embeddings barely move. A 2026 preprint by Guerin Green of Novel Cognition examined this problem directly. The author built a labelled set of 243 English customer-support question pairs across eight domains, split into 101 heavily reworded paraphrases, 102 [https://www.accountingweb.co.uk/search?search_api_views_fulltext=near-miss%20pairs near-miss pairs] that differ by a word or by word order but need different answers, and 40 lightly reworded paraphrases. Using the open embedding model Qwen3-Embedding-0.6B, the near-miss pairs scored higher by cosine similarity (median 0.801) than the genuine paraphrases (median 0.691). The area under the ROC curve was 0.144, with a 95% bootstrap interval from 0.097 to 0.195. That means the two distributions were inverted: the pairs that should trigger a cache hit looked less similar than the pairs that should not. At a threshold of 0.85, which is the default of one commercial service, the model matched none of the 101 paraphrases and 29 of the 102 near-misses, a 28.4% false cache hit rate. False matches only reached zero above a threshold of 0.985, where 2 of the 40 near-verbatim repeats were still missed. So raising the threshold reduces false hits but also sacrifices legitimate reuse, lowering the cache hit rate and defeating the purpose of caching. The preprint also found that similarity correlated strongly with word-set Jaccard overlap (Pearson r = 0.695), even within each label. That suggests the embedding model was largely measuring lexical overlap, not deeper semantic equivalence. Adding a query-instruction prefix halved the false matches at 0.85, dropping them to 12.7%, [https://zenodo.org/records/23667256 https://zenodo.org/records/23667256] but did not change the ordering of the pairs. The core issue remained. The takeaway for engineers is that a similarity threshold is not a universal constant. It is a property of one embedding model on one traffic distribution. A threshold that works for a forum FAQ will fail for a billing support chatbot where negation and numbers matter. The preprint argues for a labelled near-miss protocol: collect pairs that look alike but need different answers, measure their similarity with your chosen embedding model, and set the threshold only after seeing where those pairs sit relative to genuine paraphrases. The pairs, scores, and analysis script are released under CC BY 4.0, so any team can recompute the statistics for their own context. Other embedding models need their own measurement; the numbers above apply only to Qwen3-Embedding-0.6B. Semantic caching is still useful, but a single threshold is a blunt instrument. The right approach is to measure the failure modes before trusting the cache.

Użytkownik:GeraldPascal657

2026-10-28T23:33:23Z

GeraldPascal657: Utworzono nową stronę "A practical guide to embedding similarity thresholds, near-miss detection, and avoiding costly semantic cache mistakes in production LLM apps. Also visit my homepage :: [https://zenodo.org/records/23667256 https://zenodo.org/records/23667256]"

A practical guide to embedding similarity thresholds, near-miss detection, and avoiding costly semantic cache mistakes in production LLM apps. Also visit my homepage :: [https://zenodo.org/records/23667256 https://zenodo.org/records/23667256]

Mazovia - Wkład użytkownika [pl]

Semantic Caching And The Problem Of Small Edits

Użytkownik:GeraldPascal657