Cachemaxxing with Jev: A Semantic Cache That Knows Q3 from Q4

• A semantic cache serves a precomputed answer when an incoming prompt expresses the same meaning as an earlier one. Embedding similarity measures shared topic, not shared answer: on our pairs, every threshold served at least half its answers wrong.
• A GoodMem space supplies most of the cache: exact repeats are served in 6 to 30 ms, before the entry is embedded, and a space per tenant, with its own roles and keys, keeps one tenant’s answers from another.
• Putting Jev in front of the cache as a judge raised precision on a held-out set from 0.46 to 1.00, with 24 of 25 paraphrases still served from the cache.
• A chat model in the same role, gpt-5.4-nano, let one wrong answer through, took a median of 696 ms against Jev’s 170 ms, and returns only YES or NO, so there is no cut-off to tune.

Acme Notes is a note-taking app that does not exist. We made it up so that we could ask a support assistant the same question several ways and watch what a cache did with each one. The assistant’s system prompt is “You are the support assistant for Acme Notes, a note-taking app. Answer in two sentences,” and the model behind it is gpt-5.4-mini, which takes between 0.8 and 1.6 seconds to produce those two sentences and is paid for every time.

A semantic cache stores the prompt and the answer, embeds the prompt, and on the next request looks for the nearest stored prompt by cosine similarity. If the nearest stored prompt is similar enough, the cache returns its answer and the model is never called. This is not the prompt caching that Anthropic and OpenAI sell, which caches the model’s internal state for a prefix repeated exactly and still writes a new answer every time.¹ A semantic cache returns an old answer to a different question, so it saves the whole model call and it can be wrong in a way prompt caching cannot. The idea has a reference implementation in GPTCache, which Zilliz released in March 2023,² and it is a policy you can switch on in the Azure API Management and Apigee gateways now.

We wanted to know how often a semantic cache serves the wrong answer, so we built one on GoodMem. The code is at github.com/PAIR-Systems-Inc/goodmem-semantic-cache.

Building it on GoodMem

A GoodMem space already provides most of these pieces. It embeds what you put in it, searches it by vector, filters on metadata, and has an owner, roles, and grants. We used the Node.js SDK, @pairsystems/goodmem 0.1.9, against a server running locally, with OpenAI’s text-embedding-3-small as the space’s embedder.

Each cache entry is one memory. Its content is the prompt, because that is what GoodMem embeds and searches. The answer and the bookkeeping go in the entry’s metadata:

await client.memories.create({
  spaceId,
  originalContent: prompt,
  chunkingConfig: { none: {} },
  metadata: {
    scope, // hash of model + system prompt (+ tenant, version)
    prompt_sha256: sha256(normalize(prompt)),
    prompt,
    response,
    expires_at: Date.now() + ttlSeconds * 1000,
  },
});

Chunking is off so that a prompt is one vector rather than several passages, any one of which could match. The scope is a hash of the model, the system prompt, the tenant, and a version number, and a lookup only ever sees entries in its own scope.

A lookup runs a metadata query first, for an entry with the same normalized prompt hash, the same scope, and an expiry in the future. If the metadata query finds nothing, the lookup retrieves the single nearest memory under the same filter:

const live =
  `CAST(val('$.scope') AS text) = '${scope}' ` +
  `AND CAST(val('$.expires_at') AS bigint) > ${Date.now()}`;

for await (const event of client.memories.retrieve({
  message: prompt,
  spaceKeys: [{ spaceId, filter: live }],
  requestedSize: 1,
  fetchMemory: true,
})) {
  // event.retrievedItem.chunk.relevanceScore, event.memoryDefinition.metadata
}

Metadata values are JSON, so the string comparison needs the cast, and the score GoodMem reports is pgvector’s negative inner product, where smaller means closer. OpenAI’s embeddings have length 1, so the inner product is the cosine, and the cache negates the score before comparing it with the threshold.

One run of the demo script

The demo script sends five requests through the cache with Jev as the judge. What follows is one run, with the times the script printed.

The first request, “How do I export my notes as PDF?”, found no cached entry. The lookup took 689 ms, the model took 1,432 ms, and the answer was that you open the note, click the More options menu, and choose Export as PDF, or select several notes and use Export from the toolbar. None of those menus exist. The cache stored the answer under the support assistant’s scope.

The second request was the same sentence with three spaces in the middle and the capital letters gone. The cache answered it in 7 ms, by the metadata query, before GoodMem had finished embedding the first entry. A new memory is embedded by a background job, which on that machine takes about 0.6 seconds, and until it finishes the entry cannot be found by vector search at all. The exact-match step covers that gap, and the demo script waits on the entry’s processing status before going on, because the next two requests need the vector.

The third request, “Can I save a note to a PDF file?”, had a similarity of 0.710 to the cached question. Jev put the probability that the cached answer could be reused at 0.90, and the cache served it in 680 ms, most of which was the embedding call and the judge.

The fourth request, “How do I import notes from a PDF?”, scored 0.732, higher than the paraphrase. A cache working on similarity alone, with any threshold that admitted the third request, would have told this user how to export. Jev put the probability at 0.02. The cache called the model, which took 1,094 ms to describe a File menu with an Import submenu and a PDF item under it. On an earlier run the same model had said that Acme Notes could not import PDFs.

The fifth request was the first question again, asked of a billing assistant whose system prompt differs by one word. That is a different scope, so the lookup saw no entries, the model answered in 829 ms, and this time the menu was called More rather than More options and the answer ended by suggesting an app update or a call to support.

Choosing a similarity threshold

Before the judge existed, we had tried to choose a similarity threshold. We wrote 51 pairs: a cached question, a new question, and whether the cached answer could be returned unchanged. The file starts with password resets. Twenty-two pairs were paraphrases, and the other 29 were questions that look alike and need different answers: Saturday against Sunday, Canada against Mexico, Australia against Austria. We scored each pair in isolation, storing the cached question and looking up the new one.

The pairs that scored highest were all wrong:

Cached question	New question	Similarity
Summarize our Q3 earnings call.	Summarize our Q4 earnings call.	0.925
How do I convert a list to a set in Python?	How do I convert a set to a list in Python?	0.920
How do I sort a list in descending order in Python?	…in ascending order in Python?	0.912
Which plans include single sign-on?	Which plans do not include single sign-on?	0.888

The paraphrase “Give me a summary of the Q3 earnings call” came fifth, at 0.871, below the Q4 question it must not be confused with. At the other end, “How do I cancel my subscription?” against “Where is the option to end my plan?” scored 0.560, and “What is the capital of Australia?” against “What is the capital of Austria?” scored 0.545. No threshold admits the cancellation paraphrase at 0.560 and refuses the Austria question at 0.545. At the best threshold on this set, 0.80, the cache served 16 answers and 8 of them were wrong. Switching the space to text-embedding-3-large moved the best precision from 0.50 to 0.58 and moved Q3 against Q4 up to 0.926.

The vCache paper, from Berkeley, Stanford, and TUM, had already measured this gap at a scale we could not: it ran 45,000 prompts through a cache and found that the hits which returned a correct answer and the hits which returned a wrong one had average similarities of 0.84 and 0.85.³ AWS benchmarked a cache on ElastiCache with Bedrock and reported that at a threshold of 0.99 it answered 23.5 percent of requests with 92.1 percent of those answers correct, and at 0.75 it answered 90.3 percent of requests with 91.2 percent correct.⁴ Raising the threshold made the cache answer a quarter as often and made the answers it gave about one percentage point more accurate. Lowering it until nearly everything hits, which is cachemaxxing, makes the savings figure look good and leaves about one answer in eleven wrong. An embedding model is trained to put questions about the same topic near each other, and a cache needs to know whether two questions have the same answer, and our 51 pairs were a small demonstration that these are not the same property.

Asking a judge instead of a threshold

One fix in the literature is to learn a threshold for each cached entry rather than one for the whole cache, which is what vCache does. Another, used by Apple’s Krites and the one we chose, is to ask a separate model whether the near match can be served.⁵ The cache treats anything above a similarity floor of 0.55 as a candidate and asks that model, the judge, whether the cached answer can be returned unchanged for the new question.

The judge in the title is Jev, a decision model from TypeSafe that OpenRouter added on September 25. Unlike a standard LLM, Jev does not generate text. Rather, you hand it the facts of the case as a JSON object and ask typed questions about them, and it answers each with a probability instead of prose. The cache asks it one question of the type TypeSafe calls a noul, a yes-or-no question answered with the probability of yes:

const response = await fetch("https://openrouter.ai/api/alpha/decisions", {
  method: "POST",
  headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "typesafe/jev-1.13",
    state: { cached_question: cachedPrompt, new_question: newPrompt },
    questions: {
      reusable: {
        type: "noul",
        instructions:
          "Can the answer already written for the cached question be returned, " +
          "unchanged, as a correct answer to the new question?",
        criteria: {
          true: "Both questions ask for the same information, even if they are worded differently.",
          false:
            "The questions differ in a way that changes the answer, such as a different " +
            "date, number, name, place, direction, or a negation.",
        },
      },
    },
  }),
});
const { answers } = await response.json();
return answers.reusable.noul; // probability of yes, 0 to 1

The cache accepts a candidate when the probability is at least 0.5. The model is pinned to typesafe/jev-1.13 rather than the latest alias so that a cut-off tuned against one version is not applied unknowingly to the next.

For comparison we also wrote a judge on gpt-5.4-nano, which gets the same question and criteria in a system prompt and is told to reply with only YES or NO. A chat model answers in text, so the best it can give the cache is 1 or 0.

Because we had written the 51 pairs and the judge instructions, the instructions could have been fitted to the pairs, which is benchmaxxing with a smaller benchmark. So we had a separate model session write a second set of 60 pairs, 25 of them paraphrases, without access to either. The pairs that session produced were harder than ours. The closest look-alike is “How do I convert Celsius to Fahrenheit?” against “How do I convert Fahrenheit to Celsius?”, at 0.948, and the set includes “How do I sort a Python list in place?” against “How do I get a sorted copy of a Python list without changing the original?”, which are list.sort() and sorted(). It also includes “What’s a good recipe for banana bread?” against “How do I set up two-factor authentication on my account?”, at 0.071, which we assume was put there to make sure something was easy.

Our first prompt for the chat judge already listed the differences that change an answer, but said nothing about when two questions count as the same. On our own pairs it reused 13 of 22 paraphrases, rejecting “Who is the CEO of the company?” against “Who runs the company?”, and on the held-out set below it reused 15 of 25. We then gave it the instructions Jev already had, which add that two questions match when they ask for the same information in different words. Jev’s instructions were written before the held-out set existed and did not change, so its held-out numbers are clean. The chat judge’s instructions were revised after we had seen its held-out result, so its numbers below are one round less clean than the word held-out suggests.

On the 60 held-out pairs:

Configuration	Wrong answers served	Paraphrases reused	Judge time, median
Similarity only, threshold 0.50	29 of 54	25 of 25	–
Similarity ≥ 0.55, gpt-5.4-nano judge	1 of 25	24 of 25	696 ms
Similarity ≥ 0.55, Jev judge	0 of 24	24 of 25	170 ms

The one wrong answer the chat judge let through was the list.sort() pair. Jev gave it 0.04. Jev’s probabilities on the look-alikes were low across the board: 0.02 for Q3 against Q4, 0.02 for the SSO negation, 0.03 for Celsius against Fahrenheit, 0.01 for banana bread against two-factor authentication. The paraphrases it was least sure of were “Who runs the company?” at 0.76 and “how can i change my password” against “How do I reset my password?” at 0.72, and we had labeled the second of those a paraphrase without thinking hard about whether resetting and changing are the same operation. Both judges disagreed with one of our labels, on whether “Which plans include single sign-on?” answers “Is SSO available on every plan?”. The chat model said no and Jev gave it 0.20. On reflection they may be right, since the second question has a yes-or-no answer and the first does not.

The evaluation sweeps the cut-off from 0.1 to 0.9 for both judges. For the chat judge every row is the same, 25 served and 1 wrong, because a judge that only ever says 1 or 0 has nothing to sweep. For Jev the rows change with the cut-off: 24 served with none wrong at every cut-off up to 0.75, and 22 at 0.9. On our own 51 pairs the 0.9 cut-off reused only 12 of the 22 paraphrases, so a strict setting costs more on casual wording than the held-out set alone suggests. The prompt was identical for every row. Only the cut-off moved. On two runs of the held-out set the count at 0.9 was 21 and then 22, so the probabilities move a little between calls. Each Jev call used 382 input tokens and cost $0.000016.

Latency on each path

These numbers are from one machine, with GoodMem and PostgreSQL local and the OpenAI and OpenRouter calls going over the internet.

Path	Time
Exact hit	6 to 30 ms
Lookup that reaches the vector search, median over 51 lookups	243 ms
Verified semantic hit, including the judge, three runs	514, 599, and 680 ms
Model call on a miss, gpt-5.4-mini	0.8 to 1.6 s

Most of the lookup is the embedding call to OpenAI, which every semantic lookup makes, so a miss costs about a quarter of a second more than calling the model directly and a verified hit about half a second. Whether that trade pays depends on how often questions repeat, and the published figures for that vary by a factor of seven. MeanCache, which studied 27,000 queries from 20 ChatGPT users in an academic setting, found that about 31 percent of a user’s queries were similar to that user’s earlier ones.⁶ SCALM, working from public chat logs, found that only 4.5 percent of queries in the MOSS dataset and 7.5 percent in LMSYS could have been answered from earlier responses.⁷ A support desk for one product sits near MeanCache’s 31 percent. Open-ended chat sits near SCALM’s 4.5 and 7.5, where a cache mostly adds a quarter of a second to every request.

Isolation, expiry, and stale answers

The part of a semantic cache that GoodMem handled without any work from us is the part the security papers are about. A cache hit returns in milliseconds and a miss in seconds, and researchers measuring that difference against GPTCache’s defaults could tell whether someone else had recently asked a question with 81.4 percent accuracy on one try and 95.4 percent after five.⁸ A paper at NDSS 2026 went the other way and planted answers in shared caches on AWS, Azure, and Alibaba Cloud that later users then received, with success rates of 82, 89, and 87 percent.⁹ A GoodMem space has an owner, roles, and grants enforced in the database query, and an API key can be issued with a ceiling that covers one space, so a cache per tenant is a space per tenant and one tenant’s key cannot read or write another’s entries. That does not remove the timing signal within a tenant, but it limits who can observe it and who can plant an answer. Each entry can also be listed, filtered by metadata, and deleted like any other memory, and the lookups show up in GoodMem’s OpenTelemetry traces.

What GoodMem did not handle is expiry. Entries carry an expires_at field that lookups filter on, but nothing removes them, so the cache has a purgeExpired() method that issues a batch delete with the same filter, and something has to call it on a schedule. Writes are not searchable by vector until the background job has embedded them, which the exact-match query covers for identical repeats and nothing covers for paraphrases. And the cache stores whatever the model said. The two answers gpt-5.4-mini gave about importing PDFs into Acme Notes, that it cannot be done and that it is under File, Import, PDF, were given on different runs. Whichever one came first would have been served to every user in that scope for the next 24 hours.

On the held-out set the cache with Jev reused 24 of the 25 paraphrases and served none of the 35 look-alikes. The paraphrase it missed was “How do I submit an expense report for reimbursement?” against “What’s the process for getting paid back for business expenses?”, which scored 0.52, below the 0.55 floor, so no judge ever saw it.

Footnotes

1. Anthropic, Prompt caching, which says cache hits “require 100% identical prompt segments” and that caching “has no effect on output token generation”; OpenAI, Prompt caching. ↩

2. Bang Fu and Di Feng, “GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings,” NLP-OSS 2023. ↩

3. Schroeder et al., “vCache: Verified Semantic Prompt Caching,” ICLR 2026. ↩

4. AWS Database Blog, “Lower cost and latency for AI using Amazon ElastiCache as a semantic cache with Amazon Bedrock,” November 2025. The test set was LMArena queries plus paraphrased variants. ↩

5. Singh et al., “Asynchronous Verified Semantic Caching for Tiered LLM Architectures,” 2026 preprint. ↩

6. Gill et al., “MeanCache: User-Centric Semantic Caching for LLM Web Services,” IPDPS 2025. ↩

7. Li et al., “SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models,” 2024. ↩

8. Song et al., “The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems,” IEEE Transactions on Information Forensics and Security. ↩

9. Wu et al., “When Cache Poisoning Meets LLM Systems: Semantic Cache Poisoning and Its Countermeasures,” NDSS 2026. ↩