semantic cache

A cache that reuses previous responses based on the semantic similarity of queries, used to reduce cost and latency in LLM applications.

1 articleLast mentioned

A semantic cache is a cache that reuses previous responses based on the semantic similarity of queries. Unlike a conventional cache, which returns a stored result only when the input string matches exactly, a semantic cache converts each query into an embedding vector and returns the stored response if a past query exceeds a similarity threshold.

In LLM applications it is used to reduce model calls, cost, and latency when similar questions recur with different wording. Open-source implementations such as GPTCache and implementations built on vector databases exist. A loose similarity threshold can return a previous answer to a question with a different meaning, so semantic caches are often combined with an exact-match cache or a validation step.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Intelligence Plane: Why Memory and Ontology, Not the Model, Define AI AgentsAI in Practice ·

    …, which lives within a single session (the working context window plus a semantic cache for repeated queries), and long-term memory, which persists across sessi…

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.