A semantic cache is a cache that reuses previous responses based on the semantic similarity of queries. Unlike a conventional cache, which returns a stored result only when the input string matches exactly, a semantic cache converts each query into an embedding vector and returns the stored response if a past query exceeds a similarity threshold.
In LLM applications it is used to reduce model calls, cost, and latency when similar questions recur with different wording. Open-source implementations such as GPTCache and implementations built on vector databases exist. A loose similarity threshold can return a previous answer to a question with a different meaning, so semantic caches are often combined with an exact-match cache or a validation step.