KV cache

Memory in which a Transformer model stores the attention keys and values of already processed tokens to avoid recomputing them during generation.

5 articlesLast mentioned

The KV cache is the technique, and the memory region, in which a Transformer-based language model stores the attention key and value tensors of tokens it has already processed so they can be reused when generating subsequent tokens. In autoregressive generation each new token must attend to all previous tokens; by keeping their keys and values in GPU memory, only the computation for the newest token needs to be added.

The size of the KV cache grows with context length and with the number of layers and attention heads in the model, so it consumes substantial GPU memory for long contexts. Common mitigations include quantizing keys and values to lower precision and prefix caching, which reuses the cache for a shared prefix across requests. If an earlier part of the conversation changes, the cache from that point onward must be recomputed.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Gyro-M vs. Gyro-S on a 16GB GPU: Higher Scores, Similar Coding ResultsAI Performance ·

    When the prompt has already been processed and stored in the key-value cache, prefill is far faster.

    Read article
  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    And changing temperature between tokens does not invalidate the KV cache, so it is technically feasible even though production systems rarely do it.

    Read article
  • ShapeLearn Qwen3.8-27B on a 16GB GPU: Agent Tests Pass 4 of 5 at 128k ContextAI Performance ·

    The IQ3XS build was chosen because it leaves room for a 128k context when paired with a Q8 KV cache.

    Read article
  • DeepSeek Opens DeepInfer Serving Engine Alongside MIT-Licensed V4.1-FlashAI News ·

    …ction layers that a hosted API depends on were left out of the release: the KV cache system (the memory that stores attention state so the model does not reproc…

    Read article
  • Local Qwen 3.8 27B Coding Test: Four Harnesses Fix Nearly the Same BugsAI Performance ·

    Changing the top of the conversation invalidates the server's KV cache prefix on every turn.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.