The KV cache is the technique, and the memory region, in which a Transformer-based language model stores the attention key and value tensors of tokens it has already processed so they can be reused when generating subsequent tokens. In autoregressive generation each new token must attend to all previous tokens; by keeping their keys and values in GPU memory, only the computation for the newest token needs to be added.
The size of the KV cache grows with context length and with the number of layers and attention heads in the model, so it consumes substantial GPU memory for long contexts. Common mitigations include quantizing keys and values to lower precision and prefix caching, which reuses the cache for a shared prefix across requests. If an earlier part of the conversation changes, the cache from that point onward must be recomputed.