Multi-Head Latent Attention

An attention variant that compresses keys and values into low-dimensional latent vectors to shrink the KV cache.

1 articleLast mentioned

Multi-Head Latent Attention (MLA) is an attention variant that stores keys and values compressed into low-dimensional latent vectors, introduced by DeepSeek in DeepSeek-V2 in 2024. During inference only the compressed vectors are kept in the KV cache and are expanded back into keys and values when needed, which greatly reduces KV cache memory compared with standard multi-head attention.

It has been used in DeepSeek-V2, DeepSeek-V3, and DeepSeek-R1, and is being adopted by other models as a way to improve memory efficiency when handling long contexts.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    Kimi K3, an open frontier model, combines Multi-Head Latent Attention with KDA linear layers, stable layer normalization and feed-forward blocks, using gating t…

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.