Multi-Head Latent Attention (MLA) is an attention variant that stores keys and values compressed into low-dimensional latent vectors, introduced by DeepSeek in DeepSeek-V2 in 2024. During inference only the compressed vectors are kept in the KV cache and are expanded back into keys and values when needed, which greatly reduces KV cache memory compared with standard multi-head attention.
It has been used in DeepSeek-V2, DeepSeek-V3, and DeepSeek-R1, and is being adopted by other models as a way to improve memory efficiency when handling long contexts.