Mamba

A sequence model architecture that processes text with a selective state space model instead of attention.

1 articleLast mentioned

Mamba is a sequence model architecture based on selective state space models, introduced by Albert Gu and Tri Dao in 2023. Instead of attention, it processes sequences with input-dependent state space operations, so compute grows linearly with sequence length and only a fixed-size state is kept during inference.

It is studied as an alternative to the transformer's quadratic cost, and open models of several sizes have been released. A successor architecture, Mamba-2, and hybrid models that interleave Mamba layers with attention layers have also been published.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    …hidden in essay text and queried at the end, Llama-2-7B holds high accuracy to 8k tokens while RWKV-v5-7B and Mamba-2.8B degrade sharply beyond 2k to 4k tokens.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.