RWKV

An open-source language model architecture and model family combining transformer-style parallel training with RNN-style efficient inference.

1 articleLast mentioned

RWKV is an open-source language model architecture, and the family of models built on it, that trains in parallel like a transformer while running inference efficiently like a recurrent neural network. It was developed under the leadership of Bo Peng and is maintained as a project under the Linux Foundation.

Instead of attention it uses a linear-complexity recurrent operation, keeping only a fixed-size state during inference so that memory does not grow with input length. Models of several generations and sizes have been released, and it is classified among linear attention architectures that compress past information into a fixed-size state.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    …hidden in essay text and queried at the end, Llama-2-7B holds high accuracy to 8k tokens while RWKV-v5-7B and Mamba-2.8B degrade sharply beyond 2k to 4k tokens.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.