RWKV is an open-source language model architecture, and the family of models built on it, that trains in parallel like a transformer while running inference efficiently like a recurrent neural network. It was developed under the leadership of Bo Peng and is maintained as a project under the Linux Foundation.
Instead of attention it uses a linear-complexity recurrent operation, keeping only a fixed-size state during inference so that memory does not grow with input length. Models of several generations and sizes have been released, and it is classified among linear attention architectures that compress past information into a fixed-size state.