Superposition is an interpretability term for the phenomenon in which a neural network stores more features than a layer has dimensions by packing them into directions that are not fully orthogonal to one another. When the number of features exceeds the number of dimensions, the model compresses rarer features into mutually interfering directions, which is offered as an explanation for polysemanticity, in which a single neuron responds to several features.
The concept became widely known through Toy Models of Superposition, published by Anthropic in 2022, which analyzed the phenomenon mathematically in small models. Methods such as sparse autoencoders, which decompose internal representations to separate the packed features, are being studied. In physics, the term refers to a quantum state in which multiple states coexist.