distributed representation

A way of encoding information in neural networks in which each concept is represented by a pattern of activity across many units.

1 articleLast mentioned

A distributed representation is a way of encoding information in a neural network in which each concept is represented by a pattern of activity across many units, and each unit takes part in representing many concepts. It contrasts with a localist representation, which dedicates a single unit to each concept.

The approach was championed in the 1980s by the parallel distributed processing (PDP) research group, including Geoffrey Hinton. Its advantages include encoding many more concepts than there are units, degrading gracefully when some units are damaged, and supporting generalization because similar concepts share similar patterns. Word embeddings, which represent words as real-valued vectors, are a typical example, and the internal representations of modern deep learning models are mostly distributed. On the other hand, because the meaning of a single unit is hard to read, distributed representations are also cited as a source of difficulty for interpretability.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Neural Networks Are Already Clean: Study Claims 90% of Neuron Poles Are LegibleAI Security ·

    …er than declaring a winner in the old debate, the essay argues that localist and distributed representations describe the same architecture at different levels.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.