positional embedding

A component that encodes each token's position as a vector so a transformer can tell the order of tokens.

1 articleLast mentioned

A positional embedding is the component that converts each token's position into a vector and adds it to the input embedding so that a transformer can tell the order of tokens. The attention operation itself is indifferent to input order, so positional information must be supplied separately. It is also called positional encoding.

Approaches include fixed sinusoidal functions, learned vectors for each position, and RoPE, which rotates queries and keys to reflect relative position. Because a model interprets positions correctly only within the range seen during training, the positional embedding scheme determines the practical limit of the context window and how a model can be extended to longer contexts.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    A positional embedding is added, because attention on its own is permutation-invariant: without positions, the model could not tell which accessible token came…

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.