softmax

A function that turns a vector of real numbers into a probability distribution summing to one, used for next-token probabilities and attention weights.

1 articleLast mentioned

Softmax is a function that converts a vector of real numbers into a probability distribution that sums to one. It applies the exponential function to each value and divides by the total, so larger values receive larger probabilities.

In language models it turns the final layer's logits into next-token probabilities over the whole vocabulary, and in attention it normalizes query-key scores into weights. Temperature is a value by which logits are divided before softmax, controlling how peaked the distribution is.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    Temperature is a divisor applied to the raw logits before the softmax renormalizes them to sum to one.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.