Softmax Turns Scores into a Distribution

The previous lesson left us with logits — raw, unbounded scores, one per vocabulary entry. Logits are the model's internal ranking of how likely each next token is. For greedy decoding (always pick the highest) or top-k filtering (pick from the k highest), logits alone are sufficient — only the ranking matters. But for probabilistic sampling, top-p (nucleus) filtering, and calibrated confidence, we need actual probabilities. Logits are not probabilities:

1. They can be negative. You cannot interpret a logit of -2.3 as a probability.
2. They do not sum to 1. You cannot sample from them or compute cumulative thresholds (needed for top-p). (Logit differences do encode relative odds — exp(z_i - z_j) = p_i/p_j — but absolute probabilities require normalization.)

Softmax fixes both problems in one operation. It converts a vector of arbitrary real numbers into a vector of positive numbers that sum to exactly 1 — a proper probability distribution. Crucially, softmax preserves the ranking: the highest logit becomes the highest probability.

Softmax has three stages. Understanding each stage separately makes the whole formula easy to follow.

The amplification effect means softmax behavior depends strongly on the spread of the logits:

• Large spread If the top logit is much larger than the second-highest (e.g., top = 10.0, second = 2.0, rest near 0), the top token dominates the probability mass. How much depends on the gap and the number of competitors: with 3 classes and a gap of 10, the top token gets >99%. With 128,000 classes all at 0, a top logit of 10 only gets ~15%. The key factor is the gap between the top logits, not the gap between the top and the average.
• Small spread If all logits are close together (e.g., 1.0, 1.1, 0.9), the probabilities are nearly uniform. The model is "uncertain" — no token stands out strongly.
• Identical logits If all logits are the same value (e.g., all zeros), every probability equals 1/|V|. Maximum uncertainty. Note that the absolute value does not matter — softmax([0, 0, 0]) and softmax([100, 100, 100]) produce the same result, because only relative differences matter.

Temperature is a scalar T that divides the logits before softmax:

Dividing by T changes the spread of the logits before the exponential sees them:

• T < 1 (e.g., 0.5): Logit differences are amplified. z/0.5 = 2z, so the spread doubles. The distribution becomes sharper — the model acts more "confident." At T → 0, softmax approaches argmax.
• T = 1: No change. Standard softmax.
• T > 1 (e.g., 2.0): Logit differences are compressed. z/2 halves the spread. The distribution becomes flatter — more tokens have a reasonable chance of being picked. At T → ∞, the distribution approaches uniform (1/|V| for each token).

Temperature does not change the model's logits. It changes how the decoding policy interprets them. This is why temperature is a runtime setting, not a model property — you can adjust it between requests.

Adjust the logits and temperature below. Watch how the probability bars change. Try: setting one logit much higher than the rest, setting all logits equal, and adjusting temperature to see sharpening and flattening.

There is one practical problem with computing softmax directly. If a logit is very large (say, 1000), then e¹⁰⁰⁰ overflows — it is too large for floating-point numbers to represent. The result is Inf, and the division produces NaN.

The standard fix is to subtract the maximum logit from all logits before exponentiating:

Why does this work? Because subtracting a constant from all logits does not change the softmax output. The proof is one line: e^{(z_i - m)} / ∑ e^{(z_j - m)} = (e^z_i · e^-m) / (∑ e^z_j · e^-m) = e^z_i / ∑ e^z_j. The e^-m cancels from numerator and denominator. The result is identical, but now the largest exponent is e⁰ = 1, which never overflows.

Every real softmax implementation uses this trick. If you see max before exp in model code, this is what it is doing.

You have seen softmax here at the output head, turning logits into next-token probabilities. In Module 4, you will see it again inside the model's layers: when tokens compare their queries against keys to decide "how much should I attend to each previous token?", softmax converts those raw comparison scores into weights that sum to 1. The same formula, the same properties (positive, sums to 1), the same numerical stability trick — just applied at a different point in the computation. When you reach that lesson, the math will already be familiar.

The full softmax formula, including temperature and the stability trick:

The inspector above uses hand-tuned logits. The widget below runs the same temperature slider on real DistilGPT-2 output. Watch how the probability distribution sharpens at low temperature and flattens at high temperature — on actual model predictions, not synthetic numbers.