What a Transformer Layer Does

So far, you know how text becomes token IDs, how token IDs become embedding vectors, how projections transform vectors, how logits score the vocabulary, and how softmax produces probabilities. The missing piece is the model itself — the part that transforms initial embeddings into the rich hidden state that the output head reads.

That model is built from layers. A "layer" (also called a "block") is a fixed sequence of operations that takes in a set of hidden states — one vector per token (the [n_tokens, d_model] matrix from M2) — and produces updated hidden states of the same shape. The model repeats this layer many times: a 32-layer model applies the same pattern 32 times, each with its own learned weights. The output of layer 1 becomes the input to layer 2, and so on.

Inside each layer, two things happen:

• Attention Tokens look at each other. Information moves between positions. This is token mixing.
• FFN Each token is transformed independently. No cross-token communication. This is per-token transformation.

That is the entire structure. Every layer does attention then FFN, with normalization and residual connections around each. A model with 32 layers repeats this pattern 32 times, each with its own learned weights.

Attention and FFN serve complementary roles, and neither can replace the other:

• Attention alone is not enough. Attention routes information between token positions. While the softmax makes the routing data-dependent and nonlinear, the final weighted sum of V vectors is linear in V — it blends existing representations without creating new per-token features. The FFN adds per-token nonlinear capacity (via activation functions like SiLU) that creates genuinely new features from the blended input.
• FFN alone is not enough. The FFN processes each token independently — it has no way to compare one token to another. Without attention, the model would treat each word in isolation, as if it had no context. Attention provides the cross-token communication.

Together, they form a complete unit: attention decides what information to gather from the sequence, and the FFN decides what to do with it. This alternating pattern — mix then transform, mix then transform — is what builds increasingly rich representations over 32+ layers.

In the early layers, the model builds basic features: which words are nearby, simple syntactic patterns, local context. In the middle layers, it assembles these into more abstract features: sentence-level meaning, entity tracking, argument structure. In the deep layers, it refines the final prediction: what token is most likely next, given everything it has gathered.

Each layer reads the full hidden-state matrix produced by the previous layer and writes a new full hidden-state matrix. The shape never changes — it is always [n_tokens, d_model] in and [n_tokens, d_model] out. What changes is the content of those vectors: the information they encode becomes richer and more prediction-relevant with each successive layer.

The remaining pages in this module will open the attention and FFN boxes. By the end of M4, you will know exactly what happens inside both operations, including the math, shapes, and code.

No new formulas. The key concept is the two-phase structure: