Linear Projections Change Representation Size

A linear projection is a matrix multiply with a learned weight matrix. It can change the width of your vectors, or keep it the same. If your hidden state has dimension 2048 and you need a query vector of dimension 256, you multiply by a learned weight matrix that maps [2048] to [256].

The weight matrix is learned during training. It decides which combinations of input features matter for the output. You do not need to understand the learning — just that the projection is a matrix multiply with a specific shape contract.

This is the bridge from linear algebra to transformer semantics. The same hidden state can be projected into different spaces with different learned matrices. That is why one input vector can become a query, a key, a value, or an FFN-expanded representation.

Another way to say this: a projection is not merely resizing a vector. It is re-expressing that vector in a new learned coordinate system. The output coordinates ask different questions of the input than the original coordinates did.

Each column of the weight matrix defines one output feature. When you compute output[j] = hidden · W[:, j], you are asking: "how much does this hidden state align with column j?"

If column 0 of W_Q has large values in the positions corresponding to "syntax-related" features and small values elsewhere, then output[0] measures "how syntactic is this token?" Column 1 might measure something entirely different. The model learns these columns during training — each one becomes a learned "question" that extracts a specific kind of information from the input.

This is why the same hidden state can be projected into Q, K, and V using different weight matrices: each matrix asks different questions of the same input. W_Q asks "what are you looking for?", W_K asks "what do you contain?", and W_V asks "what should you pass along?" — but they all start from the same hidden vector.

Projections can change the dimension of a vector. This is a core mechanism used throughout the model:

• Contract (d_model → d_head): Q, K, V projections reduce dimension from 4096 to 128 per head. This is a lossy compression — the model learns which 128 features matter for this head's role.
• Expand (d_model → d_ff): The FFN up-projection widens from 4096 to 16384. This gives the activation function a wider space to work in — more features to selectively activate or suppress.
• Same dimension (d_model → d_model): The output projection W_O maps attention output back to model width. Same size in and out, but the content is transformed.

The shape contract is always: [n, d_in] × [d_in, d_out] = [n, d_out]. The inner dimension d_in must match. The output width d_out is determined by the weight matrix shape — it is a design choice.

Focus on a single column of the weight matrix. That column defines one output coordinate. In the toy example, the first output coordinate of hidden × W_q is the dot product between hidden and the first column of W_q.

So each output coordinate is a learned weighted sum of the input coordinates. If the input vector has features the model cares about for that output role, the corresponding weights amplify or suppress those features. The full projection simply computes many such learned coordinates at once.

This is why columns of the projection matrix are such a useful mental object. Each column is one learned feature extractor for the new representation space.

Suppose the same hidden state goes through W_q, W_k, and W_v. The input row is identical in all three cases. What changes is the learned matrix.

That means each projection asks a different set of questions of the same input. One learned space may be good for "what information am I looking for?" Another may be good for "what information do I contain that others might want?" Another may be good for the actual content to be transferred forward.

The exact semantics are learned, not hand-programmed. But the structural idea is simple: change the projection matrix, and you change the operational role of the resulting vector.

Read the columns of W as output features. Each column tells you how to combine the input coordinates to produce one output coordinate.

For a whole token sequence, the same projection matrix is applied to every token row. That shared learned map is exactly what gives transformer layers their strong regular structure.

Some models also add a bias vector after the multiply: Y = XW + b, where b has shape [d_out] and is added to every row. Other models (including LLaMA) skip the bias entirely. When reading real code, check whether a bias is present; the core shape logic stays the same either way.

The toy example used one hidden vector because that is the smallest readable case. But real inference nearly always applies projections to a matrix of hidden states, not a single row.

If X has shape [n_tokens, d_model], then the projection produces Y = XW with shape [n_tokens, d_out]. Each token row is remapped into the new space, and all rows share the same learned matrix. This "one matrix, many token rows" pattern is one of the most important repeated structures in the entire model.

Now combine the running example from the embeddings page with the projection idea from this page. Start from the hidden matrix:

Using the toy matrix W_q from above, the whole sequence projects to:

The exact numbers are not the main point. The shape and interpretation are. Three token rows came in, and three token rows came out. What changed is the feature space: each row is now a query vector instead of a generic hidden-state vector.