ConceptsINTERNALS.md

Attention

The operation inside every transformer. Each token looks at the others and takes a weighted mix of what it finds.

  • A word on its own is ambiguous, like “bank”.
  • Attention gives it a score against every other word, and turns the scores into weights that add to 100%.
  • Its new meaning is the weighted average of all the words.
  • Switch the sentence and the same word lands somewhere else.

Try this: switch the sentence from River to Loan and watch the orange dot move.

Try it

This page needs JavaScript.

What just happened

  1. Each word has a vector, two numbers here: how much it says about money and how much about nature. “bank” is 0.5 and 0.5, so it could mean either.
  2. “bank” asks a question, its query. Its score for each word is a dot product: the query times that word’s key. Words that point the same way as the query score high.
  3. Softmax turns the scores into weights. In the river sentence “bank” puts on “” and on itself. An equal share would be .
  4. The output is the weighted average of the value vectors. “bank” moves to money and nature, so it leans . In the loan sentence it leans . Plain averaging would put it at only nature in the river sentence, so the weights matter.
The formula and the divisor

For all the queries at once, Vaswani and others write attention as softmax(QKᵀ / √dk) V. Q, K and V hold the queries, keys and values, and dk is the length of a key vector. Dividing by √dk is the slider here. The paper’s reason: for large dk the dot products grow large, which pushes softmax into regions where its gradients are very small. Drag the divisor up and the weights flatten toward an equal share. Drag it down and one word takes almost everything.

Here each word’s key and value are the same vector, so one picture works. In a real model they are two different learned projections of the word, and the vectors have hundreds of numbers, not two.

Where you meet it

A transformer is a stack of layers. Each layer has several attention heads running side by side, each with its own learned queries, keys and values. When a model reads your prompt, every token attends to the tokens before it. When it writes, each new token attends to everything already there.

Where it breaks

Every token scores against every other token, so the table of scores is n by n. Double the context and the work is four times as big. Drag the length up.

The weights also share a fixed 100%, so with thousands of tokens each one gets a thin slice. This page shows the cost, not how well a model uses a long context.

Check yourself

You set the divisor very high. What does “bank” attend to?

Every word equally. The scores shrink toward zero, so the softmax weights all approach one share each, and the output approaches the plain average. Attention has stopped choosing.

Why can the same word end up with two different vectors?

The weights depend on the other words in the sentence. “river” pulls “bank” toward nature, “loan” toward money. The word and its own vector are the same in both sentences. Only the mix changes.

A model reads 8,000 tokens and then 16,000. How much more attention work is that, per head?

Four times as many scores: 16,000² is 256 million and 8,000² is 64 million.

Used in

Go deeper