Attention mechanism

This mechanism was a decisive breakthrough, because it lets models capture relationships across large distances in a text. A pronoun can thus be related correctly to a noun far away. Attention is therefore the foundation of modern language models' contextual understanding.

Vividly, attention can be described as a dynamic focus: for every word the model processes, it directs its attention to all the other words of the input to differing degrees. That focus is not fixed but recalculated for every position, so the model flexibly makes the relevant connections depending on task and context.

In self-attention, every token is translated into three vectors: query, key and value. Put simply, the query represents the question "what should I pay attention to?", the key describes "what do I offer?" and the value holds the actual information passed on.

The model compares every query with all the keys and derives weights stating how strongly the individual tokens relate to each other. Those weights are then applied to the value vectors. The result is a context-enriched representation for every token, taking the surrounding language into account.

To turn the raw comparison values into meaningful weights, a normalisation is applied that turns them into a distribution whose parts add up to a whole. A weighted mixture of all the other tokens' information thus arises for every token. That step is exactly what gives self-attention its ability to bring relevant content forward and let unimportant content recede.

Through this multi-head attention the model views the text from several perspectives at once. The individual heads' results are then combined. This parallel approach raises the model's expressive power considerably and contributes substantially to the quality of modern language models.

The advantage of this variety is that meaning in language can rarely be reduced to a single level of relationship. While one head may capture the syntactic link between subject and verb, another can follow thematic connections across several sentences. Combining these specialised viewpoints allows a differentiated understanding of language unreachable with a single pattern of attention.

One peculiarity of attention is that it first treats all tokens as equal, regardless of their order. For language, though, a word's position is of central importance, since the same words in a different sequence make a different statement. So the model does not lose the order, positional information is introduced into the input separately.

This positional encoding gives every token an indication of where in the sequence it stands. Only through combining the representation of content with positional information can attention capture both what is at issue and in what arrangement the elements stand. Word order, sentence structure and logical sequence can thus be taken into account reliably.

This interplay explains why transformers can process language so flexibly. They combine attention's free, distance-independent weighting with an explicit ordering of the tokens. The result is a model able to make both near and distant connections without losing sight of the thread of sequence.

The attention mechanism is inseparable from the transformer architecture, which underpins practically every large language model today. Without attention, the level reached today in understanding and generating text would be hard to imagine.

One important practical aspect concerns the computing effort: since attention takes the relationships between all tokens into account, the effort grows steeply with sequence length. That affects how long a context a model can process efficiently and is an active field for technical improvements meant to make attention less resource-hungry.