Transformer
The transformer is a neural network architecture presented in 2017 that forms the basis of modern language models. The core idea is the attention mechanism, with which the model weights, for every word, how important all the other words in the context are. The transformer thus captures far-reaching connections in text too and processes sequences highly in parallel.
Also known as: transformer architecture, transformer model, attention model
What makes the transformer architecture special?
Before the transformer, recurrent networks dominated, processing text word by word in sequence. That was slow and made references across long distances hard to recognise. The transformer solves that by viewing all the words in a sequence at once and modelling their relationships directly.
This parallel processing makes training on modern graphics processors extremely efficient. That scalability is a main reason why models with billions of parameters can be trained at all and why today's generation of capable LLMs became possible.
How does the attention mechanism work?
Attention, often called self-attention, is the heart of the transformer. For each Token in the text, the model computes how strongly it relates to every other token. That is how it recognises, for instance, that a pronoun refers to a particular noun earlier in the sentence.
Technically this happens through three representations per token: query, key and value. Matching query against key produces weights determining how strongly other tokens' value parts feed into the new representation. Several such attention calculations run in parallel as multi-head attention and capture different kinds of relationship at once.
What building blocks make up a transformer?
A transformer is structured into several stacked layers. Each layer combines an attention block with a subsequent neural network. Techniques such as residual connections and normalisation also ensure that even very deep models can be trained stably.
Because attention itself knows no order, a so-called positional encoding adds the information about where a token sits. Depending on the purpose, pure encoders, pure decoders or a combination of both are used. Most of today's chat LLMs are based on the decoder variant.
Why is the transformer so important for AI?
Almost all leading language models, from GPT about Claude until Gemini, build on the transformer architecture. It has proved exceptionally scalable and versatile. With more data and more parameters, performance rises reliably, which is what made the rapid development of recent years possible.
The transformer is now used far beyond text, for images, audio and program code among others. That makes it one of the most influential inventions of modern AI and the shared technical basis of generative AI.
Using transformers in practice
For most applications companies do not have to train transformers themselves. They access existing models through interfaces instead and adapt them to their requirements with well-crafted prompts, additional data and a well-considered architecture.
Elisabit helps companies use transformer-based models in practice, whether for smart website features, marketing automation or tailor-made AI solutions. The focus is not on the technology alone, but on the concrete value for your business.
Frequently asked questions
When was the transformer invented?
The transformer architecture was presented in 2017 in the research paper Attention Is All You Need. It replaced earlier approaches such as recurrent networks in many areas and is today regarded as the technical basis of nearly all large language models.
What does attention mean in a transformer?
Attention describes how a model determines, for every word, which other words in the text are especially relevant. The transformer thereby captures relationships across large distances and understands context far better than earlier architectures.
Is every LLM a transformer?
Practically all widely used large language models today are based on the transformer architecture. There is research into alternative approaches, but thanks to its scalability and performance the transformer remains the clear standard.
Is the transformer only used for text?
No. Originally developed for language, the transformer is now used for images, audio, video and program code too. That versatility makes it a central architecture across generative AI.
Related terms
An LLM is an AI language model that understands and produces text by predicting the most likely next word.
Deep learning uses deep neural networks to recognise complex patterns in large volumes of data automatically.
A neural network is a computational model of connected neurons, modelled on the brain, that learns from data.
GPT is OpenAI's family of generative, transformer-based language models that understand and produce text.
Generative AI independently creates new content — text, images, audio or code — based on patterns it has learned.
Put AI to work for your business?
We help you integrate artificial intelligence into your processes, your marketing and your website — strategically and securely.
Your contact
Stefan
I look forward to hearing about your project and finding the best solution together.
