Token
A token is the smallest processing unit of a large language model (LLM). In so-called tokenisation, text is broken into tokens which, depending on the model, can consist of whole words, parts of words or single characters. Tokens are the basis of how a model processes language and significantly determine cost, processing speed and maximum context length.
Also known as: token, tokens, language token, processing unit
What is a token in AI?
A token is the smallest unit a large language model (large language model, LLM) works with. Before a model like GPT or Claude can understand or produce a text, it has to be turned into machine-readable form. This is where tokens come in: the input text is split into a sequence of smaller building blocks the model then processes numerically.
Note that a token is not necessarily a whole word. Depending on the language and Tokenizer a token can be a whole word, part of a word (a subword) or even a single character. Common words are often encoded as one token, while rare or long words are split into several. This principle lets models represent an almost unlimited variety of texts with a manageable vocabulary.
How does tokenization work?
The process in which text is broken into tokens is called tokenisation. A tokeniser applies a previously learned set of rules to split the character string into the units suited to the model in question. Each token is then assigned a unique numeric ID, since a language model computes solely with numbers, not with letters.
Modern language models mostly use subword methods. The advantage: unknown terms, technical words or proper names can be assembled from smaller, known parts. The model thus stays flexible and can process even words that never appeared whole in training.
How the split works out depends heavily on the language. English text is usually encoded particularly efficiently, while German text with its long compounds, or languages with other writing systems, often needs more tokens per unit of content.
Why are tokens decisive for cost and speed?
Tokens are not just a technical detail but the central unit of account in using language models. Most providers bill their services per token processed, with a distinction between input and output tokens. Anyone sending long input or having extensive answers produced therefore uses correspondingly more tokens and incurs higher costs.
Speed also depends directly on the token count. A model generates its answer token by token, so longer output takes more time. For production applications it is therefore worth designing prompts efficiently and avoiding needlessly long input or output.
Finally, the number of tokens also determines how much information can be processed at once. The maximum Context window of a model is measured in tokens and limits how much text you can pass in at once.
Tokens in practice: what to keep in mind
In professional AI projects, handling tokens deliberately is a real lever for efficiency and economy. A rough rule of thumb for English text is that one token covers about four characters or roughly three quarters of a word. For German text you should expect somewhat higher token use.
Anyone processing large documents, for instance as part of retrieval-augmented generation (RAG), should split content sensibly and pass only the genuinely relevant passages to the model. Costs and response times then stay manageable without losing quality. Understanding tokens is therefore an important basis for any scalable AI solution.
Many providers also offer tools for estimating a text's token use in advance. In recurring, high-volume applications in particular it is worth keeping an eye on that use and optimising prompts systematically. Even small savings per request add up across thousands of calls to noticeable effects on cost and speed. At Elisabit we help companies design AI applications so tokens are used efficiently without sacrificing quality.
Frequently asked questions
Is a token the same as a word?
No, a token does not necessarily correspond to a word. Depending on the tokeniser, a token can be a whole word, part of a word or a single character. Common words are often one token, rare or long words are split into several.
How many tokens does a German text have?
That depends on the content and the tokeniser. German texts tend to need more tokens than English ones because of long compounds. As a rough guide, one token often corresponds to about three quarters of an English word; in German the usage is usually somewhat higher.
Why are AI costs billed in tokens?
Tokens are a language model's natural unit of computation, since every processing step happens at token level. Providers therefore bill per token processed and distinguish between input and output tokens.
What is the difference between input and output tokens?
Input tokens cover the text you send to the model, such as your prompt and the context you supply. Output tokens are the answer the model produces. The two are generally counted separately and often priced differently.
Related terms
An embedding is a numerical vector representation of meaning that places similar content close together.
The context window is the maximum number of tokens a language model can take into account at once.
An LLM is an AI language model that understands and produces text by predicting the most likely next word.
The transformer is an AI architecture that captures relationships within text using the attention mechanism.
Prompt engineering is the craft of phrasing AI instructions so that language models return better results.
Put AI to work for your business?
We help you integrate artificial intelligence into your processes, your marketing and your website — strategically and securely.
Your contact
Stefan
I look forward to hearing about your project and finding the best solution together.
