Tokenisation

Tokenisation is thus the bridge between human language and the model's internal representation. Every input text is first turned into tokens, which are then converted into numerical representations and processed by the model.

The component carrying out this splitting is called the tokeniser. It has a set vocabulary of known tokens and a set of rules by which unknown text is translated into that vocabulary. Vocabulary and rules are developed together with the model and stay consistent across its lifetime, so training and later use speak the same language.

Modern language models mostly use methods breaking words into frequently occurring word parts. Common words are often represented as a single token, while rarer or compound terms are split into several sub-tokens. A great variety of language can thus be covered even with a manageable vocabulary.

A vivid example: a short, common word can correspond to exactly one token, while a long technical term falls into several. Spaces, punctuation and special characters are taken into account in tokenisation too. Which specific method a model uses is part of its architecture and influences how efficiently different languages are processed.

The great advantage of this splitting into word parts is that entirely unknown terms can be processed too. Even if a word has never occurred before, it can be assembled from known building blocks. The vocabulary thus stays manageably large without the model failing on rare or newly coined words, on proper names, technical terms or compounds for instance.

Tokens are the unit of computation for many language models. Both input and output are measured in tokens, and billing for commercial models often follows exactly that. Understanding tokenisation helps judge an application's resource needs better.

In practice it is worth steering token use deliberately. Concise, clearly phrased input, avoiding unnecessary repetition and a well-considered structuring of extensive documents can lower use considerably. In high-volume applications such savings add up, so designing the input efficiently pays directly into the economics.

Not every language is tokenised equally efficiently. Since many tokenisers are built on text collections in which certain languages are more strongly represented, those are often represented with fewer tokens per word. Other languages appearing more rarely in the training material split more often into many small sub-tokens.

For German, the tendency towards long compound words plays a part too. Compounds are mostly broken by the tokeniser into several parts, which can raise token use compared with shorter terms. Anyone planning multilingual applications should take such differences into account, since they affect cost, speed and the usable context length.

This knowledge matters particularly when content is to be processed in several languages. A text with identical meaning can require a different number of tokens depending on the language. A realistic estimate of token needs for the languages actually used helps plan budgets and context limits reliably.

For users, tokenisation matters above all because it directly affects cost, speed and the maximum amount of text that can be processed. Anyone processing large documents or automating many requests benefits from keeping an eye on token use.

Tokenisation also explains some quirks of language models. A model can have difficulty counting letters, for instance, because it perceives text not as individual characters but as tokens. A basic understanding of tokenisation therefore helps place such models' workings and limits better.

At Elisabit we factor tokenisation into the design of AI-driven solutions from the start. By structuring input efficiently and calculating token needs realistically, applications can be built that are both capable and economically viable, especially where large amounts of text or many requests are involved.