Quantization
Quantization denotes reducing the numeric precision of an AI model's weights, from 16-bit numbers to 8- or 4-bit representations for instance. Model size and computing needs thus fall considerably while inference speeds up noticeably. Quantization allows large language models to run on considerably smaller, cheaper hardware, generally with only a small loss of quality, and is therefore a key technology for economical use of AI.
Also known as: quantization, model quantization, weight quantization
What is quantization?
Quantization is a method for compressing AI models. At its core, the many numeric values a model consists of, above all the weights, are represented with lower precision. Instead of storing every number as a high-resolution floating point value with 16 or 32 bits, a coarser representation with 8 or even just 4 bits is used.
Quantization can be pictured as rounding numbers: where very fine gradations were stored before, the values are now mapped onto a coarser grid. That saves storage and computation, because smaller number formats need fewer resources and modern hardware processes them faster.
The decisive point is that large AI models are astonishingly robust against this coarsening. Since a model consists of billions of parameters, the loss of precision on individual values often barely registers in the aggregate. That very robustness makes quantization one of the most important tools for running models efficiently.
How does quantization of AI models work?
Technically, quantization maps a continuous range of values onto a limited number of discrete levels. For this, the range in which a model's weights lie is determined and divided into even intervals. Each original value is then assigned to the nearest of those intervals and stored in a considerably smaller number format.
Two approaches are distinguished in principle. In post-training quantization an already trained model is compressed afterwards, which is quick and uncomplicated. In quantization-aware training the lower precision is taken into account during training, so the model learns to deal with the coarsening, which preserves quality better under heavy reduction.
The whole model is often not treated the same. Particularly sensitive areas stay at higher precision, while less critical parts are compressed more. This mixed strategy makes for the largest possible gain in speed and memory without the quality of the answers suffering noticeably. The specific choice of method depends on the model, the target hardware and the use case.
What are the benefits of quantization?
The most important advantage is the sharp reduction in memory needed. A quantized model takes only a fraction of its original's space. Models thus fit into the memory of smaller systems and can in part even run on ordinary computers or locally on devices.
Added to this is a faster Inference, that is a shorter response time when using the model. Smaller number formats can be computed more efficiently and loaded from memory faster. That cuts users’ waiting time and considerably reduces energy use and running costs.
For companies that means more flexibility and independence. Quantization allows capable AI to run where expensive specialised hardware is unavailable or uneconomical too. Use cases that have to run locally for data protection reasons, or scenarios with a high volume of requests, benefit particularly, because the resources needed per request fall noticeably.
Quantization compared with other efficiency methods
Quantization is not the only method for making AI models more efficient and can be clearly distinguished from related approaches. Where quantization reduces the precision of existing numbers, so-called pruning takes a different route: less important connections or parameters of a model are removed entirely, so the network becomes leaner without changing the precision of the remaining values.
Another related method is Distillation, in which a smaller model is trained to imitate a larger one’s behaviour. Unlike quantization, a separate model that is more compact from the ground up arises in the process. In practice these approaches do not exclude one another but are often combined. Quantization’s strength lies in being comparatively easy to apply to existing models and is often the first step before more laborious methods are considered.
Limits, risks and economic use
Quantization is no free gain. With each further reduction in precision, the risk of losing quality rises. While moderate quantization often causes barely noticeable differences, very aggressive compression can make the model less accurate or give worse results on demanding tasks. Logical inference or following instructions precisely in particular reacts more sensitively than simple applications.
It is therefore important to test a quantized model against realistic examples before production use rather than relying on theoretical savings alone. Not every piece of hardware supports every quantization format equally well, either. The actual gains in speed depend on whether the processors or accelerators used can process the small number formats chosen efficiently.
At Elisabit we help companies find the right balance between efficiency and quality. We assess whether and to what extent quantization makes sense for a use case, check whether the available hardware is suitable and make sure that AI solutions can be run economically without the quality of results suffering.
Frequently asked questions
Does quantization reduce an AI model's quality?
With moderate quantization the loss of quality is usually small and hardly noticeable in practice. The more the precision is reduced, though, the greater the risk to accuracy. A quantized model should therefore be tested on realistic examples before production use.
What does 4-bit or 8-bit quantization mean?
These figures describe how many bits the model weights are stored with. 8-bit quantization uses coarser but memory-saving values than the usual 16-bit representation, 4-bit compresses further still. The lower the bit count, the smaller and faster the model but the greater the risk of losing quality.
When does quantization pay off for companies?
Quantization pays off particularly when AI models are to run on limited hardware, running costs have to fall or high request volumes are to be served efficiently. It is attractive for local applications on data protection grounds too. The prerequisite is that the loss of quality stays acceptable for the use case.
Is quantization the same as distillation?
No. Quantization reduces the precision of the numbers in an existing model, while distillation trains a smaller model that imitates a larger one's behaviour. Both serve efficiency and can be combined but take different technical approaches.
Does hardware matter for quantization?
Yes, considerably. The actual gains in speed and memory depend on whether the processors or accelerators used support the small number format chosen efficiently. Careful matching of model format and target system is therefore important for the savings hoped for actually to materialise.
Related terms
A method in which a small student model imitates the behaviour of a large teacher model.
Running a trained model in order to produce predictions or output.
A compact, resource-efficient language model that runs locally or cheaply, in contrast to a large LLM.
Fine-tuning is the targeted retraining of a pre-trained AI model for a specific use case.
An LLM is an AI language model that understands and produces text by predicting the most likely next word.
Put AI to work for your business?
We help you integrate artificial intelligence into your processes, your marketing and your website — strategically and securely.
Your contact
Stefan
I look forward to hearing about your project and finding the best solution together.
