https://aibussin.com/tags/inference-speed/ Recent content in Inference Speed on AIBussin — AI applications, systems and books Hugo en-US Wed, 12 Feb 2025 11:07:50 +0000 https://aibussin.com/post/quantization/ Wed, 12 Feb 2025 11:07:50 +0000 https://aibussin.com/post/quantization/ <h2 id="summary">Summary</h2> <p>Large Language Models (LLMs) are powerful, but their size can lead to slow inference speeds and high memory consumption, hindering real-world deployment. Quantization, a technique that reduces the precision of model weights, offers a powerful solution. This post will explore how to use quantization techniques like <code>bitsandbytes</code>, <code>AutoGPTQ</code>, and <code>AutoRound</code> to dramatically improve LLM inference performance.</p> <h2 id="what-is-quantization">What is Quantization?</h2> <p>Quantization reduces the computational and storage demands of a model by representing its weights with lower-precision data types. Lets imagine data is water and we hold that water in buckets, most of the time we don&rsquo;t need massive floating point buckets to hold data that can be represented by integers. <code>Quantization</code> is using smaller buckets to hold the same amount of water – you save space and can move the containers more quickly. <code>Quantization</code> trades a tiny amount of precision for significant gains in speed and memory efficiency.</p>