AightBits

Quantization refers to the process of reducing a model’s size and increasing speed by reducing precision. I often explain this by comparing LLM quantization to JPEG compression when explaining at a high level. This works conceptually but isn’t exactly accurate.

JPEG compression works on two-dimensional image data. It divides the image into small blocks and reduces precision in each block separately. Because JPEG operates on small blocks rather than the whole image at once, the computational resources required are relatively low, and the effects are spatially localized (blurring or artifacts in one area don’t usually affect distant parts of the image).

An LLM has many interconnected components and vastly more interdependent relationships. Transformers operate in high-dimensional space (representations that connect different parts of language across layers and positions), where connections extend across the entire model. These connections (weights) between parts of language (tokens) across different positions propagate through multiple layers and interact dynamically during use (inference).

When you quantize a transformer’s weights, you’re changing values that get reused thousands of times across different contexts and inputs.

That’s why calibration matters so much for LLM quantization, and why it’s so resource-intensive. It’s not just a matter of rounding (reducing precision) on individual numbers or even small groups. The system needs to evaluate how reducing precision impacts the model’s overall behavior.