By Ryze AI Team
Quantization is the single most important technique for running models on consumer hardware. It reduces the precision of model weights to save memory and speed up inference, and without it most modern models would not run on a laptop, let alone a phone. The core idea is simple: instead of storing every weight as a 32-bit float, store it as an 8-bit or 4-bit integer with a scale factor, and accept a small accuracy loss in exchange for a large gain in speed and size.
Weights in a trained model are typically stored as 32-bit floating point numbers (FP32). Quantization maps these continuous values to a smaller set of discrete integers. The most common schemes use 8-bit (INT8) or 4-bit (INT4) integers, usually with a per-channel or per-block scale that preserves the range of values within each group. Memory drops by roughly 4x for INT8 and 8x for INT4. Inference often speeds up as well, because smaller weights fit in cache and modern hardware has fast integer multiply-accumulate units. The catch is that rounding introduces error, and that error accumulates through the network.
INT8 is where most production systems land. Quality loss is usually under one percent on standard benchmarks, and most runtimes (ONNX Runtime, TensorFlow Lite, Core ML, OpenVINO) support it well with mature tooling. The standard approach is post-training quantization: you run a few hundred representative samples through the model to calibrate the scales, then convert. For models that are sensitive, quantization-aware training simulates quantization during fine-tuning and recovers most of the loss. Unless you have a tight memory budget, start with INT8 and only go lower when measurements force you to.
INT4 halves memory relative to INT8 and is what makes large language models runnable on consumer hardware. The cost is higher accuracy loss, especially for small models where each parameter carries more information. Block-wise quantization, with group sizes of 32 or 128, recovers most of the quality by giving each small block its own scale. Common formats include GPTQ, AWQ, and the GGUF quantization types. Always evaluate on your own task after quantizing to INT4, because the degradation is uneven: some capabilities survive, others collapse.
GGUF is a file format used by llama.cpp that bundles weights, tokenizer, and metadata in one file. It supports mixed quantization, where different layers use different precision. A typical recipe keeps attention and embedding layers at higher precision while quantizing feed-forward layers aggressively, because not all parts of the network are equally sensitive. Variants like Q4_K_M and Q5_K_M are tuned presets that balance size and quality. GGUF is also the easiest path to running LLMs locally, since llama.cpp handles the runtime.
Quantize after training, not during, unless you know what you are doing and have a reason. Always evaluate on your own task with your own metric, because benchmark numbers do not transfer. Prefer Q4_K_M or Q5_K_M for LLMs as a starting point. Keep embedding and output layers at higher precision if quality drops. Watch out for activation quantization, which is separate from weight quantization and can be the real bottleneck. And remember that quantization interacts with the runtime: the same INT4 weights can run at very different speeds depending on whether the kernel is optimized for your hardware.
Resources: llama.cpp, Hugging Face Optimum, and the ONNX Runtime quantization guide.