Quantize a model with llama.cpp

By Ryze AI Team

This tutorial converts a Hugging Face model to GGUF and quantizes it with llama.cpp, producing a file you can run locally.

Build llama.cpp

Clone the repository and build with CMake:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

On macOS you can also brew install llama.cpp. For GPU support, add -DGGML_CUDA=ON or -DGGML_METAL=ON.

Convert to GGUF

Install Python dependencies: pip install -r requirements.txt. Then convert:

python convert_hf_to_gguf.py ./my-model --outfile model.gguf --outtype f16

This produces an unquantized GGUF file. Verify it runs with ./build/bin/llama-cli -m model.gguf -p "Hello".

Quantize

./build/bin/llama-quantize model.gguf model-q4.gguf Q4_K_M

Q4_K_M is a good default balancing size and quality. Q5_K_M is higher quality and larger. Q3_K_M and Q2_K are smaller but degrade noticeably. For very small models, prefer higher precision.

Run and serve

Run locally with ./build/bin/llama-cli -m model-q4.gguf -p "Your prompt". Serve an OpenAI-compatible API with ./build/bin/llama-server -m model-q4.gguf --port 8080, then point any OpenAI client at http://localhost:8080/v1.

Tips

Check the original model's license before redistributing quantized versions. Test quality on your task after quantization. If a layer is sensitive, use mixed precision by editing the quantization recipe. And keep the f16 GGUF as a fallback for debugging.

Docs: llama.cpp.