By Ryze AI Team
This tutorial converts a Hugging Face model to GGUF and quantizes it with llama.cpp, producing a file you can run locally.
Clone the repository and build with CMake:
git clone https://github.com/ggerganov/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release
On macOS you can also brew install llama.cpp. For GPU support, add -DGGML_CUDA=ON or -DGGML_METAL=ON.
Install Python dependencies: pip install -r requirements.txt. Then convert:
python convert_hf_to_gguf.py ./my-model --outfile model.gguf --outtype f16
This produces an unquantized GGUF file. Verify it runs with ./build/bin/llama-cli -m model.gguf -p "Hello".
./build/bin/llama-quantize model.gguf model-q4.gguf Q4_K_M
Q4_K_M is a good default balancing size and quality. Q5_K_M is higher quality and larger. Q3_K_M and Q2_K are smaller but degrade noticeably. For very small models, prefer higher precision.
Run locally with ./build/bin/llama-cli -m model-q4.gguf -p "Your prompt". Serve an OpenAI-compatible API with ./build/bin/llama-server -m model-q4.gguf --port 8080, then point any OpenAI client at http://localhost:8080/v1.
Check the original model's license before redistributing quantized versions. Test quality on your task after quantization. If a layer is sensitive, use mixed precision by editing the quantization recipe. And keep the f16 GGUF as a fallback for debugging.
Docs: llama.cpp.