By Ryze AI Team
Ollama is the easiest way to run open models locally. It wraps model downloading, quantization, and serving behind a simple CLI and an OpenAI-compatible API, so you can go from zero to a running model in minutes.
Download the installer for macOS, Linux, or Windows from ollama.com. On Linux you can also run the install script. Verify with ollama --version. Ollama runs as a background service once installed.
Models are pulled on demand. ollama pull llama3.2:3b downloads a 3 billion parameter Llama. For smaller devices or faster iteration, try ollama pull smollm2:1.7b or ollama pull qwen2.5:0.5b. List installed models with ollama list. Models are stored in ~/.ollama/models.
ollama run llama3.2:3b opens a chat in your terminal. Type a prompt and press enter. Use /bye to exit. This is useful for quick experiments and for checking that a model behaves the way you expect before wiring it into an app.
Ollama serves an HTTP API on port 11434 by default. Start it with ollama serve if it is not already running. Generate text with a POST to http://localhost:11434/api/generate with a JSON body containing model and prompt. For chat, use /api/chat with a messages array.
Ollama also exposes http://localhost:11434/v1, which speaks the OpenAI API. Point any OpenAI SDK client at it with any string as the API key. This makes Ollama a drop-in replacement for OpenAI during local development, so you can prototype without spending money or needing network access.
Create a Modelfile to set a system prompt, temperature, context window, and other parameters. Then ollama create mymodel -f Modelfile and run it like any other. This is how you bake a persona or a task-specific instruction into a reusable model. Modelfiles can also import GGUF files from disk, which lets you run models you quantized yourself.
If generation is slow, check that you are using a quantized variant and that your machine has enough RAM. On Apple silicon, Ollama uses the GPU automatically; on other platforms it may fall back to CPU. Set OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS to control concurrency. For large models, increase the context window only as needed, since memory scales with it.
Docs: Ollama GitHub.