Deploy an LLM with FastAPI

By Ryze AI Team

FastAPI is a natural fit for serving a local model: it is fast, validates inputs with Pydantic, and generates documentation automatically. This tutorial serves a Hugging Face model behind a clean HTTP API.

Install

pip install fastapi uvicorn transformers torch. For GPU, install the CUDA build of PyTorch.

The app

from fastapi import FastAPI
from pydantic import BaseModel, Field
from transformers import pipeline

app = FastAPI(title="LLM API")
gen = pipeline("text-generation", model="gpt2", device_map="auto")

class Request(BaseModel):
    prompt: str
    max_tokens: int = Field(default=64, ge=1, le=512)
    temperature: float = Field(default=0.7, ge=0.0, le=2.0)

class Response(BaseModel):
    text: str

@app.post("/generate", response_model=Response)
def generate(r: Request):
    out = gen(r.prompt, max_new_tokens=r.max_tokens, temperature=r.temperature, do_sample=True)
    return Response(text=out[0]["generated_text"])

Run locally

uvicorn app:app --host 0.0.0.0 --port 8000. Visit http://localhost:8000/docs for the auto-generated Swagger UI. Test with curl or the docs page.

Production considerations

Load the model once at startup, not per request. Add a health endpoint that reports model readiness. Use a queue or semaphore to limit concurrency, because LLM inference is not thread-safe by default and unbounded concurrency causes out-of-memory crashes. Set request timeouts. Put the service behind a reverse proxy such as nginx or Caddy with TLS. Containerize with Docker and pin dependency versions.

Scaling

For higher throughput, use a dedicated inference server such as vLLM or TGI instead of a raw FastAPI wrapper. They handle batching, KV cache management, and continuous batching, which FastAPI alone does not. FastAPI is the right choice for prototypes, custom logic, and low-traffic services.

Docs: FastAPI.