By Ryze AI Team
FastAPI is a natural fit for serving a local model: it is fast, validates inputs with Pydantic, and generates documentation automatically. This tutorial serves a Hugging Face model behind a clean HTTP API.
pip install fastapi uvicorn transformers torch. For GPU, install the CUDA build of PyTorch.
from fastapi import FastAPI
from pydantic import BaseModel, Field
from transformers import pipeline
app = FastAPI(title="LLM API")
gen = pipeline("text-generation", model="gpt2", device_map="auto")
class Request(BaseModel):
prompt: str
max_tokens: int = Field(default=64, ge=1, le=512)
temperature: float = Field(default=0.7, ge=0.0, le=2.0)
class Response(BaseModel):
text: str
@app.post("/generate", response_model=Response)
def generate(r: Request):
out = gen(r.prompt, max_new_tokens=r.max_tokens, temperature=r.temperature, do_sample=True)
return Response(text=out[0]["generated_text"])
uvicorn app:app --host 0.0.0.0 --port 8000. Visit http://localhost:8000/docs for the auto-generated Swagger UI. Test with curl or the docs page.
Load the model once at startup, not per request. Add a health endpoint that reports model readiness. Use a queue or semaphore to limit concurrency, because LLM inference is not thread-safe by default and unbounded concurrency causes out-of-memory crashes. Set request timeouts. Put the service behind a reverse proxy such as nginx or Caddy with TLS. Containerize with Docker and pin dependency versions.
For higher throughput, use a dedicated inference server such as vLLM or TGI instead of a raw FastAPI wrapper. They handle batching, KV cache management, and continuous batching, which FastAPI alone does not. FastAPI is the right choice for prototypes, custom logic, and low-traffic services.
Docs: FastAPI.