FastAPI has become the default for serving ML models in Python. It's async, type-safe, and produces OpenAPI docs for free.
A minimal model server
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class PredictRequest(BaseModel):
text: str
@app.post("/predict")
async def predict(req: PredictRequest):
return {"label": classify(req.text)}
Going to production
- Use lifespan events to load models once at startup
- Run behind uvicorn workers matched to CPU count
- Batch requests at the application layer for throughput
That handful of patterns will take you most of the way to a production-ready inference service.