Why We Ditched Cloud GPUs for Local Inference

For a while, our AI agents ran on a 27B parameter model that looked great on paper and terrible in production — 20 to 45 second response times and periodic out-of-memory crashes on a 48GB machine.

The fix wasn't a bigger GPU. It was a smaller, better-quantized model. Switching to a 4-bit quantized 12B model running locally brought memory use down to roughly 7.5GB and cut response times to about 4.5 seconds, with zero OOM crashes since.

The tradeoff that made this possible: dropping cloud GPU dependency entirely. Inference happens locally, on hardware already owned, serving an OpenAI-compatible API — no per-token cloud bill, no cold starts, no waiting on GPU availability.

The catch is concurrency — the inference runtime holds Python's GIL during generation, so a single-worker API blocks while serving a request. That's handled with a busy lock that returns a clean 503 'agent busy' instead of letting the whole API hang. Graceful degradation beats a frozen server.