Six months ago our monthly bill for hosted language models was the kind of number that made finance ask questions. Today it is one-tenth of that, and the models are faster on our hardware than they ever were over the public internet. We did not negotiate a discount. We did not switch to a cheaper tier. We brought the models home.
The shift started as an experiment and turned into a stack. A small, well-chosen set of open-weight models running on a single consumer-grade box at the edge of our network now handles the bulk of our inference traffic. Cloud models are still in the loop, but only for the jobs that genuinely need them. Everything else runs locally, and the savings fall straight to the bottom line.
The first lesson was that bigger is not better for every prompt. A 70-billion-parameter model is impressive, but for routine summarization, classification, and structured extraction it is wildly overqualified. We profiled real traffic and found that roughly seven out of ten requests were routine. A fine-tuned 7-billion-parameter model handled them at a fraction of the latency and zero per-token cost. Routing by task, not by habit, was the unlock.
The second lesson was about hardware. We had assumed we would need a rack of accelerators. In practice, a single modern GPU with enough VRAM to hold two or three quantized models at once comfortably covered our peak load. Quantization to four-bit weights cost us a measurable but acceptable drop in quality and gave us back enormous amounts of memory and throughput. For the long tail of low-priority jobs, a CPU-only fallback with a smaller model kept the system responsive when the GPU was busy.
The third lesson was caching. Token costs compound when you re-evaluate the same context window again and again. We added a semantic cache in front of the inference layer so that near-duplicate prompts return a stored response without ever touching the model. Combined with prompt templating that strips unnecessary history, the cache hit rate climbed past forty percent within weeks.
The fourth lesson was observability. Without per-model latency, cost-per-million-tokens, and quality-spot-check dashboards, optimization is guesswork. We instrumented every call so we could see exactly which routes were profitable and which were quietly burning budget.
If you are staring at a model bill that no longer matches the value you get from it, the path is straightforward. Profile what you actually send, right-size the model to the task, quantize aggressively, cache duplicates, and measure relentlessly. The cloud stays for the hard cases. Everything else can run on your own desk.
Curious what your traffic would look like routed this way? Reply with your top three model use cases and we will share the routing breakdown we would start with.