Coder, a roughly 210-person company that builds remote development environments, found itself heading toward about $5 million a year in spending on commercial AI tools and APIs. Its response was not to replace cloud models with local ones across the board. Instead, the company drew a line by workload type: interactive developer chat stays on frontier cloud models, while automated background bots are routed through an internal gateway to a small on-premises cluster, with a cloud model as backup. The approach cut the cost of one documentation-checking bot from about $1,750 a month to under $11, while also exposing the limits of what a few desktop-sized AI boxes can do.

Why handing out local boxes did not work

The experiment started when NVIDIA and Dell co-sponsored a conference booth with Coder and supplied a Dell Pro Max GB10 (a DGX Spark-class machine) with 128 GB of unified memory. The unit retails for roughly $8,000, though pricing varies with memory configuration and market conditions. Running Coder's platform and open models on the box at the booth, with no data leaving the machine, generated immediate demand from employees who wanted one on their own desks.

Early adoption followed a pattern the company came to call the "local AI death loop":

  1. An engineer installs Ollama or vLLM on a local machine.
  2. They load a popular open-weight model such as Qwen 2.5 72B.
  3. They try it on real daily work.
  4. They hit slower responses or weaker reasoning than frontier cloud models and go back to the cloud.

The comparison used internally is a car rental: offered a Porsche and a station wagon at the same personal cost, everyone picks the Porsche. The conclusion was that buying $8,000 machines for more than 200 employees would produce idle hardware while cloud bills kept climbing.

Routing through a gateway with a "virtual model"

Rather than abandon local inference, Coder changed how AI traffic enters the company. Direct chat tools such as Claude.ai and ChatGPT still connect straight to their providers. Coding tools, coding agents, and automated workflows, however, must pass through an internal AI Gateway that handles security filtering, access control, centralized auditing, and model routing. Engineers, finance staff building automations, and product managers prototyping features all use tools behind this gateway.

In the model dropdown, next to Anthropic and OpenAI options, the team added an entry labeled "Solstice (Austin Office)." Solstice is a virtual model with failover logic:

  • When a request names Solstice, the gateway checks whether the on-premises cluster in Austin is online and has free concurrency slots.
  • If it does, the request runs locally at zero marginal token cost.
  • If the cluster is offline, out of power, or backed up, the request reroutes to Claude 3.5 Sonnet in the cloud.

The analogy offered is rooftop solar: panels cover baseline daytime demand at no variable cost, and the grid takes over during peaks or outages. Even with this frictionless option, human engineers still tended to pick pure frontier cloud models when given the choice.

Requests from developer workstations enter a gateway that splits traffic between desk mini servers and a large cloud

▲ Gateway routing between local cluster and cloud

Bots first, because bots have no preferences

The breakthrough came from targeting automated workflows instead of people. Coder found that headless automations account for about 20 percent of its coding AI spend, roughly $540,000 of about $2.7 million a year, while direct human use makes up the other 80 percent, around $2.2 million.

Four bots were selected:

  • doc-check, which scans every pull request for documentation that should change with the code.
  • Code review, which spins up multiple specialized agents to check pull requests for security issues and architectural fit.
  • Security triage, which filters a growing volume of incoming open-source security reports, including fraudulent or AI-generated submissions.
  • Module scorecard, which evaluates the quality of integrations submitted to a marketplace.

Bots were the lowest-friction entry point because they have no user-facing model selector, so the platform engineering team controls the model choice outright. Platform engineers also control both the prompt inputs and the programmatic outputs, which makes these jobs far more predictable than developers building apps from scratch.

The doc-check numbers

doc-check was built by a Coder engineer and originally ran every pull request through Claude Opus. With 50 to 100 pull requests a day, each invoking multiple tool calls across many files, the bill reached about $1,750 a month. After migration to the local cluster with a Sonnet fallback, the cost fell to under $11 a month, almost all of it electricity.

Item Before After
Model Claude Opus (cloud) Local LLM cluster, Claude Sonnet fallback
Monthly cost ~$1,750 Under $11
Annual electricity n/a ~$160 per node
Hardware payback at ~$8,000 retail n/a ~4 to 4.5 months

The first box was free, but even at full retail price the savings from this single workflow would cover the hardware in a little over four months. The bot's value goes beyond cost: it flags specific sections such as user guides and CLI references that need updates before a merge, which has reduced the technical writing team's need to chase developers for context.

Getting a smaller model to do the job

The migration was not smooth. Quality dropped sharply at first because the smaller open-weight model could not infer intent from short prompts the way Claude Opus had. The fix was prompt engineering in a style reminiscent of two or three years ago: an explicit execution sequence, mandatory tool definitions, and clear formatting rules. One key change forced the model to actually search the docs before concluding that none needed updating.

The evaluation method mattered as much as the prompts. The team first spent weeks building an automated benchmark in which a stronger LLM graded the last 100 historical doc-check runs. Each iteration took about 7 hours, and pass rates bounced between 60.4 percent and 85.4 percent without converging. The team then switched to manual grading: tweak the prompt, watch the bot's decisions arrive in a dedicated Slack channel, and judge them by hand. Between September 10 and September 12 the team shipped several prompt revisions this way, hand-graded roughly 80 consecutive runs, and then switched doc-check permanently to local inference. The self-assessment is candid that the automated harness may have been poorly built, but the manual loop was clearly faster for this narrow problem.

Scaling the cluster and splitting sync from async

Hardware grew incrementally rather than as a full rack: three small-form-factor boxes stacked in an office corner, cooled by a desk fan, plus units shipped to remote engineers running home server racks. Clustering the compact machines lets the team pool compute and memory for larger models.

A routing layer called solstice-1 divides work across the cluster:

Workload Target Example models
Asynchronous, non-urgent (doc-check, scorecards) Slower, smaller models on a single GPU Qwen 3.8
Synchronous developer interaction, higher-level automation Faster, larger models across multiple GPUs DeepSeek V4
Overflow when local nodes are saturated Cloud fallback Claude Sonnet

The specific models are treated as interchangeable. New open models arrive constantly, so the routing logic, not the model of the month, is what the team considers durable.

Idle hardware and the 56-hour issue audit

Weekly utilization data showed the problem with on-premises economics: short daytime spikes during the week and nearly empty weekends. With cloud APIs, unused capacity saves money; with purchased hardware, idle time is unrecovered capital. Developers were not adopting the local option for synchronous chat because it was noticeably slower and less capable, and the platform team deliberately avoided promoting it internally for that reason.

To fill the gap, the team turned to overnight batch work. One evening, an agent was instructed to inspect every open issue in the coder/coder GitHub repository and draft proposals to close those already fixed in the codebase, with explicit instructions not to close anything itself. The first attempts confused DeepSeek, and a slightly more structured, iterative prompt was needed before the job ran reliably.

The results:

  • The audit ran for 56 hours over a weekend across 756 open issues, spawning sub-agent sessions to investigate tickets and cross-reference pull requests.
  • It flagged about 100 issues as obsolete.
  • A manual check of the top 30 proposals found 29 correct and one false positive.
  • Automated closing was prohibited to avoid frustrating community contributors.

The run was not free. The router had been configured to treat a backed-up local queue as an outage, so queued prompts spilled over to Claude Sonnet and incurred about $130. A comparable run entirely on Claude Sonnet is estimated at roughly $1,100, so the net savings remained substantial, but the lesson is to disable aggressive fallback timers for batch jobs so they simply wait in the local queue.

Glowing mini servers in a dark office as a robotic arm sorts hundreds of index cards on a conveyor into two trays

▲ Overnight batch triage on idle local hardware

What a few boxes cannot do

The company is explicit about the ceiling. Two GB10 systems, roughly $20,000 of hardware, are nowhere near offsetting a $5 million AI budget, because most of that spend comes from real-time, interactive developer sessions where current local models feel slower and less intelligent than flagship cloud APIs. Speeding up synchronous chat requires much higher unified memory bandwidth, not just more capacity, which is why Coder has pre-ordered higher-end hardware, including the new Mac Studio, to upgrade its Austin setup.

Several other points shape the broader strategy:

  • Policy beats infrastructure for quick wins. Simply asking the heaviest internal power users to moderate their consumption is estimated to cut overall AI spend by about 30 percent, far more than complex technical infrastructure alone.
  • Do not divide spend by headcount. A handful of power users and programmatic workflows skew the averages.
  • Benchmarks are not production. Quantization scores and leaderboard numbers often fail to predict whether a model can finish a real multi-step workflow.
  • Trust smaller models less, not more. Local models have fewer built-in safeguards than frontier models, so security for agentic tasks must come from infrastructure: agent workloads run in Terraform-managed workspaces inside Kubernetes pods with restricted network policies and gVisor microVM isolation, keeping untrusted execution off developer laptops.
  • Classify before you infer. Lightweight classifiers at the gateway can block sensitive material such as payroll records before any tokens reach a model, and can decide whether a query needs an LLM at all.
  • Hosted open-weight models are a middle path. Coder already uses cloud-hosted open models to test its agent tools against different backends, and hosted open-weight models on services like AWS Bedrock cost substantially less than proprietary APIs, which makes them attractive routing targets.

The gateway itself is three layers: Coder AI Gateway for authentication, user attribution, team budgets, and audit logging; Bifrost, an open-source self-hosted gateway, for routing rules and endpoint switching; and a custom telemetry shim that feeds live hardware temperature, VRAM usage, and queue depth into Bifrost so traffic follows actual capacity.

Next on the roadmap are moving sensitive workflows such as customer call transcript analysis to local models, using local models for lightweight agent orchestration and Slack bot triggers, and filling off-peak capacity with repository audits and offline reinforcement learning pipelines built on sanitized interaction logs. For Coder, the cost reduction is secondary; the primary motivation is hands-on understanding of the architecture so it can advise its enterprise customers.

What to take from Coder's experience

The main risk in on-premises AI is not uptime or purchase price but adoption: expensive boxes that sit idle because nobody routes work to them. If you are considering local models, the practical order of operations looks like this:

  1. Put all programmatic AI traffic behind a gateway first, so budgets, filters, and routing can be added later without touching individual tools.
  2. Audit your bill for power users and bots before buying anything; a budget policy may save more than hardware.
  3. Start with proven asynchronous jobs such as doc checks, code review, and issue triage, where speed matters less than per-token cost.
  4. Rewrite prompts for smaller models with explicit step-by-step instructions, and grade 50 to 100 real outputs by hand rather than building a multi-hour benchmark harness.
  5. Keep a cloud fallback for spikes, but tune it so batch jobs queue locally instead of quietly spending cloud money.
  6. Sandbox every agent at the infrastructure level, and keep human verification on anything that touches public repositories or sensitive data.

Local inference, on this evidence, pays for itself quickly on repetitive background work and remains a poor substitute for frontier models in interactive coding. Treating those as two different problems is what made the strategy work.