On-Prem LLM Inference Infrastructure

The extraction pipeline needed a model large enough to read engineering drawings reliably. The company needed that model to run without a single byte reaching the cloud. And the budget for it was one machine.

The boards go into defense and aerospace programs, so the whole environment is export-controlled under ITAR. The box sits inside that controlled network, reachable only over company VPN, with no internet egress — network isolation rather than an air gap, which is a distinction worth being precise about. Every hosted model — every API, every cloud inference endpoint — was off the table before design started. The requirement wasn’t “prefer on-prem.” It was that nothing customer-related leaves the controlled network, ever.

That combination rules out most of the standard playbook. There is no autoscaling, no managed endpoint, no vendor to page when something breaks at two in the morning. Whatever gets deployed has to hold the entire production load on its own, and it has to come back by itself if it falls over.

The constraint set the search space before any evaluation started. Whatever ran here had to fit on one box, run with no internet egress at all, and read engineering drawings accurately enough that an engineer would trust the output without re-checking it. The first two conditions eliminated most of the field on their own — anything that needs a cluster, or an endpoint, was never a candidate.

What was left came to roughly twenty candidates, drawn from the open-weight families available for on-prem deployment — Qwen, Llama, Mistral, Gemma and others. The sweep deliberately covered parameter scale as well as family: 7–9B, 14B, 32B, 70–72B, and up to 122B. The question was never only which family reads drawings best. It was how much capability one box could hold at an acceptable speed.

Candidates were served locally during evaluation, through runtimes including Ollama and vLLM, and scored on the real task: reading fab drawings and stackups and returning the spec attributes, measured on accuracy against our own documents. Public benchmark scores were not part of the decision. A general vision-language score tells you very little about whether a model will read a dimension callout correctly, and the failure that matters here is a wrong number written confidently into a production system.

No candidate was ruled out on a first pass: prompting, sampling and serving configuration were worked through before anything was judged, and the strongest of the smaller candidates were then fine-tuned on our own drawings. The question worth answering was whether a smaller model could be taught to close the gap on this task rather than bought out of it with parameters.

The largest model in the sweep won on accuracy: Qwen3.5-122B-A10B, a 122B-parameter mixture of experts with roughly 10B parameters active per token. The sparsity is the reason it fits at all — a dense model of that size was never going to sit on one box. It still only survived as a production choice once it was heavily quantized: at full precision it did not meet latency, and at AWQ 4-bit it did, with an accuracy cost small enough to accept. The deployed system is that trade — the biggest model that fits, sparse where it counts, compressed hard enough to be fast.

On a single isolated box, model selection is a fit problem — capability against memory against speed — and the only score that mattered was accuracy on our own documents.

The first real decision was quantization, and it was decided by testing rather than by preference. AWQ won over NVFP4 after an ARM64 driver bug and a round of memory-ceiling testing made the trade-off concrete — on one box, the question is not which format is theoretically better but which one fits and stays up.

Serving is vLLM in Docker, with a patched transformers build for the platform. The hardest failure to chase was a CUDA-graph hang that froze production outright: the fix was moving to eager execution, which costs some throughput and buys a serving layer that does not stop responding. That is the right trade when the alternative is a silent freeze.

Around all of it sits a systemd unit whose only job is to notice the process is unhealthy and bring it back. That path was tested rather than waited for: killing the service deliberately and timing the whole return puts it at 136 seconds to healthy, unattended. Since the CUDA-graph fix went in the box has run 91 days continuously with zero restarts, across 47,973 requests and 488.5M tokens with no server errors — the recovery path is insurance that has not been needed, which is exactly why it was worth measuring in advance rather than during an incident.

I set the machine up myself, in sequence: hardware, operating system, drivers, CUDA, container runtime. All of it inside an isolated network with no internet egress, which changes the character of the work more than it sounds like it should. There is no package download, no pulling an image, no copying a fix off a forum onto the box. Every dependency has to be identified in advance, staged deliberately, and carried in — and anything you forgot is another trip.

On top of that came the hardening already described: a patched transformers build for the platform, eager execution to get around the CUDA-graph hang, and the systemd unit that brings the service back on its own.

There was no vendor to escalate to and no reference deployment to copy. That is the reason the failure modes are written down here at all.

It started out serving several people at once — engineers hitting the model interactively, up to about ten concurrently, which is the shape you reach for by default when a machine is shared. A concurrency sweep against the live service showed that was the wrong shape. Aggregate throughput saturates at 32.8 output tokens per second — only 3.45× the single-user rate — and it gets there by 16 concurrent requests. Past that, doubling concurrency buys 8% more throughput and doubles end-to-end latency.

The interesting part is why, because it is not memory. KV cache sits at 2% utilisation median and peaked at 32.5% all time. The ceiling is the scheduler’s prefill budget: against a max-num-batched-tokens of 4096 and an average prompt of 10,138 tokens, a single prompt occupies the entire prefill budget for three consecutive steps. Requests queue behind each other’s prefill with 95% of the cache sitting free.

That is the workload, not a misconfiguration. This service reads long documents and emits short answers — a prompt-to-output ratio of 36 to 1, so 97% of the token work is reading. Concurrency helps a workload that generates. It does very little for one that reads.

So it moved to a queue. An incoming package is written to a database, a spooler picks it up and hands it to the extractor, and jobs run one at a time, continuously. Each job gets the whole machine.

That took people out of the trigger path entirely. Nobody drags a job in, and nothing waits on someone remembering to. Quoting engineers open the interface and work through jobs that are already done. Interactive capacity still exists for anyone who wants it — nothing depends on it.

The instinct on a shared machine is to serve more people at once. Measuring it showed the ceiling was neither memory nor hardware but a scheduler budget, and that serving one job at a time was both faster and simpler for work that was never interactive to begin with.