From laptop to server: choosing, securing and sizing your own LLM stack
Running a model on a laptop takes five minutes. Serving it to 200 colleagues with acceptable latency, without exposing a port to the internet, is another job. This is the stack we recommend in September 2026, and how to size it.
Ollama, LM Studio and llama.cpp are excellent for one person or a small team; vLLM and SGLang are built for many simultaneous users on GPUs. The gap comes from serving techniques, not the model: in a Red Hat test on one A100, vLLM peaked at 793 tokens per second where default Ollama reached 41.
Self-hosting also makes you the operator: SentinelLABS and Censys counted 175,108 exposed Ollama hosts, and 2026 brought critical flaws in Ollama, vLLM and llama.cpp. Our recommendation for 20 to 200 people: one engine behind an authenticated gateway on a private network, monitored from day one, sized by concurrency and kept while volume or data residency justify it.
Four engines, three scenarios
Versions as of September 26, 2026; these projects ship every few weeks.
| Engine | Version | What it brings | Best for |
|---|---|---|---|
| Ollama | v0.34.4 (Sept 23) | Simple install, model library, OpenAI- and Anthropic-compatible API; one request per model by default | One person, prototypes |
| LM Studio | 0.4.25 (Sept 19) | App on llama.cpp and MLX; llmster daemon with continuous batching | Desktops, Mac teams |
| llama.cpp (llama-server) | v0.5.0 (Sept 23) | Almost any hardware, GGUF, continuous batching, multi-model router mode | One machine, modest concurrency |
| vLLM | v0.30.0 (Sept 22) | PagedAttention, continuous batching, prefix caching, speculative decoding, FP8 and FP4 | Multi-user production on GPUs |
| SGLang | v0.5.20 (Sept 18) | RadixAttention, prefill/decode disaggregation, expert parallelism | Agents, RAG, large MoE models |
| TensorRT-LLM | v1.2.1 (Apr 20) | NVIDIA's own engine, NVIDIA GPUs only | NVIDIA fleets with engineering capacity |
Ollama, LM Studio and llamafile share llama.cpp under the hood, and for a single user configuration matters more than the app, per Mozilla.ai's September 2026 benchmark. The real divide is concurrency. Avoid starting on Hugging Face's TGI (in maintenance mode since December 2025), mlx_lm.server or TabbyAPI, which their own authors don't recommend for production.
Four techniques that make a server
They explain why one GPU can serve one user or fifty.
- Continuous batching. The engine adds and removes requests at every generation step instead of waiting for a batch to finish, so the GPU never idles. It drives Red Hat's result with Llama 3.1 8B on one A100: vLLM peaked at 793 tokens per second with a P99 latency of 80 ms, default Ollama at 41 and 673 ms. Red Hat co-founded the vLLM-based llm-d project and tested Ollama 0.9.2 (2025): read it as an order of magnitude.
- PagedAttention. The KV cache is stored in small pages, like virtual memory, rather than one block per request. Its paper (SOSP 2023) reported near-zero waste and 2 to 4 times the throughput of earlier systems at equal latency.
- Prefix caching. When requests share a beginning (system prompt, tools, documents), the engine reuses the cache already computed. vLLM enables it by default and SGLang's RadixAttention is built around it; SGLang v0.5.20 raised hit rates from 43.8% to 60.8% in its own test. For agents and RAG it is often the largest gain.
- Speculative decoding. A draft model or extra heads (EAGLE-3, MTP) propose several tokens that the large model checks in one pass. Per vLLM's docs the gain is high at low load and moderate at high load: it mostly helps latency for a few users.
OpenAI-compatible APIs, a gateway and model versions
Your applications should not know which engine answers them.
vLLM, SGLang, llama-server, LM Studio and Ollama (partly) speak the OpenAI API, and several speak Anthropic's Messages API. Switching between a local model and a provider comes down to the base URL, but tool calls, structured output and some parameters differ: test each route.
Put a gateway between apps and engines. An OpenAI-compatible proxy such as LiteLLM Proxy adds per-team keys, budgets, rate limits, routing, fallback to a contracted provider and logs; a reverse proxy (nginx, Envoy) is the minimum. On clusters, llm-d (CNCF Sandbox since March 2026) and NVIDIA Dynamo (v1.5.0) add cache-aware routing on top of vLLM or SGLang.
Versioning models
- Pin the file hash, quantization, chat template, engine version and sampling parameters.
- Expose stable aliases at the gateway (for example chat-default) so a model swap never touches application code.
- Plan for cold starts: Ollama unloads idle models after 5 minutes by default; llama-server's router mode keeps up to 4 loaded.
What to measure: p95 latency, tokens per second, queue depth
If you only watch GPU utilization, your users will report saturation first.
| Signal | Why it matters | vLLM metric |
|---|---|---|
| Time to first token (p50, p95) | Perceived speed in chat | vllm:time_to_first_token_seconds |
| Inter-token latency (p95) | Streaming fluency | vllm:inter_token_latency_seconds |
| End-to-end latency (p95) | Agents and batches | vllm:e2e_request_latency_seconds |
| Queue depth | Missing capacity | vllm:num_requests_waiting |
| KV cache usage | Memory saturation | vllm:kv_cache_usage_perc |
| Generated tokens | Throughput and cost | vllm:generation_tokens |
llama-server exposes /metrics only with --metrics, including gauges such as llamacpp:requests_deferred for its queue; for Ollama and the rest, measure at the gateway. Before launch, load-test at your target concurrency with real prompts, for example with GuideLLM, which Red Hat used.
If the queue does not return to zero between peaks, or p95 time to first token exceeds your target (say 2 seconds for chat), you are short of capacity. Check slots, context length and quantization before buying GPUs.
Security: an inference port never faces the internet
Local does not mean safe; it means the security is yours.
In 2024, Wiz disclosed Probllama (CVE-2024-37032), an Ollama flaw that could lead to remote code execution, noting that Ollama ships without authentication. In January 2026, SentinelLABS and Censys reported 175,108 exposed Ollama hosts in 130 countries, nearly half with tool calling enabled.
The engines ship critical bugs too. In 2026: CVE-2026-7482 in Ollama (before 0.17.1, CVSS 9.1), where a crafted GGUF sent to the unauthenticated /api/create endpoint leaks memory that may hold API keys and other users' conversations; CVE-2026-48746 in vLLM (fixed in 0.22.0, CVSS 9.1), an API-key bypass; and CVE-2026-34159 in llama.cpp (before b8492, CVSS 9.8), remote code execution via the RPC backend with mere TCP access.
Its security guide is blunt: --api-key protects only some path prefixes, other endpoints have no authentication, and traffic between nodes is unencrypted, so it belongs on an isolated network. It recommends a reverse proxy that allowlists only the endpoints users need. Ollama and llama-server listen on 127.0.0.1 by default: keep it that way.
Model files need the same care. Pickle, PyTorch's classic format, can run code when loaded (JFrog found about 100 malicious models on Hugging Face in 2024); safetensors passed a Trail of Bits audit with no critical code-execution flaw. GGUF holds no code, but its parsers have had overflows and its templates can hide instructions (see AI supply chain security).
- Engines on localhost or a private subnet, in unprivileged containers without internet egress; the only door is the gateway, with TLS, SSO or per-app keys.
- Block model pull and create in production (in Ollama, /api/pull, /api/create, /api/push); load models only from an internal registry with verified hashes.
- Monthly patching: NVD lists dozens of 2026 vulnerabilities for vLLM, llama.cpp and Ollama.
- Log metadata by default (user, model, tokens, latency) and full prompts only when needed, with defined retention.
A reference architecture for 20 to 200 people
Six layers inside your network, with one controlled exit to a provider.
| Layer | What we recommend | Why |
|---|---|---|
| Interfaces | Open WebUI or an internal chat, IDE assistants, apps via the API | One front door for staff |
| Identity | Company SSO on the interface, one key per application at the gateway | Know who used what; revoke in one place |
| Gateway | OpenAI-compatible proxy behind TLS | Quotas, aliases, logs, fallback to a contracted provider |
| Inference | vLLM or SGLang on one GPU server, plus a second for redundancy; llama-server or Ollama for developers | Concurrency in production, simplicity in development |
| Models | A general model, an embedding model and, if needed, a small fast one, from an internal registry | Reproducibility and rollback |
| Operations | Prometheus metrics, alerts on queue and p95, monthly patching, evaluation before every change | No surprises for users |
Mind the license: Open WebUI allows removing its branding only up to 50 users in 30 days; above that, keep it or buy an enterprise license. Models in open-weight models in 2026, machines in local AI hardware and, if you want us to build it, custom software.
A sizing method in five steps
Size for simultaneous requests and context, then measure.
- Estimate concurrency. Requests in flight ≈ active users at peak × requests per user per minute × average duration in minutes. Illustrative assumptions: 60 of 200 people active at peak, one request every two minutes, 15 seconds each: 60 × 0.5 × 0.25 ≈ 8.
- Measure context as the p95 of input plus output tokens in real prompts, not the model's maximum.
- Add up memory: weights (parameters × bits ÷ 8), KV cache (per-token size × context × requests) and margin. Qwen3-32B in FP8 takes about 33 GB and its cache 256 KiB per token in 16-bit: 8 requests of 16,384 tokens add 32 GiB, or 16 GiB in FP8. It fits one 80 GB GPU with room to spare (see quantization without myths).
- Load-test the chosen engine and quantization at that concurrency, and compare p95 time to first token and inter-token latency with your targets.
- Keep headroom and re-measure. Our rule: plan for 1.5 times the measured peak and review queue depth monthly with real traffic.
When to stop self-hosting
Self-hosting is a means, not a goal, and the numbers often favor an API.
API prices for open-weight models are low: on September 26, 2026, OpenRouter listed gpt-oss-120b at $0.15 per million input tokens and $0.60 output at mainstream providers. An analysis by Pan et al. (2025) finds on-premise viable mainly above 50 million tokens a month or under strict residency rules, although a small model on one RTX 5090 pays back in about three months. And a 2026 case study found prompt caching cut API costs by 88.6%, below a shared local GPU.
Our rule: keep a workload with a provider if volume is under about 50 million tokens a month without a residency mandate, if you need frontier capability (Epoch AI puts open-weight models about four months behind in 2026), if GPUs would sit idle, or if nobody can own patching and on-call. If data cannot leave or volume is high and steady, go local. The usual answer: a hybrid routed by the gateway.
Colombia and Europe. In Colombia, sending personal data to a foreign provider is an international transfer or transmission (Law 1581 of 2012, Decree 1377 of 2013); the United States is on the SIC's adequacy list, with accountability still required. In the EU, the GDPR requires a processing agreement (Article 28) and a transfer basis (Chapter V). A local server avoids those questions for sensitive data, not your other obligations. More in the real cost of AI.
Key takeaways
- Ollama, LM Studio and llama-server fit one person or a small team; vLLM (v0.30.0) and SGLang (v0.5.20) fit multi-user production on GPUs.
- Continuous batching, PagedAttention and prefix caching explain gaps like 793 versus 41 tokens per second; speculative decoding mainly helps at low load.
- Never expose an inference port: 175,108 exposed Ollama hosts were observed and 2026 brought CVSS 9+ flaws in Ollama, vLLM and llama.cpp.
- Size by simultaneous requests, p95 context and KV cache; watch p95 time to first token and queue depth from day one.
- Below about 50 million tokens a month and without residency constraints, an API is usually cheaper; the gateway makes the hybrid easy.
Sources
- vLLM v0.30.0 release notes
- Security guide
- Production metrics
- Ollama vs. vLLM: a deep dive into performance benchmarking
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- SGLang repository and release notes (v0.5.20)
- llama-server README (options, metrics, defaults)
- Ollama FAQ (network binding, concurrency, keep-alive)
- Silent Brothers: Ollama hosts form anonymous AI network beyond platform guardrails
- CVE-2026-7482: Ollama GGUF heap out-of-bounds read
- Pickle scanning
- A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services
Editorial note: this analysis reflects the public information available on the review date. Models, prices and rules change fast; every third-party figure links to its source, and our opinions are labeled as such. Spotted an error? Write to contact@slash-digital.io.
The questions we hear often
Can I use Ollama in production for my team?
For a few users, with care: by default it processes one request per model at a time and queues up to 512 more. For dozens of simultaneous users, a continuous-batching engine such as vLLM or SGLang serves far more per GPU.
Is vLLM's --api-key enough to expose it?
No. vLLM's security guide says the key protects only some path prefixes, and CVE-2026-48746 allowed bypassing it until version 0.22.0. Keep the engine on a private network, behind a gateway with TLS and an endpoint allowlist.
How many GPUs does a 100-person company need?
It depends on simultaneous requests and context length, not on headcount. Estimate concurrency, add up weights and KV cache, and load-test. We recommend starting with one server sized that way and adding a second for redundancy once usage is proven.
Is GGUF safer than pickle?
GGUF and safetensors store data, not code, while pickle can execute code when loaded. But GGUF parsers have had memory bugs and chat templates can carry hidden instructions, so verify publishers and hashes and keep engines patched.
When is self-hosting cheaper than an API?
With high, steady volume (Pan et al. point to 50 million tokens a month or more), well-used GPUs and a team to run them, or when data residency is mandatory. Otherwise, APIs for open-weight models, from a few cents to about a dollar per million tokens, are hard to beat.
More analysis to read next
Local AI in 2026: which hardware to buy and what runs on it
RTX 5090, RTX PRO 6000, DGX Spark, Ryzen AI Max+ and Mac Studio M5: memory math, what fits where, buy-vs-rent costs and picks by team size.
Local AI and open weightsQuantization without myths: GGUF, AWQ, FP8 and FP4
What 4, 5 or 8 bits really cost in quality, memory and speed, which format runs where (GGUF, AWQ, EXL3, FP8, NVFP4) and how to validate a quant.
CybersecurityAI supply chain security: models, packages and MCP servers
Models that run code, hallucinated packages, malicious MCP servers and skills: 2025–2026 incidents, controls that work and the EU CRA reporting duties.
Where to next
Let's put it in production
Tell us your challenge. We reply within 24 business hours with an honest first read: if we can help, we'll say how; if not, we'll say who can.
I reply personally. No endless forms, no canned replies.