Local AI hardware in 2026: buy memory first, then bandwidth

Why memory decides: capacity, then bandwidth

Capacity decides whether a model fits: its weights plus the working memory of the conversation. Bandwidth decides how fast it answers, because every generated token means reading the weights again; NVIDIA notes that in this decode phase memory transfer, not compute, dominates latency.

Rule of thumb: single-user speed tops out near bandwidth divided by the bytes read per token, and real systems reach only part of it (Databricks measures that share as bandwidth utilization). With the same 7B model at 4-bit, an M5 Pro (307 GB/s) generates about 66 tokens per second and an M5 Max (614 GB/s) about 120, in community benchmarks. Reading the prompt (prefill) depends more on compute than on bandwidth.

Mixture-of-experts models read only their active parameters. On the same DGX Spark (273 GB/s), LMSYS measured a dense Llama 3.1 70B in FP8 at 2.7 tokens per second; llama.cpp benchmarks show gpt-oss-120b, with 5.1B active parameters, at about 61. Large-memory, modest-bandwidth machines suit MoE models, not big dense ones.

The memory formula, with a worked example

Memory needed is roughly the weights, plus the KV cache (the per-token memory of the conversation), plus a margin:

weights (GB) ≈ parameters (billions) × bits per weight ÷ 8
KV cache per token (bytes) = 2 × layers × KV heads × head dimension × bytes per value
total ≈ weights + KV cache per token × context tokens + about 20% margin

Bits per weight: 16 in BF16, about 8.5 for Q8_0 and about 4.8 for Q4_K_M (per IBM's Granite 4.2 files). The margin is EleutherAI's heuristic; the KV cache comes from the model's config.json.

Only 16 of the model's 64 layers use full attention, hence 64 KiB per token; the older Qwen3-32B needs 256 KiB, or 32 GiB at 131K tokens, more than the whole card. According to Ollama, an 8-bit KV cache halves that memory with a very small loss in precision.

Hardware tiers in September 2026

Vendor specifications and US launch prices.

The RTX 5090 and RTX PRO 6000 share 1,792 GB/s and generate at similar speed on models that fit both (about 205 and 215 tokens per second with gpt-oss-20b, per LMSYS); the PRO card adds three times the memory, with ECC, and the RTX PRO 5000 (48 or 72 GB, 1,344 GB/s) sits in between. DGX Spark, Ryzen AI Max+ PCs and Macs trade bandwidth for capacity; only the M5 Ultra approaches a discrete GPU.

Status: the M5 Mac Studio shipped on September 22, 2026, the new M6 starts in the Mac mini at up to 32 GB, and the RTX 5090 Laptop GPU has 24 GB. Above the desktop, Microsoft announced a Windows DGX Station (GB300) for the fourth quarter of 2026 that it says runs models of up to 1 trillion parameters. In datacenters: NVIDIA's H100 (80 GB), H200 (141 GB), B200 and B300, and AMD's Instinct MI300X (192 GB) and MI355X (288 GB).

What runs on each tier

Published file sizes; add the KV cache and a margin.

Fitting is not speed: on 128 GB machines, prefer models with few active parameters. And vendor claims are ceilings: NVIDIA and AMD cite up to 200 billion parameters for DGX Spark and Ryzen AI Max+, already about 100 GB of weights at 4 bits.

Buy, rent or pay per token: a TCO method

Compare the monthly cost of the same job at equal quality: hardware over its useful life, energy and maintenance hours, against the API bill for the same tokens. Illustrative example: hardware, energy and labor figures are our assumptions; API prices are list prices as of September 26, 2026.

At this volume the local machine costs about twice the same model on Groq, and beats a frontier API only if a 27B model is good enough. Break-even with Groq comes near 170 million tokens a month with this mix, if a batching engine can serve that load. Operations, not the GPU, dominate: without them, local costs $115.

Pan et al. find that a small model on one RTX 5090 pays back against commercial APIs in about three months, but that on-premises mainly pays above 50 million tokens a month or under strict residency rules. Peng et al. (2026) found that prompt caching made an API cheaper than a shared on-premises GPU, and GPU performance per dollar improves about 49% a year (Epoch AI).

Power, noise and availability

Power and noise. The RTX 5090 draws up to 575 W (NVIDIA recommends a 1,000 W supply) and the RTX PRO 6000 600 W, or 300 W in its Max-Q version with the same memory and bandwidth. DGX Spark runs on 240 W in a 15 cm box weighing 1.2 kg. We found no comparable noise data; our rule is that a 575 to 600 W card belongs in a ventilated room or a rack.

Availability and software. DGX Spark also comes in OEM versions from Acer, ASUS, Dell, GIGABYTE, HP, Lenovo and MSI. Check format support: NVFP4 needs Blackwell GPUs, and MLX needs macOS 26.2 or later for the M5's accelerators.

Buying from Colombia and Europe

Colombia. Our prices are US launch prices; in Colombia, taxes, freight, the distributor's margin and the exchange rate move the final cost, so we publish no peso figure. Get quotes in USD and COP from at least two distributors, with warranty and delivery terms; prefer OEM versions with local invoicing and support, and check the circuit and UPS before installing a 1,000 W workstation.

Renting is also legal: Law 1581 allows transfers to countries on the SIC's adequacy list, which includes the United States.

Europe. Owned hardware or a SecNumCloud-qualified provider, which ANSSI designed against extraterritorial laws, answers the US CLOUD Act, which reaches data held by US providers wherever it is stored. For the models, see open-weight models in 2026.

Our recommendations by team size

Starting points, to check against your own models and volumes.

For a team, concurrency matters more than peak speed: in Red Hat's test on one A100, vLLM reached 793 tokens per second, while Ollama with default settings plateaued at 41. For production, rent first, measure real utilization and buy only if the GPUs stay busy; Lenovo's sponsored analysis puts the threshold near five hours of use a day against on-demand cloud. More in how to run a local LLM server.