Weights = parameters × bits per weight ÷ 8. “As released” uses the real checkpoint size, which often mixes FP4, FP8 and BF16. Mixture-of-experts models hold every expert in memory, though each token uses only a few.

KV cache = values per token × bytes × tokens, for each layer that keeps a full cache, per request in flight. Most 2026 models keep a full cache in only some layers: linear-attention and Mamba layers (Qwen3.5+, Nemotron, Kimi K3, GLM-5.3 Flash) hold a small fixed state, sliding-window layers (Gemma 4, gpt-oss, MiMo, Step) keep only recent tokens, and latent or compressed attention (DeepSeek V4, GLM-5.3, Mistral 4) stores far less per token.

Speed: each decode step reads the weights it needs plus every request’s cache once. Step time = max(bytes ÷ bandwidth, FLOPs ÷ compute) + overhead, so batching shares one weight read across many users until compute or cache reads take over.

First token: prefill is compute-bound (2 × active parameters × prompt tokens, plus attention), and new prompts queue behind each other under load.

Data centre: every split of the GPUs is tried — one copy of the model across all of them, or several copies side by side — and the one fastest for each user is reported. vLLM / SGLang reserve 90% of each GPU. Beyond 8 GPUs the model spans two servers.

My machine: llama.cpp and Ollama split layers across cards, so two cards give the memory of both at roughly the speed of one. Macs let the GPU use about 67–75% of unified memory by default (sudo sysctl iogpu.wired_limit_mb=… raises it). Weights that don’t fit spill into system RAM at RAM speed.

Model specs are read from each model’s config.json on Hugging Face (November 2026). Where a name doesn’t state its active parameters they are computed from expert sizes and shown with a ~.

Accuracy: memory is reliable. Speed is usually within ±30% and depends on runtime, drivers and settings. Prices are rough mid-2026 ballparks. Benchmark before you buy.