CPU and NVMe offload for inference without a p99 collapse

VRAM shortages can be covered with RAM and NVMe, but the cost rarely looks the way memory diagrams suggest. The model stops failing with OOM, but users get a long pause before the first token or stuttering during an answer that has already started. For a chat service, this is worse than an honest context limit: the interface looks healthy, while the latency tail ruins the experience for the most expensive requests.

CPU and NVMe offload for inference should be treated as a trade of capacity for predictability. RAM can serve as a working tier when the GPU-to-CPU connection is fast and contention is tightly controlled. NVMe works as a cold tier when state can be loaded in advance or reused, but it is not an invisible HBM extension inside the generation loop.

Weights and KV-cache create different kinds of latency

Weight offload and KV-cache offload solve different OOM problems, so one flag cannot replace the other. Weights are needed on every model pass. KV-cache grows with history length, the number of active sequences, the number of attention layers, the number of KV heads, and storage precision.

When some weights remain in RAM, the GPU accesses them during every forward pass. In current vLLM configurations, --cpu-offload-gb uses UVA and pinned host memory. The documentation explicitly warns that this mode requires a fast CPU-GPU interconnect because some parameters are used on the fly during every forward pass. This is primarily a risk to TPOT and ITL: the decoder has to wait for the data again and again.

KV-cache works differently. After prefill, it contains K and V for tokens that have already been processed. If an active session has lost the required blocks from VRAM, they must be brought back before attention to the history can continue. The pause may appear as poor TTFT after a cache miss or as an ITL spike during a long answer. In vLLM, the CPU KV buffer size is set separately with --kv-offloading-size; with tensor parallelism, this is the total amount across TP ranks, not the amount per card. Confusing these two figures often produces a calculation that promised 64 GiB across four GPUs but allocated it to the entire process.

There is also a third concept that is regularly mixed up with offload: a saved prefix. Prefix caching or a host-memory prompt cache can eliminate repeated prefill for the same system instruction. That is useful, but it does not mean that an arbitrary active session can decode painlessly from RAM or an SSD. Its requirements for timely data access are different.

NVMe should not sit in the decode path

NVMe can be used to store cold KV blocks and reuse prefixes, but synchronous SSD reads for every decode step almost always turn the latency tail into a lottery. Even a fast local drive shares queues with the file system, logging, model loading, and neighboring containers. A virtual cloud disk adds another layer that is not visible in the hardware specifications.

The problem is not that an SSD is slower than HBM. That is obvious. The problem is granularity: decode performs many short, dependent steps. One miss on a cold KV block can delay a sequence, while the scheduler continues serving others. The user sees not a slower average response, but a long random pause after text has already appeared.

There are three sensible ways to use NVMe.

• Store prepared KV states for long, repeated prefixes when they can be loaded before the user response begins.
• Keep a cold tier for sessions that can tolerate delayed recovery, such as batch processing, asynchronous document analysis, and draft reports.
• Preserve state across restarts or between prefill and decode roles if the transport and retention policy allow it.

For an interactive assistant, do not promise users a 128k context merely because NVMe can hold the corresponding amount of data. The promise should be based on p99 TTFT and p99 ITL under a realistic request mix. If the product requires streaming text without noticeable pauses, the SSD should remain a preparation and recovery tier, not part of the hot loop.

First calculate what does not fit

You cannot configure offload from a single VRAM figure. Break memory down into weights, persistent work buffers, graphs and temporary tensors, KV-cache, fragmentation headroom, and space for batch spikes. The model's weight size is only a lower bound.

For a rough KV-cache estimate, use:

KV bytes ≈ tokens × layers × kv_heads × head_dim × 2 × bytes_per_element

The 2 multiplier represents K and V. With GQA, kv_heads is smaller than the number of attention heads, so you cannot simply use the first number shown on the model card. For quantized KV-cache, use the actual element size, including scale and block metadata, instead of two bytes for FP16. The exact figure depends on the engine and quantization scheme, but the order of magnitude matters more than the third decimal place.

Here is a calculation worth doing before launch: with 32 layers, 8 KV heads, head_dim=128, and FP16, one token uses approximately 128 KiB of KV-cache. A 32,768-token context requires about 4 GiB for one sequence. Eight simultaneous sessions of this size already require about 32 GiB, without internal block alignment or scheduler headroom. If the model fits but the service fails only with long chats, moving weights to the CPU is treating the wrong problem.

Also check what context actually arrives through the API. Teams often test 32k synthetic tokens, then add a large system instruction, tool-call history, RAG documents, and repeated history after a failed retry in production. Log the input token count together with response length and the model-route identifier.

p99 is broken by the copy queue, not average speed

Average throughput does not answer whether offload is usable. It can remain acceptable while several requests wait for host memory, DMA, or a freed KV block. At low load, these waits are rare. As contention grows, they accumulate.

Imagine a service with four ordinary 2k-token chats and one 48k-token request that generates 1,000 tokens. While the long session fits on the GPU, it is expensive in memory but predictable. After its KV is evicted to RAM, every block brought back competes for PCIe bandwidth with accesses to offloaded weights and copies for prefill on new requests. If the same host is also exporting container logs to NVMe, a third queue appears. Average TPOT may look reasonable while p99 ITL increases tenfold.

A single-request test cannot capture this scenario. You need at least four slices:

• one request to see the clean cost of offload without queuing;
• fixed concurrency at 2, 4, 8, and then the product limit;
• short and long contexts with the same generation length;
• a mixed profile where long requests make up a small but steady share.

Also cap the queue in front of the engine. An infinite queue hides overload: you get high throughput and a p99 that includes an indeterminate wait. Controlled rejection or lowering the maximum context is more honest than a stream of responses that hangs for tens of seconds.

The vLLM documentation for bench serve separates --request-rate and --max-concurrency: the first controls request arrivals, while the second limits the number of requests actually being executed. That is exactly the distinction needed for an overload test, rather than just a polished result with --request-rate inf.

Test three memory tiers with the same profile

A comparison is meaningful only when all variants receive the same inputs, output length, sampling settings, concurrency limit, and warm-up. Do not change weight quantization, batch size, and offload at the same time. Otherwise, you will not know which trade-off bought memory and which one damaged latency.

Here is a minimal matrix for one model and one GPU configuration. It does not provide universal figures, but it shows the degradation pattern on your hardware.

For each variant, run tests at 2k, 8k, 32k, and your maximum product context length. Keep generation length fixed, for example at 256 or 512 tokens. Use ignore_eos only for synthetic testing so that an early EOS does not undermine comparability.

Example online test command:

vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/chat/completions \
  --model local-model \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 256 \
  --num-prompts 160 \
  --request-rate 2.0 \
  --max-concurrency 8 \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el

The expected output should include more than throughput. It should contain P99 TTFT, P99 TPOT, and P99 ITL blocks. This is the format produced by vllm bench serve itself. Save the raw JSON results and the versions of the driver, CUDA, engine, model, and tokenizer. Comparing CSV files after a vLLM upgrade is useless without this information.

Synthetic tests do not replace production traces. After them, repeat the test using anonymized request-length distributions, real tool-calling templates, and the actual share of repeated prefixes. But start with the synthetic matrix. It quickly shows where the curve breaks.

RAM offload requires PCIe and NUMA checks

CPU RAM is not one uniform resource with identical latency. On a two-socket server, the GPU may be attached to one NUMA node while the process, pinned memory, or NVMe interrupts are placed on another. The data path then crosses the inter-socket bus, and the missing milliseconds appear only under load.

Before testing, record the topology:

nvidia-smi topo -m
numactl --hardware
lspci -tv

In the nvidia-smi topo -m output, check the path from each GPU to the CPU and neighboring GPUs. In the numactl --hardware output, check which NUMA node owns the memory where the process will run. Do not infer speed from the words PCIe Gen5 in the server specification: the card may be installed in a slot with fewer lanes or share a root complex with a network card and a drive.

Bind the process's CPU affinity and memory policy to the node closest to the GPU. Use this as a hypothesis test, not as a permanent setting:

numactl --cpunodebind=0 --membind=0 \
  vllm serve local-model --cpu-offload-gb 12

Compare this with a run on another NUMA node under the same load. If p99 changes noticeably, you have found a physical cause rather than a scheduler mystery.

Pinned memory is usually needed for fast GPU transfers. NVIDIA writes that page-locked memory provides the highest transfer bandwidth between the host and the device, but recommends not pinning memory without a reason. This matters twice as much in inference: excessive pinning takes regular RAM away from the file cache and the cold NVMe tier. When the OS starts struggling with memory pressure, the offload benefit quickly disappears.

Do not combine weight and KV offload without separate limits

CPU weight offload and KV-cache offload can be combined, but this should be a last resort, not the default mode. Both streams use RAM and CPU-GPU bandwidth and compete for the operating system's memory resources. When one limit is set on a residual basis, the system looks stable until the first batch of long requests.

Separate the budgets on paper and in the configuration. Leave RAM for the OS, page cache, logs, tokenizer workers, the network stack, and emergency bursts. Do not assign all remaining memory to pinned memory. Set CPU weight offload to cover a small shortfall before model startup, not as a way to virtually double the GPU. Set a separate KV size and a separate limit for concurrent sequences.

In vLLM, these mechanisms are indeed separate: --cpu-offload-gb applies to weight offload, while --kv-offloading-size enables CPU KV-cache offload through the selected backend. This does not guarantee independent performance. It does let you avoid mixing two different sources of memory usage at the configuration stage.

A common recommendation is: "Since RAM is cheap, set a large offload value and let the engine figure it out." That is a poor choice for an SLO. The scheduler may use available memory efficiently, but it does not know which user scenarios can tolerate occasional 500 ms pauses and which cannot. The limit should come from the permitted p99, not from the maximum DIMM capacity.

Lower KV precision is often better than moving KV

When KV-cache itself is the bottleneck, lower cache precision should be the first candidate if the model and engine provide acceptable quality. This reduces the volume of hot data and lowers the chance of eviction. Unlike RAM offload, quantized KV remains close to the computation and does not add another DMA queue on every miss.

Do not check only a general quality benchmark. Use tasks that are sensitive to long context: finding a clause in a contract, matching table rows, multi-step tool calling, and answering questions about a document with distracting passages. Compare accuracy with FP16 KV and the selected scheme. If the model begins confidently losing details in the distant context, the memory saving was illusory.

llama.cpp separately documents support for K-cache quantization on CUDA and other backends, while its performance documentation reminds us that CPU-thread settings can limit speed even when the GPU is being used. This is a useful reminder: reducing KV does not remove the need to check the CPU. A compressed cache can save VRAM, but prefill, copies, and request handling still use the host.

For many services, a sensible order is: remove unnecessary history, enable prefix caching for repeated prefixes, evaluate KV quantization, limit concurrency for long sessions, and only then move KV to RAM. It is not the most impressive collection of flags, but it usually keeps p99 under control.

NVMe needs a cold-tier contract

If you add NVMe, define its role in the system as a contract. Which data goes to disk? When is it considered cold? Can a request continue generating before the data returns? Who cleans up the files? What happens when the disk fills or the process restarts? Without answers, this is not an architecture but a hope that the file system will take care of it.

A practical contract looks like this: hot KV for active interactive sequences lives only in VRAM; RAM holds a limited reserve for recently evicted blocks; NVMe stores only prefixes and states that can be restored before a new work stage begins; on a miss, the interactive path recomputes the prefix or moves the task to asynchronous processing. This may increase TTFT in a rare case, but it does not interrupt text in the middle of a response.

Track three separate signals: device latency, queue depth, and cache-hit percentage. A fast average read will not save the system if rare tail reads coincide with peak prefill. Do not place model files, database logs, and cold KV on the same NVMe without measuring mixed load. The storage queue is shared even when the directories are different.

For teams that care about local data placement and controlled model routing, AI Router can serve as an external OpenAI-compatible gateway, while the inference hot path itself should have explicit context and queue limits. Offload does not remove requirements for data residency, auditing, or PII masking: it only changes where tensors physically reside at a particular moment.

Make the decision at the degradation boundary

A successful offload test does not end with the answer "variant D is faster." It ends with a boundary: at what context length and concurrency do p99 TTFT or p99 ITL stop meeting the SLO? Only then can you decide what to do with the product.

If CPU weight offload causes a small p99 increase with one request but degrades sharply at concurrency 8, it can remain available for internal tasks with a queue and a separate pool. If RAM KV offload keeps p99 within limits at 8k but fails at 32k, set a separate limit for long sessions instead of using one max_model_len for everyone. If NVMe is useful only for repeated prefixes, define it as a prefix cache and do not call it a VRAM extension.

Memory is not free just because it is not on the GPU. Give each memory tier a distinct job, measure the tails separately for prefill and decode, and reject any configuration that fits the model at the cost of an unpredictable response.