The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens: 83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save: peak +432 MiB. Leg c, output != reference: c sent turn 2 again, so the server rolled back to the checkpoint at the prompt end and processed 4 tokens, while the reference processed 27 from turn 1. These are different reuse points. A controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a restore, a cold save and a restore) is bit-identical, top-3 logprobs included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh and has not run. Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock -s file sleep) leaked the lock into sleep, so the SIGTERM landed in an ordinary evict_kv release (the full KV shadow, F4). Not the cache; the driver is fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
92 KiB
FORK.md - the RX 580 llama.cpp fork
Maintainer documentation for this private fork. Read it before you rebase, change, or deploy anything. The rebase and deploy procedure is in docs/fork-rebase-runbook.md, which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-10-25):
- Branch
fable5/vram-ondemand-rebase-20261024on upstream3f03b1772. The fork commits aregit log --oneline origin/master..HEAD; that list includes documentation and script commits (this file among them), so count it rather than trusting a number written here. Section 2 maps every commit to a feature. - Production runs
2fc9f9c35(deployed 2026-10-25 13:01 UTC,system_fingerprintb11906-2fc9f9c35; rollback pointer/root/DEPLOY-ROLLBACK-commit-10.txt=6c480f7e7, the previous production). It was validated with the fullfork-validate.shset (37 PASS, 0 FAIL, runbook section 6). Any later commit that touches compiled code needs the same validation before a redeploy. - Branch
fable5/disk-prompt-cacheadds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on top. It is not deployed; its validation (GPU and 35B included) is in F11. - Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line. The commit messages are detailed and are the primary record of each mechanism
and measurement; read them with
git show. - Commit trailers (owner's convention): this fork is private and never contributes upstream.
Agent-written commits end with
Co-Authored-By: Claude <model> <noreply@anthropic.com>(some older ones sayAssisted-by: Claude). AGENTS.md asks agents forAssisted-by:instead ofCo-authored-by:and forbids AI-written commit messages and PR descriptions; those are upstream llama.cpp's contribution rules and do not apply to commits that stay in this fork. Anything meant for upstream must be written by the owner under those rules.
Contents
- Purpose and target
- Commit map
- Features
- F1 MoE routing-ids readback skip
- F2 GCN flash-attention mask_opt gate (and the shmem-staging note)
- F3 Vulkan concat tiled-transpose fast path
- F4 On-demand device residency (library side)
- F5 On-demand mmproj (multimodal encoder)
- F6 Cross-process VRAM arbiter (server side)
- F7 Server shutdown path
- F8 Cold-teardown guards
- F9 Benchmark harness and validation script
- F10 Page-cache drop after a copying load
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- Dead ends
- Configuration that matters
- Measurement methodology and traps
- Known gaps and open risks
1. Purpose and target
The fork exists to serve large MoE models well on one specific machine:
- GPU: AMD RX 580 8 GB (Polaris, GCN gfx803), RADV Vulkan driver. llama.cpp reports
fp16: 0(RADV does not exposeshaderFloat16on GFX8), no int-dot, no matrix cores, wave64. PCIe 3.0 x16. RADV exposes no transfer-only queue family on this card. - Host: Ryzen 5 3600 (6c/12t), 62 GB RAM, a Talos Kubernetes node shared with about 60 other pods (CPU contention matters for decode numbers, see section 6). The GPU runs at a fixed 120 W power cap.
- Primary model: Qwen3.6-35B-A3B (unsloth UD-Q4_K_XL, 20.8 GiB), a hybrid MoE: 40 layers, 30 of
them gated delta-net (recurrent), 10 full attention, head_dim 256, 256 experts with 8 used. All
experts live in host RAM (
--n-cpu-moe 40); only the backbone, KV, recurrent state and compute buffers sit in VRAM. - Serving: llama-swap under supervisord keeps several llama-server processes loaded at once (the 35B main models, a Qwen3.5-4B "task" model OpenWebUI calls for titles and tags, gemma-4-26B multimodal, others). They cannot all fit in 8 GB, so they time-share the GPU through the fork's VRAM arbiter (F4-F7) without restarting processes, which would throw away the RAM-only prompt cache.
- Priority: prefill (prompt processing) speed first, decode second. Agentic use pushes large tool results through the model, so long-context prefill is what users wait on.
What the fork adds, in short:
- three Polaris prefill wins (F1, F2, F3), all exact (token-identical to mainline);
- on-demand release/restore of everything a model holds in VRAM, plus a cross-process arbiter that hands the GPU from one llama-server to another in well under a second for small models;
- fixes to the server's sleep, wake and shutdown paths that the arbiter exposed;
- a benchmark harness and a post-rebase validation script.
2. Commit map
Oldest first. "Feature" points at the section below.
| commit | subject | feature |
|---|---|---|
c72cb424e |
llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert | F1 |
fb6a8a076 |
ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes | F2 |
ff2a96d2d |
vulkan: record why flash-attn shmem staging stays off for GCN | F2 |
3df719e1e |
ggml-vulkan: tiled transpose fast-path for concat with transposed source | F3 |
481be8b28 |
server: on-demand VRAM sharing to time-share one GPU between models | F4, F6 |
17165fe2a |
server: optionally evict the KV cache too (Phase 2 of VRAM sharing) | F4 |
958257cd6 |
server: keep the sleep-wake path active when the VRAM arbiter is enabled | F6 |
b9df854de |
server: coordinate model load with the VRAM arbiter to avoid load-time OOM | F6 |
8d80f0673 |
llama: guard buffer iteration against released device buffers | F8 |
2feb6aeb2 |
server: restore weights/KV on wake, not at decode, so KV eviction is safe | F6 |
bf2c1a699 |
server: also free the compute-graph scheduler on KV eviction | F4 |
9857a45f0 |
llama: evict recurrent/SSM state on device release | F4 |
9b555f1aa |
ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc | F4 |
ad27d4921 |
llama: free backend compute scratch on cold; guard memory_breakdown | F4, F8 |
0a0f3ab6d |
mtmd: release/restore the encoder weights from VRAM on demand | F4, F5 |
5e99ac198 |
server: on-demand mmproj - free encoder VRAM on the text path | F5 |
b7c131219 |
docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends | docs |
5e429d4a9 |
ggml-vulkan: skip the concat transpose fast path below one tile | F3 |
4c4486f8e |
scripts: add the RX 580 benchmark harness | F9 |
878d1338f |
server: replace the VRAM doorbell with a want-lock | F6 |
53f907771 |
llama: skip the compute-buffer size check when the context is cold | F8 |
7ac2014a8 |
llama: release/restore the MoE cache with the device weights | F4 |
54a7f534b |
scripts: rx580-bench - use -lm none, document the MoE cache option | F9 |
11044b35e |
server: run the shutdown handler off the signal-interrupted thread | F7 |
6c480f7e7 |
server: release the GPU token promptly on shutdown | F7 |
e085c347d |
ggml-vulkan: env knob for the concat transpose fast path | F3 |
149a8cb0a |
scripts: add fork-validate.sh, post-rebase validation of the fork | F9 |
bfbaf15b2 |
docs: FORK.md and rebase runbook for the RX 580 fork | docs |
f5c0d8f28 |
server: fix stale VRAM-arbiter comments and the sleep log line | F6 |
f664b0f20 |
llama: env knob for the routing-ids readback skip | F1 |
62d442e7b |
scripts: rx580-bench - decode at tg256 depth 4096, not tg32 | F9 |
2512290f9 |
scripts: fork-validate.sh - gate --perf on medians, not means | F9 |
80e662a2a |
scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 |
ceab1a91f |
scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
e5fc474b8 |
scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
cce0ac1f0 |
llama: drop the model file's page cache after a copying load | F10 |
f24c3582c |
llama: do not release CPU extra buffers (CPU_REPACK) on device release | F4 |
dd851fd0c |
llama: read a released KV/recurrent state from its host shadow | F4, F11 |
2874dcb10 |
server: disk prompt cache, save slots on shutdown and restore after a restart | F11 |
2c6c517ba |
scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to cce0ac1f0 is in production (2fc9f9c35, the documentation commit after it);
6c480f7e7 was the previous production commit. The last four rows are on fable5/disk-prompt-cache
only (hashes of that branch), not deployed.
3. Features
Every feature section uses the same headings: What and why; Where; How it works; Invariants that must hold; Failure modes and incidents; Rebase hazards; Validation.
F1. MoE routing-ids readback skip
What and why. With experts host-resident, each split uploads the experts it needs. To choose them, the expert-copy callback reads the routing ids back to the host. The ids are produced on the same GPU the upload goes to, so the readback forces a full pipeline flush: 47 per eval on a 48-layer MoE. At prefill batch sizes every expert is used anyway (2048 tokens x 10 experts over 256 experts), so the readback buys nothing. Measured on Laguna-S-2.1 118B IQ2_M, all experts on host (before it was ported to the current callback): pp2048 +4.7% at depth 32k, +0.9% at 16k, flat at depth 0, tg +3%.
Where. src/llama-context.cpp, llama_context::sched_copy_experts (registered with
ggml_backend_sched_set_copy_callback in the scheduler setup). Commit c72cb424e.
How it works. When ggml_nelements(ids) >= 4 * n_expert, it skips
ggml_backend_tensor_get_async + ggml_backend_synchronize, clears st.ids_data and sets
st.used.assign(n_expert, true). The existing grouping then copies the whole expert tensor as one
range. Below the threshold the upstream readback runs unchanged. For Qwen3.6-35B (256 experts, 8
used) the skip applies to any ubatch of 128 tokens or more; decode (8 ids) never reaches it.
Invariants that must hold.
- Exactness: uploading an expert no id selects cannot change the result, because
mul_mat_idonly reads the rows the ids point at. Any upstream change that makes unselected experts matter (for example a kernel that reads every uploaded expert) breaks this argument. - With
--moe-cache-mibon, the all-true set still goes throughmoe_cache->copy_experts, which reads the host slot maps (no device readback) and copies cached experts device-to-device. Still exact. st.idscaching stays coherent: a repeat call with the same ids tensor reuses the all-true set, which is what the threshold would decide again; a differentn_expertrecomputes.- The small-batch MoE-cache slot path (
llama_moe_cache::copy) has its own readback and is not touched.
Failure modes and incidents. None in production. Historically the change lived in
ggml_backend_sched_compute_splits in ggml/src/ggml-backend.cpp; upstream #29943 (6753a033f)
deleted that block and moved expert copying into llama_context::sched_copy_experts. The original
cherry-pick therefore had nothing to apply to and was re-authored as c72cb424e. A -X theirs
resolution would have resurrected the deleted scheduler block.
Rebase hazards.
- Any change to
sched_copy_expertsor theststate struct (ids,used,ids_data). - Upstream
701fd25b3(MUL_MAT_ID id -1 skips work) left aGGML_ASSERT(id >= 0 ...)in the readback branch; the skip branch reads no ids, so it is unaffected, but watch for upstream semantics around negative ids. - If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.
Validation. fork-validate.sh --moe-exact MODEL (needs the GPU to itself, run it in the
--perf swapdown window with the 35B): reads expert_count/expert_used_count from the GGUF,
computes the threshold ceil(4 * n_expert / n_expert_used) tokens per ubatch (128 on the 35B), and
compares 64 greedy tokens after a prompt of about twice that, every expert on the host:
candidate == reference (identical-ref), and with the LLAMA_MOE_IDS_SKIP knob (f664b0f20;
=0 forces the readback, =log reports each skip on stderr) candidate == candidate with the skip
forced off (identical-noskip, which proves the skip exact even when the reference build carries
it) and the log shows skips, all at >= 4 * n_expert ids (skip-engaged). First GPU run
2026-10-25 (35B, 2fc9f9c35 vs 6c480f7e7): prompt 397 tokens, 39 skips logged, 0 with the skip forced
off, output identical in all three runs, 78 s. Also --ops (the MUL_MAT_ID
kernel, not this callback), --perf (prefill regression) and --moecache (the cache path through
this callback).
F2. GCN flash-attention mask_opt gate (and the shmem-staging note)
What and why. mask_opt skips fully-masked causal blocks and drops the per-block mask load+add
on fully-visible blocks. Upstream disables it on AMD GCN for head sizes <= 256. On Polaris attention
at depth is compute-bound on the softmax path (exp runs at about 1/4 VALU rate, no matrix cores), so
skipping that work is a real win. Measured on Qwen3.5-35B-A3B (head_dim 256): pp2048 +1% at depth 0,
+7.8% at 16k (176 -> 190), +12.3% at 32k (136 -> 153), growing with depth. Perplexity bit-identical
(5.1668 on and off).
Where. ggml/src/ggml-vulkan/ggml-vulkan.cpp, ggml_vk_flash_attn, the use_mask_opt
condition. Commit fb6a8a076. The shmem-staging note is a comment in get_fa_tuning_params_scalar
(commit ff2a96d2d, no behaviour change).
How it works. The GCN clause HSK > 256 || HSV > 256 becomes HSK >= 256 || HSV >= 256. The
existing large-mask conditions (nem1 >= 32, nem0 * nem1 > 32768,
nem0 >= block_cols * 16) keep it off for decode.
Invariants that must hold.
- Only GCN with head size >= 256 is affected. Do not lower the threshold: relaxing it below 256 was -18.5% at 16k on Laguna (head_dim 128).
- Upstream's
!use_sparseterm must stay in the condition (sparse flash-attention came in with upstreamfc82583e6). shmem_stagingstays NVIDIA-only. Measured on GCN at head size 128: -6.7% at 16k, -7.4% at 32k. Thekvshstride ofD/4+1dwords is 4 mod 32, an 8-way LDS bank conflict on wave64 (the+1padding is tuned for warp32), and the extra shared memory cuts occupancy. If anyone retries it, fix the padding to +2/+3 first.
Failure modes and incidents. A -X theirs resolution in a trial rebase silently dropped
!use_sparse from the condition. No production incident.
Rebase hazards. Any upstream rewrite of use_mask_opt, of the FA tuning parameters, or of
architecture detection (vk_device_architecture::AMD_GCN). Upstream 733f57a99 added a path that
copies strided f16/bf16 K/V into prealloc_x scratch for flash-attention on AMD non-RDNA4 (so GCN
is included) when neq1 >= 256; measured at d32768 it adds about 68 MiB of VRAM and no visible
long-context prefill loss, but it has no env toggle, so its speed effect could not be isolated.
Validation. fork-validate.sh --ops (FLASH_ATTN_EXT, 5332 cases on Vulkan0) and --perf
(pp2048 at depth 16k and 32k is where this feature shows).
F3. Vulkan concat tiled-transpose fast path
What and why. Delta-net layers build their conv input with ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0). The generic concat shader reads the transposed source with an
uncoalesced stride: about 5.6 ms for a 16 MB concat on the RX 580 against about 130 us of memory
bandwidth. On Qwen3.6-35B the fast path alone is +14% pp8192 and accounts for more than all of the
fork's prefill win at depth 0 (the rest of the fork costs about 1% there). In the validation run on
Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 1945 us without.
Where. ggml/src/ggml-vulkan/ggml-vulkan.cpp: ggml_vk_concat (gate lambda
src_transposed_2d and the s1_trans/s0_trans checks) and ggml_vk_concat_transpose_fastpath.
The graph pattern that matters here comes from llm_build_delta_net_base::build_conv_state in
src/models/delta-net-base.cpp (qwen35, qwen35moe, qwen3next). Commits 3df719e1e (fast path),
5e429d4a9 (decode gate), e085c347d (env knob).
Other graphs that build ggml_concat(state, ggml_transpose(x), 0) (found with
git grep -n "ggml_concat.*ggml_transpose" src/models/ plus the two-line form in delta-net-base and
lfm-base): qwen4exp.cpp (build_conv_state_at), bailingmoe3.cpp, clef.cpp, kimi-linear.cpp,
kimi-k3.cpp, glm5-next.cpp, mamba-base.cpp (two sites, the Mamba/Mamba2 conv and the hybrids
built on it), plamo2.cpp, and lfm-base.cpp (causal short conv; the transpose is assigned to bx
first). The fast path takes any of them when the gate holds (transposed source a plain 2D transpose
of a contiguous f32/f16 tensor, one sequence, ne[0] >= 32), so a change to the gate or the shader
affects all of them. Only the delta-net path (Qwen3.5/3.6) has been measured and validated; the
--concat check uses qwen35.
How it works. For a concat along dim 0 where one source is stored transposed (nb[1] == type_size, nb[0] == ne[1] * type_size) and the other source and dst are contiguous, it issues two
dispatches: pipeline_cpy_32/_16 copies the contiguous source into its dst columns, and
pipeline_cpy_transpose_32/_16 (the shared-memory 32x32 tiled transpose in copy_transpose.comp)
writes the transposed source into the other columns, both with custom push constants and a dst
offset. It ends with ggml_vk_sync_buffers. Everything else falls through to the generic path.
The gate requires: f32 or f16 with all three types equal (quantized and integer types stay generic);
matching ne[1..3]; ne[2] == 1 && ne[3] == 1 on the transposed source (so it fires only with one
sequence per ubatch, which --parallel 1 gives); the 16-bit doffset bound; and
ne[0] >= 32 (TILE_DIM) on the transposed source.
The decode gate (5e429d4a9): at one token the transposed source is a single column, so there is no
stride to fix, but the path still costs two dispatches and the unconditional ggml_vk_sync_buffers,
a full pipeline barrier (the generic path emits none). That barrier, not the half-empty transpose,
was nearly all of a -17% tg256 regression, because the path ran 30 times per decoded token on the
35B. The transposed source is qkv_mixed transposed, so its ne[0] is the ubatch token count:
prefill keeps the fast path, decode and small speculative drafts take the generic one. Measured
with the gate: decode back to parity (gated/clean 1.023), prefill win retained (gated/ungated 0.998,
pp8192 +11% over clean), generation byte-identical.
The knob (e085c347d): GGML_VK_CONCAT_FASTPATH=0 forces the generic path, =log prints
ggml_vk_concat: transpose fast path, transposed ne0=N to stderr for every fast-path dispatch. It is
read once (function-local static). Not set in production.
Invariants that must hold.
- The fast path fires at prefill and never at decode on a delta-net model.
- Output is identical to the generic path (it is a pure copy/transpose).
- The
types_okgate keeps quantized types off the path: the dispatches are raw f32/f16 element copies.
Failure modes and incidents.
- The decode regression above shipped in the deployed fork before the gate existed. It looked tg-neutral on the old upstream only because tg32's +-14% spread hid a 17% effect (section 6).
- If upstream changes
build_conv_state(for example aggml_conton the transpose), the fast path silently stops firing. Nothing crashes; prefill just loses about 14%. Only the concat check or a perf A/B notices.
Rebase hazards.
- Pipeline renames: upstream
733f57a99renamedpipeline_cpy_f32_f32/pipeline_cpy_f16_f16topipeline_cpy_32/pipeline_cpy_16. The pick applied cleanly and did not compile. - Upstream
f172be756madeggml_vk_concatnon-static (declared inggml-vulkan-common.h) and split helpers into headers; the fast-path helper stays static. - Shape changes in
build_conv_state/qkv_mixed([ch, n_seq_tokens, n_seqs]reshaped from a contiguousbuild_lora_mmoutput), changes tocopy_transpose.comporcopy.comppush constants, or tovk_op_unary_push_constants/get_misalign_bytes. - Upstream concat now handles quantized types; keep
types_ok. - The stock
test_concatcases intests/test-backend-ops.cppnever produce a transposed source and always havene[2]/ne[3]> 1, sotest-backend-ops -o CONCATdoes NOT exercise this path.
Validation. fork-validate.sh --concat (needs a build with the knob, i.e. this branch or
--concat-build): counts fast-path dispatches in a prefill run (must be > 0, all with ne0 >= 32)
and a decode run (must be 0 while CONCAT nodes did execute), then ties the unmodified candidate to
the fast path by CONCAT timing (candidate about equal to knob-on and at least 2x faster than
knob-off). Last run: 72 dispatches at prefill (all ne0=512), 0 over 594 CONCAT nodes at decode.
Also --perf.
F4. On-demand device residency (library side)
What and why. Several always-loaded models must share 8 GB. Co-residence is impossible (the 35B
at --n-cpu-moe 40 uses most of the card; the 4B task model needs about 2.5-3.3 GB). Restarting a
process to free VRAM throws away the RAM-only prompt cache: on Laguna a repeat turn went from a
21,960 ms re-prefill to 225 ms once the process stayed alive across a task-model call. So a model
must be able to drop everything it holds in VRAM and get it back without a reload.
Measured footprints:
- 4B: warm 3327 MB, weights released 724 MB, restored and correct.
- 4B at 32k ctx: cold weights-only 1750 MB (KV stays), cold with KV evicted 726 MB.
- gemma-26B: warm 7147 MB, cold with weights+KV but scheduler kept 718 MB, cold with the scheduler freed too 51 MB.
- A cold 4B before the encoder work held 644 MB that turned out to be its mmproj (F5).
- Re-warm cost: about 0.6 s per cold-to-warm switch weights-only (tiny prompt: 1.98 s cold vs 1.40 s warm); 2-3 s with KV eviction (it adds the KV D2H/H2D and a scheduler reserve).
Where.
src/llama-model.cpp/.h:llama_model::release_device_weights(keep_shadow),restore_device_weights(),weights_resident(); per-buffer metadatallama_model::impl::weight_release(weight_release_slot), filled inload_tensorsandload_tensors_hadamard.src/llama-memory.h:llama_memory_i::release_device_buffers(keep_shadow)/restore_device_buffers()(default no-op). Implemented insrc/llama-kv-cache.cpp(llama_kv_cache) andsrc/llama-memory-recurrent.cpp;llama_memory_hybridandllama_kv_cache_iswadelegate to their children.src/llama-moe-cache.cpp/.h:llama_moe_cache::release_device(),restore_device(),resident(),impl::invalidate().ggml/include/ggml-backend.h,ggml/src/ggml-backend-impl.h,ggml/src/ggml-backend.cpp: optionalggml_backend_i::free_scratchandggml_backend_free_scratch(); Vulkan implementationggml_backend_vk_free_scratchinggml-vulkan.cpp.src/llama-context.cpp/.h:llama_context::release_device(evict_kv),restore_device(),discard_device(),device_resident(), memberskv_device_evicted,device_discarded; the auto-restore at the top ofllama_context::decode.include/llama.h:llama_context_release_device(ctx, evict_kv),llama_context_restore_device,llama_context_discard_device,llama_context_weights_resident.tools/mtmd/clip.cpp/.h,tools/mtmd/mtmd.cpp/.h:clip_release_device,clip_restore_device,clip_weights_resident,mtmd_release_device,mtmd_restore_device.- Commits
481be8b28,17165fe2a,bf2c1a699,9857a45f0,9b555f1aa,ad27d4921,0a0f3ab6d,7ac2014a8,6c480f7e7(discard).
How it works. Tensor metadata (ggml_context structs) is durable; device buffers are ephemeral.
- Weights: on release, for each releasable buffer, capture a host shadow once (weights are
read-only), free the buffer, null every tensor's
buffer/data. On restore,ggml_backend_alloc_ctx_tensors_from_buftreallocates an identical layout (re-pointing the tensors), the shadow is uploaded, usage is set to WEIGHTS. Only single-buffer, alloc-path, non-host buffers are releasable: the host-pointer path (mmap or lazy-mapped),no_allocdummy buffers and host (Vulkan_Host, i.e. pinned RAM) buffers stay as they are. - KV and recurrent state: same pattern, but the shadow is re-captured on every release (the data is read-write) and dropped after restore.
- Scheduler: with
evict_kv,sched.reset()frees the worst-case compute buffer andsched_need_reserve = true; restore callssched_reserve()(which goes throughsched_new(), re-registering the expert-copy callback and upstream's realloc callback). - Backend scratch: with
evict_kv,ggml_backend_free_scratch()on each backend. On Vulkan this discards the pending compute context, synchronizes, destroysprealloc_x,prealloc_y,prealloc_split_k,prealloc_add_rms_partials,sync_staging, and zeroes their sizes and theprealloc_y_last_*tracking. They are reallocated lazily on the next compute. Other backends leave the hook null (no-op). - MoE cache (
--moe-cache-mib, upstream #29887): released in both modes. The banks are freed and the cache is emptied, not shadowed: every SLRU is rebuilt fresh, itsslot_map[il]pointers are re-pointed at the host slot-map tensors, and the host slot maps are cleared to 0xff (every slot -1). On restore the banks are reallocated (which also re-inits their views), cleared to 0 and marked WEIGHTS; the cache refills from the host experts. - Multimodal encoder:
clip_release_device/clip_restore_deviceuse the weights pattern (shadow once); no-op for a CPU-backed encoder.mtmd_*wrap the vision and audio contexts. llama_context::release_device(evict_kv): synchronize, release weights, release MoE cache, and ifevict_kv: release memory buffers, free sched, free scratch.restore_device(): weights, MoE cache, memory buffers,sched_reserve()if needed, thenggml_backend_sched_synchronizeso uploads finish before any compute.discard_device(): likerelease_device(true)withkeep_shadow = falseeverywhere (no host copies), setsdevice_discarded; used only before exit.decode()callsrestore_device()if!device_resident(), as a safety net. The server restores explicitly on wake (F6), because some KV access happens before decode.
Invariants that must hold.
- Shadows are stored compactly in stable tensor-iteration order, skipping view tensors
(
view_src != NULL), and restored with the same iteration. Never index a shadow byt->data - base. weight_releasestays parallel topimpl->ctxs_bufs: every push toctxs_bufsneeds a matching push toweight_release(non-releasable slots useemplace_back()).- The caller synchronizes before any release;
release_devicedoes this itself. - Restore order: weights, MoE cache, KV/recurrent, scheduler reserve, scheduler synchronize. The MoE cache banks must be back before the scheduler is rebuilt.
- After a MoE-cache release no slot map may point at an expert: banks come back zeroed, so a stale slot map silently reads zero weights.
- Every iterator over
ctxs_bufs(model, KV cache, recurrent) must tolerate a null buffer, and every use ofschedmust tolerate null while cold withevict_kv(see F8). device_resident()covers weights and MoE cache; it gates the early returns, the decode auto-restore andllama_context_weights_resident().restore_device()afterdiscard_device()aborts (GGML_ASSERT(!device_discarded)).- Greedy output is byte-identical across any number of release/restore cycles, and the prompt cache
survives (
timings.prompt_ndrops on a repeated prompt after a cycle).
Failure modes and incidents.
- The first shadow implementation indexed by
t->data - baseand produced garbage after restore (layout and padding differ). Fixed by the compact iteration-order shadow. - Restoring KV lazily at decode crashed iSWA models (gemma) with
GGML_ASSERT(buf != NULL && "tensor buffer not set"):update_slots()creates an SWA checkpoint that reads the KV viaggml_backend_tensor_getbefore decode. Fixed by restoring on wake (2feb6aeb2). - The recurrent state was left resident (default no-op), 201 MB on the 4B, until
9857a45f0. - A model with weights on the CPU device (
-dev none/-ngl 0) segfaulted (exit 139) inrelease_device_weights()the first time the arbiter put it to sleep: CPU_REPACK buffers are not host buffers, soload_tensorsmarked them releasable, and the shadow capture called their NULLget_tensor. It also crashes production2fc9f9c35, but no production arbiter model has CPU-device weights. Fixed inf24c3582c: only buffers on a non-CPU device are releasable. Found by the--diskcachecheck (F11), whose arbiter cases run CPU only. - The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed
resident in a cold model until
bf2c1a699/ad27d4921. - The MoE cache was not released at all before
7ac2014a8. Reallocating its banks without invalidating would have produced silent wrong weights. - In the 20261024 rebase
moe_cache_slru::init()turned out to reset only the probation segment (head[1],tail[1]andn_protectedsurvive), so re-running it on a used cache corrupts the lists.invalidate()now assigns a freshmoe_cache_slru()beforeinit().
Rebase hazards.
- New push sites into
ctxs_bufs(upstream'sload_tensors_hadamard()added two in the 20261024 rebase; without matchingweight_releaseentriesrelease_device_weights()reads past the end). - Changes to buffer creation in
load_tensors(thehost_ptr_pathcondition,ctx_bufs/buf_ptrsbookkeeping; upstream replacedctx_buf_maps.emplace_back(ctx, buf_map)withctx_bufs.emplace_back(ctx, std::move(buf_ptrs))in the 20261024 rebase). - The
llama_memory_iinterface. New memory types that do not overriderelease_device_bufferskeep their KV resident underevict_kv; at the time of writing that isllama_kv_cache_msa,_dsa,_dsa_iswa,_dsv4andllama_memory_hybrid_iswa. - MoE-cache internals (
groups,slru,slot_map,buf_host,devices). Upstream33d6135cbchanged LRU to SLRU (g.lru->g.slru); the pick applied cleanly and did not compile. llama_contextmembers next to ours (upstreamn_reallocs_same/diff,copy_experts_info),sched_reserve()semantics, and thedecode()signature (nowdecode(const llama_batch_ext &)).ggml_backend_ilayout (free_scratchmust stay the last member aftergraph_optimize; other backends rely on zero-init) and the Vulkan context's prealloc field set (compare withggml_vk_cleanup).
Validation. fork-validate.sh --arbiter: b-greedy-identical (15 cycles each side equal to a
no-handoff reference), b-kv-survives (prompt_n drops after evict/restore, output identical),
c-image-after-restore. --moecache MODEL (needs a cache large enough to hit; 256 MiB on gemma-26B
had 0 hits and proves nothing, 2048 MiB had 6651 hits). --arbiter v-vram-release /
v-vram-restore check that VRAM really drops: a baseline of mem_info_vram_used before any scratch
server, then server A warm and alone, forced cold by a fake waiter on want.lock: the card-wide delta
must drop by >= 70% of what A added and come back after the next request. Production's own share is
re-read from the DRM fdinfo of every process the script did not start; if it moved by > 64 MiB the
check SKIPs. First run (2026-10-25, 0.8B with mmproj, ctx 8192, LLAMA_SLEEP_EVICT_KV=1): warm
+546 MiB, cold +4 MiB, restored +546 MiB.
F5. On-demand mmproj (multimodal encoder)
What and why. A multimodal model's vision encoder sat in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the encoder lives in RAM and enters VRAM only for an encode. Measured on the
35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered
correctly, text after the image correct.
Where. tools/server/server-context.cpp: encode_mtmd_chunk (upstream 9871df591 split the
encode out of process_mtmd_chunk; encode_mtmd_chunk is called from process_mtmd_chunk and
directly from the batched-embedding path in the slot update, so the restore covers both), the
mtmd_release_device(mctx) after mtmd_init_from_file in load_model, vram_go_cold,
vram_share_exit. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
How it works. After load the encoder is released (shadow captured). Before
mtmd_batch_encode, mtmd_restore_device(mctx); if that fails (out of VRAM) and on-demand is set, it
releases the LLM backbone weights only (llama_context_release_device(ctx_tgt, false): read-only
shadow, so a cheap free with no D2H, KV and prompt cache untouched) and retries. After the encode it
releases the encoder FIRST, then restores the weights, so the peak stays within VRAM.
Invariants that must hold.
- Free the encoder before re-uploading the weights.
- The donor is the weights, never the KV (the KV is read-write and would need a D2H every time).
vram_ensure_warm()does not restore the encoder; only the encode path does.
Failure modes and incidents. Before this, a cold text model with an auto-downloaded mmproj
(-hf pulls one when the repo ships it) held 644 MB that no release path touched. The text-only 4B
task model now runs with --no-mmproj, which is the right fix for any model that never sees images.
Rebase hazards. Renames or restructuring of encode_mtmd_chunk and the mtmd init block in
load_model; clip_ctx fields (buf, ctx_data, sched, no_alloc).
Validation. fork-validate.sh --arbiter case c-image-after-restore (server A runs with
--mmproj and LLAMA_MMPROJ_ONDEMAND=1; a 128x128 red PPM must be answered "red" after A was cold,
and text right after must equal the reference). Post-deploy: one image request to gemma-26B.
F6. Cross-process VRAM arbiter (server side)
What and why. Several llama-server processes under llama-swap share one GPU: exactly one is warm
(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production:
60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between
the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last
fork-validate.sh run before the 20261024 deploy: 30 handoffs, median 680 ms, max 1231 ms (the
2026-10-25 run of the same build read 1050 ms, max 1562 ms: handoff latency moves with node load,
which is why it is now A/B'd against the reference in one session).
Where. tools/server/server-context.cpp: vram_arena_open, vram_acquire_for_load,
vram_share_init, vram_should_yield, vram_want_acquire, vram_want_release, vram_take_token,
vram_ensure_warm, vram_go_cold, vram_share_exit, vram_share_shutdown, the VRAM-only branch of
handle_sleeping_state, the vram_ensure_warm() call before decode in the slot update path, and
server_res_generator (bypass_sleep). tools/server/server-queue.cpp/.h: on_should_yield,
start_loop (should_sleep, idle tick, post-wake grace), wait_until_no_sleep, is_stopping.
Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
How it works.
- Enabled by
LLAMA_SLEEP_VRAM_ONLY(any value). The arena directory isLLAMA_VRAM_ARENA(default/dev/shm/llama-vram) with two files. token.lock:flock(LOCK_EX)is the baton. Rule: a process has device buffers iff it holds it.want.lock: a waiter holds itLOCK_SHwhile waiting and drops it once it owns the token. The holder probes it withflock(LOCK_EX | LOCK_NB)on a SECOND fd: failure means somebody is waiting. The probe needs its own open file description, because flock on the waiter's own fd would convert that lock instead of conflicting with it. The kernel drops a dead waiter's lock, so nothing goes stale.- Load:
load_model()callsvram_acquire_for_load()beforecommon_init_from_params(), so the current holder releases before this model uploads anything. The loaded model stays warm holding the token and serves its first request without a re-warm. - Yield:
vram_share_init()registersvram_should_yieldwithqueue_tasks.on_should_yield.start_loop()'sshould_sleep()calls it on the loop thread, undermutex_tasks, only when the task queue is empty andupdate_slotshas returned. While the callback is set the idle wait ticks every 10 ms instead of 1 s. After a wake the server holds the GPU for at least 100 ms (yield_grace_ms). Sleep runs the sleeping callbacks; in VRAM-only modehandle_sleeping_state(true)callsvram_go_cold()(release device, also the encoder; thenflock(LOCK_UN)) instead of upstream's full unload. - Wake: a request calls
wait_until_no_sleep().handle_sleeping_state(false)callsvram_ensure_warm(): take the token (register on want.lock, then polltoken.lockwithLOCK_EX | LOCK_NBevery 2 ms, aborting whenqueue_tasks.is_stopping()), restore ctx_tgt and ctx_dft, and only then setvram_cold = false. The decode path callsvram_ensure_warm()again (no-op when warm) and throws "server is shutting down" if it returns false. server_res_generatorno longer bypasses the wake path when--sleep-idle-secondsis unset ifLLAMA_SLEEP_VRAM_ONLYis set, because the arbiter can put a server to sleep without idle-sleep.- Queue fixes:
wait_until_no_sleep()loopswhile (sleeping && !stopping)and re-assertsreq_stop_sleepingeach time (withnotify_all); the sleep wait also wakes on a non-empty task queue; the idlewait_foronly leaves the loop on a real task or termination.
Invariants that must hold.
- Resident iff holding
token.lock;vram_cold == falsemeans both.vram_ensure_warm()setsvram_cold = falseonly AFTER the restore. What that order protects today, reasoned from the code: the only thing that gives the token away isvram_should_yield(), called fromshould_sleep()on the loop thread.vram_ensure_warm()also runs only on the loop thread, synchronously: from the wake callback (handle_sleeping_state(false), called insidestart_loop()withmutex_tasksheld) and before decode in the slot update path (outsideyield_to_queue()). Soshould_sleep()cannot run between the token acquire and the end of the restore, whichever order the two lines are in. The measured failure (attempt 1 below: handoffs of 11m20s and 3m57s) happened under the retired doorbell design, where a separate warden thread read the cold flag to decide whether to honour or discard rings, so an earlyfalselet it accept stale rings during the restore. Under the inline want-lock probe the order is defensive: it keeps the flag's meaning exact for any future reader off the loop thread, and keeps a release out of a half-restored model if the restore or the probe ever moves (for example intoyield_to_queue()). Either way, do not change it without an A/B of ordinary handoff latency (fork-validate.sh --arbiter,a-handoff-latency). vram_acquire_for_load()setsvram_cold = falseas soon as it holds the token, before the upload. Nothing reads the flag during the load today (the loop is not running yet andon_should_yieldis only registered invram_share_init(), which sets it again), so the early set is redundant now. It records "holding the token" at the moment it becomes true, because the opposite error is the dangerous one: a model that is warm and holds the token whilevram_coldsays cold never yields (vram_should_yield()returns false for a cold model), which strands every waiter, and its next wake would try to restore weights that were never released.- Release only happens from the idle branch of
start_loop(), on the loop thread, never insideyield_to_queue()or while a slot is generating: a holder finishes its current generation before it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both). - The yield decision reads live kernel state at the point of use. Do not route it through a flag set by another thread (see attempts 3-4 below).
- Every exit from
vram_take_token()drops want.lock, or the holder yields forever. - The probe fd is separate from the waiter fd.
flockcalls retry on EINTR: the server's signal handlers are installed without SA_RESTART.- Restore on wake, before
update_slots(), so KV eviction is safe for SWA checkpoints. - Every model that can be warm on the GPU at the same time must use the same arena. A non-arbiter process (a benchmark, a manually started server) can OOM the warm model. In production the env is set for all of llama-swap, so all its children participate.
Failure modes and incidents.
- Wake-path hang (
958257cd6): with no--sleep-idle-seconds,bypass_sleepskippedwait_until_no_sleep(), so a server slept by the arbiter never woke and requests hung. - Load-time OOM (
b9df854de): weights were uploaded before the arbiter was active; loading gemma while the 4B held VRAM could exceed 8 GB. After the fix the peak was 7771 MB. - The doorbell design (replaced by
878d1338f) was edge-triggered: a waiter wrote<arena>/doorbell/<pid>and the holder's warden thread turned the inotify event into a yield flag. Rings were lost in three windows: while the holder restored (it advertised cold and discarded foreign rings), while the holder loaded (the inotify watch did not exist yet, and the kernel does not queue events for a missing watch), and when a cold waiter consumed a third process's ring. Every loss was permanent:flock(LOCK_EX)had no timeout and production had no idle-sleep. The blocked flock heldmutex_tasks, so the whole server stopped answering while/healthstill returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by the second day). - Failed fix attempt 1 (doorbell design): moving
vram_cold = falsebefore the restore made the holder's warden honour stale rings and hand the token back 2-5 s after taking it without serving. Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. See the first invariant for what this means under the current design. - Failed fix attempt 2: a re-ring loop (
LOCK_NB+ re-ring every ~1 s). Queued rings from an already satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become level-triggered. - Failed attempts 3-4: want.lock was sound (mutual exclusion, kill -9 safety, load-window coverage),
but the warden thread still delivered its decision as a flag.
start_loop()holdsmutex_tasksfromshould_sleep()through the callbacks to the wait, so the warden'srequest_yield()was always admitted after the flag had been consumed, latching a release for the next wake (a server woke and re-slept 138 ms later without serving). It also exposed the upstream lost wakeup inwait_until_no_sleep(). Result: 0.494 s median, 1/60 stranded. The fix deleted the warden andrequest_yield/yield_requestedand probes inline inshould_sleep(). - Measured deviations that are required: the 10 ms idle tick (about 0.2% of one core per warm server); a waiter that polls rather than retrying on a coarse grid (a 200 ms retry grid cost 0.503 s against 0.336 s, because the holder only releases after the waiter asks); the 100 ms post-wake grace (without it a permanent waiter caused a livelock: one request took 23.15 s over 138 sleep/wake cycles).
- Process lesson from attempts 1-2: validate an ordinary handoff latency A/B, not only the pathological case. Two production deploys were burned before that rule existed.
- Lost diagnostic: the old periodic "still waiting for the GPU token after N s" warning is gone; a
stuck token now shows one
VRAM arbiter: waiting for the GPU tokenline and then silence. Check/proc/locks(see the runbook).
Rebase hazards. This is the most fragile area, because it sits inside upstream's server loop.
server_queue::start_loop,wait_until_no_sleep,process_new_tasks,yield_to_queueand the sleeping-callback vector. Upstream refactored sleep (947fd9bb2) and made decode run insideyield_to_queue(22b8e310b); re-check after any such change that a release still cannot run under a live graph.handle_sleeping_state,load_modelordering, the slot update path where decode happens,server_res_generator'sbypass_sleep, renames inserver-context.cpp.- Anything that touches the KV in
update_slots()before decode.
Validation. fork-validate.sh --arbiter with two real servers on a scratch arena:
a-handoffs (30 alternating, 0 stranded, 0 non-200, the server that answered holds token.lock),
a-handoff-latency (fresh server pairs of --build and --ref in the same session, blocks
ref, cand, cand, ref x 16 handoffs; FAIL if the candidate median exceeds the reference median by more
than max(25%, 200 ms); first run 2026-10-25, reb1024 against itself beside production: 1050.5 vs
1082.5 ms), v-vram-* (F4), b-*, c-*, g-kill9-waiter (kill -9 a waiter registered on want.lock while the holder streams:
want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in
F7 and a log scan for asserts and Vulkan errors. In production, /proc/locks must show the warm
model holding WRITE on token.lock.
F7. Server shutdown path
What and why. llama-swap evicts models with SIGTERM, so shutdown happens all the time, often while the arbiter is mid-release or mid-wait. Two bugs made shutdown keep the GPU token and stall every other model.
Where. tools/server/server.cpp: signal_handler, shutdown_pipe, struct shutdown_thread,
sig_thread in llama_server. tools/server/server-context.cpp: server_context::start_loop (calls
vram_share_exit() after the loop returns), server_context::terminate (terminates the result queue
first), vram_share_exit, the destructor's vram_share_shutdown(). tools/server/server-queue.cpp:
server_queue::terminate (sets atomic stopping before taking the mutex), process_new_tasks,
server_response::recv_with_timeout, server_response::terminate,
server_response_reader::next. Commits 11044b35e, 6c480f7e7.
How it works.
- The signal handler only writes the signal number to a self-pipe (async-signal-safe); a dedicated
thread reads it and runs
shutdown_handler. The pipe fds areFD_CLOEXEC. Ifpipe()fails the old direct call is used. The thread object is declared after the server objects, so it is joined (a 0 byte unblocks it) before they are destroyed. A second signal still callsexit(1). terminate()setsstopping(atomic) before lockingmutex_tasks, then stops the result queue so HTTP handlers waiting for results get "server is shutting down" (HTTP 503, or an SSE error event);recv_with_timeoutchecks!runningbefore waiting and returns null.- A token wait polls with
LOCK_NBevery 2 ms and aborts onis_stopping(); the aborted wake stays cold, skips the restore, and the loop exits beforeupdate_slots. - After
start_loop()returns,vram_share_exit()frees all device buffers withllama_context_discard_device()(no host shadows), releases the encoder, and only then unlockstoken.lock. - What llama-swap (v239 on the pod) does when a child exits on its own (crash, OOM kill, external
signal), verified 2026-10-25 with a scratch llama-swap instance and a CPU-only child, for both
kill -9and an external SIGTERM: it logs[WARN] group: running <model> exited: [<model>] upstream exited unexpectedly, does not restart it eagerly (no new process in the 2 s after the kill), and starts a new process on the next request for that model (new PID, health check, HTTP 200). The production log since the last pod restart has no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers the restart pays a full model load (about 26 s for the 35B with-lm none; the first 35B request after the 2026-10-25 deploy took 34 s including load and answer), and under the arbiter the new process takes the token throughvram_acquire_for_load()like any load. A process that wedges instead of exiting (the incidents below) is never restarted:/healthkeeps answering.
Invariants that must hold.
- Nothing that takes
mutex_tasksruns in signal context. - The token is released only after this process's VRAM is free (the next model may start uploading the moment the lock drops).
- A shutting-down server never restores weights.
- SIGTERM in any state exits 0 within about 2 s and frees the token. With the disk prompt cache (F11) the exit also writes each slot's state: about 1 s more per GB of state, after the token is free.
Failure modes and incidents.
- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside
a sleeping callback (under
mutex_tasks, during a release); the handler'sterminate()lockedmutex_tasksagain on the same thread. The process wedged holdingtoken.lock, every other model queued behind it, and only SIGKILL worked. Reproduced 6/6 off production; 8/8 clean with the fix. The first suspicion (the multimodal path) was wrong. gdb on the wedged process showed the main thread infutex(mutex_tasks)with theyield_to_queueworker idle. - Token kept on shutdown: with a request in flight the HTTP handler waited forever for results, main
blocked joining it, and the token was released only in the destructor after that join; with a
token wait in progress the blocking flock held
mutex_tasks; and the old destructor unlocked the token BEFOREdestroy(), so the next model could upload into still-occupied VRAM. - After the fix: off production SIGTERM mid-generation or mid-wait exits in about 0.2 s with exit 0 and frees the token in 5-70 ms; in production about 0.7-1 s.
Rebase hazards. server.cpp's signal setup and llama_server object lifetimes;
server_response and server_response_reader (upstream 4e201388c switched recv_with_timeout to a
fixed deadline: keep the order "check !running, return null, then wait_until(lock, deadline)");
server_context::start_loop/terminate. Router mode (server-models.cpp,
server_models_sse_client::next) only checks should_stop() and can spin at shutdown now that
recv_with_timeout returns null at once; llama-swap does not use router mode.
Validation. fork-validate.sh --arbiter: d-term-in-release (gdb breakpoint on
llama_context_release_device, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and
the token), e-term-waiting (exit 0 within 1.5 s, never took the token), f-term-streaming
(exit 0, SSE error event, token free within 2.5 s), h-shutdown-warm, h-shutdown-cold. Last run
(2026-10-25, 2fc9f9c35, d by log polling): 750 ms, 225 ms, 126 ms (token free in 23 ms),
108/159/159 ms. Post-deploy: a forced llama-swap eviction
must log VRAM arbiter: loop stopped, device memory freed, releasing the GPU token and exit by itself.
F8. Cold-teardown guards
What and why. A cold model has null buffers in its ctxs_bufs and, with evict_kv, a null
scheduler. Upstream code never had to consider either state.
Where. src/llama-context.cpp: llama_context::~llama_context (sched && in the
compute-buffer size check), llama_context::memory_breakdown (if (sched) around the compute part;
upstream's if (moe_cache) breakdown block stays outside it). Null-buffer skips in
llama_model::memory_breakdown, llama_kv_cache::{clear, memory_breakdown, total_size},
llama_memory_recurrent::{clear, memory_breakdown, total_size}. Commits 8d80f0673, ad27d4921, 53f907771.
How it works. Guards at the call sites. The asserts in ggml (GGML_ASSERT(buffer),
GGML_ASSERT(sched)) are correct and stay; every other caller reserves the scheduler first.
Invariants that must hold. A server shut down while cold exits 0. Any new upstream code that
iterates ctxs_bufs or calls into sched on a path reachable while cold (shutdown, memory
breakdown, stats) needs the same guard.
Failure modes and incidents.
GGML_ASSERT(buffer)on shutdown of a cold model (seen 7 times in production logs) from memory-breakdown/size/clear on released buffers.GGML_ASSERT(sched)inmemory_breakdownfor a coldevict_kvmodel being torn down.GGML_ASSERT(sched)in the destructor's size check: every cold shutdown exited 134 instead of 0, so under llama-swap an ordinary stop looked like a crashed child, left a backtrace in the log, and the rest of the destructor never ran. Repro: arbiter env on, one request, force a yield withflock -s <arena>/want.lock -c 'sleep 6', then SIGINT while cold.
Rebase hazards. New iteration over ctxs_bufs, new sched users in teardown or breakdown
paths. Watch item: llama_kv_cache::update() takes lctx->get_sched(), which can be null; it is
safe today only because its single caller runs after sched_reserve().
Validation. fork-validate.sh --arbiter: h-shutdown-cold (exit 0 while the other server holds
the token) and log-scan; --moecache shutdown.
F9. Benchmark harness and validation script
What and why. The measurement method on this box had to be rediscovered several times (section 6). The harness encodes it; the validation script encodes the checks that each past incident required, so a rebase can be validated by someone with no context.
Where. scripts/rx580-bench/ (run.sh workstation wrapper; bench.sh pod orchestrator;
ppbench.py corpus prefill through llama-server; lbsweep.sh llama-bench sweep; summarize.py;
README.md with methodology and reference values). scripts/fork-validate.sh. Commits 4c4486f8e,
54a7f534b, 149a8cb0a, 62d442e7b, 2512290f9, 80e662a2a, ceab1a91f, e5fc474b8.
How it works.
rx580-bench: stops llama-swap, waits for an idle GPU, runs fixed slices ofpan-tadeusz.txt(4096/16384/32768 tokens, byte-identical across builds) through/completionwithcache_prompt: false, plus a llama-bench sweep (prefill pp2048/pp8192 at depth 0, decode tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and rounds; restarts llama-swap from an EXIT trap. Results append to/root/bench/results.txt.fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL] [--moe-exact MODEL] [--diskcache]: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under /tmp, a scratch arena (refuses/dev/shm/llama-vram*), ports 8091-8093, unsets inheritedGGML_*/LLAMA_*env, kills only PIDs it started (PID-reuse safe). Default with no check flag is--ops --concat --arbiter.--perf-report TSVre-aggregates a savedperf.tsvoffline. See the header of the script for what each check proves.- Which phase needs what:
--ops --concatrun beside production;--arbitertoo only if 1536 MiB of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs in the swapdown window, about 2 minutes);--perf,--moe-exactand--moecacheneed llama-swap stopped (the first two refuse otherwise;--moecachedoes not check, but loads a MoE model and must not share the card either).--diskcacheruns beside production (CPU only); its GPU leg SKIPs below 1536 MiB of free VRAM.
Reference values (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, --perf on 2026-10-24,
production 6c480f7e7 and the previous production 5751c977b): pp2048 about 274-286 t/s,
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. On 2026-10-25
(medians, 6c480f7e7 / 2fc9f9c35): pp2048 274.5/273.8, d16384 200.5/200.4, d32768 157.6/157.6,
tg256@d4096 16.8/16.7. The corpus
harness and older numbers are in scripts/rx580-bench/README.md. A fresh run far outside these
means something else is wrong (another GPU user, thermal state, a bad build) before it means a
regression or a win.
Invariants that must hold. Never benchmark with llama-swap running (--perf and --moe-exact
refuse, and any other llama process blocks them). --perf gates prefill at -5% per depth on the
MEDIAN of all repetition samples of each build (never a mean: CPU contention produces single
catastrophic samples), and peak VRAM at 95% of the card; decode is reported, not gated (CPU-bound and
noisy here).
Failure modes and incidents. Upstream #28334 removed --mmap/--no-mmap/--mlock and
llama-bench's -mmp: the server refused to start and llama-bench silently produced no rows until
54a7f534b switched to -lm none. The perf and MoE-cache checks leave tens of GB of model files in page
cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024
deploy (see the runbook). That pod restart also emptied /tmp (every earlier fork-validate.*
scratch dir with its raw samples is gone) and removed gdb, which is not in the pod image; --arbiter
case d-term-in-release then falls back to polling the log.
Rebase hazards. CLI flag renames in common/arg.cpp and tools/llama-bench; log line texts the
script greps for (server entering sleeping state, transpose fast path, transposed ne0=,
no longer waiting for the GPU token, released .* MiB of MoE cache, MoE cache size =,
llama_moe_cache: .* hits =, sched_copy_experts: ids readback skipped, N/M tests passed,
Backend Vulkan0: OK, disk cache: restored, disk cache: copied/saved, prompts of this model,
size limit, removing, disk cache: failed to write); llama-bench's jsonl fields (n_prompt, n_gen, n_depth, samples_ts) and
markdown layout; the GGUF keys <arch>.expert_count / <arch>.expert_used_count; the DRM fdinfo
keys drm-client-id / drm-memory-vram.
Validation. Known gaps of the script: peak VRAM is recorded per arm, not per test. The first
candidate arm has now twice shown an unexplained spike of about 1.6 GiB over the other arms
(2026-10-25: ref 5476, cand 7062, cand 5767, ref 5476 MiB); the cause is open (per-test peaks would
locate it), and it stays far below the 95% gate. The SIGTERM-in-release breakpoint is at function
entry, not after the host-shadow capture. The v-vram-release SKIP path (production moving during
the measurement) has not been exercised. --moe-exact and the median-gated --perf first ran on the
GPU on 2026-10-25 (runbook section 6).
F10. Page-cache drop after a copying load
What and why. With -lm none the loader read()s the GGUF into its own buffers, but the kernel
keeps the file in page cache afterwards, charged to the pod's cgroup: about 20 GB after a 35B load.
The pod is BestEffort on a shared node, and on 2026-10-24 the Talos OOM controller killed it right
after a model load under exactly that pressure (MemFree < 1 GB, file cache about 20 GB). Once the
data is copied those pages are pure waste, so the loader advises the kernel to drop them.
Where. src/llama-mmap.h/.cpp: llama_file::drop_cache(offset, len).
src/llama-model-loader.cpp, llama_model_loader::load_all_data: the per-tensor drop at the end
of the copy (non-mapping) branch and the whole-file drop in the final cleanup
(size_done >= size_data). tools/mtmd/clip.cpp, clip_model_loader::load_tensors: whole-file drop
after the tensor reads. Commit cce0ac1f0.
How it works.
drop_cache()isposix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED)under#ifdef __linux__(no-op elsewhere);len0 means to the end of the file. Errors are logged at debug level only.load_all_data()drops each tensor's range right after it is read (all three copy paths: straight into a host buffer, the async pinned-upload chunks, the staging buffer), so the cache never grows past about one tensor during the load. The final cleanup drops every file whole, which catches the GGUF metadata and the partial pages at tensor edges that a range drop leaves.- Both run only when
mappings.empty(), i.e. the loader holds no mapping of any of its files: notmmap,mmap+mlockorauto, and no lazy tensors (-lzm; with lazy tensorsinit_mappings()maps every file even under-lm none). Split GGUFs are covered (every entry offiles). - clip.cpp always copies the encoder through an ifstream (no mmap path exists), so it opens the
file again after the reads and drops it whole, unless
no_alloc. - Choice: per-tensor plus a final drop rather than a post-load drop only, because the per-tensor
drop also removes the peak during the load (the incident window: anon and file cache grow together)
for three extra lines. It costs about 0.27 s more than a post-load-only drop on the test model
below. Direct IO (
-lm dio, upstream's) was measured as the alternative and is not better (section 4).
Invariants that must hold.
- Never drop for a file this loader maps: a mapped page would just be faulted back in (the kernel also skips pages that are mapped into a process, so a violation costs refaults, not correctness).
- Drop only after the bytes are in our own buffers. Nothing in the fork re-reads the GGUF after load: the VRAM release/restore (F4) uses host shadows, not the file.
- The drop is a hint: a failure must not fail the load.
Failure modes and incidents. None. Deployed 2026-10-25 (2fc9f9c35); on a 35B GPU load in
production the node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB. Costs to know
about:
- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a 5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few seconds on a 26 s load.
- A reload of the same model can no longer hit a warm page cache (a warm-cache reload of the test model took 4.6 s against 5.9 s cold). Under llama-swap a reload happens only after an eviction or a crash.
TENSOR_DUPLICATEDtensors read the same range twice; the second read now comes from disk (small tensors only).
Rebase hazards. The structure of load_all_data() (the from_mapping branch and its three copy
paths, the final cleanup block), init_mappings() creating mappings for lazy tensors, new load modes
in llama_load_mode (any new mode that maps must leave mappings non-empty), and the tensor-read
loop in clip.cpp load_tensors.
Validation. Not in fork-validate.sh. By hand on the pod with a scratch llama-server on a
scratch port, CPU only (-dev none -ngl 0 --no-mmproj-offload, no VRAM), cold cache (evict the test
files with python os.posix_fadvise(..., DONTNEED) first): page residency of the model and mmproj
files by mincore before and after the load (must be 0 after with -lm none, unchanged with
-lm mmap), the pod cgroup's file/anon in
/sys/fs/cgroup/kubepods/besteffort/pod<uid>/memory.stat, host MemFree, load time, and a greedy
request. First run (2026-10-25, gemma-4-E2B UD-Q4_K_XL 2.5 GiB + mmproj BF16 0.9 GiB, production
6c480f7e7 and the candidate with the drops disabled as controls, 4 rotated rounds):
| drops off (control) | post-load only | per-tensor + final (committed) | |
|---|---|---|---|
| model / mmproj resident after load | 2431-2444 / 940 MiB | 0 / 0 | 0 / 0 |
| peak model residency during load | 2.1-2.5 GiB | 2.1-2.4 GiB | 1.2-1.5 GiB (largest tensor) |
pod cgroup file after load |
+3.37 GiB | +0 | +0 |
| host MemFree after load | 0.6 GB | 2.5 GB | 2.5-2.8 GB |
| load time (median, production 5.85 s) | 5.87 s | 6.56 s | 6.82 s |
Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it counts clean page cache as
available. Greedy output byte-identical in all arms. -lm mmap control: the model file stays fully
resident and mapped (2498 MiB, file_mapped +2.5 GiB) after load and after a request, and only the
copied mmproj is dropped. -lzm on control: no drop (the file is mapped), 1252 MiB of copied
tensors stay cached. The GPU upload path and a 35B load were first exercised by the 2026-10-25
deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B
request took 34 s including load and answer).
F11. Disk prompt cache (save slots on shutdown, restore after a restart)
What and why. Only one large model fits in RAM. Switching models in OpenWebUI makes llama-swap
SIGTERM the previous one, and switching back re-prefills the whole conversation, because the RAM
prompt cache died with the process (F4's reason for the arbiter). A 30k-token chat costs about 150 s
on the 35B at about 200 t/s. With --cache-disk-path DIR, each slot with a prompt is saved to DIR on
shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the
file, so only the suffix is processed. Measured on Qwen3.5-0.8B CPU only: a 12k-token turn 2 went
from a 72 s prefill to 1.2 s, of which the read took 307 ms and the restore 18 ms. Off by default.
Upstream PR #28092 (--cache-disk) was the reference for ideas, not ported (1700 lines). It
livelocked the kernel on AMD + Vulkan by DMA-ing GPU state into a MAP_SHARED mmap'd file, and it never
saved the slot in use on shutdown, which is the main case here.
Where.
tools/server/server-task.h/.cpp:server_prompt_disk_entry,server_prompt_disk_cache(n_reuse,find,has,load,save,remove); file IO helpersdisk_cache_io,disk_cache_open.tools/server/server-context.cpp:DISK_CACHE_MIN_TOKENS,disk_cache_key,disk_cache_collect,disk_cache_flush,disk_cache_load; the init inload_modelafter the RAM cache; thedisk_cache_loadcall at the end ofget_available_slot;server_context::start_loop(collect,vram_share_exit, flush).common/arg.cpp,common/common.h:--cache-disk-path(cache_disk_path),--cache-disk-size(cache_disk_mib, default 4096, 0 = no limit).- Library (cold save):
llama_context_restore_memory_host()(include/llama.h,llama_context::restore_memory_host);llama_memory_i::restore_device_buffers(bool to_host)inllama-memory.h,llama-kv-cache,llama-memory-recurrent,-hybrid,-kv-cache-iswa; the helperllama_memory_wrap_host()inllama-memory.cpp. - Commits
dd851fd0c,2874dcb10,2c6c517ba(andf24c3582c, F4).
How it works.
- Save, after
start_loop()'s loop returns. Firstdisk_cache_collect()copies the state of every slot with at least 256 text tokens to host memory (llama_state_seq_get_data_ext, flags NONE) while the device buffers still exist. Thenvram_share_exit()frees the VRAM and releases the GPU token as before. Thendisk_cache_flush()writes the files. The token release therefore waits only for the device-to-host copy, never for disk IO. - A slot is skipped when its KV does not cover its token list (
seq_pos_max + 1 != n_tokens: the last batch was not decoded, a shutdown during a token wait), when it holds media, and when the same token list is already on disk. - Cold server: with
LLAMA_SLEEP_EVICT_KVthe KV and recurrent state live only in host shadows (F4) and their tensors have no buffer.llama_context_restore_memory_host()wraps each shadow as a CPU_Mapped buffer (ggml_backend_cpu_buffer_from_ptr) and points the tensors into it, so there is no copy, no allocation and no device use. It setsdevice_discarded. Without evict_kv the KV is still in VRAM and is read directly. - File,
<dir>/<key hash>-<token hash>.lpcd: a 48-byte header (magicLPCD, version, key hash, n_tokens, n_ckpt, payload size, payload hash), then 8-byte padded sections: tokens, checkpoint table, state size, state, the state of each checkpoint. The hash is FNV-1a over the 64-bit words of the payload. Write: plain fwrite to.tmp, fdatasync,POSIX_FADV_DONTNEED, rename. Read: plain fread, then DONTNEED. There is no mmap anywhere. - Index: at startup, from the headers, tokens and checkpoint tables of the files of this key. A bad
magic, a file size that differs from the header, or an inconsistent table means the file is removed.
Other keys and other versions are counted and left alone. A
.tmpolder than 1 h is removed. - Key: the build commit, the model path + size + mtime, LoRA adapters and control vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, K/V types, flash-attn type, and the RoPE/YaRN parameters.
- Restore: at the end of
get_available_slot(), after the RAM cache, only while warm, withcache_prompton and no media tokens. The file that reuses the most of the new prompt is loaded if it reuses at least 256 more tokens than the slot holds. The slot's prompt goes to the RAM cache first. Then the slot is cleared and the state is set; a failure clears the slot and removes the file. - Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt,
or, for a recurrent or SWA model, a saved checkpoint at or below the common prefix can be used. A
plain-KV model can be cut back to any prefix. Exact prefix alone would almost never fire: the Qwen
template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is not a
prefix of the next OpenWebUI turn. Checkpoints are exact snapshots, and
update_slots()rolls back to one exactly as it does for the RAM cache; the recurrent state itself is never cut back. Restored checkpoints getid_task = -1, so one that fails to load means a re-prefill, not an abort. - Size limit: before a write, the least recently used
*.lpcdfiles of every model in DIR are removed until the new file fits. A restore updates the file's mtime. Entries the new prompt extends are removed after the write and do not count against the limit. A write also needs its size + 1 GiB free on the file system, so the cache never fills the PVC. - Disabled with speculative decoding, with a warning: the draft state is not saved.
Invariants that must hold.
- Never restore while cold or without the GPU token (
disk_cache_loadreturns whenvram_cold).get_available_slot()runs on the loop thread after the wake, like the RAM cache'sprompt_save. - Copy the state before
vram_share_exit()(it discards the device buffers) and write after it (the token is not held for disk IO). - Never serve a file that fails the size or hash check. Never mmap the file: a DMA into a shared mapping is what livelocked upstream's version on AMD.
- A write is atomic (tmp + rename), and any IO error leaves the server serving and the exit clean.
- The key must cover everything that changes the state layout or the KV values. Add a new
state-changing parameter to
disk_cache_key(). llama_context_restore_memory_host()is for exit only; the wrapped buffers point into the shadows, which must outlive them (both are members of the same memory object).
Failure modes and incidents.
- The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself (CPU_REPACK, F4), not in the save.
- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full
KV allocation at exit (about 2.7 GB for the 35B at ctx 131072) on a node that has had an
OOM-controller kill. Replaced by the in-place wrap. The 32-byte alignment that
cpu_buffer_from_ptrasserts is met by rounding the base down; those leading bytes are never read. - Writes to
/tmp(node overlay) stalled once: 5.8 s for 64.5 MiB, against 171 ms normally. - Costs to know about. llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a child still there 30 s after SIGTERM (measured 2026-10-25 with a scratch
llama-swap and dummy upstreams: term to exit 4.0 s, next start 2 ms later; a child that ignored SIGTERM
was replaced at +30.0 s). So the exit write adds directly to a model switch: about 1 s per GB on the
/rootPVC (0.66 GB/s with fdatasync). A save cut off by the SIGKILL leaves only a.tmp. - Only slots are saved. A conversation that sits only in the RAM cache when the process stops is lost.
- Every deploy empties the cache (the key holds the build commit), and so does any change to the model file or the flags in the key.
Rebase hazards.
server_context::start_loopandvram_share_exitordering (F7);get_available_slotand the RAM cache (prompt_save,prompt_load,server_prompt_cache_state);server_promptandcommon_prompt_checkpointfields (new fields must be saved, or the format version bumped).- The checkpoint rollback in
update_slots(pos_min_thold,TAG_PROMPT_LOGITS), which decides how much of a restored file is reused. - The llama state format (
state_write/state_readof the memory types). The key holds the build commit, so a format change cannot load an old file; it only empties the cache. llama_memory_i::restore_device_buffersand the shadow layout of F4 (device_buffer_shadow):llama_memory_wrap_hostrelies on the compact iteration-order shadow.- Flag names in
common/arg.cpp.
Validation. fork-validate.sh --diskcache (2c6c517ba), beside production. The model is the small
hybrid, CPU only (-dev none -ngl 0), with chat requests and greedy decoding, and a reference server
that never restarts. Cases: restore, restore-arbiter and restore-cold (turn 1, SIGTERM, restart,
turn 2 restored; prompt_n equal to the reference; identical answer), no-prefix, key-ctx,
key-ctk, key-model, corrupt (truncated and byte-flipped), lru, write-error (RLIMIT_FSIZE,
SIGXFSZ ignored), log-scan, and gpu-warm/gpu-cold when 1536 MiB of VRAM is free. First run
(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219
MiB free).
| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill |
|---|---|---|---|---|---|---|
| 573-token chat | 64.5 MiB, 2 ckpt | 18 ms | 171 ms | 314 ms (token free 56 ms vs 28) | 166 + 3 ms | prompt_n 58 of 596, 462 vs 2913 ms |
| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms |
| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s |
Swapdown window (2026-10-25, started by the owner, /root/arms/diskcache-window.sh):
--diskcache --arbiter 29 PASS, 0 FAIL, GPU legs included.
35B (q35.py, production 35B flags with --cache-disk-path on a scratch dir, arbiter env, an
18760-token pan-tadeusz chat; log /root/arms/diskcache-test/q35.log):
| leg | result |
|---|---|
| turn 1 prefill (reference and candidate) | 18760 tokens in 83.4 s (224.8 t/s), outputs identical |
| warm save on SIGTERM (a) | 556.5 MiB (2 checkpoints), copy 323 ms, write 960 ms; token free in 328 ms (6 ms without the cache), exit 1952 ms (607 ms without), peak RSS +430 MiB |
| restore after the restart (b) | read 860 ms, restore 114 ms; turn 2 prompt_n 27 of 18850 in 3.0 s (the reference that never restarted: 27 in 2.0 s); output identical |
| cold save on SIGTERM (b, KV in the host shadow) | 746.7 MiB (5 checkpoints), copy 310 ms, write 1403 ms, exit 2928 ms, peak RSS +432 MiB: the in-place wrap adds no copy |
| restore of the cold save (c, turn 2 sent again) | read 1189 ms, restore 75 ms, prompt_n 4; output differs from the reference's turn 2 (see below) |
Two anomalies in leg c, both explained:
- Output != reference: c sent turn 2 again, so the saved list (turn 2 + 63 generated tokens) extends
the new prompt.
update_slots()then rolls back to the checkpoint at the prompt end minus 4 and processes 4 tokens, while the reference processed turn 2 from turn 1's state (27 tokens). These are different reuse points, not the same computation. The controlled test on Qwen3.5-0.8B (/root/arms/diskcache-test/cmp.py) compares the same reuse point. Server X never restarts and runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the GPU (-ngl 99); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU to itself:/root/arms/diskcache-window2.sh(about 6 minutes, under swapdown) is ready and not run. Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the GPU), not proven on that model. - Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
The driver's fake waiter was
flock -s want.lock sleep 120; killing flock leftsleepholding want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right after its load and again right after its request (log 0.37.50). That ordinary arbiter release recaptures the full KV allocation as a host shadow, about 2.6 GB at ctx 131072 (F4), and the SIGTERM arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The driver now holds the lock in the process it kills, likefork-validate.shdoes.
Recommended configuration (not applied): append --cache-disk-path /root/prompt-cache --cache-disk-size 4096 to the moe_offload_args and moe_offload_gemma macros. The directory is
created on first start. /root had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
applies. Models only pick up the flags when they restart.
4. Dead ends
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry without a new reason.
| idea | result | mechanism |
|---|---|---|
Weight pinning (GGML_CUDA_REGISTER_HOST and the host-register chain, from a third-party CUDA fork whose +64% was on an RTX 3060) |
+17% pp2048 in isolated llama-bench (252 -> 294), fails in serving | RADV cannot import file-backed mmap pages; the pre-stage fallback fails (ErrorOutOfDeviceMemory for the ~18 GB expert region) or doubles host RAM. -lm none serves faster (Qwen3.6-35B 278 -> 286). Dropped in the 2026-09-09 rebase. |
Expert-upload prefetch (GGML_SCHED_PREFETCH_EXPERTS) |
-20% pp (other runs -9% to -22%), no tg benefit | Slots allocate in host-pinned GTT, not VRAM, so it stages host to host and still pays the PCIe read. =1 means the default 3 slots. Dropped. |
GGML_VK_ASYNC_USE_TRANSFER_QUEUE=1 |
flat (-0.1%), byte-identical | GCN is excluded from prefers_transfer_queue and RADV exposes no transfer-only queue family here; the per-layer upload-then-compute order leaves nothing to overlap. A standalone Vulkan microbench did show 60-73% of an H2D copy on queue family 1 hidden behind compute, so double-buffered prefetch inside one backend context remains an untested idea (estimated +10-15%, unproven). |
Fractional -ncmoe |
no measured win | Config only uses integers (22/40/48); upstream refactored -ncmoe into llm_add_n_cpu_ffn_overrides. Dropped. |
MoE cache --moe-cache-mib 512 / 1536 / 2304 (Qwen3.6-35B, ncmoe 40, ctx 131072) |
tg256@d4096 -42% / +5% / +12%; prefill +1-3%; idle VRAM 6.15 / 7.19 / 7.96 GB vs 5.66 | Small caches thrash (8-of-256 routing over 40 layers) and add a per-layer ids readback; only ~2.3 GiB pays and leaves no VRAM for a long KV. Not enabled. |
| MTP speculative decoding, Qwen3.6-35B | decode -39% (n1), -37% (n2), -50% (n3), -44% (n4), best -19% (n2, p-min 0.75); prefill -3 to -6% | A spec step costs ~180 ms vs 52 ms for plain decode; a 2-token verify alone is only 1.4x a 1-token step, so ~100 ms/step is draft machinery. The nextn layer adds 1.25 GiB VRAM. Entries removed. |
| MTP, gemma-4-26B-A4B | decode -57.9% (n1), prefill pp4k -34% | Same mechanism (~210 ms vs ~50 ms per step); MTP needs --swa-full, which alone costs -10.5% decode, -32% pp4k and saturates VRAM. Entries removed. |
| MTP, gemma E4B | decode -10.9% | Acceptance 0.41-0.67. The +19.5% pp seen with it comes from --swa-full, not MTP. |
--parallel 1 on gemma |
no effect (-1.0%, +2.1%) | The Qwen gain is probably tied to its recurrent state being sized per sequence (unverified). |
-ub 4096 -b 4096 (Laguna) |
flat (90.0 vs 90.9) | Fewer expert uploads cancelled by intra-ubatch attention growth. |
mul_mat_id tile-padding theory (larger/odd -ub) |
predicted 2.00x waste and a 1.94x cliff; measured +3.8% and -5.6%, no sawtooth | Theory dead; kills the mul_mat_id tile-selection shader work (an unbuilt, unverified WIP exists on branch fable5/mmid-tile-wip, with an unresolved correctness concern in its row_ids fill). |
-b sweeps |
non-monotonic (2048/8192 high, 4096/12288 low) | -b 2048 is already at the top. |
-ncmoe below 40 on the 35B |
34: +2.9% pp but 7.23 GiB peak at tiny context; 28: better at short context, worse at depth (137 vs 170 t/s at d32k); <= 24 collapses (~122) | Resident experts are evicted by KV/activation growth; cannot survive a 128k context. |
-ncmoe 44 on Laguna |
+1.5% | Does not fit at 64k context. Keep 48. |
fp16 on Polaris (Mesa 26.1.6 + radv_enable_float16_gfx8=true, fp16: 1) |
pp2048 -4.7%, -5.3% at d16k, tg +4% | Hurts prefill (the priority). Mesa 26.1.6 alone: +3.3% prefill, below the 5% bar; adopting it needs a pod image change. The pod's Mesa 25.0.7 has no such option. |
| KV quant on Qwen (q4_0, q8_0) | f16 beats q4_0 by +40% pp at 64k; q8_0 170 -> 128 t/s at depth; q4_0 with upstream PR #30738 still -16% vs f16 at d32k (133.6 vs 159.4) | No int-dot/fp16 here, so dequant is pure compute; the hybrid model's KV is small anyway. Keep f16. (PR #30738 was +15% at d16k on gemma-26B with q4_0 KV; not merged upstream as of the 20261024 rebase.) |
-fa 0 |
5x slower at depth (170 -> 35 at d32k) | Keep flash attention on. |
FA scalar block_rows 16 -> 32 |
3.5x regression at d32k (135 -> 38) | ~42 KB shmem per workgroup at head_dim 256, one workgroup per SIMD, occupancy collapse. |
GGML_VK_DISABLE_GRAPH_OPTIMIZE=1 |
restores tg32 at n_ctx=32 (10.44 -> 15.61), nothing at real depth | Upstream #27812's view-alias fix makes KV views look dependent and collapses barrier-free batches; only visible when per-node work is tiny. It is a correctness fix; do not disable it in serving. |
--no-mmap/-lm none for Laguna-S 118B |
does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. |
Laguna --cache-reuse, --swa-full, -t 12, KV quant |
ruled out | --cache-reuse is force-disabled for iSWA when base and SWA sizes differ. |
CPU-only task model (-ngl 0) to avoid handoffs |
rejected | At --ctx-size 8192 titles could face 8k tokens of CPU prefill. The arbiter is the answer. |
-lm dio (direct IO) instead of the F10 page-cache drop |
load +0.9 s on 3.4 GiB (6.89 vs 5.99 s), the same cost as the drop; 175 MiB of the model file and all of the mmproj stay cached | Synchronous 64 MiB O_DIRECT chunks with no readahead; the GGUF metadata goes through a separate buffered FILE and clip.cpp has no direct-IO path. Measured CPU only; reads into pinned Vulkan buffers may fall back to buffered IO (EFAULT or EINVAL). Not adopted. |
The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256 tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.
Where the time goes (Laguna-S-2.1 118B IQ2_M, all experts on host, GGML_VK_PERF_LOGGER=1, depth 0,
22.65 s per 2048-token eval, 17.18 s of it GPU-busy, so about 24% is H2D stall; moved here from the
old README section): expert MUL_MAT_ID 52%, attention projections 24%, FLASH_ATTN_EXT 16%,
everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s while dense MUL_MAT q5_K/q6_K in
the same graph reaches 3096-3691, so closing that gap is the largest remaining opportunity on this
card, not the transfer path. On interleaved-SWA models the sliding-window layers keep a small KV
(n_kv pinned at n_swa * n_seq_max + n_ubatch), so they cost the same at any depth; on Laguna 36 of
48 layers are O(1) in depth and the whole high-context slowdown comes from the 12 full-attention
layers.
5. Configuration that matters
/root/config.yaml (llama-swap) and /root/supervisord.conf on the pod are the source of truth;
this section explains the choices. Values as of the 20261024 deploy.
Qwen 35B-A3B MoE entries (macro moe_offload_args, applied to all 35B-A3B variants):
--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1,
context 131072.
-lm none: load weights into RAM (upstream removed--no-mmap; the old spelling makes the server refuse to start). Faster in serving than mmap and recommended for--n-cpu-moe. Exception: Laguna-S, which must use mmap (GTT limit, section 4).-ub 2048 -b 2048: the biggest prefill lever. ub512 = 97 t/s, ub1024 = 159, ub2048 = 254 on the 35B; the per-eval expert upload is amortized over the ubatch.--n-cpu-moe 40: all experts on host. Lower values are faster at short context but run out of VRAM at long context (section 4). Measured idle VRAM at ctx 131072: 5.66 GB.--parallel 1: +22.9% decode on the Qwen 35B (15.5 -> 19.1 t/s), prefill +0.3-1.2%, +64 MiB VRAM. Without it the server creates 4 slots and a unified KV. Two concurrent requests queue on slot 0. With one slot a ubatch never holds more than one sequence, which the concat fast path'sne[2] == 1condition (F3) needs.-t 6: six physical cores; more threads hurt decode.-fa 1: essential (section 4).- f16 KV (no
-ctk/-ctv): section 4.
Other models:
- gemma-4-26B-A4B: macro
moe_offload_gemma, same as above but--n-cpu-moe 22(8 layers of experts resident; gemma's SWA keeps KV small at depth). ncmoe sweep pp2048: 30 = 262, 24 = 266, 18 = 272, 14 = 157 (VRAM collapse); 22 keeps headroom for 128k context (7.40 of 8 GB at 128k). Its KV cache is-ctk q4_0 -ctv q4_0, set in thegemma4_sampling/gemma4_nothink_samplingmacros (not inmoe_offload_gemma); note gemma has no--parallel 1(no measured gain there). - Small models that fit in VRAM (Qwen3.5-4B, gemma E4B/E2B and similar) use auto-fit
(
common_args); forcing experts to host would slow them. - The 4B task model runs with
--no-mmprojand ctx 8192, and is preloaded. - Laguna-S-2.1 118B IQ2_M (configured as
Laguna-S-2.1-IQ2_M, matrix varlag): mmap,--n-cpu-moe 48 -b 2048 -ub 2048 -fa 1 --parallel 1, ctx 65536. Its exclusivity set must allow the task model (lag & q4task); making it exclusive with the task model killed it on every title request and forced full re-prefills.--parallel 1frees 223 MiB of VRAM at 64k context there (4 auto slots size the SWA cachen_swa * n_seq_max + n_ubatch, 4096 cells instead of 2560).
Environment, set once for all of llama-swap in [program:llama-swap] environment= of
/root/supervisord.conf:
LLAMA_SLEEP_VRAM_ONLY=1: enables the arbiter and VRAM-only sleep (F6).LLAMA_VRAM_ARENA=/dev/shm/llama-vram: the shared arena. Every co-loaded GPU model must use it.LLAMA_SLEEP_EVICT_KV=1: cold models also free KV, recurrent state, scheduler and scratch (F4), so the warm model gets nearly the whole card. Costs a 2-3 s re-warm instead of ~0.6 s.LLAMA_MMPROJ_ONDEMAND=1: encoder in VRAM only during encodes (F5). No-op for models without an mmproj.- No
--sleep-idle-seconds: release is purely demand-driven via want.lock. GGML_CUDA_REGISTER_HOST=1may still be present from the pinning era; it is a no-op with-lm none.
Not set, on purpose: GGML_VK_DISABLE_GRAPH_OPTIMIZE, GGML_SCHED_PREFETCH_EXPERTS,
GGML_VK_ASYNC_USE_TRANSFER_QUEUE, GGML_VK_CONCAT_FASTPATH, --moe-cache-mib.
--cache-disk-path (F11) is not set yet: it is not deployed; the recommended flags are in F11.
Machine-level:
- GPU power cap fixed at 120 W by
/root/fan-control.sh(a separate supervisord program; re-asserted every second; default/max cap 145 W). The two GPU hangs of 2026-10-09 happened before the cap was fixed (it used to switch between 120 and 145 W by temperature); heavy runs since have been clean. - Kernel argument
amdgpu.lockup_timeout=0,120000,0,0(compute ring 120 s) via Talosmachine.install.extraKernelArgs. Without it long 64k-context submissions hit the compute-ring watchdog (ring comp_1.x.0 timeout). - Upstream changed the default
--portfrom 8080 to 9931; the config passes--port ${PORT}explicitly, so it is unaffected.
6. Measurement methodology and traps
- Cross-run variance is about 7%. llama-bench's within-run error bars understate cross-invocation variance by about 14x (the same config read 276.7 +/- 0.9 and 296.4 +/- 0.6 in two invocations). Put every comparison you draw a conclusion from inside one llama-bench invocation (comma-separated sweeps). Two builds cannot share an invocation, so interleave them over at least 3 rounds and compare medians. Treat any cross-build delta under about 5% as unproven.
- Alternate arm order. A fixed order gave the later slot about a 9% advantage and nearly produced a false regression finding. Use orders like ref, cand, cand, ref.
- Node CPU contention. The node runs about 60 pods; llama-bench once got 4.67 of 12 hardware
threads while asking for 6. With
--n-cpu-moe 40every MoE layer decodes on the CPU, so stolen threads map onto tg almost directly. Prefill is nearly immune (held +-0.3% while tg32 swung +-14%). Expect occasional catastrophic tg samples; use medians, never means. Historical tg figures taken under different load are not comparable. - Never use tg32. At the default depth (n_ctx 32) decode is launch/barrier bound and measures
command-stream overhead: two upstream commits read -36% at tg32 and were at parity at tg256 (and +7%
at d512). Use tg256 (or tg128) with an explicit depth (
-p 0 -n 256 -d 4096). tg is not monotonic in n_ctx on this box (n_ctx 256 measured slower than 32 and 544), so only compare same-depth cells. -b Vulkanvs-b Vulkan0.test-backend-ops -b Vulkanmatches nothing, runs 0 cases and still prints OK with exit 0. The backend isVulkan0. Always check the case count.- ubatch larger than the prompt never fills. With
-ub 2048use prompt lengths that are multiples of 2048. A partial trailing ubatch costs a full expert upload. -tbis not a llama-bench flag (server only). Any bad flag makes llama-bench print usage and exit, which looks like a run that produced no rows. Always check the row count.-ncmoeis llama-bench's spelling of--n-cpu-moe.- llama-bench prints its table only after all
-dvalues finish. No output does not mean hung. - Nothing else may use the GPU during a measurement; stop llama-swap (
bench.shand/root/bench/moe/swapdown.shdo this and restart it from a trap). - Check the build finished before benchmarking it (library mtime earlier than the run start). Never overlap a build with a big benchmark (host OOM).
- Prompt-cache reuse is
timings.prompt_n, nottokens_evaluated(which is always the full prompt length). - Per-process VRAM from
/proc/<pid>/fdinfo/*drm-memory-vrammust be deduplicated bydrm-client-id; summing all fds double-counts shared buffers.drm-memory-gttis system RAM, not VRAM. Whole-card usage:/sys/class/drm/card*/device/mem_info_vram_used.GGML_VK_MEMORY_LOGGER=1logs every Vulkan buffer alloc/free with a running device total to stderr. - The concat fast path is invisible to test-backend-ops (F3); validate it with the knob, perf
logger timing, or byte-identical greedy generation (temperature 0, top_k 1, fixed seed,
cache_prompt: false). - Do not run
llama-clidetached with closed stdin on the pod: it can loop forever printing prompts (it once wrote a 65 GB log). Use llama-bench-dor the server harness.
7. Known gaps and open risks
llama_context::restore_device()ignores theboolreturned byrestore_device_weights(),moe_cache->restore_device()andmemory->restore_device_buffers(). On a restore OOM,device_resident()stays false, decode retries the restore on every call, and a graph reading a null buffer would crash. Only the mmproj path handles a failed restore.- Memory types without a
release_device_buffersoverride keep their KV underevict_kv(F4). - Router mode can spin at shutdown (F7); not used here.
- The unexplained ~1.6 GiB peak-VRAM spike in the first candidate
--perfarm, seen twice (F9). - Not reported upstream: the
wait_until_no_sleep()lost wakeup and the sleep wait that ignored new tasks (both upstream bugs fixed in878d1338f), and the graph-optimize alias cost at tiny context. Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them. - F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate; its memory effect is verified in production.
- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the
35B, only on the 0.8B (CPU and GPU).
/root/arms/diskcache-window2.shis ready (about 6 minutes of downtime). - A SIGTERM during an arbiter release with
LLAMA_SLEEP_EVICT_KV(F4, not F11) pays that release's host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time (token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that shadow while it is cold. - F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB.