Files
llama.cpp/FORK.md
T

77 KiB

FORK.md - the RX 580 llama.cpp fork

Maintainer documentation for this private fork. Read it before you rebase, change, or deploy anything. The rebase and deploy procedure is in docs/fork-rebase-runbook.md, which also holds the operations reference (pod access, files, recovery).

Status at the time of writing (2026-10-25):

  • Branch fable5/vram-ondemand-rebase-20261024 on upstream 3f03b1772. The fork commits are git log --oneline origin/master..HEAD; that list includes documentation and script commits (this file among them), so count it rather than trusting a number written here. Section 2 maps every commit to a feature.
  • Production runs 6c480f7e7. Everything after it on the branch is validation tooling, env-gated A/B knobs, comment fixes, documentation and one loader behaviour change (F10, the page-cache drop); none of it was deployed. The knobs and the comment fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation.
  • Commit hashes below refer to that branch. They change on every rebase; find a commit again by its subject line. The commit messages are detailed and are the primary record of each mechanism and measurement; read them with git show.
  • Commit trailers (owner's convention): this fork is private and never contributes upstream. Agent-written commits end with Co-Authored-By: Claude <model> <noreply@anthropic.com> (some older ones say Assisted-by: Claude). AGENTS.md asks agents for Assisted-by: instead of Co-authored-by: and forbids AI-written commit messages and PR descriptions; those are upstream llama.cpp's contribution rules and do not apply to commits that stay in this fork. Anything meant for upstream must be written by the owner under those rules.

Contents

  1. Purpose and target
  2. Commit map
  3. Features
    • F1 MoE routing-ids readback skip
    • F2 GCN flash-attention mask_opt gate (and the shmem-staging note)
    • F3 Vulkan concat tiled-transpose fast path
    • F4 On-demand device residency (library side)
    • F5 On-demand mmproj (multimodal encoder)
    • F6 Cross-process VRAM arbiter (server side)
    • F7 Server shutdown path
    • F8 Cold-teardown guards
    • F9 Benchmark harness and validation script
    • F10 Page-cache drop after a copying load
  4. Dead ends
  5. Configuration that matters
  6. Measurement methodology and traps
  7. Known gaps and open risks

1. Purpose and target

The fork exists to serve large MoE models well on one specific machine:

  • GPU: AMD RX 580 8 GB (Polaris, GCN gfx803), RADV Vulkan driver. llama.cpp reports fp16: 0 (RADV does not expose shaderFloat16 on GFX8), no int-dot, no matrix cores, wave64. PCIe 3.0 x16. RADV exposes no transfer-only queue family on this card.
  • Host: Ryzen 5 3600 (6c/12t), 62 GB RAM, a Talos Kubernetes node shared with about 60 other pods (CPU contention matters for decode numbers, see section 6). The GPU runs at a fixed 120 W power cap.
  • Primary model: Qwen3.6-35B-A3B (unsloth UD-Q4_K_XL, 20.8 GiB), a hybrid MoE: 40 layers, 30 of them gated delta-net (recurrent), 10 full attention, head_dim 256, 256 experts with 8 used. All experts live in host RAM (--n-cpu-moe 40); only the backbone, KV, recurrent state and compute buffers sit in VRAM.
  • Serving: llama-swap under supervisord keeps several llama-server processes loaded at once (the 35B main models, a Qwen3.5-4B "task" model OpenWebUI calls for titles and tags, gemma-4-26B multimodal, others). They cannot all fit in 8 GB, so they time-share the GPU through the fork's VRAM arbiter (F4-F7) without restarting processes, which would throw away the RAM-only prompt cache.
  • Priority: prefill (prompt processing) speed first, decode second. Agentic use pushes large tool results through the model, so long-context prefill is what users wait on.

What the fork adds, in short:

  • three Polaris prefill wins (F1, F2, F3), all exact (token-identical to mainline);
  • on-demand release/restore of everything a model holds in VRAM, plus a cross-process arbiter that hands the GPU from one llama-server to another in well under a second for small models;
  • fixes to the server's sleep, wake and shutdown paths that the arbiter exposed;
  • a benchmark harness and a post-rebase validation script.

2. Commit map

Oldest first. "Feature" points at the section below.

commit subject feature
c72cb424e llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert F1
fb6a8a076 ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes F2
ff2a96d2d vulkan: record why flash-attn shmem staging stays off for GCN F2
3df719e1e ggml-vulkan: tiled transpose fast-path for concat with transposed source F3
481be8b28 server: on-demand VRAM sharing to time-share one GPU between models F4, F6
17165fe2a server: optionally evict the KV cache too (Phase 2 of VRAM sharing) F4
958257cd6 server: keep the sleep-wake path active when the VRAM arbiter is enabled F6
b9df854de server: coordinate model load with the VRAM arbiter to avoid load-time OOM F6
8d80f0673 llama: guard buffer iteration against released device buffers F8
2feb6aeb2 server: restore weights/KV on wake, not at decode, so KV eviction is safe F6
bf2c1a699 server: also free the compute-graph scheduler on KV eviction F4
9857a45f0 llama: evict recurrent/SSM state on device release F4
9b555f1aa ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc F4
ad27d4921 llama: free backend compute scratch on cold; guard memory_breakdown F4, F8
0a0f3ab6d mtmd: release/restore the encoder weights from VRAM on demand F4, F5
5e99ac198 server: on-demand mmproj - free encoder VRAM on the text path F5
b7c131219 docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends docs
5e429d4a9 ggml-vulkan: skip the concat transpose fast path below one tile F3
4c4486f8e scripts: add the RX 580 benchmark harness F9
878d1338f server: replace the VRAM doorbell with a want-lock F6
53f907771 llama: skip the compute-buffer size check when the context is cold F8
7ac2014a8 llama: release/restore the MoE cache with the device weights F4
54a7f534b scripts: rx580-bench - use -lm none, document the MoE cache option F9
11044b35e server: run the shutdown handler off the signal-interrupted thread F7
6c480f7e7 server: release the GPU token promptly on shutdown F7
e085c347d ggml-vulkan: env knob for the concat transpose fast path F3
149a8cb0a scripts: add fork-validate.sh, post-rebase validation of the fork F9
bfbaf15b2 docs: FORK.md and rebase runbook for the RX 580 fork docs
f5c0d8f28 server: fix stale VRAM-arbiter comments and the sleep log line F6
f664b0f20 llama: env knob for the routing-ids readback skip F1
62d442e7b scripts: rx580-bench - decode at tg256 depth 4096, not tg32 F9
2512290f9 scripts: fork-validate.sh - gate --perf on medians, not means F9
80e662a2a scripts: fork-validate.sh - A/B ordinary handoff latency against --ref F6, F9
ceab1a91f scripts: fork-validate.sh - check that a cold server gives its VRAM back F4, F9
e5fc474b8 scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip F1, F9
cce0ac1f0 llama: drop the model file's page cache after a copying load F10

Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code. Everything from e085c347d down is after the production commit 6c480f7e7.


3. Features

Every feature section uses the same headings: What and why; Where; How it works; Invariants that must hold; Failure modes and incidents; Rebase hazards; Validation.

F1. MoE routing-ids readback skip

What and why. With experts host-resident, each split uploads the experts it needs. To choose them, the expert-copy callback reads the routing ids back to the host. The ids are produced on the same GPU the upload goes to, so the readback forces a full pipeline flush: 47 per eval on a 48-layer MoE. At prefill batch sizes every expert is used anyway (2048 tokens x 10 experts over 256 experts), so the readback buys nothing. Measured on Laguna-S-2.1 118B IQ2_M, all experts on host (before it was ported to the current callback): pp2048 +4.7% at depth 32k, +0.9% at 16k, flat at depth 0, tg +3%.

Where. src/llama-context.cpp, llama_context::sched_copy_experts (registered with ggml_backend_sched_set_copy_callback in the scheduler setup). Commit c72cb424e.

How it works. When ggml_nelements(ids) >= 4 * n_expert, it skips ggml_backend_tensor_get_async + ggml_backend_synchronize, clears st.ids_data and sets st.used.assign(n_expert, true). The existing grouping then copies the whole expert tensor as one range. Below the threshold the upstream readback runs unchanged. For Qwen3.6-35B (256 experts, 8 used) the skip applies to any ubatch of 128 tokens or more; decode (8 ids) never reaches it.

Invariants that must hold.

  • Exactness: uploading an expert no id selects cannot change the result, because mul_mat_id only reads the rows the ids point at. Any upstream change that makes unselected experts matter (for example a kernel that reads every uploaded expert) breaks this argument.
  • With --moe-cache-mib on, the all-true set still goes through moe_cache->copy_experts, which reads the host slot maps (no device readback) and copies cached experts device-to-device. Still exact.
  • st.ids caching stays coherent: a repeat call with the same ids tensor reuses the all-true set, which is what the threshold would decide again; a different n_expert recomputes.
  • The small-batch MoE-cache slot path (llama_moe_cache::copy) has its own readback and is not touched.

Failure modes and incidents. None in production. Historically the change lived in ggml_backend_sched_compute_splits in ggml/src/ggml-backend.cpp; upstream #29943 (6753a033f) deleted that block and moved expert copying into llama_context::sched_copy_experts. The original cherry-pick therefore had nothing to apply to and was re-authored as c72cb424e. A -X theirs resolution would have resurrected the deleted scheduler block.

Rebase hazards.

  • Any change to sched_copy_experts or the st state struct (ids, used, ids_data).
  • Upstream 701fd25b3 (MUL_MAT_ID id -1 skips work) left a GGML_ASSERT(id >= 0 ...) in the readback branch; the skip branch reads no ids, so it is unaffected, but watch for upstream semantics around negative ids.
  • If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.

Validation. fork-validate.sh --moe-exact MODEL (needs the GPU to itself, run it in the --perf swapdown window with the 35B): reads expert_count/expert_used_count from the GGUF, computes the threshold ceil(4 * n_expert / n_expert_used) tokens per ubatch (128 on the 35B), and compares 64 greedy tokens after a prompt of about twice that, every expert on the host: candidate == reference (identical-ref), and with the LLAMA_MOE_IDS_SKIP knob (f664b0f20; =0 forces the readback, =log reports each skip on stderr) candidate == candidate with the skip forced off (identical-noskip, which proves the skip exact even when the reference build carries it) and the log shows skips, all at >= 4 * n_expert ids (skip-engaged). Not yet run on the GPU (added after the 20261024 deploy; only dry-run against stub servers). Also --ops (the MUL_MAT_ID kernel, not this callback), --perf (prefill regression) and --moecache (the cache path through this callback).

F2. GCN flash-attention mask_opt gate (and the shmem-staging note)

What and why. mask_opt skips fully-masked causal blocks and drops the per-block mask load+add on fully-visible blocks. Upstream disables it on AMD GCN for head sizes <= 256. On Polaris attention at depth is compute-bound on the softmax path (exp runs at about 1/4 VALU rate, no matrix cores), so skipping that work is a real win. Measured on Qwen3.5-35B-A3B (head_dim 256): pp2048 +1% at depth 0, +7.8% at 16k (176 -> 190), +12.3% at 32k (136 -> 153), growing with depth. Perplexity bit-identical (5.1668 on and off).

Where. ggml/src/ggml-vulkan/ggml-vulkan.cpp, ggml_vk_flash_attn, the use_mask_opt condition. Commit fb6a8a076. The shmem-staging note is a comment in get_fa_tuning_params_scalar (commit ff2a96d2d, no behaviour change).

How it works. The GCN clause HSK > 256 || HSV > 256 becomes HSK >= 256 || HSV >= 256. The existing large-mask conditions (nem1 >= 32, nem0 * nem1 > 32768, nem0 >= block_cols * 16) keep it off for decode.

Invariants that must hold.

  • Only GCN with head size >= 256 is affected. Do not lower the threshold: relaxing it below 256 was -18.5% at 16k on Laguna (head_dim 128).
  • Upstream's !use_sparse term must stay in the condition (sparse flash-attention came in with upstream fc82583e6).
  • shmem_staging stays NVIDIA-only. Measured on GCN at head size 128: -6.7% at 16k, -7.4% at 32k. The kvsh stride of D/4+1 dwords is 4 mod 32, an 8-way LDS bank conflict on wave64 (the +1 padding is tuned for warp32), and the extra shared memory cuts occupancy. If anyone retries it, fix the padding to +2/+3 first.

Failure modes and incidents. A -X theirs resolution in a trial rebase silently dropped !use_sparse from the condition. No production incident.

Rebase hazards. Any upstream rewrite of use_mask_opt, of the FA tuning parameters, or of architecture detection (vk_device_architecture::AMD_GCN). Upstream 733f57a99 added a path that copies strided f16/bf16 K/V into prealloc_x scratch for flash-attention on AMD non-RDNA4 (so GCN is included) when neq1 >= 256; measured at d32768 it adds about 68 MiB of VRAM and no visible long-context prefill loss, but it has no env toggle, so its speed effect could not be isolated.

Validation. fork-validate.sh --ops (FLASH_ATTN_EXT, 5332 cases on Vulkan0) and --perf (pp2048 at depth 16k and 32k is where this feature shows).

F3. Vulkan concat tiled-transpose fast path

What and why. Delta-net layers build their conv input with ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0). The generic concat shader reads the transposed source with an uncoalesced stride: about 5.6 ms for a 16 MB concat on the RX 580 against about 130 us of memory bandwidth. On Qwen3.6-35B the fast path alone is +14% pp8192 and accounts for more than all of the fork's prefill win at depth 0 (the rest of the fork costs about 1% there). In the validation run on Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 1945 us without.

Where. ggml/src/ggml-vulkan/ggml-vulkan.cpp: ggml_vk_concat (gate lambda src_transposed_2d and the s1_trans/s0_trans checks) and ggml_vk_concat_transpose_fastpath. The graph pattern that matters here comes from llm_build_delta_net_base::build_conv_state in src/models/delta-net-base.cpp (qwen35, qwen35moe, qwen3next). Commits 3df719e1e (fast path), 5e429d4a9 (decode gate), e085c347d (env knob).

Other graphs that build ggml_concat(state, ggml_transpose(x), 0) (found with git grep -n "ggml_concat.*ggml_transpose" src/models/ plus the two-line form in delta-net-base and lfm-base): qwen4exp.cpp (build_conv_state_at), bailingmoe3.cpp, clef.cpp, kimi-linear.cpp, kimi-k3.cpp, glm5-next.cpp, mamba-base.cpp (two sites, the Mamba/Mamba2 conv and the hybrids built on it), plamo2.cpp, and lfm-base.cpp (causal short conv; the transpose is assigned to bx first). The fast path takes any of them when the gate holds (transposed source a plain 2D transpose of a contiguous f32/f16 tensor, one sequence, ne[0] >= 32), so a change to the gate or the shader affects all of them. Only the delta-net path (Qwen3.5/3.6) has been measured and validated; the --concat check uses qwen35.

How it works. For a concat along dim 0 where one source is stored transposed (nb[1] == type_size, nb[0] == ne[1] * type_size) and the other source and dst are contiguous, it issues two dispatches: pipeline_cpy_32/_16 copies the contiguous source into its dst columns, and pipeline_cpy_transpose_32/_16 (the shared-memory 32x32 tiled transpose in copy_transpose.comp) writes the transposed source into the other columns, both with custom push constants and a dst offset. It ends with ggml_vk_sync_buffers. Everything else falls through to the generic path.

The gate requires: f32 or f16 with all three types equal (quantized and integer types stay generic); matching ne[1..3]; ne[2] == 1 && ne[3] == 1 on the transposed source (so it fires only with one sequence per ubatch, which --parallel 1 gives); the 16-bit doffset bound; and ne[0] >= 32 (TILE_DIM) on the transposed source.

The decode gate (5e429d4a9): at one token the transposed source is a single column, so there is no stride to fix, but the path still costs two dispatches and the unconditional ggml_vk_sync_buffers, a full pipeline barrier (the generic path emits none). That barrier, not the half-empty transpose, was nearly all of a -17% tg256 regression, because the path ran 30 times per decoded token on the 35B. The transposed source is qkv_mixed transposed, so its ne[0] is the ubatch token count: prefill keeps the fast path, decode and small speculative drafts take the generic one. Measured with the gate: decode back to parity (gated/clean 1.023), prefill win retained (gated/ungated 0.998, pp8192 +11% over clean), generation byte-identical.

The knob (e085c347d): GGML_VK_CONCAT_FASTPATH=0 forces the generic path, =log prints ggml_vk_concat: transpose fast path, transposed ne0=N to stderr for every fast-path dispatch. It is read once (function-local static). Not set in production.

Invariants that must hold.

  • The fast path fires at prefill and never at decode on a delta-net model.
  • Output is identical to the generic path (it is a pure copy/transpose).
  • The types_ok gate keeps quantized types off the path: the dispatches are raw f32/f16 element copies.

Failure modes and incidents.

  • The decode regression above shipped in the deployed fork before the gate existed. It looked tg-neutral on the old upstream only because tg32's +-14% spread hid a 17% effect (section 6).
  • If upstream changes build_conv_state (for example a ggml_cont on the transpose), the fast path silently stops firing. Nothing crashes; prefill just loses about 14%. Only the concat check or a perf A/B notices.

Rebase hazards.

  • Pipeline renames: upstream 733f57a99 renamed pipeline_cpy_f32_f32/pipeline_cpy_f16_f16 to pipeline_cpy_32/pipeline_cpy_16. The pick applied cleanly and did not compile.
  • Upstream f172be756 made ggml_vk_concat non-static (declared in ggml-vulkan-common.h) and split helpers into headers; the fast-path helper stays static.
  • Shape changes in build_conv_state / qkv_mixed ([ch, n_seq_tokens, n_seqs] reshaped from a contiguous build_lora_mm output), changes to copy_transpose.comp or copy.comp push constants, or to vk_op_unary_push_constants / get_misalign_bytes.
  • Upstream concat now handles quantized types; keep types_ok.
  • The stock test_concat cases in tests/test-backend-ops.cpp never produce a transposed source and always have ne[2]/ne[3] > 1, so test-backend-ops -o CONCAT does NOT exercise this path.

Validation. fork-validate.sh --concat (needs a build with the knob, i.e. this branch or --concat-build): counts fast-path dispatches in a prefill run (must be > 0, all with ne0 >= 32) and a decode run (must be 0 while CONCAT nodes did execute), then ties the unmodified candidate to the fast path by CONCAT timing (candidate about equal to knob-on and at least 2x faster than knob-off). Last run: 72 dispatches at prefill (all ne0=512), 0 over 594 CONCAT nodes at decode. Also --perf.

F4. On-demand device residency (library side)

What and why. Several always-loaded models must share 8 GB. Co-residence is impossible (the 35B at --n-cpu-moe 40 uses most of the card; the 4B task model needs about 2.5-3.3 GB). Restarting a process to free VRAM throws away the RAM-only prompt cache: on Laguna a repeat turn went from a 21,960 ms re-prefill to 225 ms once the process stayed alive across a task-model call. So a model must be able to drop everything it holds in VRAM and get it back without a reload.

Measured footprints:

  • 4B: warm 3327 MB, weights released 724 MB, restored and correct.
  • 4B at 32k ctx: cold weights-only 1750 MB (KV stays), cold with KV evicted 726 MB.
  • gemma-26B: warm 7147 MB, cold with weights+KV but scheduler kept 718 MB, cold with the scheduler freed too 51 MB.
  • A cold 4B before the encoder work held 644 MB that turned out to be its mmproj (F5).
  • Re-warm cost: about 0.6 s per cold-to-warm switch weights-only (tiny prompt: 1.98 s cold vs 1.40 s warm); 2-3 s with KV eviction (it adds the KV D2H/H2D and a scheduler reserve).

Where.

  • src/llama-model.cpp/.h: llama_model::release_device_weights(keep_shadow), restore_device_weights(), weights_resident(); per-buffer metadata llama_model::impl::weight_release (weight_release_slot), filled in load_tensors and load_tensors_hadamard.
  • src/llama-memory.h: llama_memory_i::release_device_buffers(keep_shadow) / restore_device_buffers() (default no-op). Implemented in src/llama-kv-cache.cpp (llama_kv_cache) and src/llama-memory-recurrent.cpp; llama_memory_hybrid and llama_kv_cache_iswa delegate to their children.
  • src/llama-moe-cache.cpp/.h: llama_moe_cache::release_device(), restore_device(), resident(), impl::invalidate().
  • ggml/include/ggml-backend.h, ggml/src/ggml-backend-impl.h, ggml/src/ggml-backend.cpp: optional ggml_backend_i::free_scratch and ggml_backend_free_scratch(); Vulkan implementation ggml_backend_vk_free_scratch in ggml-vulkan.cpp.
  • src/llama-context.cpp/.h: llama_context::release_device(evict_kv), restore_device(), discard_device(), device_resident(), members kv_device_evicted, device_discarded; the auto-restore at the top of llama_context::decode.
  • include/llama.h: llama_context_release_device(ctx, evict_kv), llama_context_restore_device, llama_context_discard_device, llama_context_weights_resident.
  • tools/mtmd/clip.cpp/.h, tools/mtmd/mtmd.cpp/.h: clip_release_device, clip_restore_device, clip_weights_resident, mtmd_release_device, mtmd_restore_device.
  • Commits 481be8b28, 17165fe2a, bf2c1a699, 9857a45f0, 9b555f1aa, ad27d4921, 0a0f3ab6d, 7ac2014a8, 6c480f7e7 (discard).

How it works. Tensor metadata (ggml_context structs) is durable; device buffers are ephemeral.

  • Weights: on release, for each releasable buffer, capture a host shadow once (weights are read-only), free the buffer, null every tensor's buffer/data. On restore, ggml_backend_alloc_ctx_tensors_from_buft reallocates an identical layout (re-pointing the tensors), the shadow is uploaded, usage is set to WEIGHTS. Only single-buffer, alloc-path, non-host buffers are releasable: the host-pointer path (mmap or lazy-mapped), no_alloc dummy buffers and host (Vulkan_Host, i.e. pinned RAM) buffers stay as they are.
  • KV and recurrent state: same pattern, but the shadow is re-captured on every release (the data is read-write) and dropped after restore.
  • Scheduler: with evict_kv, sched.reset() frees the worst-case compute buffer and sched_need_reserve = true; restore calls sched_reserve() (which goes through sched_new(), re-registering the expert-copy callback and upstream's realloc callback).
  • Backend scratch: with evict_kv, ggml_backend_free_scratch() on each backend. On Vulkan this discards the pending compute context, synchronizes, destroys prealloc_x, prealloc_y, prealloc_split_k, prealloc_add_rms_partials, sync_staging, and zeroes their sizes and the prealloc_y_last_* tracking. They are reallocated lazily on the next compute. Other backends leave the hook null (no-op).
  • MoE cache (--moe-cache-mib, upstream #29887): released in both modes. The banks are freed and the cache is emptied, not shadowed: every SLRU is rebuilt fresh, its slot_map[il] pointers are re-pointed at the host slot-map tensors, and the host slot maps are cleared to 0xff (every slot -1). On restore the banks are reallocated (which also re-inits their views), cleared to 0 and marked WEIGHTS; the cache refills from the host experts.
  • Multimodal encoder: clip_release_device / clip_restore_device use the weights pattern (shadow once); no-op for a CPU-backed encoder. mtmd_* wrap the vision and audio contexts.
  • llama_context::release_device(evict_kv): synchronize, release weights, release MoE cache, and if evict_kv: release memory buffers, free sched, free scratch. restore_device(): weights, MoE cache, memory buffers, sched_reserve() if needed, then ggml_backend_sched_synchronize so uploads finish before any compute. discard_device(): like release_device(true) with keep_shadow = false everywhere (no host copies), sets device_discarded; used only before exit.
  • decode() calls restore_device() if !device_resident(), as a safety net. The server restores explicitly on wake (F6), because some KV access happens before decode.

Invariants that must hold.

  • Shadows are stored compactly in stable tensor-iteration order, skipping view tensors (view_src != NULL), and restored with the same iteration. Never index a shadow by t->data - base.
  • weight_release stays parallel to pimpl->ctxs_bufs: every push to ctxs_bufs needs a matching push to weight_release (non-releasable slots use emplace_back()).
  • The caller synchronizes before any release; release_device does this itself.
  • Restore order: weights, MoE cache, KV/recurrent, scheduler reserve, scheduler synchronize. The MoE cache banks must be back before the scheduler is rebuilt.
  • After a MoE-cache release no slot map may point at an expert: banks come back zeroed, so a stale slot map silently reads zero weights.
  • Every iterator over ctxs_bufs (model, KV cache, recurrent) must tolerate a null buffer, and every use of sched must tolerate null while cold with evict_kv (see F8).
  • device_resident() covers weights and MoE cache; it gates the early returns, the decode auto-restore and llama_context_weights_resident().
  • restore_device() after discard_device() aborts (GGML_ASSERT(!device_discarded)).
  • Greedy output is byte-identical across any number of release/restore cycles, and the prompt cache survives (timings.prompt_n drops on a repeated prompt after a cycle).

Failure modes and incidents.

  • The first shadow implementation indexed by t->data - base and produced garbage after restore (layout and padding differ). Fixed by the compact iteration-order shadow.
  • Restoring KV lazily at decode crashed iSWA models (gemma) with GGML_ASSERT(buf != NULL && "tensor buffer not set"): update_slots() creates an SWA checkpoint that reads the KV via ggml_backend_tensor_get before decode. Fixed by restoring on wake (2feb6aeb2).
  • The recurrent state was left resident (default no-op), 201 MB on the 4B, until 9857a45f0.
  • The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed resident in a cold model until bf2c1a699 / ad27d4921.
  • The MoE cache was not released at all before 7ac2014a8. Reallocating its banks without invalidating would have produced silent wrong weights.
  • In the 20261024 rebase moe_cache_slru::init() turned out to reset only the probation segment (head[1], tail[1] and n_protected survive), so re-running it on a used cache corrupts the lists. invalidate() now assigns a fresh moe_cache_slru() before init().

Rebase hazards.

  • New push sites into ctxs_bufs (upstream's load_tensors_hadamard() added two in the 20261024 rebase; without matching weight_release entries release_device_weights() reads past the end).
  • Changes to buffer creation in load_tensors (the host_ptr_path condition, ctx_bufs/buf_ptrs bookkeeping; upstream replaced ctx_buf_maps.emplace_back(ctx, buf_map) with ctx_bufs.emplace_back(ctx, std::move(buf_ptrs)) in the 20261024 rebase).
  • The llama_memory_i interface. New memory types that do not override release_device_buffers keep their KV resident under evict_kv; at the time of writing that is llama_kv_cache_msa, _dsa, _dsa_iswa, _dsv4 and llama_memory_hybrid_iswa.
  • MoE-cache internals (groups, slru, slot_map, buf_host, devices). Upstream 33d6135cb changed LRU to SLRU (g.lru -> g.slru); the pick applied cleanly and did not compile.
  • llama_context members next to ours (upstream n_reallocs_same/diff, copy_experts_info), sched_reserve() semantics, and the decode() signature (now decode(const llama_batch_ext &)).
  • ggml_backend_i layout (free_scratch must stay the last member after graph_optimize; other backends rely on zero-init) and the Vulkan context's prealloc field set (compare with ggml_vk_cleanup).

Validation. fork-validate.sh --arbiter: b-greedy-identical (15 cycles each side equal to a no-handoff reference), b-kv-survives (prompt_n drops after evict/restore, output identical), c-image-after-restore. --moecache MODEL (needs a cache large enough to hit; 256 MiB on gemma-26B had 0 hits and proves nothing, 2048 MiB had 6651 hits). --arbiter v-vram-release / v-vram-restore check that VRAM really drops: a baseline of mem_info_vram_used before any scratch server, then server A warm and alone, forced cold by a fake waiter on want.lock: the card-wide delta must drop by >= 70% of what A added and come back after the next request. Production's own share is re-read from the DRM fdinfo of every process the script did not start; if it moved by > 64 MiB the check SKIPs. First run (2026-10-25, 0.8B with mmproj, ctx 8192, LLAMA_SLEEP_EVICT_KV=1): warm +546 MiB, cold +4 MiB, restored +546 MiB.

F5. On-demand mmproj (multimodal encoder)

What and why. A multimodal model's vision encoder sat in VRAM on every text request. With LLAMA_MMPROJ_ONDEMAND=1 the encoder lives in RAM and enters VRAM only for an encode. Measured on the 35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered correctly, text after the image correct.

Where. tools/server/server-context.cpp: encode_mtmd_chunk (upstream 9871df591 split the encode out of process_mtmd_chunk; encode_mtmd_chunk is called from process_mtmd_chunk and directly from the batched-embedding path in the slot update, so the restore covers both), the mtmd_release_device(mctx) after mtmd_init_from_file in load_model, vram_go_cold, vram_share_exit. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.

How it works. After load the encoder is released (shadow captured). Before mtmd_batch_encode, mtmd_restore_device(mctx); if that fails (out of VRAM) and on-demand is set, it releases the LLM backbone weights only (llama_context_release_device(ctx_tgt, false): read-only shadow, so a cheap free with no D2H, KV and prompt cache untouched) and retries. After the encode it releases the encoder FIRST, then restores the weights, so the peak stays within VRAM.

Invariants that must hold.

  • Free the encoder before re-uploading the weights.
  • The donor is the weights, never the KV (the KV is read-write and would need a D2H every time).
  • vram_ensure_warm() does not restore the encoder; only the encode path does.

Failure modes and incidents. Before this, a cold text model with an auto-downloaded mmproj (-hf pulls one when the repo ships it) held 644 MB that no release path touched. The text-only 4B task model now runs with --no-mmproj, which is the right fix for any model that never sees images.

Rebase hazards. Renames or restructuring of encode_mtmd_chunk and the mtmd init block in load_model; clip_ctx fields (buf, ctx_data, sched, no_alloc).

Validation. fork-validate.sh --arbiter case c-image-after-restore (server A runs with --mmproj and LLAMA_MMPROJ_ONDEMAND=1; a 128x128 red PPM must be answered "red" after A was cold, and text right after must equal the reference). Post-deploy: one image request to gemma-26B.

F6. Cross-process VRAM arbiter (server side)

What and why. Several llama-server processes under llama-swap share one GPU: exactly one is warm (holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production: 60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last fork-validate.sh run before the 20261024 deploy: 30 handoffs, median 680 ms, max 1231 ms (the 2026-10-25 run of the same build read 1050 ms, max 1562 ms: handoff latency moves with node load, which is why it is now A/B'd against the reference in one session).

Where. tools/server/server-context.cpp: vram_arena_open, vram_acquire_for_load, vram_share_init, vram_should_yield, vram_want_acquire, vram_want_release, vram_take_token, vram_ensure_warm, vram_go_cold, vram_share_exit, vram_share_shutdown, the VRAM-only branch of handle_sleeping_state, the vram_ensure_warm() call before decode in the slot update path, and server_res_generator (bypass_sleep). tools/server/server-queue.cpp/.h: on_should_yield, start_loop (should_sleep, idle tick, post-wake grace), wait_until_no_sleep, is_stopping. Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.

How it works.

  • Enabled by LLAMA_SLEEP_VRAM_ONLY (any value). The arena directory is LLAMA_VRAM_ARENA (default /dev/shm/llama-vram) with two files.
  • token.lock: flock(LOCK_EX) is the baton. Rule: a process has device buffers iff it holds it.
  • want.lock: a waiter holds it LOCK_SH while waiting and drops it once it owns the token. The holder probes it with flock(LOCK_EX | LOCK_NB) on a SECOND fd: failure means somebody is waiting. The probe needs its own open file description, because flock on the waiter's own fd would convert that lock instead of conflicting with it. The kernel drops a dead waiter's lock, so nothing goes stale.
  • Load: load_model() calls vram_acquire_for_load() before common_init_from_params(), so the current holder releases before this model uploads anything. The loaded model stays warm holding the token and serves its first request without a re-warm.
  • Yield: vram_share_init() registers vram_should_yield with queue_tasks.on_should_yield. start_loop()'s should_sleep() calls it on the loop thread, under mutex_tasks, only when the task queue is empty and update_slots has returned. While the callback is set the idle wait ticks every 10 ms instead of 1 s. After a wake the server holds the GPU for at least 100 ms (yield_grace_ms). Sleep runs the sleeping callbacks; in VRAM-only mode handle_sleeping_state(true) calls vram_go_cold() (release device, also the encoder; then flock(LOCK_UN)) instead of upstream's full unload.
  • Wake: a request calls wait_until_no_sleep(). handle_sleeping_state(false) calls vram_ensure_warm(): take the token (register on want.lock, then poll token.lock with LOCK_EX | LOCK_NB every 2 ms, aborting when queue_tasks.is_stopping()), restore ctx_tgt and ctx_dft, and only then set vram_cold = false. The decode path calls vram_ensure_warm() again (no-op when warm) and throws "server is shutting down" if it returns false.
  • server_res_generator no longer bypasses the wake path when --sleep-idle-seconds is unset if LLAMA_SLEEP_VRAM_ONLY is set, because the arbiter can put a server to sleep without idle-sleep.
  • Queue fixes: wait_until_no_sleep() loops while (sleeping && !stopping) and re-asserts req_stop_sleeping each time (with notify_all); the sleep wait also wakes on a non-empty task queue; the idle wait_for only leaves the loop on a real task or termination.

Invariants that must hold.

  • Resident iff holding token.lock; vram_cold == false means both. vram_ensure_warm() sets vram_cold = false only AFTER the restore. What that order protects today, reasoned from the code: the only thing that gives the token away is vram_should_yield(), called from should_sleep() on the loop thread. vram_ensure_warm() also runs only on the loop thread, synchronously: from the wake callback (handle_sleeping_state(false), called inside start_loop() with mutex_tasks held) and before decode in the slot update path (outside yield_to_queue()). So should_sleep() cannot run between the token acquire and the end of the restore, whichever order the two lines are in. The measured failure (attempt 1 below: handoffs of 11m20s and 3m57s) happened under the retired doorbell design, where a separate warden thread read the cold flag to decide whether to honour or discard rings, so an early false let it accept stale rings during the restore. Under the inline want-lock probe the order is defensive: it keeps the flag's meaning exact for any future reader off the loop thread, and keeps a release out of a half-restored model if the restore or the probe ever moves (for example into yield_to_queue()). Either way, do not change it without an A/B of ordinary handoff latency (fork-validate.sh --arbiter, a-handoff-latency).
  • vram_acquire_for_load() sets vram_cold = false as soon as it holds the token, before the upload. Nothing reads the flag during the load today (the loop is not running yet and on_should_yield is only registered in vram_share_init(), which sets it again), so the early set is redundant now. It records "holding the token" at the moment it becomes true, because the opposite error is the dangerous one: a model that is warm and holds the token while vram_cold says cold never yields (vram_should_yield() returns false for a cold model), which strands every waiter, and its next wake would try to restore weights that were never released.
  • Release only happens from the idle branch of start_loop(), on the loop thread, never inside yield_to_queue() or while a slot is generating: a holder finishes its current generation before it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both).
  • The yield decision reads live kernel state at the point of use. Do not route it through a flag set by another thread (see attempts 3-4 below).
  • Every exit from vram_take_token() drops want.lock, or the holder yields forever.
  • The probe fd is separate from the waiter fd.
  • flock calls retry on EINTR: the server's signal handlers are installed without SA_RESTART.
  • Restore on wake, before update_slots(), so KV eviction is safe for SWA checkpoints.
  • Every model that can be warm on the GPU at the same time must use the same arena. A non-arbiter process (a benchmark, a manually started server) can OOM the warm model. In production the env is set for all of llama-swap, so all its children participate.

Failure modes and incidents.

  • Wake-path hang (958257cd6): with no --sleep-idle-seconds, bypass_sleep skipped wait_until_no_sleep(), so a server slept by the arbiter never woke and requests hung.
  • Load-time OOM (b9df854de): weights were uploaded before the arbiter was active; loading gemma while the 4B held VRAM could exceed 8 GB. After the fix the peak was 7771 MB.
  • The doorbell design (replaced by 878d1338f) was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the holder's warden thread turned the inotify event into a yield flag. Rings were lost in three windows: while the holder restored (it advertised cold and discarded foreign rings), while the holder loaded (the inotify watch did not exist yet, and the kernel does not queue events for a missing watch), and when a cold waiter consumed a third process's ring. Every loss was permanent: flock(LOCK_EX) had no timeout and production had no idle-sleep. The blocked flock held mutex_tasks, so the whole server stopped answering while /health still returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by the second day).
  • Failed fix attempt 1 (doorbell design): moving vram_cold = false before the restore made the holder's warden honour stale rings and hand the token back 2-5 s after taking it without serving. Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. See the first invariant for what this means under the current design.
  • Failed fix attempt 2: a re-ring loop (LOCK_NB + re-ring every ~1 s). Queued rings from an already satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become level-triggered.
  • Failed attempts 3-4: want.lock was sound (mutual exclusion, kill -9 safety, load-window coverage), but the warden thread still delivered its decision as a flag. start_loop() holds mutex_tasks from should_sleep() through the callbacks to the wait, so the warden's request_yield() was always admitted after the flag had been consumed, latching a release for the next wake (a server woke and re-slept 138 ms later without serving). It also exposed the upstream lost wakeup in wait_until_no_sleep(). Result: 0.494 s median, 1/60 stranded. The fix deleted the warden and request_yield/yield_requested and probes inline in should_sleep().
  • Measured deviations that are required: the 10 ms idle tick (about 0.2% of one core per warm server); a waiter that polls rather than retrying on a coarse grid (a 200 ms retry grid cost 0.503 s against 0.336 s, because the holder only releases after the waiter asks); the 100 ms post-wake grace (without it a permanent waiter caused a livelock: one request took 23.15 s over 138 sleep/wake cycles).
  • Process lesson from attempts 1-2: validate an ordinary handoff latency A/B, not only the pathological case. Two production deploys were burned before that rule existed.
  • Lost diagnostic: the old periodic "still waiting for the GPU token after N s" warning is gone; a stuck token now shows one VRAM arbiter: waiting for the GPU token line and then silence. Check /proc/locks (see the runbook).

Rebase hazards. This is the most fragile area, because it sits inside upstream's server loop.

  • server_queue::start_loop, wait_until_no_sleep, process_new_tasks, yield_to_queue and the sleeping-callback vector. Upstream refactored sleep (947fd9bb2) and made decode run inside yield_to_queue (22b8e310b); re-check after any such change that a release still cannot run under a live graph.
  • handle_sleeping_state, load_model ordering, the slot update path where decode happens, server_res_generator's bypass_sleep, renames in server-context.cpp.
  • Anything that touches the KV in update_slots() before decode.

Validation. fork-validate.sh --arbiter with two real servers on a scratch arena: a-handoffs (30 alternating, 0 stranded, 0 non-200, the server that answered holds token.lock), a-handoff-latency (fresh server pairs of --build and --ref in the same session, blocks ref, cand, cand, ref x 16 handoffs; FAIL if the candidate median exceeds the reference median by more than max(25%, 200 ms); first run 2026-10-25, reb1024 against itself beside production: 1050.5 vs 1082.5 ms), v-vram-* (F4), b-*, c-*, g-kill9-waiter (kill -9 a waiter registered on want.lock while the holder streams: want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in F7 and a log scan for asserts and Vulkan errors. In production, /proc/locks must show the warm model holding WRITE on token.lock.

F7. Server shutdown path

What and why. llama-swap evicts models with SIGTERM, so shutdown happens all the time, often while the arbiter is mid-release or mid-wait. Two bugs made shutdown keep the GPU token and stall every other model.

Where. tools/server/server.cpp: signal_handler, shutdown_pipe, struct shutdown_thread, sig_thread in llama_server. tools/server/server-context.cpp: server_context::start_loop (calls vram_share_exit() after the loop returns), server_context::terminate (terminates the result queue first), vram_share_exit, the destructor's vram_share_shutdown(). tools/server/server-queue.cpp: server_queue::terminate (sets atomic stopping before taking the mutex), process_new_tasks, server_response::recv_with_timeout, server_response::terminate, server_response_reader::next. Commits 11044b35e, 6c480f7e7.

How it works.

  • The signal handler only writes the signal number to a self-pipe (async-signal-safe); a dedicated thread reads it and runs shutdown_handler. The pipe fds are FD_CLOEXEC. If pipe() fails the old direct call is used. The thread object is declared after the server objects, so it is joined (a 0 byte unblocks it) before they are destroyed. A second signal still calls exit(1).
  • terminate() sets stopping (atomic) before locking mutex_tasks, then stops the result queue so HTTP handlers waiting for results get "server is shutting down" (HTTP 503, or an SSE error event); recv_with_timeout checks !running before waiting and returns null.
  • A token wait polls with LOCK_NB every 2 ms and aborts on is_stopping(); the aborted wake stays cold, skips the restore, and the loop exits before update_slots.
  • After start_loop() returns, vram_share_exit() frees all device buffers with llama_context_discard_device() (no host shadows), releases the encoder, and only then unlocks token.lock.
  • What llama-swap (v239 on the pod) does when a child exits on its own (crash, OOM kill, external signal), verified 2026-10-25 with a scratch llama-swap instance and a CPU-only child, for both kill -9 and an external SIGTERM: it logs [WARN] group: running <model> exited: [<model>] upstream exited unexpectedly, does not restart it eagerly (no new process in the 2 s after the kill), and starts a new process on the next request for that model (new PID, health check, HTTP 200). The production log since the last pod restart has no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers the restart pays a full model load (about 26 s for the 35B with -lm none), and under the arbiter the new process takes the token through vram_acquire_for_load() like any load. A process that wedges instead of exiting (the incidents below) is never restarted: /health keeps answering.

Invariants that must hold.

  • Nothing that takes mutex_tasks runs in signal context.
  • The token is released only after this process's VRAM is free (the next model may start uploading the moment the lock drops).
  • A shutting-down server never restores weights.
  • SIGTERM in any state exits 0 within about 2 s and frees the token.

Failure modes and incidents.

  • SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside a sleeping callback (under mutex_tasks, during a release); the handler's terminate() locked mutex_tasks again on the same thread. The process wedged holding token.lock, every other model queued behind it, and only SIGKILL worked. Reproduced 6/6 off production; 8/8 clean with the fix. The first suspicion (the multimodal path) was wrong. gdb on the wedged process showed the main thread in futex(mutex_tasks) with the yield_to_queue worker idle.
  • Token kept on shutdown: with a request in flight the HTTP handler waited forever for results, main blocked joining it, and the token was released only in the destructor after that join; with a token wait in progress the blocking flock held mutex_tasks; and the old destructor unlocked the token BEFORE destroy(), so the next model could upload into still-occupied VRAM.
  • After the fix: off production SIGTERM mid-generation or mid-wait exits in about 0.2 s with exit 0 and frees the token in 5-70 ms; in production about 0.7-1 s.

Rebase hazards. server.cpp's signal setup and llama_server object lifetimes; server_response and server_response_reader (upstream 4e201388c switched recv_with_timeout to a fixed deadline: keep the order "check !running, return null, then wait_until(lock, deadline)"); server_context::start_loop/terminate. Router mode (server-models.cpp, server_models_sse_client::next) only checks should_stop() and can spin at shutdown now that recv_with_timeout returns null at once; llama-swap does not use router mode.

Validation. fork-validate.sh --arbiter: d-term-in-release (gdb breakpoint on llama_context_release_device, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and the token), e-term-waiting (exit 0 within 1.5 s, never took the token), f-term-streaming (exit 0, SSE error event, token free within 2.5 s), h-shutdown-warm, h-shutdown-cold. Last run: 736 ms, 200 ms, 93 ms (token free in 28 ms), 80/157/156 ms. Post-deploy: a forced llama-swap eviction must log VRAM arbiter: loop stopped, device memory freed, releasing the GPU token and exit by itself.

F8. Cold-teardown guards

What and why. A cold model has null buffers in its ctxs_bufs and, with evict_kv, a null scheduler. Upstream code never had to consider either state.

Where. src/llama-context.cpp: llama_context::~llama_context (sched && in the compute-buffer size check), llama_context::memory_breakdown (if (sched) around the compute part; upstream's if (moe_cache) breakdown block stays outside it). Null-buffer skips in llama_model::memory_breakdown, llama_kv_cache::{clear, memory_breakdown, total_size}, llama_memory_recurrent::{clear, memory_breakdown, total_size}. Commits 8d80f0673, ad27d4921, 53f907771.

How it works. Guards at the call sites. The asserts in ggml (GGML_ASSERT(buffer), GGML_ASSERT(sched)) are correct and stay; every other caller reserves the scheduler first.

Invariants that must hold. A server shut down while cold exits 0. Any new upstream code that iterates ctxs_bufs or calls into sched on a path reachable while cold (shutdown, memory breakdown, stats) needs the same guard.

Failure modes and incidents.

  • GGML_ASSERT(buffer) on shutdown of a cold model (seen 7 times in production logs) from memory-breakdown/size/clear on released buffers.
  • GGML_ASSERT(sched) in memory_breakdown for a cold evict_kv model being torn down.
  • GGML_ASSERT(sched) in the destructor's size check: every cold shutdown exited 134 instead of 0, so under llama-swap an ordinary stop looked like a crashed child, left a backtrace in the log, and the rest of the destructor never ran. Repro: arbiter env on, one request, force a yield with flock -s <arena>/want.lock -c 'sleep 6', then SIGINT while cold.

Rebase hazards. New iteration over ctxs_bufs, new sched users in teardown or breakdown paths. Watch item: llama_kv_cache::update() takes lctx->get_sched(), which can be null; it is safe today only because its single caller runs after sched_reserve().

Validation. fork-validate.sh --arbiter: h-shutdown-cold (exit 0 while the other server holds the token) and log-scan; --moecache shutdown.

F9. Benchmark harness and validation script

What and why. The measurement method on this box had to be rediscovered several times (section 6). The harness encodes it; the validation script encodes the checks that each past incident required, so a rebase can be validated by someone with no context.

Where. scripts/rx580-bench/ (run.sh workstation wrapper; bench.sh pod orchestrator; ppbench.py corpus prefill through llama-server; lbsweep.sh llama-bench sweep; summarize.py; README.md with methodology and reference values). scripts/fork-validate.sh. Commits 4c4486f8e, 54a7f534b, 149a8cb0a, 62d442e7b, 2512290f9, 80e662a2a, ceab1a91f, e5fc474b8.

How it works.

  • rx580-bench: stops llama-swap, waits for an idle GPU, runs fixed slices of pan-tadeusz.txt (4096/16384/32768 tokens, byte-identical across builds) through /completion with cache_prompt: false, plus a llama-bench sweep (prefill pp2048/pp8192 at depth 0, decode tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and rounds; restarts llama-swap from an EXIT trap. Results append to /root/bench/results.txt.
  • fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL] [--moe-exact MODEL]: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under /tmp, a scratch arena (refuses /dev/shm/llama-vram*), ports 8091-8093, unsets inherited GGML_*/LLAMA_* env, kills only PIDs it started (PID-reuse safe). Default with no check flag is --ops --concat --arbiter. --perf-report TSV re-aggregates a saved perf.tsv offline. See the header of the script for what each check proves.
  • Which phase needs what: --ops --concat --arbiter run beside production (--arbiter refuses below 1.5 GiB free VRAM); --perf, --moe-exact and --moecache need llama-swap stopped (the first two refuse otherwise; --moecache does not check, but loads a MoE model and must not share the card either).

Reference values (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, --perf on 2026-10-24, production 6c480f7e7 and the previous production 5751c977b): pp2048 about 274-286 t/s, pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. The corpus harness and older numbers are in scripts/rx580-bench/README.md. A fresh run far outside these means something else is wrong (another GPU user, thermal state, a bad build) before it means a regression or a win.

Invariants that must hold. Never benchmark with llama-swap running (--perf and --moe-exact refuse, and any other llama process blocks them). --perf gates prefill at -5% per depth on the MEDIAN of all repetition samples of each build (never a mean: CPU contention produces single catastrophic samples), and peak VRAM at 95% of the card; decode is reported, not gated (CPU-bound and noisy here).

Failure modes and incidents. Upstream #28334 removed --mmap/--no-mmap/--mlock and llama-bench's -mmp: the server refused to start and llama-bench silently produced no rows until 54a7f534b switched to -lm none. The perf and MoE-cache checks leave tens of GB of model files in page cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024 deploy (see the runbook). That pod restart also emptied /tmp (every earlier fork-validate.* scratch dir with its raw samples is gone) and removed gdb, which is not in the pod image; --arbiter case d-term-in-release then falls back to polling the log.

Rebase hazards. CLI flag renames in common/arg.cpp and tools/llama-bench; log line texts the script greps for (server entering sleeping state, transpose fast path, transposed ne0=, no longer waiting for the GPU token, released .* MiB of MoE cache, MoE cache size =, llama_moe_cache: .* hits =, sched_copy_experts: ids readback skipped, N/M tests passed, Backend Vulkan0: OK); llama-bench's jsonl fields (n_prompt, n_gen, n_depth, samples_ts) and markdown layout; the GGUF keys <arch>.expert_count / <arch>.expert_used_count; the DRM fdinfo keys drm-client-id / drm-memory-vram.

Validation. Known gaps of the script: peak VRAM is recorded per arm, not per test (one candidate arm once showed an unexplained 1.6 GiB spike that did not repeat); the SIGTERM-in-release breakpoint is at function entry, not after the host-shadow capture; --moe-exact and the median-gated --perf have not run on the GPU yet (tested offline only, see the runbook); the v-vram-release SKIP path (production moving during the measurement) has not been exercised.

F10. Page-cache drop after a copying load

What and why. With -lm none the loader read()s the GGUF into its own buffers, but the kernel keeps the file in page cache afterwards, charged to the pod's cgroup: about 20 GB after a 35B load. The pod is BestEffort on a shared node, and on 2026-10-24 the Talos OOM controller killed it right after a model load under exactly that pressure (MemFree < 1 GB, file cache about 20 GB). Once the data is copied those pages are pure waste, so the loader advises the kernel to drop them.

Where. src/llama-mmap.h/.cpp: llama_file::drop_cache(offset, len). src/llama-model-loader.cpp, llama_model_loader::load_all_data: the per-tensor drop at the end of the copy (non-mapping) branch and the whole-file drop in the final cleanup (size_done >= size_data). tools/mtmd/clip.cpp, clip_model_loader::load_tensors: whole-file drop after the tensor reads. Commit cce0ac1f0.

How it works.

  • drop_cache() is posix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED) under #ifdef __linux__ (no-op elsewhere); len 0 means to the end of the file. Errors are logged at debug level only.
  • load_all_data() drops each tensor's range right after it is read (all three copy paths: straight into a host buffer, the async pinned-upload chunks, the staging buffer), so the cache never grows past about one tensor during the load. The final cleanup drops every file whole, which catches the GGUF metadata and the partial pages at tensor edges that a range drop leaves.
  • Both run only when mappings.empty(), i.e. the loader holds no mapping of any of its files: not mmap, mmap+mlock or auto, and no lazy tensors (-lzm; with lazy tensors init_mappings() maps every file even under -lm none). Split GGUFs are covered (every entry of files).
  • clip.cpp always copies the encoder through an ifstream (no mmap path exists), so it opens the file again after the reads and drops it whole, unless no_alloc.
  • Choice: per-tensor plus a final drop rather than a post-load drop only, because the per-tensor drop also removes the peak during the load (the incident window: anon and file cache grow together) for three extra lines. It costs about 0.27 s more than a post-load-only drop on the test model below. Direct IO (-lm dio, upstream's) was measured as the alternative and is not better (section 4).

Invariants that must hold.

  • Never drop for a file this loader maps: a mapped page would just be faulted back in (the kernel also skips pages that are mapped into a process, so a violation costs refaults, not correctness).
  • Drop only after the bytes are in our own buffers. Nothing in the fork re-reads the GGUF after load: the VRAM release/restore (F4) uses host shadows, not the file.
  • The drop is a hint: a failure must not fail the load.

Failure modes and incidents. None yet (not deployed). Costs to know about:

  • Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a 5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few seconds on a 26 s load.
  • A reload of the same model can no longer hit a warm page cache (a warm-cache reload of the test model took 4.6 s against 5.9 s cold). Under llama-swap a reload happens only after an eviction or a crash.
  • TENSOR_DUPLICATED tensors read the same range twice; the second read now comes from disk (small tensors only).

Rebase hazards. The structure of load_all_data() (the from_mapping branch and its three copy paths, the final cleanup block), init_mappings() creating mappings for lazy tensors, new load modes in llama_load_mode (any new mode that maps must leave mappings non-empty), and the tensor-read loop in clip.cpp load_tensors.

Validation. Not in fork-validate.sh. By hand on the pod with a scratch llama-server on a scratch port, CPU only (-dev none -ngl 0 --no-mmproj-offload, no VRAM), cold cache (evict the test files with python os.posix_fadvise(..., DONTNEED) first): page residency of the model and mmproj files by mincore before and after the load (must be 0 after with -lm none, unchanged with -lm mmap), the pod cgroup's file/anon in /sys/fs/cgroup/kubepods/besteffort/pod<uid>/memory.stat, host MemFree, load time, and a greedy request. First run (2026-10-25, gemma-4-E2B UD-Q4_K_XL 2.5 GiB + mmproj BF16 0.9 GiB, production 6c480f7e7 and the candidate with the drops disabled as controls, 4 rotated rounds):

drops off (control) post-load only per-tensor + final (committed)
model / mmproj resident after load 2431-2444 / 940 MiB 0 / 0 0 / 0
peak model residency during load 2.1-2.5 GiB 2.1-2.4 GiB 1.2-1.5 GiB (largest tensor)
pod cgroup file after load +3.37 GiB +0 +0
host MemFree after load 0.6 GB 2.5 GB 2.5-2.8 GB
load time (median, production 5.85 s) 5.87 s 6.56 s 6.82 s

Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it counts clean page cache as available. Greedy output byte-identical in all arms. -lm mmap control: the model file stays fully resident and mapped (2498 MiB, file_mapped +2.5 GiB) after load and after a request, and only the copied mmproj is dropped. -lzm on control: no drop (the file is mapped), 1252 MiB of copied tensors stay cached. Not run: the GPU upload path (the per-tensor drop sits after all three copy paths, so it is covered by construction, but the pinned async upload was not exercised: the card had only 1.2 GiB free beside production), and a 35B load.


4. Dead ends

Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry without a new reason.

idea result mechanism
Weight pinning (GGML_CUDA_REGISTER_HOST and the host-register chain, from a third-party CUDA fork whose +64% was on an RTX 3060) +17% pp2048 in isolated llama-bench (252 -> 294), fails in serving RADV cannot import file-backed mmap pages; the pre-stage fallback fails (ErrorOutOfDeviceMemory for the ~18 GB expert region) or doubles host RAM. -lm none serves faster (Qwen3.6-35B 278 -> 286). Dropped in the 2026-09-09 rebase.
Expert-upload prefetch (GGML_SCHED_PREFETCH_EXPERTS) -20% pp (other runs -9% to -22%), no tg benefit Slots allocate in host-pinned GTT, not VRAM, so it stages host to host and still pays the PCIe read. =1 means the default 3 slots. Dropped.
GGML_VK_ASYNC_USE_TRANSFER_QUEUE=1 flat (-0.1%), byte-identical GCN is excluded from prefers_transfer_queue and RADV exposes no transfer-only queue family here; the per-layer upload-then-compute order leaves nothing to overlap. A standalone Vulkan microbench did show 60-73% of an H2D copy on queue family 1 hidden behind compute, so double-buffered prefetch inside one backend context remains an untested idea (estimated +10-15%, unproven).
Fractional -ncmoe no measured win Config only uses integers (22/40/48); upstream refactored -ncmoe into llm_add_n_cpu_ffn_overrides. Dropped.
MoE cache --moe-cache-mib 512 / 1536 / 2304 (Qwen3.6-35B, ncmoe 40, ctx 131072) tg256@d4096 -42% / +5% / +12%; prefill +1-3%; idle VRAM 6.15 / 7.19 / 7.96 GB vs 5.66 Small caches thrash (8-of-256 routing over 40 layers) and add a per-layer ids readback; only ~2.3 GiB pays and leaves no VRAM for a long KV. Not enabled.
MTP speculative decoding, Qwen3.6-35B decode -39% (n1), -37% (n2), -50% (n3), -44% (n4), best -19% (n2, p-min 0.75); prefill -3 to -6% A spec step costs ~180 ms vs 52 ms for plain decode; a 2-token verify alone is only 1.4x a 1-token step, so ~100 ms/step is draft machinery. The nextn layer adds 1.25 GiB VRAM. Entries removed.
MTP, gemma-4-26B-A4B decode -57.9% (n1), prefill pp4k -34% Same mechanism (~210 ms vs ~50 ms per step); MTP needs --swa-full, which alone costs -10.5% decode, -32% pp4k and saturates VRAM. Entries removed.
MTP, gemma E4B decode -10.9% Acceptance 0.41-0.67. The +19.5% pp seen with it comes from --swa-full, not MTP.
--parallel 1 on gemma no effect (-1.0%, +2.1%) The Qwen gain is probably tied to its recurrent state being sized per sequence (unverified).
-ub 4096 -b 4096 (Laguna) flat (90.0 vs 90.9) Fewer expert uploads cancelled by intra-ubatch attention growth.
mul_mat_id tile-padding theory (larger/odd -ub) predicted 2.00x waste and a 1.94x cliff; measured +3.8% and -5.6%, no sawtooth Theory dead; kills the mul_mat_id tile-selection shader work (an unbuilt, unverified WIP exists on branch fable5/mmid-tile-wip, with an unresolved correctness concern in its row_ids fill).
-b sweeps non-monotonic (2048/8192 high, 4096/12288 low) -b 2048 is already at the top.
-ncmoe below 40 on the 35B 34: +2.9% pp but 7.23 GiB peak at tiny context; 28: better at short context, worse at depth (137 vs 170 t/s at d32k); <= 24 collapses (~122) Resident experts are evicted by KV/activation growth; cannot survive a 128k context.
-ncmoe 44 on Laguna +1.5% Does not fit at 64k context. Keep 48.
fp16 on Polaris (Mesa 26.1.6 + radv_enable_float16_gfx8=true, fp16: 1) pp2048 -4.7%, -5.3% at d16k, tg +4% Hurts prefill (the priority). Mesa 26.1.6 alone: +3.3% prefill, below the 5% bar; adopting it needs a pod image change. The pod's Mesa 25.0.7 has no such option.
KV quant on Qwen (q4_0, q8_0) f16 beats q4_0 by +40% pp at 64k; q8_0 170 -> 128 t/s at depth; q4_0 with upstream PR #30738 still -16% vs f16 at d32k (133.6 vs 159.4) No int-dot/fp16 here, so dequant is pure compute; the hybrid model's KV is small anyway. Keep f16. (PR #30738 was +15% at d16k on gemma-26B with q4_0 KV; not merged upstream as of the 20261024 rebase.)
-fa 0 5x slower at depth (170 -> 35 at d32k) Keep flash attention on.
FA scalar block_rows 16 -> 32 3.5x regression at d32k (135 -> 38) ~42 KB shmem per workgroup at head_dim 256, one workgroup per SIMD, occupancy collapse.
GGML_VK_DISABLE_GRAPH_OPTIMIZE=1 restores tg32 at n_ctx=32 (10.44 -> 15.61), nothing at real depth Upstream #27812's view-alias fix makes KV views look dependent and collapses barrier-free batches; only visible when per-node work is tiny. It is a correctness fix; do not disable it in serving.
--no-mmap/-lm none for Laguna-S 118B does not load, OOM-killed the node 3 times Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap.
Laguna --cache-reuse, --swa-full, -t 12, KV quant ruled out --cache-reuse is force-disabled for iSWA when base and SWA sizes differ.
CPU-only task model (-ngl 0) to avoid handoffs rejected At --ctx-size 8192 titles could face 8k tokens of CPU prefill. The arbiter is the answer.
-lm dio (direct IO) instead of the F10 page-cache drop load +0.9 s on 3.4 GiB (6.89 vs 5.99 s), the same cost as the drop; 175 MiB of the model file and all of the mmproj stay cached Synchronous 64 MiB O_DIRECT chunks with no readahead; the GGUF metadata goes through a separate buffered FILE and clip.cpp has no direct-IO path. Measured CPU only; reads into pinned Vulkan buffers may fall back to buffered IO (EFAULT or EINVAL). Not adopted.

The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256 tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.

Where the time goes (Laguna-S-2.1 118B IQ2_M, all experts on host, GGML_VK_PERF_LOGGER=1, depth 0, 22.65 s per 2048-token eval, 17.18 s of it GPU-busy, so about 24% is H2D stall; moved here from the old README section): expert MUL_MAT_ID 52%, attention projections 24%, FLASH_ATTN_EXT 16%, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s while dense MUL_MAT q5_K/q6_K in the same graph reaches 3096-3691, so closing that gap is the largest remaining opportunity on this card, not the transfer path. On interleaved-SWA models the sliding-window layers keep a small KV (n_kv pinned at n_swa * n_seq_max + n_ubatch), so they cost the same at any depth; on Laguna 36 of 48 layers are O(1) in depth and the whole high-context slowdown comes from the 12 full-attention layers.

5. Configuration that matters

/root/config.yaml (llama-swap) and /root/supervisord.conf on the pod are the source of truth; this section explains the choices. Values as of the 20261024 deploy.

Qwen 35B-A3B MoE entries (macro moe_offload_args, applied to all 35B-A3B variants): --no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1, context 131072.

  • -lm none: load weights into RAM (upstream removed --no-mmap; the old spelling makes the server refuse to start). Faster in serving than mmap and recommended for --n-cpu-moe. Exception: Laguna-S, which must use mmap (GTT limit, section 4).
  • -ub 2048 -b 2048: the biggest prefill lever. ub512 = 97 t/s, ub1024 = 159, ub2048 = 254 on the 35B; the per-eval expert upload is amortized over the ubatch.
  • --n-cpu-moe 40: all experts on host. Lower values are faster at short context but run out of VRAM at long context (section 4). Measured idle VRAM at ctx 131072: 5.66 GB.
  • --parallel 1: +22.9% decode on the Qwen 35B (15.5 -> 19.1 t/s), prefill +0.3-1.2%, +64 MiB VRAM. Without it the server creates 4 slots and a unified KV. Two concurrent requests queue on slot 0. With one slot a ubatch never holds more than one sequence, which the concat fast path's ne[2] == 1 condition (F3) needs.
  • -t 6: six physical cores; more threads hurt decode.
  • -fa 1: essential (section 4).
  • f16 KV (no -ctk/-ctv): section 4.

Other models:

  • gemma-4-26B-A4B: macro moe_offload_gemma, same as above but --n-cpu-moe 22 (8 layers of experts resident; gemma's SWA keeps KV small at depth). ncmoe sweep pp2048: 30 = 262, 24 = 266, 18 = 272, 14 = 157 (VRAM collapse); 22 keeps headroom for 128k context (7.40 of 8 GB at 128k). Its KV cache is -ctk q4_0 -ctv q4_0, set in the gemma4_sampling / gemma4_nothink_sampling macros (not in moe_offload_gemma); note gemma has no --parallel 1 (no measured gain there).
  • Small models that fit in VRAM (Qwen3.5-4B, gemma E4B/E2B and similar) use auto-fit (common_args); forcing experts to host would slow them.
  • The 4B task model runs with --no-mmproj and ctx 8192, and is preloaded.
  • Laguna-S-2.1 118B IQ2_M (configured as Laguna-S-2.1-IQ2_M, matrix var lag): mmap, --n-cpu-moe 48 -b 2048 -ub 2048 -fa 1 --parallel 1, ctx 65536. Its exclusivity set must allow the task model (lag & q4task); making it exclusive with the task model killed it on every title request and forced full re-prefills. --parallel 1 frees 223 MiB of VRAM at 64k context there (4 auto slots size the SWA cache n_swa * n_seq_max + n_ubatch, 4096 cells instead of 2560).

Environment, set once for all of llama-swap in [program:llama-swap] environment= of /root/supervisord.conf:

  • LLAMA_SLEEP_VRAM_ONLY=1: enables the arbiter and VRAM-only sleep (F6).
  • LLAMA_VRAM_ARENA=/dev/shm/llama-vram: the shared arena. Every co-loaded GPU model must use it.
  • LLAMA_SLEEP_EVICT_KV=1: cold models also free KV, recurrent state, scheduler and scratch (F4), so the warm model gets nearly the whole card. Costs a 2-3 s re-warm instead of ~0.6 s.
  • LLAMA_MMPROJ_ONDEMAND=1: encoder in VRAM only during encodes (F5). No-op for models without an mmproj.
  • No --sleep-idle-seconds: release is purely demand-driven via want.lock.
  • GGML_CUDA_REGISTER_HOST=1 may still be present from the pinning era; it is a no-op with -lm none.

Not set, on purpose: GGML_VK_DISABLE_GRAPH_OPTIMIZE, GGML_SCHED_PREFETCH_EXPERTS, GGML_VK_ASYNC_USE_TRANSFER_QUEUE, GGML_VK_CONCAT_FASTPATH, --moe-cache-mib.

Machine-level:

  • GPU power cap fixed at 120 W by /root/fan-control.sh (a separate supervisord program; re-asserted every second; default/max cap 145 W). The two GPU hangs of 2026-10-09 happened before the cap was fixed (it used to switch between 120 and 145 W by temperature); heavy runs since have been clean.
  • Kernel argument amdgpu.lockup_timeout=0,120000,0,0 (compute ring 120 s) via Talos machine.install.extraKernelArgs. Without it long 64k-context submissions hit the compute-ring watchdog (ring comp_1.x.0 timeout).
  • Upstream changed the default --port from 8080 to 9931; the config passes --port ${PORT} explicitly, so it is unaffected.

6. Measurement methodology and traps

  • Cross-run variance is about 7%. llama-bench's within-run error bars understate cross-invocation variance by about 14x (the same config read 276.7 +/- 0.9 and 296.4 +/- 0.6 in two invocations). Put every comparison you draw a conclusion from inside one llama-bench invocation (comma-separated sweeps). Two builds cannot share an invocation, so interleave them over at least 3 rounds and compare medians. Treat any cross-build delta under about 5% as unproven.
  • Alternate arm order. A fixed order gave the later slot about a 9% advantage and nearly produced a false regression finding. Use orders like ref, cand, cand, ref.
  • Node CPU contention. The node runs about 60 pods; llama-bench once got 4.67 of 12 hardware threads while asking for 6. With --n-cpu-moe 40 every MoE layer decodes on the CPU, so stolen threads map onto tg almost directly. Prefill is nearly immune (held +-0.3% while tg32 swung +-14%). Expect occasional catastrophic tg samples; use medians, never means. Historical tg figures taken under different load are not comparable.
  • Never use tg32. At the default depth (n_ctx 32) decode is launch/barrier bound and measures command-stream overhead: two upstream commits read -36% at tg32 and were at parity at tg256 (and +7% at d512). Use tg256 (or tg128) with an explicit depth (-p 0 -n 256 -d 4096). tg is not monotonic in n_ctx on this box (n_ctx 256 measured slower than 32 and 544), so only compare same-depth cells.
  • -b Vulkan vs -b Vulkan0. test-backend-ops -b Vulkan matches nothing, runs 0 cases and still prints OK with exit 0. The backend is Vulkan0. Always check the case count.
  • ubatch larger than the prompt never fills. With -ub 2048 use prompt lengths that are multiples of 2048. A partial trailing ubatch costs a full expert upload.
  • -tb is not a llama-bench flag (server only). Any bad flag makes llama-bench print usage and exit, which looks like a run that produced no rows. Always check the row count. -ncmoe is llama-bench's spelling of --n-cpu-moe.
  • llama-bench prints its table only after all -d values finish. No output does not mean hung.
  • Nothing else may use the GPU during a measurement; stop llama-swap (bench.sh and /root/bench/moe/swapdown.sh do this and restart it from a trap).
  • Check the build finished before benchmarking it (library mtime earlier than the run start). Never overlap a build with a big benchmark (host OOM).
  • Prompt-cache reuse is timings.prompt_n, not tokens_evaluated (which is always the full prompt length).
  • Per-process VRAM from /proc/<pid>/fdinfo/* drm-memory-vram must be deduplicated by drm-client-id; summing all fds double-counts shared buffers. drm-memory-gtt is system RAM, not VRAM. Whole-card usage: /sys/class/drm/card*/device/mem_info_vram_used. GGML_VK_MEMORY_LOGGER=1 logs every Vulkan buffer alloc/free with a running device total to stderr.
  • The concat fast path is invisible to test-backend-ops (F3); validate it with the knob, perf logger timing, or byte-identical greedy generation (temperature 0, top_k 1, fixed seed, cache_prompt: false).
  • Do not run llama-cli detached with closed stdin on the pod: it can loop forever printing prompts (it once wrote a 65 GB log). Use llama-bench -d or the server harness.

7. Known gaps and open risks

  • llama_context::restore_device() ignores the bool returned by restore_device_weights(), moe_cache->restore_device() and memory->restore_device_buffers(). On a restore OOM, device_resident() stays false, decode retries the restore on every call, and a graph reading a null buffer would crash. Only the mmproj path handles a failed restore.
  • Memory types without a release_device_buffers override keep their KV under evict_kv (F4).
  • Router mode can spin at shutdown (F7); not used here.
  • The stale arbiter comments (KV "never evicted", encoder "rebuilt by vram_ensure_warm()", lazy wake, process_mtmd_chunk) and the "keeping KV cache" sleep log line were fixed in f5c0d8f28, after the deployed build: production still logs "keeping KV cache" while it evicts the KV.
  • The new checks (--moe-exact, a-handoff-latency, v-vram-*, median --perf) and the LLAMA_MOE_IDS_SKIP knob were written after the 20261024 deploy; the knob is syntax-checked only (not built), and only --arbiter has run on the GPU. Run the full set at the next rebase.
  • Not reported upstream: the wait_until_no_sleep() lost wakeup and the sleep wait that ignored new tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context. Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
  • F10 (page-cache drop) has not run on the GPU upload path or on a 35B load; its load-time cost there (synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate.