Files
llama.cpp/FORK.md
T
LumpiastyandClaude Opus 5.5 cb48bf2240 docs: FORK.md F11 - 35B results, the two leg-c anomalies explained
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.

Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.

Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 20:57:42 +01:00

1234 lines
92 KiB
Markdown

# FORK.md - the RX 580 llama.cpp fork
Maintainer documentation for this private fork. Read it before you rebase, change, or deploy
anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md),
which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-10-25):
- Branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772`. The fork commits are
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
(this file among them), so count it rather than trusting a number written here. Section 2 maps
every commit to a feature.
- Production runs `2fc9f9c35` (deployed 2026-10-25 13:01 UTC, `system_fingerprint`
`b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
top. It is not deployed; its validation (GPU and 35B included) is in F11.
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line. The commit messages are detailed and are the primary record of each mechanism
and measurement; read them with `git show`.
- Commit trailers (owner's convention): this fork is private and never contributes upstream.
Agent-written commits end with `Co-Authored-By: Claude <model> <noreply@anthropic.com>` (some
older ones say `Assisted-by: Claude`). AGENTS.md asks agents for `Assisted-by:` instead of
`Co-authored-by:` and forbids AI-written commit messages and PR descriptions; those are upstream
llama.cpp's contribution rules and do not apply to commits that stay in this fork. Anything meant
for upstream must be written by the owner under those rules.
## Contents
1. Purpose and target
2. Commit map
3. Features
- F1 MoE routing-ids readback skip
- F2 GCN flash-attention mask_opt gate (and the shmem-staging note)
- F3 Vulkan concat tiled-transpose fast path
- F4 On-demand device residency (library side)
- F5 On-demand mmproj (multimodal encoder)
- F6 Cross-process VRAM arbiter (server side)
- F7 Server shutdown path
- F8 Cold-teardown guards
- F9 Benchmark harness and validation script
- F10 Page-cache drop after a copying load
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
7. Known gaps and open risks
---
## 1. Purpose and target
The fork exists to serve large MoE models well on one specific machine:
- **GPU:** AMD RX 580 8 GB (Polaris, GCN gfx803), RADV Vulkan driver. llama.cpp reports
`fp16: 0` (RADV does not expose `shaderFloat16` on GFX8), no int-dot, no matrix cores, wave64.
PCIe 3.0 x16. RADV exposes no transfer-only queue family on this card.
- **Host:** Ryzen 5 3600 (6c/12t), 62 GB RAM, a Talos Kubernetes node shared with about 60 other pods
(CPU contention matters for decode numbers, see section 6). The GPU runs at a fixed 120 W power cap.
- **Primary model:** Qwen3.6-35B-A3B (unsloth UD-Q4_K_XL, 20.8 GiB), a hybrid MoE: 40 layers, 30 of
them gated delta-net (recurrent), 10 full attention, head_dim 256, 256 experts with 8 used. All
experts live in host RAM (`--n-cpu-moe 40`); only the backbone, KV, recurrent state and compute
buffers sit in VRAM.
- **Serving:** llama-swap under supervisord keeps several llama-server processes loaded at once
(the 35B main models, a Qwen3.5-4B "task" model OpenWebUI calls for titles and tags, gemma-4-26B
multimodal, others). They cannot all fit in 8 GB, so they time-share the GPU through the fork's
VRAM arbiter (F4-F7) without restarting processes, which would throw away the RAM-only prompt cache.
- **Priority:** prefill (prompt processing) speed first, decode second. Agentic use pushes large tool
results through the model, so long-context prefill is what users wait on.
What the fork adds, in short:
- three Polaris prefill wins (F1, F2, F3), all exact (token-identical to mainline);
- on-demand release/restore of everything a model holds in VRAM, plus a cross-process arbiter that
hands the GPU from one llama-server to another in well under a second for small models;
- fixes to the server's sleep, wake and shutdown paths that the arbiter exposed;
- a benchmark harness and a post-rebase validation script.
## 2. Commit map
Oldest first. "Feature" points at the section below.
| commit | subject | feature |
|---|---|---|
| c72cb424e | llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert | F1 |
| fb6a8a076 | ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes | F2 |
| ff2a96d2d | vulkan: record why flash-attn shmem staging stays off for GCN | F2 |
| 3df719e1e | ggml-vulkan: tiled transpose fast-path for concat with transposed source | F3 |
| 481be8b28 | server: on-demand VRAM sharing to time-share one GPU between models | F4, F6 |
| 17165fe2a | server: optionally evict the KV cache too (Phase 2 of VRAM sharing) | F4 |
| 958257cd6 | server: keep the sleep-wake path active when the VRAM arbiter is enabled | F6 |
| b9df854de | server: coordinate model load with the VRAM arbiter to avoid load-time OOM | F6 |
| 8d80f0673 | llama: guard buffer iteration against released device buffers | F8 |
| 2feb6aeb2 | server: restore weights/KV on wake, not at decode, so KV eviction is safe | F6 |
| bf2c1a699 | server: also free the compute-graph scheduler on KV eviction | F4 |
| 9857a45f0 | llama: evict recurrent/SSM state on device release | F4 |
| 9b555f1aa | ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc | F4 |
| ad27d4921 | llama: free backend compute scratch on cold; guard memory_breakdown | F4, F8 |
| 0a0f3ab6d | mtmd: release/restore the encoder weights from VRAM on demand | F4, F5 |
| 5e99ac198 | server: on-demand mmproj - free encoder VRAM on the text path | F5 |
| b7c131219 | docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends | docs |
| 5e429d4a9 | ggml-vulkan: skip the concat transpose fast path below one tile | F3 |
| 4c4486f8e | scripts: add the RX 580 benchmark harness | F9 |
| 878d1338f | server: replace the VRAM doorbell with a want-lock | F6 |
| 53f907771 | llama: skip the compute-buffer size check when the context is cold | F8 |
| 7ac2014a8 | llama: release/restore the MoE cache with the device weights | F4 |
| 54a7f534b | scripts: rx580-bench - use -lm none, document the MoE cache option | F9 |
| 11044b35e | server: run the shutdown handler off the signal-interrupted thread | F7 |
| 6c480f7e7 | server: release the GPU token promptly on shutdown | F7 |
| e085c347d | ggml-vulkan: env knob for the concat transpose fast path | F3 |
| 149a8cb0a | scripts: add fork-validate.sh, post-rebase validation of the fork | F9 |
| bfbaf15b2 | docs: FORK.md and rebase runbook for the RX 580 fork | docs |
| f5c0d8f28 | server: fix stale VRAM-arbiter comments and the sleep log line | F6 |
| f664b0f20 | llama: env knob for the routing-ids readback skip | F1 |
| 62d442e7b | scripts: rx580-bench - decode at tg256 depth 4096, not tg32 | F9 |
| 2512290f9 | scripts: fork-validate.sh - gate --perf on medians, not means | F9 |
| 80e662a2a | scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 |
| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 |
| f24c3582c | llama: do not release CPU extra buffers (CPU_REPACK) on device release | F4 |
| dd851fd0c | llama: read a released KV/recurrent state from its host shadow | F4, F11 |
| 2874dcb10 | server: disk prompt cache, save slots on shutdown and restore after a restart | F11 |
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
`6c480f7e7` was the previous production commit. The last four rows are on `fable5/disk-prompt-cache`
only (hashes of that branch), not deployed.
---
## 3. Features
Every feature section uses the same headings: What and why; Where; How it works; Invariants that
must hold; Failure modes and incidents; Rebase hazards; Validation.
### F1. MoE routing-ids readback skip
**What and why.** With experts host-resident, each split uploads the experts it needs. To choose
them, the expert-copy callback reads the routing ids back to the host. The ids are produced on the
same GPU the upload goes to, so the readback forces a full pipeline flush: 47 per eval on a
48-layer MoE. At prefill batch sizes every expert is used anyway (2048 tokens x 10 experts over 256
experts), so the readback buys nothing. Measured on Laguna-S-2.1 118B IQ2_M, all experts on host
(before it was ported to the current callback): pp2048 +4.7% at depth 32k, +0.9% at 16k, flat at
depth 0, tg +3%.
**Where.** `src/llama-context.cpp`, `llama_context::sched_copy_experts` (registered with
`ggml_backend_sched_set_copy_callback` in the scheduler setup). Commit c72cb424e.
**How it works.** When `ggml_nelements(ids) >= 4 * n_expert`, it skips
`ggml_backend_tensor_get_async` + `ggml_backend_synchronize`, clears `st.ids_data` and sets
`st.used.assign(n_expert, true)`. The existing grouping then copies the whole expert tensor as one
range. Below the threshold the upstream readback runs unchanged. For Qwen3.6-35B (256 experts, 8
used) the skip applies to any ubatch of 128 tokens or more; decode (8 ids) never reaches it.
**Invariants that must hold.**
- Exactness: uploading an expert no id selects cannot change the result, because `mul_mat_id` only
reads the rows the ids point at. Any upstream change that makes unselected experts matter (for
example a kernel that reads every uploaded expert) breaks this argument.
- With `--moe-cache-mib` on, the all-true set still goes through `moe_cache->copy_experts`, which
reads the host slot maps (no device readback) and copies cached experts device-to-device. Still exact.
- `st.ids` caching stays coherent: a repeat call with the same ids tensor reuses the all-true set,
which is what the threshold would decide again; a different `n_expert` recomputes.
- The small-batch MoE-cache slot path (`llama_moe_cache::copy`) has its own readback and is not touched.
**Failure modes and incidents.** None in production. Historically the change lived in
`ggml_backend_sched_compute_splits` in `ggml/src/ggml-backend.cpp`; upstream #29943 (`6753a033f`)
deleted that block and moved expert copying into `llama_context::sched_copy_experts`. The original
cherry-pick therefore had nothing to apply to and was re-authored as c72cb424e. A `-X theirs`
resolution would have resurrected the deleted scheduler block.
**Rebase hazards.**
- Any change to `sched_copy_experts` or the `st` state struct (`ids`, `used`, `ids_data`).
- Upstream `701fd25b3` (MUL_MAT_ID id -1 skips work) left a `GGML_ASSERT(id >= 0 ...)` in the
readback branch; the skip branch reads no ids, so it is unaffected, but watch for upstream
semantics around negative ids.
- If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.
**Validation.** `fork-validate.sh --moe-exact MODEL` (needs the GPU to itself, run it in the
`--perf` swapdown window with the 35B): reads `expert_count`/`expert_used_count` from the GGUF,
computes the threshold ceil(4 * n_expert / n_expert_used) tokens per ubatch (128 on the 35B), and
compares 64 greedy tokens after a prompt of about twice that, every expert on the host:
candidate == reference (`identical-ref`), and with the `LLAMA_MOE_IDS_SKIP` knob (f664b0f20;
`=0` forces the readback, `=log` reports each skip on stderr) candidate == candidate with the skip
forced off (`identical-noskip`, which proves the skip exact even when the reference build carries
it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). First GPU run
2026-10-25 (35B, 2fc9f9c35 vs 6c480f7e7): prompt 397 tokens, 39 skips logged, 0 with the skip forced
off, output identical in all three runs, 78 s. Also `--ops` (the MUL_MAT_ID
kernel, not this callback), `--perf` (prefill regression) and `--moecache` (the cache path through
this callback).
### F2. GCN flash-attention mask_opt gate (and the shmem-staging note)
**What and why.** `mask_opt` skips fully-masked causal blocks and drops the per-block mask load+add
on fully-visible blocks. Upstream disables it on AMD GCN for head sizes <= 256. On Polaris attention
at depth is compute-bound on the softmax path (exp runs at about 1/4 VALU rate, no matrix cores), so
skipping that work is a real win. Measured on Qwen3.5-35B-A3B (head_dim 256): pp2048 +1% at depth 0,
+7.8% at 16k (176 -> 190), +12.3% at 32k (136 -> 153), growing with depth. Perplexity bit-identical
(5.1668 on and off).
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`, `ggml_vk_flash_attn`, the `use_mask_opt`
condition. Commit fb6a8a076. The shmem-staging note is a comment in `get_fa_tuning_params_scalar`
(commit ff2a96d2d, no behaviour change).
**How it works.** The GCN clause `HSK > 256 || HSV > 256` becomes `HSK >= 256 || HSV >= 256`. The
existing large-mask conditions (`nem1 >= 32`, `nem0 * nem1 > 32768`,
`nem0 >= block_cols * 16`) keep it off for decode.
**Invariants that must hold.**
- Only GCN with head size >= 256 is affected. Do not lower the threshold: relaxing it below 256 was
-18.5% at 16k on Laguna (head_dim 128).
- Upstream's `!use_sparse` term must stay in the condition (sparse flash-attention came in with
upstream `fc82583e6`).
- `shmem_staging` stays NVIDIA-only. Measured on GCN at head size 128: -6.7% at 16k, -7.4% at 32k.
The `kvsh` stride of `D/4+1` dwords is 4 mod 32, an 8-way LDS bank conflict on wave64 (the `+1`
padding is tuned for warp32), and the extra shared memory cuts occupancy. If anyone retries it,
fix the padding to +2/+3 first.
**Failure modes and incidents.** A `-X theirs` resolution in a trial rebase silently dropped
`!use_sparse` from the condition. No production incident.
**Rebase hazards.** Any upstream rewrite of `use_mask_opt`, of the FA tuning parameters, or of
architecture detection (`vk_device_architecture::AMD_GCN`). Upstream `733f57a99` added a path that
copies strided f16/bf16 K/V into `prealloc_x` scratch for flash-attention on AMD non-RDNA4 (so GCN
is included) when `neq1 >= 256`; measured at d32768 it adds about 68 MiB of VRAM and no visible
long-context prefill loss, but it has no env toggle, so its speed effect could not be isolated.
**Validation.** `fork-validate.sh --ops` (FLASH_ATTN_EXT, 5332 cases on Vulkan0) and `--perf`
(pp2048 at depth 16k and 32k is where this feature shows).
### F3. Vulkan concat tiled-transpose fast path
**What and why.** Delta-net layers build their conv input with `ggml_concat(conv_states,
ggml_transpose(qkv_mixed), 0)`. The generic concat shader reads the transposed source with an
uncoalesced stride: about 5.6 ms for a 16 MB concat on the RX 580 against about 130 us of memory
bandwidth. On Qwen3.6-35B the fast path alone is +14% pp8192 and accounts for more than all of the
fork's prefill win at depth 0 (the rest of the fork costs about 1% there). In the validation run on
Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 1945 us without.
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`: `ggml_vk_concat` (gate lambda
`src_transposed_2d` and the `s1_trans`/`s0_trans` checks) and `ggml_vk_concat_transpose_fastpath`.
The graph pattern that matters here comes from `llm_build_delta_net_base::build_conv_state` in
`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next). Commits 3df719e1e (fast path),
5e429d4a9 (decode gate), e085c347d (env knob).
Other graphs that build `ggml_concat(state, ggml_transpose(x), 0)` (found with
`git grep -n "ggml_concat.*ggml_transpose" src/models/` plus the two-line form in delta-net-base and
lfm-base): `qwen4exp.cpp` (`build_conv_state_at`), `bailingmoe3.cpp`, `clef.cpp`, `kimi-linear.cpp`,
`kimi-k3.cpp`, `glm5-next.cpp`, `mamba-base.cpp` (two sites, the Mamba/Mamba2 conv and the hybrids
built on it), `plamo2.cpp`, and `lfm-base.cpp` (causal short conv; the transpose is assigned to `bx`
first). The fast path takes any of them when the gate holds (transposed source a plain 2D transpose
of a contiguous f32/f16 tensor, one sequence, `ne[0] >= 32`), so a change to the gate or the shader
affects all of them. Only the delta-net path (Qwen3.5/3.6) has been measured and validated; the
`--concat` check uses qwen35.
**How it works.** For a concat along dim 0 where one source is stored transposed (`nb[1] ==
type_size`, `nb[0] == ne[1] * type_size`) and the other source and dst are contiguous, it issues two
dispatches: `pipeline_cpy_32`/`_16` copies the contiguous source into its dst columns, and
`pipeline_cpy_transpose_32`/`_16` (the shared-memory 32x32 tiled transpose in `copy_transpose.comp`)
writes the transposed source into the other columns, both with custom push constants and a dst
offset. It ends with `ggml_vk_sync_buffers`. Everything else falls through to the generic path.
The gate requires: f32 or f16 with all three types equal (quantized and integer types stay generic);
matching ne[1..3]; `ne[2] == 1 && ne[3] == 1` on the transposed source (so it fires only with one
sequence per ubatch, which `--parallel 1` gives); the 16-bit `doffset` bound; and
`ne[0] >= 32` (TILE_DIM) on the transposed source.
The decode gate (5e429d4a9): at one token the transposed source is a single column, so there is no
stride to fix, but the path still costs two dispatches and the unconditional `ggml_vk_sync_buffers`,
a full pipeline barrier (the generic path emits none). That barrier, not the half-empty transpose,
was nearly all of a -17% tg256 regression, because the path ran 30 times per decoded token on the
35B. The transposed source is `qkv_mixed` transposed, so its `ne[0]` is the ubatch token count:
prefill keeps the fast path, decode and small speculative drafts take the generic one. Measured
with the gate: decode back to parity (gated/clean 1.023), prefill win retained (gated/ungated 0.998,
pp8192 +11% over clean), generation byte-identical.
The knob (e085c347d): `GGML_VK_CONCAT_FASTPATH=0` forces the generic path, `=log` prints
`ggml_vk_concat: transpose fast path, transposed ne0=N` to stderr for every fast-path dispatch. It is
read once (function-local static). Not set in production.
**Invariants that must hold.**
- The fast path fires at prefill and never at decode on a delta-net model.
- Output is identical to the generic path (it is a pure copy/transpose).
- The `types_ok` gate keeps quantized types off the path: the dispatches are raw f32/f16 element copies.
**Failure modes and incidents.**
- The decode regression above shipped in the deployed fork before the gate existed. It looked
tg-neutral on the old upstream only because tg32's +-14% spread hid a 17% effect (section 6).
- If upstream changes `build_conv_state` (for example a `ggml_cont` on the transpose), the fast path
silently stops firing. Nothing crashes; prefill just loses about 14%. Only the concat check or a
perf A/B notices.
**Rebase hazards.**
- Pipeline renames: upstream `733f57a99` renamed `pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to
`pipeline_cpy_32`/`pipeline_cpy_16`. The pick applied cleanly and did not compile.
- Upstream `f172be756` made `ggml_vk_concat` non-static (declared in `ggml-vulkan-common.h`) and
split helpers into headers; the fast-path helper stays static.
- Shape changes in `build_conv_state` / `qkv_mixed` (`[ch, n_seq_tokens, n_seqs]` reshaped from a
contiguous `build_lora_mm` output), changes to `copy_transpose.comp` or `copy.comp` push constants,
or to `vk_op_unary_push_constants` / `get_misalign_bytes`.
- Upstream concat now handles quantized types; keep `types_ok`.
- The stock `test_concat` cases in `tests/test-backend-ops.cpp` never produce a transposed source and
always have `ne[2]`/`ne[3]` > 1, so `test-backend-ops -o CONCAT` does NOT exercise this path.
**Validation.** `fork-validate.sh --concat` (needs a build with the knob, i.e. this branch or
`--concat-build`): counts fast-path dispatches in a prefill run (must be > 0, all with ne0 >= 32)
and a decode run (must be 0 while CONCAT nodes did execute), then ties the unmodified candidate to
the fast path by CONCAT timing (candidate about equal to knob-on and at least 2x faster than
knob-off). Last run: 72 dispatches at prefill (all ne0=512), 0 over 594 CONCAT nodes at decode.
Also `--perf`.
### F4. On-demand device residency (library side)
**What and why.** Several always-loaded models must share 8 GB. Co-residence is impossible (the 35B
at `--n-cpu-moe 40` uses most of the card; the 4B task model needs about 2.5-3.3 GB). Restarting a
process to free VRAM throws away the RAM-only prompt cache: on Laguna a repeat turn went from a
21,960 ms re-prefill to 225 ms once the process stayed alive across a task-model call. So a model
must be able to drop everything it holds in VRAM and get it back without a reload.
Measured footprints:
- 4B: warm 3327 MB, weights released 724 MB, restored and correct.
- 4B at 32k ctx: cold weights-only 1750 MB (KV stays), cold with KV evicted 726 MB.
- gemma-26B: warm 7147 MB, cold with weights+KV but scheduler kept 718 MB, cold with the scheduler
freed too 51 MB.
- A cold 4B before the encoder work held 644 MB that turned out to be its mmproj (F5).
- Re-warm cost: about 0.6 s per cold-to-warm switch weights-only (tiny prompt: 1.98 s cold vs 1.40 s
warm); 2-3 s with KV eviction (it adds the KV D2H/H2D and a scheduler reserve).
**Where.**
- `src/llama-model.cpp`/`.h`: `llama_model::release_device_weights(keep_shadow)`,
`restore_device_weights()`, `weights_resident()`; per-buffer metadata
`llama_model::impl::weight_release` (`weight_release_slot`), filled in `load_tensors` and
`load_tensors_hadamard`.
- `src/llama-memory.h`: `llama_memory_i::release_device_buffers(keep_shadow)` /
`restore_device_buffers()` (default no-op). Implemented in `src/llama-kv-cache.cpp`
(`llama_kv_cache`) and `src/llama-memory-recurrent.cpp`; `llama_memory_hybrid` and
`llama_kv_cache_iswa` delegate to their children.
- `src/llama-moe-cache.cpp`/`.h`: `llama_moe_cache::release_device()`, `restore_device()`,
`resident()`, `impl::invalidate()`.
- `ggml/include/ggml-backend.h`, `ggml/src/ggml-backend-impl.h`, `ggml/src/ggml-backend.cpp`:
optional `ggml_backend_i::free_scratch` and `ggml_backend_free_scratch()`; Vulkan implementation
`ggml_backend_vk_free_scratch` in `ggml-vulkan.cpp`.
- `src/llama-context.cpp`/`.h`: `llama_context::release_device(evict_kv)`, `restore_device()`,
`discard_device()`, `device_resident()`, members `kv_device_evicted`, `device_discarded`; the
auto-restore at the top of `llama_context::decode`.
- `include/llama.h`: `llama_context_release_device(ctx, evict_kv)`, `llama_context_restore_device`,
`llama_context_discard_device`, `llama_context_weights_resident`.
- `tools/mtmd/clip.cpp`/`.h`, `tools/mtmd/mtmd.cpp`/`.h`: `clip_release_device`,
`clip_restore_device`, `clip_weights_resident`, `mtmd_release_device`, `mtmd_restore_device`.
- Commits 481be8b28, 17165fe2a, bf2c1a699, 9857a45f0, 9b555f1aa, ad27d4921, 0a0f3ab6d, 7ac2014a8,
6c480f7e7 (discard).
**How it works.** Tensor metadata (`ggml_context` structs) is durable; device buffers are ephemeral.
- Weights: on release, for each releasable buffer, capture a host shadow once (weights are
read-only), free the buffer, null every tensor's `buffer`/`data`. On restore,
`ggml_backend_alloc_ctx_tensors_from_buft` reallocates an identical layout (re-pointing the
tensors), the shadow is uploaded, usage is set to WEIGHTS. Only single-buffer, alloc-path, non-host
buffers are releasable: the host-pointer path (mmap or lazy-mapped), `no_alloc` dummy buffers and
host (`Vulkan_Host`, i.e. pinned RAM) buffers stay as they are.
- KV and recurrent state: same pattern, but the shadow is re-captured on every release (the data is
read-write) and dropped after restore.
- Scheduler: with `evict_kv`, `sched.reset()` frees the worst-case compute buffer and
`sched_need_reserve = true`; restore calls `sched_reserve()` (which goes through `sched_new()`,
re-registering the expert-copy callback and upstream's realloc callback).
- Backend scratch: with `evict_kv`, `ggml_backend_free_scratch()` on each backend. On Vulkan this
discards the pending compute context, synchronizes, destroys `prealloc_x`, `prealloc_y`,
`prealloc_split_k`, `prealloc_add_rms_partials`, `sync_staging`, and zeroes their sizes and the
`prealloc_y_last_*` tracking. They are reallocated lazily on the next compute. Other backends leave
the hook null (no-op).
- MoE cache (`--moe-cache-mib`, upstream #29887): released in both modes. The banks are freed and the
cache is emptied, not shadowed: every SLRU is rebuilt fresh, its `slot_map[il]` pointers are
re-pointed at the host slot-map tensors, and the host slot maps are cleared to 0xff (every slot
-1). On restore the banks are reallocated (which also re-inits their views), cleared to 0 and
marked WEIGHTS; the cache refills from the host experts.
- Multimodal encoder: `clip_release_device` / `clip_restore_device` use the weights pattern (shadow
once); no-op for a CPU-backed encoder. `mtmd_*` wrap the vision and audio contexts.
- `llama_context::release_device(evict_kv)`: synchronize, release weights, release MoE cache, and if
`evict_kv`: release memory buffers, free sched, free scratch.
`restore_device()`: weights, MoE cache, memory buffers, `sched_reserve()` if needed, then
`ggml_backend_sched_synchronize` so uploads finish before any compute.
`discard_device()`: like `release_device(true)` with `keep_shadow = false` everywhere (no host
copies), sets `device_discarded`; used only before exit.
- `decode()` calls `restore_device()` if `!device_resident()`, as a safety net. The server restores
explicitly on wake (F6), because some KV access happens before decode.
**Invariants that must hold.**
- Shadows are stored compactly in stable tensor-iteration order, skipping view tensors
(`view_src != NULL`), and restored with the same iteration. Never index a shadow by
`t->data - base`.
- `weight_release` stays parallel to `pimpl->ctxs_bufs`: every push to `ctxs_bufs` needs a matching
push to `weight_release` (non-releasable slots use `emplace_back()`).
- The caller synchronizes before any release; `release_device` does this itself.
- Restore order: weights, MoE cache, KV/recurrent, scheduler reserve, scheduler synchronize. The MoE
cache banks must be back before the scheduler is rebuilt.
- After a MoE-cache release no slot map may point at an expert: banks come back zeroed, so a stale
slot map silently reads zero weights.
- Every iterator over `ctxs_bufs` (model, KV cache, recurrent) must tolerate a null buffer, and every
use of `sched` must tolerate null while cold with `evict_kv` (see F8).
- `device_resident()` covers weights and MoE cache; it gates the early returns, the decode
auto-restore and `llama_context_weights_resident()`.
- `restore_device()` after `discard_device()` aborts (`GGML_ASSERT(!device_discarded)`).
- Greedy output is byte-identical across any number of release/restore cycles, and the prompt cache
survives (`timings.prompt_n` drops on a repeated prompt after a cycle).
**Failure modes and incidents.**
- The first shadow implementation indexed by `t->data - base` and produced garbage after restore
(layout and padding differ). Fixed by the compact iteration-order shadow.
- Restoring KV lazily at decode crashed iSWA models (gemma) with
`GGML_ASSERT(buf != NULL && "tensor buffer not set")`: `update_slots()` creates an SWA checkpoint
that reads the KV via `ggml_backend_tensor_get` before decode. Fixed by restoring on wake (2feb6aeb2).
- The recurrent state was left resident (default no-op), 201 MB on the 4B, until 9857a45f0.
- A model with weights on the CPU device (`-dev none` / `-ngl 0`) segfaulted (exit 139) in
`release_device_weights()` the first time the arbiter put it to sleep: CPU_REPACK buffers are not
host buffers, so `load_tensors` marked them releasable, and the shadow capture called their NULL
`get_tensor`. It also crashes production `2fc9f9c35`, but no production arbiter model has CPU-device
weights. Fixed in f24c3582c: only buffers on a non-CPU device are releasable. Found by the
`--diskcache` check (F11), whose arbiter cases run CPU only.
- The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed
resident in a cold model until bf2c1a699 / ad27d4921.
- The MoE cache was not released at all before 7ac2014a8. Reallocating its banks without
invalidating would have produced silent wrong weights.
- In the 20261024 rebase `moe_cache_slru::init()` turned out to reset only the probation segment
(`head[1]`, `tail[1]` and `n_protected` survive), so re-running it on a used cache corrupts the
lists. `invalidate()` now assigns a fresh `moe_cache_slru()` before `init()`.
**Rebase hazards.**
- New push sites into `ctxs_bufs` (upstream's `load_tensors_hadamard()` added two in the 20261024
rebase; without matching `weight_release` entries `release_device_weights()` reads past the end).
- Changes to buffer creation in `load_tensors` (the `host_ptr_path` condition, `ctx_bufs`/`buf_ptrs`
bookkeeping; upstream replaced `ctx_buf_maps.emplace_back(ctx, buf_map)` with
`ctx_bufs.emplace_back(ctx, std::move(buf_ptrs))` in the 20261024 rebase).
- The `llama_memory_i` interface. New memory types that do not override `release_device_buffers`
keep their KV resident under `evict_kv`; at the time of writing that is `llama_kv_cache_msa`,
`_dsa`, `_dsa_iswa`, `_dsv4` and `llama_memory_hybrid_iswa`.
- MoE-cache internals (`groups`, `slru`, `slot_map`, `buf_host`, `devices`). Upstream `33d6135cb`
changed LRU to SLRU (`g.lru` -> `g.slru`); the pick applied cleanly and did not compile.
- `llama_context` members next to ours (upstream `n_reallocs_same/diff`, `copy_experts_info`),
`sched_reserve()` semantics, and the `decode()` signature (now `decode(const llama_batch_ext &)`).
- `ggml_backend_i` layout (`free_scratch` must stay the last member after `graph_optimize`; other
backends rely on zero-init) and the Vulkan context's prealloc field set (compare with
`ggml_vk_cleanup`).
**Validation.** `fork-validate.sh --arbiter`: `b-greedy-identical` (15 cycles each side equal to a
no-handoff reference), `b-kv-survives` (prompt_n drops after evict/restore, output identical),
`c-image-after-restore`. `--moecache MODEL` (needs a cache large enough to hit; 256 MiB on gemma-26B
had 0 hits and proves nothing, 2048 MiB had 6651 hits). `--arbiter` `v-vram-release` /
`v-vram-restore` check that VRAM really drops: a baseline of `mem_info_vram_used` before any scratch
server, then server A warm and alone, forced cold by a fake waiter on want.lock: the card-wide delta
must drop by >= 70% of what A added and come back after the next request. Production's own share is
re-read from the DRM fdinfo of every process the script did not start; if it moved by > 64 MiB the
check SKIPs. First run (2026-10-25, 0.8B with mmproj, ctx 8192, `LLAMA_SLEEP_EVICT_KV=1`): warm
+546 MiB, cold +4 MiB, restored +546 MiB.
### F5. On-demand mmproj (multimodal encoder)
**What and why.** A multimodal model's vision encoder sat in VRAM on every text request. With
`LLAMA_MMPROJ_ONDEMAND=1` the encoder lives in RAM and enters VRAM only for an encode. Measured on the
35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered
correctly, text after the image correct.
**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream `9871df591` split the
encode out of `process_mtmd_chunk`; `encode_mtmd_chunk` is called from `process_mtmd_chunk` and
directly from the batched-embedding path in the slot update, so the restore covers both), the
`mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`, `vram_go_cold`,
`vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
**How it works.** After load the encoder is released (shadow captured). Before
`mtmd_batch_encode`, `mtmd_restore_device(mctx)`; if that fails (out of VRAM) and on-demand is set, it
releases the LLM backbone weights only (`llama_context_release_device(ctx_tgt, false)`: read-only
shadow, so a cheap free with no D2H, KV and prompt cache untouched) and retries. After the encode it
releases the encoder FIRST, then restores the weights, so the peak stays within VRAM.
**Invariants that must hold.**
- Free the encoder before re-uploading the weights.
- The donor is the weights, never the KV (the KV is read-write and would need a D2H every time).
- `vram_ensure_warm()` does not restore the encoder; only the encode path does.
**Failure modes and incidents.** Before this, a cold text model with an auto-downloaded mmproj
(`-hf` pulls one when the repo ships it) held 644 MB that no release path touched. The text-only 4B
task model now runs with `--no-mmproj`, which is the right fix for any model that never sees images.
**Rebase hazards.** Renames or restructuring of `encode_mtmd_chunk` and the mtmd init block in
`load_model`; `clip_ctx` fields (`buf`, `ctx_data`, `sched`, `no_alloc`).
**Validation.** `fork-validate.sh --arbiter` case `c-image-after-restore` (server A runs with
`--mmproj` and `LLAMA_MMPROJ_ONDEMAND=1`; a 128x128 red PPM must be answered "red" after A was cold,
and text right after must equal the reference). Post-deploy: one image request to gemma-26B.
### F6. Cross-process VRAM arbiter (server side)
**What and why.** Several llama-server processes under llama-swap share one GPU: exactly one is warm
(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production:
60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between
the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last
`fork-validate.sh` run before the 20261024 deploy: 30 handoffs, median 680 ms, max 1231 ms (the
2026-10-25 run of the same build read 1050 ms, max 1562 ms: handoff latency moves with node load,
which is why it is now A/B'd against the reference in one session).
**Where.** `tools/server/server-context.cpp`: `vram_arena_open`, `vram_acquire_for_load`,
`vram_share_init`, `vram_should_yield`, `vram_want_acquire`, `vram_want_release`, `vram_take_token`,
`vram_ensure_warm`, `vram_go_cold`, `vram_share_exit`, `vram_share_shutdown`, the VRAM-only branch of
`handle_sleeping_state`, the `vram_ensure_warm()` call before decode in the slot update path, and
`server_res_generator` (`bypass_sleep`). `tools/server/server-queue.cpp`/`.h`: `on_should_yield`,
`start_loop` (`should_sleep`, idle tick, post-wake grace), `wait_until_no_sleep`, `is_stopping`.
Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
**How it works.**
- Enabled by `LLAMA_SLEEP_VRAM_ONLY` (any value). The arena directory is `LLAMA_VRAM_ARENA` (default
`/dev/shm/llama-vram`) with two files.
- `token.lock`: `flock(LOCK_EX)` is the baton. Rule: a process has device buffers iff it holds it.
- `want.lock`: a waiter holds it `LOCK_SH` while waiting and drops it once it owns the token. The
holder probes it with `flock(LOCK_EX | LOCK_NB)` on a SECOND fd: failure means somebody is waiting.
The probe needs its own open file description, because flock on the waiter's own fd would convert
that lock instead of conflicting with it. The kernel drops a dead waiter's lock, so nothing goes stale.
- Load: `load_model()` calls `vram_acquire_for_load()` before `common_init_from_params()`, so the
current holder releases before this model uploads anything. The loaded model stays warm holding the
token and serves its first request without a re-warm.
- Yield: `vram_share_init()` registers `vram_should_yield` with `queue_tasks.on_should_yield`.
`start_loop()`'s `should_sleep()` calls it on the loop thread, under `mutex_tasks`, only when the
task queue is empty and `update_slots` has returned. While the callback is set the idle wait ticks
every 10 ms instead of 1 s. After a wake the server holds the GPU for at least 100 ms
(`yield_grace_ms`). Sleep runs the sleeping callbacks; in VRAM-only mode `handle_sleeping_state(true)`
calls `vram_go_cold()` (release device, also the encoder; then `flock(LOCK_UN)`) instead of
upstream's full unload.
- Wake: a request calls `wait_until_no_sleep()`. `handle_sleeping_state(false)` calls
`vram_ensure_warm()`: take the token (register on want.lock, then poll `token.lock` with
`LOCK_EX | LOCK_NB` every 2 ms, aborting when `queue_tasks.is_stopping()`), restore ctx_tgt and
ctx_dft, and only then set `vram_cold = false`. The decode path calls `vram_ensure_warm()` again
(no-op when warm) and throws "server is shutting down" if it returns false.
- `server_res_generator` no longer bypasses the wake path when `--sleep-idle-seconds` is unset if
`LLAMA_SLEEP_VRAM_ONLY` is set, because the arbiter can put a server to sleep without idle-sleep.
- Queue fixes: `wait_until_no_sleep()` loops `while (sleeping && !stopping)` and re-asserts
`req_stop_sleeping` each time (with `notify_all`); the sleep wait also wakes on a non-empty task
queue; the idle `wait_for` only leaves the loop on a real task or termination.
**Invariants that must hold.**
- Resident iff holding `token.lock`; `vram_cold == false` means both. `vram_ensure_warm()` sets
`vram_cold = false` only AFTER the restore. What that order protects today, reasoned from the code:
the only thing that gives the token away is `vram_should_yield()`, called from `should_sleep()` on
the loop thread. `vram_ensure_warm()` also runs only on the loop thread, synchronously: from the
wake callback (`handle_sleeping_state(false)`, called inside `start_loop()` with `mutex_tasks`
held) and before decode in the slot update path (outside `yield_to_queue()`). So `should_sleep()`
cannot run between the token acquire and the end of the restore, whichever order the two lines
are in. The measured failure (attempt 1 below: handoffs of 11m20s and 3m57s) happened under the
retired doorbell design, where a separate warden thread read the cold flag to decide whether to
honour or discard rings, so an early `false` let it accept stale rings during the restore. Under
the inline want-lock probe the order is defensive: it keeps the flag's meaning exact for any future
reader off the loop thread, and keeps a release out of a half-restored model if the restore or the
probe ever moves (for example into `yield_to_queue()`). Either way, do not change it without an A/B
of ordinary handoff latency (`fork-validate.sh --arbiter`, `a-handoff-latency`).
- `vram_acquire_for_load()` sets `vram_cold = false` as soon as it holds the token, before the upload.
Nothing reads the flag during the load today (the loop is not running yet and `on_should_yield` is
only registered in `vram_share_init()`, which sets it again), so the early set is redundant now.
It records "holding the token" at the moment it becomes true, because the opposite error is the
dangerous one: a model that is warm and holds the token while `vram_cold` says cold never yields
(`vram_should_yield()` returns false for a cold model), which strands every waiter, and its next
wake would try to restore weights that were never released.
- Release only happens from the idle branch of `start_loop()`, on the loop thread, never inside
`yield_to_queue()` or while a slot is generating: a holder finishes its current generation before
it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both).
- The yield decision reads live kernel state at the point of use. Do not route it through a flag set
by another thread (see attempts 3-4 below).
- Every exit from `vram_take_token()` drops want.lock, or the holder yields forever.
- The probe fd is separate from the waiter fd.
- `flock` calls retry on EINTR: the server's signal handlers are installed without SA_RESTART.
- Restore on wake, before `update_slots()`, so KV eviction is safe for SWA checkpoints.
- Every model that can be warm on the GPU at the same time must use the same arena. A non-arbiter
process (a benchmark, a manually started server) can OOM the warm model. In production the env is
set for all of llama-swap, so all its children participate.
**Failure modes and incidents.**
- Wake-path hang (958257cd6): with no `--sleep-idle-seconds`, `bypass_sleep` skipped
`wait_until_no_sleep()`, so a server slept by the arbiter never woke and requests hung.
- Load-time OOM (b9df854de): weights were uploaded before the arbiter was active; loading gemma while
the 4B held VRAM could exceed 8 GB. After the fix the peak was 7771 MB.
- The doorbell design (replaced by 878d1338f) was edge-triggered: a waiter wrote
`<arena>/doorbell/<pid>` and the holder's warden thread turned the inotify event into a yield flag.
Rings were lost in three windows: while the holder restored (it advertised cold and discarded
foreign rings), while the holder loaded (the inotify watch did not exist yet, and the kernel does
not queue events for a missing watch), and when a cold waiter consumed a third process's ring.
Every loss was permanent: `flock(LOCK_EX)` had no timeout and production had no idle-sleep. The
blocked flock held `mutex_tasks`, so the whole server stopped answering while `/health` still
returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by
the second day).
- Failed fix attempt 1 (doorbell design): moving `vram_cold = false` before the restore made the
holder's warden honour stale rings and hand the token back 2-5 s after taking it without serving.
Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. See the first invariant
for what this means under the current design.
- Failed fix attempt 2: a re-ring loop (`LOCK_NB` + re-ring every ~1 s). Queued rings from an already
satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s
baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become
level-triggered.
- Failed attempts 3-4: want.lock was sound (mutual exclusion, kill -9 safety, load-window coverage),
but the warden thread still delivered its decision as a flag. `start_loop()` holds `mutex_tasks`
from `should_sleep()` through the callbacks to the wait, so the warden's `request_yield()` was always
admitted after the flag had been consumed, latching a release for the next wake (a server woke and
re-slept 138 ms later without serving). It also exposed the upstream lost wakeup in
`wait_until_no_sleep()`. Result: 0.494 s median, 1/60 stranded. The fix deleted the warden and
`request_yield`/`yield_requested` and probes inline in `should_sleep()`.
- Measured deviations that are required: the 10 ms idle tick (about 0.2% of one core per warm
server); a waiter that polls rather than retrying on a coarse grid (a 200 ms retry grid cost 0.503 s
against 0.336 s, because the holder only releases after the waiter asks); the 100 ms post-wake
grace (without it a permanent waiter caused a livelock: one request took 23.15 s over 138
sleep/wake cycles).
- Process lesson from attempts 1-2: validate an ordinary handoff latency A/B, not only the
pathological case. Two production deploys were burned before that rule existed.
- Lost diagnostic: the old periodic "still waiting for the GPU token after N s" warning is gone; a
stuck token now shows one `VRAM arbiter: waiting for the GPU token` line and then silence. Check
`/proc/locks` (see the runbook).
**Rebase hazards.** This is the most fragile area, because it sits inside upstream's server loop.
- `server_queue::start_loop`, `wait_until_no_sleep`, `process_new_tasks`, `yield_to_queue` and the
sleeping-callback vector. Upstream refactored sleep (`947fd9bb2`) and made decode run inside
`yield_to_queue` (`22b8e310b`); re-check after any such change that a release still cannot run
under a live graph.
- `handle_sleeping_state`, `load_model` ordering, the slot update path where decode happens,
`server_res_generator`'s `bypass_sleep`, renames in `server-context.cpp`.
- Anything that touches the KV in `update_slots()` before decode.
**Validation.** `fork-validate.sh --arbiter` with two real servers on a scratch arena:
`a-handoffs` (30 alternating, 0 stranded, 0 non-200, the server that answered holds `token.lock`),
`a-handoff-latency` (fresh server pairs of `--build` and `--ref` in the same session, blocks
ref, cand, cand, ref x 16 handoffs; FAIL if the candidate median exceeds the reference median by more
than max(25%, 200 ms); first run 2026-10-25, reb1024 against itself beside production: 1050.5 vs
1082.5 ms), `v-vram-*` (F4), `b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams:
want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in
F7 and a log scan for asserts and Vulkan errors. In production, `/proc/locks` must show the warm
model holding WRITE on `token.lock`.
### F7. Server shutdown path
**What and why.** llama-swap evicts models with SIGTERM, so shutdown happens all the time, often
while the arbiter is mid-release or mid-wait. Two bugs made shutdown keep the GPU token and stall
every other model.
**Where.** `tools/server/server.cpp`: `signal_handler`, `shutdown_pipe`, `struct shutdown_thread`,
`sig_thread` in `llama_server`. `tools/server/server-context.cpp`: `server_context::start_loop` (calls
`vram_share_exit()` after the loop returns), `server_context::terminate` (terminates the result queue
first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/server/server-queue.cpp`:
`server_queue::terminate` (sets atomic `stopping` before taking the mutex), `process_new_tasks`,
`server_response::recv_with_timeout`, `server_response::terminate`,
`server_response_reader::next`. Commits 11044b35e, 6c480f7e7.
**How it works.**
- The signal handler only writes the signal number to a self-pipe (async-signal-safe); a dedicated
thread reads it and runs `shutdown_handler`. The pipe fds are `FD_CLOEXEC`. If `pipe()` fails the old
direct call is used. The thread object is declared after the server objects, so it is joined (a 0
byte unblocks it) before they are destroyed. A second signal still calls `exit(1)`.
- `terminate()` sets `stopping` (atomic) before locking `mutex_tasks`, then stops the result queue so
HTTP handlers waiting for results get "server is shutting down" (HTTP 503, or an SSE error event);
`recv_with_timeout` checks `!running` before waiting and returns null.
- A token wait polls with `LOCK_NB` every 2 ms and aborts on `is_stopping()`; the aborted wake stays
cold, skips the restore, and the loop exits before `update_slots`.
- After `start_loop()` returns, `vram_share_exit()` frees all device buffers with
`llama_context_discard_device()` (no host shadows), releases the encoder, and only then unlocks
`token.lock`.
- What llama-swap (v239 on the pod) does when a child exits on its own (crash, OOM kill, external
signal), verified 2026-10-25 with a scratch llama-swap instance and a CPU-only child, for both
`kill -9` and an external SIGTERM: it logs
`[WARN] group: running <model> exited: [<model>] upstream exited unexpectedly`, does not restart it
eagerly (no new process in the 2 s after the kill), and starts a new process on the next request
for that model (new PID, health check, HTTP 200). The production log since the last pod restart has
no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers
the restart pays a full model load (about 26 s for the 35B with `-lm none`; the first 35B request
after the 2026-10-25 deploy took 34 s including load and answer), and under the arbiter
the new process takes the token through `vram_acquire_for_load()` like any load. A process that
wedges instead of exiting (the incidents below) is never restarted: `/health` keeps answering.
**Invariants that must hold.**
- Nothing that takes `mutex_tasks` runs in signal context.
- The token is released only after this process's VRAM is free (the next model may start uploading
the moment the lock drops).
- A shutting-down server never restores weights.
- SIGTERM in any state exits 0 within about 2 s and frees the token. With the disk prompt cache (F11)
the exit also writes each slot's state: about 1 s more per GB of state, after the token is free.
**Failure modes and incidents.**
- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside
a sleeping callback (under `mutex_tasks`, during a release); the handler's `terminate()` locked
`mutex_tasks` again on the same thread. The process wedged holding `token.lock`, every other model
queued behind it, and only SIGKILL worked. Reproduced 6/6 off production; 8/8 clean with the fix.
The first suspicion (the multimodal path) was wrong. gdb on the wedged process showed the main
thread in `futex(mutex_tasks)` with the `yield_to_queue` worker idle.
- Token kept on shutdown: with a request in flight the HTTP handler waited forever for results, main
blocked joining it, and the token was released only in the destructor after that join; with a
token wait in progress the blocking flock held `mutex_tasks`; and the old destructor unlocked the
token BEFORE `destroy()`, so the next model could upload into still-occupied VRAM.
- After the fix: off production SIGTERM mid-generation or mid-wait exits in about 0.2 s with exit 0
and frees the token in 5-70 ms; in production about 0.7-1 s.
**Rebase hazards.** `server.cpp`'s signal setup and `llama_server` object lifetimes;
`server_response` and `server_response_reader` (upstream `4e201388c` switched `recv_with_timeout` to a
fixed deadline: keep the order "check `!running`, return null, then `wait_until(lock, deadline)`");
`server_context::start_loop`/`terminate`. Router mode (`server-models.cpp`,
`server_models_sse_client::next`) only checks `should_stop()` and can spin at shutdown now that
`recv_with_timeout` returns null at once; llama-swap does not use router mode.
**Validation.** `fork-validate.sh --arbiter`: `d-term-in-release` (gdb breakpoint on
`llama_context_release_device`, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and
the token), `e-term-waiting` (exit 0 within 1.5 s, never took the token), `f-term-streaming`
(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run
(2026-10-25, 2fc9f9c35, `d` by log polling): 750 ms, 225 ms, 126 ms (token free in 23 ms),
108/159/159 ms. Post-deploy: a forced llama-swap eviction
must log `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token` and exit by itself.
### F8. Cold-teardown guards
**What and why.** A cold model has null buffers in its `ctxs_bufs` and, with `evict_kv`, a null
scheduler. Upstream code never had to consider either state.
**Where.** `src/llama-context.cpp`: `llama_context::~llama_context` (`sched &&` in the
compute-buffer size check), `llama_context::memory_breakdown` (`if (sched)` around the compute part;
upstream's `if (moe_cache)` breakdown block stays outside it). Null-buffer skips in
`llama_model::memory_breakdown`, `llama_kv_cache::{clear, memory_breakdown, total_size}`,
`llama_memory_recurrent::{clear, memory_breakdown, total_size}`. Commits 8d80f0673, ad27d4921, 53f907771.
**How it works.** Guards at the call sites. The asserts in ggml (`GGML_ASSERT(buffer)`,
`GGML_ASSERT(sched)`) are correct and stay; every other caller reserves the scheduler first.
**Invariants that must hold.** A server shut down while cold exits 0. Any new upstream code that
iterates `ctxs_bufs` or calls into `sched` on a path reachable while cold (shutdown, memory
breakdown, stats) needs the same guard.
**Failure modes and incidents.**
- `GGML_ASSERT(buffer)` on shutdown of a cold model (seen 7 times in production logs) from
memory-breakdown/size/clear on released buffers.
- `GGML_ASSERT(sched)` in `memory_breakdown` for a cold `evict_kv` model being torn down.
- `GGML_ASSERT(sched)` in the destructor's size check: every cold shutdown exited 134 instead of 0, so
under llama-swap an ordinary stop looked like a crashed child, left a backtrace in the log, and the
rest of the destructor never ran. Repro: arbiter env on, one request, force a yield with
`flock -s <arena>/want.lock -c 'sleep 6'`, then SIGINT while cold.
**Rebase hazards.** New iteration over `ctxs_bufs`, new `sched` users in teardown or breakdown
paths. Watch item: `llama_kv_cache::update()` takes `lctx->get_sched()`, which can be null; it is
safe today only because its single caller runs after `sched_reserve()`.
**Validation.** `fork-validate.sh --arbiter`: `h-shutdown-cold` (exit 0 while the other server holds
the token) and `log-scan`; `--moecache` `shutdown`.
### F9. Benchmark harness and validation script
**What and why.** The measurement method on this box had to be rediscovered several times (section
6). The harness encodes it; the validation script encodes the checks that each past incident
required, so a rebase can be validated by someone with no context.
**Where.** `scripts/rx580-bench/` (`run.sh` workstation wrapper; `bench.sh` pod orchestrator;
`ppbench.py` corpus prefill through llama-server; `lbsweep.sh` llama-bench sweep; `summarize.py`;
`README.md` with methodology and reference values). `scripts/fork-validate.sh`. Commits 4c4486f8e,
54a7f534b, 149a8cb0a, 62d442e7b, 2512290f9, 80e662a2a, ceab1a91f, e5fc474b8.
**How it works.**
- `rx580-bench`: stops llama-swap, waits for an idle GPU, runs fixed slices of `pan-tadeusz.txt`
(4096/16384/32768 tokens, byte-identical across builds) through `/completion` with
`cache_prompt: false`, plus a llama-bench sweep (prefill pp2048/pp8192 at depth 0, decode
tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and
rounds; restarts llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]
[--moe-exact MODEL] [--diskcache]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a
scratch dir under /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets
inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check
flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline.
See the header of the script for what each check proves.
- Which phase needs what: `--ops --concat` run beside production; `--arbiter` too only if 1536 MiB
of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs
in the swapdown window, about 2 minutes); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share
the card either). `--diskcache` runs beside production (CPU only); its GPU leg SKIPs below 1536 MiB
of free VRAM.
**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24,
production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s,
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. On 2026-10-25
(medians, `6c480f7e7` / `2fc9f9c35`): pp2048 274.5/273.8, d16384 200.5/200.4, d32768 157.6/157.6,
tg256@d4096 16.8/16.7. The corpus
harness and older numbers are in `scripts/rx580-bench/README.md`. A fresh run far outside these
means something else is wrong (another GPU user, thermal state, a bad build) before it means a
regression or a win.
**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` and `--moe-exact`
refuse, and any other llama process blocks them). `--perf` gates prefill at -5% per depth on the
MEDIAN of all repetition samples of each build (never a mean: CPU contention produces single
catastrophic samples), and peak VRAM at 95% of the card; decode is reported, not gated (CPU-bound and
noisy here).
**Failure modes and incidents.** Upstream #28334 removed `--mmap`/`--no-mmap`/`--mlock` and
llama-bench's `-mmp`: the server refused to start and llama-bench silently produced no rows until
54a7f534b switched to `-lm none`. The perf and MoE-cache checks leave tens of GB of model files in page
cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024
deploy (see the runbook). That pod restart also emptied `/tmp` (every earlier `fork-validate.*`
scratch dir with its raw samples is gone) and removed gdb, which is not in the pod image; `--arbiter`
case `d-term-in-release` then falls back to polling the log.
**Rebase hazards.** CLI flag renames in `common/arg.cpp` and `tools/llama-bench`; log line texts the
script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`,
`no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`,
`llama_moe_cache: .* hits =`, `sched_copy_experts: ids readback skipped`, `N/M tests passed`,
`Backend Vulkan0: OK`, `disk cache: restored`, `disk cache: copied`/`saved`, `prompts of this model`,
`size limit, removing`, `disk cache: failed to write`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and
markdown layout; the GGUF keys `<arch>.expert_count` / `<arch>.expert_used_count`; the DRM fdinfo
keys `drm-client-id` / `drm-memory-vram`.
**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test. The first
candidate arm has now twice shown an unexplained spike of about 1.6 GiB over the other arms
(2026-10-25: ref 5476, cand 7062, cand 5767, ref 5476 MiB); the cause is open (per-test peaks would
locate it), and it stays far below the 95% gate. The SIGTERM-in-release breakpoint is at function
entry, not after the host-shadow capture. The `v-vram-release` SKIP path (production moving during
the measurement) has not been exercised. `--moe-exact` and the median-gated `--perf` first ran on the
GPU on 2026-10-25 (runbook section 6).
### F10. Page-cache drop after a copying load
**What and why.** With `-lm none` the loader read()s the GGUF into its own buffers, but the kernel
keeps the file in page cache afterwards, charged to the pod's cgroup: about 20 GB after a 35B load.
The pod is BestEffort on a shared node, and on 2026-10-24 the Talos OOM controller killed it right
after a model load under exactly that pressure (MemFree < 1 GB, file cache about 20 GB). Once the
data is copied those pages are pure waste, so the loader advises the kernel to drop them.
**Where.** `src/llama-mmap.h`/`.cpp`: `llama_file::drop_cache(offset, len)`.
`src/llama-model-loader.cpp`, `llama_model_loader::load_all_data`: the per-tensor drop at the end
of the copy (non-mapping) branch and the whole-file drop in the final cleanup
(`size_done >= size_data`). `tools/mtmd/clip.cpp`, `clip_model_loader::load_tensors`: whole-file drop
after the tensor reads. Commit cce0ac1f0.
**How it works.**
- `drop_cache()` is `posix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED)` under `#ifdef __linux__`
(no-op elsewhere); `len` 0 means to the end of the file. Errors are logged at debug level only.
- `load_all_data()` drops each tensor's range right after it is read (all three copy paths: straight
into a host buffer, the async pinned-upload chunks, the staging buffer), so the cache never grows
past about one tensor during the load. The final cleanup drops every file whole, which catches the
GGUF metadata and the partial pages at tensor edges that a range drop leaves.
- Both run only when `mappings.empty()`, i.e. the loader holds no mapping of any of its files: not
`mmap`, `mmap+mlock` or `auto`, and no lazy tensors (`-lzm`; with lazy tensors `init_mappings()`
maps every file even under `-lm none`). Split GGUFs are covered (every entry of `files`).
- clip.cpp always copies the encoder through an ifstream (no mmap path exists), so it opens the
file again after the reads and drops it whole, unless `no_alloc`.
- Choice: per-tensor plus a final drop rather than a post-load drop only, because the per-tensor
drop also removes the peak during the load (the incident window: anon and file cache grow together)
for three extra lines. It costs about 0.27 s more than a post-load-only drop on the test model
below. Direct IO (`-lm dio`, upstream's) was measured as the alternative and is not better (section 4).
**Invariants that must hold.**
- Never drop for a file this loader maps: a mapped page would just be faulted back in (the kernel
also skips pages that are mapped into a process, so a violation costs refaults, not correctness).
- Drop only after the bytes are in our own buffers. Nothing in the fork re-reads the GGUF after
load: the VRAM release/restore (F4) uses host shadows, not the file.
- The drop is a hint: a failure must not fail the load.
**Failure modes and incidents.** None. Deployed 2026-10-25 (`2fc9f9c35`); on a 35B GPU load in
production the node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB. Costs to know
about:
- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a
5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few
seconds on a 26 s load.
- A reload of the same model can no longer hit a warm page cache (a warm-cache reload of the test
model took 4.6 s against 5.9 s cold). Under llama-swap a reload happens only after an eviction or
a crash.
- `TENSOR_DUPLICATED` tensors read the same range twice; the second read now comes from disk (small
tensors only).
**Rebase hazards.** The structure of `load_all_data()` (the `from_mapping` branch and its three copy
paths, the final cleanup block), `init_mappings()` creating mappings for lazy tensors, new load modes
in `llama_load_mode` (any new mode that maps must leave `mappings` non-empty), and the tensor-read
loop in clip.cpp `load_tensors`.
**Validation.** Not in `fork-validate.sh`. By hand on the pod with a scratch llama-server on a
scratch port, CPU only (`-dev none -ngl 0 --no-mmproj-offload`, no VRAM), cold cache (evict the test
files with python `os.posix_fadvise(..., DONTNEED)` first): page residency of the model and mmproj
files by `mincore` before and after the load (must be 0 after with `-lm none`, unchanged with
`-lm mmap`), the pod cgroup's `file`/`anon` in
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/memory.stat`, host `MemFree`, load time, and a greedy
request. First run (2026-10-25, gemma-4-E2B UD-Q4_K_XL 2.5 GiB + mmproj BF16 0.9 GiB, production
6c480f7e7 and the candidate with the drops disabled as controls, 4 rotated rounds):
| | drops off (control) | post-load only | per-tensor + final (committed) |
|---|---|---|---|
| model / mmproj resident after load | 2431-2444 / 940 MiB | 0 / 0 | 0 / 0 |
| peak model residency during load | 2.1-2.5 GiB | 2.1-2.4 GiB | 1.2-1.5 GiB (largest tensor) |
| pod cgroup `file` after load | +3.37 GiB | +0 | +0 |
| host MemFree after load | 0.6 GB | 2.5 GB | 2.5-2.8 GB |
| load time (median, production 5.85 s) | 5.87 s | 6.56 s | 6.82 s |
Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it counts clean page cache as
available. Greedy output byte-identical in all arms. `-lm mmap` control: the model file stays fully
resident and mapped (2498 MiB, `file_mapped` +2.5 GiB) after load and after a request, and only the
copied mmproj is dropped. `-lzm on` control: no drop (the file is mapped), 1252 MiB of copied
tensors stay cached. The GPU upload path and a 35B load were first exercised by the 2026-10-25
deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B
request took 34 s including load and answer).
### F11. Disk prompt cache (save slots on shutdown, restore after a restart)
**What and why.** Only one large model fits in RAM. Switching models in OpenWebUI makes llama-swap
SIGTERM the previous one, and switching back re-prefills the whole conversation, because the RAM
prompt cache died with the process (F4's reason for the arbiter). A 30k-token chat costs about 150 s
on the 35B at about 200 t/s. With `--cache-disk-path DIR`, each slot with a prompt is saved to DIR on
shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the
file, so only the suffix is processed. Measured on Qwen3.5-0.8B CPU only: a 12k-token turn 2 went
from a 72 s prefill to 1.2 s, of which the read took 307 ms and the restore 18 ms. Off by default.
Upstream PR #28092 (`--cache-disk`) was the reference for ideas, not ported (1700 lines). It
livelocked the kernel on AMD + Vulkan by DMA-ing GPU state into a MAP_SHARED mmap'd file, and it never
saved the slot in use on shutdown, which is the main case here.
**Where.**
- `tools/server/server-task.h`/`.cpp`: `server_prompt_disk_entry`, `server_prompt_disk_cache`
(`n_reuse`, `find`, `has`, `load`, `save`, `remove`); file IO helpers `disk_cache_io`,
`disk_cache_open`.
- `tools/server/server-context.cpp`: `DISK_CACHE_MIN_TOKENS`, `disk_cache_key`,
`disk_cache_collect`, `disk_cache_flush`, `disk_cache_load`; the init in `load_model` after the RAM
cache; the `disk_cache_load` call at the end of `get_available_slot`; `server_context::start_loop`
(collect, `vram_share_exit`, flush).
- `common/arg.cpp`, `common/common.h`: `--cache-disk-path` (`cache_disk_path`), `--cache-disk-size`
(`cache_disk_mib`, default 4096, 0 = no limit).
- Library (cold save): `llama_context_restore_memory_host()` (`include/llama.h`,
`llama_context::restore_memory_host`); `llama_memory_i::restore_device_buffers(bool to_host)` in
`llama-memory.h`, `llama-kv-cache`, `llama-memory-recurrent`, `-hybrid`, `-kv-cache-iswa`; the helper
`llama_memory_wrap_host()` in `llama-memory.cpp`.
- Commits dd851fd0c, 2874dcb10, 2c6c517ba (and f24c3582c, F4).
**How it works.**
- Save, after `start_loop()`'s loop returns. First `disk_cache_collect()` copies the state of every
slot with at least 256 text tokens to host memory (`llama_state_seq_get_data_ext`, flags NONE)
while the device buffers still exist. Then `vram_share_exit()` frees the VRAM and releases the GPU
token as before. Then `disk_cache_flush()` writes the files. The token release therefore waits
only for the device-to-host copy, never for disk IO.
- A slot is skipped when its KV does not cover its token list (`seq_pos_max + 1 != n_tokens`: the last
batch was not decoded, a shutdown during a token wait), when it holds media, and when the same
token list is already on disk.
- Cold server: with `LLAMA_SLEEP_EVICT_KV` the KV and recurrent state live only in host shadows (F4)
and their tensors have no buffer. `llama_context_restore_memory_host()` wraps each shadow as a
CPU_Mapped buffer (`ggml_backend_cpu_buffer_from_ptr`) and points the tensors into it, so there is
no copy, no allocation and no device use. It sets `device_discarded`. Without evict_kv the KV is still
in VRAM and is read directly.
- File, `<dir>/<key hash>-<token hash>.lpcd`: a 48-byte header (magic `LPCD`, version, key hash,
n_tokens, n_ckpt, payload size, payload hash), then 8-byte padded sections: tokens, checkpoint table,
state size, state, the state of each checkpoint. The hash is FNV-1a over the 64-bit words of the
payload. Write: plain fwrite to `.tmp`, fdatasync, `POSIX_FADV_DONTNEED`, rename. Read: plain fread,
then DONTNEED. There is no mmap anywhere.
- Index: at startup, from the headers, tokens and checkpoint tables of the files of this key. A bad
magic, a file size that differs from the header, or an inconsistent table means the file is removed.
Other keys and other versions are counted and left alone. A `.tmp` older than 1 h is removed.
- Key: the build commit, the model path + size + mtime, LoRA adapters and control vectors, n_ctx,
n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, K/V types, flash-attn type, and the RoPE/YaRN
parameters.
- Restore: at the end of `get_available_slot()`, after the RAM cache, only while warm, with
`cache_prompt` on and no media tokens. The file that reuses the most of the new prompt is loaded if
it reuses at least 256 more tokens than the slot holds. The slot's prompt goes to the RAM cache
first. Then the slot is cleared and the state is set; a failure clears the slot and removes the file.
- Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt,
or, for a recurrent or SWA model, a saved checkpoint at or below the common prefix can be used. A
plain-KV model can be cut back to any prefix. Exact prefix alone would almost never fire: the Qwen
template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is not a
prefix of the next OpenWebUI turn. Checkpoints are exact snapshots, and `update_slots()` rolls back
to one exactly as it does for the RAM cache; the recurrent state itself is never cut back.
Restored checkpoints get `id_task = -1`, so one that fails to load means a re-prefill, not an abort.
- Size limit: before a write, the least recently used `*.lpcd` files of every model in DIR are removed
until the new file fits. A restore updates the file's mtime. Entries the new prompt extends are
removed after the write and do not count against the limit. A write also needs its size + 1 GiB
free on the file system, so the cache never fills the PVC.
- Disabled with speculative decoding, with a warning: the draft state is not saved.
**Invariants that must hold.**
- Never restore while cold or without the GPU token (`disk_cache_load` returns when `vram_cold`).
`get_available_slot()` runs on the loop thread after the wake, like the RAM cache's `prompt_save`.
- Copy the state before `vram_share_exit()` (it discards the device buffers) and write after it (the
token is not held for disk IO).
- Never serve a file that fails the size or hash check. Never mmap the file: a DMA into a shared
mapping is what livelocked upstream's version on AMD.
- A write is atomic (tmp + rename), and any IO error leaves the server serving and the exit clean.
- The key must cover everything that changes the state layout or the KV values. Add a new
state-changing parameter to `disk_cache_key()`.
- `llama_context_restore_memory_host()` is for exit only; the wrapped buffers point into the shadows,
which must outlive them (both are members of the same memory object).
**Failure modes and incidents.**
- The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself
(CPU_REPACK, F4), not in the save.
- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full
KV allocation at exit (about 2.7 GB for the 35B at ctx 131072) on a node that has had an
OOM-controller kill. Replaced by the in-place wrap. The 32-byte alignment that
`cpu_buffer_from_ptr` asserts is met by rounding the base down; those leading bytes are never read.
- Writes to `/tmp` (node overlay) stalled once: 5.8 s for 64.5 MiB, against 171 ms normally.
- Costs to know about. llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a child still there 30 s after SIGTERM (measured 2026-10-25 with a scratch
llama-swap and dummy upstreams: term to exit 4.0 s, next start 2 ms later; a child that ignored SIGTERM
was replaced at +30.0 s). So the exit write adds directly to a model switch: about 1 s per GB on the
`/root` PVC (0.66 GB/s with fdatasync). A save cut off by the SIGKILL leaves only a `.tmp`.
- Only slots are saved. A conversation that sits only in the RAM cache when the process stops is lost.
- Every deploy empties the cache (the key holds the build commit), and so does any change to the
model file or the flags in the key.
**Rebase hazards.**
- `server_context::start_loop` and `vram_share_exit` ordering (F7); `get_available_slot` and the RAM
cache (`prompt_save`, `prompt_load`, `server_prompt_cache_state`); `server_prompt` and
`common_prompt_checkpoint` fields (new fields must be saved, or the format version bumped).
- The checkpoint rollback in `update_slots` (`pos_min_thold`, `TAG_PROMPT_LOGITS`), which decides how
much of a restored file is reused.
- The llama state format (`state_write`/`state_read` of the memory types). The key holds the build
commit, so a format change cannot load an old file; it only empties the cache.
- `llama_memory_i::restore_device_buffers` and the shadow layout of F4 (`device_buffer_shadow`):
`llama_memory_wrap_host` relies on the compact iteration-order shadow.
- Flag names in `common/arg.cpp`.
**Validation.** `fork-validate.sh --diskcache` (2c6c517ba), beside production. The model is the small
hybrid, CPU only (`-dev none -ngl 0`), with chat requests and greedy decoding, and a reference server
that never restarts. Cases: `restore`, `restore-arbiter` and `restore-cold` (turn 1, SIGTERM, restart,
turn 2 restored; prompt_n equal to the reference; identical answer), `no-prefix`, `key-ctx`,
`key-ctk`, `key-model`, `corrupt` (truncated and byte-flipped), `lru`, `write-error` (RLIMIT_FSIZE,
SIGXFSZ ignored), `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free. First run
(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219
MiB free).
| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill |
|---|---|---|---|---|---|---|
| 573-token chat | 64.5 MiB, 2 ckpt | 18 ms | 171 ms | 314 ms (token free 56 ms vs 28) | 166 + 3 ms | prompt_n 58 of 596, 462 vs 2913 ms |
| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms |
| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s |
Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.sh`):
`--diskcache --arbiter` 29 PASS, 0 FAIL, GPU legs included.
35B (`q35.py`, production 35B flags with `--cache-disk-path` on a scratch dir, arbiter env, an
18760-token pan-tadeusz chat; log `/root/arms/diskcache-test/q35.log`):
| leg | result |
|---|---|
| turn 1 prefill (reference and candidate) | 18760 tokens in 83.4 s (224.8 t/s), outputs identical |
| warm save on SIGTERM (a) | 556.5 MiB (2 checkpoints), copy 323 ms, write 960 ms; token free in 328 ms (6 ms without the cache), exit 1952 ms (607 ms without), peak RSS +430 MiB |
| restore after the restart (b) | read 860 ms, restore 114 ms; turn 2 prompt_n 27 of 18850 in 3.0 s (the reference that never restarted: 27 in 2.0 s); output identical |
| cold save on SIGTERM (b, KV in the host shadow) | 746.7 MiB (5 checkpoints), copy 310 ms, write 1403 ms, exit 2928 ms, peak RSS +432 MiB: the in-place wrap adds no copy |
| restore of the cold save (c, turn 2 sent again) | read 1189 ms, restore 75 ms, prompt_n 4; output differs from the reference's turn 2 (see below) |
Two anomalies in leg c, both explained:
- Output != reference: c sent turn 2 again, so the saved list (turn 2 + 63 generated tokens) extends
the new prompt. `update_slots()` then rolls back to the checkpoint at the prompt end minus 4 and
processes 4 tokens, while the reference processed turn 2 from turn 1's state (27 tokens). These are
different reuse points, not the same computation. The controlled test on Qwen3.5-0.8B
(`/root/arms/diskcache-test/cmp.py`) compares the same reuse point. Server X never restarts and
runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced
cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the
GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU
to itself: `/root/arms/diskcache-window2.sh` (about 6 minutes, under swapdown) is ready and not run.
Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the
GPU), not proven on that model.
- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
The driver's fake waiter was `flock -s want.lock sleep 120`; killing flock left `sleep` holding
want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right
after its load and again right after its request (log 0.37.50). That ordinary arbiter release
recaptures the full KV allocation as a host shadow, about 2.6 GB at ctx 131072 (F4), and the SIGTERM
arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The
driver now holds the lock in the process it kills, like `fork-validate.sh` does.
Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache
--cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is
created on first start. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
applies. Models only pick up the flags when they restart.
---
## 4. Dead ends
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
without a new reason.
| idea | result | mechanism |
|---|---|---|
| Weight pinning (`GGML_CUDA_REGISTER_HOST` and the host-register chain, from a third-party CUDA fork whose +64% was on an RTX 3060) | +17% pp2048 in isolated llama-bench (252 -> 294), fails in serving | RADV cannot import file-backed mmap pages; the pre-stage fallback fails (`ErrorOutOfDeviceMemory` for the ~18 GB expert region) or doubles host RAM. `-lm none` serves faster (Qwen3.6-35B 278 -> 286). Dropped in the 2026-09-09 rebase. |
| Expert-upload prefetch (`GGML_SCHED_PREFETCH_EXPERTS`) | -20% pp (other runs -9% to -22%), no tg benefit | Slots allocate in host-pinned GTT, not VRAM, so it stages host to host and still pays the PCIe read. `=1` means the default 3 slots. Dropped. |
| `GGML_VK_ASYNC_USE_TRANSFER_QUEUE=1` | flat (-0.1%), byte-identical | GCN is excluded from `prefers_transfer_queue` and RADV exposes no transfer-only queue family here; the per-layer upload-then-compute order leaves nothing to overlap. A standalone Vulkan microbench did show 60-73% of an H2D copy on queue family 1 hidden behind compute, so double-buffered prefetch inside one backend context remains an untested idea (estimated +10-15%, unproven). |
| Fractional `-ncmoe` | no measured win | Config only uses integers (22/40/48); upstream refactored `-ncmoe` into `llm_add_n_cpu_ffn_overrides`. Dropped. |
| MoE cache `--moe-cache-mib` 512 / 1536 / 2304 (Qwen3.6-35B, ncmoe 40, ctx 131072) | tg256@d4096 -42% / +5% / +12%; prefill +1-3%; idle VRAM 6.15 / 7.19 / 7.96 GB vs 5.66 | Small caches thrash (8-of-256 routing over 40 layers) and add a per-layer ids readback; only ~2.3 GiB pays and leaves no VRAM for a long KV. Not enabled. |
| MTP speculative decoding, Qwen3.6-35B | decode -39% (n1), -37% (n2), -50% (n3), -44% (n4), best -19% (n2, p-min 0.75); prefill -3 to -6% | A spec step costs ~180 ms vs 52 ms for plain decode; a 2-token verify alone is only 1.4x a 1-token step, so ~100 ms/step is draft machinery. The nextn layer adds 1.25 GiB VRAM. Entries removed. |
| MTP, gemma-4-26B-A4B | decode -57.9% (n1), prefill pp4k -34% | Same mechanism (~210 ms vs ~50 ms per step); MTP needs `--swa-full`, which alone costs -10.5% decode, -32% pp4k and saturates VRAM. Entries removed. |
| MTP, gemma E4B | decode -10.9% | Acceptance 0.41-0.67. The +19.5% pp seen with it comes from `--swa-full`, not MTP. |
| `--parallel 1` on gemma | no effect (-1.0%, +2.1%) | The Qwen gain is probably tied to its recurrent state being sized per sequence (unverified). |
| `-ub 4096 -b 4096` (Laguna) | flat (90.0 vs 90.9) | Fewer expert uploads cancelled by intra-ubatch attention growth. |
| mul_mat_id tile-padding theory (larger/odd `-ub`) | predicted 2.00x waste and a 1.94x cliff; measured +3.8% and -5.6%, no sawtooth | Theory dead; kills the mul_mat_id tile-selection shader work (an unbuilt, unverified WIP exists on branch `fable5/mmid-tile-wip`, with an unresolved correctness concern in its row_ids fill). |
| `-b` sweeps | non-monotonic (2048/8192 high, 4096/12288 low) | `-b 2048` is already at the top. |
| `-ncmoe` below 40 on the 35B | 34: +2.9% pp but 7.23 GiB peak at tiny context; 28: better at short context, worse at depth (137 vs 170 t/s at d32k); <= 24 collapses (~122) | Resident experts are evicted by KV/activation growth; cannot survive a 128k context. |
| `-ncmoe` 44 on Laguna | +1.5% | Does not fit at 64k context. Keep 48. |
| fp16 on Polaris (Mesa 26.1.6 + `radv_enable_float16_gfx8=true`, `fp16: 1`) | pp2048 -4.7%, -5.3% at d16k, tg +4% | Hurts prefill (the priority). Mesa 26.1.6 alone: +3.3% prefill, below the 5% bar; adopting it needs a pod image change. The pod's Mesa 25.0.7 has no such option. |
| KV quant on Qwen (q4_0, q8_0) | f16 beats q4_0 by +40% pp at 64k; q8_0 170 -> 128 t/s at depth; q4_0 with upstream PR #30738 still -16% vs f16 at d32k (133.6 vs 159.4) | No int-dot/fp16 here, so dequant is pure compute; the hybrid model's KV is small anyway. Keep f16. (PR #30738 was +15% at d16k on gemma-26B with q4_0 KV; not merged upstream as of the 20261024 rebase.) |
| `-fa 0` | 5x slower at depth (170 -> 35 at d32k) | Keep flash attention on. |
| FA scalar `block_rows` 16 -> 32 | 3.5x regression at d32k (135 -> 38) | ~42 KB shmem per workgroup at head_dim 256, one workgroup per SIMD, occupancy collapse. |
| `GGML_VK_DISABLE_GRAPH_OPTIMIZE=1` | restores tg32 at n_ctx=32 (10.44 -> 15.61), nothing at real depth | Upstream #27812's view-alias fix makes KV views look dependent and collapses barrier-free batches; only visible when per-node work is tiny. It is a correctness fix; do not disable it in serving. |
| `--no-mmap`/`-lm none` for Laguna-S 118B | does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. |
| Laguna `--cache-reuse`, `--swa-full`, `-t 12`, KV quant | ruled out | `--cache-reuse` is force-disabled for iSWA when base and SWA sizes differ. |
| CPU-only task model (`-ngl 0`) to avoid handoffs | rejected | At `--ctx-size 8192` titles could face 8k tokens of CPU prefill. The arbiter is the answer. |
| `-lm dio` (direct IO) instead of the F10 page-cache drop | load +0.9 s on 3.4 GiB (6.89 vs 5.99 s), the same cost as the drop; 175 MiB of the model file and all of the mmproj stay cached | Synchronous 64 MiB O_DIRECT chunks with no readahead; the GGUF metadata goes through a separate buffered FILE and clip.cpp has no direct-IO path. Measured CPU only; reads into pinned Vulkan buffers may fall back to buffered IO (`EFAULT` or `EINVAL`). Not adopted. |
The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256
tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.
Where the time goes (Laguna-S-2.1 118B IQ2_M, all experts on host, `GGML_VK_PERF_LOGGER=1`, depth 0,
22.65 s per 2048-token eval, 17.18 s of it GPU-busy, so about 24% is H2D stall; moved here from the
old README section): expert `MUL_MAT_ID` 52%, attention projections 24%, `FLASH_ATTN_EXT` 16%,
everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s while dense `MUL_MAT` q5_K/q6_K in
the same graph reaches 3096-3691, so closing that gap is the largest remaining opportunity on this
card, not the transfer path. On interleaved-SWA models the sliding-window layers keep a small KV
(`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`), so they cost the same at any depth; on Laguna 36 of
48 layers are O(1) in depth and the whole high-context slowdown comes from the 12 full-attention
layers.
## 5. Configuration that matters
`/root/config.yaml` (llama-swap) and `/root/supervisord.conf` on the pod are the source of truth;
this section explains the choices. Values as of the 20261024 deploy.
Qwen 35B-A3B MoE entries (macro `moe_offload_args`, applied to all 35B-A3B variants):
`--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1`,
context 131072.
- `-lm none`: load weights into RAM (upstream removed `--no-mmap`; the old spelling makes the server
refuse to start). Faster in serving than mmap and recommended for `--n-cpu-moe`. Exception:
Laguna-S, which must use mmap (GTT limit, section 4).
- `-ub 2048 -b 2048`: the biggest prefill lever. ub512 = 97 t/s, ub1024 = 159, ub2048 = 254 on the
35B; the per-eval expert upload is amortized over the ubatch.
- `--n-cpu-moe 40`: all experts on host. Lower values are faster at short context but run out of VRAM
at long context (section 4). Measured idle VRAM at ctx 131072: 5.66 GB.
- `--parallel 1`: +22.9% decode on the Qwen 35B (15.5 -> 19.1 t/s), prefill +0.3-1.2%, +64 MiB VRAM.
Without it the server creates 4 slots and a unified KV. Two concurrent requests queue on slot 0.
With one slot a ubatch never holds more than one sequence, which the concat fast path's
`ne[2] == 1` condition (F3) needs.
- `-t 6`: six physical cores; more threads hurt decode.
- `-fa 1`: essential (section 4).
- f16 KV (no `-ctk/-ctv`): section 4.
Other models:
- gemma-4-26B-A4B: macro `moe_offload_gemma`, same as above but `--n-cpu-moe 22` (8 layers of experts
resident; gemma's SWA keeps KV small at depth). ncmoe sweep pp2048: 30 = 262, 24 = 266, 18 = 272,
14 = 157 (VRAM collapse); 22 keeps headroom for 128k context (7.40 of 8 GB at 128k).
Its KV cache is `-ctk q4_0 -ctv q4_0`, set in the `gemma4_sampling` / `gemma4_nothink_sampling`
macros (not in `moe_offload_gemma`); note gemma has no `--parallel 1` (no measured gain there).
- Small models that fit in VRAM (Qwen3.5-4B, gemma E4B/E2B and similar) use auto-fit
(`common_args`); forcing experts to host would slow them.
- The 4B task model runs with `--no-mmproj` and ctx 8192, and is preloaded.
- Laguna-S-2.1 118B IQ2_M (configured as `Laguna-S-2.1-IQ2_M`, matrix var `lag`): mmap, `--n-cpu-moe 48 -b 2048 -ub 2048 -fa 1 --parallel 1`,
ctx 65536. Its exclusivity set must allow the task model (`lag & q4task`); making it exclusive with
the task model killed it on every title request and forced full re-prefills. `--parallel 1` frees
223 MiB of VRAM at 64k context there (4 auto slots size the SWA cache `n_swa * n_seq_max +
n_ubatch`, 4096 cells instead of 2560).
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
`/root/supervisord.conf`:
- `LLAMA_SLEEP_VRAM_ONLY=1`: enables the arbiter and VRAM-only sleep (F6).
- `LLAMA_VRAM_ARENA=/dev/shm/llama-vram`: the shared arena. Every co-loaded GPU model must use it.
- `LLAMA_SLEEP_EVICT_KV=1`: cold models also free KV, recurrent state, scheduler and scratch (F4),
so the warm model gets nearly the whole card. Costs a 2-3 s re-warm instead of ~0.6 s.
- `LLAMA_MMPROJ_ONDEMAND=1`: encoder in VRAM only during encodes (F5). No-op for models without an mmproj.
- No `--sleep-idle-seconds`: release is purely demand-driven via want.lock.
- `GGML_CUDA_REGISTER_HOST=1` may still be present from the pinning era; it is a no-op with `-lm none`.
Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`,
`GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`.
`--cache-disk-path` (F11) is not set yet: it is not deployed; the recommended flags are in F11.
Machine-level:
- GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted
every second; default/max cap 145 W). The two GPU hangs of 2026-10-09 happened before the cap was
fixed (it used to switch between 120 and 145 W by temperature); heavy runs since have been clean.
- Kernel argument `amdgpu.lockup_timeout=0,120000,0,0` (compute ring 120 s) via Talos
`machine.install.extraKernelArgs`. Without it long 64k-context submissions hit the compute-ring
watchdog (`ring comp_1.x.0 timeout`).
- Upstream changed the default `--port` from 8080 to 9931; the config passes `--port ${PORT}`
explicitly, so it is unaffected.
## 6. Measurement methodology and traps
- **Cross-run variance is about 7%.** llama-bench's within-run error bars understate cross-invocation
variance by about 14x (the same config read 276.7 +/- 0.9 and 296.4 +/- 0.6 in two invocations).
Put every comparison you draw a conclusion from inside one llama-bench invocation
(comma-separated sweeps). Two builds cannot share an invocation, so interleave them over at least 3
rounds and compare medians. Treat any cross-build delta under about 5% as unproven.
- **Alternate arm order.** A fixed order gave the later slot about a 9% advantage and nearly produced
a false regression finding. Use orders like ref, cand, cand, ref.
- **Node CPU contention.** The node runs about 60 pods; llama-bench once got 4.67 of 12 hardware
threads while asking for 6. With `--n-cpu-moe 40` every MoE layer decodes on the CPU, so stolen
threads map onto tg almost directly. Prefill is nearly immune (held +-0.3% while tg32 swung +-14%).
Expect occasional catastrophic tg samples; use medians, never means. Historical tg figures taken
under different load are not comparable.
- **Never use tg32.** At the default depth (n_ctx 32) decode is launch/barrier bound and measures
command-stream overhead: two upstream commits read -36% at tg32 and were at parity at tg256 (and +7%
at d512). Use tg256 (or tg128) with an explicit depth (`-p 0 -n 256 -d 4096`). tg is not monotonic in
n_ctx on this box (n_ctx 256 measured slower than 32 and 544), so only compare same-depth cells.
- **`-b Vulkan` vs `-b Vulkan0`.** `test-backend-ops -b Vulkan` matches nothing, runs 0 cases and
still prints OK with exit 0. The backend is `Vulkan0`. Always check the case count.
- **ubatch larger than the prompt never fills.** With `-ub 2048` use prompt lengths that are
multiples of 2048. A partial trailing ubatch costs a full expert upload.
- **`-tb` is not a llama-bench flag** (server only). Any bad flag makes llama-bench print usage and
exit, which looks like a run that produced no rows. Always check the row count. `-ncmoe` is
llama-bench's spelling of `--n-cpu-moe`.
- **llama-bench prints its table only after all `-d` values finish.** No output does not mean hung.
- **Nothing else may use the GPU** during a measurement; stop llama-swap (`bench.sh` and
`/root/bench/moe/swapdown.sh` do this and restart it from a trap).
- **Check the build finished** before benchmarking it (library mtime earlier than the run start).
Never overlap a build with a big benchmark (host OOM).
- **Prompt-cache reuse is `timings.prompt_n`,** not `tokens_evaluated` (which is always the full
prompt length).
- **Per-process VRAM** from `/proc/<pid>/fdinfo/*` `drm-memory-vram` must be deduplicated by
`drm-client-id`; summing all fds double-counts shared buffers. `drm-memory-gtt` is system RAM, not
VRAM. Whole-card usage: `/sys/class/drm/card*/device/mem_info_vram_used`.
`GGML_VK_MEMORY_LOGGER=1` logs every Vulkan buffer alloc/free with a running device total to stderr.
- **The concat fast path is invisible to test-backend-ops** (F3); validate it with the knob, perf
logger timing, or byte-identical greedy generation (temperature 0, top_k 1, fixed seed,
`cache_prompt: false`).
- **Do not run `llama-cli` detached with closed stdin** on the pod: it can loop forever printing
prompts (it once wrote a 65 GB log). Use llama-bench `-d` or the server harness.
## 7. Known gaps and open risks
- `llama_context::restore_device()` ignores the `bool` returned by `restore_device_weights()`,
`moe_cache->restore_device()` and `memory->restore_device_buffers()`. On a restore OOM,
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
null buffer would crash. Only the mmproj path handles a failed restore.
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
- Router mode can spin at shutdown (F7); not used here.
- The unexplained ~1.6 GiB peak-VRAM spike in the first candidate `--perf` arm, seen twice (F9).
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
is an estimate; its memory effect is verified in production.
- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the
35B, only on the 0.8B (CPU and GPU). `/root/arms/diskcache-window2.sh` is ready (about 6 minutes of
downtime).
- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's
host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time
(token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that
shadow while it is cold.
- F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key
holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB.