docs: FORK.md and rebase runbook for the RX 580 fork

FORK.md documents every fork feature (what and why with measured numbers,
where, how it works, invariants, past failures, rebase hazards, validation),
the dead ends measured on this hardware, the production configuration and
the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation,
deploy, rollback and GPU-recovery procedure plus an operations reference.
The README fork section links both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 00:18:56 +02:00
1 parent 149a8cb0ae
commit bfbaf15b2a
3 files changed
+1258

No files matched your search

+837
View File
@@ -0,0 +1,837 @@
# FORK.md - the RX 580 llama.cpp fork
Maintainer documentation for this private fork. Read it before you rebase, change, or deploy
anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md),
which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-10-25):
- Branch `fable5/vram-ondemand-rebase-20261024`: upstream `3f03b1772` plus 27 fork commits.
- Production runs `6c480f7e7` (the first 25 commits). The 2 commits after it, `e085c347d` (concat
env knob) and `149a8cb0a` (`scripts/fork-validate.sh`), are validation-only and were not deployed.
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line (`git log --oneline origin/master..HEAD`). The commit messages are detailed and
are the primary record of each mechanism and measurement; read them with `git show`.
## Contents
1. Purpose and target
2. Commit map
3. Features
- F1 MoE routing-ids readback skip
- F2 GCN flash-attention mask_opt gate (and the shmem-staging note)
- F3 Vulkan concat tiled-transpose fast path
- F4 On-demand device residency (library side)
- F5 On-demand mmproj (multimodal encoder)
- F6 Cross-process VRAM arbiter (server side)
- F7 Server shutdown path
- F8 Cold-teardown guards
- F9 Benchmark harness and validation script
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
7. Known gaps and open risks
---
## 1. Purpose and target
The fork exists to serve large MoE models well on one specific machine:
- **GPU:** AMD RX 580 8 GB (Polaris, GCN gfx803), RADV Vulkan driver. llama.cpp reports
`fp16: 0` (RADV does not expose `shaderFloat16` on GFX8), no int-dot, no matrix cores, wave64.
PCIe 3.0 x16. RADV exposes no transfer-only queue family on this card.
- **Host:** Ryzen 5 3600 (6c/12t), 62 GB RAM, a Talos Kubernetes node shared with about 60 other pods
(CPU contention matters for decode numbers, see section 6). The GPU runs at a fixed 120 W power cap.
- **Primary model:** Qwen3.6-35B-A3B (unsloth UD-Q4_K_XL, 20.8 GiB), a hybrid MoE: 40 layers, 30 of
them gated delta-net (recurrent), 10 full attention, head_dim 256, 256 experts with 8 used. All
experts live in host RAM (`--n-cpu-moe 40`); only the backbone, KV, recurrent state and compute
buffers sit in VRAM.
- **Serving:** llama-swap under supervisord keeps several llama-server processes loaded at once
(the 35B main models, a Qwen3.5-4B "task" model OpenWebUI calls for titles and tags, gemma-4-26B
multimodal, others). They cannot all fit in 8 GB, so they time-share the GPU through the fork's
VRAM arbiter (F4-F7) without restarting processes, which would throw away the RAM-only prompt cache.
- **Priority:** prefill (prompt processing) speed first, decode second. Agentic use pushes large tool
results through the model, so long-context prefill is what users wait on.
What the fork adds, in short:
- three Polaris prefill wins (F1, F2, F3), all exact (token-identical to mainline);
- on-demand release/restore of everything a model holds in VRAM, plus a cross-process arbiter that
hands the GPU from one llama-server to another in well under a second for small models;
- fixes to the server's sleep, wake and shutdown paths that the arbiter exposed;
- a benchmark harness and a post-rebase validation script.
## 2. Commit map
Oldest first. "Feature" points at the section below.
| commit | subject | feature |
|---|---|---|
| c72cb424e | llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert | F1 |
| fb6a8a076 | ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes | F2 |
| ff2a96d2d | vulkan: record why flash-attn shmem staging stays off for GCN | F2 |
| 3df719e1e | ggml-vulkan: tiled transpose fast-path for concat with transposed source | F3 |
| 481be8b28 | server: on-demand VRAM sharing to time-share one GPU between models | F4, F6 |
| 17165fe2a | server: optionally evict the KV cache too (Phase 2 of VRAM sharing) | F4 |
| 958257cd6 | server: keep the sleep-wake path active when the VRAM arbiter is enabled | F6 |
| b9df854de | server: coordinate model load with the VRAM arbiter to avoid load-time OOM | F6 |
| 8d80f0673 | llama: guard buffer iteration against released device buffers | F8 |
| 2feb6aeb2 | server: restore weights/KV on wake, not at decode, so KV eviction is safe | F6 |
| bf2c1a699 | server: also free the compute-graph scheduler on KV eviction | F4 |
| 9857a45f0 | llama: evict recurrent/SSM state on device release | F4 |
| 9b555f1aa | ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc | F4 |
| ad27d4921 | llama: free backend compute scratch on cold; guard memory_breakdown | F4, F8 |
| 0a0f3ab6d | mtmd: release/restore the encoder weights from VRAM on demand | F4, F5 |
| 5e99ac198 | server: on-demand mmproj - free encoder VRAM on the text path | F5 |
| b7c131219 | docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends | docs |
| 5e429d4a9 | ggml-vulkan: skip the concat transpose fast path below one tile | F3 |
| 4c4486f8e | scripts: add the RX 580 benchmark harness | F9 |
| 878d1338f | server: replace the VRAM doorbell with a want-lock | F6 |
| 53f907771 | llama: skip the compute-buffer size check when the context is cold | F8 |
| 7ac2014a8 | llama: release/restore the MoE cache with the device weights | F4 |
| 54a7f534b | scripts: rx580-bench - use -lm none, document the MoE cache option | F9 |
| 11044b35e | server: run the shutdown handler off the signal-interrupted thread | F7 |
| 6c480f7e7 | server: release the GPU token promptly on shutdown | F7 |
| e085c347d | ggml-vulkan: env knob for the concat transpose fast path | F3 |
| 149a8cb0a | scripts: add fork-validate.sh, post-rebase validation of the fork | F9 |
---
## 3. Features
Every feature section uses the same headings: What and why; Where; How it works; Invariants that
must hold; Failure modes and incidents; Rebase hazards; Validation.
### F1. MoE routing-ids readback skip
**What and why.** With experts host-resident, each split uploads the experts it needs. To choose
them, the expert-copy callback reads the routing ids back to the host. The ids are produced on the
same GPU the upload goes to, so the readback forces a full pipeline flush: 47 per eval on a
48-layer MoE. At prefill batch sizes every expert is used anyway (2048 tokens x 10 experts over 256
experts), so the readback buys nothing. Measured on Laguna-S-2.1 118B IQ2_M, all experts on host
(before it was ported to the current callback): pp2048 +4.7% at depth 32k, +0.9% at 16k, flat at
depth 0, tg +3%.
**Where.** `src/llama-context.cpp`, `llama_context::sched_copy_experts` (registered with
`ggml_backend_sched_set_copy_callback` in the scheduler setup). Commit c72cb424e.
**How it works.** When `ggml_nelements(ids) >= 4 * n_expert`, it skips
`ggml_backend_tensor_get_async` + `ggml_backend_synchronize`, clears `st.ids_data` and sets
`st.used.assign(n_expert, true)`. The existing grouping then copies the whole expert tensor as one
range. Below the threshold the upstream readback runs unchanged. For Qwen3.6-35B (256 experts, 8
used) the skip applies to any ubatch of 128 tokens or more; decode (8 ids) never reaches it.
**Invariants that must hold.**
- Exactness: uploading an expert no id selects cannot change the result, because `mul_mat_id` only
reads the rows the ids point at. Any upstream change that makes unselected experts matter (for
example a kernel that reads every uploaded expert) breaks this argument.
- With `--moe-cache-mib` on, the all-true set still goes through `moe_cache->copy_experts`, which
reads the host slot maps (no device readback) and copies cached experts device-to-device. Still exact.
- `st.ids` caching stays coherent: a repeat call with the same ids tensor reuses the all-true set,
which is what the threshold would decide again; a different `n_expert` recomputes.
- The small-batch MoE-cache slot path (`llama_moe_cache::copy`) has its own readback and is not touched.
**Failure modes and incidents.** None in production. Historically the change lived in
`ggml_backend_sched_compute_splits` in `ggml/src/ggml-backend.cpp`; upstream #29943 (`6753a033f`)
deleted that block and moved expert copying into `llama_context::sched_copy_experts`. The original
cherry-pick therefore had nothing to apply to and was re-authored as c72cb424e. A `-X theirs`
resolution would have resurrected the deleted scheduler block.
**Rebase hazards.**
- Any change to `sched_copy_experts` or the `st` state struct (`ids`, `used`, `ids_data`).
- Upstream `701fd25b3` (MUL_MAT_ID id -1 skips work) left a `GGML_ASSERT(id >= 0 ...)` in the
readback branch; the skip branch reads no ids, so it is unaffected, but watch for upstream
semantics around negative ids.
- If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.
**Validation.** No dedicated check. `fork-validate.sh --ops` covers the MUL_MAT_ID kernel (not this
callback); `--perf` catches a prefill regression; `--moecache` exercises the cache path through this
callback. After a rebase that touches this function, also compare greedy output against the reference
build on the 35B.
### F2. GCN flash-attention mask_opt gate (and the shmem-staging note)
**What and why.** `mask_opt` skips fully-masked causal blocks and drops the per-block mask load+add
on fully-visible blocks. Upstream disables it on AMD GCN for head sizes <= 256. On Polaris attention
at depth is compute-bound on the softmax path (exp runs at about 1/4 VALU rate, no matrix cores), so
skipping that work is a real win. Measured on Qwen3.5-35B-A3B (head_dim 256): pp2048 +1% at depth 0,
+7.8% at 16k (176 -> 190), +12.3% at 32k (136 -> 153), growing with depth. Perplexity bit-identical
(5.1668 on and off).
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`, `ggml_vk_flash_attn`, the `use_mask_opt`
condition. Commit fb6a8a076. The shmem-staging note is a comment in `get_fa_tuning_params_scalar`
(commit ff2a96d2d, no behaviour change).
**How it works.** The GCN clause `HSK > 256 || HSV > 256` becomes `HSK >= 256 || HSV >= 256`. The
existing large-mask conditions (`nem1 >= 32`, `nem0 * nem1 > 32768`,
`nem0 >= block_cols * 16`) keep it off for decode.
**Invariants that must hold.**
- Only GCN with head size >= 256 is affected. Do not lower the threshold: relaxing it below 256 was
-18.5% at 16k on Laguna (head_dim 128).
- Upstream's `!use_sparse` term must stay in the condition (sparse flash-attention came in with
upstream `fc82583e6`).
- `shmem_staging` stays NVIDIA-only. Measured on GCN at head size 128: -6.7% at 16k, -7.4% at 32k.
The `kvsh` stride of `D/4+1` dwords is 4 mod 32, an 8-way LDS bank conflict on wave64 (the `+1`
padding is tuned for warp32), and the extra shared memory cuts occupancy. If anyone retries it,
fix the padding to +2/+3 first.
**Failure modes and incidents.** A `-X theirs` resolution in a trial rebase silently dropped
`!use_sparse` from the condition. No production incident.
**Rebase hazards.** Any upstream rewrite of `use_mask_opt`, of the FA tuning parameters, or of
architecture detection (`vk_device_architecture::AMD_GCN`). Upstream `733f57a99` added a path that
copies strided f16/bf16 K/V into `prealloc_x` scratch for flash-attention on AMD non-RDNA4 (so GCN
is included) when `neq1 >= 256`; measured at d32768 it adds about 68 MiB of VRAM and no visible
long-context prefill loss, but it has no env toggle, so its speed effect could not be isolated.
**Validation.** `fork-validate.sh --ops` (FLASH_ATTN_EXT, 5332 cases on Vulkan0) and `--perf`
(pp2048 at depth 16k and 32k is where this feature shows).
### F3. Vulkan concat tiled-transpose fast path
**What and why.** Delta-net layers build their conv input with `ggml_concat(conv_states,
ggml_transpose(qkv_mixed), 0)`. The generic concat shader reads the transposed source with an
uncoalesced stride: about 5.6 ms for a 16 MB concat on the RX 580 against about 130 us of memory
bandwidth. On Qwen3.6-35B the fast path alone is +14% pp8192 and accounts for more than all of the
fork's prefill win at depth 0 (the rest of the fork costs about 1% there). In the validation run on
Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 1945 us without.
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`: `ggml_vk_concat` (gate lambda
`src_transposed_2d` and the `s1_trans`/`s0_trans` checks) and `ggml_vk_concat_transpose_fastpath`.
The graph pattern comes from `llm_build_delta_net_base::build_conv_state` in
`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next); qwen4exp's `build_conv_state_at` uses
the same pattern. Commits 3df719e1e (fast path), 5e429d4a9 (decode gate), e085c347d (env knob).
**How it works.** For a concat along dim 0 where one source is stored transposed (`nb[1] ==
type_size`, `nb[0] == ne[1] * type_size`) and the other source and dst are contiguous, it issues two
dispatches: `pipeline_cpy_32`/`_16` copies the contiguous source into its dst columns, and
`pipeline_cpy_transpose_32`/`_16` (the shared-memory 32x32 tiled transpose in `copy_transpose.comp`)
writes the transposed source into the other columns, both with custom push constants and a dst
offset. It ends with `ggml_vk_sync_buffers`. Everything else falls through to the generic path.
The gate requires: f32 or f16 with all three types equal (quantized and integer types stay generic);
matching ne[1..3]; `ne[2] == 1 && ne[3] == 1` on the transposed source (so it fires only with one
sequence per ubatch, which `--parallel 1` gives); the 16-bit `doffset` bound; and
`ne[0] >= 32` (TILE_DIM) on the transposed source.
The decode gate (5e429d4a9): at one token the transposed source is a single column, so there is no
stride to fix, but the path still costs two dispatches and the unconditional `ggml_vk_sync_buffers`,
a full pipeline barrier (the generic path emits none). That barrier, not the half-empty transpose,
was nearly all of a -17% tg256 regression, because the path ran 30 times per decoded token on the
35B. The transposed source is `qkv_mixed` transposed, so its `ne[0]` is the ubatch token count:
prefill keeps the fast path, decode and small speculative drafts take the generic one. Measured
with the gate: decode back to parity (gated/clean 1.023), prefill win retained (gated/ungated 0.998,
pp8192 +11% over clean), generation byte-identical.
The knob (e085c347d): `GGML_VK_CONCAT_FASTPATH=0` forces the generic path, `=log` prints
`ggml_vk_concat: transpose fast path, transposed ne0=N` to stderr for every fast-path dispatch. It is
read once (function-local static). Not set in production.
**Invariants that must hold.**
- The fast path fires at prefill and never at decode on a delta-net model.
- Output is identical to the generic path (it is a pure copy/transpose).
- The `types_ok` gate keeps quantized types off the path: the dispatches are raw f32/f16 element copies.
**Failure modes and incidents.**
- The decode regression above shipped in the deployed fork before the gate existed. It looked
tg-neutral on the old upstream only because tg32's +-14% spread hid a 17% effect (section 6).
- If upstream changes `build_conv_state` (for example a `ggml_cont` on the transpose), the fast path
silently stops firing. Nothing crashes; prefill just loses about 14%. Only the concat check or a
perf A/B notices.
**Rebase hazards.**
- Pipeline renames: upstream `733f57a99` renamed `pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to
`pipeline_cpy_32`/`pipeline_cpy_16`. The pick applied cleanly and did not compile.
- Upstream `f172be756` made `ggml_vk_concat` non-static (declared in `ggml-vulkan-common.h`) and
split helpers into headers; the fast-path helper stays static.
- Shape changes in `build_conv_state` / `qkv_mixed` (`[ch, n_seq_tokens, n_seqs]` reshaped from a
contiguous `build_lora_mm` output), changes to `copy_transpose.comp` or `copy.comp` push constants,
or to `vk_op_unary_push_constants` / `get_misalign_bytes`.
- Upstream concat now handles quantized types; keep `types_ok`.
- The stock `test_concat` cases in `tests/test-backend-ops.cpp` never produce a transposed source and
always have `ne[2]`/`ne[3]` > 1, so `test-backend-ops -o CONCAT` does NOT exercise this path.
**Validation.** `fork-validate.sh --concat` (needs a build with the knob, i.e. this branch or
`--concat-build`): counts fast-path dispatches in a prefill run (must be > 0, all with ne0 >= 32)
and a decode run (must be 0 while CONCAT nodes did execute), then ties the unmodified candidate to
the fast path by CONCAT timing (candidate about equal to knob-on and at least 2x faster than
knob-off). Last run: 72 dispatches at prefill (all ne0=512), 0 over 594 CONCAT nodes at decode.
Also `--perf`.
### F4. On-demand device residency (library side)
**What and why.** Several always-loaded models must share 8 GB. Co-residence is impossible (the 35B
at `--n-cpu-moe 40` uses most of the card; the 4B task model needs about 2.5-3.3 GB). Restarting a
process to free VRAM throws away the RAM-only prompt cache: on Laguna a repeat turn went from a
21,960 ms re-prefill to 225 ms once the process stayed alive across a task-model call. So a model
must be able to drop everything it holds in VRAM and get it back without a reload.
Measured footprints:
- 4B: warm 3327 MB, weights released 724 MB, restored and correct.
- 4B at 32k ctx: cold weights-only 1750 MB (KV stays), cold with KV evicted 726 MB.
- gemma-26B: warm 7147 MB, cold with weights+KV but scheduler kept 718 MB, cold with the scheduler
freed too 51 MB.
- A cold 4B before the encoder work held 644 MB that turned out to be its mmproj (F5).
- Re-warm cost: about 0.6 s per cold-to-warm switch weights-only (tiny prompt: 1.98 s cold vs 1.40 s
warm); 2-3 s with KV eviction (it adds the KV D2H/H2D and a scheduler reserve).
**Where.**
- `src/llama-model.cpp`/`.h`: `llama_model::release_device_weights(keep_shadow)`,
`restore_device_weights()`, `weights_resident()`; per-buffer metadata
`llama_model::impl::weight_release` (`weight_release_slot`), filled in `load_tensors` and
`load_tensors_hadamard`.
- `src/llama-memory.h`: `llama_memory_i::release_device_buffers(keep_shadow)` /
`restore_device_buffers()` (default no-op). Implemented in `src/llama-kv-cache.cpp`
(`llama_kv_cache`) and `src/llama-memory-recurrent.cpp`; `llama_memory_hybrid` and
`llama_kv_cache_iswa` delegate to their children.
- `src/llama-moe-cache.cpp`/`.h`: `llama_moe_cache::release_device()`, `restore_device()`,
`resident()`, `impl::invalidate()`.
- `ggml/include/ggml-backend.h`, `ggml/src/ggml-backend-impl.h`, `ggml/src/ggml-backend.cpp`:
optional `ggml_backend_i::free_scratch` and `ggml_backend_free_scratch()`; Vulkan implementation
`ggml_backend_vk_free_scratch` in `ggml-vulkan.cpp`.
- `src/llama-context.cpp`/`.h`: `llama_context::release_device(evict_kv)`, `restore_device()`,
`discard_device()`, `device_resident()`, members `kv_device_evicted`, `device_discarded`; the
auto-restore at the top of `llama_context::decode`.
- `include/llama.h`: `llama_context_release_device(ctx, evict_kv)`, `llama_context_restore_device`,
`llama_context_discard_device`, `llama_context_weights_resident`.
- `tools/mtmd/clip.cpp`/`.h`, `tools/mtmd/mtmd.cpp`/`.h`: `clip_release_device`,
`clip_restore_device`, `clip_weights_resident`, `mtmd_release_device`, `mtmd_restore_device`.
- Commits 481be8b28, 17165fe2a, bf2c1a699, 9857a45f0, 9b555f1aa, ad27d4921, 0a0f3ab6d, 7ac2014a8,
6c480f7e7 (discard).
**How it works.** Tensor metadata (`ggml_context` structs) is durable; device buffers are ephemeral.
- Weights: on release, for each releasable buffer, capture a host shadow once (weights are
read-only), free the buffer, null every tensor's `buffer`/`data`. On restore,
`ggml_backend_alloc_ctx_tensors_from_buft` reallocates an identical layout (re-pointing the
tensors), the shadow is uploaded, usage is set to WEIGHTS. Only single-buffer, alloc-path, non-host
buffers are releasable: the host-pointer path (mmap or lazy-mapped), `no_alloc` dummy buffers and
host (`Vulkan_Host`, i.e. pinned RAM) buffers stay as they are.
- KV and recurrent state: same pattern, but the shadow is re-captured on every release (the data is
read-write) and dropped after restore.
- Scheduler: with `evict_kv`, `sched.reset()` frees the worst-case compute buffer and
`sched_need_reserve = true`; restore calls `sched_reserve()` (which goes through `sched_new()`,
re-registering the expert-copy callback and upstream's realloc callback).
- Backend scratch: with `evict_kv`, `ggml_backend_free_scratch()` on each backend. On Vulkan this
discards the pending compute context, synchronizes, destroys `prealloc_x`, `prealloc_y`,
`prealloc_split_k`, `prealloc_add_rms_partials`, `sync_staging`, and zeroes their sizes and the
`prealloc_y_last_*` tracking. They are reallocated lazily on the next compute. Other backends leave
the hook null (no-op).
- MoE cache (`--moe-cache-mib`, upstream #29887): released in both modes. The banks are freed and the
cache is emptied, not shadowed: every SLRU is rebuilt fresh, its `slot_map[il]` pointers are
re-pointed at the host slot-map tensors, and the host slot maps are cleared to 0xff (every slot
-1). On restore the banks are reallocated (which also re-inits their views), cleared to 0 and
marked WEIGHTS; the cache refills from the host experts.
- Multimodal encoder: `clip_release_device` / `clip_restore_device` use the weights pattern (shadow
once); no-op for a CPU-backed encoder. `mtmd_*` wrap the vision and audio contexts.
- `llama_context::release_device(evict_kv)`: synchronize, release weights, release MoE cache, and if
`evict_kv`: release memory buffers, free sched, free scratch.
`restore_device()`: weights, MoE cache, memory buffers, `sched_reserve()` if needed, then
`ggml_backend_sched_synchronize` so uploads finish before any compute.
`discard_device()`: like `release_device(true)` with `keep_shadow = false` everywhere (no host
copies), sets `device_discarded`; used only before exit.
- `decode()` calls `restore_device()` if `!device_resident()`, as a safety net. The server restores
explicitly on wake (F6), because some KV access happens before decode.
**Invariants that must hold.**
- Shadows are stored compactly in stable tensor-iteration order, skipping view tensors
(`view_src != NULL`), and restored with the same iteration. Never index a shadow by
`t->data - base`.
- `weight_release` stays parallel to `pimpl->ctxs_bufs`: every push to `ctxs_bufs` needs a matching
push to `weight_release` (non-releasable slots use `emplace_back()`).
- The caller synchronizes before any release; `release_device` does this itself.
- Restore order: weights, MoE cache, KV/recurrent, scheduler reserve, scheduler synchronize. The MoE
cache banks must be back before the scheduler is rebuilt.
- After a MoE-cache release no slot map may point at an expert: banks come back zeroed, so a stale
slot map silently reads zero weights.
- Every iterator over `ctxs_bufs` (model, KV cache, recurrent) must tolerate a null buffer, and every
use of `sched` must tolerate null while cold with `evict_kv` (see F8).
- `device_resident()` covers weights and MoE cache; it gates the early returns, the decode
auto-restore and `llama_context_weights_resident()`.
- `restore_device()` after `discard_device()` aborts (`GGML_ASSERT(!device_discarded)`).
- Greedy output is byte-identical across any number of release/restore cycles, and the prompt cache
survives (`timings.prompt_n` drops on a repeated prompt after a cycle).
**Failure modes and incidents.**
- The first shadow implementation indexed by `t->data - base` and produced garbage after restore
(layout and padding differ). Fixed by the compact iteration-order shadow.
- Restoring KV lazily at decode crashed iSWA models (gemma) with
`GGML_ASSERT(buf != NULL && "tensor buffer not set")`: `update_slots()` creates an SWA checkpoint
that reads the KV via `ggml_backend_tensor_get` before decode. Fixed by restoring on wake (2feb6aeb2).
- The recurrent state was left resident (default no-op), 201 MB on the 4B, until 9857a45f0.
- The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed
resident in a cold model until bf2c1a699 / ad27d4921.
- The MoE cache was not released at all before 7ac2014a8. Reallocating its banks without
invalidating would have produced silent wrong weights.
- In the 20261024 rebase `moe_cache_slru::init()` turned out to reset only the probation segment
(`head[1]`, `tail[1]` and `n_protected` survive), so re-running it on a used cache corrupts the
lists. `invalidate()` now assigns a fresh `moe_cache_slru()` before `init()`.
**Rebase hazards.**
- New push sites into `ctxs_bufs` (upstream's `load_tensors_hadamard()` added two in the 20261024
rebase; without matching `weight_release` entries `release_device_weights()` reads past the end).
- Changes to buffer creation in `load_tensors` (the `host_ptr_path` condition, `ctx_bufs`/`buf_ptrs`
bookkeeping; upstream replaced `ctx_buf_maps.emplace_back(ctx, buf_map)` with
`ctx_bufs.emplace_back(ctx, std::move(buf_ptrs))` in the 20261024 rebase).
- The `llama_memory_i` interface. New memory types that do not override `release_device_buffers`
keep their KV resident under `evict_kv`; at the time of writing that is `llama_kv_cache_msa`,
`_dsa`, `_dsa_iswa`, `_dsv4` and `llama_memory_hybrid_iswa`.
- MoE-cache internals (`groups`, `slru`, `slot_map`, `buf_host`, `devices`). Upstream `33d6135cb`
changed LRU to SLRU (`g.lru` -> `g.slru`); the pick applied cleanly and did not compile.
- `llama_context` members next to ours (upstream `n_reallocs_same/diff`, `copy_experts_info`),
`sched_reserve()` semantics, and the `decode()` signature (now `decode(const llama_batch_ext &)`).
- `ggml_backend_i` layout (`free_scratch` must stay the last member after `graph_optimize`; other
backends rely on zero-init) and the Vulkan context's prealloc field set (compare with
`ggml_vk_cleanup`).
**Validation.** `fork-validate.sh --arbiter`: `b-greedy-identical` (15 cycles each side equal to a
no-handoff reference), `b-kv-survives` (prompt_n drops after evict/restore, output identical),
`c-image-after-restore`. `--moecache MODEL` (needs a cache large enough to hit; 256 MiB on gemma-26B
had 0 hits and proves nothing, 2048 MiB had 6651 hits). The script does not check that VRAM actually
drops on release; check that by hand with `cat /sys/class/drm/card*/device/mem_info_vram_used`
before and after forcing a release.
### F5. On-demand mmproj (multimodal encoder)
**What and why.** A multimodal model's vision encoder sat in VRAM on every text request. With
`LLAMA_MMPROJ_ONDEMAND=1` the encoder lives in RAM and enters VRAM only for an encode. Measured on the
35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered
correctly, text after the image correct.
**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream renamed it from
`process_mtmd_chunk`), the `mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`,
`vram_go_cold`, `vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
**How it works.** After load the encoder is released (shadow captured). Before
`mtmd_batch_encode`, `mtmd_restore_device(mctx)`; if that fails (out of VRAM) and on-demand is set, it
releases the LLM backbone weights only (`llama_context_release_device(ctx_tgt, false)`: read-only
shadow, so a cheap free with no D2H, KV and prompt cache untouched) and retries. After the encode it
releases the encoder FIRST, then restores the weights, so the peak stays within VRAM.
**Invariants that must hold.**
- Free the encoder before re-uploading the weights.
- The donor is the weights, never the KV (the KV is read-write and would need a D2H every time).
- `vram_ensure_warm()` does not restore the encoder; only the encode path does.
**Failure modes and incidents.** Before this, a cold text model with an auto-downloaded mmproj
(`-hf` pulls one when the repo ships it) held 644 MB that no release path touched. The text-only 4B
task model now runs with `--no-mmproj`, which is the right fix for any model that never sees images.
**Rebase hazards.** Renames or restructuring of `encode_mtmd_chunk` and the mtmd init block in
`load_model`; `clip_ctx` fields (`buf`, `ctx_data`, `sched`, `no_alloc`).
**Validation.** `fork-validate.sh --arbiter` case `c-image-after-restore` (server A runs with
`--mmproj` and `LLAMA_MMPROJ_ONDEMAND=1`; a 128x128 red PPM must be answered "red" after A was cold,
and text right after must equal the reference). Post-deploy: one image request to gemma-26B.
### F6. Cross-process VRAM arbiter (server side)
**What and why.** Several llama-server processes under llama-swap share one GPU: exactly one is warm
(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production:
60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between
the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last
`fork-validate.sh` run: 30 handoffs, median 680 ms, max 1231 ms.
**Where.** `tools/server/server-context.cpp`: `vram_arena_open`, `vram_acquire_for_load`,
`vram_share_init`, `vram_should_yield`, `vram_want_acquire`, `vram_want_release`, `vram_take_token`,
`vram_ensure_warm`, `vram_go_cold`, `vram_share_exit`, `vram_share_shutdown`, the VRAM-only branch of
`handle_sleeping_state`, the `vram_ensure_warm()` call before decode in the slot update path, and
`server_res_generator` (`bypass_sleep`). `tools/server/server-queue.cpp`/`.h`: `on_should_yield`,
`start_loop` (`should_sleep`, idle tick, post-wake grace), `wait_until_no_sleep`, `is_stopping`.
Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
**How it works.**
- Enabled by `LLAMA_SLEEP_VRAM_ONLY` (any value). The arena directory is `LLAMA_VRAM_ARENA` (default
`/dev/shm/llama-vram`) with two files.
- `token.lock`: `flock(LOCK_EX)` is the baton. Rule: a process has device buffers iff it holds it.
- `want.lock`: a waiter holds it `LOCK_SH` while waiting and drops it once it owns the token. The
holder probes it with `flock(LOCK_EX | LOCK_NB)` on a SECOND fd: failure means somebody is waiting.
The probe needs its own open file description, because flock on the waiter's own fd would convert
that lock instead of conflicting with it. The kernel drops a dead waiter's lock, so nothing goes stale.
- Load: `load_model()` calls `vram_acquire_for_load()` before `common_init_from_params()`, so the
current holder releases before this model uploads anything. The loaded model stays warm holding the
token and serves its first request without a re-warm.
- Yield: `vram_share_init()` registers `vram_should_yield` with `queue_tasks.on_should_yield`.
`start_loop()`'s `should_sleep()` calls it on the loop thread, under `mutex_tasks`, only when the
task queue is empty and `update_slots` has returned. While the callback is set the idle wait ticks
every 10 ms instead of 1 s. After a wake the server holds the GPU for at least 100 ms
(`yield_grace_ms`). Sleep runs the sleeping callbacks; in VRAM-only mode `handle_sleeping_state(true)`
calls `vram_go_cold()` (release device, also the encoder; then `flock(LOCK_UN)`) instead of
upstream's full unload.
- Wake: a request calls `wait_until_no_sleep()`. `handle_sleeping_state(false)` calls
`vram_ensure_warm()`: take the token (register on want.lock, then poll `token.lock` with
`LOCK_EX | LOCK_NB` every 2 ms, aborting when `queue_tasks.is_stopping()`), restore ctx_tgt and
ctx_dft, and only then set `vram_cold = false`. The decode path calls `vram_ensure_warm()` again
(no-op when warm) and throws "server is shutting down" if it returns false.
- `server_res_generator` no longer bypasses the wake path when `--sleep-idle-seconds` is unset if
`LLAMA_SLEEP_VRAM_ONLY` is set, because the arbiter can put a server to sleep without idle-sleep.
- Queue fixes: `wait_until_no_sleep()` loops `while (sleeping && !stopping)` and re-asserts
`req_stop_sleeping` each time (with `notify_all`); the sleep wait also wakes on a non-empty task
queue; the idle `wait_for` only leaves the loop on a real task or termination.
**Invariants that must hold.**
- Resident iff holding `token.lock`. `vram_cold = false` is set only AFTER the restore, so the loop
cannot give the token away while still bringing weights up.
- Release only happens from the idle branch of `start_loop()`, on the loop thread, never inside
`yield_to_queue()` or while a slot is generating: a holder finishes its current generation before
it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both).
- The yield decision reads live kernel state at the point of use. Do not route it through a flag set
by another thread (see attempts 3-4 below).
- Every exit from `vram_take_token()` drops want.lock, or the holder yields forever.
- The probe fd is separate from the waiter fd.
- `flock` calls retry on EINTR: the server's signal handlers are installed without SA_RESTART.
- Restore on wake, before `update_slots()`, so KV eviction is safe for SWA checkpoints.
- Every model that can be warm on the GPU at the same time must use the same arena. A non-arbiter
process (a benchmark, a manually started server) can OOM the warm model. In production the env is
set for all of llama-swap, so all its children participate.
**Failure modes and incidents.**
- Wake-path hang (958257cd6): with no `--sleep-idle-seconds`, `bypass_sleep` skipped
`wait_until_no_sleep()`, so a server slept by the arbiter never woke and requests hung.
- Load-time OOM (b9df854de): weights were uploaded before the arbiter was active; loading gemma while
the 4B held VRAM could exceed 8 GB. After the fix the peak was 7771 MB.
- The doorbell design (replaced by 878d1338f) was edge-triggered: a waiter wrote
`<arena>/doorbell/<pid>` and the holder's warden thread turned the inotify event into a yield flag.
Rings were lost in three windows: while the holder restored (it advertised cold and discarded
foreign rings), while the holder loaded (the inotify watch did not exist yet, and the kernel does
not queue events for a missing watch), and when a cold waiter consumed a third process's ring.
Every loss was permanent: `flock(LOCK_EX)` had no timeout and production had no idle-sleep. The
blocked flock held `mutex_tasks`, so the whole server stopped answering while `/health` still
returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by
the second day).
- Failed fix attempt 1: moving `vram_cold = false` before the restore made the holder honour stale
rings and hand the token back 2-5 s after taking it without serving. Handoffs took 11m20s and 3m57s
against 5 s and 20 s before. Rolled back.
- Failed fix attempt 2: a re-ring loop (`LOCK_NB` + re-ring every ~1 s). Queued rings from an already
satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s
baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become
level-triggered.
- Failed attempts 3-4: want.lock was sound (mutual exclusion, kill -9 safety, load-window coverage),
but the warden thread still delivered its decision as a flag. `start_loop()` holds `mutex_tasks`
from `should_sleep()` through the callbacks to the wait, so the warden's `request_yield()` was always
admitted after the flag had been consumed, latching a release for the next wake (a server woke and
re-slept 138 ms later without serving). It also exposed the upstream lost wakeup in
`wait_until_no_sleep()`. Result: 0.494 s median, 1/60 stranded. The fix deleted the warden and
`request_yield`/`yield_requested` and probes inline in `should_sleep()`.
- Measured deviations that are required: the 10 ms idle tick (about 0.2% of one core per warm
server); a waiter that polls rather than retrying on a coarse grid (a 200 ms retry grid cost 0.503 s
against 0.336 s, because the holder only releases after the waiter asks); the 100 ms post-wake
grace (without it a permanent waiter caused a livelock: one request took 23.15 s over 138
sleep/wake cycles).
- Process lesson from attempts 1-2: validate an ordinary handoff latency A/B, not only the
pathological case. Two production deploys were burned before that rule existed.
- Lost diagnostic: the old periodic "still waiting for the GPU token after N s" warning is gone; a
stuck token now shows one `VRAM arbiter: waiting for the GPU token` line and then silence. Check
`/proc/locks` (see the runbook).
**Rebase hazards.** This is the most fragile area, because it sits inside upstream's server loop.
- `server_queue::start_loop`, `wait_until_no_sleep`, `process_new_tasks`, `yield_to_queue` and the
sleeping-callback vector. Upstream refactored sleep (`947fd9bb2`) and made decode run inside
`yield_to_queue` (`22b8e310b`); re-check after any such change that a release still cannot run
under a live graph.
- `handle_sleeping_state`, `load_model` ordering, the slot update path where decode happens,
`server_res_generator`'s `bypass_sleep`, renames in `server-context.cpp`.
- Anything that touches the KV in `update_slots()` before decode.
**Validation.** `fork-validate.sh --arbiter` with two real servers on a scratch arena:
`a-handoffs` (30 alternating, 0 stranded, 0 non-200, the server that answered holds `token.lock`),
`b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams:
want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in
F7 and a log scan for asserts and Vulkan errors. In production, `/proc/locks` must show the warm
model holding WRITE on `token.lock`.
### F7. Server shutdown path
**What and why.** llama-swap evicts models with SIGTERM, so shutdown happens all the time, often
while the arbiter is mid-release or mid-wait. Two bugs made shutdown keep the GPU token and stall
every other model.
**Where.** `tools/server/server.cpp`: `signal_handler`, `shutdown_pipe`, `struct shutdown_thread`,
`sig_thread` in `llama_server`. `tools/server/server-context.cpp`: `server_context::start_loop` (calls
`vram_share_exit()` after the loop returns), `server_context::terminate` (terminates the result queue
first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/server/server-queue.cpp`:
`server_queue::terminate` (sets atomic `stopping` before taking the mutex), `process_new_tasks`,
`server_response::recv_with_timeout`, `server_response::terminate`,
`server_response_reader::next`. Commits 11044b35e, 6c480f7e7.
**How it works.**
- The signal handler only writes the signal number to a self-pipe (async-signal-safe); a dedicated
thread reads it and runs `shutdown_handler`. The pipe fds are `FD_CLOEXEC`. If `pipe()` fails the old
direct call is used. The thread object is declared after the server objects, so it is joined (a 0
byte unblocks it) before they are destroyed. A second signal still calls `exit(1)`.
- `terminate()` sets `stopping` (atomic) before locking `mutex_tasks`, then stops the result queue so
HTTP handlers waiting for results get "server is shutting down" (HTTP 503, or an SSE error event);
`recv_with_timeout` checks `!running` before waiting and returns null.
- A token wait polls with `LOCK_NB` every 2 ms and aborts on `is_stopping()`; the aborted wake stays
cold, skips the restore, and the loop exits before `update_slots`.
- After `start_loop()` returns, `vram_share_exit()` frees all device buffers with
`llama_context_discard_device()` (no host shadows), releases the encoder, and only then unlocks
`token.lock`.
**Invariants that must hold.**
- Nothing that takes `mutex_tasks` runs in signal context.
- The token is released only after this process's VRAM is free (the next model may start uploading
the moment the lock drops).
- A shutting-down server never restores weights.
- SIGTERM in any state exits 0 within about 2 s and frees the token.
**Failure modes and incidents.**
- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside
a sleeping callback (under `mutex_tasks`, during a release); the handler's `terminate()` locked
`mutex_tasks` again on the same thread. The process wedged holding `token.lock`, every other model
queued behind it, and only SIGKILL worked. Reproduced 6/6 off production; 8/8 clean with the fix.
The first suspicion (the multimodal path) was wrong. gdb on the wedged process showed the main
thread in `futex(mutex_tasks)` with the `yield_to_queue` worker idle.
- Token kept on shutdown: with a request in flight the HTTP handler waited forever for results, main
blocked joining it, and the token was released only in the destructor after that join; with a
token wait in progress the blocking flock held `mutex_tasks`; and the old destructor unlocked the
token BEFORE `destroy()`, so the next model could upload into still-occupied VRAM.
- After the fix: off production SIGTERM mid-generation or mid-wait exits in about 0.2 s with exit 0
and frees the token in 5-70 ms; in production about 0.7-1 s.
**Rebase hazards.** `server.cpp`'s signal setup and `llama_server` object lifetimes;
`server_response` and `server_response_reader` (upstream `4e201388c` switched `recv_with_timeout` to a
fixed deadline: keep the order "check `!running`, return null, then `wait_until(lock, deadline)`");
`server_context::start_loop`/`terminate`. Router mode (`server-models.cpp`,
`server_models_sse_client::next`) only checks `should_stop()` and can spin at shutdown now that
`recv_with_timeout` returns null at once; llama-swap does not use router mode.
**Validation.** `fork-validate.sh --arbiter`: `d-term-in-release` (gdb breakpoint on
`llama_context_release_device`, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and
the token), `e-term-waiting` (exit 0 within 1.5 s, never took the token), `f-term-streaming`
(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run:
736 ms, 200 ms, 93 ms (token free in 28 ms), 80/157/156 ms. Post-deploy: a forced llama-swap eviction
must log `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token` and exit by itself.
### F8. Cold-teardown guards
**What and why.** A cold model has null buffers in its `ctxs_bufs` and, with `evict_kv`, a null
scheduler. Upstream code never had to consider either state.
**Where.** `src/llama-context.cpp`: `llama_context::~llama_context` (`sched &&` in the
compute-buffer size check), `llama_context::memory_breakdown` (`if (sched)` around the compute part;
upstream's `if (moe_cache)` breakdown block stays outside it). Null-buffer skips in
`llama_model::memory_breakdown`, `llama_kv_cache::{clear, memory_breakdown, total_size}`,
`llama_memory_recurrent::{clear, memory_breakdown, total_size}`. Commits 8d80f0673, ad27d4921, 53f907771.
**How it works.** Guards at the call sites. The asserts in ggml (`GGML_ASSERT(buffer)`,
`GGML_ASSERT(sched)`) are correct and stay; every other caller reserves the scheduler first.
**Invariants that must hold.** A server shut down while cold exits 0. Any new upstream code that
iterates `ctxs_bufs` or calls into `sched` on a path reachable while cold (shutdown, memory
breakdown, stats) needs the same guard.
**Failure modes and incidents.**
- `GGML_ASSERT(buffer)` on shutdown of a cold model (seen 7 times in production logs) from
memory-breakdown/size/clear on released buffers.
- `GGML_ASSERT(sched)` in `memory_breakdown` for a cold `evict_kv` model being torn down.
- `GGML_ASSERT(sched)` in the destructor's size check: every cold shutdown exited 134 instead of 0, so
under llama-swap an ordinary stop looked like a crashed child, left a backtrace in the log, and the
rest of the destructor never ran. Repro: arbiter env on, one request, force a yield with
`flock -s <arena>/want.lock -c 'sleep 6'`, then SIGINT while cold.
**Rebase hazards.** New iteration over `ctxs_bufs`, new `sched` users in teardown or breakdown
paths. Watch item: `llama_kv_cache::update()` takes `lctx->get_sched()`, which can be null; it is
safe today only because its single caller runs after `sched_reserve()`.
**Validation.** `fork-validate.sh --arbiter`: `h-shutdown-cold` (exit 0 while the other server holds
the token) and `log-scan`; `--moecache` `shutdown`.
### F9. Benchmark harness and validation script
**What and why.** The measurement method on this box had to be rediscovered several times (section
6). The harness encodes it; the validation script encodes the checks that each past incident
required, so a rebase can be validated by someone with no context.
**Where.** `scripts/rx580-bench/` (`run.sh` workstation wrapper; `bench.sh` pod orchestrator;
`ppbench.py` corpus prefill through llama-server; `lbsweep.sh` llama-bench sweep; `summarize.py`;
`README.md` with methodology and reference values). `scripts/fork-validate.sh`. Commits 4c4486f8e,
54a7f534b, 149a8cb0a.
**How it works.**
- `rx580-bench`: stops llama-swap, waits for an idle GPU, runs fixed slices of `pan-tadeusz.txt`
(4096/16384/32768 tokens, byte-identical across builds) through `/completion` with
`cache_prompt: false`, plus a llama-bench sweep, interleaved across arms and rounds; restarts
llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]`:
runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under
/tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets inherited
`GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check flag is
`--ops --concat --arbiter`. See the header of the script for what each check proves.
**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` refuses, and any
other llama process blocks it). `--perf` gates prefill at -5% per depth and peak VRAM at 95% of the
card; decode is reported, not gated (CPU-bound and noisy here).
**Failure modes and incidents.** Upstream #28334 removed `--mmap`/`--no-mmap`/`--mlock` and
llama-bench's `-mmp`: the server refused to start and llama-bench silently produced no rows until
54a7f534b switched to `-lm none`. The perf and MoE-cache checks leave tens of GB of model files in page
cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024
deploy (see the runbook).
**Rebase hazards.** CLI flag renames in `common/arg.cpp` and `tools/llama-bench`; log line texts the
script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`,
`no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`,
`llama_moe_cache: .* hits =`, `N/M tests passed`, `Backend Vulkan0: OK`); the llama-bench markdown
table layout.
**Validation.** Known gaps of the script: it does not check that VRAM drops on release; peak VRAM is
recorded per arm, not per test (one candidate arm once showed an unexplained 1.6 GiB spike that did
not repeat); the SIGTERM-in-release breakpoint is at function entry, not after the host-shadow
capture; nothing covers F1 directly.
---
## 4. Dead ends
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
without a new reason.
| idea | result | mechanism |
|---|---|---|
| Weight pinning (`GGML_CUDA_REGISTER_HOST` and the host-register chain, from a third-party CUDA fork whose +64% was on an RTX 3060) | +17% pp2048 in isolated llama-bench (252 -> 294), fails in serving | RADV cannot import file-backed mmap pages; the pre-stage fallback fails (`ErrorOutOfDeviceMemory` for the ~18 GB expert region) or doubles host RAM. `-lm none` serves faster (Qwen3.6-35B 278 -> 286). Dropped in the 2026-09-09 rebase. |
| Expert-upload prefetch (`GGML_SCHED_PREFETCH_EXPERTS`) | -20% pp (other runs -9% to -22%), no tg benefit | Slots allocate in host-pinned GTT, not VRAM, so it stages host to host and still pays the PCIe read. `=1` means the default 3 slots. Dropped. |
| `GGML_VK_ASYNC_USE_TRANSFER_QUEUE=1` | flat (-0.1%), byte-identical | GCN is excluded from `prefers_transfer_queue` and RADV exposes no transfer-only queue family here; the per-layer upload-then-compute order leaves nothing to overlap. A standalone Vulkan microbench did show 60-73% of an H2D copy on queue family 1 hidden behind compute, so double-buffered prefetch inside one backend context remains an untested idea (estimated +10-15%, unproven). |
| Fractional `-ncmoe` | no measured win | Config only uses integers (22/40/48); upstream refactored `-ncmoe` into `llm_add_n_cpu_ffn_overrides`. Dropped. |
| MoE cache `--moe-cache-mib` 512 / 1536 / 2304 (Qwen3.6-35B, ncmoe 40, ctx 131072) | tg256@d4096 -42% / +5% / +12%; prefill +1-3%; idle VRAM 6.15 / 7.19 / 7.96 GB vs 5.66 | Small caches thrash (8-of-256 routing over 40 layers) and add a per-layer ids readback; only ~2.3 GiB pays and leaves no VRAM for a long KV. Not enabled. |
| MTP speculative decoding, Qwen3.6-35B | decode -39% (n1), -37% (n2), -50% (n3), -44% (n4), best -19% (n2, p-min 0.75); prefill -3 to -6% | A spec step costs ~180 ms vs 52 ms for plain decode; a 2-token verify alone is only 1.4x a 1-token step, so ~100 ms/step is draft machinery. The nextn layer adds 1.25 GiB VRAM. Entries removed. |
| MTP, gemma-4-26B-A4B | decode -57.9% (n1), prefill pp4k -34% | Same mechanism (~210 ms vs ~50 ms per step); MTP needs `--swa-full`, which alone costs -10.5% decode, -32% pp4k and saturates VRAM. Entries removed. |
| MTP, gemma E4B | decode -10.9% | Acceptance 0.41-0.67. The +19.5% pp seen with it comes from `--swa-full`, not MTP. |
| `--parallel 1` on gemma | no effect (-1.0%, +2.1%) | The Qwen gain is probably tied to its recurrent state being sized per sequence (unverified). |
| `-ub 4096 -b 4096` (Laguna) | flat (90.0 vs 90.9) | Fewer expert uploads cancelled by intra-ubatch attention growth. |
| mul_mat_id tile-padding theory (larger/odd `-ub`) | predicted 2.00x waste and a 1.94x cliff; measured +3.8% and -5.6%, no sawtooth | Theory dead; kills the mul_mat_id tile-selection shader work (an unbuilt, unverified WIP exists on branch `fable5/mmid-tile-wip`, with an unresolved correctness concern in its row_ids fill). |
| `-b` sweeps | non-monotonic (2048/8192 high, 4096/12288 low) | `-b 2048` is already at the top. |
| `-ncmoe` below 40 on the 35B | 34: +2.9% pp but 7.23 GiB peak at tiny context; 28: better at short context, worse at depth (137 vs 170 t/s at d32k); <= 24 collapses (~122) | Resident experts are evicted by KV/activation growth; cannot survive a 128k context. |
| `-ncmoe` 44 on Laguna | +1.5% | Does not fit at 64k context. Keep 48. |
| fp16 on Polaris (Mesa 26.1.6 + `radv_enable_float16_gfx8=true`, `fp16: 1`) | pp2048 -4.7%, -5.3% at d16k, tg +4% | Hurts prefill (the priority). Mesa 26.1.6 alone: +3.3% prefill, below the 5% bar; adopting it needs a pod image change. The pod's Mesa 25.0.7 has no such option. |
| KV quant on Qwen (q4_0, q8_0) | f16 beats q4_0 by +40% pp at 64k; q8_0 170 -> 128 t/s at depth; q4_0 with upstream PR #30738 still -16% vs f16 at d32k (133.6 vs 159.4) | No int-dot/fp16 here, so dequant is pure compute; the hybrid model's KV is small anyway. Keep f16. (PR #30738 was +15% at d16k on gemma-26B with q4_0 KV; not merged upstream as of the 20261024 rebase.) |
| `-fa 0` | 5x slower at depth (170 -> 35 at d32k) | Keep flash attention on. |
| FA scalar `block_rows` 16 -> 32 | 3.5x regression at d32k (135 -> 38) | ~42 KB shmem per workgroup at head_dim 256, one workgroup per SIMD, occupancy collapse. |
| `GGML_VK_DISABLE_GRAPH_OPTIMIZE=1` | restores tg32 at n_ctx=32 (10.44 -> 15.61), nothing at real depth | Upstream #27812's view-alias fix makes KV views look dependent and collapses barrier-free batches; only visible when per-node work is tiny. It is a correctness fix; do not disable it in serving. |
| `--no-mmap`/`-lm none` for Laguna-S 118B | does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. |
| Laguna `--cache-reuse`, `--swa-full`, `-t 12`, KV quant | ruled out | `--cache-reuse` is force-disabled for iSWA when base and SWA sizes differ. |
| CPU-only task model (`-ngl 0`) to avoid handoffs | rejected | At `--ctx-size 8192` titles could face 8k tokens of CPU prefill. The arbiter is the answer. |
The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256
tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.
## 5. Configuration that matters
`/root/config.yaml` (llama-swap) and `/root/supervisord.conf` on the pod are the source of truth;
this section explains the choices. Values as of the 20261024 deploy.
Qwen 35B-A3B MoE entries (macro `moe_offload_args`, applied to all 35B-A3B variants):
`--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1`,
context 131072.
- `-lm none`: load weights into RAM (upstream removed `--no-mmap`; the old spelling makes the server
refuse to start). Faster in serving than mmap and recommended for `--n-cpu-moe`. Exception:
Laguna-S, which must use mmap (GTT limit, section 4).
- `-ub 2048 -b 2048`: the biggest prefill lever. ub512 = 97 t/s, ub1024 = 159, ub2048 = 254 on the
35B; the per-eval expert upload is amortized over the ubatch.
- `--n-cpu-moe 40`: all experts on host. Lower values are faster at short context but run out of VRAM
at long context (section 4). Measured idle VRAM at ctx 131072: 5.66 GB.
- `--parallel 1`: +22.9% decode on the Qwen 35B (15.5 -> 19.1 t/s), prefill +0.3-1.2%, +64 MiB VRAM.
Without it the server creates 4 slots and a unified KV. Two concurrent requests queue on slot 0.
With one slot a ubatch never holds more than one sequence, which the concat fast path's
`ne[2] == 1` condition (F3) needs.
- `-t 6`: six physical cores; more threads hurt decode.
- `-fa 1`: essential (section 4).
- f16 KV (no `-ctk/-ctv`): section 4.
Other models:
- gemma-4-26B-A4B: macro `moe_offload_gemma`, same as above but `--n-cpu-moe 22` (8 layers of experts
resident; gemma's SWA keeps KV small at depth). ncmoe sweep pp2048: 30 = 262, 24 = 266, 18 = 272,
14 = 157 (VRAM collapse); 22 keeps headroom for 128k context (7.40 of 8 GB at 128k).
Its KV cache is `-ctk q4_0 -ctv q4_0`, set in the `gemma4_sampling` / `gemma4_nothink_sampling`
macros (not in `moe_offload_gemma`); note gemma has no `--parallel 1` (no measured gain there).
- Small models that fit in VRAM (Qwen3.5-4B, gemma E4B/E2B and similar) use auto-fit
(`common_args`); forcing experts to host would slow them.
- The 4B task model runs with `--no-mmproj` and ctx 8192, and is preloaded.
- Laguna-S-2.1 118B IQ2_M (configured as `Laguna-S-2.1-IQ2_M`, matrix var `lag`): mmap, `--n-cpu-moe 48 -b 2048 -ub 2048 -fa 1 --parallel 1`,
ctx 65536. Its exclusivity set must allow the task model (`lag & q4task`); making it exclusive with
the task model killed it on every title request and forced full re-prefills.
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
`/root/supervisord.conf`:
- `LLAMA_SLEEP_VRAM_ONLY=1`: enables the arbiter and VRAM-only sleep (F6).
- `LLAMA_VRAM_ARENA=/dev/shm/llama-vram`: the shared arena. Every co-loaded GPU model must use it.
- `LLAMA_SLEEP_EVICT_KV=1`: cold models also free KV, recurrent state, scheduler and scratch (F4),
so the warm model gets nearly the whole card. Costs a 2-3 s re-warm instead of ~0.6 s.
- `LLAMA_MMPROJ_ONDEMAND=1`: encoder in VRAM only during encodes (F5). No-op for models without an mmproj.
- No `--sleep-idle-seconds`: release is purely demand-driven via want.lock.
- `GGML_CUDA_REGISTER_HOST=1` may still be present from the pinning era; it is a no-op with `-lm none`.
Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`,
`GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`.
Machine-level:
- GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted
every second; default/max cap 145 W). The two GPU hangs of 2026-10-09 happened before the cap was
fixed (it used to switch between 120 and 145 W by temperature); heavy runs since have been clean.
- Kernel argument `amdgpu.lockup_timeout=0,120000,0,0` (compute ring 120 s) via Talos
`machine.install.extraKernelArgs`. Without it long 64k-context submissions hit the compute-ring
watchdog (`ring comp_1.x.0 timeout`).
- Upstream changed the default `--port` from 8080 to 9931; the config passes `--port ${PORT}`
explicitly, so it is unaffected.
## 6. Measurement methodology and traps
- **Cross-run variance is about 7%.** llama-bench's within-run error bars understate cross-invocation
variance by about 14x (the same config read 276.7 +/- 0.9 and 296.4 +/- 0.6 in two invocations).
Put every comparison you draw a conclusion from inside one llama-bench invocation
(comma-separated sweeps). Two builds cannot share an invocation, so interleave them over at least 3
rounds and compare medians. Treat any cross-build delta under about 5% as unproven.
- **Alternate arm order.** A fixed order gave the later slot about a 9% advantage and nearly produced
a false regression finding. Use orders like ref, cand, cand, ref.
- **Node CPU contention.** The node runs about 60 pods; llama-bench once got 4.67 of 12 hardware
threads while asking for 6. With `--n-cpu-moe 40` every MoE layer decodes on the CPU, so stolen
threads map onto tg almost directly. Prefill is nearly immune (held +-0.3% while tg32 swung +-14%).
Expect occasional catastrophic tg samples; use medians, never means. Historical tg figures taken
under different load are not comparable.
- **Never use tg32.** At the default depth (n_ctx 32) decode is launch/barrier bound and measures
command-stream overhead: two upstream commits read -36% at tg32 and were at parity at tg256 (and +7%
at d512). Use tg256 (or tg128) with an explicit depth (`-p 0 -n 256 -d 4096`). tg is not monotonic in
n_ctx on this box (n_ctx 256 measured slower than 32 and 544), so only compare same-depth cells.
- **`-b Vulkan` vs `-b Vulkan0`.** `test-backend-ops -b Vulkan` matches nothing, runs 0 cases and
still prints OK with exit 0. The backend is `Vulkan0`. Always check the case count.
- **ubatch larger than the prompt never fills.** With `-ub 2048` use prompt lengths that are
multiples of 2048. A partial trailing ubatch costs a full expert upload.
- **`-tb` is not a llama-bench flag** (server only). Any bad flag makes llama-bench print usage and
exit, which looks like a run that produced no rows. Always check the row count. `-ncmoe` is
llama-bench's spelling of `--n-cpu-moe`.
- **llama-bench prints its table only after all `-d` values finish.** No output does not mean hung.
- **Nothing else may use the GPU** during a measurement; stop llama-swap (`bench.sh` and
`/root/bench/moe/swapdown.sh` do this and restart it from a trap).
- **Check the build finished** before benchmarking it (library mtime earlier than the run start).
Never overlap a build with a big benchmark (host OOM).
- **Prompt-cache reuse is `timings.prompt_n`,** not `tokens_evaluated` (which is always the full
prompt length).
- **Per-process VRAM** from `/proc/<pid>/fdinfo/*` `drm-memory-vram` must be deduplicated by
`drm-client-id`; summing all fds double-counts shared buffers. `drm-memory-gtt` is system RAM, not
VRAM. Whole-card usage: `/sys/class/drm/card*/device/mem_info_vram_used`.
`GGML_VK_MEMORY_LOGGER=1` logs every Vulkan buffer alloc/free with a running device total to stderr.
- **The concat fast path is invisible to test-backend-ops** (F3); validate it with the knob, perf
logger timing, or byte-identical greedy generation (temperature 0, top_k 1, fixed seed,
`cache_prompt: false`).
- **Do not run `llama-cli` detached with closed stdin** on the pod: it can loop forever printing
prompts (it once wrote a 65 GB log). Use llama-bench `-d` or the server harness.
## 7. Known gaps and open risks
- `llama_context::restore_device()` ignores the `bool` returned by `restore_device_weights()`,
`moe_cache->restore_device()` and `memory->restore_device_buffers()`. On a restore OOM,
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
null buffer would crash. Only the mmproj path handles a failed restore.
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
- Router mode can spin at shutdown (F7); not used here.
- Two code comments are stale: the arbiter block comment in `server-context.cpp` says the KV cache is
never evicted (it is, with `LLAMA_SLEEP_EVICT_KV`), and `vram_go_cold` says the encoder is rebuilt by
`vram_ensure_warm()` (it is restored only at encode time).
- The README fork section says "the two code changes below are auto-on"; the fork now has more code
changes than that. Treat this file as the complete list.
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
+4
View File
@@ -19,6 +19,10 @@
## This fork - Polaris / GCN tuning for large MoE models
Maintainer documentation for this fork (every custom feature, its invariants and rebase hazards, the
measured dead ends, and the rebase/deploy procedure): [FORK.md](FORK.md) and
[docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md).
Changes and measurements for running large MoE models with their experts offloaded to system RAM
(`--n-cpu-moe`) on an old GCN card. The two code changes below are auto-on, need no flag, and are
**token-identical** to mainline. Everything else here is tuning guidance.
+417
View File
@@ -0,0 +1,417 @@
# Fork rebase and deploy runbook (RX 580 fork)
Step-by-step procedure for moving the fork onto a new upstream and putting it into production, as it
has evolved over the 2026-09-09, 2026-10-09 and 2026-10-24 rebases. Read [FORK.md](../FORK.md) first:
it lists every fork feature with its invariants and rebase hazards, and this runbook refers to it.
Ground rules:
- **Never build on the workstation.** All builds run on the RX 580 pod. Locally, rely on careful reading.
- **Never resolve a conflict with `-X theirs` or `-X ours`.** Resolve every hunk by hand.
- **"Applied cleanly" is not "correct".** Re-read every clean pick against the new upstream.
- **Never `pkill -f` on the pod.** It matches the `kubectl exec` shell and kills it. Kill by explicit PID.
- **Never leave the pod not serving.** Anything that stops llama-swap must restart it from a trap.
- Keep the previous production branch untouched as the safety net. Do not push for the owner; the owner
reviews and pushes.
Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a compiler; 5. Build on
the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU;
11. Operations reference.
---
## 1. Survey upstream
Local remotes: `origin` is upstream (github.com/ggerganov/llama.cpp), `gitea` is the fork
(git@gitea.lumpiasty.xyz:Lumpiasty/llama.cpp.git, SSH push from the workstation). On the pod,
`origin` is the same gitea repository over HTTPS (fetch only).
```sh
git fetch origin
OLD_BASE=$(git merge-base origin/master HEAD) # the old upstream base (still an ancestor of the new master)
OLD_TIP=$(git rev-parse HEAD)
git log --oneline "$OLD_BASE"..HEAD # the fork commits, oldest last
git log --oneline "$OLD_BASE"..origin/master | wc -l # size of the upstream delta
```
Then look at upstream changes in the areas FORK.md lists as hazards:
```sh
git log --oneline "$OLD_BASE"..origin/master -- \
src/llama-context.cpp src/llama-context.h src/llama-model.cpp src/llama-model.h \
src/llama-kv-cache.cpp src/llama-kv-cache.h src/llama-kv-cache-iswa.cpp \
src/llama-memory.h src/llama-memory-recurrent.cpp src/llama-memory-hybrid.cpp \
src/llama-moe-cache.cpp src/llama-moe-cache.h src/models/delta-net-base.cpp \
ggml/src/ggml-vulkan/ ggml/src/ggml-backend.cpp ggml/src/ggml-backend-impl.h ggml/include/ggml-backend.h \
tools/server/server-queue.cpp tools/server/server-queue.h tools/server/server-context.cpp tools/server/server.cpp \
tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/
```
For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past
decisions:
- 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional `-ncmoe` (all measured
losses or no win on this card, FORK.md section 4).
- 2026-10-09: the routing-ids readback skip had nothing to apply to (upstream #29943 deleted its block
in `ggml-backend.cpp`); it was re-authored in `llama_context::sched_copy_experts`. Upstream's new MoE
cache (#29887) needed a new fork commit so the VRAM arbiter releases and invalidates it.
- Also check CLI changes that break production config or the scripts (upstream #28334 removed
`--no-mmap`/`--mmap`/`--mlock` and llama-bench `-mmp`; the default `--port` changed to 9931).
## 2. Trial rebase in a scratch worktree
Measure the conflicts before doing the real thing:
```sh
git worktree add /tmp/trial-rebase origin/master
cd /tmp/trial-rebase
git cherry-pick "$OLD_BASE".."$OLD_TIP" # stop at each conflict, note file + hunk, then:
git cherry-pick --abort # or resolve roughly to see the next one
cd - && git worktree remove --force /tmp/trial-rebase
```
Write down, per commit: clean / conflict (which file, which hunk, what both sides did). That list
becomes the brief for the real rebase.
## 3. The real rebase
```sh
NEW=fable5/vram-ondemand-rebase-$(date +%Y%m%d)
git switch -c "$NEW" origin/master
git cherry-pick <first-fork-commit> # one at a time, in the original order
```
Rules:
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
trailer the owner asks for.
- Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms.
- Fold each fix into the commit it belongs to so every commit builds on its own:
`git commit --fixup=<sha>` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`.
- Leave the previous production branch untouched.
**What `-X theirs` silently broke in a trial (2026-10-09).** It "applied" every commit and produced
broken code:
- resurrected the scheduler readback block upstream had deleted from `ggml-backend.cpp`;
- dropped `!use_sparse` from the GCN `use_mask_opt` condition;
- dropped upstream's `copy_experts_info` (compile error);
- dropped upstream's `is_empty()` in the recurrent memory;
- dropped upstream's MoE-cache memory breakdown block;
- left a static/extern mismatch with the new `ggml-vulkan-common.h` declarations (compile error).
The correct resolutions in that rebase: keep upstream's `n_embd` width check plus our decode
auto-restore; `copy_experts_info` plus our `kv_device_evicted`; `is_empty()` plus our `dev_shadows`;
upstream's `if (moe_cache)` breakdown block outside our `if (sched)` guard; make `ggml_vk_concat` and
`ggml_backend_vk_free` non-static to match the new header while our helpers stay static.
**Clean picks that were wrong (2026-10-24).**
- Concat fast path (3df719e1e): applied cleanly, did not compile. Upstream `733f57a99` renamed
`pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to `pipeline_cpy_32`/`pipeline_cpy_16` (bitwise copies,
same push constants).
- MoE cache release/restore (7ac2014a8): applied cleanly, did not compile (upstream `33d6135cb` changed
LRU to SLRU, `g.lru` -> `g.slru`) and had a runtime bug: `moe_cache_slru::init()` resets only the
probation segment, so re-initialising a used cache corrupts its lists. `invalidate()` now assigns a
fresh `moe_cache_slru()` first.
- VRAM sharing core (481be8b28): besides its one conflict, upstream's new `load_tensors_hadamard()`
pushes two more entries into `pimpl->ctxs_bufs`. Without matching `weight_release.emplace_back()`
calls, `release_device_weights()` reads past the end of `weight_release`. Nothing conflicted.
- Conflicts that round: `ctx_buf_maps.emplace_back` replaced upstream by `ctx_bufs.emplace_back(ctx,
std::move(buf_ptrs))` (keep upstream's line, our release-slot block before it); `llama-context.h`
(keep upstream's `n_reallocs_same/diff` beside our `kv_device_evicted`); `recv_with_timeout`
(upstream `4e201388c` uses a fixed deadline: keep our order "check `!running`, return null", then
`wait_until(lock, deadline)`).
**Semantic re-checks to do for every rebase, clean or not** (details in FORK.md):
- F1: `sched_copy_experts` and its `st` struct unchanged in meaning.
- F2: `use_mask_opt` still contains upstream's terms plus our `>= 256`.
- F3: `build_conv_state` still emits `ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0)` with
`qkv_mixed` contiguous `[ch, n_seq_tokens, n_seqs]`; pipeline names; `types_ok` still excludes
quantized types.
- F4: every `ctxs_bufs` push in `src/llama-model.cpp` has a `weight_release` push
(`git grep -n "ctxs_bufs.emplace_back\|weight_release" src/llama-model.cpp`); list memory types
(`git grep -n "public llama_memory_i" src/`) and note any without a release override; MoE-cache
internals; `sched_reserve()` still goes through `sched_new()` and re-registers callbacks;
`free_scratch` still the last `ggml_backend_i` member; Vulkan prealloc fields match `ggml_vk_cleanup`.
- F6/F7: `server_queue::start_loop` still evaluates `should_sleep()` only from the idle branch, never
inside `yield_to_queue()`; `handle_sleeping_state`, `server_res_generator`, `load_model` ordering;
signal setup in `server.cpp`.
- F8: new iterators over `ctxs_bufs` or users of `sched` on teardown paths.
- Scripts: flag spellings used by `scripts/rx580-bench/*` and `scripts/fork-validate.sh`, and the log
lines `fork-validate.sh` greps for (FORK.md F9).
- Production config: every flag in `/root/config.yaml` still exists in `common/arg.cpp`.
## 4. Checks without a compiler
Before handing the branch over for a build:
```sh
git diff origin/master..HEAD | grep -n '^[+-]\(<<<<<<<\|=======\|>>>>>>>\)' # no conflict markers
git diff origin/master..HEAD | grep -nP '^\+.*[^\x00-\x7F]' # ASCII only in added lines
git range-diff "$OLD_BASE".."$OLD_TIP" origin/master..HEAD # patch vs patch
```
- The range-diff must show only the edits you made on purpose. In 2026-10-24 the whole patch was
identical to the old one except for 4 deliberate edits.
- Brace balance per changed file: the change in `{`/`}` balance from new base to new tip should equal
the change from old base to old tip.
- Every identifier in added lines must exist on the new master or be defined by the fork. This catches
a renamed function but not a renamed member (like `lru` -> `slru`); re-read member accesses by hand.
- Write down every spot where you would normally lean on the compiler and hand that list to whoever
builds.
Then the owner pushes the branch to gitea.
## 5. Build on the pod, in a scratch worktree
Production keeps serving during this step.
```sh
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '
set -e
B=fable5/vram-ondemand-rebase-YYYYMMDD
W=/root/arms/rebYYYYMMDD
git -C /root/llama.cpp fetch origin "$B"
git -C /root/llama.cpp worktree add "$W" FETCH_HEAD
cd "$W"
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF
nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8
'
```
- Build only the targets you need; the full `all` target fails on the UI assets (needs npm/network).
If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed).
- `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes.
- Report compile errors verbatim against the list from step 4.
- Do not overlap a build with a large benchmark (host OOM). Disk is limited (about 22 GB free; each
build tree is about 700 MB); remove scratch worktrees you no longer need with
`git -C /root/llama.cpp worktree remove <dir>`.
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
holding a `kubectl exec` open for hours.
## 6. Validate with `scripts/fork-validate.sh`
Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of
the script describes each check.
```sh
# phase 1, beside production (needs >= 1.5 GiB VRAM free for --arbiter; it refuses otherwise)
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat --arbiter
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
cat > /root/arms/fv-perf.sh <<'EOF'
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf \
--moecache /root/.cache/huggingface/hub/<gemma-4-26B-A4B snapshot>/<file>.gguf
EOF
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
```
- `/root/bench/moe/swapdown.sh <script>` (pod-only helper, not in the repo) stops llama-swap, waits for
an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints
`SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200.
- The `--concat` check needs a build with the `GGML_VK_CONCAT_FASTPATH` knob (in the fork since
e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`).
- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits,
which proves nothing).
- Phase 2 costs about 27 minutes of downtime and leaves tens of GB of model files in page cache.
- Last full run (2026-10-24, 33 PASS, 0 FAIL): perf candidate vs reference pp2048 -4.1%, -2.2% at d16k,
-0.7% at d32k (gate is -5%), tg256@d4096 -0.2%.
- Manual checks the script does not do: VRAM actually drops when a server goes cold
(`cat /sys/class/drm/card*/device/mem_info_vram_used` before and after); greedy output of the 35B
against the reference if `sched_copy_experts` changed.
- Any arbiter change: A/B the ordinary handoff latency against the current binary, not only the
pathological case. Two deploys were lost by skipping that.
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
## 7. Deploy
Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 minutes.
1. **Rollback pointer.** Record the current production commit in the next free pointer file:
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 9 (holds 5751c977b)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-10.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea):
```sh
cd /root/llama.cpp
git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD
git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD <validated-commit>
```
3. **Memory.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM victim. With
`-lm none` each load reads the GGUF through the page cache, and a recent benchmark leaves tens of GB
of model files cached. During the first 2026-10-24 deploy the Talos OOM controller killed the whole
pod 17 s into verification (cause likely, not proven, memory pressure). Before stopping llama-swap,
evict the model files' page cache (unprivileged, safe while servers run: `-lm none` servers hold
their weights in anonymous memory, and pages mapped by an mmap model are not dropped) and check
memory:
```sh
python3 - <<'EOF'
import glob, os
n = 0
for root in ('/root/.cache/huggingface/hub', '/root/models'):
for p in glob.glob(root + '/**/*.gguf', recursive=True):
fd = os.open(p, os.O_RDONLY)
try:
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
n += 1
finally:
os.close(fd)
print('evicted', n, 'files')
EOF
grep -E 'MemAvailable|MemFree' /proc/meminfo
```
Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by
models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache
from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the
container is the HOST root cgroup; the pod's own cgroup is under
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/`.
4. **Stop llama-swap and reap orphans.**
```sh
supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,ppid,etimes,args | grep '[l]lama-server'
kill -TERM <pid> <pid> # explicit PIDs only, never pkill -f
ps -eo pid,args | grep '[l]lama-server' # must be empty
```
`supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1),
one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM.
5. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
```sh
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12
```
Check that the libraries in `build/bin` have fresh mtimes.
6. **Start and probe.**
```sh
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200
```
Sample `/proc/meminfo` every few seconds during the first model loads.
## 8. Post-deploy checks
All through llama-swap on :8080. A request helper:
```sh
ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; }
holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; }
```
1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers;
`holder` shows the 35B's PID (check with `ps -o args -p <pid>`).
2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is
correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch
(2026-10-24: 3.1-5.6 s).
3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF:UD-Q4_K_XL` (a small
solid-colour image, ask for the colour). This also forces an eviction in the llama-swap matrix: the
evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed,
releasing the GPU token`, with no kill escalation.
4. Text to gemma and to the 4B after the eviction; `holder` correct each time.
5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931).
6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for
`GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none.
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged.
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
shows a process holding WRITE on `token.lock` that is not generating. Inspect it with
`gdb -p <pid> -batch -ex 'thread apply all bt'` (gdb is installed in the pod) before killing it.
## 9. Rollback
```sh
supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
cd /root/llama.cpp
git checkout -B <previous-production-branch> "$(cat /root/DEPLOY-ROLLBACK-commit-N.txt)"
cmake --build build --target llama-server llama-bench -j 12
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
```
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in
production. Keep the candidate worktree for a later redeploy.
Pointer files 2 to 9 exist; for example commit-7 holds `e70486447`, commit-8 `64543a435`, and commit-9
`5751c977b`, the rollback target for the current production `6c480f7e7`.
## 10. Recovering a hung GPU
Symptoms: a process that survives SIGKILL (stuck in the driver); `ring sdma1 timeout` or
`ring comp_... timeout` in the kernel log; after a reboot the supervisord pod sits in FailedMount with
`/dev/kfd is not a character device` / `/dev/dri is not a directory`, and
`talosctl get pcidevices` lists no VGA device (the RX 580 and its HDMI audio function are missing).
A warm reboot does not bring the card back (classic Polaris failed reset).
The node (`anapistula-delrosalae`, Talos, bare metal) has a NanoKVM at http://192.168.255.8.
Credentials: ask the owner. Procedure (owner's instructions, follow exactly):
1. Soft shutdown: **one short press** of the power button through the NanoKVM.
2. **Never use the reset button.** It does not cut GPU power and is useless here.
3. Wait until the power LED goes off.
4. Press the power button once more to boot.
5. Only if the LED is still on 10+ minutes after the short press: one **long press of 8 seconds** to
force power off, wait for the LED to go off, then a short press to boot. With the GPU off the bus the
soft shutdown has hung before (Talos services stopped, LED stayed on), and the long press was needed.
Talos access from the workstation:
```sh
export TALOSCONFIG=/home/user/Projects/klaster/talos/generated/talosconfig
nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices
nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg
```
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active
(`/tmp/fan-control.log` shows `power1_cap ... -> 120000000`). The two 2026-10-09 hangs both came at the
first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork
and a clean-upstream build hung, so they were treated as hardware/power, not code.
## 11. Operations reference
Pod access:
```sh
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts
```
Wiring: k8s service `llama:11434` -> pod port 8080 (selector `app=llama-swap`) -> llama-swap `:8080`;
a Caddy proxy fronts `llama.lumpiasty.xyz`. llama-swap must listen on `:8080`.
Files (all under `/root`, a persistent volume):
- `/root/llama.cpp`: production tree, built in `build/` (shared libraries, so the real code is in
`build/bin/libllama.so` and `libggml-*.so`). Scratch worktrees go in `/root/arms/`.
- `/root/supervisord.conf`: supervisord (PID 1). `[program:llama-swap]` runs
`/root/llama-swap -config /root/config.yaml -listen :8080 -watch-config`; its `environment=` sets
`PATH` and `LD_LIBRARY_PATH` to `/root/llama.cpp/build/bin` and the arbiter variables (FORK.md section 5).
- `/root/config.yaml`: llama-swap models, macros (`moe_offload_args`, `moe_offload_gemma`,
`common_args`), matrix/exclusivity sets, preload. Backups `/root/config.yaml.bak-*`. Edit only with
the owner's approval.
- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program).
- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`).
- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers.
- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/`
directory from the old design may remain). Never point a test at it.
Reloads: a `config.yaml` change is picked up by `-watch-config`, but a running model keeps its old
arguments until it restarts (`supervisorctl -c /root/supervisord.conf restart llama-swap`). An
`environment=` change needs `supervisorctl -c /root/supervisord.conf reread && supervisorctl -c
/root/supervisord.conf update`.
Useful commands:
- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`.
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
- The pod has `curl`, `python3`, `gdb`, `flock`; no `jq`.
Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2).
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do
not transfer: about 2.25x the RX 580's prefill, 13% slower decode, and the GCN-only mask_opt change does
not apply. Do not change its system config.
Not a fork problem: OpenWebUI's stop button does not cancel generation (its Redis task-command listener
never handles the stop); llama-server and llama-swap both cancel correctly on client disconnect.