diff --git a/FORK.md b/FORK.md new file mode 100644 index 000000000..a3d2a88f7 --- /dev/null +++ b/FORK.md @@ -0,0 +1,837 @@ +# FORK.md - the RX 580 llama.cpp fork + +Maintainer documentation for this private fork. Read it before you rebase, change, or deploy +anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md), +which also holds the operations reference (pod access, files, recovery). + +Status at the time of writing (2026-10-25): + +- Branch `fable5/vram-ondemand-rebase-20261024`: upstream `3f03b1772` plus 27 fork commits. +- Production runs `6c480f7e7` (the first 25 commits). The 2 commits after it, `e085c347d` (concat + env knob) and `149a8cb0a` (`scripts/fork-validate.sh`), are validation-only and were not deployed. +- Commit hashes below refer to that branch. They change on every rebase; find a commit again by + its subject line (`git log --oneline origin/master..HEAD`). The commit messages are detailed and + are the primary record of each mechanism and measurement; read them with `git show`. + +## Contents + +1. Purpose and target +2. Commit map +3. Features + - F1 MoE routing-ids readback skip + - F2 GCN flash-attention mask_opt gate (and the shmem-staging note) + - F3 Vulkan concat tiled-transpose fast path + - F4 On-demand device residency (library side) + - F5 On-demand mmproj (multimodal encoder) + - F6 Cross-process VRAM arbiter (server side) + - F7 Server shutdown path + - F8 Cold-teardown guards + - F9 Benchmark harness and validation script +4. Dead ends +5. Configuration that matters +6. Measurement methodology and traps +7. Known gaps and open risks + +--- + +## 1. Purpose and target + +The fork exists to serve large MoE models well on one specific machine: + +- **GPU:** AMD RX 580 8 GB (Polaris, GCN gfx803), RADV Vulkan driver. llama.cpp reports + `fp16: 0` (RADV does not expose `shaderFloat16` on GFX8), no int-dot, no matrix cores, wave64. + PCIe 3.0 x16. RADV exposes no transfer-only queue family on this card. +- **Host:** Ryzen 5 3600 (6c/12t), 62 GB RAM, a Talos Kubernetes node shared with about 60 other pods + (CPU contention matters for decode numbers, see section 6). The GPU runs at a fixed 120 W power cap. +- **Primary model:** Qwen3.6-35B-A3B (unsloth UD-Q4_K_XL, 20.8 GiB), a hybrid MoE: 40 layers, 30 of + them gated delta-net (recurrent), 10 full attention, head_dim 256, 256 experts with 8 used. All + experts live in host RAM (`--n-cpu-moe 40`); only the backbone, KV, recurrent state and compute + buffers sit in VRAM. +- **Serving:** llama-swap under supervisord keeps several llama-server processes loaded at once + (the 35B main models, a Qwen3.5-4B "task" model OpenWebUI calls for titles and tags, gemma-4-26B + multimodal, others). They cannot all fit in 8 GB, so they time-share the GPU through the fork's + VRAM arbiter (F4-F7) without restarting processes, which would throw away the RAM-only prompt cache. +- **Priority:** prefill (prompt processing) speed first, decode second. Agentic use pushes large tool + results through the model, so long-context prefill is what users wait on. + +What the fork adds, in short: + +- three Polaris prefill wins (F1, F2, F3), all exact (token-identical to mainline); +- on-demand release/restore of everything a model holds in VRAM, plus a cross-process arbiter that + hands the GPU from one llama-server to another in well under a second for small models; +- fixes to the server's sleep, wake and shutdown paths that the arbiter exposed; +- a benchmark harness and a post-rebase validation script. + +## 2. Commit map + +Oldest first. "Feature" points at the section below. + +| commit | subject | feature | +|---|---|---| +| c72cb424e | llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert | F1 | +| fb6a8a076 | ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes | F2 | +| ff2a96d2d | vulkan: record why flash-attn shmem staging stays off for GCN | F2 | +| 3df719e1e | ggml-vulkan: tiled transpose fast-path for concat with transposed source | F3 | +| 481be8b28 | server: on-demand VRAM sharing to time-share one GPU between models | F4, F6 | +| 17165fe2a | server: optionally evict the KV cache too (Phase 2 of VRAM sharing) | F4 | +| 958257cd6 | server: keep the sleep-wake path active when the VRAM arbiter is enabled | F6 | +| b9df854de | server: coordinate model load with the VRAM arbiter to avoid load-time OOM | F6 | +| 8d80f0673 | llama: guard buffer iteration against released device buffers | F8 | +| 2feb6aeb2 | server: restore weights/KV on wake, not at decode, so KV eviction is safe | F6 | +| bf2c1a699 | server: also free the compute-graph scheduler on KV eviction | F4 | +| 9857a45f0 | llama: evict recurrent/SSM state on device release | F4 | +| 9b555f1aa | ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc | F4 | +| ad27d4921 | llama: free backend compute scratch on cold; guard memory_breakdown | F4, F8 | +| 0a0f3ab6d | mtmd: release/restore the encoder weights from VRAM on demand | F4, F5 | +| 5e99ac198 | server: on-demand mmproj - free encoder VRAM on the text path | F5 | +| b7c131219 | docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends | docs | +| 5e429d4a9 | ggml-vulkan: skip the concat transpose fast path below one tile | F3 | +| 4c4486f8e | scripts: add the RX 580 benchmark harness | F9 | +| 878d1338f | server: replace the VRAM doorbell with a want-lock | F6 | +| 53f907771 | llama: skip the compute-buffer size check when the context is cold | F8 | +| 7ac2014a8 | llama: release/restore the MoE cache with the device weights | F4 | +| 54a7f534b | scripts: rx580-bench - use -lm none, document the MoE cache option | F9 | +| 11044b35e | server: run the shutdown handler off the signal-interrupted thread | F7 | +| 6c480f7e7 | server: release the GPU token promptly on shutdown | F7 | +| e085c347d | ggml-vulkan: env knob for the concat transpose fast path | F3 | +| 149a8cb0a | scripts: add fork-validate.sh, post-rebase validation of the fork | F9 | + +--- + +## 3. Features + +Every feature section uses the same headings: What and why; Where; How it works; Invariants that +must hold; Failure modes and incidents; Rebase hazards; Validation. + +### F1. MoE routing-ids readback skip + +**What and why.** With experts host-resident, each split uploads the experts it needs. To choose +them, the expert-copy callback reads the routing ids back to the host. The ids are produced on the +same GPU the upload goes to, so the readback forces a full pipeline flush: 47 per eval on a +48-layer MoE. At prefill batch sizes every expert is used anyway (2048 tokens x 10 experts over 256 +experts), so the readback buys nothing. Measured on Laguna-S-2.1 118B IQ2_M, all experts on host +(before it was ported to the current callback): pp2048 +4.7% at depth 32k, +0.9% at 16k, flat at +depth 0, tg +3%. + +**Where.** `src/llama-context.cpp`, `llama_context::sched_copy_experts` (registered with +`ggml_backend_sched_set_copy_callback` in the scheduler setup). Commit c72cb424e. + +**How it works.** When `ggml_nelements(ids) >= 4 * n_expert`, it skips +`ggml_backend_tensor_get_async` + `ggml_backend_synchronize`, clears `st.ids_data` and sets +`st.used.assign(n_expert, true)`. The existing grouping then copies the whole expert tensor as one +range. Below the threshold the upstream readback runs unchanged. For Qwen3.6-35B (256 experts, 8 +used) the skip applies to any ubatch of 128 tokens or more; decode (8 ids) never reaches it. + +**Invariants that must hold.** +- Exactness: uploading an expert no id selects cannot change the result, because `mul_mat_id` only + reads the rows the ids point at. Any upstream change that makes unselected experts matter (for + example a kernel that reads every uploaded expert) breaks this argument. +- With `--moe-cache-mib` on, the all-true set still goes through `moe_cache->copy_experts`, which + reads the host slot maps (no device readback) and copies cached experts device-to-device. Still exact. +- `st.ids` caching stays coherent: a repeat call with the same ids tensor reuses the all-true set, + which is what the threshold would decide again; a different `n_expert` recomputes. +- The small-batch MoE-cache slot path (`llama_moe_cache::copy`) has its own readback and is not touched. + +**Failure modes and incidents.** None in production. Historically the change lived in +`ggml_backend_sched_compute_splits` in `ggml/src/ggml-backend.cpp`; upstream #29943 (`6753a033f`) +deleted that block and moved expert copying into `llama_context::sched_copy_experts`. The original +cherry-pick therefore had nothing to apply to and was re-authored as c72cb424e. A `-X theirs` +resolution would have resurrected the deleted scheduler block. + +**Rebase hazards.** +- Any change to `sched_copy_experts` or the `st` state struct (`ids`, `used`, `ids_data`). +- Upstream `701fd25b3` (MUL_MAT_ID id -1 skips work) left a `GGML_ASSERT(id >= 0 ...)` in the + readback branch; the skip branch reads no ids, so it is unaffected, but watch for upstream + semantics around negative ids. +- If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one. + +**Validation.** No dedicated check. `fork-validate.sh --ops` covers the MUL_MAT_ID kernel (not this +callback); `--perf` catches a prefill regression; `--moecache` exercises the cache path through this +callback. After a rebase that touches this function, also compare greedy output against the reference +build on the 35B. + +### F2. GCN flash-attention mask_opt gate (and the shmem-staging note) + +**What and why.** `mask_opt` skips fully-masked causal blocks and drops the per-block mask load+add +on fully-visible blocks. Upstream disables it on AMD GCN for head sizes <= 256. On Polaris attention +at depth is compute-bound on the softmax path (exp runs at about 1/4 VALU rate, no matrix cores), so +skipping that work is a real win. Measured on Qwen3.5-35B-A3B (head_dim 256): pp2048 +1% at depth 0, ++7.8% at 16k (176 -> 190), +12.3% at 32k (136 -> 153), growing with depth. Perplexity bit-identical +(5.1668 on and off). + +**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`, `ggml_vk_flash_attn`, the `use_mask_opt` +condition. Commit fb6a8a076. The shmem-staging note is a comment in `get_fa_tuning_params_scalar` +(commit ff2a96d2d, no behaviour change). + +**How it works.** The GCN clause `HSK > 256 || HSV > 256` becomes `HSK >= 256 || HSV >= 256`. The +existing large-mask conditions (`nem1 >= 32`, `nem0 * nem1 > 32768`, +`nem0 >= block_cols * 16`) keep it off for decode. + +**Invariants that must hold.** +- Only GCN with head size >= 256 is affected. Do not lower the threshold: relaxing it below 256 was + -18.5% at 16k on Laguna (head_dim 128). +- Upstream's `!use_sparse` term must stay in the condition (sparse flash-attention came in with + upstream `fc82583e6`). +- `shmem_staging` stays NVIDIA-only. Measured on GCN at head size 128: -6.7% at 16k, -7.4% at 32k. + The `kvsh` stride of `D/4+1` dwords is 4 mod 32, an 8-way LDS bank conflict on wave64 (the `+1` + padding is tuned for warp32), and the extra shared memory cuts occupancy. If anyone retries it, + fix the padding to +2/+3 first. + +**Failure modes and incidents.** A `-X theirs` resolution in a trial rebase silently dropped +`!use_sparse` from the condition. No production incident. + +**Rebase hazards.** Any upstream rewrite of `use_mask_opt`, of the FA tuning parameters, or of +architecture detection (`vk_device_architecture::AMD_GCN`). Upstream `733f57a99` added a path that +copies strided f16/bf16 K/V into `prealloc_x` scratch for flash-attention on AMD non-RDNA4 (so GCN +is included) when `neq1 >= 256`; measured at d32768 it adds about 68 MiB of VRAM and no visible +long-context prefill loss, but it has no env toggle, so its speed effect could not be isolated. + +**Validation.** `fork-validate.sh --ops` (FLASH_ATTN_EXT, 5332 cases on Vulkan0) and `--perf` +(pp2048 at depth 16k and 32k is where this feature shows). + +### F3. Vulkan concat tiled-transpose fast path + +**What and why.** Delta-net layers build their conv input with `ggml_concat(conv_states, +ggml_transpose(qkv_mixed), 0)`. The generic concat shader reads the transposed source with an +uncoalesced stride: about 5.6 ms for a 16 MB concat on the RX 580 against about 130 us of memory +bandwidth. On Qwen3.6-35B the fast path alone is +14% pp8192 and accounts for more than all of the +fork's prefill win at depth 0 (the rest of the fork costs about 1% there). In the validation run on +Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 1945 us without. + +**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`: `ggml_vk_concat` (gate lambda +`src_transposed_2d` and the `s1_trans`/`s0_trans` checks) and `ggml_vk_concat_transpose_fastpath`. +The graph pattern comes from `llm_build_delta_net_base::build_conv_state` in +`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next); qwen4exp's `build_conv_state_at` uses +the same pattern. Commits 3df719e1e (fast path), 5e429d4a9 (decode gate), e085c347d (env knob). + +**How it works.** For a concat along dim 0 where one source is stored transposed (`nb[1] == +type_size`, `nb[0] == ne[1] * type_size`) and the other source and dst are contiguous, it issues two +dispatches: `pipeline_cpy_32`/`_16` copies the contiguous source into its dst columns, and +`pipeline_cpy_transpose_32`/`_16` (the shared-memory 32x32 tiled transpose in `copy_transpose.comp`) +writes the transposed source into the other columns, both with custom push constants and a dst +offset. It ends with `ggml_vk_sync_buffers`. Everything else falls through to the generic path. + +The gate requires: f32 or f16 with all three types equal (quantized and integer types stay generic); +matching ne[1..3]; `ne[2] == 1 && ne[3] == 1` on the transposed source (so it fires only with one +sequence per ubatch, which `--parallel 1` gives); the 16-bit `doffset` bound; and +`ne[0] >= 32` (TILE_DIM) on the transposed source. + +The decode gate (5e429d4a9): at one token the transposed source is a single column, so there is no +stride to fix, but the path still costs two dispatches and the unconditional `ggml_vk_sync_buffers`, +a full pipeline barrier (the generic path emits none). That barrier, not the half-empty transpose, +was nearly all of a -17% tg256 regression, because the path ran 30 times per decoded token on the +35B. The transposed source is `qkv_mixed` transposed, so its `ne[0]` is the ubatch token count: +prefill keeps the fast path, decode and small speculative drafts take the generic one. Measured +with the gate: decode back to parity (gated/clean 1.023), prefill win retained (gated/ungated 0.998, +pp8192 +11% over clean), generation byte-identical. + +The knob (e085c347d): `GGML_VK_CONCAT_FASTPATH=0` forces the generic path, `=log` prints +`ggml_vk_concat: transpose fast path, transposed ne0=N` to stderr for every fast-path dispatch. It is +read once (function-local static). Not set in production. + +**Invariants that must hold.** +- The fast path fires at prefill and never at decode on a delta-net model. +- Output is identical to the generic path (it is a pure copy/transpose). +- The `types_ok` gate keeps quantized types off the path: the dispatches are raw f32/f16 element copies. + +**Failure modes and incidents.** +- The decode regression above shipped in the deployed fork before the gate existed. It looked + tg-neutral on the old upstream only because tg32's +-14% spread hid a 17% effect (section 6). +- If upstream changes `build_conv_state` (for example a `ggml_cont` on the transpose), the fast path + silently stops firing. Nothing crashes; prefill just loses about 14%. Only the concat check or a + perf A/B notices. + +**Rebase hazards.** +- Pipeline renames: upstream `733f57a99` renamed `pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to + `pipeline_cpy_32`/`pipeline_cpy_16`. The pick applied cleanly and did not compile. +- Upstream `f172be756` made `ggml_vk_concat` non-static (declared in `ggml-vulkan-common.h`) and + split helpers into headers; the fast-path helper stays static. +- Shape changes in `build_conv_state` / `qkv_mixed` (`[ch, n_seq_tokens, n_seqs]` reshaped from a + contiguous `build_lora_mm` output), changes to `copy_transpose.comp` or `copy.comp` push constants, + or to `vk_op_unary_push_constants` / `get_misalign_bytes`. +- Upstream concat now handles quantized types; keep `types_ok`. +- The stock `test_concat` cases in `tests/test-backend-ops.cpp` never produce a transposed source and + always have `ne[2]`/`ne[3]` > 1, so `test-backend-ops -o CONCAT` does NOT exercise this path. + +**Validation.** `fork-validate.sh --concat` (needs a build with the knob, i.e. this branch or +`--concat-build`): counts fast-path dispatches in a prefill run (must be > 0, all with ne0 >= 32) +and a decode run (must be 0 while CONCAT nodes did execute), then ties the unmodified candidate to +the fast path by CONCAT timing (candidate about equal to knob-on and at least 2x faster than +knob-off). Last run: 72 dispatches at prefill (all ne0=512), 0 over 594 CONCAT nodes at decode. +Also `--perf`. + +### F4. On-demand device residency (library side) + +**What and why.** Several always-loaded models must share 8 GB. Co-residence is impossible (the 35B +at `--n-cpu-moe 40` uses most of the card; the 4B task model needs about 2.5-3.3 GB). Restarting a +process to free VRAM throws away the RAM-only prompt cache: on Laguna a repeat turn went from a +21,960 ms re-prefill to 225 ms once the process stayed alive across a task-model call. So a model +must be able to drop everything it holds in VRAM and get it back without a reload. + +Measured footprints: +- 4B: warm 3327 MB, weights released 724 MB, restored and correct. +- 4B at 32k ctx: cold weights-only 1750 MB (KV stays), cold with KV evicted 726 MB. +- gemma-26B: warm 7147 MB, cold with weights+KV but scheduler kept 718 MB, cold with the scheduler + freed too 51 MB. +- A cold 4B before the encoder work held 644 MB that turned out to be its mmproj (F5). +- Re-warm cost: about 0.6 s per cold-to-warm switch weights-only (tiny prompt: 1.98 s cold vs 1.40 s + warm); 2-3 s with KV eviction (it adds the KV D2H/H2D and a scheduler reserve). + +**Where.** +- `src/llama-model.cpp`/`.h`: `llama_model::release_device_weights(keep_shadow)`, + `restore_device_weights()`, `weights_resident()`; per-buffer metadata + `llama_model::impl::weight_release` (`weight_release_slot`), filled in `load_tensors` and + `load_tensors_hadamard`. +- `src/llama-memory.h`: `llama_memory_i::release_device_buffers(keep_shadow)` / + `restore_device_buffers()` (default no-op). Implemented in `src/llama-kv-cache.cpp` + (`llama_kv_cache`) and `src/llama-memory-recurrent.cpp`; `llama_memory_hybrid` and + `llama_kv_cache_iswa` delegate to their children. +- `src/llama-moe-cache.cpp`/`.h`: `llama_moe_cache::release_device()`, `restore_device()`, + `resident()`, `impl::invalidate()`. +- `ggml/include/ggml-backend.h`, `ggml/src/ggml-backend-impl.h`, `ggml/src/ggml-backend.cpp`: + optional `ggml_backend_i::free_scratch` and `ggml_backend_free_scratch()`; Vulkan implementation + `ggml_backend_vk_free_scratch` in `ggml-vulkan.cpp`. +- `src/llama-context.cpp`/`.h`: `llama_context::release_device(evict_kv)`, `restore_device()`, + `discard_device()`, `device_resident()`, members `kv_device_evicted`, `device_discarded`; the + auto-restore at the top of `llama_context::decode`. +- `include/llama.h`: `llama_context_release_device(ctx, evict_kv)`, `llama_context_restore_device`, + `llama_context_discard_device`, `llama_context_weights_resident`. +- `tools/mtmd/clip.cpp`/`.h`, `tools/mtmd/mtmd.cpp`/`.h`: `clip_release_device`, + `clip_restore_device`, `clip_weights_resident`, `mtmd_release_device`, `mtmd_restore_device`. +- Commits 481be8b28, 17165fe2a, bf2c1a699, 9857a45f0, 9b555f1aa, ad27d4921, 0a0f3ab6d, 7ac2014a8, + 6c480f7e7 (discard). + +**How it works.** Tensor metadata (`ggml_context` structs) is durable; device buffers are ephemeral. + +- Weights: on release, for each releasable buffer, capture a host shadow once (weights are + read-only), free the buffer, null every tensor's `buffer`/`data`. On restore, + `ggml_backend_alloc_ctx_tensors_from_buft` reallocates an identical layout (re-pointing the + tensors), the shadow is uploaded, usage is set to WEIGHTS. Only single-buffer, alloc-path, non-host + buffers are releasable: the host-pointer path (mmap or lazy-mapped), `no_alloc` dummy buffers and + host (`Vulkan_Host`, i.e. pinned RAM) buffers stay as they are. +- KV and recurrent state: same pattern, but the shadow is re-captured on every release (the data is + read-write) and dropped after restore. +- Scheduler: with `evict_kv`, `sched.reset()` frees the worst-case compute buffer and + `sched_need_reserve = true`; restore calls `sched_reserve()` (which goes through `sched_new()`, + re-registering the expert-copy callback and upstream's realloc callback). +- Backend scratch: with `evict_kv`, `ggml_backend_free_scratch()` on each backend. On Vulkan this + discards the pending compute context, synchronizes, destroys `prealloc_x`, `prealloc_y`, + `prealloc_split_k`, `prealloc_add_rms_partials`, `sync_staging`, and zeroes their sizes and the + `prealloc_y_last_*` tracking. They are reallocated lazily on the next compute. Other backends leave + the hook null (no-op). +- MoE cache (`--moe-cache-mib`, upstream #29887): released in both modes. The banks are freed and the + cache is emptied, not shadowed: every SLRU is rebuilt fresh, its `slot_map[il]` pointers are + re-pointed at the host slot-map tensors, and the host slot maps are cleared to 0xff (every slot + -1). On restore the banks are reallocated (which also re-inits their views), cleared to 0 and + marked WEIGHTS; the cache refills from the host experts. +- Multimodal encoder: `clip_release_device` / `clip_restore_device` use the weights pattern (shadow + once); no-op for a CPU-backed encoder. `mtmd_*` wrap the vision and audio contexts. +- `llama_context::release_device(evict_kv)`: synchronize, release weights, release MoE cache, and if + `evict_kv`: release memory buffers, free sched, free scratch. + `restore_device()`: weights, MoE cache, memory buffers, `sched_reserve()` if needed, then + `ggml_backend_sched_synchronize` so uploads finish before any compute. + `discard_device()`: like `release_device(true)` with `keep_shadow = false` everywhere (no host + copies), sets `device_discarded`; used only before exit. +- `decode()` calls `restore_device()` if `!device_resident()`, as a safety net. The server restores + explicitly on wake (F6), because some KV access happens before decode. + +**Invariants that must hold.** +- Shadows are stored compactly in stable tensor-iteration order, skipping view tensors + (`view_src != NULL`), and restored with the same iteration. Never index a shadow by + `t->data - base`. +- `weight_release` stays parallel to `pimpl->ctxs_bufs`: every push to `ctxs_bufs` needs a matching + push to `weight_release` (non-releasable slots use `emplace_back()`). +- The caller synchronizes before any release; `release_device` does this itself. +- Restore order: weights, MoE cache, KV/recurrent, scheduler reserve, scheduler synchronize. The MoE + cache banks must be back before the scheduler is rebuilt. +- After a MoE-cache release no slot map may point at an expert: banks come back zeroed, so a stale + slot map silently reads zero weights. +- Every iterator over `ctxs_bufs` (model, KV cache, recurrent) must tolerate a null buffer, and every + use of `sched` must tolerate null while cold with `evict_kv` (see F8). +- `device_resident()` covers weights and MoE cache; it gates the early returns, the decode + auto-restore and `llama_context_weights_resident()`. +- `restore_device()` after `discard_device()` aborts (`GGML_ASSERT(!device_discarded)`). +- Greedy output is byte-identical across any number of release/restore cycles, and the prompt cache + survives (`timings.prompt_n` drops on a repeated prompt after a cycle). + +**Failure modes and incidents.** +- The first shadow implementation indexed by `t->data - base` and produced garbage after restore + (layout and padding differ). Fixed by the compact iteration-order shadow. +- Restoring KV lazily at decode crashed iSWA models (gemma) with + `GGML_ASSERT(buf != NULL && "tensor buffer not set")`: `update_slots()` creates an SWA checkpoint + that reads the KV via `ggml_backend_tensor_get` before decode. Fixed by restoring on wake (2feb6aeb2). +- The recurrent state was left resident (default no-op), 201 MB on the 4B, until 9857a45f0. +- The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed + resident in a cold model until bf2c1a699 / ad27d4921. +- The MoE cache was not released at all before 7ac2014a8. Reallocating its banks without + invalidating would have produced silent wrong weights. +- In the 20261024 rebase `moe_cache_slru::init()` turned out to reset only the probation segment + (`head[1]`, `tail[1]` and `n_protected` survive), so re-running it on a used cache corrupts the + lists. `invalidate()` now assigns a fresh `moe_cache_slru()` before `init()`. + +**Rebase hazards.** +- New push sites into `ctxs_bufs` (upstream's `load_tensors_hadamard()` added two in the 20261024 + rebase; without matching `weight_release` entries `release_device_weights()` reads past the end). +- Changes to buffer creation in `load_tensors` (the `host_ptr_path` condition, `ctx_bufs`/`buf_ptrs` + bookkeeping; upstream replaced `ctx_buf_maps.emplace_back(ctx, buf_map)` with + `ctx_bufs.emplace_back(ctx, std::move(buf_ptrs))` in the 20261024 rebase). +- The `llama_memory_i` interface. New memory types that do not override `release_device_buffers` + keep their KV resident under `evict_kv`; at the time of writing that is `llama_kv_cache_msa`, + `_dsa`, `_dsa_iswa`, `_dsv4` and `llama_memory_hybrid_iswa`. +- MoE-cache internals (`groups`, `slru`, `slot_map`, `buf_host`, `devices`). Upstream `33d6135cb` + changed LRU to SLRU (`g.lru` -> `g.slru`); the pick applied cleanly and did not compile. +- `llama_context` members next to ours (upstream `n_reallocs_same/diff`, `copy_experts_info`), + `sched_reserve()` semantics, and the `decode()` signature (now `decode(const llama_batch_ext &)`). +- `ggml_backend_i` layout (`free_scratch` must stay the last member after `graph_optimize`; other + backends rely on zero-init) and the Vulkan context's prealloc field set (compare with + `ggml_vk_cleanup`). + +**Validation.** `fork-validate.sh --arbiter`: `b-greedy-identical` (15 cycles each side equal to a +no-handoff reference), `b-kv-survives` (prompt_n drops after evict/restore, output identical), +`c-image-after-restore`. `--moecache MODEL` (needs a cache large enough to hit; 256 MiB on gemma-26B +had 0 hits and proves nothing, 2048 MiB had 6651 hits). The script does not check that VRAM actually +drops on release; check that by hand with `cat /sys/class/drm/card*/device/mem_info_vram_used` +before and after forcing a release. + +### F5. On-demand mmproj (multimodal encoder) + +**What and why.** A multimodal model's vision encoder sat in VRAM on every text request. With +`LLAMA_MMPROJ_ONDEMAND=1` the encoder lives in RAM and enters VRAM only for an encode. Measured on the +35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered +correctly, text after the image correct. + +**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream renamed it from +`process_mtmd_chunk`), the `mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`, +`vram_go_cold`, `vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198. + +**How it works.** After load the encoder is released (shadow captured). Before +`mtmd_batch_encode`, `mtmd_restore_device(mctx)`; if that fails (out of VRAM) and on-demand is set, it +releases the LLM backbone weights only (`llama_context_release_device(ctx_tgt, false)`: read-only +shadow, so a cheap free with no D2H, KV and prompt cache untouched) and retries. After the encode it +releases the encoder FIRST, then restores the weights, so the peak stays within VRAM. + +**Invariants that must hold.** +- Free the encoder before re-uploading the weights. +- The donor is the weights, never the KV (the KV is read-write and would need a D2H every time). +- `vram_ensure_warm()` does not restore the encoder; only the encode path does. + +**Failure modes and incidents.** Before this, a cold text model with an auto-downloaded mmproj +(`-hf` pulls one when the repo ships it) held 644 MB that no release path touched. The text-only 4B +task model now runs with `--no-mmproj`, which is the right fix for any model that never sees images. + +**Rebase hazards.** Renames or restructuring of `encode_mtmd_chunk` and the mtmd init block in +`load_model`; `clip_ctx` fields (`buf`, `ctx_data`, `sched`, `no_alloc`). + +**Validation.** `fork-validate.sh --arbiter` case `c-image-after-restore` (server A runs with +`--mmproj` and `LLAMA_MMPROJ_ONDEMAND=1`; a 128x128 red PPM must be answered "red" after A was cold, +and text right after must equal the reference). Post-deploy: one image request to gemma-26B. + +### F6. Cross-process VRAM arbiter (server side) + +**What and why.** Several llama-server processes under llama-swap share one GPU: exactly one is warm +(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production: +60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between +the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last +`fork-validate.sh` run: 30 handoffs, median 680 ms, max 1231 ms. + +**Where.** `tools/server/server-context.cpp`: `vram_arena_open`, `vram_acquire_for_load`, +`vram_share_init`, `vram_should_yield`, `vram_want_acquire`, `vram_want_release`, `vram_take_token`, +`vram_ensure_warm`, `vram_go_cold`, `vram_share_exit`, `vram_share_shutdown`, the VRAM-only branch of +`handle_sleeping_state`, the `vram_ensure_warm()` call before decode in the slot update path, and +`server_res_generator` (`bypass_sleep`). `tools/server/server-queue.cpp`/`.h`: `on_should_yield`, +`start_loop` (`should_sleep`, idle tick, post-wake grace), `wait_until_no_sleep`, `is_stopping`. +Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f. + +**How it works.** +- Enabled by `LLAMA_SLEEP_VRAM_ONLY` (any value). The arena directory is `LLAMA_VRAM_ARENA` (default + `/dev/shm/llama-vram`) with two files. +- `token.lock`: `flock(LOCK_EX)` is the baton. Rule: a process has device buffers iff it holds it. +- `want.lock`: a waiter holds it `LOCK_SH` while waiting and drops it once it owns the token. The + holder probes it with `flock(LOCK_EX | LOCK_NB)` on a SECOND fd: failure means somebody is waiting. + The probe needs its own open file description, because flock on the waiter's own fd would convert + that lock instead of conflicting with it. The kernel drops a dead waiter's lock, so nothing goes stale. +- Load: `load_model()` calls `vram_acquire_for_load()` before `common_init_from_params()`, so the + current holder releases before this model uploads anything. The loaded model stays warm holding the + token and serves its first request without a re-warm. +- Yield: `vram_share_init()` registers `vram_should_yield` with `queue_tasks.on_should_yield`. + `start_loop()`'s `should_sleep()` calls it on the loop thread, under `mutex_tasks`, only when the + task queue is empty and `update_slots` has returned. While the callback is set the idle wait ticks + every 10 ms instead of 1 s. After a wake the server holds the GPU for at least 100 ms + (`yield_grace_ms`). Sleep runs the sleeping callbacks; in VRAM-only mode `handle_sleeping_state(true)` + calls `vram_go_cold()` (release device, also the encoder; then `flock(LOCK_UN)`) instead of + upstream's full unload. +- Wake: a request calls `wait_until_no_sleep()`. `handle_sleeping_state(false)` calls + `vram_ensure_warm()`: take the token (register on want.lock, then poll `token.lock` with + `LOCK_EX | LOCK_NB` every 2 ms, aborting when `queue_tasks.is_stopping()`), restore ctx_tgt and + ctx_dft, and only then set `vram_cold = false`. The decode path calls `vram_ensure_warm()` again + (no-op when warm) and throws "server is shutting down" if it returns false. +- `server_res_generator` no longer bypasses the wake path when `--sleep-idle-seconds` is unset if + `LLAMA_SLEEP_VRAM_ONLY` is set, because the arbiter can put a server to sleep without idle-sleep. +- Queue fixes: `wait_until_no_sleep()` loops `while (sleeping && !stopping)` and re-asserts + `req_stop_sleeping` each time (with `notify_all`); the sleep wait also wakes on a non-empty task + queue; the idle `wait_for` only leaves the loop on a real task or termination. + +**Invariants that must hold.** +- Resident iff holding `token.lock`. `vram_cold = false` is set only AFTER the restore, so the loop + cannot give the token away while still bringing weights up. +- Release only happens from the idle branch of `start_loop()`, on the loop thread, never inside + `yield_to_queue()` or while a slot is generating: a holder finishes its current generation before + it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both). +- The yield decision reads live kernel state at the point of use. Do not route it through a flag set + by another thread (see attempts 3-4 below). +- Every exit from `vram_take_token()` drops want.lock, or the holder yields forever. +- The probe fd is separate from the waiter fd. +- `flock` calls retry on EINTR: the server's signal handlers are installed without SA_RESTART. +- Restore on wake, before `update_slots()`, so KV eviction is safe for SWA checkpoints. +- Every model that can be warm on the GPU at the same time must use the same arena. A non-arbiter + process (a benchmark, a manually started server) can OOM the warm model. In production the env is + set for all of llama-swap, so all its children participate. + +**Failure modes and incidents.** +- Wake-path hang (958257cd6): with no `--sleep-idle-seconds`, `bypass_sleep` skipped + `wait_until_no_sleep()`, so a server slept by the arbiter never woke and requests hung. +- Load-time OOM (b9df854de): weights were uploaded before the arbiter was active; loading gemma while + the 4B held VRAM could exceed 8 GB. After the fix the peak was 7771 MB. +- The doorbell design (replaced by 878d1338f) was edge-triggered: a waiter wrote + `/doorbell/` and the holder's warden thread turned the inotify event into a yield flag. + Rings were lost in three windows: while the holder restored (it advertised cold and discarded + foreign rings), while the holder loaded (the inotify watch did not exist yet, and the kernel does + not queue events for a missing watch), and when a cold waiter consumed a third process's ring. + Every loss was permanent: `flock(LOCK_EX)` had no timeout and production had no idle-sleep. The + blocked flock held `mutex_tasks`, so the whole server stopped answering while `/health` still + returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by + the second day). +- Failed fix attempt 1: moving `vram_cold = false` before the restore made the holder honour stale + rings and hand the token back 2-5 s after taking it without serving. Handoffs took 11m20s and 3m57s + against 5 s and 20 s before. Rolled back. +- Failed fix attempt 2: a re-ring loop (`LOCK_NB` + re-ring every ~1 s). Queued rings from an already + satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s + baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become + level-triggered. +- Failed attempts 3-4: want.lock was sound (mutual exclusion, kill -9 safety, load-window coverage), + but the warden thread still delivered its decision as a flag. `start_loop()` holds `mutex_tasks` + from `should_sleep()` through the callbacks to the wait, so the warden's `request_yield()` was always + admitted after the flag had been consumed, latching a release for the next wake (a server woke and + re-slept 138 ms later without serving). It also exposed the upstream lost wakeup in + `wait_until_no_sleep()`. Result: 0.494 s median, 1/60 stranded. The fix deleted the warden and + `request_yield`/`yield_requested` and probes inline in `should_sleep()`. +- Measured deviations that are required: the 10 ms idle tick (about 0.2% of one core per warm + server); a waiter that polls rather than retrying on a coarse grid (a 200 ms retry grid cost 0.503 s + against 0.336 s, because the holder only releases after the waiter asks); the 100 ms post-wake + grace (without it a permanent waiter caused a livelock: one request took 23.15 s over 138 + sleep/wake cycles). +- Process lesson from attempts 1-2: validate an ordinary handoff latency A/B, not only the + pathological case. Two production deploys were burned before that rule existed. +- Lost diagnostic: the old periodic "still waiting for the GPU token after N s" warning is gone; a + stuck token now shows one `VRAM arbiter: waiting for the GPU token` line and then silence. Check + `/proc/locks` (see the runbook). + +**Rebase hazards.** This is the most fragile area, because it sits inside upstream's server loop. +- `server_queue::start_loop`, `wait_until_no_sleep`, `process_new_tasks`, `yield_to_queue` and the + sleeping-callback vector. Upstream refactored sleep (`947fd9bb2`) and made decode run inside + `yield_to_queue` (`22b8e310b`); re-check after any such change that a release still cannot run + under a live graph. +- `handle_sleeping_state`, `load_model` ordering, the slot update path where decode happens, + `server_res_generator`'s `bypass_sleep`, renames in `server-context.cpp`. +- Anything that touches the KV in `update_slots()` before decode. + +**Validation.** `fork-validate.sh --arbiter` with two real servers on a scratch arena: +`a-handoffs` (30 alternating, 0 stranded, 0 non-200, the server that answered holds `token.lock`), +`b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams: +want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in +F7 and a log scan for asserts and Vulkan errors. In production, `/proc/locks` must show the warm +model holding WRITE on `token.lock`. + +### F7. Server shutdown path + +**What and why.** llama-swap evicts models with SIGTERM, so shutdown happens all the time, often +while the arbiter is mid-release or mid-wait. Two bugs made shutdown keep the GPU token and stall +every other model. + +**Where.** `tools/server/server.cpp`: `signal_handler`, `shutdown_pipe`, `struct shutdown_thread`, +`sig_thread` in `llama_server`. `tools/server/server-context.cpp`: `server_context::start_loop` (calls +`vram_share_exit()` after the loop returns), `server_context::terminate` (terminates the result queue +first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/server/server-queue.cpp`: +`server_queue::terminate` (sets atomic `stopping` before taking the mutex), `process_new_tasks`, +`server_response::recv_with_timeout`, `server_response::terminate`, +`server_response_reader::next`. Commits 11044b35e, 6c480f7e7. + +**How it works.** +- The signal handler only writes the signal number to a self-pipe (async-signal-safe); a dedicated + thread reads it and runs `shutdown_handler`. The pipe fds are `FD_CLOEXEC`. If `pipe()` fails the old + direct call is used. The thread object is declared after the server objects, so it is joined (a 0 + byte unblocks it) before they are destroyed. A second signal still calls `exit(1)`. +- `terminate()` sets `stopping` (atomic) before locking `mutex_tasks`, then stops the result queue so + HTTP handlers waiting for results get "server is shutting down" (HTTP 503, or an SSE error event); + `recv_with_timeout` checks `!running` before waiting and returns null. +- A token wait polls with `LOCK_NB` every 2 ms and aborts on `is_stopping()`; the aborted wake stays + cold, skips the restore, and the loop exits before `update_slots`. +- After `start_loop()` returns, `vram_share_exit()` frees all device buffers with + `llama_context_discard_device()` (no host shadows), releases the encoder, and only then unlocks + `token.lock`. + +**Invariants that must hold.** +- Nothing that takes `mutex_tasks` runs in signal context. +- The token is released only after this process's VRAM is free (the next model may start uploading + the moment the lock drops). +- A shutting-down server never restores weights. +- SIGTERM in any state exits 0 within about 2 s and frees the token. + +**Failure modes and incidents.** +- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside + a sleeping callback (under `mutex_tasks`, during a release); the handler's `terminate()` locked + `mutex_tasks` again on the same thread. The process wedged holding `token.lock`, every other model + queued behind it, and only SIGKILL worked. Reproduced 6/6 off production; 8/8 clean with the fix. + The first suspicion (the multimodal path) was wrong. gdb on the wedged process showed the main + thread in `futex(mutex_tasks)` with the `yield_to_queue` worker idle. +- Token kept on shutdown: with a request in flight the HTTP handler waited forever for results, main + blocked joining it, and the token was released only in the destructor after that join; with a + token wait in progress the blocking flock held `mutex_tasks`; and the old destructor unlocked the + token BEFORE `destroy()`, so the next model could upload into still-occupied VRAM. +- After the fix: off production SIGTERM mid-generation or mid-wait exits in about 0.2 s with exit 0 + and frees the token in 5-70 ms; in production about 0.7-1 s. + +**Rebase hazards.** `server.cpp`'s signal setup and `llama_server` object lifetimes; +`server_response` and `server_response_reader` (upstream `4e201388c` switched `recv_with_timeout` to a +fixed deadline: keep the order "check `!running`, return null, then `wait_until(lock, deadline)`"); +`server_context::start_loop`/`terminate`. Router mode (`server-models.cpp`, +`server_models_sse_client::next`) only checks `should_stop()` and can spin at shutdown now that +`recv_with_timeout` returns null at once; llama-swap does not use router mode. + +**Validation.** `fork-validate.sh --arbiter`: `d-term-in-release` (gdb breakpoint on +`llama_context_release_device`, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and +the token), `e-term-waiting` (exit 0 within 1.5 s, never took the token), `f-term-streaming` +(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run: +736 ms, 200 ms, 93 ms (token free in 28 ms), 80/157/156 ms. Post-deploy: a forced llama-swap eviction +must log `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token` and exit by itself. + +### F8. Cold-teardown guards + +**What and why.** A cold model has null buffers in its `ctxs_bufs` and, with `evict_kv`, a null +scheduler. Upstream code never had to consider either state. + +**Where.** `src/llama-context.cpp`: `llama_context::~llama_context` (`sched &&` in the +compute-buffer size check), `llama_context::memory_breakdown` (`if (sched)` around the compute part; +upstream's `if (moe_cache)` breakdown block stays outside it). Null-buffer skips in +`llama_model::memory_breakdown`, `llama_kv_cache::{clear, memory_breakdown, total_size}`, +`llama_memory_recurrent::{clear, memory_breakdown, total_size}`. Commits 8d80f0673, ad27d4921, 53f907771. + +**How it works.** Guards at the call sites. The asserts in ggml (`GGML_ASSERT(buffer)`, +`GGML_ASSERT(sched)`) are correct and stay; every other caller reserves the scheduler first. + +**Invariants that must hold.** A server shut down while cold exits 0. Any new upstream code that +iterates `ctxs_bufs` or calls into `sched` on a path reachable while cold (shutdown, memory +breakdown, stats) needs the same guard. + +**Failure modes and incidents.** +- `GGML_ASSERT(buffer)` on shutdown of a cold model (seen 7 times in production logs) from + memory-breakdown/size/clear on released buffers. +- `GGML_ASSERT(sched)` in `memory_breakdown` for a cold `evict_kv` model being torn down. +- `GGML_ASSERT(sched)` in the destructor's size check: every cold shutdown exited 134 instead of 0, so + under llama-swap an ordinary stop looked like a crashed child, left a backtrace in the log, and the + rest of the destructor never ran. Repro: arbiter env on, one request, force a yield with + `flock -s /want.lock -c 'sleep 6'`, then SIGINT while cold. + +**Rebase hazards.** New iteration over `ctxs_bufs`, new `sched` users in teardown or breakdown +paths. Watch item: `llama_kv_cache::update()` takes `lctx->get_sched()`, which can be null; it is +safe today only because its single caller runs after `sched_reserve()`. + +**Validation.** `fork-validate.sh --arbiter`: `h-shutdown-cold` (exit 0 while the other server holds +the token) and `log-scan`; `--moecache` `shutdown`. + +### F9. Benchmark harness and validation script + +**What and why.** The measurement method on this box had to be rediscovered several times (section +6). The harness encodes it; the validation script encodes the checks that each past incident +required, so a rebase can be validated by someone with no context. + +**Where.** `scripts/rx580-bench/` (`run.sh` workstation wrapper; `bench.sh` pod orchestrator; +`ppbench.py` corpus prefill through llama-server; `lbsweep.sh` llama-bench sweep; `summarize.py`; +`README.md` with methodology and reference values). `scripts/fork-validate.sh`. Commits 4c4486f8e, +54a7f534b, 149a8cb0a. + +**How it works.** +- `rx580-bench`: stops llama-swap, waits for an idle GPU, runs fixed slices of `pan-tadeusz.txt` + (4096/16384/32768 tokens, byte-identical across builds) through `/completion` with + `cache_prompt: false`, plus a llama-bench sweep, interleaved across arms and rounds; restarts + llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`. +- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]`: + runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under + /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets inherited + `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check flag is + `--ops --concat --arbiter`. See the header of the script for what each check proves. + +**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` refuses, and any +other llama process blocks it). `--perf` gates prefill at -5% per depth and peak VRAM at 95% of the +card; decode is reported, not gated (CPU-bound and noisy here). + +**Failure modes and incidents.** Upstream #28334 removed `--mmap`/`--no-mmap`/`--mlock` and +llama-bench's `-mmp`: the server refused to start and llama-bench silently produced no rows until +54a7f534b switched to `-lm none`. The perf and MoE-cache checks leave tens of GB of model files in page +cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024 +deploy (see the runbook). + +**Rebase hazards.** CLI flag renames in `common/arg.cpp` and `tools/llama-bench`; log line texts the +script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`, +`no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`, +`llama_moe_cache: .* hits =`, `N/M tests passed`, `Backend Vulkan0: OK`); the llama-bench markdown +table layout. + +**Validation.** Known gaps of the script: it does not check that VRAM drops on release; peak VRAM is +recorded per arm, not per test (one candidate arm once showed an unexplained 1.6 GiB spike that did +not repeat); the SIGTERM-in-release breakpoint is at function entry, not after the host-shadow +capture; nothing covers F1 directly. + +--- + +## 4. Dead ends + +Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry +without a new reason. + +| idea | result | mechanism | +|---|---|---| +| Weight pinning (`GGML_CUDA_REGISTER_HOST` and the host-register chain, from a third-party CUDA fork whose +64% was on an RTX 3060) | +17% pp2048 in isolated llama-bench (252 -> 294), fails in serving | RADV cannot import file-backed mmap pages; the pre-stage fallback fails (`ErrorOutOfDeviceMemory` for the ~18 GB expert region) or doubles host RAM. `-lm none` serves faster (Qwen3.6-35B 278 -> 286). Dropped in the 2026-09-09 rebase. | +| Expert-upload prefetch (`GGML_SCHED_PREFETCH_EXPERTS`) | -20% pp (other runs -9% to -22%), no tg benefit | Slots allocate in host-pinned GTT, not VRAM, so it stages host to host and still pays the PCIe read. `=1` means the default 3 slots. Dropped. | +| `GGML_VK_ASYNC_USE_TRANSFER_QUEUE=1` | flat (-0.1%), byte-identical | GCN is excluded from `prefers_transfer_queue` and RADV exposes no transfer-only queue family here; the per-layer upload-then-compute order leaves nothing to overlap. A standalone Vulkan microbench did show 60-73% of an H2D copy on queue family 1 hidden behind compute, so double-buffered prefetch inside one backend context remains an untested idea (estimated +10-15%, unproven). | +| Fractional `-ncmoe` | no measured win | Config only uses integers (22/40/48); upstream refactored `-ncmoe` into `llm_add_n_cpu_ffn_overrides`. Dropped. | +| MoE cache `--moe-cache-mib` 512 / 1536 / 2304 (Qwen3.6-35B, ncmoe 40, ctx 131072) | tg256@d4096 -42% / +5% / +12%; prefill +1-3%; idle VRAM 6.15 / 7.19 / 7.96 GB vs 5.66 | Small caches thrash (8-of-256 routing over 40 layers) and add a per-layer ids readback; only ~2.3 GiB pays and leaves no VRAM for a long KV. Not enabled. | +| MTP speculative decoding, Qwen3.6-35B | decode -39% (n1), -37% (n2), -50% (n3), -44% (n4), best -19% (n2, p-min 0.75); prefill -3 to -6% | A spec step costs ~180 ms vs 52 ms for plain decode; a 2-token verify alone is only 1.4x a 1-token step, so ~100 ms/step is draft machinery. The nextn layer adds 1.25 GiB VRAM. Entries removed. | +| MTP, gemma-4-26B-A4B | decode -57.9% (n1), prefill pp4k -34% | Same mechanism (~210 ms vs ~50 ms per step); MTP needs `--swa-full`, which alone costs -10.5% decode, -32% pp4k and saturates VRAM. Entries removed. | +| MTP, gemma E4B | decode -10.9% | Acceptance 0.41-0.67. The +19.5% pp seen with it comes from `--swa-full`, not MTP. | +| `--parallel 1` on gemma | no effect (-1.0%, +2.1%) | The Qwen gain is probably tied to its recurrent state being sized per sequence (unverified). | +| `-ub 4096 -b 4096` (Laguna) | flat (90.0 vs 90.9) | Fewer expert uploads cancelled by intra-ubatch attention growth. | +| mul_mat_id tile-padding theory (larger/odd `-ub`) | predicted 2.00x waste and a 1.94x cliff; measured +3.8% and -5.6%, no sawtooth | Theory dead; kills the mul_mat_id tile-selection shader work (an unbuilt, unverified WIP exists on branch `fable5/mmid-tile-wip`, with an unresolved correctness concern in its row_ids fill). | +| `-b` sweeps | non-monotonic (2048/8192 high, 4096/12288 low) | `-b 2048` is already at the top. | +| `-ncmoe` below 40 on the 35B | 34: +2.9% pp but 7.23 GiB peak at tiny context; 28: better at short context, worse at depth (137 vs 170 t/s at d32k); <= 24 collapses (~122) | Resident experts are evicted by KV/activation growth; cannot survive a 128k context. | +| `-ncmoe` 44 on Laguna | +1.5% | Does not fit at 64k context. Keep 48. | +| fp16 on Polaris (Mesa 26.1.6 + `radv_enable_float16_gfx8=true`, `fp16: 1`) | pp2048 -4.7%, -5.3% at d16k, tg +4% | Hurts prefill (the priority). Mesa 26.1.6 alone: +3.3% prefill, below the 5% bar; adopting it needs a pod image change. The pod's Mesa 25.0.7 has no such option. | +| KV quant on Qwen (q4_0, q8_0) | f16 beats q4_0 by +40% pp at 64k; q8_0 170 -> 128 t/s at depth; q4_0 with upstream PR #30738 still -16% vs f16 at d32k (133.6 vs 159.4) | No int-dot/fp16 here, so dequant is pure compute; the hybrid model's KV is small anyway. Keep f16. (PR #30738 was +15% at d16k on gemma-26B with q4_0 KV; not merged upstream as of the 20261024 rebase.) | +| `-fa 0` | 5x slower at depth (170 -> 35 at d32k) | Keep flash attention on. | +| FA scalar `block_rows` 16 -> 32 | 3.5x regression at d32k (135 -> 38) | ~42 KB shmem per workgroup at head_dim 256, one workgroup per SIMD, occupancy collapse. | +| `GGML_VK_DISABLE_GRAPH_OPTIMIZE=1` | restores tg32 at n_ctx=32 (10.44 -> 15.61), nothing at real depth | Upstream #27812's view-alias fix makes KV views look dependent and collapses barrier-free batches; only visible when per-node work is tiny. It is a correctness fix; do not disable it in serving. | +| `--no-mmap`/`-lm none` for Laguna-S 118B | does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. | +| Laguna `--cache-reuse`, `--swa-full`, `-t 12`, KV quant | ruled out | `--cache-reuse` is force-disabled for iSWA when base and SWA sizes differ. | +| CPU-only task model (`-ngl 0`) to avoid handoffs | rejected | At `--ctx-size 8192` titles could face 8k tokens of CPU prefill. The arbiter is the answer. | + +The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256 +tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch. + +## 5. Configuration that matters + +`/root/config.yaml` (llama-swap) and `/root/supervisord.conf` on the pod are the source of truth; +this section explains the choices. Values as of the 20261024 deploy. + +Qwen 35B-A3B MoE entries (macro `moe_offload_args`, applied to all 35B-A3B variants): +`--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1`, +context 131072. + +- `-lm none`: load weights into RAM (upstream removed `--no-mmap`; the old spelling makes the server + refuse to start). Faster in serving than mmap and recommended for `--n-cpu-moe`. Exception: + Laguna-S, which must use mmap (GTT limit, section 4). +- `-ub 2048 -b 2048`: the biggest prefill lever. ub512 = 97 t/s, ub1024 = 159, ub2048 = 254 on the + 35B; the per-eval expert upload is amortized over the ubatch. +- `--n-cpu-moe 40`: all experts on host. Lower values are faster at short context but run out of VRAM + at long context (section 4). Measured idle VRAM at ctx 131072: 5.66 GB. +- `--parallel 1`: +22.9% decode on the Qwen 35B (15.5 -> 19.1 t/s), prefill +0.3-1.2%, +64 MiB VRAM. + Without it the server creates 4 slots and a unified KV. Two concurrent requests queue on slot 0. + With one slot a ubatch never holds more than one sequence, which the concat fast path's + `ne[2] == 1` condition (F3) needs. +- `-t 6`: six physical cores; more threads hurt decode. +- `-fa 1`: essential (section 4). +- f16 KV (no `-ctk/-ctv`): section 4. + +Other models: +- gemma-4-26B-A4B: macro `moe_offload_gemma`, same as above but `--n-cpu-moe 22` (8 layers of experts + resident; gemma's SWA keeps KV small at depth). ncmoe sweep pp2048: 30 = 262, 24 = 266, 18 = 272, + 14 = 157 (VRAM collapse); 22 keeps headroom for 128k context (7.40 of 8 GB at 128k). + Its KV cache is `-ctk q4_0 -ctv q4_0`, set in the `gemma4_sampling` / `gemma4_nothink_sampling` + macros (not in `moe_offload_gemma`); note gemma has no `--parallel 1` (no measured gain there). +- Small models that fit in VRAM (Qwen3.5-4B, gemma E4B/E2B and similar) use auto-fit + (`common_args`); forcing experts to host would slow them. +- The 4B task model runs with `--no-mmproj` and ctx 8192, and is preloaded. +- Laguna-S-2.1 118B IQ2_M (configured as `Laguna-S-2.1-IQ2_M`, matrix var `lag`): mmap, `--n-cpu-moe 48 -b 2048 -ub 2048 -fa 1 --parallel 1`, + ctx 65536. Its exclusivity set must allow the task model (`lag & q4task`); making it exclusive with + the task model killed it on every title request and forced full re-prefills. + +Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of +`/root/supervisord.conf`: +- `LLAMA_SLEEP_VRAM_ONLY=1`: enables the arbiter and VRAM-only sleep (F6). +- `LLAMA_VRAM_ARENA=/dev/shm/llama-vram`: the shared arena. Every co-loaded GPU model must use it. +- `LLAMA_SLEEP_EVICT_KV=1`: cold models also free KV, recurrent state, scheduler and scratch (F4), + so the warm model gets nearly the whole card. Costs a 2-3 s re-warm instead of ~0.6 s. +- `LLAMA_MMPROJ_ONDEMAND=1`: encoder in VRAM only during encodes (F5). No-op for models without an mmproj. +- No `--sleep-idle-seconds`: release is purely demand-driven via want.lock. +- `GGML_CUDA_REGISTER_HOST=1` may still be present from the pinning era; it is a no-op with `-lm none`. + +Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`, +`GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`. + +Machine-level: +- GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted + every second; default/max cap 145 W). The two GPU hangs of 2026-10-09 happened before the cap was + fixed (it used to switch between 120 and 145 W by temperature); heavy runs since have been clean. +- Kernel argument `amdgpu.lockup_timeout=0,120000,0,0` (compute ring 120 s) via Talos + `machine.install.extraKernelArgs`. Without it long 64k-context submissions hit the compute-ring + watchdog (`ring comp_1.x.0 timeout`). +- Upstream changed the default `--port` from 8080 to 9931; the config passes `--port ${PORT}` + explicitly, so it is unaffected. + +## 6. Measurement methodology and traps + +- **Cross-run variance is about 7%.** llama-bench's within-run error bars understate cross-invocation + variance by about 14x (the same config read 276.7 +/- 0.9 and 296.4 +/- 0.6 in two invocations). + Put every comparison you draw a conclusion from inside one llama-bench invocation + (comma-separated sweeps). Two builds cannot share an invocation, so interleave them over at least 3 + rounds and compare medians. Treat any cross-build delta under about 5% as unproven. +- **Alternate arm order.** A fixed order gave the later slot about a 9% advantage and nearly produced + a false regression finding. Use orders like ref, cand, cand, ref. +- **Node CPU contention.** The node runs about 60 pods; llama-bench once got 4.67 of 12 hardware + threads while asking for 6. With `--n-cpu-moe 40` every MoE layer decodes on the CPU, so stolen + threads map onto tg almost directly. Prefill is nearly immune (held +-0.3% while tg32 swung +-14%). + Expect occasional catastrophic tg samples; use medians, never means. Historical tg figures taken + under different load are not comparable. +- **Never use tg32.** At the default depth (n_ctx 32) decode is launch/barrier bound and measures + command-stream overhead: two upstream commits read -36% at tg32 and were at parity at tg256 (and +7% + at d512). Use tg256 (or tg128) with an explicit depth (`-p 0 -n 256 -d 4096`). tg is not monotonic in + n_ctx on this box (n_ctx 256 measured slower than 32 and 544), so only compare same-depth cells. +- **`-b Vulkan` vs `-b Vulkan0`.** `test-backend-ops -b Vulkan` matches nothing, runs 0 cases and + still prints OK with exit 0. The backend is `Vulkan0`. Always check the case count. +- **ubatch larger than the prompt never fills.** With `-ub 2048` use prompt lengths that are + multiples of 2048. A partial trailing ubatch costs a full expert upload. +- **`-tb` is not a llama-bench flag** (server only). Any bad flag makes llama-bench print usage and + exit, which looks like a run that produced no rows. Always check the row count. `-ncmoe` is + llama-bench's spelling of `--n-cpu-moe`. +- **llama-bench prints its table only after all `-d` values finish.** No output does not mean hung. +- **Nothing else may use the GPU** during a measurement; stop llama-swap (`bench.sh` and + `/root/bench/moe/swapdown.sh` do this and restart it from a trap). +- **Check the build finished** before benchmarking it (library mtime earlier than the run start). + Never overlap a build with a big benchmark (host OOM). +- **Prompt-cache reuse is `timings.prompt_n`,** not `tokens_evaluated` (which is always the full + prompt length). +- **Per-process VRAM** from `/proc//fdinfo/*` `drm-memory-vram` must be deduplicated by + `drm-client-id`; summing all fds double-counts shared buffers. `drm-memory-gtt` is system RAM, not + VRAM. Whole-card usage: `/sys/class/drm/card*/device/mem_info_vram_used`. + `GGML_VK_MEMORY_LOGGER=1` logs every Vulkan buffer alloc/free with a running device total to stderr. +- **The concat fast path is invisible to test-backend-ops** (F3); validate it with the knob, perf + logger timing, or byte-identical greedy generation (temperature 0, top_k 1, fixed seed, + `cache_prompt: false`). +- **Do not run `llama-cli` detached with closed stdin** on the pod: it can loop forever printing + prompts (it once wrote a 65 GB log). Use llama-bench `-d` or the server harness. + +## 7. Known gaps and open risks + +- `llama_context::restore_device()` ignores the `bool` returned by `restore_device_weights()`, + `moe_cache->restore_device()` and `memory->restore_device_buffers()`. On a restore OOM, + `device_resident()` stays false, decode retries the restore on every call, and a graph reading a + null buffer would crash. Only the mmproj path handles a failed restore. +- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4). +- Router mode can spin at shutdown (F7); not used here. +- Two code comments are stale: the arbiter block comment in `server-context.cpp` says the KV cache is + never evicted (it is, with `LLAMA_SLEEP_EVICT_KV`), and `vram_go_cold` says the encoder is rebuilt by + `vram_ensure_warm()` (it is restored only at encode time). +- The README fork section says "the two code changes below are auto-on"; the fork now has more code + changes than that. Treat this file as the complete list. +- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new + tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context. + Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them. diff --git a/README.md b/README.md index 0c4c4b9e0..7f127d675 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,10 @@ ## This fork - Polaris / GCN tuning for large MoE models +Maintainer documentation for this fork (every custom feature, its invariants and rebase hazards, the +measured dead ends, and the rebase/deploy procedure): [FORK.md](FORK.md) and +[docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md). + Changes and measurements for running large MoE models with their experts offloaded to system RAM (`--n-cpu-moe`) on an old GCN card. The two code changes below are auto-on, need no flag, and are **token-identical** to mainline. Everything else here is tuning guidance. diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md new file mode 100644 index 000000000..6baab37b3 --- /dev/null +++ b/docs/fork-rebase-runbook.md @@ -0,0 +1,417 @@ +# Fork rebase and deploy runbook (RX 580 fork) + +Step-by-step procedure for moving the fork onto a new upstream and putting it into production, as it +has evolved over the 2026-09-09, 2026-10-09 and 2026-10-24 rebases. Read [FORK.md](../FORK.md) first: +it lists every fork feature with its invariants and rebase hazards, and this runbook refers to it. + +Ground rules: + +- **Never build on the workstation.** All builds run on the RX 580 pod. Locally, rely on careful reading. +- **Never resolve a conflict with `-X theirs` or `-X ours`.** Resolve every hunk by hand. +- **"Applied cleanly" is not "correct".** Re-read every clean pick against the new upstream. +- **Never `pkill -f` on the pod.** It matches the `kubectl exec` shell and kills it. Kill by explicit PID. +- **Never leave the pod not serving.** Anything that stops llama-swap must restart it from a trap. +- Keep the previous production branch untouched as the safety net. Do not push for the owner; the owner + reviews and pushes. + +Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a compiler; 5. Build on +the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU; +11. Operations reference. + +--- + +## 1. Survey upstream + +Local remotes: `origin` is upstream (github.com/ggerganov/llama.cpp), `gitea` is the fork +(git@gitea.lumpiasty.xyz:Lumpiasty/llama.cpp.git, SSH push from the workstation). On the pod, +`origin` is the same gitea repository over HTTPS (fetch only). + +```sh +git fetch origin +OLD_BASE=$(git merge-base origin/master HEAD) # the old upstream base (still an ancestor of the new master) +OLD_TIP=$(git rev-parse HEAD) +git log --oneline "$OLD_BASE"..HEAD # the fork commits, oldest last +git log --oneline "$OLD_BASE"..origin/master | wc -l # size of the upstream delta +``` + +Then look at upstream changes in the areas FORK.md lists as hazards: + +```sh +git log --oneline "$OLD_BASE"..origin/master -- \ + src/llama-context.cpp src/llama-context.h src/llama-model.cpp src/llama-model.h \ + src/llama-kv-cache.cpp src/llama-kv-cache.h src/llama-kv-cache-iswa.cpp \ + src/llama-memory.h src/llama-memory-recurrent.cpp src/llama-memory-hybrid.cpp \ + src/llama-moe-cache.cpp src/llama-moe-cache.h src/models/delta-net-base.cpp \ + ggml/src/ggml-vulkan/ ggml/src/ggml-backend.cpp ggml/src/ggml-backend-impl.h ggml/include/ggml-backend.h \ + tools/server/server-queue.cpp tools/server/server-queue.h tools/server/server-context.cpp tools/server/server.cpp \ + tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/ +``` + +For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past +decisions: +- 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional `-ncmoe` (all measured + losses or no win on this card, FORK.md section 4). +- 2026-10-09: the routing-ids readback skip had nothing to apply to (upstream #29943 deleted its block + in `ggml-backend.cpp`); it was re-authored in `llama_context::sched_copy_experts`. Upstream's new MoE + cache (#29887) needed a new fork commit so the VRAM arbiter releases and invalidates it. +- Also check CLI changes that break production config or the scripts (upstream #28334 removed + `--no-mmap`/`--mmap`/`--mlock` and llama-bench `-mmp`; the default `--port` changed to 9931). + +## 2. Trial rebase in a scratch worktree + +Measure the conflicts before doing the real thing: + +```sh +git worktree add /tmp/trial-rebase origin/master +cd /tmp/trial-rebase +git cherry-pick "$OLD_BASE".."$OLD_TIP" # stop at each conflict, note file + hunk, then: +git cherry-pick --abort # or resolve roughly to see the next one +cd - && git worktree remove --force /tmp/trial-rebase +``` + +Write down, per commit: clean / conflict (which file, which hunk, what both sides did). That list +becomes the brief for the real rebase. + +## 3. The real rebase + +```sh +NEW=fable5/vram-ondemand-rebase-$(date +%Y%m%d) +git switch -c "$NEW" origin/master +git cherry-pick # one at a time, in the original order +``` + +Rules: +- Cherry-pick in the original order, keep authors and messages. New commits you author get the + trailer the owner asks for. +- Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms. +- Fold each fix into the commit it belongs to so every commit builds on its own: + `git commit --fixup=` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`. +- Leave the previous production branch untouched. + +**What `-X theirs` silently broke in a trial (2026-10-09).** It "applied" every commit and produced +broken code: +- resurrected the scheduler readback block upstream had deleted from `ggml-backend.cpp`; +- dropped `!use_sparse` from the GCN `use_mask_opt` condition; +- dropped upstream's `copy_experts_info` (compile error); +- dropped upstream's `is_empty()` in the recurrent memory; +- dropped upstream's MoE-cache memory breakdown block; +- left a static/extern mismatch with the new `ggml-vulkan-common.h` declarations (compile error). + +The correct resolutions in that rebase: keep upstream's `n_embd` width check plus our decode +auto-restore; `copy_experts_info` plus our `kv_device_evicted`; `is_empty()` plus our `dev_shadows`; +upstream's `if (moe_cache)` breakdown block outside our `if (sched)` guard; make `ggml_vk_concat` and +`ggml_backend_vk_free` non-static to match the new header while our helpers stay static. + +**Clean picks that were wrong (2026-10-24).** +- Concat fast path (3df719e1e): applied cleanly, did not compile. Upstream `733f57a99` renamed + `pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to `pipeline_cpy_32`/`pipeline_cpy_16` (bitwise copies, + same push constants). +- MoE cache release/restore (7ac2014a8): applied cleanly, did not compile (upstream `33d6135cb` changed + LRU to SLRU, `g.lru` -> `g.slru`) and had a runtime bug: `moe_cache_slru::init()` resets only the + probation segment, so re-initialising a used cache corrupts its lists. `invalidate()` now assigns a + fresh `moe_cache_slru()` first. +- VRAM sharing core (481be8b28): besides its one conflict, upstream's new `load_tensors_hadamard()` + pushes two more entries into `pimpl->ctxs_bufs`. Without matching `weight_release.emplace_back()` + calls, `release_device_weights()` reads past the end of `weight_release`. Nothing conflicted. +- Conflicts that round: `ctx_buf_maps.emplace_back` replaced upstream by `ctx_bufs.emplace_back(ctx, + std::move(buf_ptrs))` (keep upstream's line, our release-slot block before it); `llama-context.h` + (keep upstream's `n_reallocs_same/diff` beside our `kv_device_evicted`); `recv_with_timeout` + (upstream `4e201388c` uses a fixed deadline: keep our order "check `!running`, return null", then + `wait_until(lock, deadline)`). + +**Semantic re-checks to do for every rebase, clean or not** (details in FORK.md): +- F1: `sched_copy_experts` and its `st` struct unchanged in meaning. +- F2: `use_mask_opt` still contains upstream's terms plus our `>= 256`. +- F3: `build_conv_state` still emits `ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0)` with + `qkv_mixed` contiguous `[ch, n_seq_tokens, n_seqs]`; pipeline names; `types_ok` still excludes + quantized types. +- F4: every `ctxs_bufs` push in `src/llama-model.cpp` has a `weight_release` push + (`git grep -n "ctxs_bufs.emplace_back\|weight_release" src/llama-model.cpp`); list memory types + (`git grep -n "public llama_memory_i" src/`) and note any without a release override; MoE-cache + internals; `sched_reserve()` still goes through `sched_new()` and re-registers callbacks; + `free_scratch` still the last `ggml_backend_i` member; Vulkan prealloc fields match `ggml_vk_cleanup`. +- F6/F7: `server_queue::start_loop` still evaluates `should_sleep()` only from the idle branch, never + inside `yield_to_queue()`; `handle_sleeping_state`, `server_res_generator`, `load_model` ordering; + signal setup in `server.cpp`. +- F8: new iterators over `ctxs_bufs` or users of `sched` on teardown paths. +- Scripts: flag spellings used by `scripts/rx580-bench/*` and `scripts/fork-validate.sh`, and the log + lines `fork-validate.sh` greps for (FORK.md F9). +- Production config: every flag in `/root/config.yaml` still exists in `common/arg.cpp`. + +## 4. Checks without a compiler + +Before handing the branch over for a build: + +```sh +git diff origin/master..HEAD | grep -n '^[+-]\(<<<<<<<\|=======\|>>>>>>>\)' # no conflict markers +git diff origin/master..HEAD | grep -nP '^\+.*[^\x00-\x7F]' # ASCII only in added lines +git range-diff "$OLD_BASE".."$OLD_TIP" origin/master..HEAD # patch vs patch +``` + +- The range-diff must show only the edits you made on purpose. In 2026-10-24 the whole patch was + identical to the old one except for 4 deliberate edits. +- Brace balance per changed file: the change in `{`/`}` balance from new base to new tip should equal + the change from old base to old tip. +- Every identifier in added lines must exist on the new master or be defined by the fork. This catches + a renamed function but not a renamed member (like `lru` -> `slru`); re-read member accesses by hand. +- Write down every spot where you would normally lean on the compiler and hand that list to whoever + builds. + +Then the owner pushes the branch to gitea. + +## 5. Build on the pod, in a scratch worktree + +Production keeps serving during this step. + +```sh +export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig +kubectl -n llama exec -i deploy/supervisord -- sh -c ' + set -e + B=fable5/vram-ondemand-rebase-YYYYMMDD + W=/root/arms/rebYYYYMMDD + git -C /root/llama.cpp fetch origin "$B" + git -C /root/llama.cpp worktree add "$W" FETCH_HEAD + cd "$W" + cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF + nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8 +' +``` + +- Build only the targets you need; the full `all` target fails on the UI assets (needs npm/network). + If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed). +- `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes. +- Report compile errors verbatim against the list from step 4. +- Do not overlap a build with a large benchmark (host OOM). Disk is limited (about 22 GB free; each + build tree is about 700 MB); remove scratch worktrees you no longer need with + `git -C /root/llama.cpp worktree remove `. +- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 = 1.5 GiB VRAM free for --arbiter; it refuses otherwise) +bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \ + --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat --arbiter + +# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap +cat > /root/arms/fv-perf.sh <<'EOF' +bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \ + --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf \ + --moecache /root/.cache/huggingface/hub//.gguf +EOF +setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 ` (pod-only helper, not in the repo) stops llama-swap, waits for + an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints + `SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200. +- The `--concat` check needs a build with the `GGML_VK_CONCAT_FASTPATH` knob (in the fork since + e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`). +- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits, + which proves nothing). +- Phase 2 costs about 27 minutes of downtime and leaves tens of GB of model files in page cache. +- Last full run (2026-10-24, 33 PASS, 0 FAIL): perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, + -0.7% at d32k (gate is -5%), tg256@d4096 -0.2%. +- Manual checks the script does not do: VRAM actually drops when a server goes cold + (`cat /sys/class/drm/card*/device/mem_info_vram_used` before and after); greedy output of the 35B + against the reference if `sched_copy_experts` changed. +- Any arbiter change: A/B the ordinary handoff latency against the current binary, not only the + pathological case. Two deploys were lost by skipping that. + +On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs. + +## 7. Deploy + +Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 minutes. + +1. **Rollback pointer.** Record the current production commit in the next free pointer file: + ```sh + ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 9 (holds 5751c977b) + git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-10.txt + git -C /root/llama.cpp status --short # must be clean + ``` +2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea): + ```sh + cd /root/llama.cpp + git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD + git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD + ``` +3. **Memory.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM victim. With + `-lm none` each load reads the GGUF through the page cache, and a recent benchmark leaves tens of GB + of model files cached. During the first 2026-10-24 deploy the Talos OOM controller killed the whole + pod 17 s into verification (cause likely, not proven, memory pressure). Before stopping llama-swap, + evict the model files' page cache (unprivileged, safe while servers run: `-lm none` servers hold + their weights in anonymous memory, and pages mapped by an mmap model are not dropped) and check + memory: + ```sh + python3 - <<'EOF' + import glob, os + n = 0 + for root in ('/root/.cache/huggingface/hub', '/root/models'): + for p in glob.glob(root + '/**/*.gguf', recursive=True): + fd = os.open(p, os.O_RDONLY) + try: + os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) + n += 1 + finally: + os.close(fd) + print('evicted', n, 'files') + EOF + grep -E 'MemAvailable|MemFree' /proc/meminfo + ``` + Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by + models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache + from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the + container is the HOST root cgroup; the pod's own cgroup is under + `/sys/fs/cgroup/kubepods/besteffort/pod/`. +4. **Stop llama-swap and reap orphans.** + ```sh + supervisorctl -c /root/supervisord.conf stop llama-swap + ps -eo pid,ppid,etimes,args | grep '[l]lama-server' + kill -TERM # explicit PIDs only, never pkill -f + ps -eo pid,args | grep '[l]lama-server' # must be empty + ``` + `supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1), + one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM. +5. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's + `LD_LIBRARY_PATH` points there, so do not rebuild it under running servers): + ```sh + cd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12 + ``` + Check that the libraries in `build/bin` have fresh mtimes. +6. **Start and probe.** + ```sh + supervisorctl -c /root/supervisord.conf start llama-swap + curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200 + ``` + Sample `/proc/meminfo` every few seconds during the first model loads. + +## 8. Post-deploy checks + +All through llama-swap on :8080. A request helper: + +```sh +ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \ + -d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; } +holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; } +``` + +1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers; + `holder` shows the 35B's PID (check with `ps -o args -p `). +2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is + correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch + (2026-10-24: 3.1-5.6 s). +3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF:UD-Q4_K_XL` (a small + solid-colour image, ask for the colour). This also forces an eviction in the llama-swap matrix: the + evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed, + releasing the GPU token`, with no kill escalation. +4. Text to gemma and to the 4B after the eviction; `holder` correct each time. +5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931). +6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for + `GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none. +7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged. + +A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks` +shows a process holding WRITE on `token.lock` that is not generating. Inspect it with +`gdb -p -batch -ex 'thread apply all bt'` (gdb is installed in the pod) before killing it. + +## 9. Rollback + +```sh +supervisorctl -c /root/supervisord.conf stop llama-swap +ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID +cd /root/llama.cpp +git checkout -B "$(cat /root/DEPLOY-ROLLBACK-commit-N.txt)" +cmake --build build --target llama-server llama-bench -j 12 +supervisorctl -c /root/supervisord.conf start llama-swap +curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models +``` + +Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in +production. Keep the candidate worktree for a later redeploy. + +Pointer files 2 to 9 exist; for example commit-7 holds `e70486447`, commit-8 `64543a435`, and commit-9 +`5751c977b`, the rollback target for the current production `6c480f7e7`. + +## 10. Recovering a hung GPU + +Symptoms: a process that survives SIGKILL (stuck in the driver); `ring sdma1 timeout` or +`ring comp_... timeout` in the kernel log; after a reboot the supervisord pod sits in FailedMount with +`/dev/kfd is not a character device` / `/dev/dri is not a directory`, and +`talosctl get pcidevices` lists no VGA device (the RX 580 and its HDMI audio function are missing). +A warm reboot does not bring the card back (classic Polaris failed reset). + +The node (`anapistula-delrosalae`, Talos, bare metal) has a NanoKVM at http://192.168.255.8. +Credentials: ask the owner. Procedure (owner's instructions, follow exactly): + +1. Soft shutdown: **one short press** of the power button through the NanoKVM. +2. **Never use the reset button.** It does not cut GPU power and is useless here. +3. Wait until the power LED goes off. +4. Press the power button once more to boot. +5. Only if the LED is still on 10+ minutes after the short press: one **long press of 8 seconds** to + force power off, wait for the LED to go off, then a short press to boot. With the GPU off the bus the + soft shutdown has hung before (Talos services stopped, LED stayed on), and the long press was needed. + +Talos access from the workstation: + +```sh +export TALOSCONFIG=/home/user/Projects/klaster/talos/generated/talosconfig +nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices +nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg +``` + +Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active +(`/tmp/fan-control.log` shows `power1_cap ... -> 120000000`). The two 2026-10-09 hangs both came at the +first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork +and a clean-upstream build hung, so they were treated as hardware/power, not code. + +## 11. Operations reference + +Pod access: + +```sh +export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig +kubectl -n llama exec -i deploy/supervisord -- sh -c '' +kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts +``` + +Wiring: k8s service `llama:11434` -> pod port 8080 (selector `app=llama-swap`) -> llama-swap `:8080`; +a Caddy proxy fronts `llama.lumpiasty.xyz`. llama-swap must listen on `:8080`. + +Files (all under `/root`, a persistent volume): +- `/root/llama.cpp`: production tree, built in `build/` (shared libraries, so the real code is in + `build/bin/libllama.so` and `libggml-*.so`). Scratch worktrees go in `/root/arms/`. +- `/root/supervisord.conf`: supervisord (PID 1). `[program:llama-swap]` runs + `/root/llama-swap -config /root/config.yaml -listen :8080 -watch-config`; its `environment=` sets + `PATH` and `LD_LIBRARY_PATH` to `/root/llama.cpp/build/bin` and the arbiter variables (FORK.md section 5). +- `/root/config.yaml`: llama-swap models, macros (`moe_offload_args`, `moe_offload_gemma`, + `common_args`), matrix/exclusivity sets, preload. Backups `/root/config.yaml.bak-*`. Edit only with + the owner's approval. +- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program). +- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`). +- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers. +- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/` + directory from the old design may remain). Never point a test at it. + +Reloads: a `config.yaml` change is picked up by `-watch-config`, but a running model keeps its old +arguments until it restarts (`supervisorctl -c /root/supervisord.conf restart llama-swap`). An +`environment=` change needs `supervisorctl -c /root/supervisord.conf reread && supervisorctl -c +/root/supervisord.conf update`. + +Useful commands: +- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`. +- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock` + (WRITE = holder on token.lock, READ = registered waiters on want.lock). +- The pod has `curl`, `python3`, `gdb`, `flock`; no `jq`. + +Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2). +Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do +not transfer: about 2.25x the RX 580's prefill, 13% slower decode, and the GCN-only mask_opt change does +not apply. Do not change its system config. + +Not a fork problem: OpenWebUI's stop button does not cancel generation (its Redis task-command listener +never handles the stop); llama-server and llama-swap both cancel correctly on client disconnect.