The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens: 83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save: peak +432 MiB. Leg c, output != reference: c sent turn 2 again, so the server rolled back to the checkpoint at the prompt end and processed 4 tokens, while the reference processed 27 from turn 1. These are different reuse points. A controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a restore, a cold save and a restore) is bit-identical, top-3 logprobs included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh and has not run. Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock -s file sleep) leaked the lock into sleep, so the SIGTERM landed in an ordinary evict_kv release (the full KV shadow, F4). Not the cache; the driver is fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1234 lines
92 KiB
Markdown
1234 lines
92 KiB
Markdown
# FORK.md - the RX 580 llama.cpp fork
|
|
|
|
Maintainer documentation for this private fork. Read it before you rebase, change, or deploy
|
|
anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md),
|
|
which also holds the operations reference (pod access, files, recovery).
|
|
|
|
Status at the time of writing (2026-10-25):
|
|
|
|
- Branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772`. The fork commits are
|
|
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
|
|
(this file among them), so count it rather than trusting a number written here. Section 2 maps
|
|
every commit to a feature.
|
|
- Production runs `2fc9f9c35` (deployed 2026-10-25 13:01 UTC, `system_fingerprint`
|
|
`b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the
|
|
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
|
|
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
|
|
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
|
|
top. It is not deployed; its validation (GPU and 35B included) is in F11.
|
|
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
|
|
its subject line. The commit messages are detailed and are the primary record of each mechanism
|
|
and measurement; read them with `git show`.
|
|
- Commit trailers (owner's convention): this fork is private and never contributes upstream.
|
|
Agent-written commits end with `Co-Authored-By: Claude <model> <noreply@anthropic.com>` (some
|
|
older ones say `Assisted-by: Claude`). AGENTS.md asks agents for `Assisted-by:` instead of
|
|
`Co-authored-by:` and forbids AI-written commit messages and PR descriptions; those are upstream
|
|
llama.cpp's contribution rules and do not apply to commits that stay in this fork. Anything meant
|
|
for upstream must be written by the owner under those rules.
|
|
|
|
## Contents
|
|
|
|
1. Purpose and target
|
|
2. Commit map
|
|
3. Features
|
|
- F1 MoE routing-ids readback skip
|
|
- F2 GCN flash-attention mask_opt gate (and the shmem-staging note)
|
|
- F3 Vulkan concat tiled-transpose fast path
|
|
- F4 On-demand device residency (library side)
|
|
- F5 On-demand mmproj (multimodal encoder)
|
|
- F6 Cross-process VRAM arbiter (server side)
|
|
- F7 Server shutdown path
|
|
- F8 Cold-teardown guards
|
|
- F9 Benchmark harness and validation script
|
|
- F10 Page-cache drop after a copying load
|
|
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
|
4. Dead ends
|
|
5. Configuration that matters
|
|
6. Measurement methodology and traps
|
|
7. Known gaps and open risks
|
|
|
|
---
|
|
|
|
## 1. Purpose and target
|
|
|
|
The fork exists to serve large MoE models well on one specific machine:
|
|
|
|
- **GPU:** AMD RX 580 8 GB (Polaris, GCN gfx803), RADV Vulkan driver. llama.cpp reports
|
|
`fp16: 0` (RADV does not expose `shaderFloat16` on GFX8), no int-dot, no matrix cores, wave64.
|
|
PCIe 3.0 x16. RADV exposes no transfer-only queue family on this card.
|
|
- **Host:** Ryzen 5 3600 (6c/12t), 62 GB RAM, a Talos Kubernetes node shared with about 60 other pods
|
|
(CPU contention matters for decode numbers, see section 6). The GPU runs at a fixed 120 W power cap.
|
|
- **Primary model:** Qwen3.6-35B-A3B (unsloth UD-Q4_K_XL, 20.8 GiB), a hybrid MoE: 40 layers, 30 of
|
|
them gated delta-net (recurrent), 10 full attention, head_dim 256, 256 experts with 8 used. All
|
|
experts live in host RAM (`--n-cpu-moe 40`); only the backbone, KV, recurrent state and compute
|
|
buffers sit in VRAM.
|
|
- **Serving:** llama-swap under supervisord keeps several llama-server processes loaded at once
|
|
(the 35B main models, a Qwen3.5-4B "task" model OpenWebUI calls for titles and tags, gemma-4-26B
|
|
multimodal, others). They cannot all fit in 8 GB, so they time-share the GPU through the fork's
|
|
VRAM arbiter (F4-F7) without restarting processes, which would throw away the RAM-only prompt cache.
|
|
- **Priority:** prefill (prompt processing) speed first, decode second. Agentic use pushes large tool
|
|
results through the model, so long-context prefill is what users wait on.
|
|
|
|
What the fork adds, in short:
|
|
|
|
- three Polaris prefill wins (F1, F2, F3), all exact (token-identical to mainline);
|
|
- on-demand release/restore of everything a model holds in VRAM, plus a cross-process arbiter that
|
|
hands the GPU from one llama-server to another in well under a second for small models;
|
|
- fixes to the server's sleep, wake and shutdown paths that the arbiter exposed;
|
|
- a benchmark harness and a post-rebase validation script.
|
|
|
|
## 2. Commit map
|
|
|
|
Oldest first. "Feature" points at the section below.
|
|
|
|
| commit | subject | feature |
|
|
|---|---|---|
|
|
| c72cb424e | llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert | F1 |
|
|
| fb6a8a076 | ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes | F2 |
|
|
| ff2a96d2d | vulkan: record why flash-attn shmem staging stays off for GCN | F2 |
|
|
| 3df719e1e | ggml-vulkan: tiled transpose fast-path for concat with transposed source | F3 |
|
|
| 481be8b28 | server: on-demand VRAM sharing to time-share one GPU between models | F4, F6 |
|
|
| 17165fe2a | server: optionally evict the KV cache too (Phase 2 of VRAM sharing) | F4 |
|
|
| 958257cd6 | server: keep the sleep-wake path active when the VRAM arbiter is enabled | F6 |
|
|
| b9df854de | server: coordinate model load with the VRAM arbiter to avoid load-time OOM | F6 |
|
|
| 8d80f0673 | llama: guard buffer iteration against released device buffers | F8 |
|
|
| 2feb6aeb2 | server: restore weights/KV on wake, not at decode, so KV eviction is safe | F6 |
|
|
| bf2c1a699 | server: also free the compute-graph scheduler on KV eviction | F4 |
|
|
| 9857a45f0 | llama: evict recurrent/SSM state on device release | F4 |
|
|
| 9b555f1aa | ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc | F4 |
|
|
| ad27d4921 | llama: free backend compute scratch on cold; guard memory_breakdown | F4, F8 |
|
|
| 0a0f3ab6d | mtmd: release/restore the encoder weights from VRAM on demand | F4, F5 |
|
|
| 5e99ac198 | server: on-demand mmproj - free encoder VRAM on the text path | F5 |
|
|
| b7c131219 | docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends | docs |
|
|
| 5e429d4a9 | ggml-vulkan: skip the concat transpose fast path below one tile | F3 |
|
|
| 4c4486f8e | scripts: add the RX 580 benchmark harness | F9 |
|
|
| 878d1338f | server: replace the VRAM doorbell with a want-lock | F6 |
|
|
| 53f907771 | llama: skip the compute-buffer size check when the context is cold | F8 |
|
|
| 7ac2014a8 | llama: release/restore the MoE cache with the device weights | F4 |
|
|
| 54a7f534b | scripts: rx580-bench - use -lm none, document the MoE cache option | F9 |
|
|
| 11044b35e | server: run the shutdown handler off the signal-interrupted thread | F7 |
|
|
| 6c480f7e7 | server: release the GPU token promptly on shutdown | F7 |
|
|
| e085c347d | ggml-vulkan: env knob for the concat transpose fast path | F3 |
|
|
| 149a8cb0a | scripts: add fork-validate.sh, post-rebase validation of the fork | F9 |
|
|
| bfbaf15b2 | docs: FORK.md and rebase runbook for the RX 580 fork | docs |
|
|
| f5c0d8f28 | server: fix stale VRAM-arbiter comments and the sleep log line | F6 |
|
|
| f664b0f20 | llama: env knob for the routing-ids readback skip | F1 |
|
|
| 62d442e7b | scripts: rx580-bench - decode at tg256 depth 4096, not tg32 | F9 |
|
|
| 2512290f9 | scripts: fork-validate.sh - gate --perf on medians, not means | F9 |
|
|
| 80e662a2a | scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 |
|
|
| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
|
|
| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
|
|
| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 |
|
|
| f24c3582c | llama: do not release CPU extra buffers (CPU_REPACK) on device release | F4 |
|
|
| dd851fd0c | llama: read a released KV/recurrent state from its host shadow | F4, F11 |
|
|
| 2874dcb10 | server: disk prompt cache, save slots on shutdown and restore after a restart | F11 |
|
|
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
|
|
|
|
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
|
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
|
`6c480f7e7` was the previous production commit. The last four rows are on `fable5/disk-prompt-cache`
|
|
only (hashes of that branch), not deployed.
|
|
|
|
---
|
|
|
|
## 3. Features
|
|
|
|
Every feature section uses the same headings: What and why; Where; How it works; Invariants that
|
|
must hold; Failure modes and incidents; Rebase hazards; Validation.
|
|
|
|
### F1. MoE routing-ids readback skip
|
|
|
|
**What and why.** With experts host-resident, each split uploads the experts it needs. To choose
|
|
them, the expert-copy callback reads the routing ids back to the host. The ids are produced on the
|
|
same GPU the upload goes to, so the readback forces a full pipeline flush: 47 per eval on a
|
|
48-layer MoE. At prefill batch sizes every expert is used anyway (2048 tokens x 10 experts over 256
|
|
experts), so the readback buys nothing. Measured on Laguna-S-2.1 118B IQ2_M, all experts on host
|
|
(before it was ported to the current callback): pp2048 +4.7% at depth 32k, +0.9% at 16k, flat at
|
|
depth 0, tg +3%.
|
|
|
|
**Where.** `src/llama-context.cpp`, `llama_context::sched_copy_experts` (registered with
|
|
`ggml_backend_sched_set_copy_callback` in the scheduler setup). Commit c72cb424e.
|
|
|
|
**How it works.** When `ggml_nelements(ids) >= 4 * n_expert`, it skips
|
|
`ggml_backend_tensor_get_async` + `ggml_backend_synchronize`, clears `st.ids_data` and sets
|
|
`st.used.assign(n_expert, true)`. The existing grouping then copies the whole expert tensor as one
|
|
range. Below the threshold the upstream readback runs unchanged. For Qwen3.6-35B (256 experts, 8
|
|
used) the skip applies to any ubatch of 128 tokens or more; decode (8 ids) never reaches it.
|
|
|
|
**Invariants that must hold.**
|
|
- Exactness: uploading an expert no id selects cannot change the result, because `mul_mat_id` only
|
|
reads the rows the ids point at. Any upstream change that makes unselected experts matter (for
|
|
example a kernel that reads every uploaded expert) breaks this argument.
|
|
- With `--moe-cache-mib` on, the all-true set still goes through `moe_cache->copy_experts`, which
|
|
reads the host slot maps (no device readback) and copies cached experts device-to-device. Still exact.
|
|
- `st.ids` caching stays coherent: a repeat call with the same ids tensor reuses the all-true set,
|
|
which is what the threshold would decide again; a different `n_expert` recomputes.
|
|
- The small-batch MoE-cache slot path (`llama_moe_cache::copy`) has its own readback and is not touched.
|
|
|
|
**Failure modes and incidents.** None in production. Historically the change lived in
|
|
`ggml_backend_sched_compute_splits` in `ggml/src/ggml-backend.cpp`; upstream #29943 (`6753a033f`)
|
|
deleted that block and moved expert copying into `llama_context::sched_copy_experts`. The original
|
|
cherry-pick therefore had nothing to apply to and was re-authored as c72cb424e. A `-X theirs`
|
|
resolution would have resurrected the deleted scheduler block.
|
|
|
|
**Rebase hazards.**
|
|
- Any change to `sched_copy_experts` or the `st` state struct (`ids`, `used`, `ids_data`).
|
|
- Upstream `701fd25b3` (MUL_MAT_ID id -1 skips work) left a `GGML_ASSERT(id >= 0 ...)` in the
|
|
readback branch; the skip branch reads no ids, so it is unaffected, but watch for upstream
|
|
semantics around negative ids.
|
|
- If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.
|
|
|
|
**Validation.** `fork-validate.sh --moe-exact MODEL` (needs the GPU to itself, run it in the
|
|
`--perf` swapdown window with the 35B): reads `expert_count`/`expert_used_count` from the GGUF,
|
|
computes the threshold ceil(4 * n_expert / n_expert_used) tokens per ubatch (128 on the 35B), and
|
|
compares 64 greedy tokens after a prompt of about twice that, every expert on the host:
|
|
candidate == reference (`identical-ref`), and with the `LLAMA_MOE_IDS_SKIP` knob (f664b0f20;
|
|
`=0` forces the readback, `=log` reports each skip on stderr) candidate == candidate with the skip
|
|
forced off (`identical-noskip`, which proves the skip exact even when the reference build carries
|
|
it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). First GPU run
|
|
2026-10-25 (35B, 2fc9f9c35 vs 6c480f7e7): prompt 397 tokens, 39 skips logged, 0 with the skip forced
|
|
off, output identical in all three runs, 78 s. Also `--ops` (the MUL_MAT_ID
|
|
kernel, not this callback), `--perf` (prefill regression) and `--moecache` (the cache path through
|
|
this callback).
|
|
|
|
### F2. GCN flash-attention mask_opt gate (and the shmem-staging note)
|
|
|
|
**What and why.** `mask_opt` skips fully-masked causal blocks and drops the per-block mask load+add
|
|
on fully-visible blocks. Upstream disables it on AMD GCN for head sizes <= 256. On Polaris attention
|
|
at depth is compute-bound on the softmax path (exp runs at about 1/4 VALU rate, no matrix cores), so
|
|
skipping that work is a real win. Measured on Qwen3.5-35B-A3B (head_dim 256): pp2048 +1% at depth 0,
|
|
+7.8% at 16k (176 -> 190), +12.3% at 32k (136 -> 153), growing with depth. Perplexity bit-identical
|
|
(5.1668 on and off).
|
|
|
|
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`, `ggml_vk_flash_attn`, the `use_mask_opt`
|
|
condition. Commit fb6a8a076. The shmem-staging note is a comment in `get_fa_tuning_params_scalar`
|
|
(commit ff2a96d2d, no behaviour change).
|
|
|
|
**How it works.** The GCN clause `HSK > 256 || HSV > 256` becomes `HSK >= 256 || HSV >= 256`. The
|
|
existing large-mask conditions (`nem1 >= 32`, `nem0 * nem1 > 32768`,
|
|
`nem0 >= block_cols * 16`) keep it off for decode.
|
|
|
|
**Invariants that must hold.**
|
|
- Only GCN with head size >= 256 is affected. Do not lower the threshold: relaxing it below 256 was
|
|
-18.5% at 16k on Laguna (head_dim 128).
|
|
- Upstream's `!use_sparse` term must stay in the condition (sparse flash-attention came in with
|
|
upstream `fc82583e6`).
|
|
- `shmem_staging` stays NVIDIA-only. Measured on GCN at head size 128: -6.7% at 16k, -7.4% at 32k.
|
|
The `kvsh` stride of `D/4+1` dwords is 4 mod 32, an 8-way LDS bank conflict on wave64 (the `+1`
|
|
padding is tuned for warp32), and the extra shared memory cuts occupancy. If anyone retries it,
|
|
fix the padding to +2/+3 first.
|
|
|
|
**Failure modes and incidents.** A `-X theirs` resolution in a trial rebase silently dropped
|
|
`!use_sparse` from the condition. No production incident.
|
|
|
|
**Rebase hazards.** Any upstream rewrite of `use_mask_opt`, of the FA tuning parameters, or of
|
|
architecture detection (`vk_device_architecture::AMD_GCN`). Upstream `733f57a99` added a path that
|
|
copies strided f16/bf16 K/V into `prealloc_x` scratch for flash-attention on AMD non-RDNA4 (so GCN
|
|
is included) when `neq1 >= 256`; measured at d32768 it adds about 68 MiB of VRAM and no visible
|
|
long-context prefill loss, but it has no env toggle, so its speed effect could not be isolated.
|
|
|
|
**Validation.** `fork-validate.sh --ops` (FLASH_ATTN_EXT, 5332 cases on Vulkan0) and `--perf`
|
|
(pp2048 at depth 16k and 32k is where this feature shows).
|
|
|
|
### F3. Vulkan concat tiled-transpose fast path
|
|
|
|
**What and why.** Delta-net layers build their conv input with `ggml_concat(conv_states,
|
|
ggml_transpose(qkv_mixed), 0)`. The generic concat shader reads the transposed source with an
|
|
uncoalesced stride: about 5.6 ms for a 16 MB concat on the RX 580 against about 130 us of memory
|
|
bandwidth. On Qwen3.6-35B the fast path alone is +14% pp8192 and accounts for more than all of the
|
|
fork's prefill win at depth 0 (the rest of the fork costs about 1% there). In the validation run on
|
|
Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 1945 us without.
|
|
|
|
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`: `ggml_vk_concat` (gate lambda
|
|
`src_transposed_2d` and the `s1_trans`/`s0_trans` checks) and `ggml_vk_concat_transpose_fastpath`.
|
|
The graph pattern that matters here comes from `llm_build_delta_net_base::build_conv_state` in
|
|
`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next). Commits 3df719e1e (fast path),
|
|
5e429d4a9 (decode gate), e085c347d (env knob).
|
|
|
|
Other graphs that build `ggml_concat(state, ggml_transpose(x), 0)` (found with
|
|
`git grep -n "ggml_concat.*ggml_transpose" src/models/` plus the two-line form in delta-net-base and
|
|
lfm-base): `qwen4exp.cpp` (`build_conv_state_at`), `bailingmoe3.cpp`, `clef.cpp`, `kimi-linear.cpp`,
|
|
`kimi-k3.cpp`, `glm5-next.cpp`, `mamba-base.cpp` (two sites, the Mamba/Mamba2 conv and the hybrids
|
|
built on it), `plamo2.cpp`, and `lfm-base.cpp` (causal short conv; the transpose is assigned to `bx`
|
|
first). The fast path takes any of them when the gate holds (transposed source a plain 2D transpose
|
|
of a contiguous f32/f16 tensor, one sequence, `ne[0] >= 32`), so a change to the gate or the shader
|
|
affects all of them. Only the delta-net path (Qwen3.5/3.6) has been measured and validated; the
|
|
`--concat` check uses qwen35.
|
|
|
|
**How it works.** For a concat along dim 0 where one source is stored transposed (`nb[1] ==
|
|
type_size`, `nb[0] == ne[1] * type_size`) and the other source and dst are contiguous, it issues two
|
|
dispatches: `pipeline_cpy_32`/`_16` copies the contiguous source into its dst columns, and
|
|
`pipeline_cpy_transpose_32`/`_16` (the shared-memory 32x32 tiled transpose in `copy_transpose.comp`)
|
|
writes the transposed source into the other columns, both with custom push constants and a dst
|
|
offset. It ends with `ggml_vk_sync_buffers`. Everything else falls through to the generic path.
|
|
|
|
The gate requires: f32 or f16 with all three types equal (quantized and integer types stay generic);
|
|
matching ne[1..3]; `ne[2] == 1 && ne[3] == 1` on the transposed source (so it fires only with one
|
|
sequence per ubatch, which `--parallel 1` gives); the 16-bit `doffset` bound; and
|
|
`ne[0] >= 32` (TILE_DIM) on the transposed source.
|
|
|
|
The decode gate (5e429d4a9): at one token the transposed source is a single column, so there is no
|
|
stride to fix, but the path still costs two dispatches and the unconditional `ggml_vk_sync_buffers`,
|
|
a full pipeline barrier (the generic path emits none). That barrier, not the half-empty transpose,
|
|
was nearly all of a -17% tg256 regression, because the path ran 30 times per decoded token on the
|
|
35B. The transposed source is `qkv_mixed` transposed, so its `ne[0]` is the ubatch token count:
|
|
prefill keeps the fast path, decode and small speculative drafts take the generic one. Measured
|
|
with the gate: decode back to parity (gated/clean 1.023), prefill win retained (gated/ungated 0.998,
|
|
pp8192 +11% over clean), generation byte-identical.
|
|
|
|
The knob (e085c347d): `GGML_VK_CONCAT_FASTPATH=0` forces the generic path, `=log` prints
|
|
`ggml_vk_concat: transpose fast path, transposed ne0=N` to stderr for every fast-path dispatch. It is
|
|
read once (function-local static). Not set in production.
|
|
|
|
**Invariants that must hold.**
|
|
- The fast path fires at prefill and never at decode on a delta-net model.
|
|
- Output is identical to the generic path (it is a pure copy/transpose).
|
|
- The `types_ok` gate keeps quantized types off the path: the dispatches are raw f32/f16 element copies.
|
|
|
|
**Failure modes and incidents.**
|
|
- The decode regression above shipped in the deployed fork before the gate existed. It looked
|
|
tg-neutral on the old upstream only because tg32's +-14% spread hid a 17% effect (section 6).
|
|
- If upstream changes `build_conv_state` (for example a `ggml_cont` on the transpose), the fast path
|
|
silently stops firing. Nothing crashes; prefill just loses about 14%. Only the concat check or a
|
|
perf A/B notices.
|
|
|
|
**Rebase hazards.**
|
|
- Pipeline renames: upstream `733f57a99` renamed `pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to
|
|
`pipeline_cpy_32`/`pipeline_cpy_16`. The pick applied cleanly and did not compile.
|
|
- Upstream `f172be756` made `ggml_vk_concat` non-static (declared in `ggml-vulkan-common.h`) and
|
|
split helpers into headers; the fast-path helper stays static.
|
|
- Shape changes in `build_conv_state` / `qkv_mixed` (`[ch, n_seq_tokens, n_seqs]` reshaped from a
|
|
contiguous `build_lora_mm` output), changes to `copy_transpose.comp` or `copy.comp` push constants,
|
|
or to `vk_op_unary_push_constants` / `get_misalign_bytes`.
|
|
- Upstream concat now handles quantized types; keep `types_ok`.
|
|
- The stock `test_concat` cases in `tests/test-backend-ops.cpp` never produce a transposed source and
|
|
always have `ne[2]`/`ne[3]` > 1, so `test-backend-ops -o CONCAT` does NOT exercise this path.
|
|
|
|
**Validation.** `fork-validate.sh --concat` (needs a build with the knob, i.e. this branch or
|
|
`--concat-build`): counts fast-path dispatches in a prefill run (must be > 0, all with ne0 >= 32)
|
|
and a decode run (must be 0 while CONCAT nodes did execute), then ties the unmodified candidate to
|
|
the fast path by CONCAT timing (candidate about equal to knob-on and at least 2x faster than
|
|
knob-off). Last run: 72 dispatches at prefill (all ne0=512), 0 over 594 CONCAT nodes at decode.
|
|
Also `--perf`.
|
|
|
|
### F4. On-demand device residency (library side)
|
|
|
|
**What and why.** Several always-loaded models must share 8 GB. Co-residence is impossible (the 35B
|
|
at `--n-cpu-moe 40` uses most of the card; the 4B task model needs about 2.5-3.3 GB). Restarting a
|
|
process to free VRAM throws away the RAM-only prompt cache: on Laguna a repeat turn went from a
|
|
21,960 ms re-prefill to 225 ms once the process stayed alive across a task-model call. So a model
|
|
must be able to drop everything it holds in VRAM and get it back without a reload.
|
|
|
|
Measured footprints:
|
|
- 4B: warm 3327 MB, weights released 724 MB, restored and correct.
|
|
- 4B at 32k ctx: cold weights-only 1750 MB (KV stays), cold with KV evicted 726 MB.
|
|
- gemma-26B: warm 7147 MB, cold with weights+KV but scheduler kept 718 MB, cold with the scheduler
|
|
freed too 51 MB.
|
|
- A cold 4B before the encoder work held 644 MB that turned out to be its mmproj (F5).
|
|
- Re-warm cost: about 0.6 s per cold-to-warm switch weights-only (tiny prompt: 1.98 s cold vs 1.40 s
|
|
warm); 2-3 s with KV eviction (it adds the KV D2H/H2D and a scheduler reserve).
|
|
|
|
**Where.**
|
|
- `src/llama-model.cpp`/`.h`: `llama_model::release_device_weights(keep_shadow)`,
|
|
`restore_device_weights()`, `weights_resident()`; per-buffer metadata
|
|
`llama_model::impl::weight_release` (`weight_release_slot`), filled in `load_tensors` and
|
|
`load_tensors_hadamard`.
|
|
- `src/llama-memory.h`: `llama_memory_i::release_device_buffers(keep_shadow)` /
|
|
`restore_device_buffers()` (default no-op). Implemented in `src/llama-kv-cache.cpp`
|
|
(`llama_kv_cache`) and `src/llama-memory-recurrent.cpp`; `llama_memory_hybrid` and
|
|
`llama_kv_cache_iswa` delegate to their children.
|
|
- `src/llama-moe-cache.cpp`/`.h`: `llama_moe_cache::release_device()`, `restore_device()`,
|
|
`resident()`, `impl::invalidate()`.
|
|
- `ggml/include/ggml-backend.h`, `ggml/src/ggml-backend-impl.h`, `ggml/src/ggml-backend.cpp`:
|
|
optional `ggml_backend_i::free_scratch` and `ggml_backend_free_scratch()`; Vulkan implementation
|
|
`ggml_backend_vk_free_scratch` in `ggml-vulkan.cpp`.
|
|
- `src/llama-context.cpp`/`.h`: `llama_context::release_device(evict_kv)`, `restore_device()`,
|
|
`discard_device()`, `device_resident()`, members `kv_device_evicted`, `device_discarded`; the
|
|
auto-restore at the top of `llama_context::decode`.
|
|
- `include/llama.h`: `llama_context_release_device(ctx, evict_kv)`, `llama_context_restore_device`,
|
|
`llama_context_discard_device`, `llama_context_weights_resident`.
|
|
- `tools/mtmd/clip.cpp`/`.h`, `tools/mtmd/mtmd.cpp`/`.h`: `clip_release_device`,
|
|
`clip_restore_device`, `clip_weights_resident`, `mtmd_release_device`, `mtmd_restore_device`.
|
|
- Commits 481be8b28, 17165fe2a, bf2c1a699, 9857a45f0, 9b555f1aa, ad27d4921, 0a0f3ab6d, 7ac2014a8,
|
|
6c480f7e7 (discard).
|
|
|
|
**How it works.** Tensor metadata (`ggml_context` structs) is durable; device buffers are ephemeral.
|
|
|
|
- Weights: on release, for each releasable buffer, capture a host shadow once (weights are
|
|
read-only), free the buffer, null every tensor's `buffer`/`data`. On restore,
|
|
`ggml_backend_alloc_ctx_tensors_from_buft` reallocates an identical layout (re-pointing the
|
|
tensors), the shadow is uploaded, usage is set to WEIGHTS. Only single-buffer, alloc-path, non-host
|
|
buffers are releasable: the host-pointer path (mmap or lazy-mapped), `no_alloc` dummy buffers and
|
|
host (`Vulkan_Host`, i.e. pinned RAM) buffers stay as they are.
|
|
- KV and recurrent state: same pattern, but the shadow is re-captured on every release (the data is
|
|
read-write) and dropped after restore.
|
|
- Scheduler: with `evict_kv`, `sched.reset()` frees the worst-case compute buffer and
|
|
`sched_need_reserve = true`; restore calls `sched_reserve()` (which goes through `sched_new()`,
|
|
re-registering the expert-copy callback and upstream's realloc callback).
|
|
- Backend scratch: with `evict_kv`, `ggml_backend_free_scratch()` on each backend. On Vulkan this
|
|
discards the pending compute context, synchronizes, destroys `prealloc_x`, `prealloc_y`,
|
|
`prealloc_split_k`, `prealloc_add_rms_partials`, `sync_staging`, and zeroes their sizes and the
|
|
`prealloc_y_last_*` tracking. They are reallocated lazily on the next compute. Other backends leave
|
|
the hook null (no-op).
|
|
- MoE cache (`--moe-cache-mib`, upstream #29887): released in both modes. The banks are freed and the
|
|
cache is emptied, not shadowed: every SLRU is rebuilt fresh, its `slot_map[il]` pointers are
|
|
re-pointed at the host slot-map tensors, and the host slot maps are cleared to 0xff (every slot
|
|
-1). On restore the banks are reallocated (which also re-inits their views), cleared to 0 and
|
|
marked WEIGHTS; the cache refills from the host experts.
|
|
- Multimodal encoder: `clip_release_device` / `clip_restore_device` use the weights pattern (shadow
|
|
once); no-op for a CPU-backed encoder. `mtmd_*` wrap the vision and audio contexts.
|
|
- `llama_context::release_device(evict_kv)`: synchronize, release weights, release MoE cache, and if
|
|
`evict_kv`: release memory buffers, free sched, free scratch.
|
|
`restore_device()`: weights, MoE cache, memory buffers, `sched_reserve()` if needed, then
|
|
`ggml_backend_sched_synchronize` so uploads finish before any compute.
|
|
`discard_device()`: like `release_device(true)` with `keep_shadow = false` everywhere (no host
|
|
copies), sets `device_discarded`; used only before exit.
|
|
- `decode()` calls `restore_device()` if `!device_resident()`, as a safety net. The server restores
|
|
explicitly on wake (F6), because some KV access happens before decode.
|
|
|
|
**Invariants that must hold.**
|
|
- Shadows are stored compactly in stable tensor-iteration order, skipping view tensors
|
|
(`view_src != NULL`), and restored with the same iteration. Never index a shadow by
|
|
`t->data - base`.
|
|
- `weight_release` stays parallel to `pimpl->ctxs_bufs`: every push to `ctxs_bufs` needs a matching
|
|
push to `weight_release` (non-releasable slots use `emplace_back()`).
|
|
- The caller synchronizes before any release; `release_device` does this itself.
|
|
- Restore order: weights, MoE cache, KV/recurrent, scheduler reserve, scheduler synchronize. The MoE
|
|
cache banks must be back before the scheduler is rebuilt.
|
|
- After a MoE-cache release no slot map may point at an expert: banks come back zeroed, so a stale
|
|
slot map silently reads zero weights.
|
|
- Every iterator over `ctxs_bufs` (model, KV cache, recurrent) must tolerate a null buffer, and every
|
|
use of `sched` must tolerate null while cold with `evict_kv` (see F8).
|
|
- `device_resident()` covers weights and MoE cache; it gates the early returns, the decode
|
|
auto-restore and `llama_context_weights_resident()`.
|
|
- `restore_device()` after `discard_device()` aborts (`GGML_ASSERT(!device_discarded)`).
|
|
- Greedy output is byte-identical across any number of release/restore cycles, and the prompt cache
|
|
survives (`timings.prompt_n` drops on a repeated prompt after a cycle).
|
|
|
|
**Failure modes and incidents.**
|
|
- The first shadow implementation indexed by `t->data - base` and produced garbage after restore
|
|
(layout and padding differ). Fixed by the compact iteration-order shadow.
|
|
- Restoring KV lazily at decode crashed iSWA models (gemma) with
|
|
`GGML_ASSERT(buf != NULL && "tensor buffer not set")`: `update_slots()` creates an SWA checkpoint
|
|
that reads the KV via `ggml_backend_tensor_get` before decode. Fixed by restoring on wake (2feb6aeb2).
|
|
- The recurrent state was left resident (default no-op), 201 MB on the 4B, until 9857a45f0.
|
|
- A model with weights on the CPU device (`-dev none` / `-ngl 0`) segfaulted (exit 139) in
|
|
`release_device_weights()` the first time the arbiter put it to sleep: CPU_REPACK buffers are not
|
|
host buffers, so `load_tensors` marked them releasable, and the shadow capture called their NULL
|
|
`get_tensor`. It also crashes production `2fc9f9c35`, but no production arbiter model has CPU-device
|
|
weights. Fixed in f24c3582c: only buffers on a non-CPU device are releasable. Found by the
|
|
`--diskcache` check (F11), whose arbiter cases run CPU only.
|
|
- The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed
|
|
resident in a cold model until bf2c1a699 / ad27d4921.
|
|
- The MoE cache was not released at all before 7ac2014a8. Reallocating its banks without
|
|
invalidating would have produced silent wrong weights.
|
|
- In the 20261024 rebase `moe_cache_slru::init()` turned out to reset only the probation segment
|
|
(`head[1]`, `tail[1]` and `n_protected` survive), so re-running it on a used cache corrupts the
|
|
lists. `invalidate()` now assigns a fresh `moe_cache_slru()` before `init()`.
|
|
|
|
**Rebase hazards.**
|
|
- New push sites into `ctxs_bufs` (upstream's `load_tensors_hadamard()` added two in the 20261024
|
|
rebase; without matching `weight_release` entries `release_device_weights()` reads past the end).
|
|
- Changes to buffer creation in `load_tensors` (the `host_ptr_path` condition, `ctx_bufs`/`buf_ptrs`
|
|
bookkeeping; upstream replaced `ctx_buf_maps.emplace_back(ctx, buf_map)` with
|
|
`ctx_bufs.emplace_back(ctx, std::move(buf_ptrs))` in the 20261024 rebase).
|
|
- The `llama_memory_i` interface. New memory types that do not override `release_device_buffers`
|
|
keep their KV resident under `evict_kv`; at the time of writing that is `llama_kv_cache_msa`,
|
|
`_dsa`, `_dsa_iswa`, `_dsv4` and `llama_memory_hybrid_iswa`.
|
|
- MoE-cache internals (`groups`, `slru`, `slot_map`, `buf_host`, `devices`). Upstream `33d6135cb`
|
|
changed LRU to SLRU (`g.lru` -> `g.slru`); the pick applied cleanly and did not compile.
|
|
- `llama_context` members next to ours (upstream `n_reallocs_same/diff`, `copy_experts_info`),
|
|
`sched_reserve()` semantics, and the `decode()` signature (now `decode(const llama_batch_ext &)`).
|
|
- `ggml_backend_i` layout (`free_scratch` must stay the last member after `graph_optimize`; other
|
|
backends rely on zero-init) and the Vulkan context's prealloc field set (compare with
|
|
`ggml_vk_cleanup`).
|
|
|
|
**Validation.** `fork-validate.sh --arbiter`: `b-greedy-identical` (15 cycles each side equal to a
|
|
no-handoff reference), `b-kv-survives` (prompt_n drops after evict/restore, output identical),
|
|
`c-image-after-restore`. `--moecache MODEL` (needs a cache large enough to hit; 256 MiB on gemma-26B
|
|
had 0 hits and proves nothing, 2048 MiB had 6651 hits). `--arbiter` `v-vram-release` /
|
|
`v-vram-restore` check that VRAM really drops: a baseline of `mem_info_vram_used` before any scratch
|
|
server, then server A warm and alone, forced cold by a fake waiter on want.lock: the card-wide delta
|
|
must drop by >= 70% of what A added and come back after the next request. Production's own share is
|
|
re-read from the DRM fdinfo of every process the script did not start; if it moved by > 64 MiB the
|
|
check SKIPs. First run (2026-10-25, 0.8B with mmproj, ctx 8192, `LLAMA_SLEEP_EVICT_KV=1`): warm
|
|
+546 MiB, cold +4 MiB, restored +546 MiB.
|
|
|
|
### F5. On-demand mmproj (multimodal encoder)
|
|
|
|
**What and why.** A multimodal model's vision encoder sat in VRAM on every text request. With
|
|
`LLAMA_MMPROJ_ONDEMAND=1` the encoder lives in RAM and enters VRAM only for an encode. Measured on the
|
|
35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered
|
|
correctly, text after the image correct.
|
|
|
|
**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream `9871df591` split the
|
|
encode out of `process_mtmd_chunk`; `encode_mtmd_chunk` is called from `process_mtmd_chunk` and
|
|
directly from the batched-embedding path in the slot update, so the restore covers both), the
|
|
`mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`, `vram_go_cold`,
|
|
`vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
|
|
|
|
**How it works.** After load the encoder is released (shadow captured). Before
|
|
`mtmd_batch_encode`, `mtmd_restore_device(mctx)`; if that fails (out of VRAM) and on-demand is set, it
|
|
releases the LLM backbone weights only (`llama_context_release_device(ctx_tgt, false)`: read-only
|
|
shadow, so a cheap free with no D2H, KV and prompt cache untouched) and retries. After the encode it
|
|
releases the encoder FIRST, then restores the weights, so the peak stays within VRAM.
|
|
|
|
**Invariants that must hold.**
|
|
- Free the encoder before re-uploading the weights.
|
|
- The donor is the weights, never the KV (the KV is read-write and would need a D2H every time).
|
|
- `vram_ensure_warm()` does not restore the encoder; only the encode path does.
|
|
|
|
**Failure modes and incidents.** Before this, a cold text model with an auto-downloaded mmproj
|
|
(`-hf` pulls one when the repo ships it) held 644 MB that no release path touched. The text-only 4B
|
|
task model now runs with `--no-mmproj`, which is the right fix for any model that never sees images.
|
|
|
|
**Rebase hazards.** Renames or restructuring of `encode_mtmd_chunk` and the mtmd init block in
|
|
`load_model`; `clip_ctx` fields (`buf`, `ctx_data`, `sched`, `no_alloc`).
|
|
|
|
**Validation.** `fork-validate.sh --arbiter` case `c-image-after-restore` (server A runs with
|
|
`--mmproj` and `LLAMA_MMPROJ_ONDEMAND=1`; a 128x128 red PPM must be answered "red" after A was cold,
|
|
and text right after must equal the reference). Post-deploy: one image request to gemma-26B.
|
|
|
|
### F6. Cross-process VRAM arbiter (server side)
|
|
|
|
**What and why.** Several llama-server processes under llama-swap share one GPU: exactly one is warm
|
|
(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production:
|
|
60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between
|
|
the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last
|
|
`fork-validate.sh` run before the 20261024 deploy: 30 handoffs, median 680 ms, max 1231 ms (the
|
|
2026-10-25 run of the same build read 1050 ms, max 1562 ms: handoff latency moves with node load,
|
|
which is why it is now A/B'd against the reference in one session).
|
|
|
|
**Where.** `tools/server/server-context.cpp`: `vram_arena_open`, `vram_acquire_for_load`,
|
|
`vram_share_init`, `vram_should_yield`, `vram_want_acquire`, `vram_want_release`, `vram_take_token`,
|
|
`vram_ensure_warm`, `vram_go_cold`, `vram_share_exit`, `vram_share_shutdown`, the VRAM-only branch of
|
|
`handle_sleeping_state`, the `vram_ensure_warm()` call before decode in the slot update path, and
|
|
`server_res_generator` (`bypass_sleep`). `tools/server/server-queue.cpp`/`.h`: `on_should_yield`,
|
|
`start_loop` (`should_sleep`, idle tick, post-wake grace), `wait_until_no_sleep`, `is_stopping`.
|
|
Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
|
|
|
|
**How it works.**
|
|
- Enabled by `LLAMA_SLEEP_VRAM_ONLY` (any value). The arena directory is `LLAMA_VRAM_ARENA` (default
|
|
`/dev/shm/llama-vram`) with two files.
|
|
- `token.lock`: `flock(LOCK_EX)` is the baton. Rule: a process has device buffers iff it holds it.
|
|
- `want.lock`: a waiter holds it `LOCK_SH` while waiting and drops it once it owns the token. The
|
|
holder probes it with `flock(LOCK_EX | LOCK_NB)` on a SECOND fd: failure means somebody is waiting.
|
|
The probe needs its own open file description, because flock on the waiter's own fd would convert
|
|
that lock instead of conflicting with it. The kernel drops a dead waiter's lock, so nothing goes stale.
|
|
- Load: `load_model()` calls `vram_acquire_for_load()` before `common_init_from_params()`, so the
|
|
current holder releases before this model uploads anything. The loaded model stays warm holding the
|
|
token and serves its first request without a re-warm.
|
|
- Yield: `vram_share_init()` registers `vram_should_yield` with `queue_tasks.on_should_yield`.
|
|
`start_loop()`'s `should_sleep()` calls it on the loop thread, under `mutex_tasks`, only when the
|
|
task queue is empty and `update_slots` has returned. While the callback is set the idle wait ticks
|
|
every 10 ms instead of 1 s. After a wake the server holds the GPU for at least 100 ms
|
|
(`yield_grace_ms`). Sleep runs the sleeping callbacks; in VRAM-only mode `handle_sleeping_state(true)`
|
|
calls `vram_go_cold()` (release device, also the encoder; then `flock(LOCK_UN)`) instead of
|
|
upstream's full unload.
|
|
- Wake: a request calls `wait_until_no_sleep()`. `handle_sleeping_state(false)` calls
|
|
`vram_ensure_warm()`: take the token (register on want.lock, then poll `token.lock` with
|
|
`LOCK_EX | LOCK_NB` every 2 ms, aborting when `queue_tasks.is_stopping()`), restore ctx_tgt and
|
|
ctx_dft, and only then set `vram_cold = false`. The decode path calls `vram_ensure_warm()` again
|
|
(no-op when warm) and throws "server is shutting down" if it returns false.
|
|
- `server_res_generator` no longer bypasses the wake path when `--sleep-idle-seconds` is unset if
|
|
`LLAMA_SLEEP_VRAM_ONLY` is set, because the arbiter can put a server to sleep without idle-sleep.
|
|
- Queue fixes: `wait_until_no_sleep()` loops `while (sleeping && !stopping)` and re-asserts
|
|
`req_stop_sleeping` each time (with `notify_all`); the sleep wait also wakes on a non-empty task
|
|
queue; the idle `wait_for` only leaves the loop on a real task or termination.
|
|
|
|
**Invariants that must hold.**
|
|
- Resident iff holding `token.lock`; `vram_cold == false` means both. `vram_ensure_warm()` sets
|
|
`vram_cold = false` only AFTER the restore. What that order protects today, reasoned from the code:
|
|
the only thing that gives the token away is `vram_should_yield()`, called from `should_sleep()` on
|
|
the loop thread. `vram_ensure_warm()` also runs only on the loop thread, synchronously: from the
|
|
wake callback (`handle_sleeping_state(false)`, called inside `start_loop()` with `mutex_tasks`
|
|
held) and before decode in the slot update path (outside `yield_to_queue()`). So `should_sleep()`
|
|
cannot run between the token acquire and the end of the restore, whichever order the two lines
|
|
are in. The measured failure (attempt 1 below: handoffs of 11m20s and 3m57s) happened under the
|
|
retired doorbell design, where a separate warden thread read the cold flag to decide whether to
|
|
honour or discard rings, so an early `false` let it accept stale rings during the restore. Under
|
|
the inline want-lock probe the order is defensive: it keeps the flag's meaning exact for any future
|
|
reader off the loop thread, and keeps a release out of a half-restored model if the restore or the
|
|
probe ever moves (for example into `yield_to_queue()`). Either way, do not change it without an A/B
|
|
of ordinary handoff latency (`fork-validate.sh --arbiter`, `a-handoff-latency`).
|
|
- `vram_acquire_for_load()` sets `vram_cold = false` as soon as it holds the token, before the upload.
|
|
Nothing reads the flag during the load today (the loop is not running yet and `on_should_yield` is
|
|
only registered in `vram_share_init()`, which sets it again), so the early set is redundant now.
|
|
It records "holding the token" at the moment it becomes true, because the opposite error is the
|
|
dangerous one: a model that is warm and holds the token while `vram_cold` says cold never yields
|
|
(`vram_should_yield()` returns false for a cold model), which strands every waiter, and its next
|
|
wake would try to restore weights that were never released.
|
|
- Release only happens from the idle branch of `start_loop()`, on the loop thread, never inside
|
|
`yield_to_queue()` or while a slot is generating: a holder finishes its current generation before
|
|
it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both).
|
|
- The yield decision reads live kernel state at the point of use. Do not route it through a flag set
|
|
by another thread (see attempts 3-4 below).
|
|
- Every exit from `vram_take_token()` drops want.lock, or the holder yields forever.
|
|
- The probe fd is separate from the waiter fd.
|
|
- `flock` calls retry on EINTR: the server's signal handlers are installed without SA_RESTART.
|
|
- Restore on wake, before `update_slots()`, so KV eviction is safe for SWA checkpoints.
|
|
- Every model that can be warm on the GPU at the same time must use the same arena. A non-arbiter
|
|
process (a benchmark, a manually started server) can OOM the warm model. In production the env is
|
|
set for all of llama-swap, so all its children participate.
|
|
|
|
**Failure modes and incidents.**
|
|
- Wake-path hang (958257cd6): with no `--sleep-idle-seconds`, `bypass_sleep` skipped
|
|
`wait_until_no_sleep()`, so a server slept by the arbiter never woke and requests hung.
|
|
- Load-time OOM (b9df854de): weights were uploaded before the arbiter was active; loading gemma while
|
|
the 4B held VRAM could exceed 8 GB. After the fix the peak was 7771 MB.
|
|
- The doorbell design (replaced by 878d1338f) was edge-triggered: a waiter wrote
|
|
`<arena>/doorbell/<pid>` and the holder's warden thread turned the inotify event into a yield flag.
|
|
Rings were lost in three windows: while the holder restored (it advertised cold and discarded
|
|
foreign rings), while the holder loaded (the inotify watch did not exist yet, and the kernel does
|
|
not queue events for a missing watch), and when a cold waiter consumed a third process's ring.
|
|
Every loss was permanent: `flock(LOCK_EX)` had no timeout and production had no idle-sleep. The
|
|
blocked flock held `mutex_tasks`, so the whole server stopped answering while `/health` still
|
|
returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by
|
|
the second day).
|
|
- Failed fix attempt 1 (doorbell design): moving `vram_cold = false` before the restore made the
|
|
holder's warden honour stale rings and hand the token back 2-5 s after taking it without serving.
|
|
Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. See the first invariant
|
|
for what this means under the current design.
|
|
- Failed fix attempt 2: a re-ring loop (`LOCK_NB` + re-ring every ~1 s). Queued rings from an already
|
|
satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s
|
|
baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become
|
|
level-triggered.
|
|
- Failed attempts 3-4: want.lock was sound (mutual exclusion, kill -9 safety, load-window coverage),
|
|
but the warden thread still delivered its decision as a flag. `start_loop()` holds `mutex_tasks`
|
|
from `should_sleep()` through the callbacks to the wait, so the warden's `request_yield()` was always
|
|
admitted after the flag had been consumed, latching a release for the next wake (a server woke and
|
|
re-slept 138 ms later without serving). It also exposed the upstream lost wakeup in
|
|
`wait_until_no_sleep()`. Result: 0.494 s median, 1/60 stranded. The fix deleted the warden and
|
|
`request_yield`/`yield_requested` and probes inline in `should_sleep()`.
|
|
- Measured deviations that are required: the 10 ms idle tick (about 0.2% of one core per warm
|
|
server); a waiter that polls rather than retrying on a coarse grid (a 200 ms retry grid cost 0.503 s
|
|
against 0.336 s, because the holder only releases after the waiter asks); the 100 ms post-wake
|
|
grace (without it a permanent waiter caused a livelock: one request took 23.15 s over 138
|
|
sleep/wake cycles).
|
|
- Process lesson from attempts 1-2: validate an ordinary handoff latency A/B, not only the
|
|
pathological case. Two production deploys were burned before that rule existed.
|
|
- Lost diagnostic: the old periodic "still waiting for the GPU token after N s" warning is gone; a
|
|
stuck token now shows one `VRAM arbiter: waiting for the GPU token` line and then silence. Check
|
|
`/proc/locks` (see the runbook).
|
|
|
|
**Rebase hazards.** This is the most fragile area, because it sits inside upstream's server loop.
|
|
- `server_queue::start_loop`, `wait_until_no_sleep`, `process_new_tasks`, `yield_to_queue` and the
|
|
sleeping-callback vector. Upstream refactored sleep (`947fd9bb2`) and made decode run inside
|
|
`yield_to_queue` (`22b8e310b`); re-check after any such change that a release still cannot run
|
|
under a live graph.
|
|
- `handle_sleeping_state`, `load_model` ordering, the slot update path where decode happens,
|
|
`server_res_generator`'s `bypass_sleep`, renames in `server-context.cpp`.
|
|
- Anything that touches the KV in `update_slots()` before decode.
|
|
|
|
**Validation.** `fork-validate.sh --arbiter` with two real servers on a scratch arena:
|
|
`a-handoffs` (30 alternating, 0 stranded, 0 non-200, the server that answered holds `token.lock`),
|
|
`a-handoff-latency` (fresh server pairs of `--build` and `--ref` in the same session, blocks
|
|
ref, cand, cand, ref x 16 handoffs; FAIL if the candidate median exceeds the reference median by more
|
|
than max(25%, 200 ms); first run 2026-10-25, reb1024 against itself beside production: 1050.5 vs
|
|
1082.5 ms), `v-vram-*` (F4), `b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams:
|
|
want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in
|
|
F7 and a log scan for asserts and Vulkan errors. In production, `/proc/locks` must show the warm
|
|
model holding WRITE on `token.lock`.
|
|
|
|
### F7. Server shutdown path
|
|
|
|
**What and why.** llama-swap evicts models with SIGTERM, so shutdown happens all the time, often
|
|
while the arbiter is mid-release or mid-wait. Two bugs made shutdown keep the GPU token and stall
|
|
every other model.
|
|
|
|
**Where.** `tools/server/server.cpp`: `signal_handler`, `shutdown_pipe`, `struct shutdown_thread`,
|
|
`sig_thread` in `llama_server`. `tools/server/server-context.cpp`: `server_context::start_loop` (calls
|
|
`vram_share_exit()` after the loop returns), `server_context::terminate` (terminates the result queue
|
|
first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/server/server-queue.cpp`:
|
|
`server_queue::terminate` (sets atomic `stopping` before taking the mutex), `process_new_tasks`,
|
|
`server_response::recv_with_timeout`, `server_response::terminate`,
|
|
`server_response_reader::next`. Commits 11044b35e, 6c480f7e7.
|
|
|
|
**How it works.**
|
|
- The signal handler only writes the signal number to a self-pipe (async-signal-safe); a dedicated
|
|
thread reads it and runs `shutdown_handler`. The pipe fds are `FD_CLOEXEC`. If `pipe()` fails the old
|
|
direct call is used. The thread object is declared after the server objects, so it is joined (a 0
|
|
byte unblocks it) before they are destroyed. A second signal still calls `exit(1)`.
|
|
- `terminate()` sets `stopping` (atomic) before locking `mutex_tasks`, then stops the result queue so
|
|
HTTP handlers waiting for results get "server is shutting down" (HTTP 503, or an SSE error event);
|
|
`recv_with_timeout` checks `!running` before waiting and returns null.
|
|
- A token wait polls with `LOCK_NB` every 2 ms and aborts on `is_stopping()`; the aborted wake stays
|
|
cold, skips the restore, and the loop exits before `update_slots`.
|
|
- After `start_loop()` returns, `vram_share_exit()` frees all device buffers with
|
|
`llama_context_discard_device()` (no host shadows), releases the encoder, and only then unlocks
|
|
`token.lock`.
|
|
- What llama-swap (v239 on the pod) does when a child exits on its own (crash, OOM kill, external
|
|
signal), verified 2026-10-25 with a scratch llama-swap instance and a CPU-only child, for both
|
|
`kill -9` and an external SIGTERM: it logs
|
|
`[WARN] group: running <model> exited: [<model>] upstream exited unexpectedly`, does not restart it
|
|
eagerly (no new process in the 2 s after the kill), and starts a new process on the next request
|
|
for that model (new PID, health check, HTTP 200). The production log since the last pod restart has
|
|
no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers
|
|
the restart pays a full model load (about 26 s for the 35B with `-lm none`; the first 35B request
|
|
after the 2026-10-25 deploy took 34 s including load and answer), and under the arbiter
|
|
the new process takes the token through `vram_acquire_for_load()` like any load. A process that
|
|
wedges instead of exiting (the incidents below) is never restarted: `/health` keeps answering.
|
|
|
|
**Invariants that must hold.**
|
|
- Nothing that takes `mutex_tasks` runs in signal context.
|
|
- The token is released only after this process's VRAM is free (the next model may start uploading
|
|
the moment the lock drops).
|
|
- A shutting-down server never restores weights.
|
|
- SIGTERM in any state exits 0 within about 2 s and frees the token. With the disk prompt cache (F11)
|
|
the exit also writes each slot's state: about 1 s more per GB of state, after the token is free.
|
|
|
|
**Failure modes and incidents.**
|
|
- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside
|
|
a sleeping callback (under `mutex_tasks`, during a release); the handler's `terminate()` locked
|
|
`mutex_tasks` again on the same thread. The process wedged holding `token.lock`, every other model
|
|
queued behind it, and only SIGKILL worked. Reproduced 6/6 off production; 8/8 clean with the fix.
|
|
The first suspicion (the multimodal path) was wrong. gdb on the wedged process showed the main
|
|
thread in `futex(mutex_tasks)` with the `yield_to_queue` worker idle.
|
|
- Token kept on shutdown: with a request in flight the HTTP handler waited forever for results, main
|
|
blocked joining it, and the token was released only in the destructor after that join; with a
|
|
token wait in progress the blocking flock held `mutex_tasks`; and the old destructor unlocked the
|
|
token BEFORE `destroy()`, so the next model could upload into still-occupied VRAM.
|
|
- After the fix: off production SIGTERM mid-generation or mid-wait exits in about 0.2 s with exit 0
|
|
and frees the token in 5-70 ms; in production about 0.7-1 s.
|
|
|
|
**Rebase hazards.** `server.cpp`'s signal setup and `llama_server` object lifetimes;
|
|
`server_response` and `server_response_reader` (upstream `4e201388c` switched `recv_with_timeout` to a
|
|
fixed deadline: keep the order "check `!running`, return null, then `wait_until(lock, deadline)`");
|
|
`server_context::start_loop`/`terminate`. Router mode (`server-models.cpp`,
|
|
`server_models_sse_client::next`) only checks `should_stop()` and can spin at shutdown now that
|
|
`recv_with_timeout` returns null at once; llama-swap does not use router mode.
|
|
|
|
**Validation.** `fork-validate.sh --arbiter`: `d-term-in-release` (gdb breakpoint on
|
|
`llama_context_release_device`, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and
|
|
the token), `e-term-waiting` (exit 0 within 1.5 s, never took the token), `f-term-streaming`
|
|
(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run
|
|
(2026-10-25, 2fc9f9c35, `d` by log polling): 750 ms, 225 ms, 126 ms (token free in 23 ms),
|
|
108/159/159 ms. Post-deploy: a forced llama-swap eviction
|
|
must log `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token` and exit by itself.
|
|
|
|
### F8. Cold-teardown guards
|
|
|
|
**What and why.** A cold model has null buffers in its `ctxs_bufs` and, with `evict_kv`, a null
|
|
scheduler. Upstream code never had to consider either state.
|
|
|
|
**Where.** `src/llama-context.cpp`: `llama_context::~llama_context` (`sched &&` in the
|
|
compute-buffer size check), `llama_context::memory_breakdown` (`if (sched)` around the compute part;
|
|
upstream's `if (moe_cache)` breakdown block stays outside it). Null-buffer skips in
|
|
`llama_model::memory_breakdown`, `llama_kv_cache::{clear, memory_breakdown, total_size}`,
|
|
`llama_memory_recurrent::{clear, memory_breakdown, total_size}`. Commits 8d80f0673, ad27d4921, 53f907771.
|
|
|
|
**How it works.** Guards at the call sites. The asserts in ggml (`GGML_ASSERT(buffer)`,
|
|
`GGML_ASSERT(sched)`) are correct and stay; every other caller reserves the scheduler first.
|
|
|
|
**Invariants that must hold.** A server shut down while cold exits 0. Any new upstream code that
|
|
iterates `ctxs_bufs` or calls into `sched` on a path reachable while cold (shutdown, memory
|
|
breakdown, stats) needs the same guard.
|
|
|
|
**Failure modes and incidents.**
|
|
- `GGML_ASSERT(buffer)` on shutdown of a cold model (seen 7 times in production logs) from
|
|
memory-breakdown/size/clear on released buffers.
|
|
- `GGML_ASSERT(sched)` in `memory_breakdown` for a cold `evict_kv` model being torn down.
|
|
- `GGML_ASSERT(sched)` in the destructor's size check: every cold shutdown exited 134 instead of 0, so
|
|
under llama-swap an ordinary stop looked like a crashed child, left a backtrace in the log, and the
|
|
rest of the destructor never ran. Repro: arbiter env on, one request, force a yield with
|
|
`flock -s <arena>/want.lock -c 'sleep 6'`, then SIGINT while cold.
|
|
|
|
**Rebase hazards.** New iteration over `ctxs_bufs`, new `sched` users in teardown or breakdown
|
|
paths. Watch item: `llama_kv_cache::update()` takes `lctx->get_sched()`, which can be null; it is
|
|
safe today only because its single caller runs after `sched_reserve()`.
|
|
|
|
**Validation.** `fork-validate.sh --arbiter`: `h-shutdown-cold` (exit 0 while the other server holds
|
|
the token) and `log-scan`; `--moecache` `shutdown`.
|
|
|
|
### F9. Benchmark harness and validation script
|
|
|
|
**What and why.** The measurement method on this box had to be rediscovered several times (section
|
|
6). The harness encodes it; the validation script encodes the checks that each past incident
|
|
required, so a rebase can be validated by someone with no context.
|
|
|
|
**Where.** `scripts/rx580-bench/` (`run.sh` workstation wrapper; `bench.sh` pod orchestrator;
|
|
`ppbench.py` corpus prefill through llama-server; `lbsweep.sh` llama-bench sweep; `summarize.py`;
|
|
`README.md` with methodology and reference values). `scripts/fork-validate.sh`. Commits 4c4486f8e,
|
|
54a7f534b, 149a8cb0a, 62d442e7b, 2512290f9, 80e662a2a, ceab1a91f, e5fc474b8.
|
|
|
|
**How it works.**
|
|
- `rx580-bench`: stops llama-swap, waits for an idle GPU, runs fixed slices of `pan-tadeusz.txt`
|
|
(4096/16384/32768 tokens, byte-identical across builds) through `/completion` with
|
|
`cache_prompt: false`, plus a llama-bench sweep (prefill pp2048/pp8192 at depth 0, decode
|
|
tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and
|
|
rounds; restarts llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
|
|
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]
|
|
[--moe-exact MODEL] [--diskcache]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a
|
|
scratch dir under /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets
|
|
inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check
|
|
flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline.
|
|
See the header of the script for what each check proves.
|
|
- Which phase needs what: `--ops --concat` run beside production; `--arbiter` too only if 1536 MiB
|
|
of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs
|
|
in the swapdown window, about 2 minutes); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
|
|
first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share
|
|
the card either). `--diskcache` runs beside production (CPU only); its GPU leg SKIPs below 1536 MiB
|
|
of free VRAM.
|
|
|
|
**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24,
|
|
production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s,
|
|
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. On 2026-10-25
|
|
(medians, `6c480f7e7` / `2fc9f9c35`): pp2048 274.5/273.8, d16384 200.5/200.4, d32768 157.6/157.6,
|
|
tg256@d4096 16.8/16.7. The corpus
|
|
harness and older numbers are in `scripts/rx580-bench/README.md`. A fresh run far outside these
|
|
means something else is wrong (another GPU user, thermal state, a bad build) before it means a
|
|
regression or a win.
|
|
|
|
**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` and `--moe-exact`
|
|
refuse, and any other llama process blocks them). `--perf` gates prefill at -5% per depth on the
|
|
MEDIAN of all repetition samples of each build (never a mean: CPU contention produces single
|
|
catastrophic samples), and peak VRAM at 95% of the card; decode is reported, not gated (CPU-bound and
|
|
noisy here).
|
|
|
|
**Failure modes and incidents.** Upstream #28334 removed `--mmap`/`--no-mmap`/`--mlock` and
|
|
llama-bench's `-mmp`: the server refused to start and llama-bench silently produced no rows until
|
|
54a7f534b switched to `-lm none`. The perf and MoE-cache checks leave tens of GB of model files in page
|
|
cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024
|
|
deploy (see the runbook). That pod restart also emptied `/tmp` (every earlier `fork-validate.*`
|
|
scratch dir with its raw samples is gone) and removed gdb, which is not in the pod image; `--arbiter`
|
|
case `d-term-in-release` then falls back to polling the log.
|
|
|
|
**Rebase hazards.** CLI flag renames in `common/arg.cpp` and `tools/llama-bench`; log line texts the
|
|
script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`,
|
|
`no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`,
|
|
`llama_moe_cache: .* hits =`, `sched_copy_experts: ids readback skipped`, `N/M tests passed`,
|
|
`Backend Vulkan0: OK`, `disk cache: restored`, `disk cache: copied`/`saved`, `prompts of this model`,
|
|
`size limit, removing`, `disk cache: failed to write`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and
|
|
markdown layout; the GGUF keys `<arch>.expert_count` / `<arch>.expert_used_count`; the DRM fdinfo
|
|
keys `drm-client-id` / `drm-memory-vram`.
|
|
|
|
**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test. The first
|
|
candidate arm has now twice shown an unexplained spike of about 1.6 GiB over the other arms
|
|
(2026-10-25: ref 5476, cand 7062, cand 5767, ref 5476 MiB); the cause is open (per-test peaks would
|
|
locate it), and it stays far below the 95% gate. The SIGTERM-in-release breakpoint is at function
|
|
entry, not after the host-shadow capture. The `v-vram-release` SKIP path (production moving during
|
|
the measurement) has not been exercised. `--moe-exact` and the median-gated `--perf` first ran on the
|
|
GPU on 2026-10-25 (runbook section 6).
|
|
|
|
### F10. Page-cache drop after a copying load
|
|
|
|
**What and why.** With `-lm none` the loader read()s the GGUF into its own buffers, but the kernel
|
|
keeps the file in page cache afterwards, charged to the pod's cgroup: about 20 GB after a 35B load.
|
|
The pod is BestEffort on a shared node, and on 2026-10-24 the Talos OOM controller killed it right
|
|
after a model load under exactly that pressure (MemFree < 1 GB, file cache about 20 GB). Once the
|
|
data is copied those pages are pure waste, so the loader advises the kernel to drop them.
|
|
|
|
**Where.** `src/llama-mmap.h`/`.cpp`: `llama_file::drop_cache(offset, len)`.
|
|
`src/llama-model-loader.cpp`, `llama_model_loader::load_all_data`: the per-tensor drop at the end
|
|
of the copy (non-mapping) branch and the whole-file drop in the final cleanup
|
|
(`size_done >= size_data`). `tools/mtmd/clip.cpp`, `clip_model_loader::load_tensors`: whole-file drop
|
|
after the tensor reads. Commit cce0ac1f0.
|
|
|
|
**How it works.**
|
|
- `drop_cache()` is `posix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED)` under `#ifdef __linux__`
|
|
(no-op elsewhere); `len` 0 means to the end of the file. Errors are logged at debug level only.
|
|
- `load_all_data()` drops each tensor's range right after it is read (all three copy paths: straight
|
|
into a host buffer, the async pinned-upload chunks, the staging buffer), so the cache never grows
|
|
past about one tensor during the load. The final cleanup drops every file whole, which catches the
|
|
GGUF metadata and the partial pages at tensor edges that a range drop leaves.
|
|
- Both run only when `mappings.empty()`, i.e. the loader holds no mapping of any of its files: not
|
|
`mmap`, `mmap+mlock` or `auto`, and no lazy tensors (`-lzm`; with lazy tensors `init_mappings()`
|
|
maps every file even under `-lm none`). Split GGUFs are covered (every entry of `files`).
|
|
- clip.cpp always copies the encoder through an ifstream (no mmap path exists), so it opens the
|
|
file again after the reads and drops it whole, unless `no_alloc`.
|
|
- Choice: per-tensor plus a final drop rather than a post-load drop only, because the per-tensor
|
|
drop also removes the peak during the load (the incident window: anon and file cache grow together)
|
|
for three extra lines. It costs about 0.27 s more than a post-load-only drop on the test model
|
|
below. Direct IO (`-lm dio`, upstream's) was measured as the alternative and is not better (section 4).
|
|
|
|
**Invariants that must hold.**
|
|
- Never drop for a file this loader maps: a mapped page would just be faulted back in (the kernel
|
|
also skips pages that are mapped into a process, so a violation costs refaults, not correctness).
|
|
- Drop only after the bytes are in our own buffers. Nothing in the fork re-reads the GGUF after
|
|
load: the VRAM release/restore (F4) uses host shadows, not the file.
|
|
- The drop is a hint: a failure must not fail the load.
|
|
|
|
**Failure modes and incidents.** None. Deployed 2026-10-25 (`2fc9f9c35`); on a 35B GPU load in
|
|
production the node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB. Costs to know
|
|
about:
|
|
- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a
|
|
5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few
|
|
seconds on a 26 s load.
|
|
- A reload of the same model can no longer hit a warm page cache (a warm-cache reload of the test
|
|
model took 4.6 s against 5.9 s cold). Under llama-swap a reload happens only after an eviction or
|
|
a crash.
|
|
- `TENSOR_DUPLICATED` tensors read the same range twice; the second read now comes from disk (small
|
|
tensors only).
|
|
|
|
**Rebase hazards.** The structure of `load_all_data()` (the `from_mapping` branch and its three copy
|
|
paths, the final cleanup block), `init_mappings()` creating mappings for lazy tensors, new load modes
|
|
in `llama_load_mode` (any new mode that maps must leave `mappings` non-empty), and the tensor-read
|
|
loop in clip.cpp `load_tensors`.
|
|
|
|
**Validation.** Not in `fork-validate.sh`. By hand on the pod with a scratch llama-server on a
|
|
scratch port, CPU only (`-dev none -ngl 0 --no-mmproj-offload`, no VRAM), cold cache (evict the test
|
|
files with python `os.posix_fadvise(..., DONTNEED)` first): page residency of the model and mmproj
|
|
files by `mincore` before and after the load (must be 0 after with `-lm none`, unchanged with
|
|
`-lm mmap`), the pod cgroup's `file`/`anon` in
|
|
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/memory.stat`, host `MemFree`, load time, and a greedy
|
|
request. First run (2026-10-25, gemma-4-E2B UD-Q4_K_XL 2.5 GiB + mmproj BF16 0.9 GiB, production
|
|
6c480f7e7 and the candidate with the drops disabled as controls, 4 rotated rounds):
|
|
|
|
| | drops off (control) | post-load only | per-tensor + final (committed) |
|
|
|---|---|---|---|
|
|
| model / mmproj resident after load | 2431-2444 / 940 MiB | 0 / 0 | 0 / 0 |
|
|
| peak model residency during load | 2.1-2.5 GiB | 2.1-2.4 GiB | 1.2-1.5 GiB (largest tensor) |
|
|
| pod cgroup `file` after load | +3.37 GiB | +0 | +0 |
|
|
| host MemFree after load | 0.6 GB | 2.5 GB | 2.5-2.8 GB |
|
|
| load time (median, production 5.85 s) | 5.87 s | 6.56 s | 6.82 s |
|
|
|
|
Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it counts clean page cache as
|
|
available. Greedy output byte-identical in all arms. `-lm mmap` control: the model file stays fully
|
|
resident and mapped (2498 MiB, `file_mapped` +2.5 GiB) after load and after a request, and only the
|
|
copied mmproj is dropped. `-lzm on` control: no drop (the file is mapped), 1252 MiB of copied
|
|
tensors stay cached. The GPU upload path and a 35B load were first exercised by the 2026-10-25
|
|
deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B
|
|
request took 34 s including load and answer).
|
|
|
|
### F11. Disk prompt cache (save slots on shutdown, restore after a restart)
|
|
|
|
**What and why.** Only one large model fits in RAM. Switching models in OpenWebUI makes llama-swap
|
|
SIGTERM the previous one, and switching back re-prefills the whole conversation, because the RAM
|
|
prompt cache died with the process (F4's reason for the arbiter). A 30k-token chat costs about 150 s
|
|
on the 35B at about 200 t/s. With `--cache-disk-path DIR`, each slot with a prompt is saved to DIR on
|
|
shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the
|
|
file, so only the suffix is processed. Measured on Qwen3.5-0.8B CPU only: a 12k-token turn 2 went
|
|
from a 72 s prefill to 1.2 s, of which the read took 307 ms and the restore 18 ms. Off by default.
|
|
Upstream PR #28092 (`--cache-disk`) was the reference for ideas, not ported (1700 lines). It
|
|
livelocked the kernel on AMD + Vulkan by DMA-ing GPU state into a MAP_SHARED mmap'd file, and it never
|
|
saved the slot in use on shutdown, which is the main case here.
|
|
|
|
**Where.**
|
|
- `tools/server/server-task.h`/`.cpp`: `server_prompt_disk_entry`, `server_prompt_disk_cache`
|
|
(`n_reuse`, `find`, `has`, `load`, `save`, `remove`); file IO helpers `disk_cache_io`,
|
|
`disk_cache_open`.
|
|
- `tools/server/server-context.cpp`: `DISK_CACHE_MIN_TOKENS`, `disk_cache_key`,
|
|
`disk_cache_collect`, `disk_cache_flush`, `disk_cache_load`; the init in `load_model` after the RAM
|
|
cache; the `disk_cache_load` call at the end of `get_available_slot`; `server_context::start_loop`
|
|
(collect, `vram_share_exit`, flush).
|
|
- `common/arg.cpp`, `common/common.h`: `--cache-disk-path` (`cache_disk_path`), `--cache-disk-size`
|
|
(`cache_disk_mib`, default 4096, 0 = no limit).
|
|
- Library (cold save): `llama_context_restore_memory_host()` (`include/llama.h`,
|
|
`llama_context::restore_memory_host`); `llama_memory_i::restore_device_buffers(bool to_host)` in
|
|
`llama-memory.h`, `llama-kv-cache`, `llama-memory-recurrent`, `-hybrid`, `-kv-cache-iswa`; the helper
|
|
`llama_memory_wrap_host()` in `llama-memory.cpp`.
|
|
- Commits dd851fd0c, 2874dcb10, 2c6c517ba (and f24c3582c, F4).
|
|
|
|
**How it works.**
|
|
- Save, after `start_loop()`'s loop returns. First `disk_cache_collect()` copies the state of every
|
|
slot with at least 256 text tokens to host memory (`llama_state_seq_get_data_ext`, flags NONE)
|
|
while the device buffers still exist. Then `vram_share_exit()` frees the VRAM and releases the GPU
|
|
token as before. Then `disk_cache_flush()` writes the files. The token release therefore waits
|
|
only for the device-to-host copy, never for disk IO.
|
|
- A slot is skipped when its KV does not cover its token list (`seq_pos_max + 1 != n_tokens`: the last
|
|
batch was not decoded, a shutdown during a token wait), when it holds media, and when the same
|
|
token list is already on disk.
|
|
- Cold server: with `LLAMA_SLEEP_EVICT_KV` the KV and recurrent state live only in host shadows (F4)
|
|
and their tensors have no buffer. `llama_context_restore_memory_host()` wraps each shadow as a
|
|
CPU_Mapped buffer (`ggml_backend_cpu_buffer_from_ptr`) and points the tensors into it, so there is
|
|
no copy, no allocation and no device use. It sets `device_discarded`. Without evict_kv the KV is still
|
|
in VRAM and is read directly.
|
|
- File, `<dir>/<key hash>-<token hash>.lpcd`: a 48-byte header (magic `LPCD`, version, key hash,
|
|
n_tokens, n_ckpt, payload size, payload hash), then 8-byte padded sections: tokens, checkpoint table,
|
|
state size, state, the state of each checkpoint. The hash is FNV-1a over the 64-bit words of the
|
|
payload. Write: plain fwrite to `.tmp`, fdatasync, `POSIX_FADV_DONTNEED`, rename. Read: plain fread,
|
|
then DONTNEED. There is no mmap anywhere.
|
|
- Index: at startup, from the headers, tokens and checkpoint tables of the files of this key. A bad
|
|
magic, a file size that differs from the header, or an inconsistent table means the file is removed.
|
|
Other keys and other versions are counted and left alone. A `.tmp` older than 1 h is removed.
|
|
- Key: the build commit, the model path + size + mtime, LoRA adapters and control vectors, n_ctx,
|
|
n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, K/V types, flash-attn type, and the RoPE/YaRN
|
|
parameters.
|
|
- Restore: at the end of `get_available_slot()`, after the RAM cache, only while warm, with
|
|
`cache_prompt` on and no media tokens. The file that reuses the most of the new prompt is loaded if
|
|
it reuses at least 256 more tokens than the slot holds. The slot's prompt goes to the RAM cache
|
|
first. Then the slot is cleared and the state is set; a failure clears the slot and removes the file.
|
|
- Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt,
|
|
or, for a recurrent or SWA model, a saved checkpoint at or below the common prefix can be used. A
|
|
plain-KV model can be cut back to any prefix. Exact prefix alone would almost never fire: the Qwen
|
|
template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is not a
|
|
prefix of the next OpenWebUI turn. Checkpoints are exact snapshots, and `update_slots()` rolls back
|
|
to one exactly as it does for the RAM cache; the recurrent state itself is never cut back.
|
|
Restored checkpoints get `id_task = -1`, so one that fails to load means a re-prefill, not an abort.
|
|
- Size limit: before a write, the least recently used `*.lpcd` files of every model in DIR are removed
|
|
until the new file fits. A restore updates the file's mtime. Entries the new prompt extends are
|
|
removed after the write and do not count against the limit. A write also needs its size + 1 GiB
|
|
free on the file system, so the cache never fills the PVC.
|
|
- Disabled with speculative decoding, with a warning: the draft state is not saved.
|
|
|
|
**Invariants that must hold.**
|
|
- Never restore while cold or without the GPU token (`disk_cache_load` returns when `vram_cold`).
|
|
`get_available_slot()` runs on the loop thread after the wake, like the RAM cache's `prompt_save`.
|
|
- Copy the state before `vram_share_exit()` (it discards the device buffers) and write after it (the
|
|
token is not held for disk IO).
|
|
- Never serve a file that fails the size or hash check. Never mmap the file: a DMA into a shared
|
|
mapping is what livelocked upstream's version on AMD.
|
|
- A write is atomic (tmp + rename), and any IO error leaves the server serving and the exit clean.
|
|
- The key must cover everything that changes the state layout or the KV values. Add a new
|
|
state-changing parameter to `disk_cache_key()`.
|
|
- `llama_context_restore_memory_host()` is for exit only; the wrapped buffers point into the shadows,
|
|
which must outlive them (both are members of the same memory object).
|
|
|
|
**Failure modes and incidents.**
|
|
- The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself
|
|
(CPU_REPACK, F4), not in the save.
|
|
- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full
|
|
KV allocation at exit (about 2.7 GB for the 35B at ctx 131072) on a node that has had an
|
|
OOM-controller kill. Replaced by the in-place wrap. The 32-byte alignment that
|
|
`cpu_buffer_from_ptr` asserts is met by rounding the base down; those leading bytes are never read.
|
|
- Writes to `/tmp` (node overlay) stalled once: 5.8 s for 64.5 MiB, against 171 ms normally.
|
|
- Costs to know about. llama-swap v239 waits for the old process to exit before it starts the next
|
|
model, and SIGKILLs a child still there 30 s after SIGTERM (measured 2026-10-25 with a scratch
|
|
llama-swap and dummy upstreams: term to exit 4.0 s, next start 2 ms later; a child that ignored SIGTERM
|
|
was replaced at +30.0 s). So the exit write adds directly to a model switch: about 1 s per GB on the
|
|
`/root` PVC (0.66 GB/s with fdatasync). A save cut off by the SIGKILL leaves only a `.tmp`.
|
|
- Only slots are saved. A conversation that sits only in the RAM cache when the process stops is lost.
|
|
- Every deploy empties the cache (the key holds the build commit), and so does any change to the
|
|
model file or the flags in the key.
|
|
|
|
**Rebase hazards.**
|
|
- `server_context::start_loop` and `vram_share_exit` ordering (F7); `get_available_slot` and the RAM
|
|
cache (`prompt_save`, `prompt_load`, `server_prompt_cache_state`); `server_prompt` and
|
|
`common_prompt_checkpoint` fields (new fields must be saved, or the format version bumped).
|
|
- The checkpoint rollback in `update_slots` (`pos_min_thold`, `TAG_PROMPT_LOGITS`), which decides how
|
|
much of a restored file is reused.
|
|
- The llama state format (`state_write`/`state_read` of the memory types). The key holds the build
|
|
commit, so a format change cannot load an old file; it only empties the cache.
|
|
- `llama_memory_i::restore_device_buffers` and the shadow layout of F4 (`device_buffer_shadow`):
|
|
`llama_memory_wrap_host` relies on the compact iteration-order shadow.
|
|
- Flag names in `common/arg.cpp`.
|
|
|
|
**Validation.** `fork-validate.sh --diskcache` (2c6c517ba), beside production. The model is the small
|
|
hybrid, CPU only (`-dev none -ngl 0`), with chat requests and greedy decoding, and a reference server
|
|
that never restarts. Cases: `restore`, `restore-arbiter` and `restore-cold` (turn 1, SIGTERM, restart,
|
|
turn 2 restored; prompt_n equal to the reference; identical answer), `no-prefix`, `key-ctx`,
|
|
`key-ctk`, `key-model`, `corrupt` (truncated and byte-flipped), `lru`, `write-error` (RLIMIT_FSIZE,
|
|
SIGXFSZ ignored), `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free. First run
|
|
(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219
|
|
MiB free).
|
|
|
|
| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill |
|
|
|---|---|---|---|---|---|---|
|
|
| 573-token chat | 64.5 MiB, 2 ckpt | 18 ms | 171 ms | 314 ms (token free 56 ms vs 28) | 166 + 3 ms | prompt_n 58 of 596, 462 vs 2913 ms |
|
|
| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms |
|
|
| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s |
|
|
|
|
Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.sh`):
|
|
`--diskcache --arbiter` 29 PASS, 0 FAIL, GPU legs included.
|
|
|
|
35B (`q35.py`, production 35B flags with `--cache-disk-path` on a scratch dir, arbiter env, an
|
|
18760-token pan-tadeusz chat; log `/root/arms/diskcache-test/q35.log`):
|
|
|
|
| leg | result |
|
|
|---|---|
|
|
| turn 1 prefill (reference and candidate) | 18760 tokens in 83.4 s (224.8 t/s), outputs identical |
|
|
| warm save on SIGTERM (a) | 556.5 MiB (2 checkpoints), copy 323 ms, write 960 ms; token free in 328 ms (6 ms without the cache), exit 1952 ms (607 ms without), peak RSS +430 MiB |
|
|
| restore after the restart (b) | read 860 ms, restore 114 ms; turn 2 prompt_n 27 of 18850 in 3.0 s (the reference that never restarted: 27 in 2.0 s); output identical |
|
|
| cold save on SIGTERM (b, KV in the host shadow) | 746.7 MiB (5 checkpoints), copy 310 ms, write 1403 ms, exit 2928 ms, peak RSS +432 MiB: the in-place wrap adds no copy |
|
|
| restore of the cold save (c, turn 2 sent again) | read 1189 ms, restore 75 ms, prompt_n 4; output differs from the reference's turn 2 (see below) |
|
|
|
|
Two anomalies in leg c, both explained:
|
|
- Output != reference: c sent turn 2 again, so the saved list (turn 2 + 63 generated tokens) extends
|
|
the new prompt. `update_slots()` then rolls back to the checkpoint at the prompt end minus 4 and
|
|
processes 4 tokens, while the reference processed turn 2 from turn 1's state (27 tokens). These are
|
|
different reuse points, not the same computation. The controlled test on Qwen3.5-0.8B
|
|
(`/root/arms/diskcache-test/cmp.py`) compares the same reuse point. Server X never restarts and
|
|
runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced
|
|
cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the
|
|
GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU
|
|
to itself: `/root/arms/diskcache-window2.sh` (about 6 minutes, under swapdown) is ready and not run.
|
|
Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the
|
|
GPU), not proven on that model.
|
|
- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
|
|
The driver's fake waiter was `flock -s want.lock sleep 120`; killing flock left `sleep` holding
|
|
want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right
|
|
after its load and again right after its request (log 0.37.50). That ordinary arbiter release
|
|
recaptures the full KV allocation as a host shadow, about 2.6 GB at ctx 131072 (F4), and the SIGTERM
|
|
arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The
|
|
driver now holds the lock in the process it kills, like `fork-validate.sh` does.
|
|
|
|
Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache
|
|
--cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is
|
|
created on first start. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
|
|
applies. Models only pick up the flags when they restart.
|
|
|
|
---
|
|
|
|
## 4. Dead ends
|
|
|
|
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
|
|
without a new reason.
|
|
|
|
| idea | result | mechanism |
|
|
|---|---|---|
|
|
| Weight pinning (`GGML_CUDA_REGISTER_HOST` and the host-register chain, from a third-party CUDA fork whose +64% was on an RTX 3060) | +17% pp2048 in isolated llama-bench (252 -> 294), fails in serving | RADV cannot import file-backed mmap pages; the pre-stage fallback fails (`ErrorOutOfDeviceMemory` for the ~18 GB expert region) or doubles host RAM. `-lm none` serves faster (Qwen3.6-35B 278 -> 286). Dropped in the 2026-09-09 rebase. |
|
|
| Expert-upload prefetch (`GGML_SCHED_PREFETCH_EXPERTS`) | -20% pp (other runs -9% to -22%), no tg benefit | Slots allocate in host-pinned GTT, not VRAM, so it stages host to host and still pays the PCIe read. `=1` means the default 3 slots. Dropped. |
|
|
| `GGML_VK_ASYNC_USE_TRANSFER_QUEUE=1` | flat (-0.1%), byte-identical | GCN is excluded from `prefers_transfer_queue` and RADV exposes no transfer-only queue family here; the per-layer upload-then-compute order leaves nothing to overlap. A standalone Vulkan microbench did show 60-73% of an H2D copy on queue family 1 hidden behind compute, so double-buffered prefetch inside one backend context remains an untested idea (estimated +10-15%, unproven). |
|
|
| Fractional `-ncmoe` | no measured win | Config only uses integers (22/40/48); upstream refactored `-ncmoe` into `llm_add_n_cpu_ffn_overrides`. Dropped. |
|
|
| MoE cache `--moe-cache-mib` 512 / 1536 / 2304 (Qwen3.6-35B, ncmoe 40, ctx 131072) | tg256@d4096 -42% / +5% / +12%; prefill +1-3%; idle VRAM 6.15 / 7.19 / 7.96 GB vs 5.66 | Small caches thrash (8-of-256 routing over 40 layers) and add a per-layer ids readback; only ~2.3 GiB pays and leaves no VRAM for a long KV. Not enabled. |
|
|
| MTP speculative decoding, Qwen3.6-35B | decode -39% (n1), -37% (n2), -50% (n3), -44% (n4), best -19% (n2, p-min 0.75); prefill -3 to -6% | A spec step costs ~180 ms vs 52 ms for plain decode; a 2-token verify alone is only 1.4x a 1-token step, so ~100 ms/step is draft machinery. The nextn layer adds 1.25 GiB VRAM. Entries removed. |
|
|
| MTP, gemma-4-26B-A4B | decode -57.9% (n1), prefill pp4k -34% | Same mechanism (~210 ms vs ~50 ms per step); MTP needs `--swa-full`, which alone costs -10.5% decode, -32% pp4k and saturates VRAM. Entries removed. |
|
|
| MTP, gemma E4B | decode -10.9% | Acceptance 0.41-0.67. The +19.5% pp seen with it comes from `--swa-full`, not MTP. |
|
|
| `--parallel 1` on gemma | no effect (-1.0%, +2.1%) | The Qwen gain is probably tied to its recurrent state being sized per sequence (unverified). |
|
|
| `-ub 4096 -b 4096` (Laguna) | flat (90.0 vs 90.9) | Fewer expert uploads cancelled by intra-ubatch attention growth. |
|
|
| mul_mat_id tile-padding theory (larger/odd `-ub`) | predicted 2.00x waste and a 1.94x cliff; measured +3.8% and -5.6%, no sawtooth | Theory dead; kills the mul_mat_id tile-selection shader work (an unbuilt, unverified WIP exists on branch `fable5/mmid-tile-wip`, with an unresolved correctness concern in its row_ids fill). |
|
|
| `-b` sweeps | non-monotonic (2048/8192 high, 4096/12288 low) | `-b 2048` is already at the top. |
|
|
| `-ncmoe` below 40 on the 35B | 34: +2.9% pp but 7.23 GiB peak at tiny context; 28: better at short context, worse at depth (137 vs 170 t/s at d32k); <= 24 collapses (~122) | Resident experts are evicted by KV/activation growth; cannot survive a 128k context. |
|
|
| `-ncmoe` 44 on Laguna | +1.5% | Does not fit at 64k context. Keep 48. |
|
|
| fp16 on Polaris (Mesa 26.1.6 + `radv_enable_float16_gfx8=true`, `fp16: 1`) | pp2048 -4.7%, -5.3% at d16k, tg +4% | Hurts prefill (the priority). Mesa 26.1.6 alone: +3.3% prefill, below the 5% bar; adopting it needs a pod image change. The pod's Mesa 25.0.7 has no such option. |
|
|
| KV quant on Qwen (q4_0, q8_0) | f16 beats q4_0 by +40% pp at 64k; q8_0 170 -> 128 t/s at depth; q4_0 with upstream PR #30738 still -16% vs f16 at d32k (133.6 vs 159.4) | No int-dot/fp16 here, so dequant is pure compute; the hybrid model's KV is small anyway. Keep f16. (PR #30738 was +15% at d16k on gemma-26B with q4_0 KV; not merged upstream as of the 20261024 rebase.) |
|
|
| `-fa 0` | 5x slower at depth (170 -> 35 at d32k) | Keep flash attention on. |
|
|
| FA scalar `block_rows` 16 -> 32 | 3.5x regression at d32k (135 -> 38) | ~42 KB shmem per workgroup at head_dim 256, one workgroup per SIMD, occupancy collapse. |
|
|
| `GGML_VK_DISABLE_GRAPH_OPTIMIZE=1` | restores tg32 at n_ctx=32 (10.44 -> 15.61), nothing at real depth | Upstream #27812's view-alias fix makes KV views look dependent and collapses barrier-free batches; only visible when per-node work is tiny. It is a correctness fix; do not disable it in serving. |
|
|
| `--no-mmap`/`-lm none` for Laguna-S 118B | does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. |
|
|
| Laguna `--cache-reuse`, `--swa-full`, `-t 12`, KV quant | ruled out | `--cache-reuse` is force-disabled for iSWA when base and SWA sizes differ. |
|
|
| CPU-only task model (`-ngl 0`) to avoid handoffs | rejected | At `--ctx-size 8192` titles could face 8k tokens of CPU prefill. The arbiter is the answer. |
|
|
| `-lm dio` (direct IO) instead of the F10 page-cache drop | load +0.9 s on 3.4 GiB (6.89 vs 5.99 s), the same cost as the drop; 175 MiB of the model file and all of the mmproj stay cached | Synchronous 64 MiB O_DIRECT chunks with no readahead; the GGUF metadata goes through a separate buffered FILE and clip.cpp has no direct-IO path. Measured CPU only; reads into pinned Vulkan buffers may fall back to buffered IO (`EFAULT` or `EINVAL`). Not adopted. |
|
|
|
|
The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256
|
|
tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.
|
|
|
|
Where the time goes (Laguna-S-2.1 118B IQ2_M, all experts on host, `GGML_VK_PERF_LOGGER=1`, depth 0,
|
|
22.65 s per 2048-token eval, 17.18 s of it GPU-busy, so about 24% is H2D stall; moved here from the
|
|
old README section): expert `MUL_MAT_ID` 52%, attention projections 24%, `FLASH_ATTN_EXT` 16%,
|
|
everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s while dense `MUL_MAT` q5_K/q6_K in
|
|
the same graph reaches 3096-3691, so closing that gap is the largest remaining opportunity on this
|
|
card, not the transfer path. On interleaved-SWA models the sliding-window layers keep a small KV
|
|
(`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`), so they cost the same at any depth; on Laguna 36 of
|
|
48 layers are O(1) in depth and the whole high-context slowdown comes from the 12 full-attention
|
|
layers.
|
|
|
|
## 5. Configuration that matters
|
|
|
|
`/root/config.yaml` (llama-swap) and `/root/supervisord.conf` on the pod are the source of truth;
|
|
this section explains the choices. Values as of the 20261024 deploy.
|
|
|
|
Qwen 35B-A3B MoE entries (macro `moe_offload_args`, applied to all 35B-A3B variants):
|
|
`--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1`,
|
|
context 131072.
|
|
|
|
- `-lm none`: load weights into RAM (upstream removed `--no-mmap`; the old spelling makes the server
|
|
refuse to start). Faster in serving than mmap and recommended for `--n-cpu-moe`. Exception:
|
|
Laguna-S, which must use mmap (GTT limit, section 4).
|
|
- `-ub 2048 -b 2048`: the biggest prefill lever. ub512 = 97 t/s, ub1024 = 159, ub2048 = 254 on the
|
|
35B; the per-eval expert upload is amortized over the ubatch.
|
|
- `--n-cpu-moe 40`: all experts on host. Lower values are faster at short context but run out of VRAM
|
|
at long context (section 4). Measured idle VRAM at ctx 131072: 5.66 GB.
|
|
- `--parallel 1`: +22.9% decode on the Qwen 35B (15.5 -> 19.1 t/s), prefill +0.3-1.2%, +64 MiB VRAM.
|
|
Without it the server creates 4 slots and a unified KV. Two concurrent requests queue on slot 0.
|
|
With one slot a ubatch never holds more than one sequence, which the concat fast path's
|
|
`ne[2] == 1` condition (F3) needs.
|
|
- `-t 6`: six physical cores; more threads hurt decode.
|
|
- `-fa 1`: essential (section 4).
|
|
- f16 KV (no `-ctk/-ctv`): section 4.
|
|
|
|
Other models:
|
|
- gemma-4-26B-A4B: macro `moe_offload_gemma`, same as above but `--n-cpu-moe 22` (8 layers of experts
|
|
resident; gemma's SWA keeps KV small at depth). ncmoe sweep pp2048: 30 = 262, 24 = 266, 18 = 272,
|
|
14 = 157 (VRAM collapse); 22 keeps headroom for 128k context (7.40 of 8 GB at 128k).
|
|
Its KV cache is `-ctk q4_0 -ctv q4_0`, set in the `gemma4_sampling` / `gemma4_nothink_sampling`
|
|
macros (not in `moe_offload_gemma`); note gemma has no `--parallel 1` (no measured gain there).
|
|
- Small models that fit in VRAM (Qwen3.5-4B, gemma E4B/E2B and similar) use auto-fit
|
|
(`common_args`); forcing experts to host would slow them.
|
|
- The 4B task model runs with `--no-mmproj` and ctx 8192, and is preloaded.
|
|
- Laguna-S-2.1 118B IQ2_M (configured as `Laguna-S-2.1-IQ2_M`, matrix var `lag`): mmap, `--n-cpu-moe 48 -b 2048 -ub 2048 -fa 1 --parallel 1`,
|
|
ctx 65536. Its exclusivity set must allow the task model (`lag & q4task`); making it exclusive with
|
|
the task model killed it on every title request and forced full re-prefills. `--parallel 1` frees
|
|
223 MiB of VRAM at 64k context there (4 auto slots size the SWA cache `n_swa * n_seq_max +
|
|
n_ubatch`, 4096 cells instead of 2560).
|
|
|
|
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
|
|
`/root/supervisord.conf`:
|
|
- `LLAMA_SLEEP_VRAM_ONLY=1`: enables the arbiter and VRAM-only sleep (F6).
|
|
- `LLAMA_VRAM_ARENA=/dev/shm/llama-vram`: the shared arena. Every co-loaded GPU model must use it.
|
|
- `LLAMA_SLEEP_EVICT_KV=1`: cold models also free KV, recurrent state, scheduler and scratch (F4),
|
|
so the warm model gets nearly the whole card. Costs a 2-3 s re-warm instead of ~0.6 s.
|
|
- `LLAMA_MMPROJ_ONDEMAND=1`: encoder in VRAM only during encodes (F5). No-op for models without an mmproj.
|
|
- No `--sleep-idle-seconds`: release is purely demand-driven via want.lock.
|
|
- `GGML_CUDA_REGISTER_HOST=1` may still be present from the pinning era; it is a no-op with `-lm none`.
|
|
|
|
Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`,
|
|
`GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`.
|
|
`--cache-disk-path` (F11) is not set yet: it is not deployed; the recommended flags are in F11.
|
|
|
|
Machine-level:
|
|
- GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted
|
|
every second; default/max cap 145 W). The two GPU hangs of 2026-10-09 happened before the cap was
|
|
fixed (it used to switch between 120 and 145 W by temperature); heavy runs since have been clean.
|
|
- Kernel argument `amdgpu.lockup_timeout=0,120000,0,0` (compute ring 120 s) via Talos
|
|
`machine.install.extraKernelArgs`. Without it long 64k-context submissions hit the compute-ring
|
|
watchdog (`ring comp_1.x.0 timeout`).
|
|
- Upstream changed the default `--port` from 8080 to 9931; the config passes `--port ${PORT}`
|
|
explicitly, so it is unaffected.
|
|
|
|
## 6. Measurement methodology and traps
|
|
|
|
- **Cross-run variance is about 7%.** llama-bench's within-run error bars understate cross-invocation
|
|
variance by about 14x (the same config read 276.7 +/- 0.9 and 296.4 +/- 0.6 in two invocations).
|
|
Put every comparison you draw a conclusion from inside one llama-bench invocation
|
|
(comma-separated sweeps). Two builds cannot share an invocation, so interleave them over at least 3
|
|
rounds and compare medians. Treat any cross-build delta under about 5% as unproven.
|
|
- **Alternate arm order.** A fixed order gave the later slot about a 9% advantage and nearly produced
|
|
a false regression finding. Use orders like ref, cand, cand, ref.
|
|
- **Node CPU contention.** The node runs about 60 pods; llama-bench once got 4.67 of 12 hardware
|
|
threads while asking for 6. With `--n-cpu-moe 40` every MoE layer decodes on the CPU, so stolen
|
|
threads map onto tg almost directly. Prefill is nearly immune (held +-0.3% while tg32 swung +-14%).
|
|
Expect occasional catastrophic tg samples; use medians, never means. Historical tg figures taken
|
|
under different load are not comparable.
|
|
- **Never use tg32.** At the default depth (n_ctx 32) decode is launch/barrier bound and measures
|
|
command-stream overhead: two upstream commits read -36% at tg32 and were at parity at tg256 (and +7%
|
|
at d512). Use tg256 (or tg128) with an explicit depth (`-p 0 -n 256 -d 4096`). tg is not monotonic in
|
|
n_ctx on this box (n_ctx 256 measured slower than 32 and 544), so only compare same-depth cells.
|
|
- **`-b Vulkan` vs `-b Vulkan0`.** `test-backend-ops -b Vulkan` matches nothing, runs 0 cases and
|
|
still prints OK with exit 0. The backend is `Vulkan0`. Always check the case count.
|
|
- **ubatch larger than the prompt never fills.** With `-ub 2048` use prompt lengths that are
|
|
multiples of 2048. A partial trailing ubatch costs a full expert upload.
|
|
- **`-tb` is not a llama-bench flag** (server only). Any bad flag makes llama-bench print usage and
|
|
exit, which looks like a run that produced no rows. Always check the row count. `-ncmoe` is
|
|
llama-bench's spelling of `--n-cpu-moe`.
|
|
- **llama-bench prints its table only after all `-d` values finish.** No output does not mean hung.
|
|
- **Nothing else may use the GPU** during a measurement; stop llama-swap (`bench.sh` and
|
|
`/root/bench/moe/swapdown.sh` do this and restart it from a trap).
|
|
- **Check the build finished** before benchmarking it (library mtime earlier than the run start).
|
|
Never overlap a build with a big benchmark (host OOM).
|
|
- **Prompt-cache reuse is `timings.prompt_n`,** not `tokens_evaluated` (which is always the full
|
|
prompt length).
|
|
- **Per-process VRAM** from `/proc/<pid>/fdinfo/*` `drm-memory-vram` must be deduplicated by
|
|
`drm-client-id`; summing all fds double-counts shared buffers. `drm-memory-gtt` is system RAM, not
|
|
VRAM. Whole-card usage: `/sys/class/drm/card*/device/mem_info_vram_used`.
|
|
`GGML_VK_MEMORY_LOGGER=1` logs every Vulkan buffer alloc/free with a running device total to stderr.
|
|
- **The concat fast path is invisible to test-backend-ops** (F3); validate it with the knob, perf
|
|
logger timing, or byte-identical greedy generation (temperature 0, top_k 1, fixed seed,
|
|
`cache_prompt: false`).
|
|
- **Do not run `llama-cli` detached with closed stdin** on the pod: it can loop forever printing
|
|
prompts (it once wrote a 65 GB log). Use llama-bench `-d` or the server harness.
|
|
|
|
## 7. Known gaps and open risks
|
|
|
|
- `llama_context::restore_device()` ignores the `bool` returned by `restore_device_weights()`,
|
|
`moe_cache->restore_device()` and `memory->restore_device_buffers()`. On a restore OOM,
|
|
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
|
|
null buffer would crash. Only the mmproj path handles a failed restore.
|
|
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
|
|
- Router mode can spin at shutdown (F7); not used here.
|
|
- The unexplained ~1.6 GiB peak-VRAM spike in the first candidate `--perf` arm, seen twice (F9).
|
|
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
|
|
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
|
|
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
|
|
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
|
|
is an estimate; its memory effect is verified in production.
|
|
- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the
|
|
35B, only on the 0.8B (CPU and GPU). `/root/arms/diskcache-window2.sh` is ready (about 6 minutes of
|
|
downtime).
|
|
- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's
|
|
host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time
|
|
(token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that
|
|
shadow while it is cold.
|
|
- F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key
|
|
holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB.
|