Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
1672 lines
125 KiB
Markdown
1672 lines
125 KiB
Markdown
# FORK.md - the RX 580 llama.cpp fork
|
|
|
|
Maintainer documentation for this private fork. Read it before you rebase, change, or deploy
|
|
anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md),
|
|
which also holds the operations reference (pod access, files, recovery).
|
|
|
|
Status at the time of writing (2026-11-22):
|
|
|
|
- Production runs `b13c41355` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
|
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, and F13, the
|
|
per-pass loop views for Inkling), deployed 2026-11-22 08:47 with the Inkling placement change
|
|
(section 5, F13). Validated with the full `fork-validate.sh` set in one window (52 PASS, 0 FAIL,
|
|
0 SKIP; runbook section 6). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-15.txt` = `3621025e8`
|
|
(build copy `/root/arms/build-3621025e8`, config backup `/root/config.yaml.bak-20261122-predeploy`;
|
|
restore both, the old build has no F13); next free pointer 16.
|
|
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12683-b13c41355`) and changes
|
|
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
|
|
(section 3, "2026-11-21").
|
|
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
|
|
`adc2aa3ec` until 2026-11-21), kept untouched as the safety net. The fork commits are
|
|
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
|
|
(this file among them), so count it rather than trusting a number written here. Section 2 maps
|
|
every commit to a feature.
|
|
- The disk prompt cache is ON in production (F11). `2fc9f9c35` was validated with the full
|
|
`fork-validate.sh` set (37 PASS, 0 FAIL). Any later commit that touches compiled code needs the
|
|
same validation before a redeploy.
|
|
- Branch `fable5/disk-prompt-cache` added F11 (disk prompt cache) and a CPU_REPACK release fix (F4).
|
|
A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); the fix (adc2aa3ec) passed
|
|
the 35B run with the production command line on 2026-11-21 and is deployed.
|
|
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
|
|
its subject line. The commit messages are detailed and are the primary record of each mechanism
|
|
and measurement; read them with `git show`.
|
|
- Commit trailers (owner's convention): this fork is private and never contributes upstream.
|
|
Agent-written commits end with `Co-Authored-By: Claude <model> <noreply@anthropic.com>` (some
|
|
older ones say `Assisted-by: Claude`). AGENTS.md asks agents for `Assisted-by:` instead of
|
|
`Co-authored-by:` and forbids AI-written commit messages and PR descriptions; those are upstream
|
|
llama.cpp's contribution rules and do not apply to commits that stay in this fork. Anything meant
|
|
for upstream must be written by the owner under those rules.
|
|
|
|
## Contents
|
|
|
|
1. Purpose and target
|
|
2. Commit map
|
|
3. Features
|
|
- F1 MoE routing-ids readback skip
|
|
- F2 GCN flash-attention mask_opt gate (and the shmem-staging note)
|
|
- F3 Vulkan concat tiled-transpose fast path
|
|
- F4 On-demand device residency (library side)
|
|
- F5 On-demand mmproj (multimodal encoder)
|
|
- F6 Cross-process VRAM arbiter (server side)
|
|
- F7 Server shutdown path
|
|
- F8 Cold-teardown guards
|
|
- F9 Benchmark harness and validation script
|
|
- F10 Page-cache drop after a copying load
|
|
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
|
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
|
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
|
- F14 User-message batch-break merge (one ubatch per short turn)
|
|
4. Dead ends
|
|
5. Configuration that matters
|
|
6. Measurement methodology and traps
|
|
7. Known gaps and open risks
|
|
|
|
---
|
|
|
|
## 1. Purpose and target
|
|
|
|
The fork exists to serve large MoE models well on one specific machine:
|
|
|
|
- **GPU:** AMD RX 580 8 GB (Polaris, GCN gfx803), RADV Vulkan driver. llama.cpp reports
|
|
`fp16: 0` (RADV does not expose `shaderFloat16` on GFX8), no int-dot, no matrix cores, wave64.
|
|
PCIe 3.0 x16. RADV exposes no transfer-only queue family on this card.
|
|
- **Host:** Ryzen 5 3600 (6c/12t), 62 GB RAM, a Talos Kubernetes node shared with about 60 other pods
|
|
(CPU contention matters for decode numbers, see section 6). The GPU runs at a fixed 120 W power cap.
|
|
- **Primary model:** Qwen3.6-35B-A3B (unsloth UD-Q4_K_XL, 20.8 GiB), a hybrid MoE: 40 layers, 30 of
|
|
them gated delta-net (recurrent), 10 full attention, head_dim 256, 256 experts with 8 used. All
|
|
experts live in host RAM (`--n-cpu-moe 40`); only the backbone, KV, recurrent state and compute
|
|
buffers sit in VRAM.
|
|
- **Serving:** llama-swap under supervisord keeps several llama-server processes loaded at once
|
|
(the 35B main models, a Qwen3.5-4B "task" model OpenWebUI calls for titles and tags, gemma-4-26B
|
|
multimodal, others). They cannot all fit in 8 GB, so they time-share the GPU through the fork's
|
|
VRAM arbiter (F4-F7) without restarting processes, which would throw away the RAM-only prompt cache.
|
|
- **Priority:** prefill (prompt processing) speed first, decode second. Agentic use pushes large tool
|
|
results through the model, so long-context prefill is what users wait on.
|
|
|
|
What the fork adds, in short:
|
|
|
|
- three Polaris prefill wins (F1, F2, F3), all exact (token-identical to mainline);
|
|
- on-demand release/restore of everything a model holds in VRAM, plus a cross-process arbiter that
|
|
hands the GPU from one llama-server to another in well under a second for small models;
|
|
- fixes to the server's sleep, wake and shutdown paths that the arbiter exposed;
|
|
- a benchmark harness and a post-rebase validation script.
|
|
|
|
## 2. Commit map
|
|
|
|
Oldest first. "Feature" points at the section below.
|
|
|
|
| commit | subject | feature |
|
|
|---|---|---|
|
|
| c72cb424e | llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert | F1 |
|
|
| fb6a8a076 | ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes | F2 |
|
|
| ff2a96d2d | vulkan: record why flash-attn shmem staging stays off for GCN | F2 |
|
|
| 3df719e1e | ggml-vulkan: tiled transpose fast-path for concat with transposed source | F3 |
|
|
| 481be8b28 | server: on-demand VRAM sharing to time-share one GPU between models | F4, F6 |
|
|
| 17165fe2a | server: optionally evict the KV cache too (Phase 2 of VRAM sharing) | F4 |
|
|
| 958257cd6 | server: keep the sleep-wake path active when the VRAM arbiter is enabled | F6 |
|
|
| b9df854de | server: coordinate model load with the VRAM arbiter to avoid load-time OOM | F6 |
|
|
| 8d80f0673 | llama: guard buffer iteration against released device buffers | F8 |
|
|
| 2feb6aeb2 | server: restore weights/KV on wake, not at decode, so KV eviction is safe | F6 |
|
|
| bf2c1a699 | server: also free the compute-graph scheduler on KV eviction | F4 |
|
|
| 9857a45f0 | llama: evict recurrent/SSM state on device release | F4 |
|
|
| 9b555f1aa | ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc | F4 |
|
|
| ad27d4921 | llama: free backend compute scratch on cold; guard memory_breakdown | F4, F8 |
|
|
| 0a0f3ab6d | mtmd: release/restore the encoder weights from VRAM on demand | F4, F5 |
|
|
| 5e99ac198 | server: on-demand mmproj - free encoder VRAM on the text path | F5 |
|
|
| b7c131219 | docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends | docs |
|
|
| 5e429d4a9 | ggml-vulkan: skip the concat transpose fast path below one tile | F3 |
|
|
| 4c4486f8e | scripts: add the RX 580 benchmark harness | F9 |
|
|
| 878d1338f | server: replace the VRAM doorbell with a want-lock | F6 |
|
|
| 53f907771 | llama: skip the compute-buffer size check when the context is cold | F8 |
|
|
| 7ac2014a8 | llama: release/restore the MoE cache with the device weights | F4 |
|
|
| 54a7f534b | scripts: rx580-bench - use -lm none, document the MoE cache option | F9 |
|
|
| 11044b35e | server: run the shutdown handler off the signal-interrupted thread | F7 |
|
|
| 6c480f7e7 | server: release the GPU token promptly on shutdown | F7 |
|
|
| e085c347d | ggml-vulkan: env knob for the concat transpose fast path | F3 |
|
|
| 149a8cb0a | scripts: add fork-validate.sh, post-rebase validation of the fork | F9 |
|
|
| bfbaf15b2 | docs: FORK.md and rebase runbook for the RX 580 fork | docs |
|
|
| f5c0d8f28 | server: fix stale VRAM-arbiter comments and the sleep log line | F6 |
|
|
| f664b0f20 | llama: env knob for the routing-ids readback skip | F1 |
|
|
| 62d442e7b | scripts: rx580-bench - decode at tg256 depth 4096, not tg32 | F9 |
|
|
| 2512290f9 | scripts: fork-validate.sh - gate --perf on medians, not means | F9 |
|
|
| 80e662a2a | scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 |
|
|
| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
|
|
| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
|
|
| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 |
|
|
| f24c3582c | llama: do not release CPU extra buffers (CPU_REPACK) on device release | F4 |
|
|
| dd851fd0c | llama: read a released KV/recurrent state from its host shadow | F4, F11 |
|
|
| 2874dcb10 | server: disk prompt cache, save slots on shutdown and restore after a restart | F11 |
|
|
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
|
|
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
|
|
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
|
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
|
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (on `fable5/userstart-split`, not deployed) | F14 |
|
|
|
|
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
|
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
|
`6c480f7e7` was the previous production commit. The last four rows are on `fable5/disk-prompt-cache`
|
|
only (hashes of that branch), not deployed.
|
|
|
|
---
|
|
|
|
## 3. Features
|
|
|
|
Every feature section uses the same headings: What and why; Where; How it works; Invariants that
|
|
must hold; Failure modes and incidents; Rebase hazards; Validation.
|
|
|
|
### F1. MoE routing-ids readback skip
|
|
|
|
**What and why.** With experts host-resident, each split uploads the experts it needs. To choose
|
|
them, the expert-copy callback reads the routing ids back to the host. The ids are produced on the
|
|
same GPU the upload goes to, so the readback forces a full pipeline flush: 47 per eval on a
|
|
48-layer MoE. At prefill batch sizes every expert is used anyway (2048 tokens x 10 experts over 256
|
|
experts), so the readback buys nothing. Measured on Laguna-S-2.1 118B IQ2_M, all experts on host
|
|
(before it was ported to the current callback): pp2048 +4.7% at depth 32k, +0.9% at 16k, flat at
|
|
depth 0, tg +3%.
|
|
|
|
**Where.** `src/llama-context.cpp`, `llama_context::sched_copy_experts` (registered with
|
|
`ggml_backend_sched_set_copy_callback` in the scheduler setup). Commit c72cb424e.
|
|
|
|
**How it works.** When `ggml_nelements(ids) >= 4 * n_expert`, it skips
|
|
`ggml_backend_tensor_get_async` + `ggml_backend_synchronize`, clears `st.ids_data` and sets
|
|
`st.used.assign(n_expert, true)`. The existing grouping then copies the whole expert tensor as one
|
|
range. Below the threshold the upstream readback runs unchanged. For Qwen3.6-35B (256 experts, 8
|
|
used) the skip applies to any ubatch of 128 tokens or more; decode (8 ids) never reaches it.
|
|
|
|
**Invariants that must hold.**
|
|
- Exactness: uploading an expert no id selects cannot change the result, because `mul_mat_id` only
|
|
reads the rows the ids point at. Any upstream change that makes unselected experts matter (for
|
|
example a kernel that reads every uploaded expert) breaks this argument.
|
|
- With `--moe-cache-mib` on, the all-true set still goes through `moe_cache->copy_experts`, which
|
|
reads the host slot maps (no device readback) and copies cached experts device-to-device. Still exact.
|
|
- `st.ids` caching stays coherent: a repeat call with the same ids tensor reuses the all-true set,
|
|
which is what the threshold would decide again; a different `n_expert` recomputes.
|
|
- The small-batch MoE-cache slot path (`llama_moe_cache::copy`) has its own readback and is not touched.
|
|
|
|
**Failure modes and incidents.** None in production. Historically the change lived in
|
|
`ggml_backend_sched_compute_splits` in `ggml/src/ggml-backend.cpp`; upstream #29943 (`6753a033f`)
|
|
deleted that block and moved expert copying into `llama_context::sched_copy_experts`. The original
|
|
cherry-pick therefore had nothing to apply to and was re-authored as c72cb424e. A `-X theirs`
|
|
resolution would have resurrected the deleted scheduler block.
|
|
|
|
**Rebase hazards.**
|
|
- Any change to `sched_copy_experts` or the `st` state struct (`ids`, `used`, `ids_data`).
|
|
- Upstream `701fd25b3` (MUL_MAT_ID id -1 skips work) left a `GGML_ASSERT(id >= 0 ...)` in the
|
|
readback branch; the skip branch reads no ids, so it is unaffected, but watch for upstream
|
|
semantics around negative ids.
|
|
- If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.
|
|
|
|
**Validation.** `fork-validate.sh --moe-exact MODEL` (needs the GPU to itself, run it in the
|
|
`--perf` swapdown window with the 35B): reads `expert_count`/`expert_used_count` from the GGUF,
|
|
computes the threshold ceil(4 * n_expert / n_expert_used) tokens per ubatch (128 on the 35B), and
|
|
compares 64 greedy tokens after a prompt of about twice that, every expert on the host:
|
|
candidate == reference (`identical-ref`), and with the `LLAMA_MOE_IDS_SKIP` knob (f664b0f20;
|
|
`=0` forces the readback, `=log` reports each skip on stderr) candidate == candidate with the skip
|
|
forced off (`identical-noskip`, which proves the skip exact even when the reference build carries
|
|
it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). First GPU run
|
|
2026-10-25 (35B, 2fc9f9c35 vs 6c480f7e7): prompt 397 tokens, 39 skips logged, 0 with the skip forced
|
|
off, output identical in all three runs, 78 s. Also `--ops` (the MUL_MAT_ID
|
|
kernel, not this callback), `--perf` (prefill regression) and `--moecache` (the cache path through
|
|
this callback).
|
|
|
|
### F2. GCN flash-attention mask_opt gate (and the shmem-staging note)
|
|
|
|
**What and why.** `mask_opt` skips fully-masked causal blocks and drops the per-block mask load+add
|
|
on fully-visible blocks. Upstream disables it on AMD GCN for head sizes <= 256. On Polaris attention
|
|
at depth is compute-bound on the softmax path (exp runs at about 1/4 VALU rate, no matrix cores), so
|
|
skipping that work is a real win. Measured on Qwen3.5-35B-A3B (head_dim 256): pp2048 +1% at depth 0,
|
|
+7.8% at 16k (176 -> 190), +12.3% at 32k (136 -> 153), growing with depth. Perplexity bit-identical
|
|
(5.1668 on and off).
|
|
|
|
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`, `ggml_vk_flash_attn`, the `use_mask_opt`
|
|
condition. Commit fb6a8a076. The shmem-staging note is a comment in `get_fa_tuning_params_scalar`
|
|
(commit ff2a96d2d, no behaviour change).
|
|
|
|
**How it works.** The GCN clause `HSK > 256 || HSV > 256` becomes `HSK >= 256 || HSV >= 256`. The
|
|
existing large-mask conditions (`nem1 >= 32`, `nem0 * nem1 > 32768`,
|
|
`nem0 >= block_cols * 16`) keep it off for decode.
|
|
|
|
**Invariants that must hold.**
|
|
- Only GCN with head size >= 256 is affected. Do not lower the threshold: relaxing it below 256 was
|
|
-18.5% at 16k on Laguna (head_dim 128).
|
|
- Upstream's `!use_sparse` term must stay in the condition (sparse flash-attention came in with
|
|
upstream `fc82583e6`).
|
|
- `shmem_staging` stays NVIDIA-only. Measured on GCN at head size 128: -6.7% at 16k, -7.4% at 32k.
|
|
The `kvsh` stride of `D/4+1` dwords is 4 mod 32, an 8-way LDS bank conflict on wave64 (the `+1`
|
|
padding is tuned for warp32), and the extra shared memory cuts occupancy. If anyone retries it,
|
|
fix the padding to +2/+3 first.
|
|
|
|
**Failure modes and incidents.** A `-X theirs` resolution in a trial rebase silently dropped
|
|
`!use_sparse` from the condition. No production incident.
|
|
|
|
**Rebase hazards.** Any upstream rewrite of `use_mask_opt`, of the FA tuning parameters, or of
|
|
architecture detection (`vk_device_architecture::AMD_GCN`). Upstream `733f57a99` added a path that
|
|
copies strided f16/bf16 K/V into `prealloc_x` scratch for flash-attention on AMD non-RDNA4 (so GCN
|
|
is included) when `neq1 >= 256`; measured at d32768 it adds about 68 MiB of VRAM and no visible
|
|
long-context prefill loss, but it has no env toggle, so its speed effect could not be isolated.
|
|
|
|
**Validation.** `fork-validate.sh --ops` (FLASH_ATTN_EXT, 5332 cases on Vulkan0) and `--perf`
|
|
(pp2048 at depth 16k and 32k is where this feature shows).
|
|
|
|
### F3. Vulkan concat tiled-transpose fast path
|
|
|
|
**What and why.** Delta-net layers build their conv input with `ggml_concat(conv_states,
|
|
ggml_transpose(qkv_mixed), 0)`. The generic concat shader reads the transposed source with an
|
|
uncoalesced stride: about 5.6 ms for a 16 MB concat on the RX 580 against about 130 us of memory
|
|
bandwidth. On Qwen3.6-35B the fast path alone is +14% pp8192 and accounts for more than all of the
|
|
fork's prefill win at depth 0 (the rest of the fork costs about 1% there). In the validation run on
|
|
Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 1945 us without.
|
|
|
|
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`: `ggml_vk_concat` (gate lambda
|
|
`src_transposed_2d` and the `s1_trans`/`s0_trans` checks) and `ggml_vk_concat_transpose_fastpath`.
|
|
The graph pattern that matters here comes from `llm_build_delta_net_base::build_conv_state` in
|
|
`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next). Commits 3df719e1e (fast path),
|
|
5e429d4a9 (decode gate), e085c347d (env knob).
|
|
|
|
Other graphs that build `ggml_concat(state, ggml_transpose(x), 0)` (found with
|
|
`git grep -n "ggml_concat.*ggml_transpose" src/models/` plus the two-line form in delta-net-base and
|
|
lfm-base): `qwen4exp.cpp` (`build_conv_state_at`), `bailingmoe3.cpp`, `clef.cpp`, `kimi-linear.cpp`,
|
|
`kimi-k3.cpp`, `glm5-next.cpp`, `mamba-base.cpp` (two sites, the Mamba/Mamba2 conv and the hybrids
|
|
built on it), `plamo2.cpp`, and `lfm-base.cpp` (causal short conv; the transpose is assigned to `bx`
|
|
first). The fast path takes any of them when the gate holds (transposed source a plain 2D transpose
|
|
of a contiguous f32/f16 tensor, one sequence, `ne[0] >= 32`), so a change to the gate or the shader
|
|
affects all of them. Only the delta-net path (Qwen3.5/3.6) has been measured and validated; the
|
|
`--concat` check uses qwen35.
|
|
|
|
**How it works.** For a concat along dim 0 where one source is stored transposed (`nb[1] ==
|
|
type_size`, `nb[0] == ne[1] * type_size`) and the other source and dst are contiguous, it issues two
|
|
dispatches: `pipeline_cpy_32`/`_16` copies the contiguous source into its dst columns, and
|
|
`pipeline_cpy_transpose_32`/`_16` (the shared-memory 32x32 tiled transpose in `copy_transpose.comp`)
|
|
writes the transposed source into the other columns, both with custom push constants and a dst
|
|
offset. It ends with `ggml_vk_sync_buffers`. Everything else falls through to the generic path.
|
|
|
|
The gate requires: f32 or f16 with all three types equal (quantized and integer types stay generic);
|
|
matching ne[1..3]; `ne[2] == 1 && ne[3] == 1` on the transposed source (so it fires only with one
|
|
sequence per ubatch, which `--parallel 1` gives); the 16-bit `doffset` bound; and
|
|
`ne[0] >= 32` (TILE_DIM) on the transposed source.
|
|
|
|
The decode gate (5e429d4a9): at one token the transposed source is a single column, so there is no
|
|
stride to fix, but the path still costs two dispatches and the unconditional `ggml_vk_sync_buffers`,
|
|
a full pipeline barrier (the generic path emits none). That barrier, not the half-empty transpose,
|
|
was nearly all of a -17% tg256 regression, because the path ran 30 times per decoded token on the
|
|
35B. The transposed source is `qkv_mixed` transposed, so its `ne[0]` is the ubatch token count:
|
|
prefill keeps the fast path, decode and small speculative drafts take the generic one. Measured
|
|
with the gate: decode back to parity (gated/clean 1.023), prefill win retained (gated/ungated 0.998,
|
|
pp8192 +11% over clean), generation byte-identical.
|
|
|
|
The knob (e085c347d): `GGML_VK_CONCAT_FASTPATH=0` forces the generic path, `=log` prints
|
|
`ggml_vk_concat: transpose fast path, transposed ne0=N` to stderr for every fast-path dispatch. It is
|
|
read once (function-local static). Not set in production.
|
|
|
|
**Invariants that must hold.**
|
|
- The fast path fires at prefill and never at decode on a delta-net model.
|
|
- Output is identical to the generic path (it is a pure copy/transpose).
|
|
- The `types_ok` gate keeps quantized types off the path: the dispatches are raw f32/f16 element copies.
|
|
|
|
**Failure modes and incidents.**
|
|
- The decode regression above shipped in the deployed fork before the gate existed. It looked
|
|
tg-neutral on the old upstream only because tg32's +-14% spread hid a 17% effect (section 6).
|
|
- If upstream changes `build_conv_state` (for example a `ggml_cont` on the transpose), the fast path
|
|
silently stops firing. Nothing crashes; prefill just loses about 14%. Only the concat check or a
|
|
perf A/B notices.
|
|
|
|
**Rebase hazards.**
|
|
- Pipeline renames: upstream `733f57a99` renamed `pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to
|
|
`pipeline_cpy_32`/`pipeline_cpy_16`. The pick applied cleanly and did not compile.
|
|
- Upstream `f172be756` made `ggml_vk_concat` non-static (declared in `ggml-vulkan-common.h`) and
|
|
split helpers into headers; the fast-path helper stays static.
|
|
- Shape changes in `build_conv_state` / `qkv_mixed` (`[ch, n_seq_tokens, n_seqs]` reshaped from a
|
|
contiguous `build_lora_mm` output), changes to `copy_transpose.comp` or `copy.comp` push constants,
|
|
or to `vk_op_unary_push_constants` / `get_misalign_bytes`.
|
|
- Upstream concat now handles quantized types; keep `types_ok`.
|
|
- The stock `test_concat` cases in `tests/test-backend-ops.cpp` never produce a transposed source and
|
|
always have `ne[2]`/`ne[3]` > 1, so `test-backend-ops -o CONCAT` does NOT exercise this path.
|
|
|
|
**Validation.** `fork-validate.sh --concat` (needs a build with the knob, i.e. this branch or
|
|
`--concat-build`): counts fast-path dispatches in a prefill run (must be > 0, all with ne0 >= 32)
|
|
and a decode run (must be 0 while CONCAT nodes did execute), then ties the unmodified candidate to
|
|
the fast path by CONCAT timing (candidate about equal to knob-on and at least 2x faster than
|
|
knob-off). Last run: 72 dispatches at prefill (all ne0=512), 0 over 594 CONCAT nodes at decode.
|
|
Also `--perf`.
|
|
|
|
### F4. On-demand device residency (library side)
|
|
|
|
**What and why.** Several always-loaded models must share 8 GB. Co-residence is impossible (the 35B
|
|
at `--n-cpu-moe 40` uses most of the card; the 4B task model needs about 2.5-3.3 GB). Restarting a
|
|
process to free VRAM throws away the RAM-only prompt cache: on Laguna a repeat turn went from a
|
|
21,960 ms re-prefill to 225 ms once the process stayed alive across a task-model call. So a model
|
|
must be able to drop everything it holds in VRAM and get it back without a reload.
|
|
|
|
Measured footprints:
|
|
- 4B: warm 3327 MB, weights released 724 MB, restored and correct.
|
|
- 4B at 32k ctx: cold weights-only 1750 MB (KV stays), cold with KV evicted 726 MB.
|
|
- gemma-26B: warm 7147 MB, cold with weights+KV but scheduler kept 718 MB, cold with the scheduler
|
|
freed too 51 MB.
|
|
- A cold 4B before the encoder work held 644 MB that turned out to be its mmproj (F5).
|
|
- Re-warm cost: about 0.6 s per cold-to-warm switch weights-only (tiny prompt: 1.98 s cold vs 1.40 s
|
|
warm); 2-3 s with KV eviction (it adds the KV D2H/H2D and a scheduler reserve).
|
|
|
|
**Where.**
|
|
- `src/llama-model.cpp`/`.h`: `llama_model::release_device_weights(keep_shadow)`,
|
|
`restore_device_weights()`, `weights_resident()`; per-buffer metadata
|
|
`llama_model::impl::weight_release` (`weight_release_slot`), filled in `load_tensors` and
|
|
`load_tensors_hadamard`.
|
|
- `src/llama-memory.h`: `llama_memory_i::release_device_buffers(keep_shadow)` /
|
|
`restore_device_buffers()` (default no-op). Implemented in `src/llama-kv-cache.cpp`
|
|
(`llama_kv_cache`) and `src/llama-memory-recurrent.cpp`; `llama_memory_hybrid` and
|
|
`llama_kv_cache_iswa` delegate to their children.
|
|
- `src/llama-moe-cache.cpp`/`.h`: `llama_moe_cache::release_device()`, `restore_device()`,
|
|
`resident()`, `impl::invalidate()`.
|
|
- `ggml/include/ggml-backend.h`, `ggml/src/ggml-backend-impl.h`, `ggml/src/ggml-backend.cpp`:
|
|
optional `ggml_backend_i::free_scratch` and `ggml_backend_free_scratch()`; Vulkan implementation
|
|
`ggml_backend_vk_free_scratch` in `ggml-vulkan.cpp`.
|
|
- `src/llama-context.cpp`/`.h`: `llama_context::release_device(evict_kv)`, `restore_device()`,
|
|
`discard_device()`, `device_resident()`, members `kv_device_evicted`, `device_discarded`; the
|
|
auto-restore at the top of `llama_context::decode`.
|
|
- `include/llama.h`: `llama_context_release_device(ctx, evict_kv)`, `llama_context_restore_device`,
|
|
`llama_context_discard_device`, `llama_context_weights_resident`.
|
|
- `tools/mtmd/clip.cpp`/`.h`, `tools/mtmd/mtmd.cpp`/`.h`: `clip_release_device`,
|
|
`clip_restore_device`, `clip_weights_resident`, `mtmd_release_device`, `mtmd_restore_device`.
|
|
- Commits 481be8b28, 17165fe2a, bf2c1a699, 9857a45f0, 9b555f1aa, ad27d4921, 0a0f3ab6d, 7ac2014a8,
|
|
6c480f7e7 (discard).
|
|
|
|
**How it works.** Tensor metadata (`ggml_context` structs) is durable; device buffers are ephemeral.
|
|
|
|
- Weights: on release, for each releasable buffer, capture a host shadow once (weights are
|
|
read-only), free the buffer, null every tensor's `buffer`/`data`. On restore,
|
|
`ggml_backend_alloc_ctx_tensors_from_buft` reallocates an identical layout (re-pointing the
|
|
tensors), the shadow is uploaded, usage is set to WEIGHTS. Only single-buffer, alloc-path, non-host
|
|
buffers are releasable: the host-pointer path (mmap or lazy-mapped), `no_alloc` dummy buffers and
|
|
host (`Vulkan_Host`, i.e. pinned RAM) buffers stay as they are.
|
|
- KV and recurrent state: same pattern, but the shadow is re-captured on every release (the data is
|
|
read-write) and dropped after restore.
|
|
- Scheduler: with `evict_kv`, `sched.reset()` frees the worst-case compute buffer and
|
|
`sched_need_reserve = true`; restore calls `sched_reserve()` (which goes through `sched_new()`,
|
|
re-registering the expert-copy callback and upstream's realloc callback).
|
|
- Backend scratch: with `evict_kv`, `ggml_backend_free_scratch()` on each backend. On Vulkan this
|
|
discards the pending compute context, synchronizes, destroys `prealloc_x`, `prealloc_y`,
|
|
`prealloc_split_k`, `prealloc_add_rms_partials`, `sync_staging`, and zeroes their sizes and the
|
|
`prealloc_y_last_*` tracking. They are reallocated lazily on the next compute. Other backends leave
|
|
the hook null (no-op).
|
|
- MoE cache (`--moe-cache-mib`, upstream #29887): released in both modes. The banks are freed and the
|
|
cache is emptied, not shadowed: every SLRU is rebuilt fresh, its `slot_map[il]` pointers are
|
|
re-pointed at the host slot-map tensors, and the host slot maps are cleared to 0xff (every slot
|
|
-1). On restore the banks are reallocated (which also re-inits their views), cleared to 0 and
|
|
marked WEIGHTS; the cache refills from the host experts.
|
|
- Multimodal encoder: `clip_release_device` / `clip_restore_device` use the weights pattern (shadow
|
|
once); no-op for a CPU-backed encoder. `mtmd_*` wrap the vision and audio contexts.
|
|
- `llama_context::release_device(evict_kv)`: synchronize, release weights, release MoE cache, and if
|
|
`evict_kv`: release memory buffers, free sched, free scratch.
|
|
`restore_device()`: weights, MoE cache, memory buffers, `sched_reserve()` if needed, then
|
|
`ggml_backend_sched_synchronize` so uploads finish before any compute.
|
|
`discard_device()`: like `release_device(true)` with `keep_shadow = false` everywhere (no host
|
|
copies), sets `device_discarded`; used only before exit.
|
|
- `decode()` calls `restore_device()` if `!device_resident()`, as a safety net. The server restores
|
|
explicitly on wake (F6), because some KV access happens before decode.
|
|
|
|
**Invariants that must hold.**
|
|
- Shadows are stored compactly in stable tensor-iteration order, skipping view tensors
|
|
(`view_src != NULL`), and restored with the same iteration. Never index a shadow by
|
|
`t->data - base`.
|
|
- `weight_release` stays parallel to `pimpl->ctxs_bufs`: every push to `ctxs_bufs` needs a matching
|
|
push to `weight_release` (non-releasable slots use `emplace_back()`).
|
|
- The caller synchronizes before any release; `release_device` does this itself.
|
|
- Restore order: weights, MoE cache, KV/recurrent, scheduler reserve, scheduler synchronize. The MoE
|
|
cache banks must be back before the scheduler is rebuilt.
|
|
- After a MoE-cache release no slot map may point at an expert: banks come back zeroed, so a stale
|
|
slot map silently reads zero weights.
|
|
- Every iterator over `ctxs_bufs` (model, KV cache, recurrent) must tolerate a null buffer, and every
|
|
use of `sched` must tolerate null while cold with `evict_kv` (see F8).
|
|
- `device_resident()` covers weights and MoE cache; it gates the early returns, the decode
|
|
auto-restore and `llama_context_weights_resident()`.
|
|
- `restore_device()` after `discard_device()` aborts (`GGML_ASSERT(!device_discarded)`).
|
|
- Greedy output is byte-identical across any number of release/restore cycles, and the prompt cache
|
|
survives (`timings.prompt_n` drops on a repeated prompt after a cycle).
|
|
|
|
**Failure modes and incidents.**
|
|
- The first shadow implementation indexed by `t->data - base` and produced garbage after restore
|
|
(layout and padding differ). Fixed by the compact iteration-order shadow.
|
|
- Restoring KV lazily at decode crashed iSWA models (gemma) with
|
|
`GGML_ASSERT(buf != NULL && "tensor buffer not set")`: `update_slots()` creates an SWA checkpoint
|
|
that reads the KV via `ggml_backend_tensor_get` before decode. Fixed by restoring on wake (2feb6aeb2).
|
|
- The recurrent state was left resident (default no-op), 201 MB on the 4B, until 9857a45f0.
|
|
- A model with weights on the CPU device (`-dev none` / `-ngl 0`) segfaulted (exit 139) in
|
|
`release_device_weights()` the first time the arbiter put it to sleep: CPU_REPACK buffers are not
|
|
host buffers, so `load_tensors` marked them releasable, and the shadow capture called their NULL
|
|
`get_tensor`. It also crashes production `2fc9f9c35`, but no production arbiter model has CPU-device
|
|
weights. Fixed in f24c3582c: only buffers on a non-CPU device are releasable. Found by the
|
|
`--diskcache` check (F11), whose arbiter cases run CPU only.
|
|
- The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed
|
|
resident in a cold model until bf2c1a699 / ad27d4921.
|
|
- The MoE cache was not released at all before 7ac2014a8. Reallocating its banks without
|
|
invalidating would have produced silent wrong weights.
|
|
- In the 20261024 rebase `moe_cache_slru::init()` turned out to reset only the probation segment
|
|
(`head[1]`, `tail[1]` and `n_protected` survive), so re-running it on a used cache corrupts the
|
|
lists. `invalidate()` now assigns a fresh `moe_cache_slru()` before `init()`.
|
|
|
|
**Rebase hazards.**
|
|
- New push sites into `ctxs_bufs` (upstream's `load_tensors_hadamard()` added two in the 20261024
|
|
rebase; without matching `weight_release` entries `release_device_weights()` reads past the end).
|
|
- Changes to buffer creation in `load_tensors` (the `host_ptr_path` condition, `ctx_bufs`/`buf_ptrs`
|
|
bookkeeping; upstream replaced `ctx_buf_maps.emplace_back(ctx, buf_map)` with
|
|
`ctx_bufs.emplace_back(ctx, std::move(buf_ptrs))` in the 20261024 rebase).
|
|
- The `llama_memory_i` interface. New memory types that do not override `release_device_buffers`
|
|
keep their KV resident under `evict_kv`; at the time of writing that is `llama_kv_cache_msa`,
|
|
`_dsa`, `_dsa_iswa`, `_dsv4` and `llama_memory_hybrid_iswa`.
|
|
- MoE-cache internals (`groups`, `slru`, `slot_map`, `buf_host`, `devices`). Upstream `33d6135cb`
|
|
changed LRU to SLRU (`g.lru` -> `g.slru`); the pick applied cleanly and did not compile.
|
|
- `llama_context` members next to ours (upstream `n_reallocs_same/diff`, `copy_experts_info`),
|
|
`sched_reserve()` semantics, and the `decode()` signature (now `decode(const llama_batch_ext &)`).
|
|
- `ggml_backend_i` layout (`free_scratch` must stay the last member after `graph_optimize`; other
|
|
backends rely on zero-init) and the Vulkan context's prealloc field set (compare with
|
|
`ggml_vk_cleanup`).
|
|
|
|
**Validation.** `fork-validate.sh --arbiter`: `b-greedy-identical` (15 cycles each side equal to a
|
|
no-handoff reference), `b-kv-survives` (prompt_n drops after evict/restore, output identical),
|
|
`c-image-after-restore`. `--moecache MODEL` (needs a cache large enough to hit; 256 MiB on gemma-26B
|
|
had 0 hits and proves nothing, 2048 MiB had 6651 hits). `--arbiter` `v-vram-release` /
|
|
`v-vram-restore` check that VRAM really drops: a baseline of `mem_info_vram_used` before any scratch
|
|
server, then server A warm and alone, forced cold by a fake waiter on want.lock: the card-wide delta
|
|
must drop by >= 70% of what A added and come back after the next request. Production's own share is
|
|
re-read from the DRM fdinfo of every process the script did not start; if it moved by > 64 MiB the
|
|
check SKIPs. First run (2026-10-25, 0.8B with mmproj, ctx 8192, `LLAMA_SLEEP_EVICT_KV=1`): warm
|
|
+546 MiB, cold +4 MiB, restored +546 MiB.
|
|
|
|
### F5. On-demand mmproj (multimodal encoder)
|
|
|
|
**What and why.** A multimodal model's vision encoder sat in VRAM on every text request. With
|
|
`LLAMA_MMPROJ_ONDEMAND=1` the encoder lives in RAM and enters VRAM only for an encode. Measured on the
|
|
35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered
|
|
correctly, text after the image correct.
|
|
|
|
**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream `9871df591` split the
|
|
encode out of `process_mtmd_chunk`; `encode_mtmd_chunk` is called from `process_mtmd_chunk` and
|
|
directly from the batched-embedding path in the slot update, so the restore covers both), the
|
|
`mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`, `vram_go_cold`,
|
|
`vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
|
|
|
|
**How it works.** After load the encoder is released (shadow captured). Before
|
|
`mtmd_batch_encode`, `mtmd_restore_device(mctx)`; if that fails (out of VRAM) and on-demand is set, it
|
|
releases the LLM backbone weights only (`llama_context_release_device(ctx_tgt, false)`: read-only
|
|
shadow, so a cheap free with no D2H, KV and prompt cache untouched) and retries. After the encode it
|
|
releases the encoder FIRST, then restores the weights, so the peak stays within VRAM.
|
|
|
|
**Invariants that must hold.**
|
|
- Free the encoder before re-uploading the weights.
|
|
- The donor is the weights, never the KV (the KV is read-write and would need a D2H every time).
|
|
- `vram_ensure_warm()` does not restore the encoder; only the encode path does.
|
|
|
|
**Failure modes and incidents.** Before this, a cold text model with an auto-downloaded mmproj
|
|
(`-hf` pulls one when the repo ships it) held 644 MB that no release path touched. The text-only 4B
|
|
task model now runs with `--no-mmproj`, which is the right fix for any model that never sees images.
|
|
|
|
**Rebase hazards.** Renames or restructuring of `encode_mtmd_chunk` and the mtmd init block in
|
|
`load_model`; `clip_ctx` fields (`buf`, `ctx_data`, `sched`, `no_alloc`).
|
|
|
|
**Validation.** `fork-validate.sh --arbiter` case `c-image-after-restore` (server A runs with
|
|
`--mmproj` and `LLAMA_MMPROJ_ONDEMAND=1`; a 128x128 red PPM must be answered "red" after A was cold,
|
|
and text right after must equal the reference). Post-deploy: one image request to gemma-26B.
|
|
|
|
### F6. Cross-process VRAM arbiter (server side)
|
|
|
|
**What and why.** Several llama-server processes under llama-swap share one GPU: exactly one is warm
|
|
(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production:
|
|
60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between
|
|
the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last
|
|
`fork-validate.sh` run before the 20261024 deploy: 30 handoffs, median 680 ms, max 1231 ms (the
|
|
2026-10-25 run of the same build read 1050 ms, max 1562 ms: handoff latency moves with node load,
|
|
which is why it is now A/B'd against the reference in one session).
|
|
|
|
**Where.** `tools/server/server-context.cpp`: `vram_arena_open`, `vram_acquire_for_load`,
|
|
`vram_share_init`, `vram_should_yield`, `vram_want_acquire`, `vram_want_release`, `vram_take_token`,
|
|
`vram_ensure_warm`, `vram_go_cold`, `vram_share_exit`, `vram_share_shutdown`, the VRAM-only branch of
|
|
`handle_sleeping_state`, the `vram_ensure_warm()` call before decode in the slot update path, and
|
|
`server_res_generator` (`bypass_sleep`). `tools/server/server-queue.cpp`/`.h`: `on_should_yield`,
|
|
`start_loop` (`should_sleep`, idle tick, post-wake grace), `wait_until_no_sleep`, `is_stopping`.
|
|
Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
|
|
|
|
**How it works.**
|
|
- Enabled by `LLAMA_SLEEP_VRAM_ONLY` (any value). The arena directory is `LLAMA_VRAM_ARENA` (default
|
|
`/dev/shm/llama-vram`) with two files.
|
|
- `token.lock`: `flock(LOCK_EX)` is the baton. Rule: a process has device buffers iff it holds it.
|
|
- `want.lock`: a waiter holds it `LOCK_SH` while waiting and drops it once it owns the token. The
|
|
holder probes it with `flock(LOCK_EX | LOCK_NB)` on a SECOND fd: failure means somebody is waiting.
|
|
The probe needs its own open file description, because flock on the waiter's own fd would convert
|
|
that lock instead of conflicting with it. The kernel drops a dead waiter's lock, so nothing goes stale.
|
|
- Load: `load_model()` calls `vram_acquire_for_load()` before `common_init_from_params()`, so the
|
|
current holder releases before this model uploads anything. The loaded model stays warm holding the
|
|
token and serves its first request without a re-warm.
|
|
- Yield: `vram_share_init()` registers `vram_should_yield` with `queue_tasks.on_should_yield`.
|
|
`start_loop()`'s `should_sleep()` calls it on the loop thread, under `mutex_tasks`, only when the
|
|
task queue is empty and `update_slots` has returned. While the callback is set the idle wait ticks
|
|
every 10 ms instead of 1 s. After a wake the server holds the GPU for at least 100 ms
|
|
(`yield_grace_ms`). Sleep runs the sleeping callbacks; in VRAM-only mode `handle_sleeping_state(true)`
|
|
calls `vram_go_cold()` (release device, also the encoder; then `flock(LOCK_UN)`) instead of
|
|
upstream's full unload.
|
|
- Wake: a request calls `wait_until_no_sleep()`. `handle_sleeping_state(false)` calls
|
|
`vram_ensure_warm()`: take the token (register on want.lock, then poll `token.lock` with
|
|
`LOCK_EX | LOCK_NB` every 2 ms, aborting when `queue_tasks.is_stopping()`), restore ctx_tgt and
|
|
ctx_dft, and only then set `vram_cold = false`. The decode path calls `vram_ensure_warm()` again
|
|
(no-op when warm) and throws "server is shutting down" if it returns false.
|
|
- `server_res_generator` no longer bypasses the wake path when `--sleep-idle-seconds` is unset if
|
|
`LLAMA_SLEEP_VRAM_ONLY` is set, because the arbiter can put a server to sleep without idle-sleep.
|
|
- Queue fixes: `wait_until_no_sleep()` loops `while (sleeping && !stopping)` and re-asserts
|
|
`req_stop_sleeping` each time (with `notify_all`); the sleep wait also wakes on a non-empty task
|
|
queue; the idle `wait_for` only leaves the loop on a real task or termination.
|
|
|
|
**Invariants that must hold.**
|
|
- Resident iff holding `token.lock`; `vram_cold == false` means both. `vram_ensure_warm()` sets
|
|
`vram_cold = false` only AFTER the restore. What that order protects today, reasoned from the code:
|
|
the only thing that gives the token away is `vram_should_yield()`, called from `should_sleep()` on
|
|
the loop thread. `vram_ensure_warm()` also runs only on the loop thread, synchronously: from the
|
|
wake callback (`handle_sleeping_state(false)`, called inside `start_loop()` with `mutex_tasks`
|
|
held) and before decode in the slot update path (outside `yield_to_queue()`). So `should_sleep()`
|
|
cannot run between the token acquire and the end of the restore, whichever order the two lines
|
|
are in. The measured failure (attempt 1 below: handoffs of 11m20s and 3m57s) happened under the
|
|
retired doorbell design, where a separate warden thread read the cold flag to decide whether to
|
|
honour or discard rings, so an early `false` let it accept stale rings during the restore. Under
|
|
the inline want-lock probe the order is defensive: it keeps the flag's meaning exact for any future
|
|
reader off the loop thread, and keeps a release out of a half-restored model if the restore or the
|
|
probe ever moves (for example into `yield_to_queue()`). Either way, do not change it without an A/B
|
|
of ordinary handoff latency (`fork-validate.sh --arbiter`, `a-handoff-latency`).
|
|
- `vram_acquire_for_load()` sets `vram_cold = false` as soon as it holds the token, before the upload.
|
|
Nothing reads the flag during the load today (the loop is not running yet and `on_should_yield` is
|
|
only registered in `vram_share_init()`, which sets it again), so the early set is redundant now.
|
|
It records "holding the token" at the moment it becomes true, because the opposite error is the
|
|
dangerous one: a model that is warm and holds the token while `vram_cold` says cold never yields
|
|
(`vram_should_yield()` returns false for a cold model), which strands every waiter, and its next
|
|
wake would try to restore weights that were never released.
|
|
- Release only happens from the idle branch of `start_loop()`, on the loop thread, never inside
|
|
`yield_to_queue()` or while a slot is generating: a holder finishes its current generation before
|
|
it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both).
|
|
- The yield decision reads live kernel state at the point of use. Do not route it through a flag set
|
|
by another thread (see attempts 3-4 below).
|
|
- Every exit from `vram_take_token()` drops want.lock, or the holder yields forever.
|
|
- The probe fd is separate from the waiter fd.
|
|
- `flock` calls retry on EINTR: the server's signal handlers are installed without SA_RESTART.
|
|
- Restore on wake, before `update_slots()`, so KV eviction is safe for SWA checkpoints.
|
|
- Every model that can be warm on the GPU at the same time must use the same arena. A non-arbiter
|
|
process (a benchmark, a manually started server) can OOM the warm model. In production the env is
|
|
set for all of llama-swap, so all its children participate.
|
|
|
|
**Failure modes and incidents.**
|
|
- Wake-path hang (958257cd6): with no `--sleep-idle-seconds`, `bypass_sleep` skipped
|
|
`wait_until_no_sleep()`, so a server slept by the arbiter never woke and requests hung.
|
|
- Load-time OOM (b9df854de): weights were uploaded before the arbiter was active; loading gemma while
|
|
the 4B held VRAM could exceed 8 GB. After the fix the peak was 7771 MB.
|
|
- The doorbell design (replaced by 878d1338f) was edge-triggered: a waiter wrote
|
|
`<arena>/doorbell/<pid>` and the holder's warden thread turned the inotify event into a yield flag.
|
|
Rings were lost in three windows: while the holder restored (it advertised cold and discarded
|
|
foreign rings), while the holder loaded (the inotify watch did not exist yet, and the kernel does
|
|
not queue events for a missing watch), and when a cold waiter consumed a third process's ring.
|
|
Every loss was permanent: `flock(LOCK_EX)` had no timeout and production had no idle-sleep. The
|
|
blocked flock held `mutex_tasks`, so the whole server stopped answering while `/health` still
|
|
returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by
|
|
the second day).
|
|
- Failed fix attempt 1 (doorbell design): moving `vram_cold = false` before the restore made the
|
|
holder's warden honour stale rings and hand the token back 2-5 s after taking it without serving.
|
|
Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. See the first invariant
|
|
for what this means under the current design.
|
|
- Failed fix attempt 2: a re-ring loop (`LOCK_NB` + re-ring every ~1 s). Queued rings from an already
|
|
satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s
|
|
baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become
|
|
level-triggered.
|
|
- Failed attempts 3-4: want.lock was sound (mutual exclusion, kill -9 safety, load-window coverage),
|
|
but the warden thread still delivered its decision as a flag. `start_loop()` holds `mutex_tasks`
|
|
from `should_sleep()` through the callbacks to the wait, so the warden's `request_yield()` was always
|
|
admitted after the flag had been consumed, latching a release for the next wake (a server woke and
|
|
re-slept 138 ms later without serving). It also exposed the upstream lost wakeup in
|
|
`wait_until_no_sleep()`. Result: 0.494 s median, 1/60 stranded. The fix deleted the warden and
|
|
`request_yield`/`yield_requested` and probes inline in `should_sleep()`.
|
|
- Measured deviations that are required: the 10 ms idle tick (about 0.2% of one core per warm
|
|
server); a waiter that polls rather than retrying on a coarse grid (a 200 ms retry grid cost 0.503 s
|
|
against 0.336 s, because the holder only releases after the waiter asks); the 100 ms post-wake
|
|
grace (without it a permanent waiter caused a livelock: one request took 23.15 s over 138
|
|
sleep/wake cycles).
|
|
- Process lesson from attempts 1-2: validate an ordinary handoff latency A/B, not only the
|
|
pathological case. Two production deploys were burned before that rule existed.
|
|
- Lost diagnostic: the old periodic "still waiting for the GPU token after N s" warning is gone; a
|
|
stuck token now shows one `VRAM arbiter: waiting for the GPU token` line and then silence. Check
|
|
`/proc/locks` (see the runbook).
|
|
|
|
**Rebase hazards.** This is the most fragile area, because it sits inside upstream's server loop.
|
|
- `server_queue::start_loop`, `wait_until_no_sleep`, `process_new_tasks`, `yield_to_queue` and the
|
|
sleeping-callback vector. Upstream refactored sleep (`947fd9bb2`) and made decode run inside
|
|
`yield_to_queue` (`22b8e310b`); re-check after any such change that a release still cannot run
|
|
under a live graph.
|
|
- `handle_sleeping_state`, `load_model` ordering, the slot update path where decode happens,
|
|
`server_res_generator`'s `bypass_sleep`, renames in `server-context.cpp`.
|
|
- Anything that touches the KV in `update_slots()` before decode.
|
|
|
|
**Validation.** `fork-validate.sh --arbiter` with two real servers on a scratch arena:
|
|
`a-handoffs` (30 alternating, 0 stranded, 0 non-200, the server that answered holds `token.lock`),
|
|
`a-handoff-latency` (fresh server pairs of `--build` and `--ref` in the same session, blocks
|
|
ref, cand, cand, ref x 16 handoffs; FAIL if the candidate median exceeds the reference median by more
|
|
than max(25%, 200 ms); first run 2026-10-25, reb1024 against itself beside production: 1050.5 vs
|
|
1082.5 ms), `v-vram-*` (F4), `b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams:
|
|
want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in
|
|
F7 and a log scan for asserts and Vulkan errors. In production, `/proc/locks` must show the warm
|
|
model holding WRITE on `token.lock`.
|
|
|
|
### F7. Server shutdown path
|
|
|
|
**What and why.** llama-swap evicts models with SIGTERM, so shutdown happens all the time, often
|
|
while the arbiter is mid-release or mid-wait. Two bugs made shutdown keep the GPU token and stall
|
|
every other model.
|
|
|
|
**Where.** `tools/server/server.cpp`: `signal_handler`, `shutdown_pipe`, `struct shutdown_thread`,
|
|
`sig_thread` in `llama_server`. `tools/server/server-context.cpp`: `server_context::start_loop` (calls
|
|
`vram_share_exit()` after the loop returns), `server_context::terminate` (terminates the result queue
|
|
first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/server/server-queue.cpp`:
|
|
`server_queue::terminate` (sets atomic `stopping` before taking the mutex), `process_new_tasks`,
|
|
`server_response::recv_with_timeout`, `server_response::terminate`,
|
|
`server_response_reader::next`. Commits 11044b35e, 6c480f7e7.
|
|
|
|
**How it works.**
|
|
- The signal handler only writes the signal number to a self-pipe (async-signal-safe); a dedicated
|
|
thread reads it and runs `shutdown_handler`. The pipe fds are `FD_CLOEXEC`. If `pipe()` fails the old
|
|
direct call is used. The thread object is declared after the server objects, so it is joined (a 0
|
|
byte unblocks it) before they are destroyed. A second signal still calls `exit(1)`.
|
|
- `terminate()` sets `stopping` (atomic) before locking `mutex_tasks`, then stops the result queue so
|
|
HTTP handlers waiting for results get "server is shutting down" (HTTP 503, or an SSE error event);
|
|
`recv_with_timeout` checks `!running` before waiting and returns null.
|
|
- A token wait polls with `LOCK_NB` every 2 ms and aborts on `is_stopping()`; the aborted wake stays
|
|
cold, skips the restore, and the loop exits before `update_slots`.
|
|
- After `start_loop()` returns, `vram_share_exit()` frees all device buffers with
|
|
`llama_context_discard_device()` (no host shadows), releases the encoder, and only then unlocks
|
|
`token.lock`.
|
|
- What llama-swap (v239 on the pod) does when a child exits on its own (crash, OOM kill, external
|
|
signal), verified 2026-10-25 with a scratch llama-swap instance and a CPU-only child, for both
|
|
`kill -9` and an external SIGTERM: it logs
|
|
`[WARN] group: running <model> exited: [<model>] upstream exited unexpectedly`, does not restart it
|
|
eagerly (no new process in the 2 s after the kill), and starts a new process on the next request
|
|
for that model (new PID, health check, HTTP 200). The production log since the last pod restart has
|
|
no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers
|
|
the restart pays a full model load (about 26 s for the 35B with `-lm none`; the first 35B request
|
|
after the 2026-10-25 deploy took 34 s including load and answer), and under the arbiter
|
|
the new process takes the token through `vram_acquire_for_load()` like any load. A process that
|
|
wedges instead of exiting (the incidents below) is never restarted: `/health` keeps answering.
|
|
|
|
**Invariants that must hold.**
|
|
- Nothing that takes `mutex_tasks` runs in signal context.
|
|
- The token is released only after this process's VRAM is free (the next model may start uploading
|
|
the moment the lock drops).
|
|
- A shutting-down server never restores weights.
|
|
- SIGTERM in any state exits 0 within about 2 s and frees the token. With the disk prompt cache (F11)
|
|
the exit also writes each slot's state: about 1 s more per GB of state, after the token is free.
|
|
|
|
**Failure modes and incidents.**
|
|
- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside
|
|
a sleeping callback (under `mutex_tasks`, during a release); the handler's `terminate()` locked
|
|
`mutex_tasks` again on the same thread. The process wedged holding `token.lock`, every other model
|
|
queued behind it, and only SIGKILL worked. Reproduced 6/6 off production; 8/8 clean with the fix.
|
|
The first suspicion (the multimodal path) was wrong. gdb on the wedged process showed the main
|
|
thread in `futex(mutex_tasks)` with the `yield_to_queue` worker idle.
|
|
- Token kept on shutdown: with a request in flight the HTTP handler waited forever for results, main
|
|
blocked joining it, and the token was released only in the destructor after that join; with a
|
|
token wait in progress the blocking flock held `mutex_tasks`; and the old destructor unlocked the
|
|
token BEFORE `destroy()`, so the next model could upload into still-occupied VRAM.
|
|
- After the fix: off production SIGTERM mid-generation or mid-wait exits in about 0.2 s with exit 0
|
|
and frees the token in 5-70 ms; in production about 0.7-1 s.
|
|
|
|
**Rebase hazards.** `server.cpp`'s signal setup and `llama_server` object lifetimes;
|
|
`server_response` and `server_response_reader` (upstream `4e201388c` switched `recv_with_timeout` to a
|
|
fixed deadline: keep the order "check `!running`, return null, then `wait_until(lock, deadline)`");
|
|
`server_context::start_loop`/`terminate`. Router mode (`server-models.cpp`,
|
|
`server_models_sse_client::next`) only checks `should_stop()` and can spin at shutdown now that
|
|
`recv_with_timeout` returns null at once; llama-swap does not use router mode.
|
|
|
|
**Validation.** `fork-validate.sh --arbiter`: `d-term-in-release` (gdb breakpoint on
|
|
`llama_context_release_device`, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and
|
|
the token), `e-term-waiting` (exit 0 within 1.5 s, never took the token), `f-term-streaming`
|
|
(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run
|
|
(2026-10-25, 2fc9f9c35, `d` by log polling): 750 ms, 225 ms, 126 ms (token free in 23 ms),
|
|
108/159/159 ms. Post-deploy: a forced llama-swap eviction
|
|
must log `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token` and exit by itself.
|
|
|
|
### F8. Cold-teardown guards
|
|
|
|
**What and why.** A cold model has null buffers in its `ctxs_bufs` and, with `evict_kv`, a null
|
|
scheduler. Upstream code never had to consider either state.
|
|
|
|
**Where.** `src/llama-context.cpp`: `llama_context::~llama_context` (`sched &&` in the
|
|
compute-buffer size check), `llama_context::memory_breakdown` (`if (sched)` around the compute part;
|
|
upstream's `if (moe_cache)` breakdown block stays outside it). Null-buffer skips in
|
|
`llama_model::memory_breakdown`, `llama_kv_cache::{clear, memory_breakdown, total_size}`,
|
|
`llama_memory_recurrent::{clear, memory_breakdown, total_size}`. Commits 8d80f0673, ad27d4921, 53f907771.
|
|
|
|
**How it works.** Guards at the call sites. The asserts in ggml (`GGML_ASSERT(buffer)`,
|
|
`GGML_ASSERT(sched)`) are correct and stay; every other caller reserves the scheduler first.
|
|
|
|
**Invariants that must hold.** A server shut down while cold exits 0. Any new upstream code that
|
|
iterates `ctxs_bufs` or calls into `sched` on a path reachable while cold (shutdown, memory
|
|
breakdown, stats) needs the same guard.
|
|
|
|
**Failure modes and incidents.**
|
|
- `GGML_ASSERT(buffer)` on shutdown of a cold model (seen 7 times in production logs) from
|
|
memory-breakdown/size/clear on released buffers.
|
|
- `GGML_ASSERT(sched)` in `memory_breakdown` for a cold `evict_kv` model being torn down.
|
|
- `GGML_ASSERT(sched)` in the destructor's size check: every cold shutdown exited 134 instead of 0, so
|
|
under llama-swap an ordinary stop looked like a crashed child, left a backtrace in the log, and the
|
|
rest of the destructor never ran. Repro: arbiter env on, one request, force a yield with
|
|
`flock -s <arena>/want.lock -c 'sleep 6'`, then SIGINT while cold.
|
|
|
|
**Rebase hazards.** New iteration over `ctxs_bufs`, new `sched` users in teardown or breakdown
|
|
paths. Watch item: `llama_kv_cache::update()` takes `lctx->get_sched()`, which can be null; it is
|
|
safe today only because its single caller runs after `sched_reserve()`.
|
|
|
|
**Validation.** `fork-validate.sh --arbiter`: `h-shutdown-cold` (exit 0 while the other server holds
|
|
the token) and `log-scan`; `--moecache` `shutdown`.
|
|
|
|
### F9. Benchmark harness and validation script
|
|
|
|
**What and why.** The measurement method on this box had to be rediscovered several times (section
|
|
6). The harness encodes it; the validation script encodes the checks that each past incident
|
|
required, so a rebase can be validated by someone with no context.
|
|
|
|
**Where.** `scripts/rx580-bench/` (`run.sh` workstation wrapper; `bench.sh` pod orchestrator;
|
|
`ppbench.py` corpus prefill through llama-server; `lbsweep.sh` llama-bench sweep; `summarize.py`;
|
|
`README.md` with methodology and reference values). `scripts/fork-validate.sh`. Commits 4c4486f8e,
|
|
54a7f534b, 149a8cb0a, 62d442e7b, 2512290f9, 80e662a2a, ceab1a91f, e5fc474b8.
|
|
|
|
**How it works.**
|
|
- `rx580-bench`: stops llama-swap, waits for an idle GPU, runs fixed slices of `pan-tadeusz.txt`
|
|
(4096/16384/32768 tokens, byte-identical across builds) through `/completion` with
|
|
`cache_prompt: false`, plus a llama-bench sweep (prefill pp2048/pp8192 at depth 0, decode
|
|
tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and
|
|
rounds; restarts llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
|
|
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]
|
|
[--moe-exact MODEL] [--diskcache]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a
|
|
scratch dir under /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets
|
|
inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check
|
|
flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline.
|
|
See the header of the script for what each check proves.
|
|
- Which phase needs what: `--ops --concat` run beside production; `--arbiter` too only if 1536 MiB
|
|
of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs
|
|
in the swapdown window, about 2 minutes); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
|
|
first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share
|
|
the card either). `--diskcache` runs beside production (CPU only); its GPU leg SKIPs below 1536 MiB
|
|
of free VRAM.
|
|
|
|
**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24,
|
|
production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s,
|
|
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. On 2026-10-25
|
|
(medians, `6c480f7e7` / `2fc9f9c35`): pp2048 274.5/273.8, d16384 200.5/200.4, d32768 157.6/157.6,
|
|
tg256@d4096 16.8/16.7. The corpus
|
|
harness and older numbers are in `scripts/rx580-bench/README.md`. A fresh run far outside these
|
|
means something else is wrong (another GPU user, thermal state, a bad build) before it means a
|
|
regression or a win.
|
|
|
|
**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` and `--moe-exact`
|
|
refuse, and any other llama process blocks them). `--perf` gates prefill at -5% per depth on the
|
|
MEDIAN of all repetition samples of each build (never a mean: CPU contention produces single
|
|
catastrophic samples), and peak VRAM at 95% of the card; decode is reported, not gated (CPU-bound and
|
|
noisy here).
|
|
|
|
**Failure modes and incidents.** Upstream #28334 removed `--mmap`/`--no-mmap`/`--mlock` and
|
|
llama-bench's `-mmp`: the server refused to start and llama-bench silently produced no rows until
|
|
54a7f534b switched to `-lm none`. The perf and MoE-cache checks leave tens of GB of model files in page
|
|
cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024
|
|
deploy (see the runbook). That pod restart also emptied `/tmp` (every earlier `fork-validate.*`
|
|
scratch dir with its raw samples is gone) and removed gdb, which is not in the pod image; `--arbiter`
|
|
case `d-term-in-release` then falls back to polling the log.
|
|
|
|
**Rebase hazards.** CLI flag renames in `common/arg.cpp` and `tools/llama-bench`; log line texts the
|
|
script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`,
|
|
`no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`,
|
|
`llama_moe_cache: .* hits =`, `sched_copy_experts: ids readback skipped`, `N/M tests passed`,
|
|
`Backend Vulkan0: OK`, `disk cache: restored`, `disk cache: copied`/`saved`, `prompts of this model`,
|
|
`size limit, removing`, `disk cache: failed to write`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and
|
|
markdown layout; the GGUF keys `<arch>.expert_count` / `<arch>.expert_used_count`; the DRM fdinfo
|
|
keys `drm-client-id` / `drm-memory-vram`.
|
|
|
|
**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test. The first
|
|
candidate arm has now twice shown an unexplained spike of about 1.6 GiB over the other arms
|
|
(2026-10-25: ref 5476, cand 7062, cand 5767, ref 5476 MiB); the cause is open (per-test peaks would
|
|
locate it), and it stays far below the 95% gate. The SIGTERM-in-release breakpoint is at function
|
|
entry, not after the host-shadow capture. The `v-vram-release` SKIP path (production moving during
|
|
the measurement) has not been exercised. `--moe-exact` and the median-gated `--perf` first ran on the
|
|
GPU on 2026-10-25 (runbook section 6).
|
|
|
|
### F10. Page-cache drop after a copying load
|
|
|
|
**What and why.** With `-lm none` the loader read()s the GGUF into its own buffers, but the kernel
|
|
keeps the file in page cache afterwards, charged to the pod's cgroup: about 20 GB after a 35B load.
|
|
The pod is BestEffort on a shared node, and on 2026-10-24 the Talos OOM controller killed it right
|
|
after a model load under exactly that pressure (MemFree < 1 GB, file cache about 20 GB). Once the
|
|
data is copied those pages are pure waste, so the loader advises the kernel to drop them.
|
|
|
|
**Where.** `src/llama-mmap.h`/`.cpp`: `llama_file::drop_cache(offset, len)`.
|
|
`src/llama-model-loader.cpp`, `llama_model_loader::load_all_data`: the per-tensor drop at the end
|
|
of the copy (non-mapping) branch and the whole-file drop in the final cleanup
|
|
(`size_done >= size_data`). `tools/mtmd/clip.cpp`, `clip_model_loader::load_tensors`: whole-file drop
|
|
after the tensor reads. Commit cce0ac1f0.
|
|
|
|
**How it works.**
|
|
- `drop_cache()` is `posix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED)` under `#ifdef __linux__`
|
|
(no-op elsewhere); `len` 0 means to the end of the file. Errors are logged at debug level only.
|
|
- `load_all_data()` drops each tensor's range right after it is read (all three copy paths: straight
|
|
into a host buffer, the async pinned-upload chunks, the staging buffer), so the cache never grows
|
|
past about one tensor during the load. The final cleanup drops every file whole, which catches the
|
|
GGUF metadata and the partial pages at tensor edges that a range drop leaves.
|
|
- Both run only when `mappings.empty()`, i.e. the loader holds no mapping of any of its files: not
|
|
`mmap`, `mmap+mlock` or `auto`, and no lazy tensors (`-lzm`; with lazy tensors `init_mappings()`
|
|
maps every file even under `-lm none`). Split GGUFs are covered (every entry of `files`).
|
|
- clip.cpp always copies the encoder through an ifstream (no mmap path exists), so it opens the
|
|
file again after the reads and drops it whole, unless `no_alloc`.
|
|
- Choice: per-tensor plus a final drop rather than a post-load drop only, because the per-tensor
|
|
drop also removes the peak during the load (the incident window: anon and file cache grow together)
|
|
for three extra lines. It costs about 0.27 s more than a post-load-only drop on the test model
|
|
below. Direct IO (`-lm dio`, upstream's) was measured as the alternative and is not better (section 4).
|
|
|
|
**Invariants that must hold.**
|
|
- Never drop for a file this loader maps: a mapped page would just be faulted back in (the kernel
|
|
also skips pages that are mapped into a process, so a violation costs refaults, not correctness).
|
|
- Drop only after the bytes are in our own buffers. Nothing in the fork re-reads the GGUF after
|
|
load: the VRAM release/restore (F4) uses host shadows, not the file.
|
|
- The drop is a hint: a failure must not fail the load.
|
|
|
|
**Failure modes and incidents.** None. Deployed 2026-10-25 (`2fc9f9c35`); on a 35B GPU load in
|
|
production the node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB. Costs to know
|
|
about:
|
|
- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a
|
|
5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few
|
|
seconds on a 26 s load.
|
|
- A reload of the same model can no longer hit a warm page cache (a warm-cache reload of the test
|
|
model took 4.6 s against 5.9 s cold). Under llama-swap a reload happens only after an eviction or
|
|
a crash.
|
|
- `TENSOR_DUPLICATED` tensors read the same range twice; the second read now comes from disk (small
|
|
tensors only).
|
|
|
|
**Rebase hazards.** The structure of `load_all_data()` (the `from_mapping` branch and its three copy
|
|
paths, the final cleanup block), `init_mappings()` creating mappings for lazy tensors, new load modes
|
|
in `llama_load_mode` (any new mode that maps must leave `mappings` non-empty), and the tensor-read
|
|
loop in clip.cpp `load_tensors`.
|
|
|
|
**Validation.** Not in `fork-validate.sh`. By hand on the pod with a scratch llama-server on a
|
|
scratch port, CPU only (`-dev none -ngl 0 --no-mmproj-offload`, no VRAM), cold cache (evict the test
|
|
files with python `os.posix_fadvise(..., DONTNEED)` first): page residency of the model and mmproj
|
|
files by `mincore` before and after the load (must be 0 after with `-lm none`, unchanged with
|
|
`-lm mmap`), the pod cgroup's `file`/`anon` in
|
|
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/memory.stat`, host `MemFree`, load time, and a greedy
|
|
request. First run (2026-10-25, gemma-4-E2B UD-Q4_K_XL 2.5 GiB + mmproj BF16 0.9 GiB, production
|
|
6c480f7e7 and the candidate with the drops disabled as controls, 4 rotated rounds):
|
|
|
|
| | drops off (control) | post-load only | per-tensor + final (committed) |
|
|
|---|---|---|---|
|
|
| model / mmproj resident after load | 2431-2444 / 940 MiB | 0 / 0 | 0 / 0 |
|
|
| peak model residency during load | 2.1-2.5 GiB | 2.1-2.4 GiB | 1.2-1.5 GiB (largest tensor) |
|
|
| pod cgroup `file` after load | +3.37 GiB | +0 | +0 |
|
|
| host MemFree after load | 0.6 GB | 2.5 GB | 2.5-2.8 GB |
|
|
| load time (median, production 5.85 s) | 5.87 s | 6.56 s | 6.82 s |
|
|
|
|
Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it counts clean page cache as
|
|
available. Greedy output byte-identical in all arms. `-lm mmap` control: the model file stays fully
|
|
resident and mapped (2498 MiB, `file_mapped` +2.5 GiB) after load and after a request, and only the
|
|
copied mmproj is dropped. `-lzm on` control: no drop (the file is mapped), 1252 MiB of copied
|
|
tensors stay cached. The GPU upload path and a 35B load were first exercised by the 2026-10-25
|
|
deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B
|
|
request took 34 s including load and answer).
|
|
|
|
### F11. Disk prompt cache (save slots on shutdown, restore after a restart)
|
|
|
|
**What and why.** Only one large model fits in RAM. Switching models in OpenWebUI makes llama-swap
|
|
SIGTERM the previous one, and switching back re-prefills the whole conversation, because the RAM
|
|
prompt cache died with the process (F4's reason for the arbiter). A 30k-token chat costs about 150 s
|
|
on the 35B at about 200 t/s. With `--cache-disk-path DIR`, each slot with a prompt is saved to DIR on
|
|
shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the
|
|
file, so only the suffix is processed. Measured on Qwen3.5-0.8B CPU only: a 12k-token turn 2 went
|
|
from a 72 s prefill to 1.2 s, of which the read took 307 ms and the restore 18 ms. Off by default.
|
|
Upstream PR #28092 (`--cache-disk`) was the reference for ideas, not ported (1700 lines). It
|
|
livelocked the kernel on AMD + Vulkan by DMA-ing GPU state into a MAP_SHARED mmap'd file, and it never
|
|
saved the slot in use on shutdown, which is the main case here.
|
|
|
|
**Where.**
|
|
- `tools/server/server-task.h`/`.cpp`: `server_prompt_disk_entry`, `server_prompt_disk_cache`
|
|
(`n_reuse`, `find`, `has`, `load`, `save`, `remove`); file IO helpers `disk_cache_io`,
|
|
`disk_cache_open`.
|
|
- `tools/server/server-context.cpp`: `DISK_CACHE_MIN_TOKENS`, `disk_cache_key`,
|
|
`disk_cache_collect`, `disk_cache_flush`, `disk_cache_load`; the init in `load_model` after the RAM
|
|
cache; the `disk_cache_load` call at the end of `get_available_slot`; `server_context::start_loop`
|
|
(collect, `vram_share_exit`, flush).
|
|
- `common/arg.cpp`, `common/common.h`: `--cache-disk-path` (`cache_disk_path`), `--cache-disk-size`
|
|
(`cache_disk_mib`, default 4096, 0 = no limit).
|
|
- Library (cold save): `llama_context_restore_memory_host()` (`include/llama.h`,
|
|
`llama_context::restore_memory_host`); `llama_memory_i::restore_device_buffers(bool to_host)` in
|
|
`llama-memory.h`, `llama-kv-cache`, `llama-memory-recurrent`, `-hybrid`, `-kv-cache-iswa`; the helper
|
|
`llama_memory_wrap_host()` in `llama-memory.cpp`.
|
|
- Commits dd851fd0c, 2874dcb10, 2c6c517ba (and f24c3582c, F4).
|
|
|
|
**How it works.**
|
|
- Save, after `start_loop()`'s loop returns. First `disk_cache_collect()` copies the state of every
|
|
slot with at least 256 text tokens to host memory (`llama_state_seq_get_data_ext`, flags NONE)
|
|
while the device buffers still exist. Then `vram_share_exit()` frees the VRAM and releases the GPU
|
|
token as before. Then `disk_cache_flush()` writes the files. The token release therefore waits
|
|
only for the device-to-host copy, never for disk IO.
|
|
- A slot is skipped when its KV does not cover its token list (`seq_pos_max + 1 != n_tokens`: the last
|
|
batch was not decoded, a shutdown during a token wait), when it holds media, and when the same
|
|
token list is already on disk.
|
|
- Cold server: with `LLAMA_SLEEP_EVICT_KV` the KV and recurrent state live only in host shadows (F4)
|
|
and their tensors have no buffer. `llama_context_restore_memory_host()` wraps each shadow as a
|
|
CPU_Mapped buffer (`ggml_backend_cpu_buffer_from_ptr`) and points the tensors into it, so there is
|
|
no copy, no allocation and no device use. It sets `device_discarded`. Without evict_kv the KV is still
|
|
in VRAM and is read directly.
|
|
- File, `<dir>/<key hash>-<token hash>.lpcd`: a 48-byte header (magic `LPCD`, version, key hash,
|
|
n_tokens, n_ckpt, payload size, payload hash), then 8-byte padded sections: tokens, checkpoint table,
|
|
state size, state, the state of each checkpoint. The hash is FNV-1a over the 64-bit words of the
|
|
payload. Write: plain fwrite to `.tmp`, fdatasync, `POSIX_FADV_DONTNEED`, rename. Read: plain fread,
|
|
then DONTNEED. There is no mmap anywhere.
|
|
- Index: at startup, from the headers, tokens and checkpoint tables of the files of this key. A bad
|
|
magic, a file size that differs from the header, or an inconsistent table means the file is removed.
|
|
Other keys and other versions are counted and left alone. A `.tmp` older than 1 h is removed.
|
|
- Key: the build commit, the model path + size + mtime, LoRA adapters and control vectors, n_ctx,
|
|
n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, K/V types, flash-attn type, and the RoPE/YaRN
|
|
parameters.
|
|
- Restore: at the end of `get_available_slot()`, after the RAM cache, only while warm, with
|
|
`cache_prompt` on and no media tokens. The file that reuses the most of the new prompt is loaded if
|
|
it reuses at least 256 more tokens than the slot holds. The slot's prompt goes to the RAM cache
|
|
first. Then the slot is cleared and the state is set; a failure clears the slot and removes the file.
|
|
- Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt,
|
|
or, for a recurrent or SWA model, a saved checkpoint at or below the common prefix can be used. A
|
|
plain-KV model can be cut back to any prefix. Exact prefix alone would almost never fire: the Qwen
|
|
template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is not a
|
|
prefix of the next OpenWebUI turn. Checkpoints are exact snapshots, and `update_slots()` rolls back
|
|
to one exactly as it does for the RAM cache; the recurrent state itself is never cut back.
|
|
Restored checkpoints get `id_task = -1`, so one that fails to load means a re-prefill, not an abort.
|
|
- Size limit: before a write, the least recently used `*.lpcd` files of every model in DIR are removed
|
|
until the new file fits. A restore updates the file's mtime. Entries the new prompt extends are
|
|
removed after the write and do not count against the limit. A write also needs its size + 1 GiB
|
|
free on the file system, so the cache never fills the PVC.
|
|
- Disabled with speculative decoding, with a warning: the draft state is not saved.
|
|
|
|
**Invariants that must hold.**
|
|
- Never restore while cold or without the GPU token (`disk_cache_load` returns when `vram_cold`).
|
|
`get_available_slot()` runs on the loop thread after the wake, like the RAM cache's `prompt_save`.
|
|
- Copy the state before `vram_share_exit()` (it discards the device buffers) and write after it (the
|
|
token is not held for disk IO).
|
|
- Never serve a file that fails the size or hash check. Never mmap the file: a DMA into a shared
|
|
mapping is what livelocked upstream's version on AMD.
|
|
- A write is atomic (tmp + rename), and any IO error leaves the server serving and the exit clean.
|
|
- The key must cover everything that changes the state layout or the KV values. Add a new
|
|
state-changing parameter to `disk_cache_key()`.
|
|
- `llama_context_restore_memory_host()` is for exit only; the wrapped buffers point into the shadows,
|
|
which must outlive them (both are members of the same memory object).
|
|
|
|
**Failure modes and incidents.**
|
|
- Production incident (2026-10-25, deploy of cb48bf224 with `--cache-disk-path`, rolled back): the first
|
|
request to the 35B aborted in `disk_cache_load()`. `server_tokens::get_tokens()` has
|
|
`GGML_ASSERT(!has_mtmd)`, and `has_mtmd` is true whenever an mmproj is loaded, even for text-only
|
|
prompts. Production's 35B and gemma-26B always load one (`-hf` pulls it; there is no
|
|
`--no-mmproj`). `disk_cache_collect()` and `save()` had the same call, so the shutdown save would
|
|
have aborted too. Every test had used `--no-mmproj` (`q35.py` and `--diskcache`), which hid it.
|
|
Fixed: the feature takes the token list from `get_text_tokens()`, and a prompt is text-only when
|
|
that list is as long as `size()` (`disk_cache_text`). A text-only prompt is cached with an mmproj
|
|
loaded. A prompt with media is never saved or restored. When the slot itself holds media, its reuse
|
|
estimate is `get_common_prefix()`. Lesson (runbook section 6): test with production's exact command line.
|
|
- The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself
|
|
(CPU_REPACK, F4), not in the save.
|
|
- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full
|
|
KV allocation at exit (about 2.7 GB for the 35B at ctx 131072) on a node that has had an
|
|
OOM-controller kill. Replaced by the in-place wrap. The 32-byte alignment that
|
|
`cpu_buffer_from_ptr` asserts is met by rounding the base down; those leading bytes are never read.
|
|
- Writes to `/tmp` (node overlay) stalled once: 5.8 s for 64.5 MiB, against 171 ms normally.
|
|
- Costs to know about. llama-swap v239 waits for the old process to exit before it starts the next
|
|
model, and SIGKILLs a child still there 30 s after SIGTERM (measured 2026-10-25 with a scratch
|
|
llama-swap and dummy upstreams: term to exit 4.0 s, next start 2 ms later; a child that ignored SIGTERM
|
|
was replaced at +30.0 s). So the exit write adds directly to a model switch: about 1 s per GB on the
|
|
`/root` PVC (0.66 GB/s with fdatasync). A save cut off by the SIGKILL leaves only a `.tmp`.
|
|
- Only slots are saved. A conversation that sits only in the RAM cache when the process stops is lost.
|
|
- Every deploy empties the cache (the key holds the build commit), and so does any change to the
|
|
model file or the flags in the key.
|
|
|
|
**Rebase hazards.**
|
|
- `server_context::start_loop` and `vram_share_exit` ordering (F7); `get_available_slot` and the RAM
|
|
cache (`prompt_save`, `prompt_load`, `server_prompt_cache_state`); `server_prompt` and
|
|
`common_prompt_checkpoint` fields (new fields must be saved, or the format version bumped).
|
|
- The checkpoint rollback in `update_slots` (`pos_min_thold`, `TAG_PROMPT_LOGITS`), which decides how
|
|
much of a restored file is reused.
|
|
- The llama state format (`state_write`/`state_read` of the memory types). The key holds the build
|
|
commit, so a format change cannot load an old file; it only empties the cache.
|
|
- `llama_memory_i::restore_device_buffers` and the shadow layout of F4 (`device_buffer_shadow`):
|
|
`llama_memory_wrap_host` relies on the compact iteration-order shadow.
|
|
- Flag names in `common/arg.cpp`.
|
|
|
|
**Validation.** `fork-validate.sh --diskcache` (2c6c517ba), beside production. The model is the small
|
|
hybrid, CPU only (`-dev none -ngl 0`), with chat requests and greedy decoding, and a reference server
|
|
that never restarts. Cases: `restore`, `restore-arbiter` and `restore-cold` (turn 1, SIGTERM, restart,
|
|
turn 2 restored; prompt_n equal to the reference; identical answer), `no-prefix`, `key-ctx`,
|
|
`key-ctk`, `key-model`, `corrupt` (truncated and byte-flipped), `lru`, `write-error` (RLIMIT_FSIZE,
|
|
SIGXFSZ ignored), `restore-mmproj` (the same restore with the 0.8B's mmproj loaded),
|
|
`mmproj-image`, `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free.
|
|
`mmproj-image` sends an image (answered "red", not restored), then the text chat (restored, identical),
|
|
then an image again, and SIGTERMs with media in the slot (no state copied, exit 0). First run
|
|
(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219
|
|
MiB free). After the mmproj fix (beside production, 5.3 GB VRAM free): 15 PASS, 0 FAIL, 0 SKIP.
|
|
|
|
| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill |
|
|
|---|---|---|---|---|---|---|
|
|
| 573-token chat | 64.5 MiB, 2 ckpt | 18 ms | 171 ms | 314 ms (token free 56 ms vs 28) | 166 + 3 ms | prompt_n 58 of 596, 462 vs 2913 ms |
|
|
| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms |
|
|
| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s |
|
|
|
|
Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.sh`):
|
|
`--diskcache --arbiter` 29 PASS, 0 FAIL, GPU legs included.
|
|
|
|
35B (`q35.py`, the production 35B flags but WITH `--no-mmproj`, unlike production, which hid the
|
|
incident above; `--cache-disk-path` on a scratch dir, arbiter env, an 18760-token pan-tadeusz chat;
|
|
log `/root/arms/diskcache-test/q35.log`). `q35.py` now uses the exact production entry (`-hf`, so
|
|
the mmproj loads, nothing extra). That run (`/root/arms/diskcache-window3.sh`, about 8 minutes under
|
|
swapdown, which also covers the same-reuse-point identity and an image request on the 35B) was run
|
|
2026-11-21 on adc2aa3ec (log `/root/arms/diskcache-test/q35-mm.log`) and passed all legs: 18.8k-token
|
|
chat, prefill 84.6 s -> 4.7 s for turn 2 after the restart with identical output; save 557 MB in
|
|
1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image answered and not saved. The
|
|
table below is the earlier `--no-mmproj` run:
|
|
|
|
| leg | result |
|
|
|---|---|
|
|
| turn 1 prefill (reference and candidate) | 18760 tokens in 83.4 s (224.8 t/s), outputs identical |
|
|
| warm save on SIGTERM (a) | 556.5 MiB (2 checkpoints), copy 323 ms, write 960 ms; token free in 328 ms (6 ms without the cache), exit 1952 ms (607 ms without), peak RSS +430 MiB |
|
|
| restore after the restart (b) | read 860 ms, restore 114 ms; turn 2 prompt_n 27 of 18850 in 3.0 s (the reference that never restarted: 27 in 2.0 s); output identical |
|
|
| cold save on SIGTERM (b, KV in the host shadow) | 746.7 MiB (5 checkpoints), copy 310 ms, write 1403 ms, exit 2928 ms, peak RSS +432 MiB: the in-place wrap adds no copy |
|
|
| restore of the cold save (c, turn 2 sent again) | read 1189 ms, restore 75 ms, prompt_n 4; output differs from the reference's turn 2 (see below) |
|
|
|
|
Two anomalies in leg c, both explained:
|
|
- Output != reference: c sent turn 2 again, so the saved list (turn 2 + 63 generated tokens) extends
|
|
the new prompt. `update_slots()` then rolls back to the checkpoint at the prompt end minus 4 and
|
|
processes 4 tokens, while the reference processed turn 2 from turn 1's state (27 tokens). These are
|
|
different reuse points, not the same computation. The controlled test on Qwen3.5-0.8B
|
|
(`/root/arms/diskcache-test/cmp.py`) compares the same reuse point. Server X never restarts and
|
|
runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced
|
|
cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the
|
|
GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU
|
|
to itself: `/root/arms/diskcache-window3.sh` (production command line, reference "turn 2 again" against
|
|
leg c; about 8 minutes under swapdown). It ran 2026-11-21 and passed (same-reuse-point identity:
|
|
identical output), so the difference above is the reuse point. It supersedes
|
|
`diskcache-window2.sh`, whose `cmp.py` runs with `--no-mmproj`.
|
|
- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
|
|
The driver's fake waiter was `flock -s want.lock sleep 120`; killing flock left `sleep` holding
|
|
want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right
|
|
after its load and again right after its request (log 0.37.50). That ordinary arbiter release
|
|
recaptures the full KV allocation as a host shadow, about 2.6 GB at ctx 131072 (F4), and the SIGTERM
|
|
arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The
|
|
driver now holds the lock in the process it kills, like `fork-validate.sh` does.
|
|
|
|
Configuration (applied in production 2026-11-21): `--cache-disk-path /root/prompt-cache
|
|
--cache-disk-size 4096` appended to the `moe_offload_args` and `moe_offload_gemma` macros in
|
|
`/root/config.yaml` (backup `/root/config.yaml.bak-diskcache-20261121`). The directory is created on
|
|
first start. Verified in production: a 35B 4.6k-token chat was saved on eviction (278.6 MiB, copy
|
|
126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms), and turn 2 prefilled
|
|
only 32 tokens; a gemma image request with the flags on works and is not cached. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
|
|
applies. Models only pick up the flags when they restart.
|
|
|
|
### F12. Temporary upstream carries (open upstream PRs, drop when merged)
|
|
|
|
**What and why.** Open upstream PRs carried unchanged until upstream merges them. Each carry is one
|
|
commit whose subject starts with `upstream-carry #<PR>:`; the author is the PR author, and the body
|
|
names the PR URL and the PR head that was carried. List them with
|
|
`git log --oneline --grep '^upstream-carry' origin/master..HEAD`. They are not fork features: never
|
|
edit them in place, re-fetch the PR instead.
|
|
|
|
Carried since 2026-11-22 (both on upstream `e6d624b9a`, for Inkling-2-Mini, section 5):
|
|
- #32735 `vulkan : support non-contiguous src in SCALE` (head `0ea4ffd00`): the Vulkan backend runs
|
|
SCALE on a non-contiguous source instead of rejecting it.
|
|
- #32740 `ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan` (head
|
|
`808fd1c4e`, four PR commits squashed): Inkling's per-query bias table (`[n_bins, n_head,
|
|
n_tokens]` plus an I32 bin index per key) is applied inside the flash-attention kernel. Before it,
|
|
the graph gathered the bias into a dense per-head mask (`n_kv x n_tokens x n_head`, F32, added to
|
|
the KQ mask and cast to F16) and passed that to FA, and the Inkling layers could not use the
|
|
`kv_rows` cell lists (dense masks only). Vulkan
|
|
binds the table and the indices in the two mask_opt descriptor slots, so `use_mask_opt` gets
|
|
`!bias` (F2). Vulkan supports it only for F16 K/V, head size 64 or 128, with subgroup shuffle and
|
|
vote (RX 580 RADV has both; Inkling is f16 KV, head size 128).
|
|
|
|
Measured on the RX 580 (2026-11-21, the previous PR head: `1fa93f902`, `d29f9d489`, `ff91e4f84`,
|
|
without `808fd1c4e`; base `3bf6b55b3` vs base + both PRs; logs `/root/arms/pr-out/` on the pod):
|
|
- Inkling pp2048 at ub1024: 91.6 -> 110.5 t/s. pp2048 at depth 16384 (ub512): 48.4 -> 73.0 t/s.
|
|
- Inkling compute buffers at ctx 131072, ub512: Vulkan0 12918 -> 7173 MiB, Vulkan_Host
|
|
12729 -> 398 MiB. At ctx 65536 the base build already reserved 15866 MiB of Vulkan0 compute (more
|
|
than the card; the driver places the excess in GTT). With the carry: 6981 MiB.
|
|
- No regression elsewhere: Qwen 35B pp2048 286.5 -> 287.9, pp2048 at 16k 208.0 -> 208.5;
|
|
gemma-26B 249.4 both.
|
|
- test-backend-ops on Vulkan0: SCALE 15/15, FLASH_ATTN_EXT 5693/5693.
|
|
`808fd1c4e` (the PR head since 2026-11-22) changes the table layout from `[n_head, n_bins, n_tokens]`
|
|
to `[n_bins, n_head, n_tokens]`, which removes one permute+cont from the Inkling graph.
|
|
|
|
Measured with the carried head (2026-11-22, candidate `3621025e8` vs production `3bf6b55b3`,
|
|
production's Inkling command line with the nothink sampling, GPU to itself; driver
|
|
`/root/arms/carry1122-ink-direct.py`, logs `/root/arms/carry1122-p2.log`, `-p3.log`):
|
|
|
|
| build, args | 8.2K-token prefill | 13.9K | 28.4K | process VRAM / GTT after load |
|
|
|---|---|---|---|---|
|
|
| production, ub512, ctx 65536 | 32.7 t/s | 31.6 | 29.5 | 7209 / 28113 MiB |
|
|
| carry, ub1024, ctx 131072 | 43.7 | 42.5 | 42.5 | 7983 / 21835 MiB |
|
|
| carry, ub1024, ctx 65536 | 46.8 | 46.9 | 46.0 | 7368 / 19629 MiB |
|
|
|
|
- Device memory wanted (`-lv 4` breakdown): production ub512/65536 19424 MiB (compute 15866); carry
|
|
ub1024/131072 13388 MiB (model 1266, context 4419, compute 7702), host compute 798 MiB. Everything
|
|
above the 8 GB card lands in GTT (system RAM over PCIe): about 5.2 GB at 131072. That costs about
|
|
8% prefill against 65536, so production runs ub1024 with ctx 65536 (section 5) until the loop-layer
|
|
experts move into VRAM, which needs that headroom.
|
|
- Answers: a ~160-token first request on a fresh server (upstream #32778, all-'?' output) was
|
|
correct on every build. Pan Tadeusz fragments of 8.2K and 13.9K tokens gave coherent, correct
|
|
summaries on every build (upstream #32755 did not reproduce). The nothink entry
|
|
(`--reasoning-effort none --reasoning-budget 0`) returns repeated
|
|
`<|message_model|><|content_thinking|><|end_message|>` as content, with no tool call, for an image
|
|
request, a tool call and a 28.4K-token weather log ("which station is in Record 5?"). This happens
|
|
on every build, production `3bf6b55b3` with ub512 included, with and without
|
|
`GGML_OP_OFFLOAD_MIN_BATCH=64`, so the carry did not cause it. The think entry gets all three
|
|
right on both builds (image "Red", `get_weather({"city":"Warsaw"})`, 28.4K log "65" at 45.9 t/s
|
|
in production). Short text prompts work on the nothink entry.
|
|
- Nothink fix (2026-11-22, config only): the cause was `--reasoning-budget 0`. With effort `none`
|
|
and no budget the model still writes a short reasoning block (100-550 characters) and then
|
|
answers; a budget of 0 cut that block off at once and the model looped on the chat-structure
|
|
tokens. The macro is now `--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --reasoning-effort none`.
|
|
Verified on production `3621025e8` (pod driver `/root/arms/overnight/nothink.py`, logs
|
|
`nothink1.log`, `nothink2.log` there): short question, Polish poem, image ("Red"), tool call
|
|
(`get_weather({"city":"Warsaw"})`) and a 29.4K-token Pan Tadeusz prompt (correct, 46.0 t/s) all
|
|
OK. Effort `minimal` and a budget of 256 also work; the budget of 0 fails on all but the short
|
|
question.
|
|
|
|
**Rebase hazards.**
|
|
- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry
|
|
commit; still open -> compare `git diff <carried-head> <current PR head>` and re-carry the current
|
|
head if it moved; closed unmerged -> decide with the owner (Inkling's prefill and memory depend on
|
|
#32740, section 5).
|
|
- #32740 conflicts with F2 on `use_mask_opt` in `ggml_vk_flash_attn`. The resolution: upstream's
|
|
`!use_sparse`, the PR's `!bias &&`, the fork's GCN `HSK >= 256 || HSV >= 256` and its comment.
|
|
When the PR is merged and the carry dropped, the same line conflicts again when F2 is re-applied
|
|
on top; resolve it the same way.
|
|
- #32740 changes `llama_kv_cache::set_input_relative_pos` (new `rows` argument) and the
|
|
supports_op of every backend; a fork change in those places must follow it.
|
|
|
|
**Validation.** `test-backend-ops -b Vulkan0 -o SCALE` and `-o FLASH_ATTN_EXT` (check the case
|
|
count), and the PR's own test `tests/test-flash-attn-indexed-bias` (build target
|
|
`test-flash-attn-indexed-bias`; it runs every device, prints one line per case it ran and skips
|
|
unsupported cases silently, so count the `Vulkan0:` lines: 16 f16 cases on the RX 580, max error
|
|
about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rejects indexed bias
|
|
(the graph then uses the dense mask), so it prints no CPU lines).
|
|
Then an Inkling request through llama-swap.
|
|
|
|
### F13. Per-pass views of host weights in looped layers (inkling2_loop)
|
|
|
|
Branch `fable5/inkling-loop-views` (on top of the F12 carries), fast-forwarded into the production
|
|
branch on 2026-11-22 (deploy and results below).
|
|
|
|
**What and why.** `inkling2_loop` (Inkling-2-Mini) runs the core layers 6..17 three times; the 48
|
|
cache slots are 24 physical layers plus 2 x 12 repeats. `post_arch_tensors` makes every repeat slot
|
|
share the tensors of its physical layer (`layers[il] = layers[phys]`). With the experts in host
|
|
memory (`--n-cpu-moe 24`, `-ot exps=CPU`), `ggml_backend_sched` makes one device copy per (host
|
|
tensor, backend) per graph, hashed by tensor pointer, at the first split that reads it, and keeps it
|
|
allocated until the last reader (`ggml_backend_sched_split_graph`, pass 5 and the graph_copy input
|
|
nodes). Two consequences:
|
|
- Memory: the expert copies of all 12 loop layers are alive at the same time (about 547 MiB per
|
|
layer on UD-Q4_K_XL, 6.5 GiB in total). That is most of the ~7 GiB Vulkan0 compute buffer in the
|
|
F12 measurements.
|
|
- Correctness (upstream issue #32755, "GGGG" output): the copy callback (`sched_copy_experts`,
|
|
F1) runs once, for the first reader. When it reads the routing ids back (ubatches below the F1
|
|
skip threshold of `4 * n_expert` ids, and at or above the Vulkan MUL_MAT_ID offload batch of 32
|
|
tokens: 32-63 tokens on Inkling-2-Mini), it uploads only the experts pass 0 selected. Passes 1 and
|
|
2 reuse that copy and read stale VRAM for every other expert. Ubatches of 64+ tokens are correct
|
|
(F1 uploads every expert), ubatches below 32 run MUL_MAT_ID on the CPU (no copy).
|
|
|
|
**Where.** `src/llama-model.h` (`llama_model::post_load_tensors`, a new optional hook),
|
|
`src/llama-model.cpp` (`llama_model_base::load_tensors` calls it on the no_alloc return and after
|
|
`load_tensors_hadamard()`), `src/models/models.h` and `src/models/inkling2-loop.cpp`
|
|
(`llama_model_inkling2_loop::post_load_tensors`, member `ctx_visits`). Commit 312c29c2b.
|
|
|
|
**How it works.** After the weights are loaded, for every slot of pass 1 and 2 (`il >= n_phys`), each
|
|
tensor pointer in its `llama_layer` (walked as an array of pointers, as `llama_context::opt_init`
|
|
does) that sits in a host buffer (`ggml_backend_buffer_is_host`) is replaced with
|
|
`ggml_view_tensor` of it, named `<name>.visit<pass>`, initialized with `ggml_backend_view_init`
|
|
(same buffer and data). On the no_alloc pass (memory fit) only `view->buffer` is set to the dummy
|
|
buffer, so the scheduler still sees a WEIGHTS buffer. The view is a leaf (op NONE, `view_src` set),
|
|
so the scheduler hashes it separately: each pass gets its own copy, freed after the last reader of
|
|
that pass, and the copy callback sees the MUL_MAT_ID of that pass and its ids (`st.ids` differs, so
|
|
F1 recomputes the expert set). The pointer-keyed weight maps of `llama_model` (`w_scales`,
|
|
`prec_policy.prec_acc`/`prec_src1`, `hdmd.rot`) get entries for the views. The views live in
|
|
`ctx_visits`, a context of the model outside `ctxs_bufs`. The startup log prints
|
|
`post_load_tensors: N views of host weights for the loop passes after the first (X MiB, not copied)`.
|
|
|
|
Knob: `LLAMA_LOOP_HOST_VIEWS=0` keeps the shared tensors (the upstream behaviour), read once at load.
|
|
Default on.
|
|
|
|
**Invariants that must hold.**
|
|
- Only host buffers get views. Device weights stay shared (used in place, no copy, so no problem),
|
|
and so do CPU extra buffers (CPU_REPACK is not a host buffer; its ops run on the CPU in place).
|
|
- F4: views point only at host buffers, which `release_device_weights` never releases, and they
|
|
are not in `ctxs_bufs`, so the shadows, `weight_release`, `memory_breakdown`, `n_tensors()` and
|
|
`tensors_by_name` do not see them. A device weight is released and restored in place, and the
|
|
loop slots keep pointing at the same tensor.
|
|
- F1: the skip/readback decision is per copy and keys on the ids tensor of the first node of the
|
|
split, so it stays per pass.
|
|
- `llama_layer` must hold only `ggml_tensor *` members (a `static_assert` checks the size only).
|
|
|
|
**Cost.** Pass 1 and 2 now upload their own experts: per ubatch the host-to-device expert traffic
|
|
goes from 24 to 48 layer uploads (about +13 GiB per full ubatch on UD-Q4_K_XL). Measured where the
|
|
old layout still fits in VRAM (llama-bench pp2048 at depth 0, `-ncmoe 24`, B C C B in one window):
|
|
views 107.10 / 107.53 t/s, off 110.80 / 111.53, so -3.5%. Decode is unaffected (single-token
|
|
MUL_MAT_ID runs on the CPU). In the server at 16k-65k context the old layout does not fit (7 GiB of
|
|
compute buffer), spills into GTT, and the views win about 1.6x at prefill (Measured, below). The
|
|
freed ~6 GiB of VRAM can hold experts instead: an expert tensor of a loop layer kept on the GPU
|
|
saves 3 uploads per ubatch and 3 CPU expert passes per token, a non-loop one saves 1, so loop layers
|
|
are the first to move to the GPU (`-ot` with the loop layers left out of the CPU pattern; placement
|
|
guide below).
|
|
|
|
**Measured (2026-11-22, pod, GPU to itself, build `b13c41355` = 312c29c2b + docs; drivers and logs in
|
|
`/root/arms/overnight/`: `f13.py`, `cmp.py`, `kld48.sh`, `fit.py`, `sweep.sh`, `STATUS.md`).**
|
|
- Memory, server with the production command line at ctx 16384, `-lv 4`: Vulkan0 compute buffer
|
|
1426 MiB with views, 7030 MiB without. Process VRAM after load: views 4179 MiB, off 7727, A
|
|
(loop experts on the GPU, no views) 7879 (A wants 10089 MiB and spills).
|
|
- Prefill, server, 7224-token raw prompt at ctx 16384: views 92.4 t/s, off 57.0, A 46.5 (production
|
|
at ctx 65536 without views: about 46).
|
|
- Correctness. First-token top-5 logprobs (`f13.py`, `n_probs 5`, knob 32 so the 32-63 window is
|
|
live): with the tail at or above 64 or below 32 tokens (N = 20, 100, 7368) views on and off are
|
|
identical (max difference 0.000); in the window (N = 40, 63, 1072, 7224) they differ (0.04-0.16).
|
|
That test cannot say which arm is right: arm A differs from both by up to 0.15 even at the
|
|
controls (different placement, different kernels), and at 32-63 tokens x 8 experts pass 0's
|
|
union already covers most of the 128 experts, so the stale copy is mostly correct data. The
|
|
decisive test is `kld48.sh`: `llama-perplexity -c 48 -b 48 -ub 48`, 30 chunks of Pan Tadeusz, so
|
|
every chunk is one 48-token ubatch, KL divergence against a base with views on and knob 64 (the
|
|
tail MUL_MAT_ID on the CPU, correct by construction):
|
|
|
|
| arm | mean KLD | max KLD | same top token | RMS dp |
|
|
|---|---|---|---|---|
|
|
| views on, knob 32 (tail on the GPU) | 0.00314 | 0.087 | 97.4% | 1.35% |
|
|
| views off, knob 32 (tail on the GPU) | 0.03163 | 0.347 | 92.2% | 3.71% |
|
|
| views off, knob 64 | 0.00000 | - | 100% | 0.00% |
|
|
|
|
Views on is at the GPU-vs-CPU kernel noise level; views off is 10x worse (the #32755 mechanism,
|
|
reproduced). So with F13 the `GGML_OP_OFFLOAD_MIN_BATCH=64` workaround is no longer needed.
|
|
|
|
**Placement guide (UD-Q4_K_XL, views on).** k = number of loop layers (6, 7, ...) whose experts stay
|
|
on the GPU: `-ot 'blk[.](<the other 24-k layers>)[.]ffn_.*_exps=CPU'` instead of `--n-cpu-moe 24`
|
|
(`/root/arms/overnight/mkot.py` and `cfgedit.py` write the pattern; `[.]` avoids backslashes in the
|
|
YAML and in llama-swap's argument splitting). Each loop layer's experts take 547 MiB. Fit, from the
|
|
`-lv 4` breakdown of a server with the production command line (mmproj on demand, so not loaded):
|
|
|
|
| ctx | k | Vulkan0 self (model + context + compute) | free | verdict |
|
|
|---|---|---|---|---|
|
|
| 32768 | 0 | 4136 (1266 + 1347 + 1522) | 3982 | fits |
|
|
| 32768 | 6 | 7416 | 699 | fits |
|
|
| 32768 | 7 | 7963 | 15 | too tight |
|
|
| 65536 | 4 | 7246 (3453 + 2371 + 1421) | 868 | fits, chosen |
|
|
| 65536 | 5 | 7793 | 128 | too tight for the mmproj |
|
|
| 65536 | 6 | - | - | spills (GTT about 900 MiB above expectation) |
|
|
| 131072 | 0 | 7784 (1266 + 4419 + 2098) | 134 | fits (before F13 it overcommitted 5.2 GB) |
|
|
| 131072 | 2 | - | - | spills (about 1.2 GB) |
|
|
|
|
fdinfo `drm-memory-gtt` is not the spill test: it includes the pinned host weights
|
|
(Vulkan_Host). Compare GTT(k) with GTT(0) - k x 547 MiB, or read the breakdown.
|
|
|
|
Speed, llama-bench, one invocation per metric (`sweep.sh`, `-fa 1 -t 6 -b 1024 -ub 1024`, knob 64,
|
|
pp r1, tg r2):
|
|
|
|
| k | pp2048 @ d8192 | pp2048 @ d16384 | tg256 @ d4096 |
|
|
|---|---|---|---|
|
|
| 0 | 97.53 | 89.62 | 5.57 |
|
|
| 4 | 102.19 | 93.90 | 6.45 |
|
|
| 6 | 104.43 | 95.27 | 6.72 |
|
|
| 7 | 105.90 | 96.90 | 7.25 |
|
|
|
|
Decode gains most (+16% at k=4, +30% at k=7): every loop layer on the GPU removes 3 of the 48 CPU
|
|
expert passes per token. Prefill gains only 5-9%: at depth the attention and the remaining uploads
|
|
dominate.
|
|
|
|
**Quants (2026-11-22, not adopted).** Unsloth's Mainline_Support UD-Q3_K_XL (13.6 GB) and UD-IQ3_XXS
|
|
(11.7 GB) are in `/root/models/inkling/`. Quality: `llama-perplexity -c 2048 --chunks 4`, KL
|
|
divergence against UD-Q4_K_XL logits (`kld.sh`), on a fixed Pan Tadeusz chunk
|
|
(`/root/bench/entext/pt-chunk.txt`, characters 20000-100000) and wikitext-2 test:
|
|
|
|
| text | quant | PPL (Q4_K_XL) | PPL ratio | mean KLD | same top token |
|
|
|---|---|---|---|---|---|
|
|
| Pan Tadeusz | Q3_K_XL | 11.45 (11.30) | +1.4% | 0.0241 | 92.5% |
|
|
| Pan Tadeusz | IQ3_XXS | 11.52 (11.30) | +1.9% | 0.0421 | 90.1% |
|
|
| wikitext-2 | Q3_K_XL | 6.251 (6.230) | +0.3% | 0.0097 | 95.6% |
|
|
| wikitext-2 | IQ3_XXS | 6.307 (6.230) | +1.2% | 0.0214 | 94.2% |
|
|
|
|
Fit at ctx 65536: UD-Q3_K_XL keeps the loop-layer experts at 547 MiB each (like Q4), so k=5 (867
|
|
MiB free); UD-IQ3_XXS has 390 MiB per layer, k=7 (952 MiB free). Speed (separate invocations, so
|
|
within the ~7% cross-run noise for prefill): Q3_K_XL k=4 105.23 / 6.83, k=6 107.31 / 7.48;
|
|
IQ3_XXS k=4 105.59 / 6.44, k=7 108.48 / 7.86 (pp2048 @ d8192 / tg256 @ d4096; Q4_K_XL k=4
|
|
102.19 / 6.45). The best case, IQ3_XXS at k=7, is +6% prefill and +22% decode (about 7.9 instead
|
|
of 6.5 t/s) for a Polish KLD of 0.042; IQ3_XXS's CPU dot product is slower per byte (at k=4 it
|
|
decodes no faster than Q4). Not worth the quality loss for the owner's Polish use.
|
|
|
|
**Production (deployed 2026-11-22, `b13c41355`, ctx 65536, k=4, no knob).** Through llama-swap
|
|
(`/root/arms/overnight/post.py`, log `post.log`): Pan Tadeusz 9312 tokens 81.6 t/s prefill, +7958
|
|
tokens 89.3 t/s, +17255 tokens (28K total) 78.0 t/s, decode 5.9-6.0 t/s, answers correct (before:
|
|
about 46 t/s prefill). Think and nothink entries answer text, an image ("Red") and a tool call. A
|
|
tiny request after the 4B task model held the GPU takes 1.3-2.4 s longer than a warm one (the
|
|
arbiter restores 3.4 GiB of device weights). Disk cache save/restore works on Inkling (1329 tokens,
|
|
410 MiB). The first request on a fresh process is slow (162 tokens 11.7 t/s prefill, 2.6 t/s decode)
|
|
since the pipelines are compiled on first use (`--no-warmup`).
|
|
|
|
**Profile (2026-11-22, `perf record` + `gpu_busy_percent`, `prof.sh`, results in
|
|
`/root/arms/overnight/prof/`).** Inkling k=4 prefill (pp2048 @ d4096, 107 t/s): GPU busy 77%, CPU
|
|
time mostly `ggml_vk_wait_for_fence` (43%) and the kernel (11%, the uploads): GPU-bound. Decode
|
|
(7.3 t/s): GPU busy 58%, CPU in the OpenMP barrier spin (libgomp 36%) and
|
|
`ggml_vec_dot_nvfp4_q8_0` (21%): the UD-Q4_K_XL experts are NVFP4, and that CPU dot product is the
|
|
decode hotspot. Qwen3.6-35B for comparison: prefill 263 t/s at GPU busy 52%, decode 13.8 t/s at
|
|
35%, `vec_dot_q4_K_q8_K` 10%. `GGML_VK_PERF_LOGGER=1` aborts at the second split
|
|
(`GGML_ASSERT(ctx->compute_ctx.expired())`, ggml-vulkan.cpp:15692), so it gives per-op GPU times
|
|
only for the first split; not investigated.
|
|
|
|
**Failure modes and known limits.**
|
|
- LoRA adapters look weights up by name (`llama_adapter_lora::get_weight`), so they do not apply to
|
|
passes 1 and 2 with views on. Not used here; set `LLAMA_LOOP_HOST_VIEWS=0` for a LoRA on Inkling.
|
|
- `--moe-cache-mib`: the cache binds expert tensors per slot (`llama_moe_cache_layer_experts` of
|
|
`model.layers[il]`), so with views each loop slot gets its own binding and slot map and caches the
|
|
same experts separately (before, the three slots collided on one binding key). Should not crash;
|
|
not tested. The MoE cache is off in production.
|
|
- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data
|
|
in a saved file cannot happen.
|
|
|
|
**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for
|
|
every reader in the scheduler, or real per-pass tensors in the loader), revert 312c29c2b; the hook has
|
|
no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
|
|
|
|
**Rebase hazards.** `post_arch_tensors` and the loop slot layout in `inkling2-loop.cpp`;
|
|
`llama_model_base::load_tensors` returns (a new return path must call the hook); new pointer-keyed
|
|
weight maps in `llama_model` (they need a view entry like `w_scales`); non-pointer members in
|
|
`llama_layer`; scheduler changes to how host weights are copied (`hv_tensor_copies`,
|
|
`ggml_backend_sched_is_host_weight`).
|
|
|
|
**Validation (pod, GPU to itself in a swapdown window).** Build `llama-server` and `llama-bench`
|
|
from this branch in a scratch worktree (runbook section 5). Use the Inkling command line from
|
|
`inkling_args` in `/root/config.yaml` (with `--mmproj`) and change only the context to 16384 (so
|
|
arm A fits) and the placement:
|
|
- A, reference: replace `--n-cpu-moe 24` with `-ot 'blk\.([0-5]|1[89]|2[0-3])\.ffn_.*_exps=CPU'`
|
|
(loop experts on the GPU, no copies for the loop; the views log line must not appear).
|
|
- B, views on: `--n-cpu-moe 24`; the log line must show the views.
|
|
- C, views off: B with `LLAMA_LOOP_HOST_VIEWS=0`.
|
|
|
|
For each arm send raw token prompts (`/tokenize` a long text, cut to N tokens, `/completion` with the
|
|
token array, `temperature 0, top_k 1, seed 42, cache_prompt false, n_predict 32, n_probs 5`) at
|
|
N = 40, 63, 1072, 6176, 7224 (tail 32-63 at ub1024), controls N = 100, 7368 (tail >= 64) and 20.
|
|
Expected:
|
|
- B == A in the first token's top-5 probabilities for every N >= 32 (prefill MUL_MAT_ID runs on the
|
|
GPU in both arms with the same data). The generated text may drift later, since decode MUL_MAT_ID
|
|
runs on the GPU in A and on the CPU in B.
|
|
- B == C byte-identical for the controls (tail >= 64 or < 32).
|
|
- C differs from A at the 32-63 tails (the bug); if it does not, the bug was not reproduced and
|
|
B's result proves nothing.
|
|
- Vulkan0 compute buffer (startup log): B about 6 GiB below C. A's model buffer carries the loop
|
|
experts instead.
|
|
Then `llama-bench -lm none -ngl 99 -ncmoe 24 -fa 1 -t 6 -b 1024 -ub 1024 -p 2048 -n 0 -r 3`, with and
|
|
without `LLAMA_LOOP_HOST_VIEWS=0`, interleaved B C C B, medians (section 6), for the prefill cost
|
|
above. After a deploy: an Inkling text request, an image request, and the same text request after a
|
|
task-model call (arbiter handoff, F4), identical at temperature 0.
|
|
|
|
---
|
|
|
|
### F14. User-message batch-break merge (one ubatch per short turn)
|
|
|
|
**What and why.** Upstream ends a prompt batch right before a user message so that a context
|
|
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
|
|
limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state
|
|
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
|
|
message is what a later request uses when it diverges inside that message (the user edits it, or the
|
|
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
|
|
template drops the previous answer's reasoning, which moves the divergence to just after
|
|
`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
|
|
restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation
|
|
prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each
|
|
ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128
|
|
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
|
|
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
|
|
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
|
|
`<|im_start|>user`, the detected delimiter).
|
|
|
|
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
|
|
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
|
|
Commit 510aa6e09.
|
|
|
|
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
|
|
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
|
|
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
|
|
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
|
|
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
|
|
the restored checkpoint at the same position, with the current task id, so the next request still
|
|
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
|
|
diverges inside that message restores the batch-start checkpoint instead and re-processes the
|
|
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
|
|
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
|
|
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
|
|
upstream behaviour everywhere.
|
|
|
|
**Invariants that must hold.**
|
|
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
|
|
divergence inside the last user message to less than `n_ubatch` tokens.
|
|
- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared
|
|
by `has_mtmd`); hence the `!has_mtmd` condition.
|
|
- It only changes where batches end, never which tokens are processed, so outputs differ from an
|
|
unmerged run only by batch-composition rounding.
|
|
|
|
**Failure modes and incidents.** None yet; not validated on the pod, not deployed.
|
|
|
|
**Rebase hazards.**
|
|
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
|
|
`checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch
|
|
start is still checkpointed when the batch ends at prompt end minus 4.
|
|
|
|
**Validation plan (pod, not yet run).**
|
|
1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the
|
|
`created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch
|
|
`[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the
|
|
last user message.
|
|
2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded),
|
|
candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn
|
|
chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read
|
|
`prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens
|
|
(the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens.
|
|
3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit
|
|
the last user message and resend: merged re-processes previous answer + message (bounded by one
|
|
ubatch), unmerged only the message; no "forcing full prompt re-processing".
|
|
4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line):
|
|
restore after a restart, `prompt_n` and output identical to the reference built from the same
|
|
commit.
|
|
|
|
---
|
|
|
|
## 4. Dead ends
|
|
|
|
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
|
|
without a new reason.
|
|
|
|
| idea | result | mechanism |
|
|
|---|---|---|
|
|
| Weight pinning (`GGML_CUDA_REGISTER_HOST` and the host-register chain, from a third-party CUDA fork whose +64% was on an RTX 3060) | +17% pp2048 in isolated llama-bench (252 -> 294), fails in serving | RADV cannot import file-backed mmap pages; the pre-stage fallback fails (`ErrorOutOfDeviceMemory` for the ~18 GB expert region) or doubles host RAM. `-lm none` serves faster (Qwen3.6-35B 278 -> 286). Dropped in the 2026-09-09 rebase. |
|
|
| Expert-upload prefetch (`GGML_SCHED_PREFETCH_EXPERTS`) | -20% pp (other runs -9% to -22%), no tg benefit | Slots allocate in host-pinned GTT, not VRAM, so it stages host to host and still pays the PCIe read. `=1` means the default 3 slots. Dropped. |
|
|
| `GGML_VK_ASYNC_USE_TRANSFER_QUEUE=1` | flat (-0.1%), byte-identical | GCN is excluded from `prefers_transfer_queue` and RADV exposes no transfer-only queue family here; the per-layer upload-then-compute order leaves nothing to overlap. A standalone Vulkan microbench did show 60-73% of an H2D copy on queue family 1 hidden behind compute, so double-buffered prefetch inside one backend context remains an untested idea (estimated +10-15%, unproven). |
|
|
| Fractional `-ncmoe` | no measured win | Config only uses integers (22/40/48); upstream refactored `-ncmoe` into `llm_add_n_cpu_ffn_overrides`. Dropped. |
|
|
| MoE cache `--moe-cache-mib` 512 / 1536 / 2304 (Qwen3.6-35B, ncmoe 40, ctx 131072) | tg256@d4096 -42% / +5% / +12%; prefill +1-3%; idle VRAM 6.15 / 7.19 / 7.96 GB vs 5.66 | Small caches thrash (8-of-256 routing over 40 layers) and add a per-layer ids readback; only ~2.3 GiB pays and leaves no VRAM for a long KV. Not enabled. |
|
|
| MTP speculative decoding, Qwen3.6-35B | decode -39% (n1), -37% (n2), -50% (n3), -44% (n4), best -19% (n2, p-min 0.75); prefill -3 to -6% | A spec step costs ~180 ms vs 52 ms for plain decode; a 2-token verify alone is only 1.4x a 1-token step, so ~100 ms/step is draft machinery. The nextn layer adds 1.25 GiB VRAM. Entries removed. |
|
|
| MTP, gemma-4-26B-A4B | decode -57.9% (n1), prefill pp4k -34% | Same mechanism (~210 ms vs ~50 ms per step); MTP needs `--swa-full`, which alone costs -10.5% decode, -32% pp4k and saturates VRAM. Entries removed. |
|
|
| MTP, gemma E4B | decode -10.9% | Acceptance 0.41-0.67. The +19.5% pp seen with it comes from `--swa-full`, not MTP. |
|
|
| `--parallel 1` on gemma | no effect (-1.0%, +2.1%) | The Qwen gain is probably tied to its recurrent state being sized per sequence (unverified). |
|
|
| `-ub 4096 -b 4096` (Laguna) | flat (90.0 vs 90.9) | Fewer expert uploads cancelled by intra-ubatch attention growth. |
|
|
| mul_mat_id tile-padding theory (larger/odd `-ub`) | predicted 2.00x waste and a 1.94x cliff; measured +3.8% and -5.6%, no sawtooth | Theory dead; kills the mul_mat_id tile-selection shader work (an unbuilt, unverified WIP exists on branch `fable5/mmid-tile-wip`, with an unresolved correctness concern in its row_ids fill). |
|
|
| `-b` sweeps | non-monotonic (2048/8192 high, 4096/12288 low) | `-b 2048` is already at the top. |
|
|
| `-ncmoe` below 40 on the 35B | 34: +2.9% pp but 7.23 GiB peak at tiny context; 28: better at short context, worse at depth (137 vs 170 t/s at d32k); <= 24 collapses (~122) | Resident experts are evicted by KV/activation growth; cannot survive a 128k context. |
|
|
| `-ncmoe` 44 on Laguna | +1.5% | Does not fit at 64k context. Keep 48. |
|
|
| fp16 on Polaris (Mesa 26.1.6 + `radv_enable_float16_gfx8=true`, `fp16: 1`) | pp2048 -4.7%, -5.3% at d16k, tg +4% | Hurts prefill (the priority). Mesa 26.1.6 alone: +3.3% prefill, below the 5% bar; adopting it needs a pod image change. The pod's Mesa 25.0.7 has no such option. |
|
|
| KV quant on Qwen (q4_0, q8_0) | f16 beats q4_0 by +40% pp at 64k; q8_0 170 -> 128 t/s at depth; q4_0 with upstream PR #30738 still -16% vs f16 at d32k (133.6 vs 159.4) | No int-dot/fp16 here, so dequant is pure compute; the hybrid model's KV is small anyway. Keep f16. (PR #30738 was +15% at d16k on gemma-26B with q4_0 KV; not merged upstream as of the 20261024 rebase.) |
|
|
| `-fa 0` | 5x slower at depth (170 -> 35 at d32k) | Keep flash attention on. |
|
|
| FA scalar `block_rows` 16 -> 32 | 3.5x regression at d32k (135 -> 38) | ~42 KB shmem per workgroup at head_dim 256, one workgroup per SIMD, occupancy collapse. |
|
|
| `GGML_VK_DISABLE_GRAPH_OPTIMIZE=1` | restores tg32 at n_ctx=32 (10.44 -> 15.61), nothing at real depth | Upstream #27812's view-alias fix makes KV views look dependent and collapses barrier-free batches; only visible when per-node work is tiny. It is a correctness fix; do not disable it in serving. |
|
|
| `--no-mmap`/`-lm none` for Laguna-S 118B | does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. |
|
|
| Laguna `--cache-reuse`, `--swa-full`, `-t 12`, KV quant | ruled out | `--cache-reuse` is force-disabled for iSWA when base and SWA sizes differ. |
|
|
| CPU-only task model (`-ngl 0`) to avoid handoffs | rejected | At `--ctx-size 8192` titles could face 8k tokens of CPU prefill. The arbiter is the answer. |
|
|
| `-lm dio` (direct IO) instead of the F10 page-cache drop | load +0.9 s on 3.4 GiB (6.89 vs 5.99 s), the same cost as the drop; 175 MiB of the model file and all of the mmproj stay cached | Synchronous 64 MiB O_DIRECT chunks with no readahead; the GGUF metadata goes through a separate buffered FILE and clip.cpp has no direct-IO path. Measured CPU only; reads into pinned Vulkan buffers may fall back to buffered IO (`EFAULT` or `EINVAL`). Not adopted. |
|
|
|
|
The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256
|
|
tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.
|
|
|
|
Where the time goes (Laguna-S-2.1 118B IQ2_M, all experts on host, `GGML_VK_PERF_LOGGER=1`, depth 0,
|
|
22.65 s per 2048-token eval, 17.18 s of it GPU-busy, so about 24% is H2D stall; moved here from the
|
|
old README section): expert `MUL_MAT_ID` 52%, attention projections 24%, `FLASH_ATTN_EXT` 16%,
|
|
everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s while dense `MUL_MAT` q5_K/q6_K in
|
|
the same graph reaches 3096-3691, so closing that gap is the largest remaining opportunity on this
|
|
card, not the transfer path. On interleaved-SWA models the sliding-window layers keep a small KV
|
|
(`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`), so they cost the same at any depth; on Laguna 36 of
|
|
48 layers are O(1) in depth and the whole high-context slowdown comes from the 12 full-attention
|
|
layers.
|
|
|
|
## 5. Configuration that matters
|
|
|
|
`/root/config.yaml` (llama-swap) and `/root/supervisord.conf` on the pod are the source of truth;
|
|
this section explains the choices. Values as of the 2026-11-22 deploy.
|
|
|
|
Qwen 35B-A3B MoE entries (macro `moe_offload_args`, applied to all 35B-A3B variants):
|
|
`--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1`,
|
|
context 131072.
|
|
|
|
- `-lm none`: load weights into RAM (upstream removed `--no-mmap`; the old spelling makes the server
|
|
refuse to start). Faster in serving than mmap and recommended for `--n-cpu-moe`. Exception:
|
|
Laguna-S, which must use mmap (GTT limit, section 4).
|
|
- `-ub 2048 -b 2048`: the biggest prefill lever. ub512 = 97 t/s, ub1024 = 159, ub2048 = 254 on the
|
|
35B; the per-eval expert upload is amortized over the ubatch.
|
|
- `--n-cpu-moe 40`: all experts on host. Lower values are faster at short context but run out of VRAM
|
|
at long context (section 4). Measured idle VRAM at ctx 131072: 5.66 GB.
|
|
- `--parallel 1`: +22.9% decode on the Qwen 35B (15.5 -> 19.1 t/s), prefill +0.3-1.2%, +64 MiB VRAM.
|
|
Without it the server creates 4 slots and a unified KV. Two concurrent requests queue on slot 0.
|
|
With one slot a ubatch never holds more than one sequence, which the concat fast path's
|
|
`ne[2] == 1` condition (F3) needs.
|
|
- `-t 6`: six physical cores; more threads hurt decode.
|
|
- `-fa 1`: essential (section 4).
|
|
- f16 KV (no `-ctk/-ctv`): section 4.
|
|
|
|
Other models:
|
|
- gemma-4-26B-A4B: macro `moe_offload_gemma`, same as above but `--n-cpu-moe 22` (8 layers of experts
|
|
resident; gemma's SWA keeps KV small at depth). ncmoe sweep pp2048: 30 = 262, 24 = 266, 18 = 272,
|
|
14 = 157 (VRAM collapse); 22 keeps headroom for 128k context (7.40 of 8 GB at 128k).
|
|
Its KV cache is `-ctk q4_0 -ctv q4_0`, set in the `gemma4_sampling` / `gemma4_nothink_sampling`
|
|
macros (not in `moe_offload_gemma`); note gemma has no `--parallel 1` (no measured gain there).
|
|
- Small models that fit in VRAM (Qwen3.5-4B, gemma E4B/E2B and similar) use auto-fit
|
|
(`common_args`); forcing experts to host would slow them.
|
|
- The 4B task model runs with `--no-mmproj` and ctx 8192, and is preloaded.
|
|
- Laguna-S-2.1 118B IQ2_M (configured as `Laguna-S-2.1-IQ2_M`, matrix var `lag`): mmap, `--n-cpu-moe 48 -b 2048 -ub 2048 -fa 1 --parallel 1`,
|
|
ctx 65536. Its exclusivity set must allow the task model (`lag & q4task`); making it exclusive with
|
|
the task model killed it on every title request and forced full re-prefills. `--parallel 1` frees
|
|
223 MiB of VRAM at 64k context there (4 auto slots size the SWA cache `n_swa * n_seq_max +
|
|
n_ubatch`, 4096 cells instead of 2560).
|
|
- Inkling-2-Mini (26B MoE, `inkling2`, 24 layers with the middle 12 looped 3x, 24 heads / 8 KV
|
|
heads, head size 128; macro `inkling_args`, entries `unsloth/Inkling-2-Mini-GGUF[-nothink]:UD-Q4_K_XL`,
|
|
matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m`
|
|
(the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it,
|
|
`-t 6`, f16 KV, `--parallel 1`, the disk prompt cache, `-b 1024 -ub 1024 --ctx-size 65536`
|
|
(since the 2026-11-22 carry of upstream #32740, F12; before: `-b 512 -ub 512`). Since the F13
|
|
deploy (2026-11-22, `b13c41355`): the experts of loop layers 6-9 stay on the GPU and all others
|
|
on the host, `-ot blk[.](0|1|2|3|4|5|10|11|12|13|14|15|16|17|18|19|20|21|22|23)[.]ffn_.*_exps=CPU`
|
|
instead of `--n-cpu-moe 24` (k=4 of the F13 placement guide: 7246 MiB of device memory, 868 MiB
|
|
left for the mmproj). The `GGML_OP_OFFLOAD_MIN_BATCH=64` env of both entries was removed: F13
|
|
makes the 32-63-token ubatches correct (KLD in F13). Other choices from the guide: ctx 32768 with
|
|
k=6, or ctx 131072 with k=0 (fits since F13, 134 MiB free, so images would spill). Smaller quants
|
|
(UD-Q3_K_XL, UD-IQ3_XXS, `/root/models/inkling/`) were measured and not adopted (F13, Quants).
|
|
Its nothink macro uses `--reasoning-effort none` (no `--reasoning-budget 0`: that budget made it
|
|
loop on chat-structure tokens, F12); even with effort none it writes a short reasoning block.
|
|
Against Qwen3.6-35B (think entries, 7 tasks, one sample each, 3500-token cap, 2026-11-22,
|
|
`/root/arms/overnight/quality.py`, `quality/quality.json`): Inkling 6/7, Qwen 5/7. Inkling found
|
|
all three bugs of a binary search where Qwen ran out of budget; both ran out on a 4-line Polish
|
|
rhyme. Inkling used about a third fewer tokens (9677 vs 14254) but took 1.7x the wall time (2283
|
|
vs 1316 s; decode 4-6 vs 13 t/s) and 4x on a 29K-token question (835 vs 211 s). Too small a
|
|
sample to call it smarter; it is clearly slower here.
|
|
|
|
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
|
|
`/root/supervisord.conf`:
|
|
- `LLAMA_SLEEP_VRAM_ONLY=1`: enables the arbiter and VRAM-only sleep (F6).
|
|
- `LLAMA_VRAM_ARENA=/dev/shm/llama-vram`: the shared arena. Every co-loaded GPU model must use it.
|
|
- `LLAMA_SLEEP_EVICT_KV=1`: cold models also free KV, recurrent state, scheduler and scratch (F4),
|
|
so the warm model gets nearly the whole card. Costs a 2-3 s re-warm instead of ~0.6 s.
|
|
- `LLAMA_MMPROJ_ONDEMAND=1`: encoder in VRAM only during encodes (F5). No-op for models without an mmproj.
|
|
- No `--sleep-idle-seconds`: release is purely demand-driven via want.lock.
|
|
- `GGML_CUDA_REGISTER_HOST=1` may still be present from the pinning era; it is a no-op with `-lm none`.
|
|
|
|
Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`,
|
|
`GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`.
|
|
`--cache-disk-path /root/prompt-cache --cache-disk-size 4096` (F11) is set in the `moe_offload_args`, `moe_offload_gemma` and `inkling_args` macros.
|
|
|
|
Machine-level:
|
|
- GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted
|
|
every second; default/max cap 145 W). The two GPU hangs of 2026-10-09 happened before the cap was
|
|
fixed (it used to switch between 120 and 145 W by temperature); heavy runs since have been clean.
|
|
- Kernel argument `amdgpu.lockup_timeout=0,120000,0,0` (compute ring 120 s) via Talos
|
|
`machine.install.extraKernelArgs`. Without it long 64k-context submissions hit the compute-ring
|
|
watchdog (`ring comp_1.x.0 timeout`).
|
|
- Upstream changed the default `--port` from 8080 to 9931; the config passes `--port ${PORT}`
|
|
explicitly, so it is unaffected.
|
|
|
|
## 6. Measurement methodology and traps
|
|
|
|
- **Cross-run variance is about 7%.** llama-bench's within-run error bars understate cross-invocation
|
|
variance by about 14x (the same config read 276.7 +/- 0.9 and 296.4 +/- 0.6 in two invocations).
|
|
Put every comparison you draw a conclusion from inside one llama-bench invocation
|
|
(comma-separated sweeps). Two builds cannot share an invocation, so interleave them over at least 3
|
|
rounds and compare medians. Treat any cross-build delta under about 5% as unproven.
|
|
- **Alternate arm order.** A fixed order gave the later slot about a 9% advantage and nearly produced
|
|
a false regression finding. Use orders like ref, cand, cand, ref.
|
|
- **Node CPU contention.** The node runs about 60 pods; llama-bench once got 4.67 of 12 hardware
|
|
threads while asking for 6. With `--n-cpu-moe 40` every MoE layer decodes on the CPU, so stolen
|
|
threads map onto tg almost directly. Prefill is nearly immune (held +-0.3% while tg32 swung +-14%).
|
|
Expect occasional catastrophic tg samples; use medians, never means. Historical tg figures taken
|
|
under different load are not comparable.
|
|
- **Never use tg32.** At the default depth (n_ctx 32) decode is launch/barrier bound and measures
|
|
command-stream overhead: two upstream commits read -36% at tg32 and were at parity at tg256 (and +7%
|
|
at d512). Use tg256 (or tg128) with an explicit depth (`-p 0 -n 256 -d 4096`). tg is not monotonic in
|
|
n_ctx on this box (n_ctx 256 measured slower than 32 and 544), so only compare same-depth cells.
|
|
- **`-b Vulkan` vs `-b Vulkan0`.** `test-backend-ops -b Vulkan` matches nothing, runs 0 cases and
|
|
still prints OK with exit 0. The backend is `Vulkan0`. Always check the case count.
|
|
- **ubatch larger than the prompt never fills.** With `-ub 2048` use prompt lengths that are
|
|
multiples of 2048. A partial trailing ubatch costs a full expert upload.
|
|
- **`-tb` is not a llama-bench flag** (server only). Any bad flag makes llama-bench print usage and
|
|
exit, which looks like a run that produced no rows. Always check the row count. `-ncmoe` is
|
|
llama-bench's spelling of `--n-cpu-moe`.
|
|
- **llama-bench prints its table only after all `-d` values finish.** No output does not mean hung.
|
|
- **Nothing else may use the GPU** during a measurement; stop llama-swap (`bench.sh` and
|
|
`/root/bench/moe/swapdown.sh` do this and restart it from a trap).
|
|
- **Check the build finished** before benchmarking it (library mtime earlier than the run start).
|
|
Never overlap a build with a big benchmark (host OOM).
|
|
- **Prompt-cache reuse is `timings.prompt_n`,** not `tokens_evaluated` (which is always the full
|
|
prompt length).
|
|
- **Per-process VRAM** from `/proc/<pid>/fdinfo/*` `drm-memory-vram` must be deduplicated by
|
|
`drm-client-id`; summing all fds double-counts shared buffers. `drm-memory-gtt` is system RAM, not
|
|
VRAM. Whole-card usage: `/sys/class/drm/card*/device/mem_info_vram_used`.
|
|
`GGML_VK_MEMORY_LOGGER=1` logs every Vulkan buffer alloc/free with a running device total to stderr.
|
|
- **The concat fast path is invisible to test-backend-ops** (F3); validate it with the knob, perf
|
|
logger timing, or byte-identical greedy generation (temperature 0, top_k 1, fixed seed,
|
|
`cache_prompt: false`).
|
|
- **Do not run `llama-cli` detached with closed stdin** on the pod: it can loop forever printing
|
|
prompts (it once wrote a 65 GB log). Use llama-bench `-d` or the server harness.
|
|
|
|
## 7. Known gaps and open risks
|
|
|
|
- `llama_context::restore_device()` ignores the `bool` returned by `restore_device_weights()`,
|
|
`moe_cache->restore_device()` and `memory->restore_device_buffers()`. On a restore OOM,
|
|
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
|
|
null buffer would crash. Only the mmproj path handles a failed restore.
|
|
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
|
|
- Inkling (F12, section 5): the nothink entry's chat-structure loop came from
|
|
`--reasoning-budget 0` and was fixed in the config on 2026-11-22 (F12). Upstream #32755 / #32778
|
|
did not reproduce here; F13 bounds the loop-pass expert copies (more in F13). Drop the F12 carries when upstream merges #32735 and #32740.
|
|
- Open after the 2026-11-21 rebase: is the fork's concat fast path (F3) now redundant with upstream's
|
|
own tiled concat (#30591, gated in the fork with `ne0 < 32` and the `GGML_VK_CONCAT_FASTPATH` knob)?
|
|
Benchmark it. Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no
|
|
(handoff median 692 ms candidate vs 1068 ms reference).
|
|
- Router mode can spin at shutdown (F7); not used here.
|
|
- The unexplained ~1.6 GiB peak-VRAM spike in the first candidate `--perf` arm, seen twice (F9).
|
|
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
|
|
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
|
|
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
|
|
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
|
|
is an estimate; its memory effect is verified in production.
|
|
- F11 (disk prompt cache): the same-reuse-point identity check and the production-command-line (mmproj
|
|
loaded) 35B run both passed on 2026-11-21 (`/root/arms/diskcache-window3.sh`, F11 Validation), and
|
|
the cache is on in production. Remaining gap: only the 35B and gemma were checked in production;
|
|
other models gain the flag only if they use those macros.
|
|
- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's
|
|
host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time
|
|
(token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that
|
|
shadow while it is cold.
|
|
- F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key
|
|
holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB.
|