- AGENTS.md: one-line pointer to FORK.md and the runbook at the top. - FORK.md: candidate status; commit map rows for the new commits; F12 state after the rebase (#32735 merged and dropped; #32740 split down to the GGML contract, the 808fd1c4e carry kept, owner decision; #32777 carried) and how to use --fit for Inkling, with the F13 interaction read (fit measures F13's compute buffer; the visit count is taken on the all-device probe, so the views do not hide the loop); F13 section on upstream #32807, the LLAMA_LOOP_FULL_COPY knob and the window A/B; F14 refinement (system-prompt checkpoint, disk cache read, lifetime) and validation plan items 5-7. - Runbook: a rebase is not done until fork-validate phases 1 and 2 pass; ssh access and the persistent rootfs; gdb persists; the shrinking-PR case for carries; the 2026-11-22 rebase record; the overlayfs lock_id issue; phase 1 results and the ordered list of what the window must run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
143 KiB
FORK.md - the RX 580 llama.cpp fork
Maintainer documentation for this private fork. Read it before you rebase, change, or deploy anything. The rebase and deploy procedure is in docs/fork-rebase-runbook.md, which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-11-22):
- Candidate, not deployed: branch
fable5/vram-ondemand-rebase-20261122on upstream5bacb7c86(19 upstream commits paste6d624b9a, including upstream's #32755 fix #32807 and the merge of #32735), with the F14 first-user-message break, theLLAMA_LOOP_FULL_COPYknob (F13) and the carry of #32777 (F12). Pod build/root/arms/reb20261122/build. Its validation and the Inkling A/Bs are in runbook section 3 ("2026-11-22") and in F12, F13 and F14 below; production stays onf3397cdeduntil they pass. - Production runs
f3397cdedon branchfable5/vram-ondemand-rebase-20261121(upstreame6d624b9a, 757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, F13, the per-pass loop views for Inkling, and F14, the user-message batch-break merge), deployed 2026-11-22 10:37 (F14 validated in its own window: 35B TTFT A/B, disk cache 15 PASS and the 35B legs; F14 measured). Rollback pointer/root/DEPLOY-ROLLBACK-commit-16.txt=b13c41355(build copy/root/arms/build-b13c41355; config unchanged by this deploy); next free pointer 17. The previous pointer 15 =3621025e8needs its config backup/root/config.yaml.bak-20261122-predeploytoo.system_fingerprintisb<build number>-<short sha>(currentlyb12687-f3397cded) and changes with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook (section 3, "2026-11-21"). - Previous branch
fable5/vram-ondemand-rebase-20261024on upstream3f03b1772(productionadc2aa3ecuntil 2026-11-21), kept untouched as the safety net. The fork commits aregit log --oneline origin/master..HEAD; that list includes documentation and script commits (this file among them), so count it rather than trusting a number written here. Section 2 maps every commit to a feature. - The disk prompt cache is ON in production (F11).
2fc9f9c35was validated with the fullfork-validate.shset (37 PASS, 0 FAIL). Any later commit that touches compiled code needs the same validation before a redeploy. - Branch
fable5/disk-prompt-cacheadded F11 (disk prompt cache) and a CPU_REPACK release fix (F4). A deploy ofcb48bf224was rolled back on 2026-10-25 (mmproj abort, F11); the fix (adc2aa3ec) passed the 35B run with the production command line on 2026-11-21 and is deployed. - Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line. The commit messages are detailed and are the primary record of each mechanism
and measurement; read them with
git show. - Commit trailers (owner's convention): this fork is private and never contributes upstream.
Agent-written commits end with
Co-Authored-By: Claude <model> <noreply@anthropic.com>(some older ones sayAssisted-by: Claude). AGENTS.md asks agents forAssisted-by:instead ofCo-authored-by:and forbids AI-written commit messages and PR descriptions; those are upstream llama.cpp's contribution rules and do not apply to commits that stay in this fork. Anything meant for upstream must be written by the owner under those rules.
Contents
- Purpose and target
- Commit map
- Features
- F1 MoE routing-ids readback skip
- F2 GCN flash-attention mask_opt gate (and the shmem-staging note)
- F3 Vulkan concat tiled-transpose fast path
- F4 On-demand device residency (library side)
- F5 On-demand mmproj (multimodal encoder)
- F6 Cross-process VRAM arbiter (server side)
- F7 Server shutdown path
- F8 Cold-teardown guards
- F9 Benchmark harness and validation script
- F10 Page-cache drop after a copying load
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
- F14 User-message batch-break merge (one ubatch per short turn)
- Dead ends
- Configuration that matters
- Measurement methodology and traps
- Known gaps and open risks
1. Purpose and target
The fork exists to serve large MoE models well on one specific machine:
- GPU: AMD RX 580 8 GB (Polaris, GCN gfx803), RADV Vulkan driver. llama.cpp reports
fp16: 0(RADV does not exposeshaderFloat16on GFX8), no int-dot, no matrix cores, wave64. PCIe 3.0 x16. RADV exposes no transfer-only queue family on this card. - Host: Ryzen 5 3600 (6c/12t), 62 GB RAM, a Talos Kubernetes node shared with about 60 other pods (CPU contention matters for decode numbers, see section 6). The GPU runs at a fixed 120 W power cap.
- Primary model: Qwen3.6-35B-A3B (unsloth UD-Q4_K_XL, 20.8 GiB), a hybrid MoE: 40 layers, 30 of
them gated delta-net (recurrent), 10 full attention, head_dim 256, 256 experts with 8 used. All
experts live in host RAM (
--n-cpu-moe 40); only the backbone, KV, recurrent state and compute buffers sit in VRAM. - Serving: llama-swap under supervisord keeps several llama-server processes loaded at once (the 35B main models, a Qwen3.5-4B "task" model OpenWebUI calls for titles and tags, gemma-4-26B multimodal, others). They cannot all fit in 8 GB, so they time-share the GPU through the fork's VRAM arbiter (F4-F7) without restarting processes, which would throw away the RAM-only prompt cache.
- Priority: prefill (prompt processing) speed first, decode second. Agentic use pushes large tool results through the model, so long-context prefill is what users wait on.
What the fork adds, in short:
- three Polaris prefill wins (F1, F2, F3), all exact (token-identical to mainline);
- on-demand release/restore of everything a model holds in VRAM, plus a cross-process arbiter that hands the GPU from one llama-server to another in well under a second for small models;
- fixes to the server's sleep, wake and shutdown paths that the arbiter exposed;
- a benchmark harness and a post-rebase validation script.
2. Commit map
Oldest first. "Feature" points at the section below.
| commit | subject | feature |
|---|---|---|
c72cb424e |
llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert | F1 |
fb6a8a076 |
ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes | F2 |
ff2a96d2d |
vulkan: record why flash-attn shmem staging stays off for GCN | F2 |
3df719e1e |
ggml-vulkan: tiled transpose fast-path for concat with transposed source | F3 |
481be8b28 |
server: on-demand VRAM sharing to time-share one GPU between models | F4, F6 |
17165fe2a |
server: optionally evict the KV cache too (Phase 2 of VRAM sharing) | F4 |
958257cd6 |
server: keep the sleep-wake path active when the VRAM arbiter is enabled | F6 |
b9df854de |
server: coordinate model load with the VRAM arbiter to avoid load-time OOM | F6 |
8d80f0673 |
llama: guard buffer iteration against released device buffers | F8 |
2feb6aeb2 |
server: restore weights/KV on wake, not at decode, so KV eviction is safe | F6 |
bf2c1a699 |
server: also free the compute-graph scheduler on KV eviction | F4 |
9857a45f0 |
llama: evict recurrent/SSM state on device release | F4 |
9b555f1aa |
ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc | F4 |
ad27d4921 |
llama: free backend compute scratch on cold; guard memory_breakdown | F4, F8 |
0a0f3ab6d |
mtmd: release/restore the encoder weights from VRAM on demand | F4, F5 |
5e99ac198 |
server: on-demand mmproj - free encoder VRAM on the text path | F5 |
b7c131219 |
docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends | docs |
5e429d4a9 |
ggml-vulkan: skip the concat transpose fast path below one tile | F3 |
4c4486f8e |
scripts: add the RX 580 benchmark harness | F9 |
878d1338f |
server: replace the VRAM doorbell with a want-lock | F6 |
53f907771 |
llama: skip the compute-buffer size check when the context is cold | F8 |
7ac2014a8 |
llama: release/restore the MoE cache with the device weights | F4 |
54a7f534b |
scripts: rx580-bench - use -lm none, document the MoE cache option | F9 |
11044b35e |
server: run the shutdown handler off the signal-interrupted thread | F7 |
6c480f7e7 |
server: release the GPU token promptly on shutdown | F7 |
e085c347d |
ggml-vulkan: env knob for the concat transpose fast path | F3 |
149a8cb0a |
scripts: add fork-validate.sh, post-rebase validation of the fork | F9 |
bfbaf15b2 |
docs: FORK.md and rebase runbook for the RX 580 fork | docs |
f5c0d8f28 |
server: fix stale VRAM-arbiter comments and the sleep log line | F6 |
f664b0f20 |
llama: env knob for the routing-ids readback skip | F1 |
62d442e7b |
scripts: rx580-bench - decode at tg256 depth 4096, not tg32 | F9 |
2512290f9 |
scripts: fork-validate.sh - gate --perf on medians, not means | F9 |
80e662a2a |
scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 |
ceab1a91f |
scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
e5fc474b8 |
scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
cce0ac1f0 |
llama: drop the model file's page cache after a copying load | F10 |
f24c3582c |
llama: do not release CPU extra buffers (CPU_REPACK) on device release | F4 |
dd851fd0c |
llama: read a released KV/recurrent state from its host shadow | F4, F11 |
2874dcb10 |
server: disk prompt cache, save slots on shutdown and restore after a restart | F11 |
2c6c517ba |
scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
d2396e4e3 |
upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
72da62cea |
upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
312c29c2b |
llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as b13c41355) |
F13 |
510aa6e09 |
server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as f3397cded) |
F14 |
e7e907b7c |
server: F14 - keep the batch break at the first user message (system-prompt checkpoint) (branch -20261122 only) |
F14 |
5f88124b3 |
llama: env knob LLAMA_LOOP_FULL_COPY=0 to skip upstream's inkling2_loop full-copy guard (#32807) (branch -20261122 only) |
F13 |
6c895a491 |
upstream-carry #32777: fit : prioritize looped MoE layers in device memory (branch -20261122 only) |
F12 |
70d39bdbc |
scripts: fork-validate.sh - scratch arenas on /dev/shm, lock_id failed on overlayfs (branch -20261122 only) |
F9 |
On fable5/vram-ondemand-rebase-20261122 the #32735 carry (d2396e4e3) is gone (merged upstream) and
the other hashes in this table differ; the last four rows are hashes of that branch.
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to cce0ac1f0 is in production (2fc9f9c35, the documentation commit after it);
6c480f7e7 was the previous production commit. The last four rows are on fable5/disk-prompt-cache
only (hashes of that branch), not deployed.
3. Features
Every feature section uses the same headings: What and why; Where; How it works; Invariants that must hold; Failure modes and incidents; Rebase hazards; Validation.
F1. MoE routing-ids readback skip
What and why. With experts host-resident, each split uploads the experts it needs. To choose them, the expert-copy callback reads the routing ids back to the host. The ids are produced on the same GPU the upload goes to, so the readback forces a full pipeline flush: 47 per eval on a 48-layer MoE. At prefill batch sizes every expert is used anyway (2048 tokens x 10 experts over 256 experts), so the readback buys nothing. Measured on Laguna-S-2.1 118B IQ2_M, all experts on host (before it was ported to the current callback): pp2048 +4.7% at depth 32k, +0.9% at 16k, flat at depth 0, tg +3%.
Where. src/llama-context.cpp, llama_context::sched_copy_experts (registered with
ggml_backend_sched_set_copy_callback in the scheduler setup). Commit c72cb424e.
How it works. When ggml_nelements(ids) >= 4 * n_expert, it skips
ggml_backend_tensor_get_async + ggml_backend_synchronize, clears st.ids_data and sets
st.used.assign(n_expert, true). The existing grouping then copies the whole expert tensor as one
range. Below the threshold the upstream readback runs unchanged. For Qwen3.6-35B (256 experts, 8
used) the skip applies to any ubatch of 128 tokens or more; decode (8 ids) never reaches it.
Invariants that must hold.
- Exactness: uploading an expert no id selects cannot change the result, because
mul_mat_idonly reads the rows the ids point at. Any upstream change that makes unselected experts matter (for example a kernel that reads every uploaded expert) breaks this argument. - With
--moe-cache-mibon, the all-true set still goes throughmoe_cache->copy_experts, which reads the host slot maps (no device readback) and copies cached experts device-to-device. Still exact. st.idscaching stays coherent: a repeat call with the same ids tensor reuses the all-true set, which is what the threshold would decide again; a differentn_expertrecomputes.- The small-batch MoE-cache slot path (
llama_moe_cache::copy) has its own readback and is not touched.
Failure modes and incidents. None in production. Historically the change lived in
ggml_backend_sched_compute_splits in ggml/src/ggml-backend.cpp; upstream #29943 (6753a033f)
deleted that block and moved expert copying into llama_context::sched_copy_experts. The original
cherry-pick therefore had nothing to apply to and was re-authored as c72cb424e. A -X theirs
resolution would have resurrected the deleted scheduler block.
Rebase hazards.
- Any change to
sched_copy_expertsor theststate struct (ids,used,ids_data). - Upstream
701fd25b3(MUL_MAT_ID id -1 skips work) left aGGML_ASSERT(id >= 0 ...)in the readback branch; the skip branch reads no ids, so it is unaffected, but watch for upstream semantics around negative ids. - If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.
Validation. fork-validate.sh --moe-exact MODEL (needs the GPU to itself, run it in the
--perf swapdown window with the 35B): reads expert_count/expert_used_count from the GGUF,
computes the threshold ceil(4 * n_expert / n_expert_used) tokens per ubatch (128 on the 35B), and
compares 64 greedy tokens after a prompt of about twice that, every expert on the host:
candidate == reference (identical-ref), and with the LLAMA_MOE_IDS_SKIP knob (f664b0f20;
=0 forces the readback, =log reports each skip on stderr) candidate == candidate with the skip
forced off (identical-noskip, which proves the skip exact even when the reference build carries
it) and the log shows skips, all at >= 4 * n_expert ids (skip-engaged). First GPU run
2026-10-25 (35B, 2fc9f9c35 vs 6c480f7e7): prompt 397 tokens, 39 skips logged, 0 with the skip forced
off, output identical in all three runs, 78 s. Also --ops (the MUL_MAT_ID
kernel, not this callback), --perf (prefill regression) and --moecache (the cache path through
this callback).
F2. GCN flash-attention mask_opt gate (and the shmem-staging note)
What and why. mask_opt skips fully-masked causal blocks and drops the per-block mask load+add
on fully-visible blocks. Upstream disables it on AMD GCN for head sizes <= 256. On Polaris attention
at depth is compute-bound on the softmax path (exp runs at about 1/4 VALU rate, no matrix cores), so
skipping that work is a real win. Measured on Qwen3.5-35B-A3B (head_dim 256): pp2048 +1% at depth 0,
+7.8% at 16k (176 -> 190), +12.3% at 32k (136 -> 153), growing with depth. Perplexity bit-identical
(5.1668 on and off).
Where. ggml/src/ggml-vulkan/ggml-vulkan.cpp, ggml_vk_flash_attn, the use_mask_opt
condition. Commit fb6a8a076. The shmem-staging note is a comment in get_fa_tuning_params_scalar
(commit ff2a96d2d, no behaviour change).
How it works. The GCN clause HSK > 256 || HSV > 256 becomes HSK >= 256 || HSV >= 256. The
existing large-mask conditions (nem1 >= 32, nem0 * nem1 > 32768,
nem0 >= block_cols * 16) keep it off for decode.
Invariants that must hold.
- Only GCN with head size >= 256 is affected. Do not lower the threshold: relaxing it below 256 was -18.5% at 16k on Laguna (head_dim 128).
- Upstream's
!use_sparseterm must stay in the condition (sparse flash-attention came in with upstreamfc82583e6). shmem_stagingstays NVIDIA-only. Measured on GCN at head size 128: -6.7% at 16k, -7.4% at 32k. Thekvshstride ofD/4+1dwords is 4 mod 32, an 8-way LDS bank conflict on wave64 (the+1padding is tuned for warp32), and the extra shared memory cuts occupancy. If anyone retries it, fix the padding to +2/+3 first.
Failure modes and incidents. A -X theirs resolution in a trial rebase silently dropped
!use_sparse from the condition. No production incident.
Rebase hazards. Any upstream rewrite of use_mask_opt, of the FA tuning parameters, or of
architecture detection (vk_device_architecture::AMD_GCN). Upstream 733f57a99 added a path that
copies strided f16/bf16 K/V into prealloc_x scratch for flash-attention on AMD non-RDNA4 (so GCN
is included) when neq1 >= 256; measured at d32768 it adds about 68 MiB of VRAM and no visible
long-context prefill loss, but it has no env toggle, so its speed effect could not be isolated.
Validation. fork-validate.sh --ops (FLASH_ATTN_EXT, 5332 cases on Vulkan0) and --perf
(pp2048 at depth 16k and 32k is where this feature shows).
F3. Vulkan concat tiled-transpose fast path
What and why. Delta-net layers build their conv input with ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0). The generic concat shader reads the transposed source with an
uncoalesced stride: about 5.6 ms for a 16 MB concat on the RX 580 against about 130 us of memory
bandwidth. On Qwen3.6-35B the fast path alone is +14% pp8192 and accounts for more than all of the
fork's prefill win at depth 0 (the rest of the fork costs about 1% there). In the validation run on
Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 1945 us without.
Where. ggml/src/ggml-vulkan/ggml-vulkan.cpp: ggml_vk_concat (gate lambda
src_transposed_2d and the s1_trans/s0_trans checks) and ggml_vk_concat_transpose_fastpath.
The graph pattern that matters here comes from llm_build_delta_net_base::build_conv_state in
src/models/delta-net-base.cpp (qwen35, qwen35moe, qwen3next). Commits 3df719e1e (fast path),
5e429d4a9 (decode gate), e085c347d (env knob).
Other graphs that build ggml_concat(state, ggml_transpose(x), 0) (found with
git grep -n "ggml_concat.*ggml_transpose" src/models/ plus the two-line form in delta-net-base and
lfm-base): qwen4exp.cpp (build_conv_state_at), bailingmoe3.cpp, clef.cpp, kimi-linear.cpp,
kimi-k3.cpp, glm5-next.cpp, mamba-base.cpp (two sites, the Mamba/Mamba2 conv and the hybrids
built on it), plamo2.cpp, and lfm-base.cpp (causal short conv; the transpose is assigned to bx
first). The fast path takes any of them when the gate holds (transposed source a plain 2D transpose
of a contiguous f32/f16 tensor, one sequence, ne[0] >= 32), so a change to the gate or the shader
affects all of them. Only the delta-net path (Qwen3.5/3.6) has been measured and validated; the
--concat check uses qwen35.
How it works. For a concat along dim 0 where one source is stored transposed (nb[1] == type_size, nb[0] == ne[1] * type_size) and the other source and dst are contiguous, it issues two
dispatches: pipeline_cpy_32/_16 copies the contiguous source into its dst columns, and
pipeline_cpy_transpose_32/_16 (the shared-memory 32x32 tiled transpose in copy_transpose.comp)
writes the transposed source into the other columns, both with custom push constants and a dst
offset. It ends with ggml_vk_sync_buffers. Everything else falls through to the generic path.
The gate requires: f32 or f16 with all three types equal (quantized and integer types stay generic);
matching ne[1..3]; ne[2] == 1 && ne[3] == 1 on the transposed source (so it fires only with one
sequence per ubatch, which --parallel 1 gives); the 16-bit doffset bound; and
ne[0] >= 32 (TILE_DIM) on the transposed source.
The decode gate (5e429d4a9): at one token the transposed source is a single column, so there is no
stride to fix, but the path still costs two dispatches and the unconditional ggml_vk_sync_buffers,
a full pipeline barrier (the generic path emits none). That barrier, not the half-empty transpose,
was nearly all of a -17% tg256 regression, because the path ran 30 times per decoded token on the
35B. The transposed source is qkv_mixed transposed, so its ne[0] is the ubatch token count:
prefill keeps the fast path, decode and small speculative drafts take the generic one. Measured
with the gate: decode back to parity (gated/clean 1.023), prefill win retained (gated/ungated 0.998,
pp8192 +11% over clean), generation byte-identical.
The knob (e085c347d): GGML_VK_CONCAT_FASTPATH=0 forces the generic path, =log prints
ggml_vk_concat: transpose fast path, transposed ne0=N to stderr for every fast-path dispatch. It is
read once (function-local static). Not set in production.
Invariants that must hold.
- The fast path fires at prefill and never at decode on a delta-net model.
- Output is identical to the generic path (it is a pure copy/transpose).
- The
types_okgate keeps quantized types off the path: the dispatches are raw f32/f16 element copies.
Failure modes and incidents.
- The decode regression above shipped in the deployed fork before the gate existed. It looked tg-neutral on the old upstream only because tg32's +-14% spread hid a 17% effect (section 6).
- If upstream changes
build_conv_state(for example aggml_conton the transpose), the fast path silently stops firing. Nothing crashes; prefill just loses about 14%. Only the concat check or a perf A/B notices.
Rebase hazards.
- Pipeline renames: upstream
733f57a99renamedpipeline_cpy_f32_f32/pipeline_cpy_f16_f16topipeline_cpy_32/pipeline_cpy_16. The pick applied cleanly and did not compile. - Upstream
f172be756madeggml_vk_concatnon-static (declared inggml-vulkan-common.h) and split helpers into headers; the fast-path helper stays static. - Shape changes in
build_conv_state/qkv_mixed([ch, n_seq_tokens, n_seqs]reshaped from a contiguousbuild_lora_mmoutput), changes tocopy_transpose.comporcopy.comppush constants, or tovk_op_unary_push_constants/get_misalign_bytes. - Upstream concat now handles quantized types; keep
types_ok. - The stock
test_concatcases intests/test-backend-ops.cppnever produce a transposed source and always havene[2]/ne[3]> 1, sotest-backend-ops -o CONCATdoes NOT exercise this path.
Validation. fork-validate.sh --concat (needs a build with the knob, i.e. this branch or
--concat-build): counts fast-path dispatches in a prefill run (must be > 0, all with ne0 >= 32)
and a decode run (must be 0 while CONCAT nodes did execute), then ties the unmodified candidate to
the fast path by CONCAT timing (candidate about equal to knob-on and at least 2x faster than
knob-off). Last run: 72 dispatches at prefill (all ne0=512), 0 over 594 CONCAT nodes at decode.
Also --perf.
F4. On-demand device residency (library side)
What and why. Several always-loaded models must share 8 GB. Co-residence is impossible (the 35B
at --n-cpu-moe 40 uses most of the card; the 4B task model needs about 2.5-3.3 GB). Restarting a
process to free VRAM throws away the RAM-only prompt cache: on Laguna a repeat turn went from a
21,960 ms re-prefill to 225 ms once the process stayed alive across a task-model call. So a model
must be able to drop everything it holds in VRAM and get it back without a reload.
Measured footprints:
- 4B: warm 3327 MB, weights released 724 MB, restored and correct.
- 4B at 32k ctx: cold weights-only 1750 MB (KV stays), cold with KV evicted 726 MB.
- gemma-26B: warm 7147 MB, cold with weights+KV but scheduler kept 718 MB, cold with the scheduler freed too 51 MB.
- A cold 4B before the encoder work held 644 MB that turned out to be its mmproj (F5).
- Re-warm cost: about 0.6 s per cold-to-warm switch weights-only (tiny prompt: 1.98 s cold vs 1.40 s warm); 2-3 s with KV eviction (it adds the KV D2H/H2D and a scheduler reserve).
Where.
src/llama-model.cpp/.h:llama_model::release_device_weights(keep_shadow),restore_device_weights(),weights_resident(); per-buffer metadatallama_model::impl::weight_release(weight_release_slot), filled inload_tensorsandload_tensors_hadamard.src/llama-memory.h:llama_memory_i::release_device_buffers(keep_shadow)/restore_device_buffers()(default no-op). Implemented insrc/llama-kv-cache.cpp(llama_kv_cache) andsrc/llama-memory-recurrent.cpp;llama_memory_hybridandllama_kv_cache_iswadelegate to their children.src/llama-moe-cache.cpp/.h:llama_moe_cache::release_device(),restore_device(),resident(),impl::invalidate().ggml/include/ggml-backend.h,ggml/src/ggml-backend-impl.h,ggml/src/ggml-backend.cpp: optionalggml_backend_i::free_scratchandggml_backend_free_scratch(); Vulkan implementationggml_backend_vk_free_scratchinggml-vulkan.cpp.src/llama-context.cpp/.h:llama_context::release_device(evict_kv),restore_device(),discard_device(),device_resident(), memberskv_device_evicted,device_discarded; the auto-restore at the top ofllama_context::decode.include/llama.h:llama_context_release_device(ctx, evict_kv),llama_context_restore_device,llama_context_discard_device,llama_context_weights_resident.tools/mtmd/clip.cpp/.h,tools/mtmd/mtmd.cpp/.h:clip_release_device,clip_restore_device,clip_weights_resident,mtmd_release_device,mtmd_restore_device.- Commits
481be8b28,17165fe2a,bf2c1a699,9857a45f0,9b555f1aa,ad27d4921,0a0f3ab6d,7ac2014a8,6c480f7e7(discard).
How it works. Tensor metadata (ggml_context structs) is durable; device buffers are ephemeral.
- Weights: on release, for each releasable buffer, capture a host shadow once (weights are
read-only), free the buffer, null every tensor's
buffer/data. On restore,ggml_backend_alloc_ctx_tensors_from_buftreallocates an identical layout (re-pointing the tensors), the shadow is uploaded, usage is set to WEIGHTS. Only single-buffer, alloc-path, non-host buffers are releasable: the host-pointer path (mmap or lazy-mapped),no_allocdummy buffers and host (Vulkan_Host, i.e. pinned RAM) buffers stay as they are. - KV and recurrent state: same pattern, but the shadow is re-captured on every release (the data is read-write) and dropped after restore.
- Scheduler: with
evict_kv,sched.reset()frees the worst-case compute buffer andsched_need_reserve = true; restore callssched_reserve()(which goes throughsched_new(), re-registering the expert-copy callback and upstream's realloc callback). - Backend scratch: with
evict_kv,ggml_backend_free_scratch()on each backend. On Vulkan this discards the pending compute context, synchronizes, destroysprealloc_x,prealloc_y,prealloc_split_k,prealloc_add_rms_partials,sync_staging, and zeroes their sizes and theprealloc_y_last_*tracking. They are reallocated lazily on the next compute. Other backends leave the hook null (no-op). - MoE cache (
--moe-cache-mib, upstream #29887): released in both modes. The banks are freed and the cache is emptied, not shadowed: every SLRU is rebuilt fresh, itsslot_map[il]pointers are re-pointed at the host slot-map tensors, and the host slot maps are cleared to 0xff (every slot -1). On restore the banks are reallocated (which also re-inits their views), cleared to 0 and marked WEIGHTS; the cache refills from the host experts. - Multimodal encoder:
clip_release_device/clip_restore_deviceuse the weights pattern (shadow once); no-op for a CPU-backed encoder.mtmd_*wrap the vision and audio contexts. llama_context::release_device(evict_kv): synchronize, release weights, release MoE cache, and ifevict_kv: release memory buffers, free sched, free scratch.restore_device(): weights, MoE cache, memory buffers,sched_reserve()if needed, thenggml_backend_sched_synchronizeso uploads finish before any compute.discard_device(): likerelease_device(true)withkeep_shadow = falseeverywhere (no host copies), setsdevice_discarded; used only before exit.decode()callsrestore_device()if!device_resident(), as a safety net. The server restores explicitly on wake (F6), because some KV access happens before decode.
Invariants that must hold.
- Shadows are stored compactly in stable tensor-iteration order, skipping view tensors
(
view_src != NULL), and restored with the same iteration. Never index a shadow byt->data - base. weight_releasestays parallel topimpl->ctxs_bufs: every push toctxs_bufsneeds a matching push toweight_release(non-releasable slots useemplace_back()).- The caller synchronizes before any release;
release_devicedoes this itself. - Restore order: weights, MoE cache, KV/recurrent, scheduler reserve, scheduler synchronize. The MoE cache banks must be back before the scheduler is rebuilt.
- After a MoE-cache release no slot map may point at an expert: banks come back zeroed, so a stale slot map silently reads zero weights.
- Every iterator over
ctxs_bufs(model, KV cache, recurrent) must tolerate a null buffer, and every use ofschedmust tolerate null while cold withevict_kv(see F8). device_resident()covers weights and MoE cache; it gates the early returns, the decode auto-restore andllama_context_weights_resident().restore_device()afterdiscard_device()aborts (GGML_ASSERT(!device_discarded)).- Greedy output is byte-identical across any number of release/restore cycles, and the prompt cache
survives (
timings.prompt_ndrops on a repeated prompt after a cycle).
Failure modes and incidents.
- The first shadow implementation indexed by
t->data - baseand produced garbage after restore (layout and padding differ). Fixed by the compact iteration-order shadow. - Restoring KV lazily at decode crashed iSWA models (gemma) with
GGML_ASSERT(buf != NULL && "tensor buffer not set"):update_slots()creates an SWA checkpoint that reads the KV viaggml_backend_tensor_getbefore decode. Fixed by restoring on wake (2feb6aeb2). - The recurrent state was left resident (default no-op), 201 MB on the 4B, until
9857a45f0. - A model with weights on the CPU device (
-dev none/-ngl 0) segfaulted (exit 139) inrelease_device_weights()the first time the arbiter put it to sleep: CPU_REPACK buffers are not host buffers, soload_tensorsmarked them releasable, and the shadow capture called their NULLget_tensor. It also crashes production2fc9f9c35, but no production arbiter model has CPU-device weights. Fixed inf24c3582c: only buffers on a non-CPU device are releasable. Found by the--diskcachecheck (F11), whose arbiter cases run CPU only. - The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed
resident in a cold model until
bf2c1a699/ad27d4921. - The MoE cache was not released at all before
7ac2014a8. Reallocating its banks without invalidating would have produced silent wrong weights. - In the 20261024 rebase
moe_cache_slru::init()turned out to reset only the probation segment (head[1],tail[1]andn_protectedsurvive), so re-running it on a used cache corrupts the lists.invalidate()now assigns a freshmoe_cache_slru()beforeinit().
Rebase hazards.
- New push sites into
ctxs_bufs(upstream'sload_tensors_hadamard()added two in the 20261024 rebase; without matchingweight_releaseentriesrelease_device_weights()reads past the end). - Changes to buffer creation in
load_tensors(thehost_ptr_pathcondition,ctx_bufs/buf_ptrsbookkeeping; upstream replacedctx_buf_maps.emplace_back(ctx, buf_map)withctx_bufs.emplace_back(ctx, std::move(buf_ptrs))in the 20261024 rebase). - The
llama_memory_iinterface. New memory types that do not overriderelease_device_bufferskeep their KV resident underevict_kv; at the time of writing that isllama_kv_cache_msa,_dsa,_dsa_iswa,_dsv4andllama_memory_hybrid_iswa. - MoE-cache internals (
groups,slru,slot_map,buf_host,devices). Upstream33d6135cbchanged LRU to SLRU (g.lru->g.slru); the pick applied cleanly and did not compile. llama_contextmembers next to ours (upstreamn_reallocs_same/diff,copy_experts_info),sched_reserve()semantics, and thedecode()signature (nowdecode(const llama_batch_ext &)).ggml_backend_ilayout (free_scratchmust stay the last member aftergraph_optimize; other backends rely on zero-init) and the Vulkan context's prealloc field set (compare withggml_vk_cleanup).
Validation. fork-validate.sh --arbiter: b-greedy-identical (15 cycles each side equal to a
no-handoff reference), b-kv-survives (prompt_n drops after evict/restore, output identical),
c-image-after-restore. --moecache MODEL (needs a cache large enough to hit; 256 MiB on gemma-26B
had 0 hits and proves nothing, 2048 MiB had 6651 hits). --arbiter v-vram-release /
v-vram-restore check that VRAM really drops: a baseline of mem_info_vram_used before any scratch
server, then server A warm and alone, forced cold by a fake waiter on want.lock: the card-wide delta
must drop by >= 70% of what A added and come back after the next request. Production's own share is
re-read from the DRM fdinfo of every process the script did not start; if it moved by > 64 MiB the
check SKIPs. First run (2026-10-25, 0.8B with mmproj, ctx 8192, LLAMA_SLEEP_EVICT_KV=1): warm
+546 MiB, cold +4 MiB, restored +546 MiB.
F5. On-demand mmproj (multimodal encoder)
What and why. A multimodal model's vision encoder sat in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the encoder lives in RAM and enters VRAM only for an encode. Measured on the
35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered
correctly, text after the image correct.
Where. tools/server/server-context.cpp: encode_mtmd_chunk (upstream 9871df591 split the
encode out of process_mtmd_chunk; encode_mtmd_chunk is called from process_mtmd_chunk and
directly from the batched-embedding path in the slot update, so the restore covers both), the
mtmd_release_device(mctx) after mtmd_init_from_file in load_model, vram_go_cold,
vram_share_exit. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
How it works. After load the encoder is released (shadow captured). Before
mtmd_batch_encode, mtmd_restore_device(mctx); if that fails (out of VRAM) and on-demand is set, it
releases the LLM backbone weights only (llama_context_release_device(ctx_tgt, false): read-only
shadow, so a cheap free with no D2H, KV and prompt cache untouched) and retries. After the encode it
releases the encoder FIRST, then restores the weights, so the peak stays within VRAM.
Invariants that must hold.
- Free the encoder before re-uploading the weights.
- The donor is the weights, never the KV (the KV is read-write and would need a D2H every time).
vram_ensure_warm()does not restore the encoder; only the encode path does.
Failure modes and incidents. Before this, a cold text model with an auto-downloaded mmproj
(-hf pulls one when the repo ships it) held 644 MB that no release path touched. The text-only 4B
task model now runs with --no-mmproj, which is the right fix for any model that never sees images.
Rebase hazards. Renames or restructuring of encode_mtmd_chunk and the mtmd init block in
load_model; clip_ctx fields (buf, ctx_data, sched, no_alloc).
Validation. fork-validate.sh --arbiter case c-image-after-restore (server A runs with
--mmproj and LLAMA_MMPROJ_ONDEMAND=1; a 128x128 red PPM must be answered "red" after A was cold,
and text right after must equal the reference). Post-deploy: one image request to gemma-26B.
F6. Cross-process VRAM arbiter (server side)
What and why. Several llama-server processes under llama-swap share one GPU: exactly one is warm
(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production:
60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between
the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last
fork-validate.sh run before the 20261024 deploy: 30 handoffs, median 680 ms, max 1231 ms (the
2026-10-25 run of the same build read 1050 ms, max 1562 ms: handoff latency moves with node load,
which is why it is now A/B'd against the reference in one session).
Where. tools/server/server-context.cpp: vram_arena_open, vram_acquire_for_load,
vram_share_init, vram_should_yield, vram_want_acquire, vram_want_release, vram_take_token,
vram_ensure_warm, vram_go_cold, vram_share_exit, vram_share_shutdown, the VRAM-only branch of
handle_sleeping_state, the vram_ensure_warm() call before decode in the slot update path, and
server_res_generator (bypass_sleep). tools/server/server-queue.cpp/.h: on_should_yield,
start_loop (should_sleep, idle tick, post-wake grace), wait_until_no_sleep, is_stopping.
Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
How it works.
- Enabled by
LLAMA_SLEEP_VRAM_ONLY(any value). The arena directory isLLAMA_VRAM_ARENA(default/dev/shm/llama-vram) with two files. token.lock:flock(LOCK_EX)is the baton. Rule: a process has device buffers iff it holds it.want.lock: a waiter holds itLOCK_SHwhile waiting and drops it once it owns the token. The holder probes it withflock(LOCK_EX | LOCK_NB)on a SECOND fd: failure means somebody is waiting. The probe needs its own open file description, because flock on the waiter's own fd would convert that lock instead of conflicting with it. The kernel drops a dead waiter's lock, so nothing goes stale.- Load:
load_model()callsvram_acquire_for_load()beforecommon_init_from_params(), so the current holder releases before this model uploads anything. The loaded model stays warm holding the token and serves its first request without a re-warm. - Yield:
vram_share_init()registersvram_should_yieldwithqueue_tasks.on_should_yield.start_loop()'sshould_sleep()calls it on the loop thread, undermutex_tasks, only when the task queue is empty andupdate_slotshas returned. While the callback is set the idle wait ticks every 10 ms instead of 1 s. After a wake the server holds the GPU for at least 100 ms (yield_grace_ms). Sleep runs the sleeping callbacks; in VRAM-only modehandle_sleeping_state(true)callsvram_go_cold()(release device, also the encoder; thenflock(LOCK_UN)) instead of upstream's full unload. - Wake: a request calls
wait_until_no_sleep().handle_sleeping_state(false)callsvram_ensure_warm(): take the token (register on want.lock, then polltoken.lockwithLOCK_EX | LOCK_NBevery 2 ms, aborting whenqueue_tasks.is_stopping()), restore ctx_tgt and ctx_dft, and only then setvram_cold = false. The decode path callsvram_ensure_warm()again (no-op when warm) and throws "server is shutting down" if it returns false. server_res_generatorno longer bypasses the wake path when--sleep-idle-secondsis unset ifLLAMA_SLEEP_VRAM_ONLYis set, because the arbiter can put a server to sleep without idle-sleep.- Queue fixes:
wait_until_no_sleep()loopswhile (sleeping && !stopping)and re-assertsreq_stop_sleepingeach time (withnotify_all); the sleep wait also wakes on a non-empty task queue; the idlewait_foronly leaves the loop on a real task or termination.
Invariants that must hold.
- Resident iff holding
token.lock;vram_cold == falsemeans both.vram_ensure_warm()setsvram_cold = falseonly AFTER the restore. What that order protects today, reasoned from the code: the only thing that gives the token away isvram_should_yield(), called fromshould_sleep()on the loop thread.vram_ensure_warm()also runs only on the loop thread, synchronously: from the wake callback (handle_sleeping_state(false), called insidestart_loop()withmutex_tasksheld) and before decode in the slot update path (outsideyield_to_queue()). Soshould_sleep()cannot run between the token acquire and the end of the restore, whichever order the two lines are in. The measured failure (attempt 1 below: handoffs of 11m20s and 3m57s) happened under the retired doorbell design, where a separate warden thread read the cold flag to decide whether to honour or discard rings, so an earlyfalselet it accept stale rings during the restore. Under the inline want-lock probe the order is defensive: it keeps the flag's meaning exact for any future reader off the loop thread, and keeps a release out of a half-restored model if the restore or the probe ever moves (for example intoyield_to_queue()). Either way, do not change it without an A/B of ordinary handoff latency (fork-validate.sh --arbiter,a-handoff-latency). vram_acquire_for_load()setsvram_cold = falseas soon as it holds the token, before the upload. Nothing reads the flag during the load today (the loop is not running yet andon_should_yieldis only registered invram_share_init(), which sets it again), so the early set is redundant now. It records "holding the token" at the moment it becomes true, because the opposite error is the dangerous one: a model that is warm and holds the token whilevram_coldsays cold never yields (vram_should_yield()returns false for a cold model), which strands every waiter, and its next wake would try to restore weights that were never released.- Release only happens from the idle branch of
start_loop(), on the loop thread, never insideyield_to_queue()or while a slot is generating: a holder finishes its current generation before it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both). - The yield decision reads live kernel state at the point of use. Do not route it through a flag set by another thread (see attempts 3-4 below).
- Every exit from
vram_take_token()drops want.lock, or the holder yields forever. - The probe fd is separate from the waiter fd.
flockcalls retry on EINTR: the server's signal handlers are installed without SA_RESTART.- Restore on wake, before
update_slots(), so KV eviction is safe for SWA checkpoints. - Every model that can be warm on the GPU at the same time must use the same arena. A non-arbiter process (a benchmark, a manually started server) can OOM the warm model. In production the env is set for all of llama-swap, so all its children participate.
Failure modes and incidents.
- Wake-path hang (
958257cd6): with no--sleep-idle-seconds,bypass_sleepskippedwait_until_no_sleep(), so a server slept by the arbiter never woke and requests hung. - Load-time OOM (
b9df854de): weights were uploaded before the arbiter was active; loading gemma while the 4B held VRAM could exceed 8 GB. After the fix the peak was 7771 MB. - The doorbell design (replaced by
878d1338f) was edge-triggered: a waiter wrote<arena>/doorbell/<pid>and the holder's warden thread turned the inotify event into a yield flag. Rings were lost in three windows: while the holder restored (it advertised cold and discarded foreign rings), while the holder loaded (the inotify watch did not exist yet, and the kernel does not queue events for a missing watch), and when a cold waiter consumed a third process's ring. Every loss was permanent:flock(LOCK_EX)had no timeout and production had no idle-sleep. The blocked flock heldmutex_tasks, so the whole server stopped answering while/healthstill returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by the second day). - Failed fix attempt 1 (doorbell design): moving
vram_cold = falsebefore the restore made the holder's warden honour stale rings and hand the token back 2-5 s after taking it without serving. Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. See the first invariant for what this means under the current design. - Failed fix attempt 2: a re-ring loop (
LOCK_NB+ re-ring every ~1 s). Queued rings from an already satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become level-triggered. - Failed attempts 3-4: want.lock was sound (mutual exclusion, kill -9 safety, load-window coverage),
but the warden thread still delivered its decision as a flag.
start_loop()holdsmutex_tasksfromshould_sleep()through the callbacks to the wait, so the warden'srequest_yield()was always admitted after the flag had been consumed, latching a release for the next wake (a server woke and re-slept 138 ms later without serving). It also exposed the upstream lost wakeup inwait_until_no_sleep(). Result: 0.494 s median, 1/60 stranded. The fix deleted the warden andrequest_yield/yield_requestedand probes inline inshould_sleep(). - Measured deviations that are required: the 10 ms idle tick (about 0.2% of one core per warm server); a waiter that polls rather than retrying on a coarse grid (a 200 ms retry grid cost 0.503 s against 0.336 s, because the holder only releases after the waiter asks); the 100 ms post-wake grace (without it a permanent waiter caused a livelock: one request took 23.15 s over 138 sleep/wake cycles).
- Process lesson from attempts 1-2: validate an ordinary handoff latency A/B, not only the pathological case. Two production deploys were burned before that rule existed.
- Lost diagnostic: the old periodic "still waiting for the GPU token after N s" warning is gone; a
stuck token now shows one
VRAM arbiter: waiting for the GPU tokenline and then silence. Check/proc/locks(see the runbook).
Rebase hazards. This is the most fragile area, because it sits inside upstream's server loop.
server_queue::start_loop,wait_until_no_sleep,process_new_tasks,yield_to_queueand the sleeping-callback vector. Upstream refactored sleep (947fd9bb2) and made decode run insideyield_to_queue(22b8e310b); re-check after any such change that a release still cannot run under a live graph.handle_sleeping_state,load_modelordering, the slot update path where decode happens,server_res_generator'sbypass_sleep, renames inserver-context.cpp.- Anything that touches the KV in
update_slots()before decode.
Validation. fork-validate.sh --arbiter with two real servers on a scratch arena:
a-handoffs (30 alternating, 0 stranded, 0 non-200, the server that answered holds token.lock),
a-handoff-latency (fresh server pairs of --build and --ref in the same session, blocks
ref, cand, cand, ref x 16 handoffs; FAIL if the candidate median exceeds the reference median by more
than max(25%, 200 ms); first run 2026-10-25, reb1024 against itself beside production: 1050.5 vs
1082.5 ms), v-vram-* (F4), b-*, c-*, g-kill9-waiter (kill -9 a waiter registered on want.lock while the holder streams:
want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in
F7 and a log scan for asserts and Vulkan errors. In production, /proc/locks must show the warm
model holding WRITE on token.lock.
F7. Server shutdown path
What and why. llama-swap evicts models with SIGTERM, so shutdown happens all the time, often while the arbiter is mid-release or mid-wait. Two bugs made shutdown keep the GPU token and stall every other model.
Where. tools/server/server.cpp: signal_handler, shutdown_pipe, struct shutdown_thread,
sig_thread in llama_server. tools/server/server-context.cpp: server_context::start_loop (calls
vram_share_exit() after the loop returns), server_context::terminate (terminates the result queue
first), vram_share_exit, the destructor's vram_share_shutdown(). tools/server/server-queue.cpp:
server_queue::terminate (sets atomic stopping before taking the mutex), process_new_tasks,
server_response::recv_with_timeout, server_response::terminate,
server_response_reader::next. Commits 11044b35e, 6c480f7e7.
How it works.
- The signal handler only writes the signal number to a self-pipe (async-signal-safe); a dedicated
thread reads it and runs
shutdown_handler. The pipe fds areFD_CLOEXEC. Ifpipe()fails the old direct call is used. The thread object is declared after the server objects, so it is joined (a 0 byte unblocks it) before they are destroyed. A second signal still callsexit(1). terminate()setsstopping(atomic) before lockingmutex_tasks, then stops the result queue so HTTP handlers waiting for results get "server is shutting down" (HTTP 503, or an SSE error event);recv_with_timeoutchecks!runningbefore waiting and returns null.- A token wait polls with
LOCK_NBevery 2 ms and aborts onis_stopping(); the aborted wake stays cold, skips the restore, and the loop exits beforeupdate_slots. - After
start_loop()returns,vram_share_exit()frees all device buffers withllama_context_discard_device()(no host shadows), releases the encoder, and only then unlockstoken.lock. - What llama-swap (v239 on the pod) does when a child exits on its own (crash, OOM kill, external
signal), verified 2026-10-25 with a scratch llama-swap instance and a CPU-only child, for both
kill -9and an external SIGTERM: it logs[WARN] group: running <model> exited: [<model>] upstream exited unexpectedly, does not restart it eagerly (no new process in the 2 s after the kill), and starts a new process on the next request for that model (new PID, health check, HTTP 200). The production log since the last pod restart has no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers the restart pays a full model load (about 26 s for the 35B with-lm none; the first 35B request after the 2026-10-25 deploy took 34 s including load and answer), and under the arbiter the new process takes the token throughvram_acquire_for_load()like any load. A process that wedges instead of exiting (the incidents below) is never restarted:/healthkeeps answering.
Invariants that must hold.
- Nothing that takes
mutex_tasksruns in signal context. - The token is released only after this process's VRAM is free (the next model may start uploading the moment the lock drops).
- A shutting-down server never restores weights.
- SIGTERM in any state exits 0 within about 2 s and frees the token. With the disk prompt cache (F11) the exit also writes each slot's state: about 1 s more per GB of state, after the token is free.
Failure modes and incidents.
- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside
a sleeping callback (under
mutex_tasks, during a release); the handler'sterminate()lockedmutex_tasksagain on the same thread. The process wedged holdingtoken.lock, every other model queued behind it, and only SIGKILL worked. Reproduced 6/6 off production; 8/8 clean with the fix. The first suspicion (the multimodal path) was wrong. gdb on the wedged process showed the main thread infutex(mutex_tasks)with theyield_to_queueworker idle. - Token kept on shutdown: with a request in flight the HTTP handler waited forever for results, main
blocked joining it, and the token was released only in the destructor after that join; with a
token wait in progress the blocking flock held
mutex_tasks; and the old destructor unlocked the token BEFOREdestroy(), so the next model could upload into still-occupied VRAM. - After the fix: off production SIGTERM mid-generation or mid-wait exits in about 0.2 s with exit 0 and frees the token in 5-70 ms; in production about 0.7-1 s.
Rebase hazards. server.cpp's signal setup and llama_server object lifetimes;
server_response and server_response_reader (upstream 4e201388c switched recv_with_timeout to a
fixed deadline: keep the order "check !running, return null, then wait_until(lock, deadline)");
server_context::start_loop/terminate. Router mode (server-models.cpp,
server_models_sse_client::next) only checks should_stop() and can spin at shutdown now that
recv_with_timeout returns null at once; llama-swap does not use router mode.
Validation. fork-validate.sh --arbiter: d-term-in-release (gdb breakpoint on
llama_context_release_device, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and
the token), e-term-waiting (exit 0 within 1.5 s, never took the token), f-term-streaming
(exit 0, SSE error event, token free within 2.5 s), h-shutdown-warm, h-shutdown-cold. Last run
(2026-10-25, 2fc9f9c35, d by log polling): 750 ms, 225 ms, 126 ms (token free in 23 ms),
108/159/159 ms. Post-deploy: a forced llama-swap eviction
must log VRAM arbiter: loop stopped, device memory freed, releasing the GPU token and exit by itself.
F8. Cold-teardown guards
What and why. A cold model has null buffers in its ctxs_bufs and, with evict_kv, a null
scheduler. Upstream code never had to consider either state.
Where. src/llama-context.cpp: llama_context::~llama_context (sched && in the
compute-buffer size check), llama_context::memory_breakdown (if (sched) around the compute part;
upstream's if (moe_cache) breakdown block stays outside it). Null-buffer skips in
llama_model::memory_breakdown, llama_kv_cache::{clear, memory_breakdown, total_size},
llama_memory_recurrent::{clear, memory_breakdown, total_size}. Commits 8d80f0673, ad27d4921, 53f907771.
How it works. Guards at the call sites. The asserts in ggml (GGML_ASSERT(buffer),
GGML_ASSERT(sched)) are correct and stay; every other caller reserves the scheduler first.
Invariants that must hold. A server shut down while cold exits 0. Any new upstream code that
iterates ctxs_bufs or calls into sched on a path reachable while cold (shutdown, memory
breakdown, stats) needs the same guard.
Failure modes and incidents.
GGML_ASSERT(buffer)on shutdown of a cold model (seen 7 times in production logs) from memory-breakdown/size/clear on released buffers.GGML_ASSERT(sched)inmemory_breakdownfor a coldevict_kvmodel being torn down.GGML_ASSERT(sched)in the destructor's size check: every cold shutdown exited 134 instead of 0, so under llama-swap an ordinary stop looked like a crashed child, left a backtrace in the log, and the rest of the destructor never ran. Repro: arbiter env on, one request, force a yield withflock -s <arena>/want.lock -c 'sleep 6', then SIGINT while cold.
Rebase hazards. New iteration over ctxs_bufs, new sched users in teardown or breakdown
paths. Watch item: llama_kv_cache::update() takes lctx->get_sched(), which can be null; it is
safe today only because its single caller runs after sched_reserve().
Validation. fork-validate.sh --arbiter: h-shutdown-cold (exit 0 while the other server holds
the token) and log-scan; --moecache shutdown.
F9. Benchmark harness and validation script
What and why. The measurement method on this box had to be rediscovered several times (section 6). The harness encodes it; the validation script encodes the checks that each past incident required, so a rebase can be validated by someone with no context.
Where. scripts/rx580-bench/ (run.sh workstation wrapper; bench.sh pod orchestrator;
ppbench.py corpus prefill through llama-server; lbsweep.sh llama-bench sweep; summarize.py;
README.md with methodology and reference values). scripts/fork-validate.sh. Commits 4c4486f8e,
54a7f534b, 149a8cb0a, 62d442e7b, 2512290f9, 80e662a2a, ceab1a91f, e5fc474b8.
How it works.
rx580-bench: stops llama-swap, waits for an idle GPU, runs fixed slices ofpan-tadeusz.txt(4096/16384/32768 tokens, byte-identical across builds) through/completionwithcache_prompt: false, plus a llama-bench sweep (prefill pp2048/pp8192 at depth 0, decode tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and rounds; restarts llama-swap from an EXIT trap. Results append to/root/bench/results.txt.fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL] [--moe-exact MODEL] [--diskcache]: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under /tmp, a scratch arena (refuses/dev/shm/llama-vram*), ports 8091-8093, unsets inheritedGGML_*/LLAMA_*env, kills only PIDs it started (PID-reuse safe). Default with no check flag is--ops --concat --arbiter.--perf-report TSVre-aggregates a savedperf.tsvoffline. See the header of the script for what each check proves.- Which phase needs what:
--ops --concatrun beside production;--arbitertoo only if 1536 MiB of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs in the swapdown window, about 2 minutes);--perf,--moe-exactand--moecacheneed llama-swap stopped (the first two refuse otherwise;--moecachedoes not check, but loads a MoE model and must not share the card either).--diskcacheruns beside production (CPU only); its GPU leg SKIPs below 1536 MiB of free VRAM.
Reference values (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, --perf on 2026-10-24,
production 6c480f7e7 and the previous production 5751c977b): pp2048 about 274-286 t/s,
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. On 2026-10-25
(medians, 6c480f7e7 / 2fc9f9c35): pp2048 274.5/273.8, d16384 200.5/200.4, d32768 157.6/157.6,
tg256@d4096 16.8/16.7. The corpus
harness and older numbers are in scripts/rx580-bench/README.md. A fresh run far outside these
means something else is wrong (another GPU user, thermal state, a bad build) before it means a
regression or a win.
Invariants that must hold. Never benchmark with llama-swap running (--perf and --moe-exact
refuse, and any other llama process blocks them). --perf gates prefill at -5% per depth on the
MEDIAN of all repetition samples of each build (never a mean: CPU contention produces single
catastrophic samples), and peak VRAM at 95% of the card; decode is reported, not gated (CPU-bound and
noisy here).
Failure modes and incidents. Upstream #28334 removed --mmap/--no-mmap/--mlock and
llama-bench's -mmp: the server refused to start and llama-bench silently produced no rows until
54a7f534b switched to -lm none. The perf and MoE-cache checks leave tens of GB of model files in page
cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024
deploy (see the runbook). That pod restart also emptied /tmp (every earlier fork-validate.*
scratch dir with its raw samples is gone) and removed gdb, which is not in the pod image; --arbiter
case d-term-in-release then falls back to polling the log.
Rebase hazards. CLI flag renames in common/arg.cpp and tools/llama-bench; log line texts the
script greps for (server entering sleeping state, transpose fast path, transposed ne0=,
no longer waiting for the GPU token, released .* MiB of MoE cache, MoE cache size =,
llama_moe_cache: .* hits =, sched_copy_experts: ids readback skipped, N/M tests passed,
Backend Vulkan0: OK, disk cache: restored, disk cache: copied/saved, prompts of this model,
size limit, removing, disk cache: failed to write); llama-bench's jsonl fields (n_prompt, n_gen, n_depth, samples_ts) and
markdown layout; the GGUF keys <arch>.expert_count / <arch>.expert_used_count; the DRM fdinfo
keys drm-client-id / drm-memory-vram.
Validation. Known gaps of the script: peak VRAM is recorded per arm, not per test. The first
candidate arm has now twice shown an unexplained spike of about 1.6 GiB over the other arms
(2026-10-25: ref 5476, cand 7062, cand 5767, ref 5476 MiB); the cause is open (per-test peaks would
locate it), and it stays far below the 95% gate. The SIGTERM-in-release breakpoint is at function
entry, not after the host-shadow capture. The v-vram-release SKIP path (production moving during
the measurement) has not been exercised. --moe-exact and the median-gated --perf first ran on the
GPU on 2026-10-25 (runbook section 6).
F10. Page-cache drop after a copying load
What and why. With -lm none the loader read()s the GGUF into its own buffers, but the kernel
keeps the file in page cache afterwards, charged to the pod's cgroup: about 20 GB after a 35B load.
The pod is BestEffort on a shared node, and on 2026-10-24 the Talos OOM controller killed it right
after a model load under exactly that pressure (MemFree < 1 GB, file cache about 20 GB). Once the
data is copied those pages are pure waste, so the loader advises the kernel to drop them.
Where. src/llama-mmap.h/.cpp: llama_file::drop_cache(offset, len).
src/llama-model-loader.cpp, llama_model_loader::load_all_data: the per-tensor drop at the end
of the copy (non-mapping) branch and the whole-file drop in the final cleanup
(size_done >= size_data). tools/mtmd/clip.cpp, clip_model_loader::load_tensors: whole-file drop
after the tensor reads. Commit cce0ac1f0.
How it works.
drop_cache()isposix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED)under#ifdef __linux__(no-op elsewhere);len0 means to the end of the file. Errors are logged at debug level only.load_all_data()drops each tensor's range right after it is read (all three copy paths: straight into a host buffer, the async pinned-upload chunks, the staging buffer), so the cache never grows past about one tensor during the load. The final cleanup drops every file whole, which catches the GGUF metadata and the partial pages at tensor edges that a range drop leaves.- Both run only when
mappings.empty(), i.e. the loader holds no mapping of any of its files: notmmap,mmap+mlockorauto, and no lazy tensors (-lzm; with lazy tensorsinit_mappings()maps every file even under-lm none). Split GGUFs are covered (every entry offiles). - clip.cpp always copies the encoder through an ifstream (no mmap path exists), so it opens the
file again after the reads and drops it whole, unless
no_alloc. - Choice: per-tensor plus a final drop rather than a post-load drop only, because the per-tensor
drop also removes the peak during the load (the incident window: anon and file cache grow together)
for three extra lines. It costs about 0.27 s more than a post-load-only drop on the test model
below. Direct IO (
-lm dio, upstream's) was measured as the alternative and is not better (section 4).
Invariants that must hold.
- Never drop for a file this loader maps: a mapped page would just be faulted back in (the kernel also skips pages that are mapped into a process, so a violation costs refaults, not correctness).
- Drop only after the bytes are in our own buffers. Nothing in the fork re-reads the GGUF after load: the VRAM release/restore (F4) uses host shadows, not the file.
- The drop is a hint: a failure must not fail the load.
Failure modes and incidents. None. Deployed 2026-10-25 (2fc9f9c35); on a 35B GPU load in
production the node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB. Costs to know
about:
- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a 5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few seconds on a 26 s load.
- A reload of the same model can no longer hit a warm page cache (a warm-cache reload of the test model took 4.6 s against 5.9 s cold). Under llama-swap a reload happens only after an eviction or a crash.
TENSOR_DUPLICATEDtensors read the same range twice; the second read now comes from disk (small tensors only).
Rebase hazards. The structure of load_all_data() (the from_mapping branch and its three copy
paths, the final cleanup block), init_mappings() creating mappings for lazy tensors, new load modes
in llama_load_mode (any new mode that maps must leave mappings non-empty), and the tensor-read
loop in clip.cpp load_tensors.
Validation. Not in fork-validate.sh. By hand on the pod with a scratch llama-server on a
scratch port, CPU only (-dev none -ngl 0 --no-mmproj-offload, no VRAM), cold cache (evict the test
files with python os.posix_fadvise(..., DONTNEED) first): page residency of the model and mmproj
files by mincore before and after the load (must be 0 after with -lm none, unchanged with
-lm mmap), the pod cgroup's file/anon in
/sys/fs/cgroup/kubepods/besteffort/pod<uid>/memory.stat, host MemFree, load time, and a greedy
request. First run (2026-10-25, gemma-4-E2B UD-Q4_K_XL 2.5 GiB + mmproj BF16 0.9 GiB, production
6c480f7e7 and the candidate with the drops disabled as controls, 4 rotated rounds):
| drops off (control) | post-load only | per-tensor + final (committed) | |
|---|---|---|---|
| model / mmproj resident after load | 2431-2444 / 940 MiB | 0 / 0 | 0 / 0 |
| peak model residency during load | 2.1-2.5 GiB | 2.1-2.4 GiB | 1.2-1.5 GiB (largest tensor) |
pod cgroup file after load |
+3.37 GiB | +0 | +0 |
| host MemFree after load | 0.6 GB | 2.5 GB | 2.5-2.8 GB |
| load time (median, production 5.85 s) | 5.87 s | 6.56 s | 6.82 s |
Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it counts clean page cache as
available. Greedy output byte-identical in all arms. -lm mmap control: the model file stays fully
resident and mapped (2498 MiB, file_mapped +2.5 GiB) after load and after a request, and only the
copied mmproj is dropped. -lzm on control: no drop (the file is mapped), 1252 MiB of copied
tensors stay cached. The GPU upload path and a 35B load were first exercised by the 2026-10-25
deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B
request took 34 s including load and answer).
F11. Disk prompt cache (save slots on shutdown, restore after a restart)
What and why. Only one large model fits in RAM. Switching models in OpenWebUI makes llama-swap
SIGTERM the previous one, and switching back re-prefills the whole conversation, because the RAM
prompt cache died with the process (F4's reason for the arbiter). A 30k-token chat costs about 150 s
on the 35B at about 200 t/s. With --cache-disk-path DIR, each slot with a prompt is saved to DIR on
shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the
file, so only the suffix is processed. Measured on Qwen3.5-0.8B CPU only: a 12k-token turn 2 went
from a 72 s prefill to 1.2 s, of which the read took 307 ms and the restore 18 ms. Off by default.
Upstream PR #28092 (--cache-disk) was the reference for ideas, not ported (1700 lines). It
livelocked the kernel on AMD + Vulkan by DMA-ing GPU state into a MAP_SHARED mmap'd file, and it never
saved the slot in use on shutdown, which is the main case here.
Where.
tools/server/server-task.h/.cpp:server_prompt_disk_entry,server_prompt_disk_cache(n_reuse,find,has,load,save,remove); file IO helpersdisk_cache_io,disk_cache_open.tools/server/server-context.cpp:DISK_CACHE_MIN_TOKENS,disk_cache_key,disk_cache_collect,disk_cache_flush,disk_cache_load; the init inload_modelafter the RAM cache; thedisk_cache_loadcall at the end ofget_available_slot;server_context::start_loop(collect,vram_share_exit, flush).common/arg.cpp,common/common.h:--cache-disk-path(cache_disk_path),--cache-disk-size(cache_disk_mib, default 4096, 0 = no limit).- Library (cold save):
llama_context_restore_memory_host()(include/llama.h,llama_context::restore_memory_host);llama_memory_i::restore_device_buffers(bool to_host)inllama-memory.h,llama-kv-cache,llama-memory-recurrent,-hybrid,-kv-cache-iswa; the helperllama_memory_wrap_host()inllama-memory.cpp. - Commits
dd851fd0c,2874dcb10,2c6c517ba(andf24c3582c, F4).
How it works.
- Save, after
start_loop()'s loop returns. Firstdisk_cache_collect()copies the state of every slot with at least 256 text tokens to host memory (llama_state_seq_get_data_ext, flags NONE) while the device buffers still exist. Thenvram_share_exit()frees the VRAM and releases the GPU token as before. Thendisk_cache_flush()writes the files. The token release therefore waits only for the device-to-host copy, never for disk IO. - A slot is skipped when its KV does not cover its token list (
seq_pos_max + 1 != n_tokens: the last batch was not decoded, a shutdown during a token wait), when it holds media, and when the same token list is already on disk. - Cold server: with
LLAMA_SLEEP_EVICT_KVthe KV and recurrent state live only in host shadows (F4) and their tensors have no buffer.llama_context_restore_memory_host()wraps each shadow as a CPU_Mapped buffer (ggml_backend_cpu_buffer_from_ptr) and points the tensors into it, so there is no copy, no allocation and no device use. It setsdevice_discarded. Without evict_kv the KV is still in VRAM and is read directly. - File,
<dir>/<key hash>-<token hash>.lpcd: a 48-byte header (magicLPCD, version, key hash, n_tokens, n_ckpt, payload size, payload hash), then 8-byte padded sections: tokens, checkpoint table, state size, state, the state of each checkpoint. The hash is FNV-1a over the 64-bit words of the payload. Write: plain fwrite to.tmp, fdatasync,POSIX_FADV_DONTNEED, rename. Read: plain fread, then DONTNEED. There is no mmap anywhere. - Index: at startup, from the headers, tokens and checkpoint tables of the files of this key. A bad
magic, a file size that differs from the header, or an inconsistent table means the file is removed.
Other keys and other versions are counted and left alone. A
.tmpolder than 1 h is removed. - Key: the build commit, the model path + size + mtime, LoRA adapters and control vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, K/V types, flash-attn type, and the RoPE/YaRN parameters.
- Restore: at the end of
get_available_slot(), after the RAM cache, only while warm, withcache_prompton and no media tokens. The file that reuses the most of the new prompt is loaded if it reuses at least 256 more tokens than the slot holds. The slot's prompt goes to the RAM cache first. Then the slot is cleared and the state is set; a failure clears the slot and removes the file. - Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt,
or, for a recurrent or SWA model, a saved checkpoint at or below the common prefix can be used. A
plain-KV model can be cut back to any prefix. Exact prefix alone would almost never fire: the Qwen
template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is not a
prefix of the next OpenWebUI turn. Checkpoints are exact snapshots, and
update_slots()rolls back to one exactly as it does for the RAM cache; the recurrent state itself is never cut back. Restored checkpoints getid_task = -1, so one that fails to load means a re-prefill, not an abort. - Size limit: before a write, the least recently used
*.lpcdfiles of every model in DIR are removed until the new file fits. A restore updates the file's mtime. Entries the new prompt extends are removed after the write and do not count against the limit. A write also needs its size + 1 GiB free on the file system, so the cache never fills the PVC. - Disabled with speculative decoding, with a warning: the draft state is not saved.
Invariants that must hold.
- Never restore while cold or without the GPU token (
disk_cache_loadreturns whenvram_cold).get_available_slot()runs on the loop thread after the wake, like the RAM cache'sprompt_save. - Copy the state before
vram_share_exit()(it discards the device buffers) and write after it (the token is not held for disk IO). - Never serve a file that fails the size or hash check. Never mmap the file: a DMA into a shared mapping is what livelocked upstream's version on AMD.
- A write is atomic (tmp + rename), and any IO error leaves the server serving and the exit clean.
- The key must cover everything that changes the state layout or the KV values. Add a new
state-changing parameter to
disk_cache_key(). llama_context_restore_memory_host()is for exit only; the wrapped buffers point into the shadows, which must outlive them (both are members of the same memory object).
Failure modes and incidents.
- Production incident (2026-10-25, deploy of
cb48bf224with--cache-disk-path, rolled back): the first request to the 35B aborted indisk_cache_load().server_tokens::get_tokens()hasGGML_ASSERT(!has_mtmd), andhas_mtmdis true whenever an mmproj is loaded, even for text-only prompts. Production's 35B and gemma-26B always load one (-hfpulls it; there is no--no-mmproj).disk_cache_collect()andsave()had the same call, so the shutdown save would have aborted too. Every test had used--no-mmproj(q35.pyand--diskcache), which hid it. Fixed: the feature takes the token list fromget_text_tokens(), and a prompt is text-only when that list is as long assize()(disk_cache_text). A text-only prompt is cached with an mmproj loaded. A prompt with media is never saved or restored. When the slot itself holds media, its reuse estimate isget_common_prefix(). Lesson (runbook section 6): test with production's exact command line. - The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself (CPU_REPACK, F4), not in the save.
- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full
KV allocation at exit (about 2.7 GB for the 35B at ctx 131072) on a node that has had an
OOM-controller kill. Replaced by the in-place wrap. The 32-byte alignment that
cpu_buffer_from_ptrasserts is met by rounding the base down; those leading bytes are never read. - Writes to
/tmp(node overlay) stalled once: 5.8 s for 64.5 MiB, against 171 ms normally. - Costs to know about. llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a child still there 30 s after SIGTERM (measured 2026-10-25 with a scratch
llama-swap and dummy upstreams: term to exit 4.0 s, next start 2 ms later; a child that ignored SIGTERM
was replaced at +30.0 s). So the exit write adds directly to a model switch: about 1 s per GB on the
/rootPVC (0.66 GB/s with fdatasync). A save cut off by the SIGKILL leaves only a.tmp. - Only slots are saved. A conversation that sits only in the RAM cache when the process stops is lost.
- Every deploy empties the cache (the key holds the build commit), and so does any change to the model file or the flags in the key.
Rebase hazards.
server_context::start_loopandvram_share_exitordering (F7);get_available_slotand the RAM cache (prompt_save,prompt_load,server_prompt_cache_state);server_promptandcommon_prompt_checkpointfields (new fields must be saved, or the format version bumped).- The checkpoint rollback in
update_slots(pos_min_thold,TAG_PROMPT_LOGITS), which decides how much of a restored file is reused. - The llama state format (
state_write/state_readof the memory types). The key holds the build commit, so a format change cannot load an old file; it only empties the cache. llama_memory_i::restore_device_buffersand the shadow layout of F4 (device_buffer_shadow):llama_memory_wrap_hostrelies on the compact iteration-order shadow.- Flag names in
common/arg.cpp.
Validation. fork-validate.sh --diskcache (2c6c517ba), beside production. The model is the small
hybrid, CPU only (-dev none -ngl 0), with chat requests and greedy decoding, and a reference server
that never restarts. Cases: restore, restore-arbiter and restore-cold (turn 1, SIGTERM, restart,
turn 2 restored; prompt_n equal to the reference; identical answer), no-prefix, key-ctx,
key-ctk, key-model, corrupt (truncated and byte-flipped), lru, write-error (RLIMIT_FSIZE,
SIGXFSZ ignored), restore-mmproj (the same restore with the 0.8B's mmproj loaded),
mmproj-image, log-scan, and gpu-warm/gpu-cold when 1536 MiB of VRAM is free.
mmproj-image sends an image (answered "red", not restored), then the text chat (restored, identical),
then an image again, and SIGTERMs with media in the slot (no state copied, exit 0). First run
(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219
MiB free). After the mmproj fix (beside production, 5.3 GB VRAM free): 15 PASS, 0 FAIL, 0 SKIP.
| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill |
|---|---|---|---|---|---|---|
| 573-token chat | 64.5 MiB, 2 ckpt | 18 ms | 171 ms | 314 ms (token free 56 ms vs 28) | 166 + 3 ms | prompt_n 58 of 596, 462 vs 2913 ms |
| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms |
| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s |
Swapdown window (2026-10-25, started by the owner, /root/arms/diskcache-window.sh):
--diskcache --arbiter 29 PASS, 0 FAIL, GPU legs included.
35B (q35.py, the production 35B flags but WITH --no-mmproj, unlike production, which hid the
incident above; --cache-disk-path on a scratch dir, arbiter env, an 18760-token pan-tadeusz chat;
log /root/arms/diskcache-test/q35.log). q35.py now uses the exact production entry (-hf, so
the mmproj loads, nothing extra). That run (/root/arms/diskcache-window3.sh, about 8 minutes under
swapdown, which also covers the same-reuse-point identity and an image request on the 35B) was run
2026-11-21 on adc2aa3ec (log /root/arms/diskcache-test/q35-mm.log) and passed all legs: 18.8k-token
chat, prefill 84.6 s -> 4.7 s for turn 2 after the restart with identical output; save 557 MB in
1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image answered and not saved. The
table below is the earlier --no-mmproj run:
| leg | result |
|---|---|
| turn 1 prefill (reference and candidate) | 18760 tokens in 83.4 s (224.8 t/s), outputs identical |
| warm save on SIGTERM (a) | 556.5 MiB (2 checkpoints), copy 323 ms, write 960 ms; token free in 328 ms (6 ms without the cache), exit 1952 ms (607 ms without), peak RSS +430 MiB |
| restore after the restart (b) | read 860 ms, restore 114 ms; turn 2 prompt_n 27 of 18850 in 3.0 s (the reference that never restarted: 27 in 2.0 s); output identical |
| cold save on SIGTERM (b, KV in the host shadow) | 746.7 MiB (5 checkpoints), copy 310 ms, write 1403 ms, exit 2928 ms, peak RSS +432 MiB: the in-place wrap adds no copy |
| restore of the cold save (c, turn 2 sent again) | read 1189 ms, restore 75 ms, prompt_n 4; output differs from the reference's turn 2 (see below) |
Two anomalies in leg c, both explained:
- Output != reference: c sent turn 2 again, so the saved list (turn 2 + 63 generated tokens) extends
the new prompt.
update_slots()then rolls back to the checkpoint at the prompt end minus 4 and processes 4 tokens, while the reference processed turn 2 from turn 1's state (27 tokens). These are different reuse points, not the same computation. The controlled test on Qwen3.5-0.8B (/root/arms/diskcache-test/cmp.py) compares the same reuse point. Server X never restarts and runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the GPU (-ngl 99); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU to itself:/root/arms/diskcache-window3.sh(production command line, reference "turn 2 again" against leg c; about 8 minutes under swapdown). It ran 2026-11-21 and passed (same-reuse-point identity: identical output), so the difference above is the reuse point. It supersedesdiskcache-window2.sh, whosecmp.pyruns with--no-mmproj. - Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
The driver's fake waiter was
flock -s want.lock sleep 120; killing flock leftsleepholding want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right after its load and again right after its request (log 0.37.50). That ordinary arbiter release recaptures the full KV allocation as a host shadow, about 2.6 GB at ctx 131072 (F4), and the SIGTERM arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The driver now holds the lock in the process it kills, likefork-validate.shdoes.
Configuration (applied in production 2026-11-21): --cache-disk-path /root/prompt-cache --cache-disk-size 4096 appended to the moe_offload_args and moe_offload_gemma macros in
/root/config.yaml (backup /root/config.yaml.bak-diskcache-20261121). The directory is created on
first start. Verified in production: a 35B 4.6k-token chat was saved on eviction (278.6 MiB, copy
126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms), and turn 2 prefilled
only 32 tokens; a gemma image request with the flags on works and is not cached. /root had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
applies. Models only pick up the flags when they restart.
F12. Temporary upstream carries (open upstream PRs, drop when merged)
What and why. Open upstream PRs carried unchanged until upstream merges them. Each carry is one
commit whose subject starts with upstream-carry #<PR>:; the author is the PR author, and the body
names the PR URL and the PR head that was carried. List them with
git log --oneline --grep '^upstream-carry' origin/master..HEAD. They are not fork features: never
edit them in place, re-fetch the PR instead.
Carried since 2026-11-22 (both on upstream e6d624b9a, for Inkling-2-Mini, section 5):
- #32735
vulkan : support non-contiguous src in SCALE(head0ea4ffd00): the Vulkan backend runs SCALE on a non-contiguous source instead of rejecting it. - #32740
ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan(head808fd1c4e, four PR commits squashed): Inkling's per-query bias table ([n_bins, n_head, n_tokens]plus an I32 bin index per key) is applied inside the flash-attention kernel. Before it, the graph gathered the bias into a dense per-head mask (n_kv x n_tokens x n_head, F32, added to the KQ mask and cast to F16) and passed that to FA, and the Inkling layers could not use thekv_rowscell lists (dense masks only). Vulkan binds the table and the indices in the two mask_opt descriptor slots, souse_mask_optgets!bias(F2). Vulkan supports it only for F16 K/V, head size 64 or 128, with subgroup shuffle and vote (RX 580 RADV has both; Inkling is f16 KV, head size 128).
State after the 2026-11-22 rebase onto 5bacb7c86 (branch fable5/vram-ondemand-rebase-20261122):
- #32735: merged upstream as
8a97fa98a, patch identical to the carried head. Carry dropped. - #32740: still open, but the PR was split on maintainer request (one backend per PR). Its head
8c6f0c0ca(8 commits) is now only the GGML contract:ggml_flash_attn_ext_add_indexed_bias, a CPU reference (the CPU backend now accepts indexed bias) and test-backend-ops cases (the standalonetest-flash-attn-indexed-biasis gone).be0bf1fbdremoved the Vulkan support (supports_op returns false forsrc[5]or indexed bias),9cd195304moved the CUDA/HIP kernels and the Inkling graph changes (llama-graph.cpp,llama-kv-cache.cpp,inkling2.cpp) out of it. No open PR carries the Vulkan part (Metal is #32808). The contract also changed in wording: the bias is added "in F32" (before: "before its F16 rounding"); the API and thesrc[6]/src[7]layout are the same. Decision: the carry of head808fd1c4eis kept unchanged. Carrying the new head would remove the Vulkan kernel and the graph use, so Inkling would fall back to the dense mask: compute buffer about 15.8 GiB at ctx 65536 instead of 7 GiB, and the k=4 placement of F13 would no longer fit. Owner: decide whether to keep carrying808fd1c4euntil a Vulkan follow-up PR appears, or to port the Vulkan part onto the new contract. Once the contract merges upstream, the carry will conflict with it inggml.h,ggml.c,ggml-cpu/ops.cppand every backend's supports_op, and has to be reduced to the Vulkan and graph parts by hand. - #32777 (new carry, owner's wish):
fit : prioritize looped MoE layers in device memory, head01945dcb0, one commit, applied cleanly. See "#32777 and--fitfor Inkling" below.
#32777 and --fit for Inkling. llama_model_layer_visits() counts how many slots share each
layer's tensors (by pointer: ffn_up_exps, else the next present weight), so inkling2_loop gets 3
for the loop layers 6..17, 1 for the others, 0 for the 24 repeat slots. --fit then (a) probes
"only the non-loop experts on the CPU" against "all experts on the CPU" and keeps the loop experts
on the GPU if that leaves more free memory, and (b) when it moves experts back to the GPU layer by
layer, ranks the layers by visit count, so loop layers go first (the order of the F13 placement
guide).
- F13 interaction, checked by reading: fit's first probe, the only one that computes the visits,
runs with fit's own defaults (fit refuses to run with a user
-ngl,-ot,-ncmoeor-ts), so every weight is in a device (dummy) buffer, no F13 view exists, and the pointer sharing is intact. Views appear only in the later probes with CPU experts, which do not recompute the visits. If the visits were ever computed with the loop experts on the host, the views would make every slot count 1 and the loop ranking would silently turn off (fix then: key ont->view_src ? t->view_src : t); not reachable through--fittoday, so not changed. - Memory estimate: fit does not model the per-pass copies; it measures every candidate placement
with a no_alloc load and a graph reserve (
common_get_device_memory_data_impl). The reserve sees the F13 views, so its compute buffer is the F13 one (about 1.4 GiB at ctx 16384, not 7 GiB). The estimate therefore already accounts for F13, and the PR's comment ("CPU-offloading repeated core layers can inflate the compute buffer") describes the build without views; with views step (a) will usually pick "all experts on the CPU" as the base and let step (b) add loop layers back. Not changed. - The mmproj: llama-server adds the mmproj estimate (about 0.8 GiB) to the fit margin of its
device before fitting, on top of
--fit-target(default 1024 MiB). F5 releases the backbone while the encoder runs, so that headroom is conservative. - Fit plans against the VRAM free when it runs. Under llama-swap it runs inside
common_init_from_params, aftervram_acquire_for_load()took the GPU token, so the previous holder has released its VRAM; a process outside the arbiter still counts against it.
How to use it for Inkling (production command line inkling_args in /root/config.yaml):
- Remove
-ngl 99and the-ot blk[.](...)[.]ffn_.*_exps=CPUpattern (either makes fit abort with "already set by user"), keep--ctx-size 65536(a user context is not changed by fit), and add--fit on(the default, but explicit) and-lv 4(fit's log and the memory breakdown are info level). Optionally--fit-target 512to give back part of the 1024 MiB margin (the mmproj estimate is still added). - Dry run without serving:
build/bin/llama-fit-paramswith the same arguments prints the chosen-c ... -ngl ... -ot "..."line on stdout. Compare its-otlist with production's k=4 (CPU experts for layers 0-5 and 10-23, GPU for 6-9). - A/B in the window (GPU to itself): the k=4
-otline against the--fitline, with F13's drivers (/root/arms/overnight/sweep.shfor llama-bench pp2048 @ d8192/d16384 and tg256 @ d4096; give llama-bench the explicit-otthat fit-params printed rather than its own-fitt, which is off by default and fits for the bench context, not 65536), plus the process VRAM/GTT after load and the-lv 4breakdown (the free column must keep about 0.8 GiB for the mmproj).
Measured on the RX 580 (2026-11-21, the previous PR head: 1fa93f902, d29f9d489, ff91e4f84,
without 808fd1c4e; base 3bf6b55b3 vs base + both PRs; logs /root/arms/pr-out/ on the pod):
- Inkling pp2048 at ub1024: 91.6 -> 110.5 t/s. pp2048 at depth 16384 (ub512): 48.4 -> 73.0 t/s.
- Inkling compute buffers at ctx 131072, ub512: Vulkan0 12918 -> 7173 MiB, Vulkan_Host 12729 -> 398 MiB. At ctx 65536 the base build already reserved 15866 MiB of Vulkan0 compute (more than the card; the driver places the excess in GTT). With the carry: 6981 MiB.
- No regression elsewhere: Qwen 35B pp2048 286.5 -> 287.9, pp2048 at 16k 208.0 -> 208.5; gemma-26B 249.4 both.
- test-backend-ops on Vulkan0: SCALE 15/15, FLASH_ATTN_EXT 5693/5693.
808fd1c4e(the PR head since 2026-11-22) changes the table layout from[n_head, n_bins, n_tokens]to[n_bins, n_head, n_tokens], which removes one permute+cont from the Inkling graph.
Measured with the carried head (2026-11-22, candidate 3621025e8 vs production 3bf6b55b3,
production's Inkling command line with the nothink sampling, GPU to itself; driver
/root/arms/carry1122-ink-direct.py, logs /root/arms/carry1122-p2.log, -p3.log):
| build, args | 8.2K-token prefill | 13.9K | 28.4K | process VRAM / GTT after load |
|---|---|---|---|---|
| production, ub512, ctx 65536 | 32.7 t/s | 31.6 | 29.5 | 7209 / 28113 MiB |
| carry, ub1024, ctx 131072 | 43.7 | 42.5 | 42.5 | 7983 / 21835 MiB |
| carry, ub1024, ctx 65536 | 46.8 | 46.9 | 46.0 | 7368 / 19629 MiB |
- Device memory wanted (
-lv 4breakdown): production ub512/65536 19424 MiB (compute 15866); carry ub1024/131072 13388 MiB (model 1266, context 4419, compute 7702), host compute 798 MiB. Everything above the 8 GB card lands in GTT (system RAM over PCIe): about 5.2 GB at 131072. That costs about 8% prefill against 65536, so production runs ub1024 with ctx 65536 (section 5) until the loop-layer experts move into VRAM, which needs that headroom. - Answers: a ~160-token first request on a fresh server (upstream #32778, all-'?' output) was
correct on every build. Pan Tadeusz fragments of 8.2K and 13.9K tokens gave coherent, correct
summaries on every build (upstream #32755 did not reproduce). The nothink entry
(
--reasoning-effort none --reasoning-budget 0) returns repeated<|message_model|><|content_thinking|><|end_message|>as content, with no tool call, for an image request, a tool call and a 28.4K-token weather log ("which station is in Record 5?"). This happens on every build, production3bf6b55b3with ub512 included, with and withoutGGML_OP_OFFLOAD_MIN_BATCH=64, so the carry did not cause it. The think entry gets all three right on both builds (image "Red",get_weather({"city":"Warsaw"}), 28.4K log "65" at 45.9 t/s in production). Short text prompts work on the nothink entry. - Nothink fix (2026-11-22, config only): the cause was
--reasoning-budget 0. With effortnoneand no budget the model still writes a short reasoning block (100-550 characters) and then answers; a budget of 0 cut that block off at once and the model looped on the chat-structure tokens. The macro is now--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --reasoning-effort none. Verified on production3621025e8(pod driver/root/arms/overnight/nothink.py, logsnothink1.log,nothink2.logthere): short question, Polish poem, image ("Red"), tool call (get_weather({"city":"Warsaw"})) and a 29.4K-token Pan Tadeusz prompt (correct, 46.0 t/s) all OK. Effortminimaland a budget of 256 also work; the budget of 0 fails on all but the short question.
Rebase hazards.
- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry
commit; still open -> compare
git diff <carried-head> <current PR head>and re-carry the current head if it moved; closed unmerged -> decide with the owner (Inkling's prefill and memory depend on #32740, section 5). - #32740 conflicts with F2 on
use_mask_optinggml_vk_flash_attn. The resolution: upstream's!use_sparse, the PR's!bias &&, the fork's GCNHSK >= 256 || HSV >= 256and its comment. When the PR is merged and the carry dropped, the same line conflicts again when F2 is re-applied on top; resolve it the same way. - #32740 changes
llama_kv_cache::set_input_relative_pos(newrowsargument) and the supports_op of every backend; a fork change in those places must follow it.
Validation. test-backend-ops -b Vulkan0 -o SCALE and -o FLASH_ATTN_EXT (check the case
count), and the PR's own test tests/test-flash-attn-indexed-bias (build target
test-flash-attn-indexed-bias; it runs every device, prints one line per case it ran and skips
unsupported cases silently, so count the Vulkan0: lines: 16 f16 cases on the RX 580, max error
about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rejects indexed bias
(the graph then uses the dense mask), so it prints no CPU lines).
Then an Inkling request through llama-swap.
F13. Per-pass views of host weights in looped layers (inkling2_loop)
Branch fable5/inkling-loop-views (on top of the F12 carries), fast-forwarded into the production
branch on 2026-11-22 (deploy and results below).
What and why. inkling2_loop (Inkling-2-Mini) runs the core layers 6..17 three times; the 48
cache slots are 24 physical layers plus 2 x 12 repeats. post_arch_tensors makes every repeat slot
share the tensors of its physical layer (layers[il] = layers[phys]). With the experts in host
memory (--n-cpu-moe 24, -ot exps=CPU), ggml_backend_sched makes one device copy per (host
tensor, backend) per graph, hashed by tensor pointer, at the first split that reads it, and keeps it
allocated until the last reader (ggml_backend_sched_split_graph, pass 5 and the graph_copy input
nodes). Two consequences:
- Memory: the expert copies of all 12 loop layers are alive at the same time (about 547 MiB per layer on UD-Q4_K_XL, 6.5 GiB in total). That is most of the ~7 GiB Vulkan0 compute buffer in the F12 measurements.
- Correctness (upstream issue #32755, "GGGG" output): the copy callback (
sched_copy_experts, F1) runs once, for the first reader. When it reads the routing ids back (ubatches below the F1 skip threshold of4 * n_expertids, and at or above the Vulkan MUL_MAT_ID offload batch of 32 tokens: 32-63 tokens on Inkling-2-Mini), it uploads only the experts pass 0 selected. Passes 1 and 2 reuse that copy and read stale VRAM for every other expert. Ubatches of 64+ tokens are correct (F1 uploads every expert), ubatches below 32 run MUL_MAT_ID on the CPU (no copy).
Where. src/llama-model.h (llama_model::post_load_tensors, a new optional hook),
src/llama-model.cpp (llama_model_base::load_tensors calls it on the no_alloc return and after
load_tensors_hadamard()), src/models/models.h and src/models/inkling2-loop.cpp
(llama_model_inkling2_loop::post_load_tensors, member ctx_visits). Commit 312c29c2b.
How it works. After the weights are loaded, for every slot of pass 1 and 2 (il >= n_phys), each
tensor pointer in its llama_layer (walked as an array of pointers, as llama_context::opt_init
does) that sits in a host buffer (ggml_backend_buffer_is_host) is replaced with
ggml_view_tensor of it, named <name>.visit<pass>, initialized with ggml_backend_view_init
(same buffer and data). On the no_alloc pass (memory fit) only view->buffer is set to the dummy
buffer, so the scheduler still sees a WEIGHTS buffer. The view is a leaf (op NONE, view_src set),
so the scheduler hashes it separately: each pass gets its own copy, freed after the last reader of
that pass, and the copy callback sees the MUL_MAT_ID of that pass and its ids (st.ids differs, so
F1 recomputes the expert set). The pointer-keyed weight maps of llama_model (w_scales,
prec_policy.prec_acc/prec_src1, hdmd.rot) get entries for the views. The views live in
ctx_visits, a context of the model outside ctxs_bufs. The startup log prints
post_load_tensors: N views of host weights for the loop passes after the first (X MiB, not copied).
Knob: LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors (the upstream behaviour), read once at load.
Default on.
Invariants that must hold.
- Only host buffers get views. Device weights stay shared (used in place, no copy, so no problem), and so do CPU extra buffers (CPU_REPACK is not a host buffer; its ops run on the CPU in place).
- F4: views point only at host buffers, which
release_device_weightsnever releases, and they are not inctxs_bufs, so the shadows,weight_release,memory_breakdown,n_tensors()andtensors_by_namedo not see them. A device weight is released and restored in place, and the loop slots keep pointing at the same tensor. - F1: the skip/readback decision is per copy and keys on the ids tensor of the first node of the split, so it stays per pass.
llama_layermust hold onlyggml_tensor *members (astatic_assertchecks the size only).
Cost. Pass 1 and 2 now upload their own experts: per ubatch the host-to-device expert traffic
goes from 24 to 48 layer uploads (about +13 GiB per full ubatch on UD-Q4_K_XL). Measured where the
old layout still fits in VRAM (llama-bench pp2048 at depth 0, -ncmoe 24, B C C B in one window):
views 107.10 / 107.53 t/s, off 110.80 / 111.53, so -3.5%. Decode is unaffected (single-token
MUL_MAT_ID runs on the CPU). In the server at 16k-65k context the old layout does not fit (7 GiB of
compute buffer), spills into GTT, and the views win about 1.6x at prefill (Measured, below). The
freed ~6 GiB of VRAM can hold experts instead: an expert tensor of a loop layer kept on the GPU
saves 3 uploads per ubatch and 3 CPU expert passes per token, a non-loop one saves 1, so loop layers
are the first to move to the GPU (-ot with the loop layers left out of the CPU pattern; placement
guide below).
Measured (2026-11-22, pod, GPU to itself, build b13c41355 = 312c29c2b + docs; drivers and logs in
/root/arms/overnight/: f13.py, cmp.py, kld48.sh, fit.py, sweep.sh, STATUS.md).
-
Memory, server with the production command line at ctx 16384,
-lv 4: Vulkan0 compute buffer 1426 MiB with views, 7030 MiB without. Process VRAM after load: views 4179 MiB, off 7727, A (loop experts on the GPU, no views) 7879 (A wants 10089 MiB and spills). -
Prefill, server, 7224-token raw prompt at ctx 16384: views 92.4 t/s, off 57.0, A 46.5 (production at ctx 65536 without views: about 46).
-
Correctness. First-token top-5 logprobs (
f13.py,n_probs 5, knob 32 so the 32-63 window is live): with the tail at or above 64 or below 32 tokens (N = 20, 100, 7368) views on and off are identical (max difference 0.000); in the window (N = 40, 63, 1072, 7224) they differ (0.04-0.16). That test cannot say which arm is right: arm A differs from both by up to 0.15 even at the controls (different placement, different kernels), and at 32-63 tokens x 8 experts pass 0's union already covers most of the 128 experts, so the stale copy is mostly correct data. The decisive test iskld48.sh:llama-perplexity -c 48 -b 48 -ub 48, 30 chunks of Pan Tadeusz, so every chunk is one 48-token ubatch, KL divergence against a base with views on and knob 64 (the tail MUL_MAT_ID on the CPU, correct by construction):arm mean KLD max KLD same top token RMS dp views on, knob 32 (tail on the GPU) 0.00314 0.087 97.4% 1.35% views off, knob 32 (tail on the GPU) 0.03163 0.347 92.2% 3.71% views off, knob 64 0.00000 - 100% 0.00% Views on is at the GPU-vs-CPU kernel noise level; views off is 10x worse (the #32755 mechanism, reproduced). So with F13 the
GGML_OP_OFFLOAD_MIN_BATCH=64workaround is no longer needed.
Placement guide (UD-Q4_K_XL, views on). k = number of loop layers (6, 7, ...) whose experts stay
on the GPU: -ot 'blk[.](<the other 24-k layers>)[.]ffn_.*_exps=CPU' instead of --n-cpu-moe 24
(/root/arms/overnight/mkot.py and cfgedit.py write the pattern; [.] avoids backslashes in the
YAML and in llama-swap's argument splitting). Each loop layer's experts take 547 MiB. Fit, from the
-lv 4 breakdown of a server with the production command line (mmproj on demand, so not loaded):
| ctx | k | Vulkan0 self (model + context + compute) | free | verdict |
|---|---|---|---|---|
| 32768 | 0 | 4136 (1266 + 1347 + 1522) | 3982 | fits |
| 32768 | 6 | 7416 | 699 | fits |
| 32768 | 7 | 7963 | 15 | too tight |
| 65536 | 4 | 7246 (3453 + 2371 + 1421) | 868 | fits, chosen |
| 65536 | 5 | 7793 | 128 | too tight for the mmproj |
| 65536 | 6 | - | - | spills (GTT about 900 MiB above expectation) |
| 131072 | 0 | 7784 (1266 + 4419 + 2098) | 134 | fits (before F13 it overcommitted 5.2 GB) |
| 131072 | 2 | - | - | spills (about 1.2 GB) |
fdinfo drm-memory-gtt is not the spill test: it includes the pinned host weights
(Vulkan_Host). Compare GTT(k) with GTT(0) - k x 547 MiB, or read the breakdown.
Speed, llama-bench, one invocation per metric (sweep.sh, -fa 1 -t 6 -b 1024 -ub 1024, knob 64,
pp r1, tg r2):
| k | pp2048 @ d8192 | pp2048 @ d16384 | tg256 @ d4096 |
|---|---|---|---|
| 0 | 97.53 | 89.62 | 5.57 |
| 4 | 102.19 | 93.90 | 6.45 |
| 6 | 104.43 | 95.27 | 6.72 |
| 7 | 105.90 | 96.90 | 7.25 |
Decode gains most (+16% at k=4, +30% at k=7): every loop layer on the GPU removes 3 of the 48 CPU expert passes per token. Prefill gains only 5-9%: at depth the attention and the remaining uploads dominate.
Quants (2026-11-22, not adopted). Unsloth's Mainline_Support UD-Q3_K_XL (13.6 GB) and UD-IQ3_XXS
(11.7 GB) are in /root/models/inkling/. Quality: llama-perplexity -c 2048 --chunks 4, KL
divergence against UD-Q4_K_XL logits (kld.sh), on a fixed Pan Tadeusz chunk
(/root/bench/entext/pt-chunk.txt, characters 20000-100000) and wikitext-2 test:
| text | quant | PPL (Q4_K_XL) | PPL ratio | mean KLD | same top token |
|---|---|---|---|---|---|
| Pan Tadeusz | Q3_K_XL | 11.45 (11.30) | +1.4% | 0.0241 | 92.5% |
| Pan Tadeusz | IQ3_XXS | 11.52 (11.30) | +1.9% | 0.0421 | 90.1% |
| wikitext-2 | Q3_K_XL | 6.251 (6.230) | +0.3% | 0.0097 | 95.6% |
| wikitext-2 | IQ3_XXS | 6.307 (6.230) | +1.2% | 0.0214 | 94.2% |
Fit at ctx 65536: UD-Q3_K_XL keeps the loop-layer experts at 547 MiB each (like Q4), so k=5 (867 MiB free); UD-IQ3_XXS has 390 MiB per layer, k=7 (952 MiB free). Speed (separate invocations, so within the ~7% cross-run noise for prefill): Q3_K_XL k=4 105.23 / 6.83, k=6 107.31 / 7.48; IQ3_XXS k=4 105.59 / 6.44, k=7 108.48 / 7.86 (pp2048 @ d8192 / tg256 @ d4096; Q4_K_XL k=4 102.19 / 6.45). The best case, IQ3_XXS at k=7, is +6% prefill and +22% decode (about 7.9 instead of 6.5 t/s) for a Polish KLD of 0.042; IQ3_XXS's CPU dot product is slower per byte (at k=4 it decodes no faster than Q4). Not worth the quality loss for the owner's Polish use.
Production (deployed 2026-11-22, b13c41355, ctx 65536, k=4, no knob). Through llama-swap
(/root/arms/overnight/post.py, log post.log): Pan Tadeusz 9312 tokens 81.6 t/s prefill, +7958
tokens 89.3 t/s, +17255 tokens (28K total) 78.0 t/s, decode 5.9-6.0 t/s, answers correct (before:
about 46 t/s prefill). Think and nothink entries answer text, an image ("Red") and a tool call. A
tiny request after the 4B task model held the GPU takes 1.3-2.4 s longer than a warm one (the
arbiter restores 3.4 GiB of device weights). Disk cache save/restore works on Inkling (1329 tokens,
410 MiB). The first request on a fresh process is slow (162 tokens 11.7 t/s prefill, 2.6 t/s decode)
since the pipelines are compiled on first use (--no-warmup).
Profile (2026-11-22, perf record + gpu_busy_percent, prof.sh, results in
/root/arms/overnight/prof/). Inkling k=4 prefill (pp2048 @ d4096, 107 t/s): GPU busy 77%, CPU
time mostly ggml_vk_wait_for_fence (43%) and the kernel (11%, the uploads): GPU-bound. Decode
(7.3 t/s): GPU busy 58%, CPU in the OpenMP barrier spin (libgomp 36%) and
ggml_vec_dot_nvfp4_q8_0 (21%): the UD-Q4_K_XL experts are NVFP4, and that CPU dot product is the
decode hotspot. Qwen3.6-35B for comparison: prefill 263 t/s at GPU busy 52%, decode 13.8 t/s at
35%, vec_dot_q4_K_q8_K 10%. GGML_VK_PERF_LOGGER=1 aborts at the second split
(GGML_ASSERT(ctx->compute_ctx.expired()), ggml-vulkan.cpp:15692), so it gives per-op GPU times
only for the first split; not investigated.
Failure modes and known limits.
- LoRA adapters look weights up by name (
llama_adapter_lora::get_weight), so they do not apply to passes 1 and 2 with views on. Not used here; setLLAMA_LOOP_HOST_VIEWS=0for a LoRA on Inkling. --moe-cache-mib: the cache binds expert tensors per slot (llama_moe_cache_layer_expertsofmodel.layers[il]), so with views each loop slot gets its own binding and slot map and caches the same experts separately (before, the three slots collided on one binding key). Should not crash; not tested. The MoE cache is off in production.- Model saving is not supported for
inkling2_loop(llama_arch_supports_save), so duplicate data in a saved file cannot happen.
Upstream #32807 (in 5bacb7c86, branch -20261122). Upstream's fix for #32755: in
sched_copy_experts, after the first-node check and before F1's logic, if (arch == LLM_ARCH_INKLING2_LOOP) return false;, so the scheduler copies every expert of a shared host
weight (a TODO points at per-pass copies; the scheduler-side alternative #32809, "copy host experts
again for each split that reads them", was closed unmerged). That fixes correctness but keeps one
device copy per (host tensor, backend) per graph, alive until the last pass: the 6.5 GiB of held
copies and the ~7 GiB compute buffer stay. So F13 is still needed for memory; for correctness
it is now redundant with upstream's guard.
- With both, each pass uploads its own copy (F13) and every copy is a whole tensor (the guard): in the 32-63-token window the readback path, which would upload only the selected experts, no longer runs. Ubatches of 64+ tokens are unchanged (F1 copies every expert anyway), below 32 the experts run on the CPU.
- Knob (fork, commit "env knob LLAMA_LOOP_FULL_COPY=0"):
LLAMA_LOOP_FULL_COPY=0skips upstream's guard. Only correct with the views on: withLLAMA_LOOP_HOST_VIEWS=0it brings #32755 back. - On this branch, "upstream's fix alone" is
LLAMA_LOOP_HOST_VIEWS=0(shared tensors plus the guard), "F13 plus upstream" is the default, "F13 alone" isLLAMA_LOOP_FULL_COPY=0.
Window A/B to run (GPU to itself, candidate build, Inkling production command line):
- Memory: process VRAM/GTT after load and the
-lv 4breakdown at ctx 65536 with production's k=4-ot, default vsLLAMA_LOOP_HOST_VIEWS=0. Expected: upstream-only spills (compute about 7 GiB, as before F13); F13 fits with about 0.8 GiB free. - Correctness:
/root/arms/overnight/kld48.sh(48-token ubatches, KLD against the knob-64 base) for default,LLAMA_LOOP_HOST_VIEWS=0andLLAMA_LOOP_FULL_COPY=0. Expected: all three at the GPU-vs-CPU noise level (mean KLD about 0.003); before #32807 views off gave 0.032. - Speed: llama-bench pp2048 @ d8192 and tg256 @ d4096 (
sweep.shsettings) and a 48-token-tail server prompt (the window where the guard costs uploads), default vsLLAMA_LOOP_FULL_COPY=0, arms interleaved in one invocation (section 6). Keep the guard unless the knob wins clearly. Then decide whether F13 stays (expected: yes, for memory).
How to drop it. If upstream fixes #32755 (for example per-reader copies or a copy-callback call for every reader in the scheduler, or real per-pass tensors in the loader), revert 312c29c2b; the hook has no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
Rebase hazards. post_arch_tensors and the loop slot layout in inkling2-loop.cpp;
llama_model_base::load_tensors returns (a new return path must call the hook); new pointer-keyed
weight maps in llama_model (they need a view entry like w_scales); non-pointer members in
llama_layer; scheduler changes to how host weights are copied (hv_tensor_copies,
ggml_backend_sched_is_host_weight).
Validation (pod, GPU to itself in a swapdown window). Build llama-server and llama-bench
from this branch in a scratch worktree (runbook section 5). Use the Inkling command line from
inkling_args in /root/config.yaml (with --mmproj) and change only the context to 16384 (so
arm A fits) and the placement:
- A, reference: replace
--n-cpu-moe 24with-ot 'blk\.([0-5]|1[89]|2[0-3])\.ffn_.*_exps=CPU'(loop experts on the GPU, no copies for the loop; the views log line must not appear). - B, views on:
--n-cpu-moe 24; the log line must show the views. - C, views off: B with
LLAMA_LOOP_HOST_VIEWS=0.
For each arm send raw token prompts (/tokenize a long text, cut to N tokens, /completion with the
token array, temperature 0, top_k 1, seed 42, cache_prompt false, n_predict 32, n_probs 5) at
N = 40, 63, 1072, 6176, 7224 (tail 32-63 at ub1024), controls N = 100, 7368 (tail >= 64) and 20.
Expected:
- B == A in the first token's top-5 probabilities for every N >= 32 (prefill MUL_MAT_ID runs on the GPU in both arms with the same data). The generated text may drift later, since decode MUL_MAT_ID runs on the GPU in A and on the CPU in B.
- B == C byte-identical for the controls (tail >= 64 or < 32).
- C differs from A at the 32-63 tails (the bug); if it does not, the bug was not reproduced and B's result proves nothing.
- Vulkan0 compute buffer (startup log): B about 6 GiB below C. A's model buffer carries the loop
experts instead.
Then
llama-bench -lm none -ngl 99 -ncmoe 24 -fa 1 -t 6 -b 1024 -ub 1024 -p 2048 -n 0 -r 3, with and withoutLLAMA_LOOP_HOST_VIEWS=0, interleaved B C C B, medians (section 6), for the prefill cost above. After a deploy: an Inkling text request, an image request, and the same text request after a task-model call (arbiter handoff, F4), identical at temperature 0.
F14. User-message batch-break merge (one ubatch per short turn)
Branch fable5/userstart-split (rebased onto the F13 production tip 166ec22e2), fast-forwarded into
the production branch on 2026-11-22 (deploy and results below). Knob: LLAMA_USERSTART_MERGE=0.
What and why. Upstream ends a prompt batch right before a user message so that a context
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
limited the earlier ones to --checkpoint-min-step). On a hybrid/recurrent model the recurrent state
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
message is what a later request uses when it diverges inside that message (the user edits it, or the
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
template drops the previous answer's reasoning, which moves the divergence to just after
<|im_start|>assistant\n, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
restores. The cost on this hardware: a turn processes [previous answer | new user message + generation prompt]. With the break that is two ubatches instead of one, and with every expert on the host each
ubatch of >= 32 tokens (GGML_OP_OFFLOAD_MIN_BATCH) uploads the experts again: all of them from 128
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
<|im_start|>user, the detected delimiter).
Where. tools/server/server-context.cpp, update_slots, the prompt-batch loop:
merge_user_starts before the loop and the !merge_user_starts term of the user-start break.
Commit 510aa6e09. On branch -20261122 the refinement adds first_user_pos before the loop, and the
user-start break checks merge_user_starts first (merged: break only at first_user_pos).
How it works. When the rest of the prompt from the current batch start fits in one ubatch
(task.n_tokens() - prompt.n_tokens() <= n_ubatch) and no media chunk was processed in this
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
the restored checkpoint at the same position, with the current task id, so the next request still
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
diverges inside that message restores the batch-start checkpoint instead and re-processes the
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
whole of a ubatch, and the fallback would not be bounded). LLAMA_USERSTART_MERGE=0 restores the
upstream behaviour everywhere.
Refinement: the system-prompt checkpoint (owner's wish, commit "F14 - keep the batch break at the
first user message", branch -20261122, not deployed). The merged path keeps one user-start
break: the one at the first user message, which is the end of the system prompt. Upstream breaks
there on a fresh prompt (no checkpoints yet), and the plain F14 had dropped it, so a new chat with
the same system prompt found no checkpoint at the system-prompt end and re-processed the system
prompt (a hybrid/recurrent state cannot be cut back without one). Now, merged or not, a prompt
processed from before its first user message ends a batch there; the next batch starts at the
user message and gets a checkpoint (user-start batch, near-end rule). A later new chat with the same
system prompt diverges inside the first user message, restores that checkpoint and processes only
its own user message and the generation prompt.
- Only fresh prompts are affected: checkpoints past the restore point are erased on reuse, so the batch reaches the first user message only when the prompt is processed from before it (a new chat, or a changed system prompt). Follow-up turns start past it and keep the full merge, including no break at the last user message (the edit cost below is unchanged).
- First user messages below 8 tokens are skipped (no system prompt, only a BOS before it).
- Cost: turn 1 of every new chat runs the system prompt as its own batch again. That was part of the U arm's turn-1 layout below (TTFT 6.08 vs 3.99 s for M, which also merged the last-user break; the system-prompt split alone was not measured). A system prompt of 32+ tokens runs on the GPU with one extra expert upload, below 32 tokens on the CPU.
- Lifetime of that checkpoint: it is the oldest one of the slot. When the list is full
(
--ctx-checkpoints, default 32), the min-step thinning keeps the oldest checkpoint and erases later ones within--checkpoint-min-step(8192) of it, and only then the FIFO drops the oldest. A very long chat can still lose it; the RAM prompt cache (and F11 on disk) keeps the earlier conversation's copy. - Disk cache (F11), read:
disk_cache_loadreplaces the slot prompt only with an entry that reuses at leastDISK_CACHE_MIN_TOKENS(256) more of the new prompt, andn_reusecounts a saved prompt that diverges inside it as its newest checkpoint at or below the divergence. A saved chat therefore offers the system-prompt checkpoint to a new chat after a restart, but only for a system prompt of 256+ tokens; it never replaces a better slot prompt. The checkpoints are saved in the file appendix and trimmed to--ctx-checkpointsfrom the front on load (only if the file has more). Nothing in it removes the break; check it in the window anyway (validation plan item 5).
Invariants that must hold.
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
divergence inside the last user message to less than
n_ubatchtokens. - The batch-start checkpoint must still be created (near-prompt-end rule,
do_checkpointnot cleared byhas_mtmd); hence the!has_mtmdcondition. - It only changes where batches end, never which tokens are processed, so outputs differ from an unmerged run only by batch-composition rounding.
Measured (2026-11-22, pod, GPU to itself in one swapdown window, build f3397cded; driver
/root/arms/usw_ab.py, window script /root/arms/usw-window.sh, logs and res.json per arm in
/root/arms/usw-ab/{M1,U1,U2,M2}/). Production 35B think entry (q36t: ${ctx_256k} ${qwen35_think_args} ${moe_offload_args}, -hf pulls the mmproj, supervisord's environment, scratch
arena and --cache-disk-path, plus -lv 4), one fresh server per arm, arms M (merged, default) and U
(LLAMA_USERSTART_MERGE=0) in the order M U U M. Each chat: system prompt, 6 user turns from
pt-chunk.txt (150, 10, 40, 150, 600, 40 tokens), synthetic previous answers of 300, 800, 1200, 500,
1300 tokens, max_tokens 8, temperature 0, streamed; then the last user message edited in the
middle and resent. TTFT in seconds (mean of the two runs per arm; the single runs in brackets),
prompt_n was identical in all four runs on every turn:
| turn | user tokens | prompt_n | M TTFT | U TTFT | M - U |
|---|---|---|---|---|---|
| 1 (fresh prompt) | 150 | 196 | 3.99 (4.43, 3.55) | 6.08 (5.73, 6.42) | -2.1 |
| 2 | 10 | 327 | 4.11 (3.75, 4.47) | 7.66 (6.75, 8.57) | -3.6 |
| 3 | 40 | 869 | 6.34 (6.62, 6.06) | 9.34 (7.26, 11.42) | -3.0 |
| 4 | 150 | 1381 | 7.20 (7.30, 7.10) | 10.95 (12.31, 9.58) | -3.7 |
| 5 | 600 | 1130 | 8.82 (7.47, 10.17) | 9.80 (9.52, 10.07) | -1.0 |
| 6 | 40 | 1370 | 8.11 (8.01, 8.20) | 9.53 (9.33, 9.72) | -1.4 |
| edit of turn 6 | 40 | M 1374, U 66 | 7.94 (8.16, 7.71) | 3.65 (4.09, 3.20) | +4.3 |
- Batch layout (
prompt processing, n_tokensand checkpoint lines at-lv 4): M runs every turn as one batch[restore point, end-4)plus the tail, with checkpoints at the batch start (superseding the restored one) and at end minus 4; U splits at the last user message (an extra checkpoint there, for example 1323 between 515 and 1380 on turn 3) and on turn 1 also at the first user message (system prompt, 23 tokens). Turn N+1 restores the end-minus-4 checkpoint of turn N in both arms. Noforcing full prompt re-processingin any run. - The 10-token turn gained too, against the plan's "no change". U split it into 308 tokens (the previous answer, on the GPU) and 19 (message, generation prompt and tail), and the 19-token part runs the experts of all 40 layers on the CPU, which costs seconds rather than nothing at 6 threads (inferred from the totals, not profiled). The 600-token turn gained least (its two parts are 508 and 618 tokens, both on the GPU, so the split costs one extra upload, about 1-2 s against noise of the same size).
- Edit of the last user message (the cost of the trade): M has no checkpoint at the user-message
start, restores the batch start and re-processes previous answer + message (1374 tokens, one
ubatch); U restores the user-start checkpoint and re-processes 66 tokens. +4.3 s per such edit.
Ordinary follow-up turns are far more frequent than edits of the last message in this use; a client
that injects per-request context into the last user message (RAG, web search) pays the same
as an edit on every turn, so set
LLAMA_USERSTART_MERGE=0for such a client. - Outputs: identical within each arm across runs; between arms turns 5 and 6 differ after a few tokens (batch-composition rounding, as the invariant says).
- Disk cache:
fork-validate.sh --diskcache(in the window, GPU leg included) 15 PASS, 0 FAIL, 0 SKIP.diskcache-window3.shlegs with production's 35B command line (q35.pycopied toq35-usw.pywith the candidate build path; log/root/arms/usw-q35-mm.log): 18.8k-token chat, turn 1 83.5 s, save 557 MB in 1.0 s, restart restore and turn 2prompt_n27 in 3.7 s, output == reference; cold save and restore identity (prompt_n4, output == reference); image answered 'Red' and not saved. The saved state after turn 2 holds 4 checkpoints instead of 5 (the user-start one is gone).
Production (deployed 2026-11-22 10:37, f3397cded, config unchanged). Deploy script
/root/arms/dep-usw.sh (log dep-usw.log): incremental in-place build, downtime 33 s. Post-deploy
checks through llama-swap (/root/arms/post-usw.py, ink-usw.py, ink2-usw.py, logs beside them):
system_fingerprint b12687-f3397cded; 35B 'Paris', then 4B / 35B handoffs twice (correct answers,
holder the answering PID each time, 2.9-6.3 s per switch); a 1140-token 35B chat, then a gemma-26B
image ('Red', the 35B exited by itself with loop stopped, device memory freed and disk cache: saved the prompt of 1157 tokens), gemma text, 4B text, then the 35B chat plus one user message:
disk cache: restored 1157 tokens, prompt_n 30 (the suffix only). Inkling nothink: text correct
(it still spends about 40 tokens on reasoning, so a 96-token budget returns an empty content), image
'Red', a 9293-token Pan Tadeusz prompt at 82.0 t/s with the right answer (F13 measured 81.6). Children
bind llama-swap's ports. Log scan clean, dmesg clean (no ring timeout, GPU hang or OOM kill), pod
restarts 0 as before.
Failure modes and incidents. None in validation or the deploy. Known cost: an edit of the last user message (or per-request context injected into it) re-processes the previous answer too, +4.3 s measured above.
Rebase hazards.
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
checkpoint_offsetsbreaks and the near-end / min-step checkpoint rules. Re-check that the batch start is still checkpointed when the batch ends at prompt end minus 4.
Validation plan (run 2026-11-22, results under Measured above).
- Pre-check at trace log level (
SLT_TRC;-lvhigh enough forLOG_TRC): per request, thecreated context checkpoint ... n_tokens = Nandrestored context checkpointlines. Merged: one batch[restore point, end-4)plus[end-4, end);LLAMA_USERSTART_MERGE=0: the extra break at the last user message. - TTFT A/B through the production 35B entry (exact
moe_offload_argscommand line, mmproj loaded), candidate with and withoutLLAMA_USERSTART_MERGE=0, arms alternated in one run: scripted 6-turn chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Readprompt_n/prompt_msfrom the timings and TTFT from the stream. Expected: no change at 10 tokens (the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens. - Reuse: turn N+1 must report the same
prompt_nin both arms (restore from end-minus-4). Then edit the last user message and resend: merged re-processes previous answer + message (bounded by one ubatch), unmerged only the message; no "forcing full prompt re-processing". - Disk cache:
fork-validate.sh --diskcacheanddiskcache-window3.sh(production command line): restore after a restart,prompt_nand output identical to the reference built from the same commit.
Additions for the refinement (branch -20261122, to run in the window with the 35B think entry's
exact command line plus -lv 4, scratch port, arena and cache dir):
5. New chat #2 with the same system prompt. Chat 1: a system prompt of about 300 tokens (above the
256-token disk-cache minimum), then 3 turns as in item 2. Expected at trace level on chat 1 turn
1: a batch [0, first user start), then a created context checkpoint ... n_tokens = <first user start>, then the merged batch to end minus 4. Then chat 2: the same system prompt, a new
first user message. Expected: restored context checkpoint ... n_tokens = <first user start>
and timings.prompt_n = chat 2's user message plus the generation prompt only (not the system
prompt); no forcing full prompt re-processing. Follow-up turns of chat 1 must still run as one
batch (no break at the last user message): the M layout of the table above.
6. The same after a restart (disk cache): run chat 1, stop the server with SIGTERM (disk cache: saved the prompt), start it again, send chat 2. Expected: disk cache: restored, then the
restore of the system-prompt checkpoint and the same prompt_n as in item 5. Also once with a
system prompt below 256 tokens: no disk restore is expected there (the minimum), only a fresh
prompt; that is the known limit, not a failure.
7. The 6-turn TTFT table of item 2 with the candidate, M only: turn 1 may be slower than 3.99 s
(the system-prompt split is back), turns 2-6 must match the M column within noise.
4. Dead ends
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry without a new reason.
| idea | result | mechanism |
|---|---|---|
Weight pinning (GGML_CUDA_REGISTER_HOST and the host-register chain, from a third-party CUDA fork whose +64% was on an RTX 3060) |
+17% pp2048 in isolated llama-bench (252 -> 294), fails in serving | RADV cannot import file-backed mmap pages; the pre-stage fallback fails (ErrorOutOfDeviceMemory for the ~18 GB expert region) or doubles host RAM. -lm none serves faster (Qwen3.6-35B 278 -> 286). Dropped in the 2026-09-09 rebase. |
Expert-upload prefetch (GGML_SCHED_PREFETCH_EXPERTS) |
-20% pp (other runs -9% to -22%), no tg benefit | Slots allocate in host-pinned GTT, not VRAM, so it stages host to host and still pays the PCIe read. =1 means the default 3 slots. Dropped. |
GGML_VK_ASYNC_USE_TRANSFER_QUEUE=1 |
flat (-0.1%), byte-identical | GCN is excluded from prefers_transfer_queue and RADV exposes no transfer-only queue family here; the per-layer upload-then-compute order leaves nothing to overlap. A standalone Vulkan microbench did show 60-73% of an H2D copy on queue family 1 hidden behind compute, so double-buffered prefetch inside one backend context remains an untested idea (estimated +10-15%, unproven). |
Fractional -ncmoe |
no measured win | Config only uses integers (22/40/48); upstream refactored -ncmoe into llm_add_n_cpu_ffn_overrides. Dropped. |
MoE cache --moe-cache-mib 512 / 1536 / 2304 (Qwen3.6-35B, ncmoe 40, ctx 131072) |
tg256@d4096 -42% / +5% / +12%; prefill +1-3%; idle VRAM 6.15 / 7.19 / 7.96 GB vs 5.66 | Small caches thrash (8-of-256 routing over 40 layers) and add a per-layer ids readback; only ~2.3 GiB pays and leaves no VRAM for a long KV. Not enabled. |
| MTP speculative decoding, Qwen3.6-35B | decode -39% (n1), -37% (n2), -50% (n3), -44% (n4), best -19% (n2, p-min 0.75); prefill -3 to -6% | A spec step costs ~180 ms vs 52 ms for plain decode; a 2-token verify alone is only 1.4x a 1-token step, so ~100 ms/step is draft machinery. The nextn layer adds 1.25 GiB VRAM. Entries removed. |
| MTP, gemma-4-26B-A4B | decode -57.9% (n1), prefill pp4k -34% | Same mechanism (~210 ms vs ~50 ms per step); MTP needs --swa-full, which alone costs -10.5% decode, -32% pp4k and saturates VRAM. Entries removed. |
| MTP, gemma E4B | decode -10.9% | Acceptance 0.41-0.67. The +19.5% pp seen with it comes from --swa-full, not MTP. |
--parallel 1 on gemma |
no effect (-1.0%, +2.1%) | The Qwen gain is probably tied to its recurrent state being sized per sequence (unverified). |
-ub 4096 -b 4096 (Laguna) |
flat (90.0 vs 90.9) | Fewer expert uploads cancelled by intra-ubatch attention growth. |
mul_mat_id tile-padding theory (larger/odd -ub) |
predicted 2.00x waste and a 1.94x cliff; measured +3.8% and -5.6%, no sawtooth | Theory dead; kills the mul_mat_id tile-selection shader work (an unbuilt, unverified WIP exists on branch fable5/mmid-tile-wip, with an unresolved correctness concern in its row_ids fill). |
-b sweeps |
non-monotonic (2048/8192 high, 4096/12288 low) | -b 2048 is already at the top. |
-ncmoe below 40 on the 35B |
34: +2.9% pp but 7.23 GiB peak at tiny context; 28: better at short context, worse at depth (137 vs 170 t/s at d32k); <= 24 collapses (~122) | Resident experts are evicted by KV/activation growth; cannot survive a 128k context. |
-ncmoe 44 on Laguna |
+1.5% | Does not fit at 64k context. Keep 48. |
fp16 on Polaris (Mesa 26.1.6 + radv_enable_float16_gfx8=true, fp16: 1) |
pp2048 -4.7%, -5.3% at d16k, tg +4% | Hurts prefill (the priority). Mesa 26.1.6 alone: +3.3% prefill, below the 5% bar; adopting it needs a pod image change. The pod's Mesa 25.0.7 has no such option. |
| KV quant on Qwen (q4_0, q8_0) | f16 beats q4_0 by +40% pp at 64k; q8_0 170 -> 128 t/s at depth; q4_0 with upstream PR #30738 still -16% vs f16 at d32k (133.6 vs 159.4) | No int-dot/fp16 here, so dequant is pure compute; the hybrid model's KV is small anyway. Keep f16. (PR #30738 was +15% at d16k on gemma-26B with q4_0 KV; not merged upstream as of the 20261024 rebase.) |
-fa 0 |
5x slower at depth (170 -> 35 at d32k) | Keep flash attention on. |
FA scalar block_rows 16 -> 32 |
3.5x regression at d32k (135 -> 38) | ~42 KB shmem per workgroup at head_dim 256, one workgroup per SIMD, occupancy collapse. |
GGML_VK_DISABLE_GRAPH_OPTIMIZE=1 |
restores tg32 at n_ctx=32 (10.44 -> 15.61), nothing at real depth | Upstream #27812's view-alias fix makes KV views look dependent and collapses barrier-free batches; only visible when per-node work is tiny. It is a correctness fix; do not disable it in serving. |
--no-mmap/-lm none for Laguna-S 118B |
does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. |
Laguna --cache-reuse, --swa-full, -t 12, KV quant |
ruled out | --cache-reuse is force-disabled for iSWA when base and SWA sizes differ. |
CPU-only task model (-ngl 0) to avoid handoffs |
rejected | At --ctx-size 8192 titles could face 8k tokens of CPU prefill. The arbiter is the answer. |
-lm dio (direct IO) instead of the F10 page-cache drop |
load +0.9 s on 3.4 GiB (6.89 vs 5.99 s), the same cost as the drop; 175 MiB of the model file and all of the mmproj stay cached | Synchronous 64 MiB O_DIRECT chunks with no readahead; the GGUF metadata goes through a separate buffered FILE and clip.cpp has no direct-IO path. Measured CPU only; reads into pinned Vulkan buffers may fall back to buffered IO (EFAULT or EINVAL). Not adopted. |
The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256 tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.
Where the time goes (Laguna-S-2.1 118B IQ2_M, all experts on host, GGML_VK_PERF_LOGGER=1, depth 0,
22.65 s per 2048-token eval, 17.18 s of it GPU-busy, so about 24% is H2D stall; moved here from the
old README section): expert MUL_MAT_ID 52%, attention projections 24%, FLASH_ATTN_EXT 16%,
everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s while dense MUL_MAT q5_K/q6_K in
the same graph reaches 3096-3691, so closing that gap is the largest remaining opportunity on this
card, not the transfer path. On interleaved-SWA models the sliding-window layers keep a small KV
(n_kv pinned at n_swa * n_seq_max + n_ubatch), so they cost the same at any depth; on Laguna 36 of
48 layers are O(1) in depth and the whole high-context slowdown comes from the 12 full-attention
layers.
5. Configuration that matters
/root/config.yaml (llama-swap) and /root/supervisord.conf on the pod are the source of truth;
this section explains the choices. Values as of the 2026-11-22 deploy.
Qwen 35B-A3B MoE entries (macro moe_offload_args, applied to all 35B-A3B variants):
--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1,
context 131072.
-lm none: load weights into RAM (upstream removed--no-mmap; the old spelling makes the server refuse to start). Faster in serving than mmap and recommended for--n-cpu-moe. Exception: Laguna-S, which must use mmap (GTT limit, section 4).-ub 2048 -b 2048: the biggest prefill lever. ub512 = 97 t/s, ub1024 = 159, ub2048 = 254 on the 35B; the per-eval expert upload is amortized over the ubatch.--n-cpu-moe 40: all experts on host. Lower values are faster at short context but run out of VRAM at long context (section 4). Measured idle VRAM at ctx 131072: 5.66 GB.--parallel 1: +22.9% decode on the Qwen 35B (15.5 -> 19.1 t/s), prefill +0.3-1.2%, +64 MiB VRAM. Without it the server creates 4 slots and a unified KV. Two concurrent requests queue on slot 0. With one slot a ubatch never holds more than one sequence, which the concat fast path'sne[2] == 1condition (F3) needs.-t 6: six physical cores; more threads hurt decode.-fa 1: essential (section 4).- f16 KV (no
-ctk/-ctv): section 4.
Other models:
- gemma-4-26B-A4B: macro
moe_offload_gemma, same as above but--n-cpu-moe 22(8 layers of experts resident; gemma's SWA keeps KV small at depth). ncmoe sweep pp2048: 30 = 262, 24 = 266, 18 = 272, 14 = 157 (VRAM collapse); 22 keeps headroom for 128k context (7.40 of 8 GB at 128k). Its KV cache is-ctk q4_0 -ctv q4_0, set in thegemma4_sampling/gemma4_nothink_samplingmacros (not inmoe_offload_gemma); note gemma has no--parallel 1(no measured gain there). - Small models that fit in VRAM (Qwen3.5-4B, gemma E4B/E2B and similar) use auto-fit
(
common_args); forcing experts to host would slow them. - The 4B task model runs with
--no-mmprojand ctx 8192, and is preloaded. - Laguna-S-2.1 118B IQ2_M (configured as
Laguna-S-2.1-IQ2_M, matrix varlag): mmap,--n-cpu-moe 48 -b 2048 -ub 2048 -fa 1 --parallel 1, ctx 65536. Its exclusivity set must allow the task model (lag & q4task); making it exclusive with the task model killed it on every title request and forced full re-prefills.--parallel 1frees 223 MiB of VRAM at 64k context there (4 auto slots size the SWA cachen_swa * n_seq_max + n_ubatch, 4096 cells instead of 2560). - Inkling-2-Mini (26B MoE,
inkling2, 24 layers with the middle 12 looped 3x, 24 heads / 8 KV heads, head size 128; macroinkling_args, entriesunsloth/Inkling-2-Mini-GGUF[-nothink]:UD-Q4_K_XL, matrix varsink/inknt): the mainline-layout GGUF in/root/models/inkling/passed with-m(the repo-root GGUFs that-hfwould pick are Unsloth's layout),--mmprojbeside it,-t 6, f16 KV,--parallel 1, the disk prompt cache,-b 1024 -ub 1024 --ctx-size 65536(since the 2026-11-22 carry of upstream #32740, F12; before:-b 512 -ub 512). Since the F13 deploy (2026-11-22,b13c41355): the experts of loop layers 6-9 stay on the GPU and all others on the host,-ot blk[.](0|1|2|3|4|5|10|11|12|13|14|15|16|17|18|19|20|21|22|23)[.]ffn_.*_exps=CPUinstead of--n-cpu-moe 24(k=4 of the F13 placement guide: 7246 MiB of device memory, 868 MiB left for the mmproj). TheGGML_OP_OFFLOAD_MIN_BATCH=64env of both entries was removed: F13 makes the 32-63-token ubatches correct (KLD in F13). Other choices from the guide: ctx 32768 with k=6, or ctx 131072 with k=0 (fits since F13, 134 MiB free, so images would spill). Smaller quants (UD-Q3_K_XL, UD-IQ3_XXS,/root/models/inkling/) were measured and not adopted (F13, Quants). Its nothink macro uses--reasoning-effort none(no--reasoning-budget 0: that budget made it loop on chat-structure tokens, F12); even with effort none it writes a short reasoning block. Against Qwen3.6-35B (think entries, 7 tasks, one sample each, 3500-token cap, 2026-11-22,/root/arms/overnight/quality.py,quality/quality.json): Inkling 6/7, Qwen 5/7. Inkling found all three bugs of a binary search where Qwen ran out of budget; both ran out on a 4-line Polish rhyme. Inkling used about a third fewer tokens (9677 vs 14254) but took 1.7x the wall time (2283 vs 1316 s; decode 4-6 vs 13 t/s) and 4x on a 29K-token question (835 vs 211 s). Too small a sample to call it smarter; it is clearly slower here.
Environment, set once for all of llama-swap in [program:llama-swap] environment= of
/root/supervisord.conf:
LLAMA_SLEEP_VRAM_ONLY=1: enables the arbiter and VRAM-only sleep (F6).LLAMA_VRAM_ARENA=/dev/shm/llama-vram: the shared arena. Every co-loaded GPU model must use it.LLAMA_SLEEP_EVICT_KV=1: cold models also free KV, recurrent state, scheduler and scratch (F4), so the warm model gets nearly the whole card. Costs a 2-3 s re-warm instead of ~0.6 s.LLAMA_MMPROJ_ONDEMAND=1: encoder in VRAM only during encodes (F5). No-op for models without an mmproj.- No
--sleep-idle-seconds: release is purely demand-driven via want.lock. GGML_CUDA_REGISTER_HOST=1may still be present from the pinning era; it is a no-op with-lm none.
Not set, on purpose: GGML_VK_DISABLE_GRAPH_OPTIMIZE, GGML_SCHED_PREFETCH_EXPERTS,
GGML_VK_ASYNC_USE_TRANSFER_QUEUE, GGML_VK_CONCAT_FASTPATH, --moe-cache-mib.
--cache-disk-path /root/prompt-cache --cache-disk-size 4096 (F11) is set in the moe_offload_args, moe_offload_gemma and inkling_args macros.
Machine-level:
- GPU power cap fixed at 120 W by
/root/fan-control.sh(a separate supervisord program; re-asserted every second; default/max cap 145 W). The two GPU hangs of 2026-10-09 happened before the cap was fixed (it used to switch between 120 and 145 W by temperature); heavy runs since have been clean. - Kernel argument
amdgpu.lockup_timeout=0,120000,0,0(compute ring 120 s) via Talosmachine.install.extraKernelArgs. Without it long 64k-context submissions hit the compute-ring watchdog (ring comp_1.x.0 timeout). - Upstream changed the default
--portfrom 8080 to 9931; the config passes--port ${PORT}explicitly, so it is unaffected.
6. Measurement methodology and traps
- Cross-run variance is about 7%. llama-bench's within-run error bars understate cross-invocation variance by about 14x (the same config read 276.7 +/- 0.9 and 296.4 +/- 0.6 in two invocations). Put every comparison you draw a conclusion from inside one llama-bench invocation (comma-separated sweeps). Two builds cannot share an invocation, so interleave them over at least 3 rounds and compare medians. Treat any cross-build delta under about 5% as unproven.
- Alternate arm order. A fixed order gave the later slot about a 9% advantage and nearly produced a false regression finding. Use orders like ref, cand, cand, ref.
- Node CPU contention. The node runs about 60 pods; llama-bench once got 4.67 of 12 hardware
threads while asking for 6. With
--n-cpu-moe 40every MoE layer decodes on the CPU, so stolen threads map onto tg almost directly. Prefill is nearly immune (held +-0.3% while tg32 swung +-14%). Expect occasional catastrophic tg samples; use medians, never means. Historical tg figures taken under different load are not comparable. - Never use tg32. At the default depth (n_ctx 32) decode is launch/barrier bound and measures
command-stream overhead: two upstream commits read -36% at tg32 and were at parity at tg256 (and +7%
at d512). Use tg256 (or tg128) with an explicit depth (
-p 0 -n 256 -d 4096). tg is not monotonic in n_ctx on this box (n_ctx 256 measured slower than 32 and 544), so only compare same-depth cells. -b Vulkanvs-b Vulkan0.test-backend-ops -b Vulkanmatches nothing, runs 0 cases and still prints OK with exit 0. The backend isVulkan0. Always check the case count.- ubatch larger than the prompt never fills. With
-ub 2048use prompt lengths that are multiples of 2048. A partial trailing ubatch costs a full expert upload. -tbis not a llama-bench flag (server only). Any bad flag makes llama-bench print usage and exit, which looks like a run that produced no rows. Always check the row count.-ncmoeis llama-bench's spelling of--n-cpu-moe.- llama-bench prints its table only after all
-dvalues finish. No output does not mean hung. - Nothing else may use the GPU during a measurement; stop llama-swap (
bench.shand/root/bench/moe/swapdown.shdo this and restart it from a trap). - Check the build finished before benchmarking it (library mtime earlier than the run start). Never overlap a build with a big benchmark (host OOM).
- Prompt-cache reuse is
timings.prompt_n, nottokens_evaluated(which is always the full prompt length). - Per-process VRAM from
/proc/<pid>/fdinfo/*drm-memory-vrammust be deduplicated bydrm-client-id; summing all fds double-counts shared buffers.drm-memory-gttis system RAM, not VRAM. Whole-card usage:/sys/class/drm/card*/device/mem_info_vram_used.GGML_VK_MEMORY_LOGGER=1logs every Vulkan buffer alloc/free with a running device total to stderr. - The concat fast path is invisible to test-backend-ops (F3); validate it with the knob, perf
logger timing, or byte-identical greedy generation (temperature 0, top_k 1, fixed seed,
cache_prompt: false). - Do not run
llama-clidetached with closed stdin on the pod: it can loop forever printing prompts (it once wrote a 65 GB log). Use llama-bench-dor the server harness.
7. Known gaps and open risks
llama_context::restore_device()ignores theboolreturned byrestore_device_weights(),moe_cache->restore_device()andmemory->restore_device_buffers(). On a restore OOM,device_resident()stays false, decode retries the restore on every call, and a graph reading a null buffer would crash. Only the mmproj path handles a failed restore.- Memory types without a
release_device_buffersoverride keep their KV underevict_kv(F4). - Inkling (F12, section 5): the nothink entry's chat-structure loop came from
--reasoning-budget 0and was fixed in the config on 2026-11-22 (F12). Upstream #32755 / #32778 did not reproduce here; F13 bounds the loop-pass expert copies (more in F13). Drop the F12 carries when upstream merges #32735 and #32740. - Open after the 2026-11-21 rebase: is the fork's concat fast path (F3) now redundant with upstream's
own tiled concat (#30591, gated in the fork with
ne0 < 32and theGGML_VK_CONCAT_FASTPATHknob)? Benchmark it. Doesn_inflightblocking the want-lock yield delay handoffs? Validation says no (handoff median 692 ms candidate vs 1068 ms reference). - Router mode can spin at shutdown (F7); not used here.
- The unexplained ~1.6 GiB peak-VRAM spike in the first candidate
--perfarm, seen twice (F9). - Not reported upstream: the
wait_until_no_sleep()lost wakeup and the sleep wait that ignored new tasks (both upstream bugs fixed in878d1338f), and the graph-optimize alias cost at tiny context. Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them. - F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate; its memory effect is verified in production.
- F11 (disk prompt cache): the same-reuse-point identity check and the production-command-line (mmproj
loaded) 35B run both passed on 2026-11-21 (
/root/arms/diskcache-window3.sh, F11 Validation), and the cache is on in production. Remaining gap: only the 35B and gemma were checked in production; other models gain the flag only if they use those macros. - A SIGTERM during an arbiter release with
LLAMA_SLEEP_EVICT_KV(F4, not F11) pays that release's host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time (token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that shadow while it is cold. - F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB.