FORK.md documents every fork feature (what and why with measured numbers, where, how it works, invariants, past failures, rebase hazards, validation), the dead ends measured on this hardware, the production configuration and the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation, deploy, rollback and GPU-recovery procedure plus an operations reference. The README fork section links both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
60 KiB
FORK.md - the RX 580 llama.cpp fork
Maintainer documentation for this private fork. Read it before you rebase, change, or deploy anything. The rebase and deploy procedure is in docs/fork-rebase-runbook.md, which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-10-25):
- Branch
fable5/vram-ondemand-rebase-20261024: upstream3f03b1772plus 27 fork commits. - Production runs
6c480f7e7(the first 25 commits). The 2 commits after it,e085c347d(concat env knob) and149a8cb0a(scripts/fork-validate.sh), are validation-only and were not deployed. - Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line (
git log --oneline origin/master..HEAD). The commit messages are detailed and are the primary record of each mechanism and measurement; read them withgit show.
Contents
- Purpose and target
- Commit map
- Features
- F1 MoE routing-ids readback skip
- F2 GCN flash-attention mask_opt gate (and the shmem-staging note)
- F3 Vulkan concat tiled-transpose fast path
- F4 On-demand device residency (library side)
- F5 On-demand mmproj (multimodal encoder)
- F6 Cross-process VRAM arbiter (server side)
- F7 Server shutdown path
- F8 Cold-teardown guards
- F9 Benchmark harness and validation script
- Dead ends
- Configuration that matters
- Measurement methodology and traps
- Known gaps and open risks
1. Purpose and target
The fork exists to serve large MoE models well on one specific machine:
- GPU: AMD RX 580 8 GB (Polaris, GCN gfx803), RADV Vulkan driver. llama.cpp reports
fp16: 0(RADV does not exposeshaderFloat16on GFX8), no int-dot, no matrix cores, wave64. PCIe 3.0 x16. RADV exposes no transfer-only queue family on this card. - Host: Ryzen 5 3600 (6c/12t), 62 GB RAM, a Talos Kubernetes node shared with about 60 other pods (CPU contention matters for decode numbers, see section 6). The GPU runs at a fixed 120 W power cap.
- Primary model: Qwen3.6-35B-A3B (unsloth UD-Q4_K_XL, 20.8 GiB), a hybrid MoE: 40 layers, 30 of
them gated delta-net (recurrent), 10 full attention, head_dim 256, 256 experts with 8 used. All
experts live in host RAM (
--n-cpu-moe 40); only the backbone, KV, recurrent state and compute buffers sit in VRAM. - Serving: llama-swap under supervisord keeps several llama-server processes loaded at once (the 35B main models, a Qwen3.5-4B "task" model OpenWebUI calls for titles and tags, gemma-4-26B multimodal, others). They cannot all fit in 8 GB, so they time-share the GPU through the fork's VRAM arbiter (F4-F7) without restarting processes, which would throw away the RAM-only prompt cache.
- Priority: prefill (prompt processing) speed first, decode second. Agentic use pushes large tool results through the model, so long-context prefill is what users wait on.
What the fork adds, in short:
- three Polaris prefill wins (F1, F2, F3), all exact (token-identical to mainline);
- on-demand release/restore of everything a model holds in VRAM, plus a cross-process arbiter that hands the GPU from one llama-server to another in well under a second for small models;
- fixes to the server's sleep, wake and shutdown paths that the arbiter exposed;
- a benchmark harness and a post-rebase validation script.
2. Commit map
Oldest first. "Feature" points at the section below.
| commit | subject | feature |
|---|---|---|
c72cb424e |
llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert | F1 |
fb6a8a076 |
ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes | F2 |
ff2a96d2d |
vulkan: record why flash-attn shmem staging stays off for GCN | F2 |
3df719e1e |
ggml-vulkan: tiled transpose fast-path for concat with transposed source | F3 |
481be8b28 |
server: on-demand VRAM sharing to time-share one GPU between models | F4, F6 |
17165fe2a |
server: optionally evict the KV cache too (Phase 2 of VRAM sharing) | F4 |
958257cd6 |
server: keep the sleep-wake path active when the VRAM arbiter is enabled | F6 |
b9df854de |
server: coordinate model load with the VRAM arbiter to avoid load-time OOM | F6 |
8d80f0673 |
llama: guard buffer iteration against released device buffers | F8 |
2feb6aeb2 |
server: restore weights/KV on wake, not at decode, so KV eviction is safe | F6 |
bf2c1a699 |
server: also free the compute-graph scheduler on KV eviction | F4 |
9857a45f0 |
llama: evict recurrent/SSM state on device release | F4 |
9b555f1aa |
ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc | F4 |
ad27d4921 |
llama: free backend compute scratch on cold; guard memory_breakdown | F4, F8 |
0a0f3ab6d |
mtmd: release/restore the encoder weights from VRAM on demand | F4, F5 |
5e99ac198 |
server: on-demand mmproj - free encoder VRAM on the text path | F5 |
b7c131219 |
docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends | docs |
5e429d4a9 |
ggml-vulkan: skip the concat transpose fast path below one tile | F3 |
4c4486f8e |
scripts: add the RX 580 benchmark harness | F9 |
878d1338f |
server: replace the VRAM doorbell with a want-lock | F6 |
53f907771 |
llama: skip the compute-buffer size check when the context is cold | F8 |
7ac2014a8 |
llama: release/restore the MoE cache with the device weights | F4 |
54a7f534b |
scripts: rx580-bench - use -lm none, document the MoE cache option | F9 |
11044b35e |
server: run the shutdown handler off the signal-interrupted thread | F7 |
6c480f7e7 |
server: release the GPU token promptly on shutdown | F7 |
e085c347d |
ggml-vulkan: env knob for the concat transpose fast path | F3 |
149a8cb0a |
scripts: add fork-validate.sh, post-rebase validation of the fork | F9 |
3. Features
Every feature section uses the same headings: What and why; Where; How it works; Invariants that must hold; Failure modes and incidents; Rebase hazards; Validation.
F1. MoE routing-ids readback skip
What and why. With experts host-resident, each split uploads the experts it needs. To choose them, the expert-copy callback reads the routing ids back to the host. The ids are produced on the same GPU the upload goes to, so the readback forces a full pipeline flush: 47 per eval on a 48-layer MoE. At prefill batch sizes every expert is used anyway (2048 tokens x 10 experts over 256 experts), so the readback buys nothing. Measured on Laguna-S-2.1 118B IQ2_M, all experts on host (before it was ported to the current callback): pp2048 +4.7% at depth 32k, +0.9% at 16k, flat at depth 0, tg +3%.
Where. src/llama-context.cpp, llama_context::sched_copy_experts (registered with
ggml_backend_sched_set_copy_callback in the scheduler setup). Commit c72cb424e.
How it works. When ggml_nelements(ids) >= 4 * n_expert, it skips
ggml_backend_tensor_get_async + ggml_backend_synchronize, clears st.ids_data and sets
st.used.assign(n_expert, true). The existing grouping then copies the whole expert tensor as one
range. Below the threshold the upstream readback runs unchanged. For Qwen3.6-35B (256 experts, 8
used) the skip applies to any ubatch of 128 tokens or more; decode (8 ids) never reaches it.
Invariants that must hold.
- Exactness: uploading an expert no id selects cannot change the result, because
mul_mat_idonly reads the rows the ids point at. Any upstream change that makes unselected experts matter (for example a kernel that reads every uploaded expert) breaks this argument. - With
--moe-cache-mibon, the all-true set still goes throughmoe_cache->copy_experts, which reads the host slot maps (no device readback) and copies cached experts device-to-device. Still exact. st.idscaching stays coherent: a repeat call with the same ids tensor reuses the all-true set, which is what the threshold would decide again; a differentn_expertrecomputes.- The small-batch MoE-cache slot path (
llama_moe_cache::copy) has its own readback and is not touched.
Failure modes and incidents. None in production. Historically the change lived in
ggml_backend_sched_compute_splits in ggml/src/ggml-backend.cpp; upstream #29943 (6753a033f)
deleted that block and moved expert copying into llama_context::sched_copy_experts. The original
cherry-pick therefore had nothing to apply to and was re-authored as c72cb424e. A -X theirs
resolution would have resurrected the deleted scheduler block.
Rebase hazards.
- Any change to
sched_copy_expertsor theststate struct (ids,used,ids_data). - Upstream
701fd25b3(MUL_MAT_ID id -1 skips work) left aGGML_ASSERT(id >= 0 ...)in the readback branch; the skip branch reads no ids, so it is unaffected, but watch for upstream semantics around negative ids. - If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.
Validation. No dedicated check. fork-validate.sh --ops covers the MUL_MAT_ID kernel (not this
callback); --perf catches a prefill regression; --moecache exercises the cache path through this
callback. After a rebase that touches this function, also compare greedy output against the reference
build on the 35B.
F2. GCN flash-attention mask_opt gate (and the shmem-staging note)
What and why. mask_opt skips fully-masked causal blocks and drops the per-block mask load+add
on fully-visible blocks. Upstream disables it on AMD GCN for head sizes <= 256. On Polaris attention
at depth is compute-bound on the softmax path (exp runs at about 1/4 VALU rate, no matrix cores), so
skipping that work is a real win. Measured on Qwen3.5-35B-A3B (head_dim 256): pp2048 +1% at depth 0,
+7.8% at 16k (176 -> 190), +12.3% at 32k (136 -> 153), growing with depth. Perplexity bit-identical
(5.1668 on and off).
Where. ggml/src/ggml-vulkan/ggml-vulkan.cpp, ggml_vk_flash_attn, the use_mask_opt
condition. Commit fb6a8a076. The shmem-staging note is a comment in get_fa_tuning_params_scalar
(commit ff2a96d2d, no behaviour change).
How it works. The GCN clause HSK > 256 || HSV > 256 becomes HSK >= 256 || HSV >= 256. The
existing large-mask conditions (nem1 >= 32, nem0 * nem1 > 32768,
nem0 >= block_cols * 16) keep it off for decode.
Invariants that must hold.
- Only GCN with head size >= 256 is affected. Do not lower the threshold: relaxing it below 256 was -18.5% at 16k on Laguna (head_dim 128).
- Upstream's
!use_sparseterm must stay in the condition (sparse flash-attention came in with upstreamfc82583e6). shmem_stagingstays NVIDIA-only. Measured on GCN at head size 128: -6.7% at 16k, -7.4% at 32k. Thekvshstride ofD/4+1dwords is 4 mod 32, an 8-way LDS bank conflict on wave64 (the+1padding is tuned for warp32), and the extra shared memory cuts occupancy. If anyone retries it, fix the padding to +2/+3 first.
Failure modes and incidents. A -X theirs resolution in a trial rebase silently dropped
!use_sparse from the condition. No production incident.
Rebase hazards. Any upstream rewrite of use_mask_opt, of the FA tuning parameters, or of
architecture detection (vk_device_architecture::AMD_GCN). Upstream 733f57a99 added a path that
copies strided f16/bf16 K/V into prealloc_x scratch for flash-attention on AMD non-RDNA4 (so GCN
is included) when neq1 >= 256; measured at d32768 it adds about 68 MiB of VRAM and no visible
long-context prefill loss, but it has no env toggle, so its speed effect could not be isolated.
Validation. fork-validate.sh --ops (FLASH_ATTN_EXT, 5332 cases on Vulkan0) and --perf
(pp2048 at depth 16k and 32k is where this feature shows).
F3. Vulkan concat tiled-transpose fast path
What and why. Delta-net layers build their conv input with ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0). The generic concat shader reads the transposed source with an
uncoalesced stride: about 5.6 ms for a 16 MB concat on the RX 580 against about 130 us of memory
bandwidth. On Qwen3.6-35B the fast path alone is +14% pp8192 and accounts for more than all of the
fork's prefill win at depth 0 (the rest of the fork costs about 1% there). In the validation run on
Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 1945 us without.
Where. ggml/src/ggml-vulkan/ggml-vulkan.cpp: ggml_vk_concat (gate lambda
src_transposed_2d and the s1_trans/s0_trans checks) and ggml_vk_concat_transpose_fastpath.
The graph pattern comes from llm_build_delta_net_base::build_conv_state in
src/models/delta-net-base.cpp (qwen35, qwen35moe, qwen3next); qwen4exp's build_conv_state_at uses
the same pattern. Commits 3df719e1e (fast path), 5e429d4a9 (decode gate), e085c347d (env knob).
How it works. For a concat along dim 0 where one source is stored transposed (nb[1] == type_size, nb[0] == ne[1] * type_size) and the other source and dst are contiguous, it issues two
dispatches: pipeline_cpy_32/_16 copies the contiguous source into its dst columns, and
pipeline_cpy_transpose_32/_16 (the shared-memory 32x32 tiled transpose in copy_transpose.comp)
writes the transposed source into the other columns, both with custom push constants and a dst
offset. It ends with ggml_vk_sync_buffers. Everything else falls through to the generic path.
The gate requires: f32 or f16 with all three types equal (quantized and integer types stay generic);
matching ne[1..3]; ne[2] == 1 && ne[3] == 1 on the transposed source (so it fires only with one
sequence per ubatch, which --parallel 1 gives); the 16-bit doffset bound; and
ne[0] >= 32 (TILE_DIM) on the transposed source.
The decode gate (5e429d4a9): at one token the transposed source is a single column, so there is no
stride to fix, but the path still costs two dispatches and the unconditional ggml_vk_sync_buffers,
a full pipeline barrier (the generic path emits none). That barrier, not the half-empty transpose,
was nearly all of a -17% tg256 regression, because the path ran 30 times per decoded token on the
35B. The transposed source is qkv_mixed transposed, so its ne[0] is the ubatch token count:
prefill keeps the fast path, decode and small speculative drafts take the generic one. Measured
with the gate: decode back to parity (gated/clean 1.023), prefill win retained (gated/ungated 0.998,
pp8192 +11% over clean), generation byte-identical.
The knob (e085c347d): GGML_VK_CONCAT_FASTPATH=0 forces the generic path, =log prints
ggml_vk_concat: transpose fast path, transposed ne0=N to stderr for every fast-path dispatch. It is
read once (function-local static). Not set in production.
Invariants that must hold.
- The fast path fires at prefill and never at decode on a delta-net model.
- Output is identical to the generic path (it is a pure copy/transpose).
- The
types_okgate keeps quantized types off the path: the dispatches are raw f32/f16 element copies.
Failure modes and incidents.
- The decode regression above shipped in the deployed fork before the gate existed. It looked tg-neutral on the old upstream only because tg32's +-14% spread hid a 17% effect (section 6).
- If upstream changes
build_conv_state(for example aggml_conton the transpose), the fast path silently stops firing. Nothing crashes; prefill just loses about 14%. Only the concat check or a perf A/B notices.
Rebase hazards.
- Pipeline renames: upstream
733f57a99renamedpipeline_cpy_f32_f32/pipeline_cpy_f16_f16topipeline_cpy_32/pipeline_cpy_16. The pick applied cleanly and did not compile. - Upstream
f172be756madeggml_vk_concatnon-static (declared inggml-vulkan-common.h) and split helpers into headers; the fast-path helper stays static. - Shape changes in
build_conv_state/qkv_mixed([ch, n_seq_tokens, n_seqs]reshaped from a contiguousbuild_lora_mmoutput), changes tocopy_transpose.comporcopy.comppush constants, or tovk_op_unary_push_constants/get_misalign_bytes. - Upstream concat now handles quantized types; keep
types_ok. - The stock
test_concatcases intests/test-backend-ops.cppnever produce a transposed source and always havene[2]/ne[3]> 1, sotest-backend-ops -o CONCATdoes NOT exercise this path.
Validation. fork-validate.sh --concat (needs a build with the knob, i.e. this branch or
--concat-build): counts fast-path dispatches in a prefill run (must be > 0, all with ne0 >= 32)
and a decode run (must be 0 while CONCAT nodes did execute), then ties the unmodified candidate to
the fast path by CONCAT timing (candidate about equal to knob-on and at least 2x faster than
knob-off). Last run: 72 dispatches at prefill (all ne0=512), 0 over 594 CONCAT nodes at decode.
Also --perf.
F4. On-demand device residency (library side)
What and why. Several always-loaded models must share 8 GB. Co-residence is impossible (the 35B
at --n-cpu-moe 40 uses most of the card; the 4B task model needs about 2.5-3.3 GB). Restarting a
process to free VRAM throws away the RAM-only prompt cache: on Laguna a repeat turn went from a
21,960 ms re-prefill to 225 ms once the process stayed alive across a task-model call. So a model
must be able to drop everything it holds in VRAM and get it back without a reload.
Measured footprints:
- 4B: warm 3327 MB, weights released 724 MB, restored and correct.
- 4B at 32k ctx: cold weights-only 1750 MB (KV stays), cold with KV evicted 726 MB.
- gemma-26B: warm 7147 MB, cold with weights+KV but scheduler kept 718 MB, cold with the scheduler freed too 51 MB.
- A cold 4B before the encoder work held 644 MB that turned out to be its mmproj (F5).
- Re-warm cost: about 0.6 s per cold-to-warm switch weights-only (tiny prompt: 1.98 s cold vs 1.40 s warm); 2-3 s with KV eviction (it adds the KV D2H/H2D and a scheduler reserve).
Where.
src/llama-model.cpp/.h:llama_model::release_device_weights(keep_shadow),restore_device_weights(),weights_resident(); per-buffer metadatallama_model::impl::weight_release(weight_release_slot), filled inload_tensorsandload_tensors_hadamard.src/llama-memory.h:llama_memory_i::release_device_buffers(keep_shadow)/restore_device_buffers()(default no-op). Implemented insrc/llama-kv-cache.cpp(llama_kv_cache) andsrc/llama-memory-recurrent.cpp;llama_memory_hybridandllama_kv_cache_iswadelegate to their children.src/llama-moe-cache.cpp/.h:llama_moe_cache::release_device(),restore_device(),resident(),impl::invalidate().ggml/include/ggml-backend.h,ggml/src/ggml-backend-impl.h,ggml/src/ggml-backend.cpp: optionalggml_backend_i::free_scratchandggml_backend_free_scratch(); Vulkan implementationggml_backend_vk_free_scratchinggml-vulkan.cpp.src/llama-context.cpp/.h:llama_context::release_device(evict_kv),restore_device(),discard_device(),device_resident(), memberskv_device_evicted,device_discarded; the auto-restore at the top ofllama_context::decode.include/llama.h:llama_context_release_device(ctx, evict_kv),llama_context_restore_device,llama_context_discard_device,llama_context_weights_resident.tools/mtmd/clip.cpp/.h,tools/mtmd/mtmd.cpp/.h:clip_release_device,clip_restore_device,clip_weights_resident,mtmd_release_device,mtmd_restore_device.- Commits
481be8b28,17165fe2a,bf2c1a699,9857a45f0,9b555f1aa,ad27d4921,0a0f3ab6d,7ac2014a8,6c480f7e7(discard).
How it works. Tensor metadata (ggml_context structs) is durable; device buffers are ephemeral.
- Weights: on release, for each releasable buffer, capture a host shadow once (weights are
read-only), free the buffer, null every tensor's
buffer/data. On restore,ggml_backend_alloc_ctx_tensors_from_buftreallocates an identical layout (re-pointing the tensors), the shadow is uploaded, usage is set to WEIGHTS. Only single-buffer, alloc-path, non-host buffers are releasable: the host-pointer path (mmap or lazy-mapped),no_allocdummy buffers and host (Vulkan_Host, i.e. pinned RAM) buffers stay as they are. - KV and recurrent state: same pattern, but the shadow is re-captured on every release (the data is read-write) and dropped after restore.
- Scheduler: with
evict_kv,sched.reset()frees the worst-case compute buffer andsched_need_reserve = true; restore callssched_reserve()(which goes throughsched_new(), re-registering the expert-copy callback and upstream's realloc callback). - Backend scratch: with
evict_kv,ggml_backend_free_scratch()on each backend. On Vulkan this discards the pending compute context, synchronizes, destroysprealloc_x,prealloc_y,prealloc_split_k,prealloc_add_rms_partials,sync_staging, and zeroes their sizes and theprealloc_y_last_*tracking. They are reallocated lazily on the next compute. Other backends leave the hook null (no-op). - MoE cache (
--moe-cache-mib, upstream #29887): released in both modes. The banks are freed and the cache is emptied, not shadowed: every SLRU is rebuilt fresh, itsslot_map[il]pointers are re-pointed at the host slot-map tensors, and the host slot maps are cleared to 0xff (every slot -1). On restore the banks are reallocated (which also re-inits their views), cleared to 0 and marked WEIGHTS; the cache refills from the host experts. - Multimodal encoder:
clip_release_device/clip_restore_deviceuse the weights pattern (shadow once); no-op for a CPU-backed encoder.mtmd_*wrap the vision and audio contexts. llama_context::release_device(evict_kv): synchronize, release weights, release MoE cache, and ifevict_kv: release memory buffers, free sched, free scratch.restore_device(): weights, MoE cache, memory buffers,sched_reserve()if needed, thenggml_backend_sched_synchronizeso uploads finish before any compute.discard_device(): likerelease_device(true)withkeep_shadow = falseeverywhere (no host copies), setsdevice_discarded; used only before exit.decode()callsrestore_device()if!device_resident(), as a safety net. The server restores explicitly on wake (F6), because some KV access happens before decode.
Invariants that must hold.
- Shadows are stored compactly in stable tensor-iteration order, skipping view tensors
(
view_src != NULL), and restored with the same iteration. Never index a shadow byt->data - base. weight_releasestays parallel topimpl->ctxs_bufs: every push toctxs_bufsneeds a matching push toweight_release(non-releasable slots useemplace_back()).- The caller synchronizes before any release;
release_devicedoes this itself. - Restore order: weights, MoE cache, KV/recurrent, scheduler reserve, scheduler synchronize. The MoE cache banks must be back before the scheduler is rebuilt.
- After a MoE-cache release no slot map may point at an expert: banks come back zeroed, so a stale slot map silently reads zero weights.
- Every iterator over
ctxs_bufs(model, KV cache, recurrent) must tolerate a null buffer, and every use ofschedmust tolerate null while cold withevict_kv(see F8). device_resident()covers weights and MoE cache; it gates the early returns, the decode auto-restore andllama_context_weights_resident().restore_device()afterdiscard_device()aborts (GGML_ASSERT(!device_discarded)).- Greedy output is byte-identical across any number of release/restore cycles, and the prompt cache
survives (
timings.prompt_ndrops on a repeated prompt after a cycle).
Failure modes and incidents.
- The first shadow implementation indexed by
t->data - baseand produced garbage after restore (layout and padding differ). Fixed by the compact iteration-order shadow. - Restoring KV lazily at decode crashed iSWA models (gemma) with
GGML_ASSERT(buf != NULL && "tensor buffer not set"):update_slots()creates an SWA checkpoint that reads the KV viaggml_backend_tensor_getbefore decode. Fixed by restoring on wake (2feb6aeb2). - The recurrent state was left resident (default no-op), 201 MB on the 4B, until
9857a45f0. - The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed
resident in a cold model until
bf2c1a699/ad27d4921. - The MoE cache was not released at all before
7ac2014a8. Reallocating its banks without invalidating would have produced silent wrong weights. - In the 20261024 rebase
moe_cache_slru::init()turned out to reset only the probation segment (head[1],tail[1]andn_protectedsurvive), so re-running it on a used cache corrupts the lists.invalidate()now assigns a freshmoe_cache_slru()beforeinit().
Rebase hazards.
- New push sites into
ctxs_bufs(upstream'sload_tensors_hadamard()added two in the 20261024 rebase; without matchingweight_releaseentriesrelease_device_weights()reads past the end). - Changes to buffer creation in
load_tensors(thehost_ptr_pathcondition,ctx_bufs/buf_ptrsbookkeeping; upstream replacedctx_buf_maps.emplace_back(ctx, buf_map)withctx_bufs.emplace_back(ctx, std::move(buf_ptrs))in the 20261024 rebase). - The
llama_memory_iinterface. New memory types that do not overriderelease_device_bufferskeep their KV resident underevict_kv; at the time of writing that isllama_kv_cache_msa,_dsa,_dsa_iswa,_dsv4andllama_memory_hybrid_iswa. - MoE-cache internals (
groups,slru,slot_map,buf_host,devices). Upstream33d6135cbchanged LRU to SLRU (g.lru->g.slru); the pick applied cleanly and did not compile. llama_contextmembers next to ours (upstreamn_reallocs_same/diff,copy_experts_info),sched_reserve()semantics, and thedecode()signature (nowdecode(const llama_batch_ext &)).ggml_backend_ilayout (free_scratchmust stay the last member aftergraph_optimize; other backends rely on zero-init) and the Vulkan context's prealloc field set (compare withggml_vk_cleanup).
Validation. fork-validate.sh --arbiter: b-greedy-identical (15 cycles each side equal to a
no-handoff reference), b-kv-survives (prompt_n drops after evict/restore, output identical),
c-image-after-restore. --moecache MODEL (needs a cache large enough to hit; 256 MiB on gemma-26B
had 0 hits and proves nothing, 2048 MiB had 6651 hits). The script does not check that VRAM actually
drops on release; check that by hand with cat /sys/class/drm/card*/device/mem_info_vram_used
before and after forcing a release.
F5. On-demand mmproj (multimodal encoder)
What and why. A multimodal model's vision encoder sat in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the encoder lives in RAM and enters VRAM only for an encode. Measured on the
35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered
correctly, text after the image correct.
Where. tools/server/server-context.cpp: encode_mtmd_chunk (upstream renamed it from
process_mtmd_chunk), the mtmd_release_device(mctx) after mtmd_init_from_file in load_model,
vram_go_cold, vram_share_exit. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
How it works. After load the encoder is released (shadow captured). Before
mtmd_batch_encode, mtmd_restore_device(mctx); if that fails (out of VRAM) and on-demand is set, it
releases the LLM backbone weights only (llama_context_release_device(ctx_tgt, false): read-only
shadow, so a cheap free with no D2H, KV and prompt cache untouched) and retries. After the encode it
releases the encoder FIRST, then restores the weights, so the peak stays within VRAM.
Invariants that must hold.
- Free the encoder before re-uploading the weights.
- The donor is the weights, never the KV (the KV is read-write and would need a D2H every time).
vram_ensure_warm()does not restore the encoder; only the encode path does.
Failure modes and incidents. Before this, a cold text model with an auto-downloaded mmproj
(-hf pulls one when the repo ships it) held 644 MB that no release path touched. The text-only 4B
task model now runs with --no-mmproj, which is the right fix for any model that never sees images.
Rebase hazards. Renames or restructuring of encode_mtmd_chunk and the mtmd init block in
load_model; clip_ctx fields (buf, ctx_data, sched, no_alloc).
Validation. fork-validate.sh --arbiter case c-image-after-restore (server A runs with
--mmproj and LLAMA_MMPROJ_ONDEMAND=1; a 128x128 red PPM must be answered "red" after A was cold,
and text right after must equal the reference). Post-deploy: one image request to gemma-26B.
F6. Cross-process VRAM arbiter (server side)
What and why. Several llama-server processes under llama-swap share one GPU: exactly one is warm
(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production:
60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between
the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last
fork-validate.sh run: 30 handoffs, median 680 ms, max 1231 ms.
Where. tools/server/server-context.cpp: vram_arena_open, vram_acquire_for_load,
vram_share_init, vram_should_yield, vram_want_acquire, vram_want_release, vram_take_token,
vram_ensure_warm, vram_go_cold, vram_share_exit, vram_share_shutdown, the VRAM-only branch of
handle_sleeping_state, the vram_ensure_warm() call before decode in the slot update path, and
server_res_generator (bypass_sleep). tools/server/server-queue.cpp/.h: on_should_yield,
start_loop (should_sleep, idle tick, post-wake grace), wait_until_no_sleep, is_stopping.
Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
How it works.
- Enabled by
LLAMA_SLEEP_VRAM_ONLY(any value). The arena directory isLLAMA_VRAM_ARENA(default/dev/shm/llama-vram) with two files. token.lock:flock(LOCK_EX)is the baton. Rule: a process has device buffers iff it holds it.want.lock: a waiter holds itLOCK_SHwhile waiting and drops it once it owns the token. The holder probes it withflock(LOCK_EX | LOCK_NB)on a SECOND fd: failure means somebody is waiting. The probe needs its own open file description, because flock on the waiter's own fd would convert that lock instead of conflicting with it. The kernel drops a dead waiter's lock, so nothing goes stale.- Load:
load_model()callsvram_acquire_for_load()beforecommon_init_from_params(), so the current holder releases before this model uploads anything. The loaded model stays warm holding the token and serves its first request without a re-warm. - Yield:
vram_share_init()registersvram_should_yieldwithqueue_tasks.on_should_yield.start_loop()'sshould_sleep()calls it on the loop thread, undermutex_tasks, only when the task queue is empty andupdate_slotshas returned. While the callback is set the idle wait ticks every 10 ms instead of 1 s. After a wake the server holds the GPU for at least 100 ms (yield_grace_ms). Sleep runs the sleeping callbacks; in VRAM-only modehandle_sleeping_state(true)callsvram_go_cold()(release device, also the encoder; thenflock(LOCK_UN)) instead of upstream's full unload. - Wake: a request calls
wait_until_no_sleep().handle_sleeping_state(false)callsvram_ensure_warm(): take the token (register on want.lock, then polltoken.lockwithLOCK_EX | LOCK_NBevery 2 ms, aborting whenqueue_tasks.is_stopping()), restore ctx_tgt and ctx_dft, and only then setvram_cold = false. The decode path callsvram_ensure_warm()again (no-op when warm) and throws "server is shutting down" if it returns false. server_res_generatorno longer bypasses the wake path when--sleep-idle-secondsis unset ifLLAMA_SLEEP_VRAM_ONLYis set, because the arbiter can put a server to sleep without idle-sleep.- Queue fixes:
wait_until_no_sleep()loopswhile (sleeping && !stopping)and re-assertsreq_stop_sleepingeach time (withnotify_all); the sleep wait also wakes on a non-empty task queue; the idlewait_foronly leaves the loop on a real task or termination.
Invariants that must hold.
- Resident iff holding
token.lock.vram_cold = falseis set only AFTER the restore, so the loop cannot give the token away while still bringing weights up. - Release only happens from the idle branch of
start_loop(), on the loop thread, never insideyield_to_queue()or while a slot is generating: a holder finishes its current generation before it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both). - The yield decision reads live kernel state at the point of use. Do not route it through a flag set by another thread (see attempts 3-4 below).
- Every exit from
vram_take_token()drops want.lock, or the holder yields forever. - The probe fd is separate from the waiter fd.
flockcalls retry on EINTR: the server's signal handlers are installed without SA_RESTART.- Restore on wake, before
update_slots(), so KV eviction is safe for SWA checkpoints. - Every model that can be warm on the GPU at the same time must use the same arena. A non-arbiter process (a benchmark, a manually started server) can OOM the warm model. In production the env is set for all of llama-swap, so all its children participate.
Failure modes and incidents.
- Wake-path hang (
958257cd6): with no--sleep-idle-seconds,bypass_sleepskippedwait_until_no_sleep(), so a server slept by the arbiter never woke and requests hung. - Load-time OOM (
b9df854de): weights were uploaded before the arbiter was active; loading gemma while the 4B held VRAM could exceed 8 GB. After the fix the peak was 7771 MB. - The doorbell design (replaced by
878d1338f) was edge-triggered: a waiter wrote<arena>/doorbell/<pid>and the holder's warden thread turned the inotify event into a yield flag. Rings were lost in three windows: while the holder restored (it advertised cold and discarded foreign rings), while the holder loaded (the inotify watch did not exist yet, and the kernel does not queue events for a missing watch), and when a cold waiter consumed a third process's ring. Every loss was permanent:flock(LOCK_EX)had no timeout and production had no idle-sleep. The blocked flock heldmutex_tasks, so the whole server stopped answering while/healthstill returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by the second day). - Failed fix attempt 1: moving
vram_cold = falsebefore the restore made the holder honour stale rings and hand the token back 2-5 s after taking it without serving. Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. - Failed fix attempt 2: a re-ring loop (
LOCK_NB+ re-ring every ~1 s). Queued rings from an already satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become level-triggered. - Failed attempts 3-4: want.lock was sound (mutual exclusion, kill -9 safety, load-window coverage),
but the warden thread still delivered its decision as a flag.
start_loop()holdsmutex_tasksfromshould_sleep()through the callbacks to the wait, so the warden'srequest_yield()was always admitted after the flag had been consumed, latching a release for the next wake (a server woke and re-slept 138 ms later without serving). It also exposed the upstream lost wakeup inwait_until_no_sleep(). Result: 0.494 s median, 1/60 stranded. The fix deleted the warden andrequest_yield/yield_requestedand probes inline inshould_sleep(). - Measured deviations that are required: the 10 ms idle tick (about 0.2% of one core per warm server); a waiter that polls rather than retrying on a coarse grid (a 200 ms retry grid cost 0.503 s against 0.336 s, because the holder only releases after the waiter asks); the 100 ms post-wake grace (without it a permanent waiter caused a livelock: one request took 23.15 s over 138 sleep/wake cycles).
- Process lesson from attempts 1-2: validate an ordinary handoff latency A/B, not only the pathological case. Two production deploys were burned before that rule existed.
- Lost diagnostic: the old periodic "still waiting for the GPU token after N s" warning is gone; a
stuck token now shows one
VRAM arbiter: waiting for the GPU tokenline and then silence. Check/proc/locks(see the runbook).
Rebase hazards. This is the most fragile area, because it sits inside upstream's server loop.
server_queue::start_loop,wait_until_no_sleep,process_new_tasks,yield_to_queueand the sleeping-callback vector. Upstream refactored sleep (947fd9bb2) and made decode run insideyield_to_queue(22b8e310b); re-check after any such change that a release still cannot run under a live graph.handle_sleeping_state,load_modelordering, the slot update path where decode happens,server_res_generator'sbypass_sleep, renames inserver-context.cpp.- Anything that touches the KV in
update_slots()before decode.
Validation. fork-validate.sh --arbiter with two real servers on a scratch arena:
a-handoffs (30 alternating, 0 stranded, 0 non-200, the server that answered holds token.lock),
b-*, c-*, g-kill9-waiter (kill -9 a waiter registered on want.lock while the holder streams:
want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in
F7 and a log scan for asserts and Vulkan errors. In production, /proc/locks must show the warm
model holding WRITE on token.lock.
F7. Server shutdown path
What and why. llama-swap evicts models with SIGTERM, so shutdown happens all the time, often while the arbiter is mid-release or mid-wait. Two bugs made shutdown keep the GPU token and stall every other model.
Where. tools/server/server.cpp: signal_handler, shutdown_pipe, struct shutdown_thread,
sig_thread in llama_server. tools/server/server-context.cpp: server_context::start_loop (calls
vram_share_exit() after the loop returns), server_context::terminate (terminates the result queue
first), vram_share_exit, the destructor's vram_share_shutdown(). tools/server/server-queue.cpp:
server_queue::terminate (sets atomic stopping before taking the mutex), process_new_tasks,
server_response::recv_with_timeout, server_response::terminate,
server_response_reader::next. Commits 11044b35e, 6c480f7e7.
How it works.
- The signal handler only writes the signal number to a self-pipe (async-signal-safe); a dedicated
thread reads it and runs
shutdown_handler. The pipe fds areFD_CLOEXEC. Ifpipe()fails the old direct call is used. The thread object is declared after the server objects, so it is joined (a 0 byte unblocks it) before they are destroyed. A second signal still callsexit(1). terminate()setsstopping(atomic) before lockingmutex_tasks, then stops the result queue so HTTP handlers waiting for results get "server is shutting down" (HTTP 503, or an SSE error event);recv_with_timeoutchecks!runningbefore waiting and returns null.- A token wait polls with
LOCK_NBevery 2 ms and aborts onis_stopping(); the aborted wake stays cold, skips the restore, and the loop exits beforeupdate_slots. - After
start_loop()returns,vram_share_exit()frees all device buffers withllama_context_discard_device()(no host shadows), releases the encoder, and only then unlockstoken.lock.
Invariants that must hold.
- Nothing that takes
mutex_tasksruns in signal context. - The token is released only after this process's VRAM is free (the next model may start uploading the moment the lock drops).
- A shutting-down server never restores weights.
- SIGTERM in any state exits 0 within about 2 s and frees the token.
Failure modes and incidents.
- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside
a sleeping callback (under
mutex_tasks, during a release); the handler'sterminate()lockedmutex_tasksagain on the same thread. The process wedged holdingtoken.lock, every other model queued behind it, and only SIGKILL worked. Reproduced 6/6 off production; 8/8 clean with the fix. The first suspicion (the multimodal path) was wrong. gdb on the wedged process showed the main thread infutex(mutex_tasks)with theyield_to_queueworker idle. - Token kept on shutdown: with a request in flight the HTTP handler waited forever for results, main
blocked joining it, and the token was released only in the destructor after that join; with a
token wait in progress the blocking flock held
mutex_tasks; and the old destructor unlocked the token BEFOREdestroy(), so the next model could upload into still-occupied VRAM. - After the fix: off production SIGTERM mid-generation or mid-wait exits in about 0.2 s with exit 0 and frees the token in 5-70 ms; in production about 0.7-1 s.
Rebase hazards. server.cpp's signal setup and llama_server object lifetimes;
server_response and server_response_reader (upstream 4e201388c switched recv_with_timeout to a
fixed deadline: keep the order "check !running, return null, then wait_until(lock, deadline)");
server_context::start_loop/terminate. Router mode (server-models.cpp,
server_models_sse_client::next) only checks should_stop() and can spin at shutdown now that
recv_with_timeout returns null at once; llama-swap does not use router mode.
Validation. fork-validate.sh --arbiter: d-term-in-release (gdb breakpoint on
llama_context_release_device, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and
the token), e-term-waiting (exit 0 within 1.5 s, never took the token), f-term-streaming
(exit 0, SSE error event, token free within 2.5 s), h-shutdown-warm, h-shutdown-cold. Last run:
736 ms, 200 ms, 93 ms (token free in 28 ms), 80/157/156 ms. Post-deploy: a forced llama-swap eviction
must log VRAM arbiter: loop stopped, device memory freed, releasing the GPU token and exit by itself.
F8. Cold-teardown guards
What and why. A cold model has null buffers in its ctxs_bufs and, with evict_kv, a null
scheduler. Upstream code never had to consider either state.
Where. src/llama-context.cpp: llama_context::~llama_context (sched && in the
compute-buffer size check), llama_context::memory_breakdown (if (sched) around the compute part;
upstream's if (moe_cache) breakdown block stays outside it). Null-buffer skips in
llama_model::memory_breakdown, llama_kv_cache::{clear, memory_breakdown, total_size},
llama_memory_recurrent::{clear, memory_breakdown, total_size}. Commits 8d80f0673, ad27d4921, 53f907771.
How it works. Guards at the call sites. The asserts in ggml (GGML_ASSERT(buffer),
GGML_ASSERT(sched)) are correct and stay; every other caller reserves the scheduler first.
Invariants that must hold. A server shut down while cold exits 0. Any new upstream code that
iterates ctxs_bufs or calls into sched on a path reachable while cold (shutdown, memory
breakdown, stats) needs the same guard.
Failure modes and incidents.
GGML_ASSERT(buffer)on shutdown of a cold model (seen 7 times in production logs) from memory-breakdown/size/clear on released buffers.GGML_ASSERT(sched)inmemory_breakdownfor a coldevict_kvmodel being torn down.GGML_ASSERT(sched)in the destructor's size check: every cold shutdown exited 134 instead of 0, so under llama-swap an ordinary stop looked like a crashed child, left a backtrace in the log, and the rest of the destructor never ran. Repro: arbiter env on, one request, force a yield withflock -s <arena>/want.lock -c 'sleep 6', then SIGINT while cold.
Rebase hazards. New iteration over ctxs_bufs, new sched users in teardown or breakdown
paths. Watch item: llama_kv_cache::update() takes lctx->get_sched(), which can be null; it is
safe today only because its single caller runs after sched_reserve().
Validation. fork-validate.sh --arbiter: h-shutdown-cold (exit 0 while the other server holds
the token) and log-scan; --moecache shutdown.
F9. Benchmark harness and validation script
What and why. The measurement method on this box had to be rediscovered several times (section 6). The harness encodes it; the validation script encodes the checks that each past incident required, so a rebase can be validated by someone with no context.
Where. scripts/rx580-bench/ (run.sh workstation wrapper; bench.sh pod orchestrator;
ppbench.py corpus prefill through llama-server; lbsweep.sh llama-bench sweep; summarize.py;
README.md with methodology and reference values). scripts/fork-validate.sh. Commits 4c4486f8e,
54a7f534b, 149a8cb0a.
How it works.
rx580-bench: stops llama-swap, waits for an idle GPU, runs fixed slices ofpan-tadeusz.txt(4096/16384/32768 tokens, byte-identical across builds) through/completionwithcache_prompt: false, plus a llama-bench sweep, interleaved across arms and rounds; restarts llama-swap from an EXIT trap. Results append to/root/bench/results.txt.fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under /tmp, a scratch arena (refuses/dev/shm/llama-vram*), ports 8091-8093, unsets inheritedGGML_*/LLAMA_*env, kills only PIDs it started (PID-reuse safe). Default with no check flag is--ops --concat --arbiter. See the header of the script for what each check proves.
Invariants that must hold. Never benchmark with llama-swap running (--perf refuses, and any
other llama process blocks it). --perf gates prefill at -5% per depth and peak VRAM at 95% of the
card; decode is reported, not gated (CPU-bound and noisy here).
Failure modes and incidents. Upstream #28334 removed --mmap/--no-mmap/--mlock and
llama-bench's -mmp: the server refused to start and llama-bench silently produced no rows until
54a7f534b switched to -lm none. The perf and MoE-cache checks leave tens of GB of model files in page
cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024
deploy (see the runbook).
Rebase hazards. CLI flag renames in common/arg.cpp and tools/llama-bench; log line texts the
script greps for (server entering sleeping state, transpose fast path, transposed ne0=,
no longer waiting for the GPU token, released .* MiB of MoE cache, MoE cache size =,
llama_moe_cache: .* hits =, N/M tests passed, Backend Vulkan0: OK); the llama-bench markdown
table layout.
Validation. Known gaps of the script: it does not check that VRAM drops on release; peak VRAM is recorded per arm, not per test (one candidate arm once showed an unexplained 1.6 GiB spike that did not repeat); the SIGTERM-in-release breakpoint is at function entry, not after the host-shadow capture; nothing covers F1 directly.
4. Dead ends
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry without a new reason.
| idea | result | mechanism |
|---|---|---|
Weight pinning (GGML_CUDA_REGISTER_HOST and the host-register chain, from a third-party CUDA fork whose +64% was on an RTX 3060) |
+17% pp2048 in isolated llama-bench (252 -> 294), fails in serving | RADV cannot import file-backed mmap pages; the pre-stage fallback fails (ErrorOutOfDeviceMemory for the ~18 GB expert region) or doubles host RAM. -lm none serves faster (Qwen3.6-35B 278 -> 286). Dropped in the 2026-09-09 rebase. |
Expert-upload prefetch (GGML_SCHED_PREFETCH_EXPERTS) |
-20% pp (other runs -9% to -22%), no tg benefit | Slots allocate in host-pinned GTT, not VRAM, so it stages host to host and still pays the PCIe read. =1 means the default 3 slots. Dropped. |
GGML_VK_ASYNC_USE_TRANSFER_QUEUE=1 |
flat (-0.1%), byte-identical | GCN is excluded from prefers_transfer_queue and RADV exposes no transfer-only queue family here; the per-layer upload-then-compute order leaves nothing to overlap. A standalone Vulkan microbench did show 60-73% of an H2D copy on queue family 1 hidden behind compute, so double-buffered prefetch inside one backend context remains an untested idea (estimated +10-15%, unproven). |
Fractional -ncmoe |
no measured win | Config only uses integers (22/40/48); upstream refactored -ncmoe into llm_add_n_cpu_ffn_overrides. Dropped. |
MoE cache --moe-cache-mib 512 / 1536 / 2304 (Qwen3.6-35B, ncmoe 40, ctx 131072) |
tg256@d4096 -42% / +5% / +12%; prefill +1-3%; idle VRAM 6.15 / 7.19 / 7.96 GB vs 5.66 | Small caches thrash (8-of-256 routing over 40 layers) and add a per-layer ids readback; only ~2.3 GiB pays and leaves no VRAM for a long KV. Not enabled. |
| MTP speculative decoding, Qwen3.6-35B | decode -39% (n1), -37% (n2), -50% (n3), -44% (n4), best -19% (n2, p-min 0.75); prefill -3 to -6% | A spec step costs ~180 ms vs 52 ms for plain decode; a 2-token verify alone is only 1.4x a 1-token step, so ~100 ms/step is draft machinery. The nextn layer adds 1.25 GiB VRAM. Entries removed. |
| MTP, gemma-4-26B-A4B | decode -57.9% (n1), prefill pp4k -34% | Same mechanism (~210 ms vs ~50 ms per step); MTP needs --swa-full, which alone costs -10.5% decode, -32% pp4k and saturates VRAM. Entries removed. |
| MTP, gemma E4B | decode -10.9% | Acceptance 0.41-0.67. The +19.5% pp seen with it comes from --swa-full, not MTP. |
--parallel 1 on gemma |
no effect (-1.0%, +2.1%) | The Qwen gain is probably tied to its recurrent state being sized per sequence (unverified). |
-ub 4096 -b 4096 (Laguna) |
flat (90.0 vs 90.9) | Fewer expert uploads cancelled by intra-ubatch attention growth. |
mul_mat_id tile-padding theory (larger/odd -ub) |
predicted 2.00x waste and a 1.94x cliff; measured +3.8% and -5.6%, no sawtooth | Theory dead; kills the mul_mat_id tile-selection shader work (an unbuilt, unverified WIP exists on branch fable5/mmid-tile-wip, with an unresolved correctness concern in its row_ids fill). |
-b sweeps |
non-monotonic (2048/8192 high, 4096/12288 low) | -b 2048 is already at the top. |
-ncmoe below 40 on the 35B |
34: +2.9% pp but 7.23 GiB peak at tiny context; 28: better at short context, worse at depth (137 vs 170 t/s at d32k); <= 24 collapses (~122) | Resident experts are evicted by KV/activation growth; cannot survive a 128k context. |
-ncmoe 44 on Laguna |
+1.5% | Does not fit at 64k context. Keep 48. |
fp16 on Polaris (Mesa 26.1.6 + radv_enable_float16_gfx8=true, fp16: 1) |
pp2048 -4.7%, -5.3% at d16k, tg +4% | Hurts prefill (the priority). Mesa 26.1.6 alone: +3.3% prefill, below the 5% bar; adopting it needs a pod image change. The pod's Mesa 25.0.7 has no such option. |
| KV quant on Qwen (q4_0, q8_0) | f16 beats q4_0 by +40% pp at 64k; q8_0 170 -> 128 t/s at depth; q4_0 with upstream PR #30738 still -16% vs f16 at d32k (133.6 vs 159.4) | No int-dot/fp16 here, so dequant is pure compute; the hybrid model's KV is small anyway. Keep f16. (PR #30738 was +15% at d16k on gemma-26B with q4_0 KV; not merged upstream as of the 20261024 rebase.) |
-fa 0 |
5x slower at depth (170 -> 35 at d32k) | Keep flash attention on. |
FA scalar block_rows 16 -> 32 |
3.5x regression at d32k (135 -> 38) | ~42 KB shmem per workgroup at head_dim 256, one workgroup per SIMD, occupancy collapse. |
GGML_VK_DISABLE_GRAPH_OPTIMIZE=1 |
restores tg32 at n_ctx=32 (10.44 -> 15.61), nothing at real depth | Upstream #27812's view-alias fix makes KV views look dependent and collapses barrier-free batches; only visible when per-node work is tiny. It is a correctness fix; do not disable it in serving. |
--no-mmap/-lm none for Laguna-S 118B |
does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. |
Laguna --cache-reuse, --swa-full, -t 12, KV quant |
ruled out | --cache-reuse is force-disabled for iSWA when base and SWA sizes differ. |
CPU-only task model (-ngl 0) to avoid handoffs |
rejected | At --ctx-size 8192 titles could face 8k tokens of CPU prefill. The arbiter is the answer. |
The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256 tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.
5. Configuration that matters
/root/config.yaml (llama-swap) and /root/supervisord.conf on the pod are the source of truth;
this section explains the choices. Values as of the 20261024 deploy.
Qwen 35B-A3B MoE entries (macro moe_offload_args, applied to all 35B-A3B variants):
--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1,
context 131072.
-lm none: load weights into RAM (upstream removed--no-mmap; the old spelling makes the server refuse to start). Faster in serving than mmap and recommended for--n-cpu-moe. Exception: Laguna-S, which must use mmap (GTT limit, section 4).-ub 2048 -b 2048: the biggest prefill lever. ub512 = 97 t/s, ub1024 = 159, ub2048 = 254 on the 35B; the per-eval expert upload is amortized over the ubatch.--n-cpu-moe 40: all experts on host. Lower values are faster at short context but run out of VRAM at long context (section 4). Measured idle VRAM at ctx 131072: 5.66 GB.--parallel 1: +22.9% decode on the Qwen 35B (15.5 -> 19.1 t/s), prefill +0.3-1.2%, +64 MiB VRAM. Without it the server creates 4 slots and a unified KV. Two concurrent requests queue on slot 0. With one slot a ubatch never holds more than one sequence, which the concat fast path'sne[2] == 1condition (F3) needs.-t 6: six physical cores; more threads hurt decode.-fa 1: essential (section 4).- f16 KV (no
-ctk/-ctv): section 4.
Other models:
- gemma-4-26B-A4B: macro
moe_offload_gemma, same as above but--n-cpu-moe 22(8 layers of experts resident; gemma's SWA keeps KV small at depth). ncmoe sweep pp2048: 30 = 262, 24 = 266, 18 = 272, 14 = 157 (VRAM collapse); 22 keeps headroom for 128k context (7.40 of 8 GB at 128k). Its KV cache is-ctk q4_0 -ctv q4_0, set in thegemma4_sampling/gemma4_nothink_samplingmacros (not inmoe_offload_gemma); note gemma has no--parallel 1(no measured gain there). - Small models that fit in VRAM (Qwen3.5-4B, gemma E4B/E2B and similar) use auto-fit
(
common_args); forcing experts to host would slow them. - The 4B task model runs with
--no-mmprojand ctx 8192, and is preloaded. - Laguna-S-2.1 118B IQ2_M (configured as
Laguna-S-2.1-IQ2_M, matrix varlag): mmap,--n-cpu-moe 48 -b 2048 -ub 2048 -fa 1 --parallel 1, ctx 65536. Its exclusivity set must allow the task model (lag & q4task); making it exclusive with the task model killed it on every title request and forced full re-prefills.
Environment, set once for all of llama-swap in [program:llama-swap] environment= of
/root/supervisord.conf:
LLAMA_SLEEP_VRAM_ONLY=1: enables the arbiter and VRAM-only sleep (F6).LLAMA_VRAM_ARENA=/dev/shm/llama-vram: the shared arena. Every co-loaded GPU model must use it.LLAMA_SLEEP_EVICT_KV=1: cold models also free KV, recurrent state, scheduler and scratch (F4), so the warm model gets nearly the whole card. Costs a 2-3 s re-warm instead of ~0.6 s.LLAMA_MMPROJ_ONDEMAND=1: encoder in VRAM only during encodes (F5). No-op for models without an mmproj.- No
--sleep-idle-seconds: release is purely demand-driven via want.lock. GGML_CUDA_REGISTER_HOST=1may still be present from the pinning era; it is a no-op with-lm none.
Not set, on purpose: GGML_VK_DISABLE_GRAPH_OPTIMIZE, GGML_SCHED_PREFETCH_EXPERTS,
GGML_VK_ASYNC_USE_TRANSFER_QUEUE, GGML_VK_CONCAT_FASTPATH, --moe-cache-mib.
Machine-level:
- GPU power cap fixed at 120 W by
/root/fan-control.sh(a separate supervisord program; re-asserted every second; default/max cap 145 W). The two GPU hangs of 2026-10-09 happened before the cap was fixed (it used to switch between 120 and 145 W by temperature); heavy runs since have been clean. - Kernel argument
amdgpu.lockup_timeout=0,120000,0,0(compute ring 120 s) via Talosmachine.install.extraKernelArgs. Without it long 64k-context submissions hit the compute-ring watchdog (ring comp_1.x.0 timeout). - Upstream changed the default
--portfrom 8080 to 9931; the config passes--port ${PORT}explicitly, so it is unaffected.
6. Measurement methodology and traps
- Cross-run variance is about 7%. llama-bench's within-run error bars understate cross-invocation variance by about 14x (the same config read 276.7 +/- 0.9 and 296.4 +/- 0.6 in two invocations). Put every comparison you draw a conclusion from inside one llama-bench invocation (comma-separated sweeps). Two builds cannot share an invocation, so interleave them over at least 3 rounds and compare medians. Treat any cross-build delta under about 5% as unproven.
- Alternate arm order. A fixed order gave the later slot about a 9% advantage and nearly produced a false regression finding. Use orders like ref, cand, cand, ref.
- Node CPU contention. The node runs about 60 pods; llama-bench once got 4.67 of 12 hardware
threads while asking for 6. With
--n-cpu-moe 40every MoE layer decodes on the CPU, so stolen threads map onto tg almost directly. Prefill is nearly immune (held +-0.3% while tg32 swung +-14%). Expect occasional catastrophic tg samples; use medians, never means. Historical tg figures taken under different load are not comparable. - Never use tg32. At the default depth (n_ctx 32) decode is launch/barrier bound and measures
command-stream overhead: two upstream commits read -36% at tg32 and were at parity at tg256 (and +7%
at d512). Use tg256 (or tg128) with an explicit depth (
-p 0 -n 256 -d 4096). tg is not monotonic in n_ctx on this box (n_ctx 256 measured slower than 32 and 544), so only compare same-depth cells. -b Vulkanvs-b Vulkan0.test-backend-ops -b Vulkanmatches nothing, runs 0 cases and still prints OK with exit 0. The backend isVulkan0. Always check the case count.- ubatch larger than the prompt never fills. With
-ub 2048use prompt lengths that are multiples of 2048. A partial trailing ubatch costs a full expert upload. -tbis not a llama-bench flag (server only). Any bad flag makes llama-bench print usage and exit, which looks like a run that produced no rows. Always check the row count.-ncmoeis llama-bench's spelling of--n-cpu-moe.- llama-bench prints its table only after all
-dvalues finish. No output does not mean hung. - Nothing else may use the GPU during a measurement; stop llama-swap (
bench.shand/root/bench/moe/swapdown.shdo this and restart it from a trap). - Check the build finished before benchmarking it (library mtime earlier than the run start). Never overlap a build with a big benchmark (host OOM).
- Prompt-cache reuse is
timings.prompt_n, nottokens_evaluated(which is always the full prompt length). - Per-process VRAM from
/proc/<pid>/fdinfo/*drm-memory-vrammust be deduplicated bydrm-client-id; summing all fds double-counts shared buffers.drm-memory-gttis system RAM, not VRAM. Whole-card usage:/sys/class/drm/card*/device/mem_info_vram_used.GGML_VK_MEMORY_LOGGER=1logs every Vulkan buffer alloc/free with a running device total to stderr. - The concat fast path is invisible to test-backend-ops (F3); validate it with the knob, perf
logger timing, or byte-identical greedy generation (temperature 0, top_k 1, fixed seed,
cache_prompt: false). - Do not run
llama-clidetached with closed stdin on the pod: it can loop forever printing prompts (it once wrote a 65 GB log). Use llama-bench-dor the server harness.
7. Known gaps and open risks
llama_context::restore_device()ignores theboolreturned byrestore_device_weights(),moe_cache->restore_device()andmemory->restore_device_buffers(). On a restore OOM,device_resident()stays false, decode retries the restore on every call, and a graph reading a null buffer would crash. Only the mmproj path handles a failed restore.- Memory types without a
release_device_buffersoverride keep their KV underevict_kv(F4). - Router mode can spin at shutdown (F7); not used here.
- Two code comments are stale: the arbiter block comment in
server-context.cppsays the KV cache is never evicted (it is, withLLAMA_SLEEP_EVICT_KV), andvram_go_coldsays the encoder is rebuilt byvram_ensure_warm()(it is restored only at encode time). - The README fork section says "the two code changes below are auto-on"; the fork now has more code changes than that. Treat this file as the complete list.
- Not reported upstream: the
wait_until_no_sleep()lost wakeup and the sleep wait that ignored new tasks (both upstream bugs fixed in878d1338f), and the graph-optimize alias cost at tiny context. Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.