|
|
|
@@ -6,13 +6,14 @@ which also holds the operations reference (pod access, files, recovery).
|
|
|
|
|
|
|
|
|
|
Status at the time of writing (2026-11-22):
|
|
|
|
|
|
|
|
|
|
- Production runs `3621025e8` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
|
|
|
|
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12), deployed
|
|
|
|
|
2026-11-22 03:52 with the Inkling retune (section 5). Validated with the full `fork-validate.sh`
|
|
|
|
|
set (phase 2 47 PASS, 0 FAIL; runbook section 6). Rollback pointer
|
|
|
|
|
`/root/DEPLOY-ROLLBACK-commit-14.txt` = `3bf6b55b3` (build copy `/root/arms/build-3bf6b55b3`,
|
|
|
|
|
config backup `/root/config.yaml.bak-20261122-inkling-ub1024-ctx65536`); next free pointer 15.
|
|
|
|
|
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12680-3621025e8`) and changes
|
|
|
|
|
- Production runs `f3397cded` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
|
|
|
|
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, F13, the
|
|
|
|
|
per-pass loop views for Inkling, and F14, the user-message batch-break merge), deployed 2026-11-22
|
|
|
|
|
10:37 (F14 validated in its own window: 35B TTFT A/B, disk cache 15 PASS and the 35B legs; F14
|
|
|
|
|
measured). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-16.txt` = `b13c41355` (build copy
|
|
|
|
|
`/root/arms/build-b13c41355`; config unchanged by this deploy); next free pointer 17. The previous
|
|
|
|
|
pointer 15 = `3621025e8` needs its config backup `/root/config.yaml.bak-20261122-predeploy` too.
|
|
|
|
|
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12687-f3397cded`) and changes
|
|
|
|
|
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
|
|
|
|
|
(section 3, "2026-11-21").
|
|
|
|
|
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
|
|
|
|
@@ -53,6 +54,8 @@ Status at the time of writing (2026-11-22):
|
|
|
|
|
- F10 Page-cache drop after a copying load
|
|
|
|
|
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
|
|
|
|
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
|
|
|
|
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
|
|
|
|
- F14 User-message batch-break merge (one ubatch per short turn)
|
|
|
|
|
4. Dead ends
|
|
|
|
|
5. Configuration that matters
|
|
|
|
|
6. Measurement methodology and traps
|
|
|
|
@@ -136,6 +139,8 @@ Oldest first. "Feature" points at the section below.
|
|
|
|
|
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
|
|
|
|
|
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
|
|
|
|
|
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
|
|
|
|
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
|
|
|
|
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
|
|
|
|
|
|
|
|
|
|
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
|
|
|
|
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
|
|
|
@@ -1151,7 +1156,16 @@ production's Inkling command line with the nothink sampling, GPU to itself; driv
|
|
|
|
|
on every build, production `3bf6b55b3` with ub512 included, with and without
|
|
|
|
|
`GGML_OP_OFFLOAD_MIN_BATCH=64`, so the carry did not cause it. The think entry gets all three
|
|
|
|
|
right on both builds (image "Red", `get_weather({"city":"Warsaw"})`, 28.4K log "65" at 45.9 t/s
|
|
|
|
|
in production). Short text prompts work on the nothink entry. Open: fix the nothink macro.
|
|
|
|
|
in production). Short text prompts work on the nothink entry.
|
|
|
|
|
- Nothink fix (2026-11-22, config only): the cause was `--reasoning-budget 0`. With effort `none`
|
|
|
|
|
and no budget the model still writes a short reasoning block (100-550 characters) and then
|
|
|
|
|
answers; a budget of 0 cut that block off at once and the model looped on the chat-structure
|
|
|
|
|
tokens. The macro is now `--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --reasoning-effort none`.
|
|
|
|
|
Verified on production `3621025e8` (pod driver `/root/arms/overnight/nothink.py`, logs
|
|
|
|
|
`nothink1.log`, `nothink2.log` there): short question, Polish poem, image ("Red"), tool call
|
|
|
|
|
(`get_weather({"city":"Warsaw"})`) and a 29.4K-token Pan Tadeusz prompt (correct, 46.0 t/s) all
|
|
|
|
|
OK. Effort `minimal` and a budget of 256 also work; the budget of 0 fails on all but the short
|
|
|
|
|
question.
|
|
|
|
|
|
|
|
|
|
**Rebase hazards.**
|
|
|
|
|
- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry
|
|
|
|
@@ -1173,6 +1187,352 @@ about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rej
|
|
|
|
|
(the graph then uses the dense mask), so it prints no CPU lines).
|
|
|
|
|
Then an Inkling request through llama-swap.
|
|
|
|
|
|
|
|
|
|
### F13. Per-pass views of host weights in looped layers (inkling2_loop)
|
|
|
|
|
|
|
|
|
|
Branch `fable5/inkling-loop-views` (on top of the F12 carries), fast-forwarded into the production
|
|
|
|
|
branch on 2026-11-22 (deploy and results below).
|
|
|
|
|
|
|
|
|
|
**What and why.** `inkling2_loop` (Inkling-2-Mini) runs the core layers 6..17 three times; the 48
|
|
|
|
|
cache slots are 24 physical layers plus 2 x 12 repeats. `post_arch_tensors` makes every repeat slot
|
|
|
|
|
share the tensors of its physical layer (`layers[il] = layers[phys]`). With the experts in host
|
|
|
|
|
memory (`--n-cpu-moe 24`, `-ot exps=CPU`), `ggml_backend_sched` makes one device copy per (host
|
|
|
|
|
tensor, backend) per graph, hashed by tensor pointer, at the first split that reads it, and keeps it
|
|
|
|
|
allocated until the last reader (`ggml_backend_sched_split_graph`, pass 5 and the graph_copy input
|
|
|
|
|
nodes). Two consequences:
|
|
|
|
|
- Memory: the expert copies of all 12 loop layers are alive at the same time (about 547 MiB per
|
|
|
|
|
layer on UD-Q4_K_XL, 6.5 GiB in total). That is most of the ~7 GiB Vulkan0 compute buffer in the
|
|
|
|
|
F12 measurements.
|
|
|
|
|
- Correctness (upstream issue #32755, "GGGG" output): the copy callback (`sched_copy_experts`,
|
|
|
|
|
F1) runs once, for the first reader. When it reads the routing ids back (ubatches below the F1
|
|
|
|
|
skip threshold of `4 * n_expert` ids, and at or above the Vulkan MUL_MAT_ID offload batch of 32
|
|
|
|
|
tokens: 32-63 tokens on Inkling-2-Mini), it uploads only the experts pass 0 selected. Passes 1 and
|
|
|
|
|
2 reuse that copy and read stale VRAM for every other expert. Ubatches of 64+ tokens are correct
|
|
|
|
|
(F1 uploads every expert), ubatches below 32 run MUL_MAT_ID on the CPU (no copy).
|
|
|
|
|
|
|
|
|
|
**Where.** `src/llama-model.h` (`llama_model::post_load_tensors`, a new optional hook),
|
|
|
|
|
`src/llama-model.cpp` (`llama_model_base::load_tensors` calls it on the no_alloc return and after
|
|
|
|
|
`load_tensors_hadamard()`), `src/models/models.h` and `src/models/inkling2-loop.cpp`
|
|
|
|
|
(`llama_model_inkling2_loop::post_load_tensors`, member `ctx_visits`). Commit 312c29c2b.
|
|
|
|
|
|
|
|
|
|
**How it works.** After the weights are loaded, for every slot of pass 1 and 2 (`il >= n_phys`), each
|
|
|
|
|
tensor pointer in its `llama_layer` (walked as an array of pointers, as `llama_context::opt_init`
|
|
|
|
|
does) that sits in a host buffer (`ggml_backend_buffer_is_host`) is replaced with
|
|
|
|
|
`ggml_view_tensor` of it, named `<name>.visit<pass>`, initialized with `ggml_backend_view_init`
|
|
|
|
|
(same buffer and data). On the no_alloc pass (memory fit) only `view->buffer` is set to the dummy
|
|
|
|
|
buffer, so the scheduler still sees a WEIGHTS buffer. The view is a leaf (op NONE, `view_src` set),
|
|
|
|
|
so the scheduler hashes it separately: each pass gets its own copy, freed after the last reader of
|
|
|
|
|
that pass, and the copy callback sees the MUL_MAT_ID of that pass and its ids (`st.ids` differs, so
|
|
|
|
|
F1 recomputes the expert set). The pointer-keyed weight maps of `llama_model` (`w_scales`,
|
|
|
|
|
`prec_policy.prec_acc`/`prec_src1`, `hdmd.rot`) get entries for the views. The views live in
|
|
|
|
|
`ctx_visits`, a context of the model outside `ctxs_bufs`. The startup log prints
|
|
|
|
|
`post_load_tensors: N views of host weights for the loop passes after the first (X MiB, not copied)`.
|
|
|
|
|
|
|
|
|
|
Knob: `LLAMA_LOOP_HOST_VIEWS=0` keeps the shared tensors (the upstream behaviour), read once at load.
|
|
|
|
|
Default on.
|
|
|
|
|
|
|
|
|
|
**Invariants that must hold.**
|
|
|
|
|
- Only host buffers get views. Device weights stay shared (used in place, no copy, so no problem),
|
|
|
|
|
and so do CPU extra buffers (CPU_REPACK is not a host buffer; its ops run on the CPU in place).
|
|
|
|
|
- F4: views point only at host buffers, which `release_device_weights` never releases, and they
|
|
|
|
|
are not in `ctxs_bufs`, so the shadows, `weight_release`, `memory_breakdown`, `n_tensors()` and
|
|
|
|
|
`tensors_by_name` do not see them. A device weight is released and restored in place, and the
|
|
|
|
|
loop slots keep pointing at the same tensor.
|
|
|
|
|
- F1: the skip/readback decision is per copy and keys on the ids tensor of the first node of the
|
|
|
|
|
split, so it stays per pass.
|
|
|
|
|
- `llama_layer` must hold only `ggml_tensor *` members (a `static_assert` checks the size only).
|
|
|
|
|
|
|
|
|
|
**Cost.** Pass 1 and 2 now upload their own experts: per ubatch the host-to-device expert traffic
|
|
|
|
|
goes from 24 to 48 layer uploads (about +13 GiB per full ubatch on UD-Q4_K_XL). Measured where the
|
|
|
|
|
old layout still fits in VRAM (llama-bench pp2048 at depth 0, `-ncmoe 24`, B C C B in one window):
|
|
|
|
|
views 107.10 / 107.53 t/s, off 110.80 / 111.53, so -3.5%. Decode is unaffected (single-token
|
|
|
|
|
MUL_MAT_ID runs on the CPU). In the server at 16k-65k context the old layout does not fit (7 GiB of
|
|
|
|
|
compute buffer), spills into GTT, and the views win about 1.6x at prefill (Measured, below). The
|
|
|
|
|
freed ~6 GiB of VRAM can hold experts instead: an expert tensor of a loop layer kept on the GPU
|
|
|
|
|
saves 3 uploads per ubatch and 3 CPU expert passes per token, a non-loop one saves 1, so loop layers
|
|
|
|
|
are the first to move to the GPU (`-ot` with the loop layers left out of the CPU pattern; placement
|
|
|
|
|
guide below).
|
|
|
|
|
|
|
|
|
|
**Measured (2026-11-22, pod, GPU to itself, build `b13c41355` = 312c29c2b + docs; drivers and logs in
|
|
|
|
|
`/root/arms/overnight/`: `f13.py`, `cmp.py`, `kld48.sh`, `fit.py`, `sweep.sh`, `STATUS.md`).**
|
|
|
|
|
- Memory, server with the production command line at ctx 16384, `-lv 4`: Vulkan0 compute buffer
|
|
|
|
|
1426 MiB with views, 7030 MiB without. Process VRAM after load: views 4179 MiB, off 7727, A
|
|
|
|
|
(loop experts on the GPU, no views) 7879 (A wants 10089 MiB and spills).
|
|
|
|
|
- Prefill, server, 7224-token raw prompt at ctx 16384: views 92.4 t/s, off 57.0, A 46.5 (production
|
|
|
|
|
at ctx 65536 without views: about 46).
|
|
|
|
|
- Correctness. First-token top-5 logprobs (`f13.py`, `n_probs 5`, knob 32 so the 32-63 window is
|
|
|
|
|
live): with the tail at or above 64 or below 32 tokens (N = 20, 100, 7368) views on and off are
|
|
|
|
|
identical (max difference 0.000); in the window (N = 40, 63, 1072, 7224) they differ (0.04-0.16).
|
|
|
|
|
That test cannot say which arm is right: arm A differs from both by up to 0.15 even at the
|
|
|
|
|
controls (different placement, different kernels), and at 32-63 tokens x 8 experts pass 0's
|
|
|
|
|
union already covers most of the 128 experts, so the stale copy is mostly correct data. The
|
|
|
|
|
decisive test is `kld48.sh`: `llama-perplexity -c 48 -b 48 -ub 48`, 30 chunks of Pan Tadeusz, so
|
|
|
|
|
every chunk is one 48-token ubatch, KL divergence against a base with views on and knob 64 (the
|
|
|
|
|
tail MUL_MAT_ID on the CPU, correct by construction):
|
|
|
|
|
|
|
|
|
|
| arm | mean KLD | max KLD | same top token | RMS dp |
|
|
|
|
|
|---|---|---|---|---|
|
|
|
|
|
| views on, knob 32 (tail on the GPU) | 0.00314 | 0.087 | 97.4% | 1.35% |
|
|
|
|
|
| views off, knob 32 (tail on the GPU) | 0.03163 | 0.347 | 92.2% | 3.71% |
|
|
|
|
|
| views off, knob 64 | 0.00000 | - | 100% | 0.00% |
|
|
|
|
|
|
|
|
|
|
Views on is at the GPU-vs-CPU kernel noise level; views off is 10x worse (the #32755 mechanism,
|
|
|
|
|
reproduced). So with F13 the `GGML_OP_OFFLOAD_MIN_BATCH=64` workaround is no longer needed.
|
|
|
|
|
|
|
|
|
|
**Placement guide (UD-Q4_K_XL, views on).** k = number of loop layers (6, 7, ...) whose experts stay
|
|
|
|
|
on the GPU: `-ot 'blk[.](<the other 24-k layers>)[.]ffn_.*_exps=CPU'` instead of `--n-cpu-moe 24`
|
|
|
|
|
(`/root/arms/overnight/mkot.py` and `cfgedit.py` write the pattern; `[.]` avoids backslashes in the
|
|
|
|
|
YAML and in llama-swap's argument splitting). Each loop layer's experts take 547 MiB. Fit, from the
|
|
|
|
|
`-lv 4` breakdown of a server with the production command line (mmproj on demand, so not loaded):
|
|
|
|
|
|
|
|
|
|
| ctx | k | Vulkan0 self (model + context + compute) | free | verdict |
|
|
|
|
|
|---|---|---|---|---|
|
|
|
|
|
| 32768 | 0 | 4136 (1266 + 1347 + 1522) | 3982 | fits |
|
|
|
|
|
| 32768 | 6 | 7416 | 699 | fits |
|
|
|
|
|
| 32768 | 7 | 7963 | 15 | too tight |
|
|
|
|
|
| 65536 | 4 | 7246 (3453 + 2371 + 1421) | 868 | fits, chosen |
|
|
|
|
|
| 65536 | 5 | 7793 | 128 | too tight for the mmproj |
|
|
|
|
|
| 65536 | 6 | - | - | spills (GTT about 900 MiB above expectation) |
|
|
|
|
|
| 131072 | 0 | 7784 (1266 + 4419 + 2098) | 134 | fits (before F13 it overcommitted 5.2 GB) |
|
|
|
|
|
| 131072 | 2 | - | - | spills (about 1.2 GB) |
|
|
|
|
|
|
|
|
|
|
fdinfo `drm-memory-gtt` is not the spill test: it includes the pinned host weights
|
|
|
|
|
(Vulkan_Host). Compare GTT(k) with GTT(0) - k x 547 MiB, or read the breakdown.
|
|
|
|
|
|
|
|
|
|
Speed, llama-bench, one invocation per metric (`sweep.sh`, `-fa 1 -t 6 -b 1024 -ub 1024`, knob 64,
|
|
|
|
|
pp r1, tg r2):
|
|
|
|
|
|
|
|
|
|
| k | pp2048 @ d8192 | pp2048 @ d16384 | tg256 @ d4096 |
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
| 0 | 97.53 | 89.62 | 5.57 |
|
|
|
|
|
| 4 | 102.19 | 93.90 | 6.45 |
|
|
|
|
|
| 6 | 104.43 | 95.27 | 6.72 |
|
|
|
|
|
| 7 | 105.90 | 96.90 | 7.25 |
|
|
|
|
|
|
|
|
|
|
Decode gains most (+16% at k=4, +30% at k=7): every loop layer on the GPU removes 3 of the 48 CPU
|
|
|
|
|
expert passes per token. Prefill gains only 5-9%: at depth the attention and the remaining uploads
|
|
|
|
|
dominate.
|
|
|
|
|
|
|
|
|
|
**Quants (2026-11-22, not adopted).** Unsloth's Mainline_Support UD-Q3_K_XL (13.6 GB) and UD-IQ3_XXS
|
|
|
|
|
(11.7 GB) are in `/root/models/inkling/`. Quality: `llama-perplexity -c 2048 --chunks 4`, KL
|
|
|
|
|
divergence against UD-Q4_K_XL logits (`kld.sh`), on a fixed Pan Tadeusz chunk
|
|
|
|
|
(`/root/bench/entext/pt-chunk.txt`, characters 20000-100000) and wikitext-2 test:
|
|
|
|
|
|
|
|
|
|
| text | quant | PPL (Q4_K_XL) | PPL ratio | mean KLD | same top token |
|
|
|
|
|
|---|---|---|---|---|---|
|
|
|
|
|
| Pan Tadeusz | Q3_K_XL | 11.45 (11.30) | +1.4% | 0.0241 | 92.5% |
|
|
|
|
|
| Pan Tadeusz | IQ3_XXS | 11.52 (11.30) | +1.9% | 0.0421 | 90.1% |
|
|
|
|
|
| wikitext-2 | Q3_K_XL | 6.251 (6.230) | +0.3% | 0.0097 | 95.6% |
|
|
|
|
|
| wikitext-2 | IQ3_XXS | 6.307 (6.230) | +1.2% | 0.0214 | 94.2% |
|
|
|
|
|
|
|
|
|
|
Fit at ctx 65536: UD-Q3_K_XL keeps the loop-layer experts at 547 MiB each (like Q4), so k=5 (867
|
|
|
|
|
MiB free); UD-IQ3_XXS has 390 MiB per layer, k=7 (952 MiB free). Speed (separate invocations, so
|
|
|
|
|
within the ~7% cross-run noise for prefill): Q3_K_XL k=4 105.23 / 6.83, k=6 107.31 / 7.48;
|
|
|
|
|
IQ3_XXS k=4 105.59 / 6.44, k=7 108.48 / 7.86 (pp2048 @ d8192 / tg256 @ d4096; Q4_K_XL k=4
|
|
|
|
|
102.19 / 6.45). The best case, IQ3_XXS at k=7, is +6% prefill and +22% decode (about 7.9 instead
|
|
|
|
|
of 6.5 t/s) for a Polish KLD of 0.042; IQ3_XXS's CPU dot product is slower per byte (at k=4 it
|
|
|
|
|
decodes no faster than Q4). Not worth the quality loss for the owner's Polish use.
|
|
|
|
|
|
|
|
|
|
**Production (deployed 2026-11-22, `b13c41355`, ctx 65536, k=4, no knob).** Through llama-swap
|
|
|
|
|
(`/root/arms/overnight/post.py`, log `post.log`): Pan Tadeusz 9312 tokens 81.6 t/s prefill, +7958
|
|
|
|
|
tokens 89.3 t/s, +17255 tokens (28K total) 78.0 t/s, decode 5.9-6.0 t/s, answers correct (before:
|
|
|
|
|
about 46 t/s prefill). Think and nothink entries answer text, an image ("Red") and a tool call. A
|
|
|
|
|
tiny request after the 4B task model held the GPU takes 1.3-2.4 s longer than a warm one (the
|
|
|
|
|
arbiter restores 3.4 GiB of device weights). Disk cache save/restore works on Inkling (1329 tokens,
|
|
|
|
|
410 MiB). The first request on a fresh process is slow (162 tokens 11.7 t/s prefill, 2.6 t/s decode)
|
|
|
|
|
since the pipelines are compiled on first use (`--no-warmup`).
|
|
|
|
|
|
|
|
|
|
**Profile (2026-11-22, `perf record` + `gpu_busy_percent`, `prof.sh`, results in
|
|
|
|
|
`/root/arms/overnight/prof/`).** Inkling k=4 prefill (pp2048 @ d4096, 107 t/s): GPU busy 77%, CPU
|
|
|
|
|
time mostly `ggml_vk_wait_for_fence` (43%) and the kernel (11%, the uploads): GPU-bound. Decode
|
|
|
|
|
(7.3 t/s): GPU busy 58%, CPU in the OpenMP barrier spin (libgomp 36%) and
|
|
|
|
|
`ggml_vec_dot_nvfp4_q8_0` (21%): the UD-Q4_K_XL experts are NVFP4, and that CPU dot product is the
|
|
|
|
|
decode hotspot. Qwen3.6-35B for comparison: prefill 263 t/s at GPU busy 52%, decode 13.8 t/s at
|
|
|
|
|
35%, `vec_dot_q4_K_q8_K` 10%. `GGML_VK_PERF_LOGGER=1` aborts at the second split
|
|
|
|
|
(`GGML_ASSERT(ctx->compute_ctx.expired())`, ggml-vulkan.cpp:15692), so it gives per-op GPU times
|
|
|
|
|
only for the first split; not investigated.
|
|
|
|
|
|
|
|
|
|
**Failure modes and known limits.**
|
|
|
|
|
- LoRA adapters look weights up by name (`llama_adapter_lora::get_weight`), so they do not apply to
|
|
|
|
|
passes 1 and 2 with views on. Not used here; set `LLAMA_LOOP_HOST_VIEWS=0` for a LoRA on Inkling.
|
|
|
|
|
- `--moe-cache-mib`: the cache binds expert tensors per slot (`llama_moe_cache_layer_experts` of
|
|
|
|
|
`model.layers[il]`), so with views each loop slot gets its own binding and slot map and caches the
|
|
|
|
|
same experts separately (before, the three slots collided on one binding key). Should not crash;
|
|
|
|
|
not tested. The MoE cache is off in production.
|
|
|
|
|
- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data
|
|
|
|
|
in a saved file cannot happen.
|
|
|
|
|
|
|
|
|
|
**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for
|
|
|
|
|
every reader in the scheduler, or real per-pass tensors in the loader), revert 312c29c2b; the hook has
|
|
|
|
|
no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
|
|
|
|
|
|
|
|
|
|
**Rebase hazards.** `post_arch_tensors` and the loop slot layout in `inkling2-loop.cpp`;
|
|
|
|
|
`llama_model_base::load_tensors` returns (a new return path must call the hook); new pointer-keyed
|
|
|
|
|
weight maps in `llama_model` (they need a view entry like `w_scales`); non-pointer members in
|
|
|
|
|
`llama_layer`; scheduler changes to how host weights are copied (`hv_tensor_copies`,
|
|
|
|
|
`ggml_backend_sched_is_host_weight`).
|
|
|
|
|
|
|
|
|
|
**Validation (pod, GPU to itself in a swapdown window).** Build `llama-server` and `llama-bench`
|
|
|
|
|
from this branch in a scratch worktree (runbook section 5). Use the Inkling command line from
|
|
|
|
|
`inkling_args` in `/root/config.yaml` (with `--mmproj`) and change only the context to 16384 (so
|
|
|
|
|
arm A fits) and the placement:
|
|
|
|
|
- A, reference: replace `--n-cpu-moe 24` with `-ot 'blk\.([0-5]|1[89]|2[0-3])\.ffn_.*_exps=CPU'`
|
|
|
|
|
(loop experts on the GPU, no copies for the loop; the views log line must not appear).
|
|
|
|
|
- B, views on: `--n-cpu-moe 24`; the log line must show the views.
|
|
|
|
|
- C, views off: B with `LLAMA_LOOP_HOST_VIEWS=0`.
|
|
|
|
|
|
|
|
|
|
For each arm send raw token prompts (`/tokenize` a long text, cut to N tokens, `/completion` with the
|
|
|
|
|
token array, `temperature 0, top_k 1, seed 42, cache_prompt false, n_predict 32, n_probs 5`) at
|
|
|
|
|
N = 40, 63, 1072, 6176, 7224 (tail 32-63 at ub1024), controls N = 100, 7368 (tail >= 64) and 20.
|
|
|
|
|
Expected:
|
|
|
|
|
- B == A in the first token's top-5 probabilities for every N >= 32 (prefill MUL_MAT_ID runs on the
|
|
|
|
|
GPU in both arms with the same data). The generated text may drift later, since decode MUL_MAT_ID
|
|
|
|
|
runs on the GPU in A and on the CPU in B.
|
|
|
|
|
- B == C byte-identical for the controls (tail >= 64 or < 32).
|
|
|
|
|
- C differs from A at the 32-63 tails (the bug); if it does not, the bug was not reproduced and
|
|
|
|
|
B's result proves nothing.
|
|
|
|
|
- Vulkan0 compute buffer (startup log): B about 6 GiB below C. A's model buffer carries the loop
|
|
|
|
|
experts instead.
|
|
|
|
|
Then `llama-bench -lm none -ngl 99 -ncmoe 24 -fa 1 -t 6 -b 1024 -ub 1024 -p 2048 -n 0 -r 3`, with and
|
|
|
|
|
without `LLAMA_LOOP_HOST_VIEWS=0`, interleaved B C C B, medians (section 6), for the prefill cost
|
|
|
|
|
above. After a deploy: an Inkling text request, an image request, and the same text request after a
|
|
|
|
|
task-model call (arbiter handoff, F4), identical at temperature 0.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### F14. User-message batch-break merge (one ubatch per short turn)
|
|
|
|
|
|
|
|
|
|
Branch `fable5/userstart-split` (rebased onto the F13 production tip `166ec22e2`), fast-forwarded into
|
|
|
|
|
the production branch on 2026-11-22 (deploy and results below). Knob: `LLAMA_USERSTART_MERGE=0`.
|
|
|
|
|
|
|
|
|
|
**What and why.** Upstream ends a prompt batch right before a user message so that a context
|
|
|
|
|
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
|
|
|
|
|
limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state
|
|
|
|
|
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
|
|
|
|
|
message is what a later request uses when it diverges inside that message (the user edits it, or the
|
|
|
|
|
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
|
|
|
|
|
template drops the previous answer's reasoning, which moves the divergence to just after
|
|
|
|
|
`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
|
|
|
|
|
restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation
|
|
|
|
|
prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each
|
|
|
|
|
ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128
|
|
|
|
|
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
|
|
|
|
|
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
|
|
|
|
|
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
|
|
|
|
|
`<|im_start|>user`, the detected delimiter).
|
|
|
|
|
|
|
|
|
|
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
|
|
|
|
|
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
|
|
|
|
|
Commit 510aa6e09.
|
|
|
|
|
|
|
|
|
|
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
|
|
|
|
|
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
|
|
|
|
|
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
|
|
|
|
|
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
|
|
|
|
|
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
|
|
|
|
|
the restored checkpoint at the same position, with the current task id, so the next request still
|
|
|
|
|
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
|
|
|
|
|
diverges inside that message restores the batch-start checkpoint instead and re-processes the
|
|
|
|
|
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
|
|
|
|
|
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
|
|
|
|
|
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
|
|
|
|
|
upstream behaviour everywhere.
|
|
|
|
|
|
|
|
|
|
**Invariants that must hold.**
|
|
|
|
|
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
|
|
|
|
|
divergence inside the last user message to less than `n_ubatch` tokens.
|
|
|
|
|
- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared
|
|
|
|
|
by `has_mtmd`); hence the `!has_mtmd` condition.
|
|
|
|
|
- It only changes where batches end, never which tokens are processed, so outputs differ from an
|
|
|
|
|
unmerged run only by batch-composition rounding.
|
|
|
|
|
|
|
|
|
|
**Measured (2026-11-22, pod, GPU to itself in one swapdown window, build `f3397cded`; driver
|
|
|
|
|
`/root/arms/usw_ab.py`, window script `/root/arms/usw-window.sh`, logs and `res.json` per arm in
|
|
|
|
|
`/root/arms/usw-ab/{M1,U1,U2,M2}/`).** Production 35B think entry (`q36t`: `${ctx_256k}
|
|
|
|
|
${qwen35_think_args} ${moe_offload_args}`, `-hf` pulls the mmproj, supervisord's environment, scratch
|
|
|
|
|
arena and `--cache-disk-path`, plus `-lv 4`), one fresh server per arm, arms M (merged, default) and U
|
|
|
|
|
(`LLAMA_USERSTART_MERGE=0`) in the order M U U M. Each chat: system prompt, 6 user turns from
|
|
|
|
|
`pt-chunk.txt` (150, 10, 40, 150, 600, 40 tokens), synthetic previous answers of 300, 800, 1200, 500,
|
|
|
|
|
1300 tokens, `max_tokens 8`, `temperature 0`, streamed; then the last user message edited in the
|
|
|
|
|
middle and resent. TTFT in seconds (mean of the two runs per arm; the single runs in brackets),
|
|
|
|
|
`prompt_n` was identical in all four runs on every turn:
|
|
|
|
|
|
|
|
|
|
| turn | user tokens | prompt_n | M TTFT | U TTFT | M - U |
|
|
|
|
|
|---|---|---|---|---|---|
|
|
|
|
|
| 1 (fresh prompt) | 150 | 196 | 3.99 (4.43, 3.55) | 6.08 (5.73, 6.42) | -2.1 |
|
|
|
|
|
| 2 | 10 | 327 | 4.11 (3.75, 4.47) | 7.66 (6.75, 8.57) | -3.6 |
|
|
|
|
|
| 3 | 40 | 869 | 6.34 (6.62, 6.06) | 9.34 (7.26, 11.42) | -3.0 |
|
|
|
|
|
| 4 | 150 | 1381 | 7.20 (7.30, 7.10) | 10.95 (12.31, 9.58) | -3.7 |
|
|
|
|
|
| 5 | 600 | 1130 | 8.82 (7.47, 10.17) | 9.80 (9.52, 10.07) | -1.0 |
|
|
|
|
|
| 6 | 40 | 1370 | 8.11 (8.01, 8.20) | 9.53 (9.33, 9.72) | -1.4 |
|
|
|
|
|
| edit of turn 6 | 40 | M 1374, U 66 | 7.94 (8.16, 7.71) | 3.65 (4.09, 3.20) | +4.3 |
|
|
|
|
|
|
|
|
|
|
- Batch layout (`prompt processing, n_tokens` and checkpoint lines at `-lv 4`): M runs every turn as
|
|
|
|
|
one batch `[restore point, end-4)` plus the tail, with checkpoints at the batch start (superseding
|
|
|
|
|
the restored one) and at end minus 4; U splits at the last user message (an extra checkpoint
|
|
|
|
|
there, for example 1323 between 515 and 1380 on turn 3) and on turn 1 also at the first user
|
|
|
|
|
message (system prompt, 23 tokens). Turn N+1 restores the end-minus-4 checkpoint of turn N in both
|
|
|
|
|
arms. No `forcing full prompt re-processing` in any run.
|
|
|
|
|
- The 10-token turn gained too, against the plan's "no change". U split it into 308 tokens (the
|
|
|
|
|
previous answer, on the GPU) and 19 (message, generation prompt and tail), and the 19-token part runs
|
|
|
|
|
the experts of all 40 layers on the CPU, which costs seconds rather than nothing at 6 threads
|
|
|
|
|
(inferred from the totals, not profiled). The 600-token turn gained
|
|
|
|
|
least (its two parts are 508 and 618 tokens, both on the GPU, so the split costs one extra
|
|
|
|
|
upload, about 1-2 s against noise of the same size).
|
|
|
|
|
- Edit of the last user message (the cost of the trade): M has no checkpoint at the user-message
|
|
|
|
|
start, restores the batch start and re-processes previous answer + message (1374 tokens, one
|
|
|
|
|
ubatch); U restores the user-start checkpoint and re-processes 66 tokens. +4.3 s per such edit.
|
|
|
|
|
Ordinary follow-up turns are far more frequent than edits of the last message in this use; a client
|
|
|
|
|
that injects per-request context into the last user message (RAG, web search) pays the same
|
|
|
|
|
as an edit on every turn, so set `LLAMA_USERSTART_MERGE=0` for such a client.
|
|
|
|
|
- Outputs: identical within each arm across runs; between arms turns 5 and 6 differ after a few
|
|
|
|
|
tokens (batch-composition rounding, as the invariant says).
|
|
|
|
|
- Disk cache: `fork-validate.sh --diskcache` (in the window, GPU leg included) 15 PASS, 0 FAIL, 0
|
|
|
|
|
SKIP. `diskcache-window3.sh` legs with production's 35B command line (`q35.py` copied to
|
|
|
|
|
`q35-usw.py` with the candidate build path; log `/root/arms/usw-q35-mm.log`): 18.8k-token chat,
|
|
|
|
|
turn 1 83.5 s, save 557 MB in 1.0 s, restart restore and turn 2 `prompt_n` 27 in 3.7 s, output ==
|
|
|
|
|
reference; cold save and restore identity (`prompt_n` 4, output == reference); image answered
|
|
|
|
|
'Red' and not saved. The saved state after turn 2 holds 4 checkpoints instead of 5 (the user-start
|
|
|
|
|
one is gone).
|
|
|
|
|
|
|
|
|
|
**Production (deployed 2026-11-22 10:37, `f3397cded`, config unchanged).** Deploy script
|
|
|
|
|
`/root/arms/dep-usw.sh` (log `dep-usw.log`): incremental in-place build, downtime 33 s. Post-deploy
|
|
|
|
|
checks through llama-swap (`/root/arms/post-usw.py`, `ink-usw.py`, `ink2-usw.py`, logs beside them):
|
|
|
|
|
`system_fingerprint` `b12687-f3397cded`; 35B 'Paris', then 4B / 35B handoffs twice (correct answers,
|
|
|
|
|
`holder` the answering PID each time, 2.9-6.3 s per switch); a 1140-token 35B chat, then a gemma-26B
|
|
|
|
|
image ('Red', the 35B exited by itself with `loop stopped, device memory freed` and `disk cache:
|
|
|
|
|
saved the prompt of 1157 tokens`), gemma text, 4B text, then the 35B chat plus one user message:
|
|
|
|
|
`disk cache: restored 1157 tokens`, `prompt_n` 30 (the suffix only). Inkling nothink: text correct
|
|
|
|
|
(it still spends about 40 tokens on reasoning, so a 96-token budget returns an empty content), image
|
|
|
|
|
'Red', a 9293-token Pan Tadeusz prompt at 82.0 t/s with the right answer (F13 measured 81.6). Children
|
|
|
|
|
bind llama-swap's ports. Log scan clean, `dmesg` clean (no ring timeout, GPU hang or OOM kill), pod
|
|
|
|
|
restarts 0 as before.
|
|
|
|
|
|
|
|
|
|
**Failure modes and incidents.** None in validation or the deploy. Known cost: an edit of the last
|
|
|
|
|
user message (or per-request context injected into it) re-processes the previous answer too, +4.3 s
|
|
|
|
|
measured above.
|
|
|
|
|
|
|
|
|
|
**Rebase hazards.**
|
|
|
|
|
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
|
|
|
|
|
`checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch
|
|
|
|
|
start is still checkpointed when the batch ends at prompt end minus 4.
|
|
|
|
|
|
|
|
|
|
**Validation plan (run 2026-11-22, results under Measured above).**
|
|
|
|
|
1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the
|
|
|
|
|
`created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch
|
|
|
|
|
`[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the
|
|
|
|
|
last user message.
|
|
|
|
|
2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded),
|
|
|
|
|
candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn
|
|
|
|
|
chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read
|
|
|
|
|
`prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens
|
|
|
|
|
(the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens.
|
|
|
|
|
3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit
|
|
|
|
|
the last user message and resend: merged re-processes previous answer + message (bounded by one
|
|
|
|
|
ubatch), unmerged only the message; no "forcing full prompt re-processing".
|
|
|
|
|
4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line):
|
|
|
|
|
restore after a restart, `prompt_n` and output identical to the reference built from the same
|
|
|
|
|
commit.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 4. Dead ends
|
|
|
|
@@ -1261,16 +1621,23 @@ Other models:
|
|
|
|
|
heads, head size 128; macro `inkling_args`, entries `unsloth/Inkling-2-Mini-GGUF[-nothink]:UD-Q4_K_XL`,
|
|
|
|
|
matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m`
|
|
|
|
|
(the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it,
|
|
|
|
|
`--n-cpu-moe 24` (all experts on host), `-t 6`, f16 KV, `--parallel 1`, the disk prompt cache.
|
|
|
|
|
Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 65536`
|
|
|
|
|
(before: `-b 512 -ub 512`; the dense per-head bias mask needed about 12.7 GB of host compute
|
|
|
|
|
buffer at 128k). 131072 loads and works with the carry, but overcommits about 5.2 GB of device
|
|
|
|
|
memory into GTT and is about 8% slower at prefill (F12). Both Inkling entries set
|
|
|
|
|
`env: ["GGML_OP_OFFLOAD_MIN_BATCH=64"]` (llama-swap per-model env): ubatches of 32-63 tokens keep
|
|
|
|
|
MUL_MAT_ID on the CPU. The looped layers share their expert tensors, and below F1's readback-skip
|
|
|
|
|
threshold (4 x 128 experts / 8 used = 64 tokens) only loop pass 0's routed experts would be
|
|
|
|
|
uploaded (the mechanism suspected behind upstream #32755). Its nothink macro uses
|
|
|
|
|
`--reasoning-effort none --reasoning-budget 0`.
|
|
|
|
|
`-t 6`, f16 KV, `--parallel 1`, the disk prompt cache, `-b 1024 -ub 1024 --ctx-size 65536`
|
|
|
|
|
(since the 2026-11-22 carry of upstream #32740, F12; before: `-b 512 -ub 512`). Since the F13
|
|
|
|
|
deploy (2026-11-22, `b13c41355`): the experts of loop layers 6-9 stay on the GPU and all others
|
|
|
|
|
on the host, `-ot blk[.](0|1|2|3|4|5|10|11|12|13|14|15|16|17|18|19|20|21|22|23)[.]ffn_.*_exps=CPU`
|
|
|
|
|
instead of `--n-cpu-moe 24` (k=4 of the F13 placement guide: 7246 MiB of device memory, 868 MiB
|
|
|
|
|
left for the mmproj). The `GGML_OP_OFFLOAD_MIN_BATCH=64` env of both entries was removed: F13
|
|
|
|
|
makes the 32-63-token ubatches correct (KLD in F13). Other choices from the guide: ctx 32768 with
|
|
|
|
|
k=6, or ctx 131072 with k=0 (fits since F13, 134 MiB free, so images would spill). Smaller quants
|
|
|
|
|
(UD-Q3_K_XL, UD-IQ3_XXS, `/root/models/inkling/`) were measured and not adopted (F13, Quants).
|
|
|
|
|
Its nothink macro uses `--reasoning-effort none` (no `--reasoning-budget 0`: that budget made it
|
|
|
|
|
loop on chat-structure tokens, F12); even with effort none it writes a short reasoning block.
|
|
|
|
|
Against Qwen3.6-35B (think entries, 7 tasks, one sample each, 3500-token cap, 2026-11-22,
|
|
|
|
|
`/root/arms/overnight/quality.py`, `quality/quality.json`): Inkling 6/7, Qwen 5/7. Inkling found
|
|
|
|
|
all three bugs of a binary search where Qwen ran out of budget; both ran out on a 4-line Polish
|
|
|
|
|
rhyme. Inkling used about a third fewer tokens (9677 vs 14254) but took 1.7x the wall time (2283
|
|
|
|
|
vs 1316 s; decode 4-6 vs 13 t/s) and 4x on a 29K-token question (835 vs 211 s). Too small a
|
|
|
|
|
sample to call it smarter; it is clearly slower here.
|
|
|
|
|
|
|
|
|
|
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
|
|
|
|
|
`/root/supervisord.conf`:
|
|
|
|
@@ -1345,10 +1712,9 @@ Machine-level:
|
|
|
|
|
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
|
|
|
|
|
null buffer would crash. Only the mmproj path handles a failed restore.
|
|
|
|
|
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
|
|
|
|
|
- Inkling (F12, section 5): the nothink entry returns repeated chat-structure tokens instead of an
|
|
|
|
|
answer for image requests, tool calls and long prompts (the think entry is fine); the config runs
|
|
|
|
|
ctx 65536 because 131072 overcommits about 5.2 GB of device memory into GTT; upstream #32755 /
|
|
|
|
|
#32778 did not reproduce here. Drop the F12 carries when upstream merges #32735 and #32740.
|
|
|
|
|
- Inkling (F12, section 5): the nothink entry's chat-structure loop came from
|
|
|
|
|
`--reasoning-budget 0` and was fixed in the config on 2026-11-22 (F12). Upstream #32755 / #32778
|
|
|
|
|
did not reproduce here; F13 bounds the loop-pass expert copies (more in F13). Drop the F12 carries when upstream merges #32735 and #32740.
|
|
|
|
|
- Open after the 2026-11-21 rebase: is the fork's concat fast path (F3) now redundant with upstream's
|
|
|
|
|
own tiled concat (#30591, gated in the fork with `ne0 < 32` and the `GGML_VK_CONCAT_FASTPATH` knob)?
|
|
|
|
|
Benchmark it. Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no
|
|
|
|
|