|
|
|
@@ -6,13 +6,14 @@ which also holds the operations reference (pod access, files, recovery).
|
|
|
|
|
|
|
|
|
|
Status at the time of writing (2026-11-22):
|
|
|
|
|
|
|
|
|
|
- Production runs `3621025e8` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
|
|
|
|
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12), deployed
|
|
|
|
|
2026-11-22 03:52 with the Inkling retune (section 5). Validated with the full `fork-validate.sh`
|
|
|
|
|
set (phase 2 47 PASS, 0 FAIL; runbook section 6). Rollback pointer
|
|
|
|
|
`/root/DEPLOY-ROLLBACK-commit-14.txt` = `3bf6b55b3` (build copy `/root/arms/build-3bf6b55b3`,
|
|
|
|
|
config backup `/root/config.yaml.bak-20261122-inkling-ub1024-ctx65536`); next free pointer 15.
|
|
|
|
|
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12680-3621025e8`) and changes
|
|
|
|
|
- Production runs `b13c41355` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
|
|
|
|
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, and F13, the
|
|
|
|
|
per-pass loop views for Inkling), deployed 2026-11-22 08:47 with the Inkling placement change
|
|
|
|
|
(section 5, F13). Validated with the full `fork-validate.sh` set in one window (52 PASS, 0 FAIL,
|
|
|
|
|
0 SKIP; runbook section 6). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-15.txt` = `3621025e8`
|
|
|
|
|
(build copy `/root/arms/build-3621025e8`, config backup `/root/config.yaml.bak-20261122-predeploy`;
|
|
|
|
|
restore both, the old build has no F13); next free pointer 16.
|
|
|
|
|
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12683-b13c41355`) and changes
|
|
|
|
|
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
|
|
|
|
|
(section 3, "2026-11-21").
|
|
|
|
|
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
|
|
|
|
@@ -53,6 +54,7 @@ Status at the time of writing (2026-11-22):
|
|
|
|
|
- F10 Page-cache drop after a copying load
|
|
|
|
|
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
|
|
|
|
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
|
|
|
|
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
|
|
|
|
4. Dead ends
|
|
|
|
|
5. Configuration that matters
|
|
|
|
|
6. Measurement methodology and traps
|
|
|
|
@@ -136,6 +138,7 @@ Oldest first. "Feature" points at the section below.
|
|
|
|
|
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
|
|
|
|
|
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
|
|
|
|
|
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
|
|
|
|
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
|
|
|
|
|
|
|
|
|
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
|
|
|
|
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
|
|
|
@@ -1151,7 +1154,16 @@ production's Inkling command line with the nothink sampling, GPU to itself; driv
|
|
|
|
|
on every build, production `3bf6b55b3` with ub512 included, with and without
|
|
|
|
|
`GGML_OP_OFFLOAD_MIN_BATCH=64`, so the carry did not cause it. The think entry gets all three
|
|
|
|
|
right on both builds (image "Red", `get_weather({"city":"Warsaw"})`, 28.4K log "65" at 45.9 t/s
|
|
|
|
|
in production). Short text prompts work on the nothink entry. Open: fix the nothink macro.
|
|
|
|
|
in production). Short text prompts work on the nothink entry.
|
|
|
|
|
- Nothink fix (2026-11-22, config only): the cause was `--reasoning-budget 0`. With effort `none`
|
|
|
|
|
and no budget the model still writes a short reasoning block (100-550 characters) and then
|
|
|
|
|
answers; a budget of 0 cut that block off at once and the model looped on the chat-structure
|
|
|
|
|
tokens. The macro is now `--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --reasoning-effort none`.
|
|
|
|
|
Verified on production `3621025e8` (pod driver `/root/arms/overnight/nothink.py`, logs
|
|
|
|
|
`nothink1.log`, `nothink2.log` there): short question, Polish poem, image ("Red"), tool call
|
|
|
|
|
(`get_weather({"city":"Warsaw"})`) and a 29.4K-token Pan Tadeusz prompt (correct, 46.0 t/s) all
|
|
|
|
|
OK. Effort `minimal` and a budget of 256 also work; the budget of 0 fails on all but the short
|
|
|
|
|
question.
|
|
|
|
|
|
|
|
|
|
**Rebase hazards.**
|
|
|
|
|
- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry
|
|
|
|
@@ -1173,6 +1185,216 @@ about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rej
|
|
|
|
|
(the graph then uses the dense mask), so it prints no CPU lines).
|
|
|
|
|
Then an Inkling request through llama-swap.
|
|
|
|
|
|
|
|
|
|
### F13. Per-pass views of host weights in looped layers (inkling2_loop)
|
|
|
|
|
|
|
|
|
|
Branch `fable5/inkling-loop-views` (on top of the F12 carries), fast-forwarded into the production
|
|
|
|
|
branch on 2026-11-22 (deploy and results below).
|
|
|
|
|
|
|
|
|
|
**What and why.** `inkling2_loop` (Inkling-2-Mini) runs the core layers 6..17 three times; the 48
|
|
|
|
|
cache slots are 24 physical layers plus 2 x 12 repeats. `post_arch_tensors` makes every repeat slot
|
|
|
|
|
share the tensors of its physical layer (`layers[il] = layers[phys]`). With the experts in host
|
|
|
|
|
memory (`--n-cpu-moe 24`, `-ot exps=CPU`), `ggml_backend_sched` makes one device copy per (host
|
|
|
|
|
tensor, backend) per graph, hashed by tensor pointer, at the first split that reads it, and keeps it
|
|
|
|
|
allocated until the last reader (`ggml_backend_sched_split_graph`, pass 5 and the graph_copy input
|
|
|
|
|
nodes). Two consequences:
|
|
|
|
|
- Memory: the expert copies of all 12 loop layers are alive at the same time (about 547 MiB per
|
|
|
|
|
layer on UD-Q4_K_XL, 6.5 GiB in total). That is most of the ~7 GiB Vulkan0 compute buffer in the
|
|
|
|
|
F12 measurements.
|
|
|
|
|
- Correctness (upstream issue #32755, "GGGG" output): the copy callback (`sched_copy_experts`,
|
|
|
|
|
F1) runs once, for the first reader. When it reads the routing ids back (ubatches below the F1
|
|
|
|
|
skip threshold of `4 * n_expert` ids, and at or above the Vulkan MUL_MAT_ID offload batch of 32
|
|
|
|
|
tokens: 32-63 tokens on Inkling-2-Mini), it uploads only the experts pass 0 selected. Passes 1 and
|
|
|
|
|
2 reuse that copy and read stale VRAM for every other expert. Ubatches of 64+ tokens are correct
|
|
|
|
|
(F1 uploads every expert), ubatches below 32 run MUL_MAT_ID on the CPU (no copy).
|
|
|
|
|
|
|
|
|
|
**Where.** `src/llama-model.h` (`llama_model::post_load_tensors`, a new optional hook),
|
|
|
|
|
`src/llama-model.cpp` (`llama_model_base::load_tensors` calls it on the no_alloc return and after
|
|
|
|
|
`load_tensors_hadamard()`), `src/models/models.h` and `src/models/inkling2-loop.cpp`
|
|
|
|
|
(`llama_model_inkling2_loop::post_load_tensors`, member `ctx_visits`). Commit 312c29c2b.
|
|
|
|
|
|
|
|
|
|
**How it works.** After the weights are loaded, for every slot of pass 1 and 2 (`il >= n_phys`), each
|
|
|
|
|
tensor pointer in its `llama_layer` (walked as an array of pointers, as `llama_context::opt_init`
|
|
|
|
|
does) that sits in a host buffer (`ggml_backend_buffer_is_host`) is replaced with
|
|
|
|
|
`ggml_view_tensor` of it, named `<name>.visit<pass>`, initialized with `ggml_backend_view_init`
|
|
|
|
|
(same buffer and data). On the no_alloc pass (memory fit) only `view->buffer` is set to the dummy
|
|
|
|
|
buffer, so the scheduler still sees a WEIGHTS buffer. The view is a leaf (op NONE, `view_src` set),
|
|
|
|
|
so the scheduler hashes it separately: each pass gets its own copy, freed after the last reader of
|
|
|
|
|
that pass, and the copy callback sees the MUL_MAT_ID of that pass and its ids (`st.ids` differs, so
|
|
|
|
|
F1 recomputes the expert set). The pointer-keyed weight maps of `llama_model` (`w_scales`,
|
|
|
|
|
`prec_policy.prec_acc`/`prec_src1`, `hdmd.rot`) get entries for the views. The views live in
|
|
|
|
|
`ctx_visits`, a context of the model outside `ctxs_bufs`. The startup log prints
|
|
|
|
|
`post_load_tensors: N views of host weights for the loop passes after the first (X MiB, not copied)`.
|
|
|
|
|
|
|
|
|
|
Knob: `LLAMA_LOOP_HOST_VIEWS=0` keeps the shared tensors (the upstream behaviour), read once at load.
|
|
|
|
|
Default on.
|
|
|
|
|
|
|
|
|
|
**Invariants that must hold.**
|
|
|
|
|
- Only host buffers get views. Device weights stay shared (used in place, no copy, so no problem),
|
|
|
|
|
and so do CPU extra buffers (CPU_REPACK is not a host buffer; its ops run on the CPU in place).
|
|
|
|
|
- F4: views point only at host buffers, which `release_device_weights` never releases, and they
|
|
|
|
|
are not in `ctxs_bufs`, so the shadows, `weight_release`, `memory_breakdown`, `n_tensors()` and
|
|
|
|
|
`tensors_by_name` do not see them. A device weight is released and restored in place, and the
|
|
|
|
|
loop slots keep pointing at the same tensor.
|
|
|
|
|
- F1: the skip/readback decision is per copy and keys on the ids tensor of the first node of the
|
|
|
|
|
split, so it stays per pass.
|
|
|
|
|
- `llama_layer` must hold only `ggml_tensor *` members (a `static_assert` checks the size only).
|
|
|
|
|
|
|
|
|
|
**Cost.** Pass 1 and 2 now upload their own experts: per ubatch the host-to-device expert traffic
|
|
|
|
|
goes from 24 to 48 layer uploads (about +13 GiB per full ubatch on UD-Q4_K_XL). Measured where the
|
|
|
|
|
old layout still fits in VRAM (llama-bench pp2048 at depth 0, `-ncmoe 24`, B C C B in one window):
|
|
|
|
|
views 107.10 / 107.53 t/s, off 110.80 / 111.53, so -3.5%. Decode is unaffected (single-token
|
|
|
|
|
MUL_MAT_ID runs on the CPU). In the server at 16k-65k context the old layout does not fit (7 GiB of
|
|
|
|
|
compute buffer), spills into GTT, and the views win about 1.6x at prefill (Measured, below). The
|
|
|
|
|
freed ~6 GiB of VRAM can hold experts instead: an expert tensor of a loop layer kept on the GPU
|
|
|
|
|
saves 3 uploads per ubatch and 3 CPU expert passes per token, a non-loop one saves 1, so loop layers
|
|
|
|
|
are the first to move to the GPU (`-ot` with the loop layers left out of the CPU pattern; placement
|
|
|
|
|
guide below).
|
|
|
|
|
|
|
|
|
|
**Measured (2026-11-22, pod, GPU to itself, build `b13c41355` = 312c29c2b + docs; drivers and logs in
|
|
|
|
|
`/root/arms/overnight/`: `f13.py`, `cmp.py`, `kld48.sh`, `fit.py`, `sweep.sh`, `STATUS.md`).**
|
|
|
|
|
- Memory, server with the production command line at ctx 16384, `-lv 4`: Vulkan0 compute buffer
|
|
|
|
|
1426 MiB with views, 7030 MiB without. Process VRAM after load: views 4179 MiB, off 7727, A
|
|
|
|
|
(loop experts on the GPU, no views) 7879 (A wants 10089 MiB and spills).
|
|
|
|
|
- Prefill, server, 7224-token raw prompt at ctx 16384: views 92.4 t/s, off 57.0, A 46.5 (production
|
|
|
|
|
at ctx 65536 without views: about 46).
|
|
|
|
|
- Correctness. First-token top-5 logprobs (`f13.py`, `n_probs 5`, knob 32 so the 32-63 window is
|
|
|
|
|
live): with the tail at or above 64 or below 32 tokens (N = 20, 100, 7368) views on and off are
|
|
|
|
|
identical (max difference 0.000); in the window (N = 40, 63, 1072, 7224) they differ (0.04-0.16).
|
|
|
|
|
That test cannot say which arm is right: arm A differs from both by up to 0.15 even at the
|
|
|
|
|
controls (different placement, different kernels), and at 32-63 tokens x 8 experts pass 0's
|
|
|
|
|
union already covers most of the 128 experts, so the stale copy is mostly correct data. The
|
|
|
|
|
decisive test is `kld48.sh`: `llama-perplexity -c 48 -b 48 -ub 48`, 30 chunks of Pan Tadeusz, so
|
|
|
|
|
every chunk is one 48-token ubatch, KL divergence against a base with views on and knob 64 (the
|
|
|
|
|
tail MUL_MAT_ID on the CPU, correct by construction):
|
|
|
|
|
|
|
|
|
|
| arm | mean KLD | max KLD | same top token | RMS dp |
|
|
|
|
|
|---|---|---|---|---|
|
|
|
|
|
| views on, knob 32 (tail on the GPU) | 0.00314 | 0.087 | 97.4% | 1.35% |
|
|
|
|
|
| views off, knob 32 (tail on the GPU) | 0.03163 | 0.347 | 92.2% | 3.71% |
|
|
|
|
|
| views off, knob 64 | 0.00000 | - | 100% | 0.00% |
|
|
|
|
|
|
|
|
|
|
Views on is at the GPU-vs-CPU kernel noise level; views off is 10x worse (the #32755 mechanism,
|
|
|
|
|
reproduced). So with F13 the `GGML_OP_OFFLOAD_MIN_BATCH=64` workaround is no longer needed.
|
|
|
|
|
|
|
|
|
|
**Placement guide (UD-Q4_K_XL, views on).** k = number of loop layers (6, 7, ...) whose experts stay
|
|
|
|
|
on the GPU: `-ot 'blk[.](<the other 24-k layers>)[.]ffn_.*_exps=CPU'` instead of `--n-cpu-moe 24`
|
|
|
|
|
(`/root/arms/overnight/mkot.py` and `cfgedit.py` write the pattern; `[.]` avoids backslashes in the
|
|
|
|
|
YAML and in llama-swap's argument splitting). Each loop layer's experts take 547 MiB. Fit, from the
|
|
|
|
|
`-lv 4` breakdown of a server with the production command line (mmproj on demand, so not loaded):
|
|
|
|
|
|
|
|
|
|
| ctx | k | Vulkan0 self (model + context + compute) | free | verdict |
|
|
|
|
|
|---|---|---|---|---|
|
|
|
|
|
| 32768 | 0 | 4136 (1266 + 1347 + 1522) | 3982 | fits |
|
|
|
|
|
| 32768 | 6 | 7416 | 699 | fits |
|
|
|
|
|
| 32768 | 7 | 7963 | 15 | too tight |
|
|
|
|
|
| 65536 | 4 | 7246 (3453 + 2371 + 1421) | 868 | fits, chosen |
|
|
|
|
|
| 65536 | 5 | 7793 | 128 | too tight for the mmproj |
|
|
|
|
|
| 65536 | 6 | - | - | spills (GTT about 900 MiB above expectation) |
|
|
|
|
|
| 131072 | 0 | 7784 (1266 + 4419 + 2098) | 134 | fits (before F13 it overcommitted 5.2 GB) |
|
|
|
|
|
| 131072 | 2 | - | - | spills (about 1.2 GB) |
|
|
|
|
|
|
|
|
|
|
fdinfo `drm-memory-gtt` is not the spill test: it includes the pinned host weights
|
|
|
|
|
(Vulkan_Host). Compare GTT(k) with GTT(0) - k x 547 MiB, or read the breakdown.
|
|
|
|
|
|
|
|
|
|
Speed, llama-bench, one invocation per metric (`sweep.sh`, `-fa 1 -t 6 -b 1024 -ub 1024`, knob 64,
|
|
|
|
|
pp r1, tg r2):
|
|
|
|
|
|
|
|
|
|
| k | pp2048 @ d8192 | pp2048 @ d16384 | tg256 @ d4096 |
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
| 0 | 97.53 | 89.62 | 5.57 |
|
|
|
|
|
| 4 | 102.19 | 93.90 | 6.45 |
|
|
|
|
|
| 6 | 104.43 | 95.27 | 6.72 |
|
|
|
|
|
| 7 | 105.90 | 96.90 | 7.25 |
|
|
|
|
|
|
|
|
|
|
Decode gains most (+16% at k=4, +30% at k=7): every loop layer on the GPU removes 3 of the 48 CPU
|
|
|
|
|
expert passes per token. Prefill gains only 5-9%: at depth the attention and the remaining uploads
|
|
|
|
|
dominate.
|
|
|
|
|
|
|
|
|
|
**Quants (2026-11-22, not adopted).** Unsloth's Mainline_Support UD-Q3_K_XL (13.6 GB) and UD-IQ3_XXS
|
|
|
|
|
(11.7 GB) are in `/root/models/inkling/`. Quality: `llama-perplexity -c 2048 --chunks 4`, KL
|
|
|
|
|
divergence against UD-Q4_K_XL logits (`kld.sh`), on a fixed Pan Tadeusz chunk
|
|
|
|
|
(`/root/bench/entext/pt-chunk.txt`, characters 20000-100000) and wikitext-2 test:
|
|
|
|
|
|
|
|
|
|
| text | quant | PPL (Q4_K_XL) | PPL ratio | mean KLD | same top token |
|
|
|
|
|
|---|---|---|---|---|---|
|
|
|
|
|
| Pan Tadeusz | Q3_K_XL | 11.45 (11.30) | +1.4% | 0.0241 | 92.5% |
|
|
|
|
|
| Pan Tadeusz | IQ3_XXS | 11.52 (11.30) | +1.9% | 0.0421 | 90.1% |
|
|
|
|
|
| wikitext-2 | Q3_K_XL | 6.251 (6.230) | +0.3% | 0.0097 | 95.6% |
|
|
|
|
|
| wikitext-2 | IQ3_XXS | 6.307 (6.230) | +1.2% | 0.0214 | 94.2% |
|
|
|
|
|
|
|
|
|
|
Fit at ctx 65536: UD-Q3_K_XL keeps the loop-layer experts at 547 MiB each (like Q4), so k=5 (867
|
|
|
|
|
MiB free); UD-IQ3_XXS has 390 MiB per layer, k=7 (952 MiB free). Speed (separate invocations, so
|
|
|
|
|
within the ~7% cross-run noise for prefill): Q3_K_XL k=4 105.23 / 6.83, k=6 107.31 / 7.48;
|
|
|
|
|
IQ3_XXS k=4 105.59 / 6.44, k=7 108.48 / 7.86 (pp2048 @ d8192 / tg256 @ d4096; Q4_K_XL k=4
|
|
|
|
|
102.19 / 6.45). The best case, IQ3_XXS at k=7, is +6% prefill and +22% decode (about 7.9 instead
|
|
|
|
|
of 6.5 t/s) for a Polish KLD of 0.042; IQ3_XXS's CPU dot product is slower per byte (at k=4 it
|
|
|
|
|
decodes no faster than Q4). Not worth the quality loss for the owner's Polish use.
|
|
|
|
|
|
|
|
|
|
**Production (deployed 2026-11-22, `b13c41355`, ctx 65536, k=4, no knob).** Through llama-swap
|
|
|
|
|
(`/root/arms/overnight/post.py`, log `post.log`): Pan Tadeusz 9312 tokens 81.6 t/s prefill, +7958
|
|
|
|
|
tokens 89.3 t/s, +17255 tokens (28K total) 78.0 t/s, decode 5.9-6.0 t/s, answers correct (before:
|
|
|
|
|
about 46 t/s prefill). Think and nothink entries answer text, an image ("Red") and a tool call. A
|
|
|
|
|
tiny request after the 4B task model held the GPU takes 1.3-2.4 s longer than a warm one (the
|
|
|
|
|
arbiter restores 3.4 GiB of device weights). Disk cache save/restore works on Inkling (1329 tokens,
|
|
|
|
|
410 MiB). The first request on a fresh process is slow (162 tokens 11.7 t/s prefill, 2.6 t/s decode)
|
|
|
|
|
since the pipelines are compiled on first use (`--no-warmup`).
|
|
|
|
|
|
|
|
|
|
**Profile (2026-11-22, `perf record` + `gpu_busy_percent`, `prof.sh`, results in
|
|
|
|
|
`/root/arms/overnight/prof/`).** Inkling k=4 prefill (pp2048 @ d4096, 107 t/s): GPU busy 77%, CPU
|
|
|
|
|
time mostly `ggml_vk_wait_for_fence` (43%) and the kernel (11%, the uploads): GPU-bound. Decode
|
|
|
|
|
(7.3 t/s): GPU busy 58%, CPU in the OpenMP barrier spin (libgomp 36%) and
|
|
|
|
|
`ggml_vec_dot_nvfp4_q8_0` (21%): the UD-Q4_K_XL experts are NVFP4, and that CPU dot product is the
|
|
|
|
|
decode hotspot. Qwen3.6-35B for comparison: prefill 263 t/s at GPU busy 52%, decode 13.8 t/s at
|
|
|
|
|
35%, `vec_dot_q4_K_q8_K` 10%. `GGML_VK_PERF_LOGGER=1` aborts at the second split
|
|
|
|
|
(`GGML_ASSERT(ctx->compute_ctx.expired())`, ggml-vulkan.cpp:15692), so it gives per-op GPU times
|
|
|
|
|
only for the first split; not investigated.
|
|
|
|
|
|
|
|
|
|
**Failure modes and known limits.**
|
|
|
|
|
- LoRA adapters look weights up by name (`llama_adapter_lora::get_weight`), so they do not apply to
|
|
|
|
|
passes 1 and 2 with views on. Not used here; set `LLAMA_LOOP_HOST_VIEWS=0` for a LoRA on Inkling.
|
|
|
|
|
- `--moe-cache-mib`: the cache binds expert tensors per slot (`llama_moe_cache_layer_experts` of
|
|
|
|
|
`model.layers[il]`), so with views each loop slot gets its own binding and slot map and caches the
|
|
|
|
|
same experts separately (before, the three slots collided on one binding key). Should not crash;
|
|
|
|
|
not tested. The MoE cache is off in production.
|
|
|
|
|
- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data
|
|
|
|
|
in a saved file cannot happen.
|
|
|
|
|
|
|
|
|
|
**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for
|
|
|
|
|
every reader in the scheduler, or real per-pass tensors in the loader), revert 312c29c2b; the hook has
|
|
|
|
|
no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
|
|
|
|
|
|
|
|
|
|
**Rebase hazards.** `post_arch_tensors` and the loop slot layout in `inkling2-loop.cpp`;
|
|
|
|
|
`llama_model_base::load_tensors` returns (a new return path must call the hook); new pointer-keyed
|
|
|
|
|
weight maps in `llama_model` (they need a view entry like `w_scales`); non-pointer members in
|
|
|
|
|
`llama_layer`; scheduler changes to how host weights are copied (`hv_tensor_copies`,
|
|
|
|
|
`ggml_backend_sched_is_host_weight`).
|
|
|
|
|
|
|
|
|
|
**Validation (pod, GPU to itself in a swapdown window).** Build `llama-server` and `llama-bench`
|
|
|
|
|
from this branch in a scratch worktree (runbook section 5). Use the Inkling command line from
|
|
|
|
|
`inkling_args` in `/root/config.yaml` (with `--mmproj`) and change only the context to 16384 (so
|
|
|
|
|
arm A fits) and the placement:
|
|
|
|
|
- A, reference: replace `--n-cpu-moe 24` with `-ot 'blk\.([0-5]|1[89]|2[0-3])\.ffn_.*_exps=CPU'`
|
|
|
|
|
(loop experts on the GPU, no copies for the loop; the views log line must not appear).
|
|
|
|
|
- B, views on: `--n-cpu-moe 24`; the log line must show the views.
|
|
|
|
|
- C, views off: B with `LLAMA_LOOP_HOST_VIEWS=0`.
|
|
|
|
|
|
|
|
|
|
For each arm send raw token prompts (`/tokenize` a long text, cut to N tokens, `/completion` with the
|
|
|
|
|
token array, `temperature 0, top_k 1, seed 42, cache_prompt false, n_predict 32, n_probs 5`) at
|
|
|
|
|
N = 40, 63, 1072, 6176, 7224 (tail 32-63 at ub1024), controls N = 100, 7368 (tail >= 64) and 20.
|
|
|
|
|
Expected:
|
|
|
|
|
- B == A in the first token's top-5 probabilities for every N >= 32 (prefill MUL_MAT_ID runs on the
|
|
|
|
|
GPU in both arms with the same data). The generated text may drift later, since decode MUL_MAT_ID
|
|
|
|
|
runs on the GPU in A and on the CPU in B.
|
|
|
|
|
- B == C byte-identical for the controls (tail >= 64 or < 32).
|
|
|
|
|
- C differs from A at the 32-63 tails (the bug); if it does not, the bug was not reproduced and
|
|
|
|
|
B's result proves nothing.
|
|
|
|
|
- Vulkan0 compute buffer (startup log): B about 6 GiB below C. A's model buffer carries the loop
|
|
|
|
|
experts instead.
|
|
|
|
|
Then `llama-bench -lm none -ngl 99 -ncmoe 24 -fa 1 -t 6 -b 1024 -ub 1024 -p 2048 -n 0 -r 3`, with and
|
|
|
|
|
without `LLAMA_LOOP_HOST_VIEWS=0`, interleaved B C C B, medians (section 6), for the prefill cost
|
|
|
|
|
above. After a deploy: an Inkling text request, an image request, and the same text request after a
|
|
|
|
|
task-model call (arbiter handoff, F4), identical at temperature 0.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 4. Dead ends
|
|
|
|
@@ -1261,16 +1483,23 @@ Other models:
|
|
|
|
|
heads, head size 128; macro `inkling_args`, entries `unsloth/Inkling-2-Mini-GGUF[-nothink]:UD-Q4_K_XL`,
|
|
|
|
|
matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m`
|
|
|
|
|
(the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it,
|
|
|
|
|
`--n-cpu-moe 24` (all experts on host), `-t 6`, f16 KV, `--parallel 1`, the disk prompt cache.
|
|
|
|
|
Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 65536`
|
|
|
|
|
(before: `-b 512 -ub 512`; the dense per-head bias mask needed about 12.7 GB of host compute
|
|
|
|
|
buffer at 128k). 131072 loads and works with the carry, but overcommits about 5.2 GB of device
|
|
|
|
|
memory into GTT and is about 8% slower at prefill (F12). Both Inkling entries set
|
|
|
|
|
`env: ["GGML_OP_OFFLOAD_MIN_BATCH=64"]` (llama-swap per-model env): ubatches of 32-63 tokens keep
|
|
|
|
|
MUL_MAT_ID on the CPU. The looped layers share their expert tensors, and below F1's readback-skip
|
|
|
|
|
threshold (4 x 128 experts / 8 used = 64 tokens) only loop pass 0's routed experts would be
|
|
|
|
|
uploaded (the mechanism suspected behind upstream #32755). Its nothink macro uses
|
|
|
|
|
`--reasoning-effort none --reasoning-budget 0`.
|
|
|
|
|
`-t 6`, f16 KV, `--parallel 1`, the disk prompt cache, `-b 1024 -ub 1024 --ctx-size 65536`
|
|
|
|
|
(since the 2026-11-22 carry of upstream #32740, F12; before: `-b 512 -ub 512`). Since the F13
|
|
|
|
|
deploy (2026-11-22, `b13c41355`): the experts of loop layers 6-9 stay on the GPU and all others
|
|
|
|
|
on the host, `-ot blk[.](0|1|2|3|4|5|10|11|12|13|14|15|16|17|18|19|20|21|22|23)[.]ffn_.*_exps=CPU`
|
|
|
|
|
instead of `--n-cpu-moe 24` (k=4 of the F13 placement guide: 7246 MiB of device memory, 868 MiB
|
|
|
|
|
left for the mmproj). The `GGML_OP_OFFLOAD_MIN_BATCH=64` env of both entries was removed: F13
|
|
|
|
|
makes the 32-63-token ubatches correct (KLD in F13). Other choices from the guide: ctx 32768 with
|
|
|
|
|
k=6, or ctx 131072 with k=0 (fits since F13, 134 MiB free, so images would spill). Smaller quants
|
|
|
|
|
(UD-Q3_K_XL, UD-IQ3_XXS, `/root/models/inkling/`) were measured and not adopted (F13, Quants).
|
|
|
|
|
Its nothink macro uses `--reasoning-effort none` (no `--reasoning-budget 0`: that budget made it
|
|
|
|
|
loop on chat-structure tokens, F12); even with effort none it writes a short reasoning block.
|
|
|
|
|
Against Qwen3.6-35B (think entries, 7 tasks, one sample each, 3500-token cap, 2026-11-22,
|
|
|
|
|
`/root/arms/overnight/quality.py`, `quality/quality.json`): Inkling 6/7, Qwen 5/7. Inkling found
|
|
|
|
|
all three bugs of a binary search where Qwen ran out of budget; both ran out on a 4-line Polish
|
|
|
|
|
rhyme. Inkling used about a third fewer tokens (9677 vs 14254) but took 1.7x the wall time (2283
|
|
|
|
|
vs 1316 s; decode 4-6 vs 13 t/s) and 4x on a 29K-token question (835 vs 211 s). Too small a
|
|
|
|
|
sample to call it smarter; it is clearly slower here.
|
|
|
|
|
|
|
|
|
|
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
|
|
|
|
|
`/root/supervisord.conf`:
|
|
|
|
@@ -1345,10 +1574,9 @@ Machine-level:
|
|
|
|
|
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
|
|
|
|
|
null buffer would crash. Only the mmproj path handles a failed restore.
|
|
|
|
|
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
|
|
|
|
|
- Inkling (F12, section 5): the nothink entry returns repeated chat-structure tokens instead of an
|
|
|
|
|
answer for image requests, tool calls and long prompts (the think entry is fine); the config runs
|
|
|
|
|
ctx 65536 because 131072 overcommits about 5.2 GB of device memory into GTT; upstream #32755 /
|
|
|
|
|
#32778 did not reproduce here. Drop the F12 carries when upstream merges #32735 and #32740.
|
|
|
|
|
- Inkling (F12, section 5): the nothink entry's chat-structure loop came from
|
|
|
|
|
`--reasoning-budget 0` and was fixed in the config on 2026-11-22 (F12). Upstream #32755 / #32778
|
|
|
|
|
did not reproduce here; F13 bounds the loop-pass expert copies (more in F13). Drop the F12 carries when upstream merges #32735 and #32740.
|
|
|
|
|
- Open after the 2026-11-21 rebase: is the fork's concat fast path (F3) now redundant with upstream's
|
|
|
|
|
own tiled concat (#30591, gated in the fork with `ne0 < 32` and the `GGML_VK_CONCAT_FASTPATH` knob)?
|
|
|
|
|
Benchmark it. Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no
|
|
|
|
|