Compare commits

...
Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 166ec22e2b docs: FORK.md section 5, Inkling vs Qwen3.6-35B quality comparison
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:13:29 +01:00
LumpiastyandClaude Opus 5.5 e8386267ec docs: FORK.md/runbook after the b13c41355 deploy - F13 measured (KLD bug-window test, fit and placement guide, quants, profile), Inkling -ot k=4 without the offload knob, nothink fix, pointer 15
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 09:14:29 +01:00
LumpiastyandClaude Opus 5.5 b13c413555 docs: FORK.md F13, per-pass views of host weights in inkling2_loop
What and why (#32755 and the 6.5 GiB of shared expert copies), the knob,
F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the
pod validation plan (reference with the loop experts on the GPU, views on,
views off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:40:39 +01:00
LumpiastyandClaude Opus 5.5 312c29c2b1 llama: inkling2_loop - per-pass views of host weights for the later loop passes
inkling2_loop runs the core layers (6..17 on Inkling-2-Mini) three times, and
post_arch_tensors makes every cache slot share the tensors of its physical
layer. With the experts in host memory (--n-cpu-moe, -ot exps=CPU) the
scheduler makes one device copy per (host tensor, backend) per graph, at the
first split that reads it, and keeps it until the last reader. So:

- the expert copies of all 12 loop layers are alive at once (about 6.5 GiB on
  UD-Q4_K_XL), which is most of the ~7 GiB Vulkan0 compute buffer;
- sched_copy_experts runs only for the first reader. When it reads the ids back
  (ubatches of 32-63 tokens: MUL_MAT_ID is offloaded from 32 tokens, the F1
  skip starts at 4*n_expert ids), it uploads only the experts that pass 0
  selected, and passes 1 and 2 read stale VRAM for the others (upstream issue
  #32755, "GGGG" output).

Add a post_load_tensors hook to llama_model (called after the weights are
loaded, and on the no_alloc path after the dummy buffers are set). In
inkling2_loop it replaces, for each slot of a later pass, every tensor in a
host buffer with a view of it (name "<name>.visit<pass>", buffer and data of
the original). The scheduler hashes tensors by pointer, so each pass gets its
own copy, freed after that pass, and the copy callback sees the ids of that
pass. Weights in device buffers and CPU extra buffers (CPU_REPACK, not host)
stay shared. The pointer-keyed weight maps (w_scales, prec_policy, hdmd.rot)
get entries for the views. The views live in a model-owned context outside
ctxs_bufs, so the F4 release/restore, tensor counts and the memory breakdown
do not see them; they point only at host buffers, which F4 never releases.

LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors (A/B knob).

Not built or run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:40:39 +01:00
6 changed files with 387 additions and 34 deletions

No files matched your search

+250 -22
View File
@@ -6,13 +6,14 @@ which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-11-22):
- Production runs `3621025e8` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12), deployed
2026-11-22 03:52 with the Inkling retune (section 5). Validated with the full `fork-validate.sh`
set (phase 2 47 PASS, 0 FAIL; runbook section 6). Rollback pointer
`/root/DEPLOY-ROLLBACK-commit-14.txt` = `3bf6b55b3` (build copy `/root/arms/build-3bf6b55b3`,
config backup `/root/config.yaml.bak-20261122-inkling-ub1024-ctx65536`); next free pointer 15.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12680-3621025e8`) and changes
- Production runs `b13c41355` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, and F13, the
per-pass loop views for Inkling), deployed 2026-11-22 08:47 with the Inkling placement change
(section 5, F13). Validated with the full `fork-validate.sh` set in one window (52 PASS, 0 FAIL,
0 SKIP; runbook section 6). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-15.txt` = `3621025e8`
(build copy `/root/arms/build-3621025e8`, config backup `/root/config.yaml.bak-20261122-predeploy`;
restore both, the old build has no F13); next free pointer 16.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12683-b13c41355`) and changes
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
(section 3, "2026-11-21").
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
@@ -53,6 +54,7 @@ Status at the time of writing (2026-11-22):
- F10 Page-cache drop after a copying load
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -136,6 +138,7 @@ Oldest first. "Feature" points at the section below.
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1151,7 +1154,16 @@ production's Inkling command line with the nothink sampling, GPU to itself; driv
on every build, production `3bf6b55b3` with ub512 included, with and without
`GGML_OP_OFFLOAD_MIN_BATCH=64`, so the carry did not cause it. The think entry gets all three
right on both builds (image "Red", `get_weather({"city":"Warsaw"})`, 28.4K log "65" at 45.9 t/s
in production). Short text prompts work on the nothink entry. Open: fix the nothink macro.
in production). Short text prompts work on the nothink entry.
- Nothink fix (2026-11-22, config only): the cause was `--reasoning-budget 0`. With effort `none`
and no budget the model still writes a short reasoning block (100-550 characters) and then
answers; a budget of 0 cut that block off at once and the model looped on the chat-structure
tokens. The macro is now `--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --reasoning-effort none`.
Verified on production `3621025e8` (pod driver `/root/arms/overnight/nothink.py`, logs
`nothink1.log`, `nothink2.log` there): short question, Polish poem, image ("Red"), tool call
(`get_weather({"city":"Warsaw"})`) and a 29.4K-token Pan Tadeusz prompt (correct, 46.0 t/s) all
OK. Effort `minimal` and a budget of 256 also work; the budget of 0 fails on all but the short
question.
**Rebase hazards.**
- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry
@@ -1173,6 +1185,216 @@ about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rej
(the graph then uses the dense mask), so it prints no CPU lines).
Then an Inkling request through llama-swap.
### F13. Per-pass views of host weights in looped layers (inkling2_loop)
Branch `fable5/inkling-loop-views` (on top of the F12 carries), fast-forwarded into the production
branch on 2026-11-22 (deploy and results below).
**What and why.** `inkling2_loop` (Inkling-2-Mini) runs the core layers 6..17 three times; the 48
cache slots are 24 physical layers plus 2 x 12 repeats. `post_arch_tensors` makes every repeat slot
share the tensors of its physical layer (`layers[il] = layers[phys]`). With the experts in host
memory (`--n-cpu-moe 24`, `-ot exps=CPU`), `ggml_backend_sched` makes one device copy per (host
tensor, backend) per graph, hashed by tensor pointer, at the first split that reads it, and keeps it
allocated until the last reader (`ggml_backend_sched_split_graph`, pass 5 and the graph_copy input
nodes). Two consequences:
- Memory: the expert copies of all 12 loop layers are alive at the same time (about 547 MiB per
layer on UD-Q4_K_XL, 6.5 GiB in total). That is most of the ~7 GiB Vulkan0 compute buffer in the
F12 measurements.
- Correctness (upstream issue #32755, "GGGG" output): the copy callback (`sched_copy_experts`,
F1) runs once, for the first reader. When it reads the routing ids back (ubatches below the F1
skip threshold of `4 * n_expert` ids, and at or above the Vulkan MUL_MAT_ID offload batch of 32
tokens: 32-63 tokens on Inkling-2-Mini), it uploads only the experts pass 0 selected. Passes 1 and
2 reuse that copy and read stale VRAM for every other expert. Ubatches of 64+ tokens are correct
(F1 uploads every expert), ubatches below 32 run MUL_MAT_ID on the CPU (no copy).
**Where.** `src/llama-model.h` (`llama_model::post_load_tensors`, a new optional hook),
`src/llama-model.cpp` (`llama_model_base::load_tensors` calls it on the no_alloc return and after
`load_tensors_hadamard()`), `src/models/models.h` and `src/models/inkling2-loop.cpp`
(`llama_model_inkling2_loop::post_load_tensors`, member `ctx_visits`). Commit 312c29c2b.
**How it works.** After the weights are loaded, for every slot of pass 1 and 2 (`il >= n_phys`), each
tensor pointer in its `llama_layer` (walked as an array of pointers, as `llama_context::opt_init`
does) that sits in a host buffer (`ggml_backend_buffer_is_host`) is replaced with
`ggml_view_tensor` of it, named `<name>.visit<pass>`, initialized with `ggml_backend_view_init`
(same buffer and data). On the no_alloc pass (memory fit) only `view->buffer` is set to the dummy
buffer, so the scheduler still sees a WEIGHTS buffer. The view is a leaf (op NONE, `view_src` set),
so the scheduler hashes it separately: each pass gets its own copy, freed after the last reader of
that pass, and the copy callback sees the MUL_MAT_ID of that pass and its ids (`st.ids` differs, so
F1 recomputes the expert set). The pointer-keyed weight maps of `llama_model` (`w_scales`,
`prec_policy.prec_acc`/`prec_src1`, `hdmd.rot`) get entries for the views. The views live in
`ctx_visits`, a context of the model outside `ctxs_bufs`. The startup log prints
`post_load_tensors: N views of host weights for the loop passes after the first (X MiB, not copied)`.
Knob: `LLAMA_LOOP_HOST_VIEWS=0` keeps the shared tensors (the upstream behaviour), read once at load.
Default on.
**Invariants that must hold.**
- Only host buffers get views. Device weights stay shared (used in place, no copy, so no problem),
and so do CPU extra buffers (CPU_REPACK is not a host buffer; its ops run on the CPU in place).
- F4: views point only at host buffers, which `release_device_weights` never releases, and they
are not in `ctxs_bufs`, so the shadows, `weight_release`, `memory_breakdown`, `n_tensors()` and
`tensors_by_name` do not see them. A device weight is released and restored in place, and the
loop slots keep pointing at the same tensor.
- F1: the skip/readback decision is per copy and keys on the ids tensor of the first node of the
split, so it stays per pass.
- `llama_layer` must hold only `ggml_tensor *` members (a `static_assert` checks the size only).
**Cost.** Pass 1 and 2 now upload their own experts: per ubatch the host-to-device expert traffic
goes from 24 to 48 layer uploads (about +13 GiB per full ubatch on UD-Q4_K_XL). Measured where the
old layout still fits in VRAM (llama-bench pp2048 at depth 0, `-ncmoe 24`, B C C B in one window):
views 107.10 / 107.53 t/s, off 110.80 / 111.53, so -3.5%. Decode is unaffected (single-token
MUL_MAT_ID runs on the CPU). In the server at 16k-65k context the old layout does not fit (7 GiB of
compute buffer), spills into GTT, and the views win about 1.6x at prefill (Measured, below). The
freed ~6 GiB of VRAM can hold experts instead: an expert tensor of a loop layer kept on the GPU
saves 3 uploads per ubatch and 3 CPU expert passes per token, a non-loop one saves 1, so loop layers
are the first to move to the GPU (`-ot` with the loop layers left out of the CPU pattern; placement
guide below).
**Measured (2026-11-22, pod, GPU to itself, build `b13c41355` = 312c29c2b + docs; drivers and logs in
`/root/arms/overnight/`: `f13.py`, `cmp.py`, `kld48.sh`, `fit.py`, `sweep.sh`, `STATUS.md`).**
- Memory, server with the production command line at ctx 16384, `-lv 4`: Vulkan0 compute buffer
1426 MiB with views, 7030 MiB without. Process VRAM after load: views 4179 MiB, off 7727, A
(loop experts on the GPU, no views) 7879 (A wants 10089 MiB and spills).
- Prefill, server, 7224-token raw prompt at ctx 16384: views 92.4 t/s, off 57.0, A 46.5 (production
at ctx 65536 without views: about 46).
- Correctness. First-token top-5 logprobs (`f13.py`, `n_probs 5`, knob 32 so the 32-63 window is
live): with the tail at or above 64 or below 32 tokens (N = 20, 100, 7368) views on and off are
identical (max difference 0.000); in the window (N = 40, 63, 1072, 7224) they differ (0.04-0.16).
That test cannot say which arm is right: arm A differs from both by up to 0.15 even at the
controls (different placement, different kernels), and at 32-63 tokens x 8 experts pass 0's
union already covers most of the 128 experts, so the stale copy is mostly correct data. The
decisive test is `kld48.sh`: `llama-perplexity -c 48 -b 48 -ub 48`, 30 chunks of Pan Tadeusz, so
every chunk is one 48-token ubatch, KL divergence against a base with views on and knob 64 (the
tail MUL_MAT_ID on the CPU, correct by construction):
| arm | mean KLD | max KLD | same top token | RMS dp |
|---|---|---|---|---|
| views on, knob 32 (tail on the GPU) | 0.00314 | 0.087 | 97.4% | 1.35% |
| views off, knob 32 (tail on the GPU) | 0.03163 | 0.347 | 92.2% | 3.71% |
| views off, knob 64 | 0.00000 | - | 100% | 0.00% |
Views on is at the GPU-vs-CPU kernel noise level; views off is 10x worse (the #32755 mechanism,
reproduced). So with F13 the `GGML_OP_OFFLOAD_MIN_BATCH=64` workaround is no longer needed.
**Placement guide (UD-Q4_K_XL, views on).** k = number of loop layers (6, 7, ...) whose experts stay
on the GPU: `-ot 'blk[.](<the other 24-k layers>)[.]ffn_.*_exps=CPU'` instead of `--n-cpu-moe 24`
(`/root/arms/overnight/mkot.py` and `cfgedit.py` write the pattern; `[.]` avoids backslashes in the
YAML and in llama-swap's argument splitting). Each loop layer's experts take 547 MiB. Fit, from the
`-lv 4` breakdown of a server with the production command line (mmproj on demand, so not loaded):
| ctx | k | Vulkan0 self (model + context + compute) | free | verdict |
|---|---|---|---|---|
| 32768 | 0 | 4136 (1266 + 1347 + 1522) | 3982 | fits |
| 32768 | 6 | 7416 | 699 | fits |
| 32768 | 7 | 7963 | 15 | too tight |
| 65536 | 4 | 7246 (3453 + 2371 + 1421) | 868 | fits, chosen |
| 65536 | 5 | 7793 | 128 | too tight for the mmproj |
| 65536 | 6 | - | - | spills (GTT about 900 MiB above expectation) |
| 131072 | 0 | 7784 (1266 + 4419 + 2098) | 134 | fits (before F13 it overcommitted 5.2 GB) |
| 131072 | 2 | - | - | spills (about 1.2 GB) |
fdinfo `drm-memory-gtt` is not the spill test: it includes the pinned host weights
(Vulkan_Host). Compare GTT(k) with GTT(0) - k x 547 MiB, or read the breakdown.
Speed, llama-bench, one invocation per metric (`sweep.sh`, `-fa 1 -t 6 -b 1024 -ub 1024`, knob 64,
pp r1, tg r2):
| k | pp2048 @ d8192 | pp2048 @ d16384 | tg256 @ d4096 |
|---|---|---|---|
| 0 | 97.53 | 89.62 | 5.57 |
| 4 | 102.19 | 93.90 | 6.45 |
| 6 | 104.43 | 95.27 | 6.72 |
| 7 | 105.90 | 96.90 | 7.25 |
Decode gains most (+16% at k=4, +30% at k=7): every loop layer on the GPU removes 3 of the 48 CPU
expert passes per token. Prefill gains only 5-9%: at depth the attention and the remaining uploads
dominate.
**Quants (2026-11-22, not adopted).** Unsloth's Mainline_Support UD-Q3_K_XL (13.6 GB) and UD-IQ3_XXS
(11.7 GB) are in `/root/models/inkling/`. Quality: `llama-perplexity -c 2048 --chunks 4`, KL
divergence against UD-Q4_K_XL logits (`kld.sh`), on a fixed Pan Tadeusz chunk
(`/root/bench/entext/pt-chunk.txt`, characters 20000-100000) and wikitext-2 test:
| text | quant | PPL (Q4_K_XL) | PPL ratio | mean KLD | same top token |
|---|---|---|---|---|---|
| Pan Tadeusz | Q3_K_XL | 11.45 (11.30) | +1.4% | 0.0241 | 92.5% |
| Pan Tadeusz | IQ3_XXS | 11.52 (11.30) | +1.9% | 0.0421 | 90.1% |
| wikitext-2 | Q3_K_XL | 6.251 (6.230) | +0.3% | 0.0097 | 95.6% |
| wikitext-2 | IQ3_XXS | 6.307 (6.230) | +1.2% | 0.0214 | 94.2% |
Fit at ctx 65536: UD-Q3_K_XL keeps the loop-layer experts at 547 MiB each (like Q4), so k=5 (867
MiB free); UD-IQ3_XXS has 390 MiB per layer, k=7 (952 MiB free). Speed (separate invocations, so
within the ~7% cross-run noise for prefill): Q3_K_XL k=4 105.23 / 6.83, k=6 107.31 / 7.48;
IQ3_XXS k=4 105.59 / 6.44, k=7 108.48 / 7.86 (pp2048 @ d8192 / tg256 @ d4096; Q4_K_XL k=4
102.19 / 6.45). The best case, IQ3_XXS at k=7, is +6% prefill and +22% decode (about 7.9 instead
of 6.5 t/s) for a Polish KLD of 0.042; IQ3_XXS's CPU dot product is slower per byte (at k=4 it
decodes no faster than Q4). Not worth the quality loss for the owner's Polish use.
**Production (deployed 2026-11-22, `b13c41355`, ctx 65536, k=4, no knob).** Through llama-swap
(`/root/arms/overnight/post.py`, log `post.log`): Pan Tadeusz 9312 tokens 81.6 t/s prefill, +7958
tokens 89.3 t/s, +17255 tokens (28K total) 78.0 t/s, decode 5.9-6.0 t/s, answers correct (before:
about 46 t/s prefill). Think and nothink entries answer text, an image ("Red") and a tool call. A
tiny request after the 4B task model held the GPU takes 1.3-2.4 s longer than a warm one (the
arbiter restores 3.4 GiB of device weights). Disk cache save/restore works on Inkling (1329 tokens,
410 MiB). The first request on a fresh process is slow (162 tokens 11.7 t/s prefill, 2.6 t/s decode)
since the pipelines are compiled on first use (`--no-warmup`).
**Profile (2026-11-22, `perf record` + `gpu_busy_percent`, `prof.sh`, results in
`/root/arms/overnight/prof/`).** Inkling k=4 prefill (pp2048 @ d4096, 107 t/s): GPU busy 77%, CPU
time mostly `ggml_vk_wait_for_fence` (43%) and the kernel (11%, the uploads): GPU-bound. Decode
(7.3 t/s): GPU busy 58%, CPU in the OpenMP barrier spin (libgomp 36%) and
`ggml_vec_dot_nvfp4_q8_0` (21%): the UD-Q4_K_XL experts are NVFP4, and that CPU dot product is the
decode hotspot. Qwen3.6-35B for comparison: prefill 263 t/s at GPU busy 52%, decode 13.8 t/s at
35%, `vec_dot_q4_K_q8_K` 10%. `GGML_VK_PERF_LOGGER=1` aborts at the second split
(`GGML_ASSERT(ctx->compute_ctx.expired())`, ggml-vulkan.cpp:15692), so it gives per-op GPU times
only for the first split; not investigated.
**Failure modes and known limits.**
- LoRA adapters look weights up by name (`llama_adapter_lora::get_weight`), so they do not apply to
passes 1 and 2 with views on. Not used here; set `LLAMA_LOOP_HOST_VIEWS=0` for a LoRA on Inkling.
- `--moe-cache-mib`: the cache binds expert tensors per slot (`llama_moe_cache_layer_experts` of
`model.layers[il]`), so with views each loop slot gets its own binding and slot map and caches the
same experts separately (before, the three slots collided on one binding key). Should not crash;
not tested. The MoE cache is off in production.
- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data
in a saved file cannot happen.
**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for
every reader in the scheduler, or real per-pass tensors in the loader), revert 312c29c2b; the hook has
no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
**Rebase hazards.** `post_arch_tensors` and the loop slot layout in `inkling2-loop.cpp`;
`llama_model_base::load_tensors` returns (a new return path must call the hook); new pointer-keyed
weight maps in `llama_model` (they need a view entry like `w_scales`); non-pointer members in
`llama_layer`; scheduler changes to how host weights are copied (`hv_tensor_copies`,
`ggml_backend_sched_is_host_weight`).
**Validation (pod, GPU to itself in a swapdown window).** Build `llama-server` and `llama-bench`
from this branch in a scratch worktree (runbook section 5). Use the Inkling command line from
`inkling_args` in `/root/config.yaml` (with `--mmproj`) and change only the context to 16384 (so
arm A fits) and the placement:
- A, reference: replace `--n-cpu-moe 24` with `-ot 'blk\.([0-5]|1[89]|2[0-3])\.ffn_.*_exps=CPU'`
(loop experts on the GPU, no copies for the loop; the views log line must not appear).
- B, views on: `--n-cpu-moe 24`; the log line must show the views.
- C, views off: B with `LLAMA_LOOP_HOST_VIEWS=0`.
For each arm send raw token prompts (`/tokenize` a long text, cut to N tokens, `/completion` with the
token array, `temperature 0, top_k 1, seed 42, cache_prompt false, n_predict 32, n_probs 5`) at
N = 40, 63, 1072, 6176, 7224 (tail 32-63 at ub1024), controls N = 100, 7368 (tail >= 64) and 20.
Expected:
- B == A in the first token's top-5 probabilities for every N >= 32 (prefill MUL_MAT_ID runs on the
GPU in both arms with the same data). The generated text may drift later, since decode MUL_MAT_ID
runs on the GPU in A and on the CPU in B.
- B == C byte-identical for the controls (tail >= 64 or < 32).
- C differs from A at the 32-63 tails (the bug); if it does not, the bug was not reproduced and
B's result proves nothing.
- Vulkan0 compute buffer (startup log): B about 6 GiB below C. A's model buffer carries the loop
experts instead.
Then `llama-bench -lm none -ngl 99 -ncmoe 24 -fa 1 -t 6 -b 1024 -ub 1024 -p 2048 -n 0 -r 3`, with and
without `LLAMA_LOOP_HOST_VIEWS=0`, interleaved B C C B, medians (section 6), for the prefill cost
above. After a deploy: an Inkling text request, an image request, and the same text request after a
task-model call (arbiter handoff, F4), identical at temperature 0.
---
## 4. Dead ends
@@ -1261,16 +1483,23 @@ Other models:
heads, head size 128; macro `inkling_args`, entries `unsloth/Inkling-2-Mini-GGUF[-nothink]:UD-Q4_K_XL`,
matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m`
(the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it,
`--n-cpu-moe 24` (all experts on host), `-t 6`, f16 KV, `--parallel 1`, the disk prompt cache.
Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 65536`
(before: `-b 512 -ub 512`; the dense per-head bias mask needed about 12.7 GB of host compute
buffer at 128k). 131072 loads and works with the carry, but overcommits about 5.2 GB of device
memory into GTT and is about 8% slower at prefill (F12). Both Inkling entries set
`env: ["GGML_OP_OFFLOAD_MIN_BATCH=64"]` (llama-swap per-model env): ubatches of 32-63 tokens keep
MUL_MAT_ID on the CPU. The looped layers share their expert tensors, and below F1's readback-skip
threshold (4 x 128 experts / 8 used = 64 tokens) only loop pass 0's routed experts would be
uploaded (the mechanism suspected behind upstream #32755). Its nothink macro uses
`--reasoning-effort none --reasoning-budget 0`.
`-t 6`, f16 KV, `--parallel 1`, the disk prompt cache, `-b 1024 -ub 1024 --ctx-size 65536`
(since the 2026-11-22 carry of upstream #32740, F12; before: `-b 512 -ub 512`). Since the F13
deploy (2026-11-22, `b13c41355`): the experts of loop layers 6-9 stay on the GPU and all others
on the host, `-ot blk[.](0|1|2|3|4|5|10|11|12|13|14|15|16|17|18|19|20|21|22|23)[.]ffn_.*_exps=CPU`
instead of `--n-cpu-moe 24` (k=4 of the F13 placement guide: 7246 MiB of device memory, 868 MiB
left for the mmproj). The `GGML_OP_OFFLOAD_MIN_BATCH=64` env of both entries was removed: F13
makes the 32-63-token ubatches correct (KLD in F13). Other choices from the guide: ctx 32768 with
k=6, or ctx 131072 with k=0 (fits since F13, 134 MiB free, so images would spill). Smaller quants
(UD-Q3_K_XL, UD-IQ3_XXS, `/root/models/inkling/`) were measured and not adopted (F13, Quants).
Its nothink macro uses `--reasoning-effort none` (no `--reasoning-budget 0`: that budget made it
loop on chat-structure tokens, F12); even with effort none it writes a short reasoning block.
Against Qwen3.6-35B (think entries, 7 tasks, one sample each, 3500-token cap, 2026-11-22,
`/root/arms/overnight/quality.py`, `quality/quality.json`): Inkling 6/7, Qwen 5/7. Inkling found
all three bugs of a binary search where Qwen ran out of budget; both ran out on a 4-line Polish
rhyme. Inkling used about a third fewer tokens (9677 vs 14254) but took 1.7x the wall time (2283
vs 1316 s; decode 4-6 vs 13 t/s) and 4x on a 29K-token question (835 vs 211 s). Too small a
sample to call it smarter; it is clearly slower here.
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
`/root/supervisord.conf`:
@@ -1345,10 +1574,9 @@ Machine-level:
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
null buffer would crash. Only the mmproj path handles a failed restore.
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
- Inkling (F12, section 5): the nothink entry returns repeated chat-structure tokens instead of an
answer for image requests, tool calls and long prompts (the think entry is fine); the config runs
ctx 65536 because 131072 overcommits about 5.2 GB of device memory into GTT; upstream #32755 /
#32778 did not reproduce here. Drop the F12 carries when upstream merges #32735 and #32740.
- Inkling (F12, section 5): the nothink entry's chat-structure loop came from
`--reasoning-budget 0` and was fixed in the config on 2026-11-22 (F12). Upstream #32755 / #32778
did not reproduce here; F13 bounds the loop-pass expert copies (more in F13). Drop the F12 carries when upstream merges #32735 and #32740.
- Open after the 2026-11-21 rebase: is the fork's concat fast path (F3) now redundant with upstream's
own tiled concat (#30591, gated in the fork with `ne0 < 32` and the `GGML_VK_CONCAT_FASTPATH` knob)?
Benchmark it. Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no
+25 -12
View File
@@ -348,14 +348,21 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- Last full run (2026-11-22, candidate `3621025e8` = `3bf6b55b3` + the F12 carries, vs reference
- Last full run (2026-11-22, candidate `b13c41355` = `3621025e8` + F13 vs reference `3621025e8`, one
swapdown window, `--ops --concat --diskcache --arbiter --perf --moe-exact --moecache`, log
`/root/arms/overnight/fv.log`): 52 PASS, 0 FAIL, 0 SKIP in 41 minutes. Perf medians ref -> cand:
pp2048 286.61 -> 284.94 (-0.6%), d16384 207.93 -> 207.96, d32768 161.93 -> 162.12, tg256@d4096
14.11 -> 14.47; handoff median ref 715 / cand 706.5 ms; peak VRAM ref 5478 / cand 7771 MiB (the
first-arm spike of FORK.md F9, this time in a candidate arm). With the card free, phase 1 and 2
can run as one invocation like this.
- Previous full run (2026-11-22, candidate `3621025e8` = `3bf6b55b3` + the F12 carries, vs reference
`3bf6b55b3`; logs `/root/arms/carry1122-p1.log`, `-p2.log`): phase 1 `--ops --concat --diskcache`
20 PASS, 1 FAIL (`concat/candidate-timing` beside a full card, see above), 1 SKIP (diskcache GPU
leg); phase 2 `--concat --arbiter --diskcache --perf --moe-exact --moecache` 47 PASS, 0 FAIL, 0
SKIP. Perf medians ref -> cand: pp2048 286.97 -> 286.32, d16384 208.31 -> 208.21, d32768 162.25 ->
162.37; handoff median ref 696 / cand 700 ms; peak VRAM ref 7771 / cand 5502 MiB. Plus SCALE
15/15 and `test-flash-attn-indexed-bias` (FORK.md F12). Window about 30 minutes.
- Previous full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
- Earlier full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
9 PASS plus the 35B disk-cache legs (`q35.py`, mmproj loaded). Perf medians ref -> cand: pp2048
259.39 -> 286.61 (+10.5%), d16384 199.26 -> 207.63 (+4.2%), d32768 156.61 -> 157.51 (+0.6%),
tg256@d4096 9.17 -> 8.85 t/s (noisy, not gated). Handoff median: candidate 692 ms, reference
@@ -404,8 +411,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 14 (holds 3bf6b55b3)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-15.txt
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 15 (holds 3621025e8)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-16.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3):
@@ -544,11 +551,17 @@ on `reasoning_content` and returns an empty `content`; for one, raise `max_token
whether any falls in the deploy window (2026-10-25: the newest was about 19 h old).
8. Inkling (FORK.md section 5, F12): judge the answer text, not only HTTP 200. The first request
after a fresh load should be a prompt of about 150 tokens (upstream #32778 gives all-'?' output
there); then a long prompt (Pan Tadeusz, 8K and 14K tokens, upstream #32755). Image and tool
requests go to the think entry: the nothink entry returns repeated
`<|message_model|><|content_thinking|><|end_message|>` for them on every build (FORK.md F12).
Check that the env of the running process has `GGML_OP_OFFLOAD_MIN_BATCH=64`
(`tr '\0' '\n' < /proc/<pid>/environ`). The memory breakdown table is printed only at `-lv 4`,
there); then a long prompt (Pan Tadeusz, 8K and 14K tokens, upstream #32755). Send an image
and a tool request to both entries: since the nothink macro dropped `--reasoning-budget 0`
(FORK.md F12) both answer them; repeated `<|message_model|><|content_thinking|><|end_message|>`
from the nothink entry means a budget of 0 is back. Since F13 a scratch server with the
production command line plus `-lv 4` must log `post_load_tensors: N views of host weights`
(production's own log level hides libllama's info lines, so the llama-swap log never shows it),
and the entries no longer set
`GGML_OP_OFFLOAD_MIN_BATCH=64` (F13 measured the 32-63-token window correct without it). The
placement is `-ot` with the loop layers 6..6+k-1 left on the GPU (FORK.md F13 placement guide);
after a change of context or quant, check the fit again (the breakdown must leave room for the
mmproj, about 0.8 GiB). The memory breakdown table is printed only at `-lv 4`,
which production does not set, so it is not in the llama-swap log: read it from a scratch server
started with production's command line plus `-lv 4`. Drivers used on 2026-11-22 (pod only):
`/root/arms/carry1122-post.py` (through llama-swap) and `/root/arms/carry1122-ink-direct.py`
@@ -600,10 +613,10 @@ chain in the exec shell: it can die with the exec session halfway, leaving llama
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
Pointer files 2 to 13 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
Pointer files 2 to 15 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, commit-12 holds
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, and commit-14 holds `3bf6b55b3`, the rollback target
for the current production `3621025e8` (next free: 15). The formats vary
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, commit-14 holds `3bf6b55b3`, and commit-15 holds `3621025e8`, the rollback target
for the current production `b13c41355` (next free: 16). The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
+3
View File
@@ -2253,6 +2253,7 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
}
if (ml.no_alloc) {
post_load_tensors(ml);
return true;
}
@@ -2283,6 +2284,8 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
load_tensors_hadamard();
post_load_tensors(ml);
return true;
}
+3
View File
@@ -1091,6 +1091,9 @@ struct llama_model {
// optional, runs after the scale pass
virtual void post_arch_tensors(llama_model_loader &) {}
// optional, runs after the weight buffers are allocated and loaded (with no_alloc: after the dummy buffers are set)
virtual void post_load_tensors(llama_model_loader &) {}
protected:
llama_model_params params;
+102
View File
@@ -1,5 +1,12 @@
#include "models.h"
#include <algorithm>
#include <cstdlib>
#include <cstring>
#include <stdexcept>
#include <unordered_map>
#include <vector>
int llama_model_inkling2_loop::layer_phys(int il) const {
return il < n_phys ? il : core_start + (il - n_phys) % (core_end - core_start);
}
@@ -69,6 +76,101 @@ void llama_model_inkling2_loop::post_arch_tensors(llama_model_loader &) {
}
}
// the scheduler makes one device copy of a host weight per graph, alive from its first to its last use, and the expert copy
// callback runs at the first use only: give each later pass its own view, so it gets its own copy with the experts of its own ids
// knob: LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors
void llama_model_inkling2_loop::post_load_tensors(llama_model_loader & ml) {
const char * env = getenv("LLAMA_LOOP_HOST_VIEWS");
if (env != nullptr && strcmp(env, "0") == 0) {
LLAMA_LOG_INFO("%s: LLAMA_LOOP_HOST_VIEWS=0, the loop passes share the host weights\n", __func__);
return;
}
// llama_layer holds only tensor pointers, see llama_context::opt_init
static_assert(sizeof(llama_layer) % sizeof(ggml_tensor *) == 0, "llama_layer must hold only tensor pointers");
const size_t n_ptr = sizeof(llama_layer)/sizeof(ggml_tensor *);
const int n_slot = (int) hparams.n_layer();
const int n_core = (int) (core_end - core_start);
// weights on a device are used in place, only host weights are copied
std::vector<std::vector<ggml_tensor *>> host(n_slot);
size_t n_views = 0;
for (int il = n_phys; il < n_slot; ++il) {
ggml_tensor ** ts = reinterpret_cast<ggml_tensor **>(&layers[il]);
for (size_t i = 0; i < n_ptr; ++i) {
ggml_tensor * t = ts[i];
if (t == nullptr || t->buffer == nullptr || !ggml_backend_buffer_is_host(t->buffer) || (!ml.no_alloc && t->data == nullptr)) {
continue;
}
if (std::find(host[il].begin(), host[il].end(), t) == host[il].end()) {
host[il].push_back(t);
}
}
n_views += host[il].size();
}
if (n_views == 0) {
return;
}
ggml_init_params init_params = {
/*.mem_size =*/ n_views*ggml_tensor_overhead(),
/*.mem_buffer =*/ nullptr,
/*.no_alloc =*/ true,
};
ctx_visits.reset(ggml_init(init_params));
if (!ctx_visits) {
throw std::runtime_error("inkling2_loop: failed to create the context of the loop views");
}
size_t n_bytes = 0;
for (int il = n_phys; il < n_slot; ++il) {
const int pass = 1 + (il - n_phys)/n_core;
std::unordered_map<const ggml_tensor *, ggml_tensor *> view_of;
for (ggml_tensor * t : host[il]) {
ggml_tensor * v = ggml_view_tensor(ctx_visits.get(), t);
ggml_format_name(v, "%s.visit%d", t->name, pass);
if (ml.no_alloc) {
// the dummy buffer has no data, it only tells the scheduler that this is a weight
v->buffer = t->buffer;
} else if (ggml_backend_view_init(v) != GGML_STATUS_SUCCESS) {
throw std::runtime_error(format("inkling2_loop: failed to initialize the view %s", v->name));
}
view_of[t] = v;
n_bytes += ggml_nbytes(t);
// the per-weight maps are keyed by the tensor
if (const auto it = w_scales.find(t); it != w_scales.end()) {
ggml_tensor * s = it->second;
w_scales.emplace(v, s);
}
if (const auto it = prec_policy.prec_acc.find(t); it != prec_policy.prec_acc.end()) {
const ggml_prec prec = it->second;
prec_policy.prec_acc.emplace(v, prec);
}
if (const auto it = prec_policy.prec_src1.find(t); it != prec_policy.prec_src1.end()) {
const ggml_prec prec = it->second;
prec_policy.prec_src1.emplace(v, prec);
}
if (const auto it = hdmd.rot.find(t); it != hdmd.rot.end()) {
const llama_hadamard_transform tr = it->second;
hdmd.rot.emplace(v, tr);
}
}
ggml_tensor ** ts = reinterpret_cast<ggml_tensor **>(&layers[il]);
for (size_t i = 0; i < n_ptr; ++i) {
if (const auto it = view_of.find(ts[i]); it != view_of.end()) {
ts[i] = it->second;
}
}
}
LLAMA_LOG_INFO("%s: %zu views of host weights for the loop passes after the first (%.2f MiB, not copied)\n",
__func__, n_views, n_bytes/1024.0/1024.0);
}
std::unique_ptr<llm_graph_context> llama_model_inkling2_loop::build_arch_graph(const llm_graph_params & params) const {
return std::make_unique<graph>(params);
}
+4
View File
@@ -2559,6 +2559,7 @@ struct llama_model_inkling2_loop : public llama_model_inkling2 {
void load_arch_hparams(llama_model_loader & ml) override;
void load_arch_tensors(llama_model_loader & ml) override;
void post_arch_tensors(llama_model_loader & ml) override;
void post_load_tensors(llama_model_loader & ml) override;
int n_layer_phys() const override { return n_phys; }
int layer_phys(int il) const;
@@ -2573,6 +2574,9 @@ struct llama_model_inkling2_loop : public llama_model_inkling2 {
// indexed by physical layer - core_start
std::vector<loop_layer> loop_layers;
// views of the host weights for the cache slots of the later passes, see post_load_tensors
ggml_context_ptr ctx_visits;
struct graph : public llama_model_inkling2::graph {
graph(const llm_graph_params & params);
};