docs: FORK.md F13, per-pass views of host weights in inkling2_loop

What and why (#32755 and the 6.5 GiB of shared expert copies), the knob,
F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the
pod validation plan (reference with the loop experts on the GPU, views on,
views off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-11-22 05:40:39 +01:00
1 parent 312c29c2b1
commit b13c413555
1 file changed
+108
+108
View File
@@ -53,6 +53,7 @@ Status at the time of writing (2026-11-22):
- F10 Page-cache drop after a copying load
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -136,6 +137,7 @@ Oldest first. "Feature" points at the section below.
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
| 28ee3b735 | llama: inkling2_loop - per-pass views of host weights for the later loop passes (branch `fable5/inkling-loop-views`, not deployed) | F13 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1173,6 +1175,112 @@ about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rej
(the graph then uses the dense mask), so it prints no CPU lines).
Then an Inkling request through llama-swap.
### F13. Per-pass views of host weights in looped layers (inkling2_loop)
Branch `fable5/inkling-loop-views` (on top of the F12 carries). Not built, run or deployed yet.
**What and why.** `inkling2_loop` (Inkling-2-Mini) runs the core layers 6..17 three times; the 48
cache slots are 24 physical layers plus 2 x 12 repeats. `post_arch_tensors` makes every repeat slot
share the tensors of its physical layer (`layers[il] = layers[phys]`). With the experts in host
memory (`--n-cpu-moe 24`, `-ot exps=CPU`), `ggml_backend_sched` makes one device copy per (host
tensor, backend) per graph, hashed by tensor pointer, at the first split that reads it, and keeps it
allocated until the last reader (`ggml_backend_sched_split_graph`, pass 5 and the graph_copy input
nodes). Two consequences:
- Memory: the expert copies of all 12 loop layers are alive at the same time (about 547 MiB per
layer on UD-Q4_K_XL, 6.5 GiB in total). That is most of the ~7 GiB Vulkan0 compute buffer in the
F12 measurements.
- Correctness (upstream issue #32755, "GGGG" output): the copy callback (`sched_copy_experts`,
F1) runs once, for the first reader. When it reads the routing ids back (ubatches below the F1
skip threshold of `4 * n_expert` ids, and at or above the Vulkan MUL_MAT_ID offload batch of 32
tokens: 32-63 tokens on Inkling-2-Mini), it uploads only the experts pass 0 selected. Passes 1 and
2 reuse that copy and read stale VRAM for every other expert. Ubatches of 64+ tokens are correct
(F1 uploads every expert), ubatches below 32 run MUL_MAT_ID on the CPU (no copy).
**Where.** `src/llama-model.h` (`llama_model::post_load_tensors`, a new optional hook),
`src/llama-model.cpp` (`llama_model_base::load_tensors` calls it on the no_alloc return and after
`load_tensors_hadamard()`), `src/models/models.h` and `src/models/inkling2-loop.cpp`
(`llama_model_inkling2_loop::post_load_tensors`, member `ctx_visits`). Commit 28ee3b735.
**How it works.** After the weights are loaded, for every slot of pass 1 and 2 (`il >= n_phys`), each
tensor pointer in its `llama_layer` (walked as an array of pointers, as `llama_context::opt_init`
does) that sits in a host buffer (`ggml_backend_buffer_is_host`) is replaced with
`ggml_view_tensor` of it, named `<name>.visit<pass>`, initialized with `ggml_backend_view_init`
(same buffer and data). On the no_alloc pass (memory fit) only `view->buffer` is set to the dummy
buffer, so the scheduler still sees a WEIGHTS buffer. The view is a leaf (op NONE, `view_src` set),
so the scheduler hashes it separately: each pass gets its own copy, freed after the last reader of
that pass, and the copy callback sees the MUL_MAT_ID of that pass and its ids (`st.ids` differs, so
F1 recomputes the expert set). The pointer-keyed weight maps of `llama_model` (`w_scales`,
`prec_policy.prec_acc`/`prec_src1`, `hdmd.rot`) get entries for the views. The views live in
`ctx_visits`, a context of the model outside `ctxs_bufs`. The startup log prints
`post_load_tensors: N views of host weights for the loop passes after the first (X MiB, not copied)`.
Knob: `LLAMA_LOOP_HOST_VIEWS=0` keeps the shared tensors (the upstream behaviour), read once at load.
Default on.
**Invariants that must hold.**
- Only host buffers get views. Device weights stay shared (used in place, no copy, so no problem),
and so do CPU extra buffers (CPU_REPACK is not a host buffer; its ops run on the CPU in place).
- F4: views point only at host buffers, which `release_device_weights` never releases, and they
are not in `ctxs_bufs`, so the shadows, `weight_release`, `memory_breakdown`, `n_tensors()` and
`tensors_by_name` do not see them. A device weight is released and restored in place, and the
loop slots keep pointing at the same tensor.
- F1: the skip/readback decision is per copy and keys on the ids tensor of the first node of the
split, so it stays per pass.
- `llama_layer` must hold only `ggml_tensor *` members (a `static_assert` checks the size only).
**Cost.** Pass 1 and 2 now upload their own experts: per ubatch the host-to-device expert traffic
goes from 24 to 48 layer uploads (about +13 GiB per full ubatch on UD-Q4_K_XL). Prefill may lose
on the order of 10% (estimate, not measured); decode is unaffected (single-token MUL_MAT_ID runs on
the CPU). The freed ~6 GiB of VRAM can hold experts instead: an expert tensor of a loop layer kept on
the GPU saves 3 uploads per ubatch, a non-loop one saves 1, so loop layers are the first to move
to the GPU (`-ot` with the loop layers left out of the CPU pattern). Not measured.
**Failure modes and known limits.**
- LoRA adapters look weights up by name (`llama_adapter_lora::get_weight`), so they do not apply to
passes 1 and 2 with views on. Not used here; set `LLAMA_LOOP_HOST_VIEWS=0` for a LoRA on Inkling.
- `--moe-cache-mib`: the cache binds expert tensors per slot (`llama_moe_cache_layer_experts` of
`model.layers[il]`), so with views each loop slot gets its own binding and slot map and caches the
same experts separately (before, the three slots collided on one binding key). Should not crash;
not tested. The MoE cache is off in production.
- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data
in a saved file cannot happen.
**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for
every reader in the scheduler, or real per-pass tensors in the loader), revert 28ee3b735; the hook has
no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
**Rebase hazards.** `post_arch_tensors` and the loop slot layout in `inkling2-loop.cpp`;
`llama_model_base::load_tensors` returns (a new return path must call the hook); new pointer-keyed
weight maps in `llama_model` (they need a view entry like `w_scales`); non-pointer members in
`llama_layer`; scheduler changes to how host weights are copied (`hv_tensor_copies`,
`ggml_backend_sched_is_host_weight`).
**Validation (pod, GPU to itself in a swapdown window).** Build `llama-server` and `llama-bench`
from this branch in a scratch worktree (runbook section 5). Use the Inkling command line from
`inkling_args` in `/root/config.yaml` (with `--mmproj`) and change only the context to 16384 (so
arm A fits) and the placement:
- A, reference: replace `--n-cpu-moe 24` with `-ot 'blk\.([0-5]|1[89]|2[0-3])\.ffn_.*_exps=CPU'`
(loop experts on the GPU, no copies for the loop; the views log line must not appear).
- B, views on: `--n-cpu-moe 24`; the log line must show the views.
- C, views off: B with `LLAMA_LOOP_HOST_VIEWS=0`.
For each arm send raw token prompts (`/tokenize` a long text, cut to N tokens, `/completion` with the
token array, `temperature 0, top_k 1, seed 42, cache_prompt false, n_predict 32, n_probs 5`) at
N = 40, 63, 1072, 6176, 7224 (tail 32-63 at ub1024), controls N = 100, 7368 (tail >= 64) and 20.
Expected:
- B == A in the first token's top-5 probabilities for every N >= 32 (prefill MUL_MAT_ID runs on the
GPU in both arms with the same data). The generated text may drift later, since decode MUL_MAT_ID
runs on the GPU in A and on the CPU in B.
- B == C byte-identical for the controls (tail >= 64 or < 32).
- C differs from A at the 32-63 tails (the bug); if it does not, the bug was not reproduced and
B's result proves nothing.
- Vulkan0 compute buffer (startup log): B about 6 GiB below C. A's model buffer carries the loop
experts instead.
Then `llama-bench -lm none -ngl 99 -ncmoe 24 -fa 1 -t 6 -b 1024 -ub 1024 -p 2048 -n 0 -r 3`, with and
without `LLAMA_LOOP_HOST_VIEWS=0`, interleaved B C C B, medians (section 6), for the prefill cost
above. After a deploy: an Inkling text request, an image request, and the same text request after a
task-model call (arbiter handoff, F4), identical at temperature 0.
---
## 4. Dead ends