docs: FORK.md F13, per-pass views of host weights in inkling2_loop
What and why (#32755 and the 6.5 GiB of shared expert copies), the knob, F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the pod validation plan (reference with the loop experts on the GPU, views on, views off). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
1 parent
312c29c2b1
commit
b13c413555
1 file changed
+108
@@ -53,6 +53,7 @@ Status at the time of writing (2026-11-22):
|
||||
- F10 Page-cache drop after a copying load
|
||||
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
||||
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
||||
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
||||
4. Dead ends
|
||||
5. Configuration that matters
|
||||
6. Measurement methodology and traps
|
||||
@@ -136,6 +137,7 @@ Oldest first. "Feature" points at the section below.
|
||||
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
|
||||
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
|
||||
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
||||
| 28ee3b735 | llama: inkling2_loop - per-pass views of host weights for the later loop passes (branch `fable5/inkling-loop-views`, not deployed) | F13 |
|
||||
|
||||
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
||||
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
||||
@@ -1173,6 +1175,112 @@ about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rej
|
||||
(the graph then uses the dense mask), so it prints no CPU lines).
|
||||
Then an Inkling request through llama-swap.
|
||||
|
||||
### F13. Per-pass views of host weights in looped layers (inkling2_loop)
|
||||
|
||||
Branch `fable5/inkling-loop-views` (on top of the F12 carries). Not built, run or deployed yet.
|
||||
|
||||
**What and why.** `inkling2_loop` (Inkling-2-Mini) runs the core layers 6..17 three times; the 48
|
||||
cache slots are 24 physical layers plus 2 x 12 repeats. `post_arch_tensors` makes every repeat slot
|
||||
share the tensors of its physical layer (`layers[il] = layers[phys]`). With the experts in host
|
||||
memory (`--n-cpu-moe 24`, `-ot exps=CPU`), `ggml_backend_sched` makes one device copy per (host
|
||||
tensor, backend) per graph, hashed by tensor pointer, at the first split that reads it, and keeps it
|
||||
allocated until the last reader (`ggml_backend_sched_split_graph`, pass 5 and the graph_copy input
|
||||
nodes). Two consequences:
|
||||
- Memory: the expert copies of all 12 loop layers are alive at the same time (about 547 MiB per
|
||||
layer on UD-Q4_K_XL, 6.5 GiB in total). That is most of the ~7 GiB Vulkan0 compute buffer in the
|
||||
F12 measurements.
|
||||
- Correctness (upstream issue #32755, "GGGG" output): the copy callback (`sched_copy_experts`,
|
||||
F1) runs once, for the first reader. When it reads the routing ids back (ubatches below the F1
|
||||
skip threshold of `4 * n_expert` ids, and at or above the Vulkan MUL_MAT_ID offload batch of 32
|
||||
tokens: 32-63 tokens on Inkling-2-Mini), it uploads only the experts pass 0 selected. Passes 1 and
|
||||
2 reuse that copy and read stale VRAM for every other expert. Ubatches of 64+ tokens are correct
|
||||
(F1 uploads every expert), ubatches below 32 run MUL_MAT_ID on the CPU (no copy).
|
||||
|
||||
**Where.** `src/llama-model.h` (`llama_model::post_load_tensors`, a new optional hook),
|
||||
`src/llama-model.cpp` (`llama_model_base::load_tensors` calls it on the no_alloc return and after
|
||||
`load_tensors_hadamard()`), `src/models/models.h` and `src/models/inkling2-loop.cpp`
|
||||
(`llama_model_inkling2_loop::post_load_tensors`, member `ctx_visits`). Commit 28ee3b735.
|
||||
|
||||
**How it works.** After the weights are loaded, for every slot of pass 1 and 2 (`il >= n_phys`), each
|
||||
tensor pointer in its `llama_layer` (walked as an array of pointers, as `llama_context::opt_init`
|
||||
does) that sits in a host buffer (`ggml_backend_buffer_is_host`) is replaced with
|
||||
`ggml_view_tensor` of it, named `<name>.visit<pass>`, initialized with `ggml_backend_view_init`
|
||||
(same buffer and data). On the no_alloc pass (memory fit) only `view->buffer` is set to the dummy
|
||||
buffer, so the scheduler still sees a WEIGHTS buffer. The view is a leaf (op NONE, `view_src` set),
|
||||
so the scheduler hashes it separately: each pass gets its own copy, freed after the last reader of
|
||||
that pass, and the copy callback sees the MUL_MAT_ID of that pass and its ids (`st.ids` differs, so
|
||||
F1 recomputes the expert set). The pointer-keyed weight maps of `llama_model` (`w_scales`,
|
||||
`prec_policy.prec_acc`/`prec_src1`, `hdmd.rot`) get entries for the views. The views live in
|
||||
`ctx_visits`, a context of the model outside `ctxs_bufs`. The startup log prints
|
||||
`post_load_tensors: N views of host weights for the loop passes after the first (X MiB, not copied)`.
|
||||
|
||||
Knob: `LLAMA_LOOP_HOST_VIEWS=0` keeps the shared tensors (the upstream behaviour), read once at load.
|
||||
Default on.
|
||||
|
||||
**Invariants that must hold.**
|
||||
- Only host buffers get views. Device weights stay shared (used in place, no copy, so no problem),
|
||||
and so do CPU extra buffers (CPU_REPACK is not a host buffer; its ops run on the CPU in place).
|
||||
- F4: views point only at host buffers, which `release_device_weights` never releases, and they
|
||||
are not in `ctxs_bufs`, so the shadows, `weight_release`, `memory_breakdown`, `n_tensors()` and
|
||||
`tensors_by_name` do not see them. A device weight is released and restored in place, and the
|
||||
loop slots keep pointing at the same tensor.
|
||||
- F1: the skip/readback decision is per copy and keys on the ids tensor of the first node of the
|
||||
split, so it stays per pass.
|
||||
- `llama_layer` must hold only `ggml_tensor *` members (a `static_assert` checks the size only).
|
||||
|
||||
**Cost.** Pass 1 and 2 now upload their own experts: per ubatch the host-to-device expert traffic
|
||||
goes from 24 to 48 layer uploads (about +13 GiB per full ubatch on UD-Q4_K_XL). Prefill may lose
|
||||
on the order of 10% (estimate, not measured); decode is unaffected (single-token MUL_MAT_ID runs on
|
||||
the CPU). The freed ~6 GiB of VRAM can hold experts instead: an expert tensor of a loop layer kept on
|
||||
the GPU saves 3 uploads per ubatch, a non-loop one saves 1, so loop layers are the first to move
|
||||
to the GPU (`-ot` with the loop layers left out of the CPU pattern). Not measured.
|
||||
|
||||
**Failure modes and known limits.**
|
||||
- LoRA adapters look weights up by name (`llama_adapter_lora::get_weight`), so they do not apply to
|
||||
passes 1 and 2 with views on. Not used here; set `LLAMA_LOOP_HOST_VIEWS=0` for a LoRA on Inkling.
|
||||
- `--moe-cache-mib`: the cache binds expert tensors per slot (`llama_moe_cache_layer_experts` of
|
||||
`model.layers[il]`), so with views each loop slot gets its own binding and slot map and caches the
|
||||
same experts separately (before, the three slots collided on one binding key). Should not crash;
|
||||
not tested. The MoE cache is off in production.
|
||||
- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data
|
||||
in a saved file cannot happen.
|
||||
|
||||
**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for
|
||||
every reader in the scheduler, or real per-pass tensors in the loader), revert 28ee3b735; the hook has
|
||||
no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
|
||||
|
||||
**Rebase hazards.** `post_arch_tensors` and the loop slot layout in `inkling2-loop.cpp`;
|
||||
`llama_model_base::load_tensors` returns (a new return path must call the hook); new pointer-keyed
|
||||
weight maps in `llama_model` (they need a view entry like `w_scales`); non-pointer members in
|
||||
`llama_layer`; scheduler changes to how host weights are copied (`hv_tensor_copies`,
|
||||
`ggml_backend_sched_is_host_weight`).
|
||||
|
||||
**Validation (pod, GPU to itself in a swapdown window).** Build `llama-server` and `llama-bench`
|
||||
from this branch in a scratch worktree (runbook section 5). Use the Inkling command line from
|
||||
`inkling_args` in `/root/config.yaml` (with `--mmproj`) and change only the context to 16384 (so
|
||||
arm A fits) and the placement:
|
||||
- A, reference: replace `--n-cpu-moe 24` with `-ot 'blk\.([0-5]|1[89]|2[0-3])\.ffn_.*_exps=CPU'`
|
||||
(loop experts on the GPU, no copies for the loop; the views log line must not appear).
|
||||
- B, views on: `--n-cpu-moe 24`; the log line must show the views.
|
||||
- C, views off: B with `LLAMA_LOOP_HOST_VIEWS=0`.
|
||||
|
||||
For each arm send raw token prompts (`/tokenize` a long text, cut to N tokens, `/completion` with the
|
||||
token array, `temperature 0, top_k 1, seed 42, cache_prompt false, n_predict 32, n_probs 5`) at
|
||||
N = 40, 63, 1072, 6176, 7224 (tail 32-63 at ub1024), controls N = 100, 7368 (tail >= 64) and 20.
|
||||
Expected:
|
||||
- B == A in the first token's top-5 probabilities for every N >= 32 (prefill MUL_MAT_ID runs on the
|
||||
GPU in both arms with the same data). The generated text may drift later, since decode MUL_MAT_ID
|
||||
runs on the GPU in A and on the CPU in B.
|
||||
- B == C byte-identical for the controls (tail >= 64 or < 32).
|
||||
- C differs from A at the 32-63 tails (the bug); if it does not, the bug was not reproduced and
|
||||
B's result proves nothing.
|
||||
- Vulkan0 compute buffer (startup log): B about 6 GiB below C. A's model buffer carries the loop
|
||||
experts instead.
|
||||
Then `llama-bench -lm none -ngl 99 -ncmoe 24 -fa 1 -t 6 -b 1024 -ub 1024 -p 2048 -n 0 -r 3`, with and
|
||||
without `LLAMA_LOOP_HOST_VIEWS=0`, interleaved B C C B, medians (section 6), for the prefill cost
|
||||
above. After a deploy: an Inkling text request, an image request, and the same text request after a
|
||||
task-model call (arbiter handoff, F4), identical at temperature 0.
|
||||
|
||||
---
|
||||
|
||||
## 4. Dead ends
|
||||
|
||||
Reference in new issue
Block a user