From b13c413555c292af3d998145e898fa233cfba6ab Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Sun, 22 Nov 2026 03:03:38 +0100 Subject: [PATCH] docs: FORK.md F13, per-pass views of host weights in inkling2_loop What and why (#32755 and the 6.5 GiB of shared expert copies), the knob, F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the pod validation plan (reference with the loop experts on the GPU, views on, views off). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ --- FORK.md | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) diff --git a/FORK.md b/FORK.md index 433830d46..4f5c11991 100644 --- a/FORK.md +++ b/FORK.md @@ -53,6 +53,7 @@ Status at the time of writing (2026-11-22): - F10 Page-cache drop after a copying load - F11 Disk prompt cache (save slots on shutdown, restore after a restart) - F12 Temporary upstream carries (open upstream PRs, drop when merged) + - F13 Per-pass views of host weights in looped layers (inkling2_loop) 4. Dead ends 5. Configuration that matters 6. Measurement methodology and traps @@ -136,6 +137,7 @@ Oldest first. "Feature" points at the section below. | 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 | | d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 | | 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 | +| 28ee3b735 | llama: inkling2_loop - per-pass views of host weights for the later loop passes (branch `fable5/inkling-loop-views`, not deployed) | F13 | Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code. Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it); @@ -1173,6 +1175,112 @@ about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rej (the graph then uses the dense mask), so it prints no CPU lines). Then an Inkling request through llama-swap. +### F13. Per-pass views of host weights in looped layers (inkling2_loop) + +Branch `fable5/inkling-loop-views` (on top of the F12 carries). Not built, run or deployed yet. + +**What and why.** `inkling2_loop` (Inkling-2-Mini) runs the core layers 6..17 three times; the 48 +cache slots are 24 physical layers plus 2 x 12 repeats. `post_arch_tensors` makes every repeat slot +share the tensors of its physical layer (`layers[il] = layers[phys]`). With the experts in host +memory (`--n-cpu-moe 24`, `-ot exps=CPU`), `ggml_backend_sched` makes one device copy per (host +tensor, backend) per graph, hashed by tensor pointer, at the first split that reads it, and keeps it +allocated until the last reader (`ggml_backend_sched_split_graph`, pass 5 and the graph_copy input +nodes). Two consequences: +- Memory: the expert copies of all 12 loop layers are alive at the same time (about 547 MiB per + layer on UD-Q4_K_XL, 6.5 GiB in total). That is most of the ~7 GiB Vulkan0 compute buffer in the + F12 measurements. +- Correctness (upstream issue #32755, "GGGG" output): the copy callback (`sched_copy_experts`, + F1) runs once, for the first reader. When it reads the routing ids back (ubatches below the F1 + skip threshold of `4 * n_expert` ids, and at or above the Vulkan MUL_MAT_ID offload batch of 32 + tokens: 32-63 tokens on Inkling-2-Mini), it uploads only the experts pass 0 selected. Passes 1 and + 2 reuse that copy and read stale VRAM for every other expert. Ubatches of 64+ tokens are correct + (F1 uploads every expert), ubatches below 32 run MUL_MAT_ID on the CPU (no copy). + +**Where.** `src/llama-model.h` (`llama_model::post_load_tensors`, a new optional hook), +`src/llama-model.cpp` (`llama_model_base::load_tensors` calls it on the no_alloc return and after +`load_tensors_hadamard()`), `src/models/models.h` and `src/models/inkling2-loop.cpp` +(`llama_model_inkling2_loop::post_load_tensors`, member `ctx_visits`). Commit 28ee3b735. + +**How it works.** After the weights are loaded, for every slot of pass 1 and 2 (`il >= n_phys`), each +tensor pointer in its `llama_layer` (walked as an array of pointers, as `llama_context::opt_init` +does) that sits in a host buffer (`ggml_backend_buffer_is_host`) is replaced with +`ggml_view_tensor` of it, named `.visit`, initialized with `ggml_backend_view_init` +(same buffer and data). On the no_alloc pass (memory fit) only `view->buffer` is set to the dummy +buffer, so the scheduler still sees a WEIGHTS buffer. The view is a leaf (op NONE, `view_src` set), +so the scheduler hashes it separately: each pass gets its own copy, freed after the last reader of +that pass, and the copy callback sees the MUL_MAT_ID of that pass and its ids (`st.ids` differs, so +F1 recomputes the expert set). The pointer-keyed weight maps of `llama_model` (`w_scales`, +`prec_policy.prec_acc`/`prec_src1`, `hdmd.rot`) get entries for the views. The views live in +`ctx_visits`, a context of the model outside `ctxs_bufs`. The startup log prints +`post_load_tensors: N views of host weights for the loop passes after the first (X MiB, not copied)`. + +Knob: `LLAMA_LOOP_HOST_VIEWS=0` keeps the shared tensors (the upstream behaviour), read once at load. +Default on. + +**Invariants that must hold.** +- Only host buffers get views. Device weights stay shared (used in place, no copy, so no problem), + and so do CPU extra buffers (CPU_REPACK is not a host buffer; its ops run on the CPU in place). +- F4: views point only at host buffers, which `release_device_weights` never releases, and they + are not in `ctxs_bufs`, so the shadows, `weight_release`, `memory_breakdown`, `n_tensors()` and + `tensors_by_name` do not see them. A device weight is released and restored in place, and the + loop slots keep pointing at the same tensor. +- F1: the skip/readback decision is per copy and keys on the ids tensor of the first node of the + split, so it stays per pass. +- `llama_layer` must hold only `ggml_tensor *` members (a `static_assert` checks the size only). + +**Cost.** Pass 1 and 2 now upload their own experts: per ubatch the host-to-device expert traffic +goes from 24 to 48 layer uploads (about +13 GiB per full ubatch on UD-Q4_K_XL). Prefill may lose +on the order of 10% (estimate, not measured); decode is unaffected (single-token MUL_MAT_ID runs on +the CPU). The freed ~6 GiB of VRAM can hold experts instead: an expert tensor of a loop layer kept on +the GPU saves 3 uploads per ubatch, a non-loop one saves 1, so loop layers are the first to move +to the GPU (`-ot` with the loop layers left out of the CPU pattern). Not measured. + +**Failure modes and known limits.** +- LoRA adapters look weights up by name (`llama_adapter_lora::get_weight`), so they do not apply to + passes 1 and 2 with views on. Not used here; set `LLAMA_LOOP_HOST_VIEWS=0` for a LoRA on Inkling. +- `--moe-cache-mib`: the cache binds expert tensors per slot (`llama_moe_cache_layer_experts` of + `model.layers[il]`), so with views each loop slot gets its own binding and slot map and caches the + same experts separately (before, the three slots collided on one binding key). Should not crash; + not tested. The MoE cache is off in production. +- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data + in a saved file cannot happen. + +**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for +every reader in the scheduler, or real per-pass tensors in the loader), revert 28ee3b735; the hook has +no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back. + +**Rebase hazards.** `post_arch_tensors` and the loop slot layout in `inkling2-loop.cpp`; +`llama_model_base::load_tensors` returns (a new return path must call the hook); new pointer-keyed +weight maps in `llama_model` (they need a view entry like `w_scales`); non-pointer members in +`llama_layer`; scheduler changes to how host weights are copied (`hv_tensor_copies`, +`ggml_backend_sched_is_host_weight`). + +**Validation (pod, GPU to itself in a swapdown window).** Build `llama-server` and `llama-bench` +from this branch in a scratch worktree (runbook section 5). Use the Inkling command line from +`inkling_args` in `/root/config.yaml` (with `--mmproj`) and change only the context to 16384 (so +arm A fits) and the placement: +- A, reference: replace `--n-cpu-moe 24` with `-ot 'blk\.([0-5]|1[89]|2[0-3])\.ffn_.*_exps=CPU'` + (loop experts on the GPU, no copies for the loop; the views log line must not appear). +- B, views on: `--n-cpu-moe 24`; the log line must show the views. +- C, views off: B with `LLAMA_LOOP_HOST_VIEWS=0`. + +For each arm send raw token prompts (`/tokenize` a long text, cut to N tokens, `/completion` with the +token array, `temperature 0, top_k 1, seed 42, cache_prompt false, n_predict 32, n_probs 5`) at +N = 40, 63, 1072, 6176, 7224 (tail 32-63 at ub1024), controls N = 100, 7368 (tail >= 64) and 20. +Expected: +- B == A in the first token's top-5 probabilities for every N >= 32 (prefill MUL_MAT_ID runs on the + GPU in both arms with the same data). The generated text may drift later, since decode MUL_MAT_ID + runs on the GPU in A and on the CPU in B. +- B == C byte-identical for the controls (tail >= 64 or < 32). +- C differs from A at the 32-63 tails (the bug); if it does not, the bug was not reproduced and + B's result proves nothing. +- Vulkan0 compute buffer (startup log): B about 6 GiB below C. A's model buffer carries the loop + experts instead. +Then `llama-bench -lm none -ngl 99 -ncmoe 24 -fa 1 -t 6 -b 1024 -ub 1024 -p 2048 -n 0 -r 3`, with and +without `LLAMA_LOOP_HOST_VIEWS=0`, interleaved B C C B, medians (section 6), for the prefill cost +above. After a deploy: an Inkling text request, an image request, and the same text request after a +task-model call (arbiter handoff, F4), identical at temperature 0. + --- ## 4. Dead ends