Compare commits

Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 2caae7e113 docs: FORK.md/runbook after the f3397cded deploy - F14 measured (35B TTFT A/B, batch layout, edit cost, disk cache), pointer 16
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:48:22 +01:00
LumpiastyandClaude Opus 5.5 f3397cded0 docs: FORK.md F14, user-message batch-break merge (not deployed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
LumpiastyandClaude Opus 5.5 510aa6e090 server: merge user-message batch breaks when the rest of the prompt fits in one ubatch
Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last
user message (and of earlier ones on a fresh prompt) so that a context
checkpoint lands there. On a hybrid/recurrent model with MoE experts on the
host, that turns a typical chat turn (previous answer + new user message, under
one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the
host experts again.

When the rest of the prompt from the current batch start fits in one ubatch,
skip the user-start break. The near-end rule still checkpoints the batch start
and the end-4 rule the prompt end, so a later prompt that diverges inside the
last user message re-processes less than one ubatch more, in the same ubatch.
Prompts longer than one ubatch, and batches after a media chunk, keep the
upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
LumpiastyandClaude Opus 5.5 166ec22e2b docs: FORK.md section 5, Inkling vs Qwen3.6-35B quality comparison
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:13:29 +01:00
LumpiastyandClaude Opus 5.5 e8386267ec docs: FORK.md/runbook after the b13c41355 deploy - F13 measured (KLD bug-window test, fit and placement guide, quants, profile), Inkling -ot k=4 without the offload knob, nothink fix, pointer 15
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 09:14:29 +01:00
LumpiastyandClaude Opus 5.5 b13c413555 docs: FORK.md F13, per-pass views of host weights in inkling2_loop
What and why (#32755 and the 6.5 GiB of shared expert copies), the knob,
F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the
pod validation plan (reference with the loop experts on the GPU, views on,
views off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:40:39 +01:00
LumpiastyandClaude Opus 5.5 312c29c2b1 llama: inkling2_loop - per-pass views of host weights for the later loop passes
inkling2_loop runs the core layers (6..17 on Inkling-2-Mini) three times, and
post_arch_tensors makes every cache slot share the tensors of its physical
layer. With the experts in host memory (--n-cpu-moe, -ot exps=CPU) the
scheduler makes one device copy per (host tensor, backend) per graph, at the
first split that reads it, and keeps it until the last reader. So:

- the expert copies of all 12 loop layers are alive at once (about 6.5 GiB on
  UD-Q4_K_XL), which is most of the ~7 GiB Vulkan0 compute buffer;
- sched_copy_experts runs only for the first reader. When it reads the ids back
  (ubatches of 32-63 tokens: MUL_MAT_ID is offloaded from 32 tokens, the F1
  skip starts at 4*n_expert ids), it uploads only the experts that pass 0
  selected, and passes 1 and 2 read stale VRAM for the others (upstream issue
  #32755, "GGGG" output).

Add a post_load_tensors hook to llama_model (called after the weights are
loaded, and on the no_alloc path after the dummy buffers are set). In
inkling2_loop it replaces, for each slot of a later pass, every tensor in a
host buffer with a view of it (name "<name>.visit<pass>", buffer and data of
the original). The scheduler hashes tensors by pointer, so each pass gets its
own copy, freed after that pass, and the copy callback sees the ids of that
pass. Weights in device buffers and CPU extra buffers (CPU_REPACK, not host)
stay shared. The pointer-keyed weight maps (w_scales, prec_policy, hdmd.rot)
get entries for the views. The views live in a model-owned context outside
ctxs_bufs, so the F4 release/restore, tensor counts and the memory breakdown
do not see them; they point only at host buffers, which F4 never releases.

LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors (A/B knob).

Not built or run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:40:39 +01:00
7 changed files with 546 additions and 35 deletions

No files matched your search

+388 -22
View File
@@ -6,13 +6,14 @@ which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-11-22):
- Production runs `3621025e8` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12), deployed
2026-11-22 03:52 with the Inkling retune (section 5). Validated with the full `fork-validate.sh`
set (phase 2 47 PASS, 0 FAIL; runbook section 6). Rollback pointer
`/root/DEPLOY-ROLLBACK-commit-14.txt` = `3bf6b55b3` (build copy `/root/arms/build-3bf6b55b3`,
config backup `/root/config.yaml.bak-20261122-inkling-ub1024-ctx65536`); next free pointer 15.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12680-3621025e8`) and changes
- Production runs `f3397cded` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, F13, the
per-pass loop views for Inkling, and F14, the user-message batch-break merge), deployed 2026-11-22
10:37 (F14 validated in its own window: 35B TTFT A/B, disk cache 15 PASS and the 35B legs; F14
measured). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-16.txt` = `b13c41355` (build copy
`/root/arms/build-b13c41355`; config unchanged by this deploy); next free pointer 17. The previous
pointer 15 = `3621025e8` needs its config backup `/root/config.yaml.bak-20261122-predeploy` too.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12687-f3397cded`) and changes
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
(section 3, "2026-11-21").
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
@@ -53,6 +54,8 @@ Status at the time of writing (2026-11-22):
- F10 Page-cache drop after a copying load
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
- F14 User-message batch-break merge (one ubatch per short turn)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -136,6 +139,8 @@ Oldest first. "Feature" points at the section below.
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1151,7 +1156,16 @@ production's Inkling command line with the nothink sampling, GPU to itself; driv
on every build, production `3bf6b55b3` with ub512 included, with and without
`GGML_OP_OFFLOAD_MIN_BATCH=64`, so the carry did not cause it. The think entry gets all three
right on both builds (image "Red", `get_weather({"city":"Warsaw"})`, 28.4K log "65" at 45.9 t/s
in production). Short text prompts work on the nothink entry. Open: fix the nothink macro.
in production). Short text prompts work on the nothink entry.
- Nothink fix (2026-11-22, config only): the cause was `--reasoning-budget 0`. With effort `none`
and no budget the model still writes a short reasoning block (100-550 characters) and then
answers; a budget of 0 cut that block off at once and the model looped on the chat-structure
tokens. The macro is now `--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --reasoning-effort none`.
Verified on production `3621025e8` (pod driver `/root/arms/overnight/nothink.py`, logs
`nothink1.log`, `nothink2.log` there): short question, Polish poem, image ("Red"), tool call
(`get_weather({"city":"Warsaw"})`) and a 29.4K-token Pan Tadeusz prompt (correct, 46.0 t/s) all
OK. Effort `minimal` and a budget of 256 also work; the budget of 0 fails on all but the short
question.
**Rebase hazards.**
- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry
@@ -1173,6 +1187,352 @@ about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rej
(the graph then uses the dense mask), so it prints no CPU lines).
Then an Inkling request through llama-swap.
### F13. Per-pass views of host weights in looped layers (inkling2_loop)
Branch `fable5/inkling-loop-views` (on top of the F12 carries), fast-forwarded into the production
branch on 2026-11-22 (deploy and results below).
**What and why.** `inkling2_loop` (Inkling-2-Mini) runs the core layers 6..17 three times; the 48
cache slots are 24 physical layers plus 2 x 12 repeats. `post_arch_tensors` makes every repeat slot
share the tensors of its physical layer (`layers[il] = layers[phys]`). With the experts in host
memory (`--n-cpu-moe 24`, `-ot exps=CPU`), `ggml_backend_sched` makes one device copy per (host
tensor, backend) per graph, hashed by tensor pointer, at the first split that reads it, and keeps it
allocated until the last reader (`ggml_backend_sched_split_graph`, pass 5 and the graph_copy input
nodes). Two consequences:
- Memory: the expert copies of all 12 loop layers are alive at the same time (about 547 MiB per
layer on UD-Q4_K_XL, 6.5 GiB in total). That is most of the ~7 GiB Vulkan0 compute buffer in the
F12 measurements.
- Correctness (upstream issue #32755, "GGGG" output): the copy callback (`sched_copy_experts`,
F1) runs once, for the first reader. When it reads the routing ids back (ubatches below the F1
skip threshold of `4 * n_expert` ids, and at or above the Vulkan MUL_MAT_ID offload batch of 32
tokens: 32-63 tokens on Inkling-2-Mini), it uploads only the experts pass 0 selected. Passes 1 and
2 reuse that copy and read stale VRAM for every other expert. Ubatches of 64+ tokens are correct
(F1 uploads every expert), ubatches below 32 run MUL_MAT_ID on the CPU (no copy).
**Where.** `src/llama-model.h` (`llama_model::post_load_tensors`, a new optional hook),
`src/llama-model.cpp` (`llama_model_base::load_tensors` calls it on the no_alloc return and after
`load_tensors_hadamard()`), `src/models/models.h` and `src/models/inkling2-loop.cpp`
(`llama_model_inkling2_loop::post_load_tensors`, member `ctx_visits`). Commit 312c29c2b.
**How it works.** After the weights are loaded, for every slot of pass 1 and 2 (`il >= n_phys`), each
tensor pointer in its `llama_layer` (walked as an array of pointers, as `llama_context::opt_init`
does) that sits in a host buffer (`ggml_backend_buffer_is_host`) is replaced with
`ggml_view_tensor` of it, named `<name>.visit<pass>`, initialized with `ggml_backend_view_init`
(same buffer and data). On the no_alloc pass (memory fit) only `view->buffer` is set to the dummy
buffer, so the scheduler still sees a WEIGHTS buffer. The view is a leaf (op NONE, `view_src` set),
so the scheduler hashes it separately: each pass gets its own copy, freed after the last reader of
that pass, and the copy callback sees the MUL_MAT_ID of that pass and its ids (`st.ids` differs, so
F1 recomputes the expert set). The pointer-keyed weight maps of `llama_model` (`w_scales`,
`prec_policy.prec_acc`/`prec_src1`, `hdmd.rot`) get entries for the views. The views live in
`ctx_visits`, a context of the model outside `ctxs_bufs`. The startup log prints
`post_load_tensors: N views of host weights for the loop passes after the first (X MiB, not copied)`.
Knob: `LLAMA_LOOP_HOST_VIEWS=0` keeps the shared tensors (the upstream behaviour), read once at load.
Default on.
**Invariants that must hold.**
- Only host buffers get views. Device weights stay shared (used in place, no copy, so no problem),
and so do CPU extra buffers (CPU_REPACK is not a host buffer; its ops run on the CPU in place).
- F4: views point only at host buffers, which `release_device_weights` never releases, and they
are not in `ctxs_bufs`, so the shadows, `weight_release`, `memory_breakdown`, `n_tensors()` and
`tensors_by_name` do not see them. A device weight is released and restored in place, and the
loop slots keep pointing at the same tensor.
- F1: the skip/readback decision is per copy and keys on the ids tensor of the first node of the
split, so it stays per pass.
- `llama_layer` must hold only `ggml_tensor *` members (a `static_assert` checks the size only).
**Cost.** Pass 1 and 2 now upload their own experts: per ubatch the host-to-device expert traffic
goes from 24 to 48 layer uploads (about +13 GiB per full ubatch on UD-Q4_K_XL). Measured where the
old layout still fits in VRAM (llama-bench pp2048 at depth 0, `-ncmoe 24`, B C C B in one window):
views 107.10 / 107.53 t/s, off 110.80 / 111.53, so -3.5%. Decode is unaffected (single-token
MUL_MAT_ID runs on the CPU). In the server at 16k-65k context the old layout does not fit (7 GiB of
compute buffer), spills into GTT, and the views win about 1.6x at prefill (Measured, below). The
freed ~6 GiB of VRAM can hold experts instead: an expert tensor of a loop layer kept on the GPU
saves 3 uploads per ubatch and 3 CPU expert passes per token, a non-loop one saves 1, so loop layers
are the first to move to the GPU (`-ot` with the loop layers left out of the CPU pattern; placement
guide below).
**Measured (2026-11-22, pod, GPU to itself, build `b13c41355` = 312c29c2b + docs; drivers and logs in
`/root/arms/overnight/`: `f13.py`, `cmp.py`, `kld48.sh`, `fit.py`, `sweep.sh`, `STATUS.md`).**
- Memory, server with the production command line at ctx 16384, `-lv 4`: Vulkan0 compute buffer
1426 MiB with views, 7030 MiB without. Process VRAM after load: views 4179 MiB, off 7727, A
(loop experts on the GPU, no views) 7879 (A wants 10089 MiB and spills).
- Prefill, server, 7224-token raw prompt at ctx 16384: views 92.4 t/s, off 57.0, A 46.5 (production
at ctx 65536 without views: about 46).
- Correctness. First-token top-5 logprobs (`f13.py`, `n_probs 5`, knob 32 so the 32-63 window is
live): with the tail at or above 64 or below 32 tokens (N = 20, 100, 7368) views on and off are
identical (max difference 0.000); in the window (N = 40, 63, 1072, 7224) they differ (0.04-0.16).
That test cannot say which arm is right: arm A differs from both by up to 0.15 even at the
controls (different placement, different kernels), and at 32-63 tokens x 8 experts pass 0's
union already covers most of the 128 experts, so the stale copy is mostly correct data. The
decisive test is `kld48.sh`: `llama-perplexity -c 48 -b 48 -ub 48`, 30 chunks of Pan Tadeusz, so
every chunk is one 48-token ubatch, KL divergence against a base with views on and knob 64 (the
tail MUL_MAT_ID on the CPU, correct by construction):
| arm | mean KLD | max KLD | same top token | RMS dp |
|---|---|---|---|---|
| views on, knob 32 (tail on the GPU) | 0.00314 | 0.087 | 97.4% | 1.35% |
| views off, knob 32 (tail on the GPU) | 0.03163 | 0.347 | 92.2% | 3.71% |
| views off, knob 64 | 0.00000 | - | 100% | 0.00% |
Views on is at the GPU-vs-CPU kernel noise level; views off is 10x worse (the #32755 mechanism,
reproduced). So with F13 the `GGML_OP_OFFLOAD_MIN_BATCH=64` workaround is no longer needed.
**Placement guide (UD-Q4_K_XL, views on).** k = number of loop layers (6, 7, ...) whose experts stay
on the GPU: `-ot 'blk[.](<the other 24-k layers>)[.]ffn_.*_exps=CPU'` instead of `--n-cpu-moe 24`
(`/root/arms/overnight/mkot.py` and `cfgedit.py` write the pattern; `[.]` avoids backslashes in the
YAML and in llama-swap's argument splitting). Each loop layer's experts take 547 MiB. Fit, from the
`-lv 4` breakdown of a server with the production command line (mmproj on demand, so not loaded):
| ctx | k | Vulkan0 self (model + context + compute) | free | verdict |
|---|---|---|---|---|
| 32768 | 0 | 4136 (1266 + 1347 + 1522) | 3982 | fits |
| 32768 | 6 | 7416 | 699 | fits |
| 32768 | 7 | 7963 | 15 | too tight |
| 65536 | 4 | 7246 (3453 + 2371 + 1421) | 868 | fits, chosen |
| 65536 | 5 | 7793 | 128 | too tight for the mmproj |
| 65536 | 6 | - | - | spills (GTT about 900 MiB above expectation) |
| 131072 | 0 | 7784 (1266 + 4419 + 2098) | 134 | fits (before F13 it overcommitted 5.2 GB) |
| 131072 | 2 | - | - | spills (about 1.2 GB) |
fdinfo `drm-memory-gtt` is not the spill test: it includes the pinned host weights
(Vulkan_Host). Compare GTT(k) with GTT(0) - k x 547 MiB, or read the breakdown.
Speed, llama-bench, one invocation per metric (`sweep.sh`, `-fa 1 -t 6 -b 1024 -ub 1024`, knob 64,
pp r1, tg r2):
| k | pp2048 @ d8192 | pp2048 @ d16384 | tg256 @ d4096 |
|---|---|---|---|
| 0 | 97.53 | 89.62 | 5.57 |
| 4 | 102.19 | 93.90 | 6.45 |
| 6 | 104.43 | 95.27 | 6.72 |
| 7 | 105.90 | 96.90 | 7.25 |
Decode gains most (+16% at k=4, +30% at k=7): every loop layer on the GPU removes 3 of the 48 CPU
expert passes per token. Prefill gains only 5-9%: at depth the attention and the remaining uploads
dominate.
**Quants (2026-11-22, not adopted).** Unsloth's Mainline_Support UD-Q3_K_XL (13.6 GB) and UD-IQ3_XXS
(11.7 GB) are in `/root/models/inkling/`. Quality: `llama-perplexity -c 2048 --chunks 4`, KL
divergence against UD-Q4_K_XL logits (`kld.sh`), on a fixed Pan Tadeusz chunk
(`/root/bench/entext/pt-chunk.txt`, characters 20000-100000) and wikitext-2 test:
| text | quant | PPL (Q4_K_XL) | PPL ratio | mean KLD | same top token |
|---|---|---|---|---|---|
| Pan Tadeusz | Q3_K_XL | 11.45 (11.30) | +1.4% | 0.0241 | 92.5% |
| Pan Tadeusz | IQ3_XXS | 11.52 (11.30) | +1.9% | 0.0421 | 90.1% |
| wikitext-2 | Q3_K_XL | 6.251 (6.230) | +0.3% | 0.0097 | 95.6% |
| wikitext-2 | IQ3_XXS | 6.307 (6.230) | +1.2% | 0.0214 | 94.2% |
Fit at ctx 65536: UD-Q3_K_XL keeps the loop-layer experts at 547 MiB each (like Q4), so k=5 (867
MiB free); UD-IQ3_XXS has 390 MiB per layer, k=7 (952 MiB free). Speed (separate invocations, so
within the ~7% cross-run noise for prefill): Q3_K_XL k=4 105.23 / 6.83, k=6 107.31 / 7.48;
IQ3_XXS k=4 105.59 / 6.44, k=7 108.48 / 7.86 (pp2048 @ d8192 / tg256 @ d4096; Q4_K_XL k=4
102.19 / 6.45). The best case, IQ3_XXS at k=7, is +6% prefill and +22% decode (about 7.9 instead
of 6.5 t/s) for a Polish KLD of 0.042; IQ3_XXS's CPU dot product is slower per byte (at k=4 it
decodes no faster than Q4). Not worth the quality loss for the owner's Polish use.
**Production (deployed 2026-11-22, `b13c41355`, ctx 65536, k=4, no knob).** Through llama-swap
(`/root/arms/overnight/post.py`, log `post.log`): Pan Tadeusz 9312 tokens 81.6 t/s prefill, +7958
tokens 89.3 t/s, +17255 tokens (28K total) 78.0 t/s, decode 5.9-6.0 t/s, answers correct (before:
about 46 t/s prefill). Think and nothink entries answer text, an image ("Red") and a tool call. A
tiny request after the 4B task model held the GPU takes 1.3-2.4 s longer than a warm one (the
arbiter restores 3.4 GiB of device weights). Disk cache save/restore works on Inkling (1329 tokens,
410 MiB). The first request on a fresh process is slow (162 tokens 11.7 t/s prefill, 2.6 t/s decode)
since the pipelines are compiled on first use (`--no-warmup`).
**Profile (2026-11-22, `perf record` + `gpu_busy_percent`, `prof.sh`, results in
`/root/arms/overnight/prof/`).** Inkling k=4 prefill (pp2048 @ d4096, 107 t/s): GPU busy 77%, CPU
time mostly `ggml_vk_wait_for_fence` (43%) and the kernel (11%, the uploads): GPU-bound. Decode
(7.3 t/s): GPU busy 58%, CPU in the OpenMP barrier spin (libgomp 36%) and
`ggml_vec_dot_nvfp4_q8_0` (21%): the UD-Q4_K_XL experts are NVFP4, and that CPU dot product is the
decode hotspot. Qwen3.6-35B for comparison: prefill 263 t/s at GPU busy 52%, decode 13.8 t/s at
35%, `vec_dot_q4_K_q8_K` 10%. `GGML_VK_PERF_LOGGER=1` aborts at the second split
(`GGML_ASSERT(ctx->compute_ctx.expired())`, ggml-vulkan.cpp:15692), so it gives per-op GPU times
only for the first split; not investigated.
**Failure modes and known limits.**
- LoRA adapters look weights up by name (`llama_adapter_lora::get_weight`), so they do not apply to
passes 1 and 2 with views on. Not used here; set `LLAMA_LOOP_HOST_VIEWS=0` for a LoRA on Inkling.
- `--moe-cache-mib`: the cache binds expert tensors per slot (`llama_moe_cache_layer_experts` of
`model.layers[il]`), so with views each loop slot gets its own binding and slot map and caches the
same experts separately (before, the three slots collided on one binding key). Should not crash;
not tested. The MoE cache is off in production.
- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data
in a saved file cannot happen.
**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for
every reader in the scheduler, or real per-pass tensors in the loader), revert 312c29c2b; the hook has
no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
**Rebase hazards.** `post_arch_tensors` and the loop slot layout in `inkling2-loop.cpp`;
`llama_model_base::load_tensors` returns (a new return path must call the hook); new pointer-keyed
weight maps in `llama_model` (they need a view entry like `w_scales`); non-pointer members in
`llama_layer`; scheduler changes to how host weights are copied (`hv_tensor_copies`,
`ggml_backend_sched_is_host_weight`).
**Validation (pod, GPU to itself in a swapdown window).** Build `llama-server` and `llama-bench`
from this branch in a scratch worktree (runbook section 5). Use the Inkling command line from
`inkling_args` in `/root/config.yaml` (with `--mmproj`) and change only the context to 16384 (so
arm A fits) and the placement:
- A, reference: replace `--n-cpu-moe 24` with `-ot 'blk\.([0-5]|1[89]|2[0-3])\.ffn_.*_exps=CPU'`
(loop experts on the GPU, no copies for the loop; the views log line must not appear).
- B, views on: `--n-cpu-moe 24`; the log line must show the views.
- C, views off: B with `LLAMA_LOOP_HOST_VIEWS=0`.
For each arm send raw token prompts (`/tokenize` a long text, cut to N tokens, `/completion` with the
token array, `temperature 0, top_k 1, seed 42, cache_prompt false, n_predict 32, n_probs 5`) at
N = 40, 63, 1072, 6176, 7224 (tail 32-63 at ub1024), controls N = 100, 7368 (tail >= 64) and 20.
Expected:
- B == A in the first token's top-5 probabilities for every N >= 32 (prefill MUL_MAT_ID runs on the
GPU in both arms with the same data). The generated text may drift later, since decode MUL_MAT_ID
runs on the GPU in A and on the CPU in B.
- B == C byte-identical for the controls (tail >= 64 or < 32).
- C differs from A at the 32-63 tails (the bug); if it does not, the bug was not reproduced and
B's result proves nothing.
- Vulkan0 compute buffer (startup log): B about 6 GiB below C. A's model buffer carries the loop
experts instead.
Then `llama-bench -lm none -ngl 99 -ncmoe 24 -fa 1 -t 6 -b 1024 -ub 1024 -p 2048 -n 0 -r 3`, with and
without `LLAMA_LOOP_HOST_VIEWS=0`, interleaved B C C B, medians (section 6), for the prefill cost
above. After a deploy: an Inkling text request, an image request, and the same text request after a
task-model call (arbiter handoff, F4), identical at temperature 0.
---
### F14. User-message batch-break merge (one ubatch per short turn)
Branch `fable5/userstart-split` (rebased onto the F13 production tip `166ec22e2`), fast-forwarded into
the production branch on 2026-11-22 (deploy and results below). Knob: `LLAMA_USERSTART_MERGE=0`.
**What and why.** Upstream ends a prompt batch right before a user message so that a context
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
message is what a later request uses when it diverges inside that message (the user edits it, or the
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
template drops the previous answer's reasoning, which moves the divergence to just after
`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation
prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each
ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
`<|im_start|>user`, the detected delimiter).
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
Commit 510aa6e09.
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
the restored checkpoint at the same position, with the current task id, so the next request still
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
diverges inside that message restores the batch-start checkpoint instead and re-processes the
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
upstream behaviour everywhere.
**Invariants that must hold.**
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
divergence inside the last user message to less than `n_ubatch` tokens.
- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared
by `has_mtmd`); hence the `!has_mtmd` condition.
- It only changes where batches end, never which tokens are processed, so outputs differ from an
unmerged run only by batch-composition rounding.
**Measured (2026-11-22, pod, GPU to itself in one swapdown window, build `f3397cded`; driver
`/root/arms/usw_ab.py`, window script `/root/arms/usw-window.sh`, logs and `res.json` per arm in
`/root/arms/usw-ab/{M1,U1,U2,M2}/`).** Production 35B think entry (`q36t`: `${ctx_256k}
${qwen35_think_args} ${moe_offload_args}`, `-hf` pulls the mmproj, supervisord's environment, scratch
arena and `--cache-disk-path`, plus `-lv 4`), one fresh server per arm, arms M (merged, default) and U
(`LLAMA_USERSTART_MERGE=0`) in the order M U U M. Each chat: system prompt, 6 user turns from
`pt-chunk.txt` (150, 10, 40, 150, 600, 40 tokens), synthetic previous answers of 300, 800, 1200, 500,
1300 tokens, `max_tokens 8`, `temperature 0`, streamed; then the last user message edited in the
middle and resent. TTFT in seconds (mean of the two runs per arm; the single runs in brackets),
`prompt_n` was identical in all four runs on every turn:
| turn | user tokens | prompt_n | M TTFT | U TTFT | M - U |
|---|---|---|---|---|---|
| 1 (fresh prompt) | 150 | 196 | 3.99 (4.43, 3.55) | 6.08 (5.73, 6.42) | -2.1 |
| 2 | 10 | 327 | 4.11 (3.75, 4.47) | 7.66 (6.75, 8.57) | -3.6 |
| 3 | 40 | 869 | 6.34 (6.62, 6.06) | 9.34 (7.26, 11.42) | -3.0 |
| 4 | 150 | 1381 | 7.20 (7.30, 7.10) | 10.95 (12.31, 9.58) | -3.7 |
| 5 | 600 | 1130 | 8.82 (7.47, 10.17) | 9.80 (9.52, 10.07) | -1.0 |
| 6 | 40 | 1370 | 8.11 (8.01, 8.20) | 9.53 (9.33, 9.72) | -1.4 |
| edit of turn 6 | 40 | M 1374, U 66 | 7.94 (8.16, 7.71) | 3.65 (4.09, 3.20) | +4.3 |
- Batch layout (`prompt processing, n_tokens` and checkpoint lines at `-lv 4`): M runs every turn as
one batch `[restore point, end-4)` plus the tail, with checkpoints at the batch start (superseding
the restored one) and at end minus 4; U splits at the last user message (an extra checkpoint
there, for example 1323 between 515 and 1380 on turn 3) and on turn 1 also at the first user
message (system prompt, 23 tokens). Turn N+1 restores the end-minus-4 checkpoint of turn N in both
arms. No `forcing full prompt re-processing` in any run.
- The 10-token turn gained too, against the plan's "no change". U split it into 308 tokens (the
previous answer, on the GPU) and 19 (message, generation prompt and tail), and the 19-token part runs
the experts of all 40 layers on the CPU, which costs seconds rather than nothing at 6 threads
(inferred from the totals, not profiled). The 600-token turn gained
least (its two parts are 508 and 618 tokens, both on the GPU, so the split costs one extra
upload, about 1-2 s against noise of the same size).
- Edit of the last user message (the cost of the trade): M has no checkpoint at the user-message
start, restores the batch start and re-processes previous answer + message (1374 tokens, one
ubatch); U restores the user-start checkpoint and re-processes 66 tokens. +4.3 s per such edit.
Ordinary follow-up turns are far more frequent than edits of the last message in this use; a client
that injects per-request context into the last user message (RAG, web search) pays the same
as an edit on every turn, so set `LLAMA_USERSTART_MERGE=0` for such a client.
- Outputs: identical within each arm across runs; between arms turns 5 and 6 differ after a few
tokens (batch-composition rounding, as the invariant says).
- Disk cache: `fork-validate.sh --diskcache` (in the window, GPU leg included) 15 PASS, 0 FAIL, 0
SKIP. `diskcache-window3.sh` legs with production's 35B command line (`q35.py` copied to
`q35-usw.py` with the candidate build path; log `/root/arms/usw-q35-mm.log`): 18.8k-token chat,
turn 1 83.5 s, save 557 MB in 1.0 s, restart restore and turn 2 `prompt_n` 27 in 3.7 s, output ==
reference; cold save and restore identity (`prompt_n` 4, output == reference); image answered
'Red' and not saved. The saved state after turn 2 holds 4 checkpoints instead of 5 (the user-start
one is gone).
**Production (deployed 2026-11-22 10:37, `f3397cded`, config unchanged).** Deploy script
`/root/arms/dep-usw.sh` (log `dep-usw.log`): incremental in-place build, downtime 33 s. Post-deploy
checks through llama-swap (`/root/arms/post-usw.py`, `ink-usw.py`, `ink2-usw.py`, logs beside them):
`system_fingerprint` `b12687-f3397cded`; 35B 'Paris', then 4B / 35B handoffs twice (correct answers,
`holder` the answering PID each time, 2.9-6.3 s per switch); a 1140-token 35B chat, then a gemma-26B
image ('Red', the 35B exited by itself with `loop stopped, device memory freed` and `disk cache:
saved the prompt of 1157 tokens`), gemma text, 4B text, then the 35B chat plus one user message:
`disk cache: restored 1157 tokens`, `prompt_n` 30 (the suffix only). Inkling nothink: text correct
(it still spends about 40 tokens on reasoning, so a 96-token budget returns an empty content), image
'Red', a 9293-token Pan Tadeusz prompt at 82.0 t/s with the right answer (F13 measured 81.6). Children
bind llama-swap's ports. Log scan clean, `dmesg` clean (no ring timeout, GPU hang or OOM kill), pod
restarts 0 as before.
**Failure modes and incidents.** None in validation or the deploy. Known cost: an edit of the last
user message (or per-request context injected into it) re-processes the previous answer too, +4.3 s
measured above.
**Rebase hazards.**
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
`checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch
start is still checkpointed when the batch ends at prompt end minus 4.
**Validation plan (run 2026-11-22, results under Measured above).**
1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the
`created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch
`[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the
last user message.
2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded),
candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn
chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read
`prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens
(the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens.
3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit
the last user message and resend: merged re-processes previous answer + message (bounded by one
ubatch), unmerged only the message; no "forcing full prompt re-processing".
4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line):
restore after a restart, `prompt_n` and output identical to the reference built from the same
commit.
---
## 4. Dead ends
@@ -1261,16 +1621,23 @@ Other models:
heads, head size 128; macro `inkling_args`, entries `unsloth/Inkling-2-Mini-GGUF[-nothink]:UD-Q4_K_XL`,
matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m`
(the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it,
`--n-cpu-moe 24` (all experts on host), `-t 6`, f16 KV, `--parallel 1`, the disk prompt cache.
Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 65536`
(before: `-b 512 -ub 512`; the dense per-head bias mask needed about 12.7 GB of host compute
buffer at 128k). 131072 loads and works with the carry, but overcommits about 5.2 GB of device
memory into GTT and is about 8% slower at prefill (F12). Both Inkling entries set
`env: ["GGML_OP_OFFLOAD_MIN_BATCH=64"]` (llama-swap per-model env): ubatches of 32-63 tokens keep
MUL_MAT_ID on the CPU. The looped layers share their expert tensors, and below F1's readback-skip
threshold (4 x 128 experts / 8 used = 64 tokens) only loop pass 0's routed experts would be
uploaded (the mechanism suspected behind upstream #32755). Its nothink macro uses
`--reasoning-effort none --reasoning-budget 0`.
`-t 6`, f16 KV, `--parallel 1`, the disk prompt cache, `-b 1024 -ub 1024 --ctx-size 65536`
(since the 2026-11-22 carry of upstream #32740, F12; before: `-b 512 -ub 512`). Since the F13
deploy (2026-11-22, `b13c41355`): the experts of loop layers 6-9 stay on the GPU and all others
on the host, `-ot blk[.](0|1|2|3|4|5|10|11|12|13|14|15|16|17|18|19|20|21|22|23)[.]ffn_.*_exps=CPU`
instead of `--n-cpu-moe 24` (k=4 of the F13 placement guide: 7246 MiB of device memory, 868 MiB
left for the mmproj). The `GGML_OP_OFFLOAD_MIN_BATCH=64` env of both entries was removed: F13
makes the 32-63-token ubatches correct (KLD in F13). Other choices from the guide: ctx 32768 with
k=6, or ctx 131072 with k=0 (fits since F13, 134 MiB free, so images would spill). Smaller quants
(UD-Q3_K_XL, UD-IQ3_XXS, `/root/models/inkling/`) were measured and not adopted (F13, Quants).
Its nothink macro uses `--reasoning-effort none` (no `--reasoning-budget 0`: that budget made it
loop on chat-structure tokens, F12); even with effort none it writes a short reasoning block.
Against Qwen3.6-35B (think entries, 7 tasks, one sample each, 3500-token cap, 2026-11-22,
`/root/arms/overnight/quality.py`, `quality/quality.json`): Inkling 6/7, Qwen 5/7. Inkling found
all three bugs of a binary search where Qwen ran out of budget; both ran out on a 4-line Polish
rhyme. Inkling used about a third fewer tokens (9677 vs 14254) but took 1.7x the wall time (2283
vs 1316 s; decode 4-6 vs 13 t/s) and 4x on a 29K-token question (835 vs 211 s). Too small a
sample to call it smarter; it is clearly slower here.
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
`/root/supervisord.conf`:
@@ -1345,10 +1712,9 @@ Machine-level:
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
null buffer would crash. Only the mmproj path handles a failed restore.
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
- Inkling (F12, section 5): the nothink entry returns repeated chat-structure tokens instead of an
answer for image requests, tool calls and long prompts (the think entry is fine); the config runs
ctx 65536 because 131072 overcommits about 5.2 GB of device memory into GTT; upstream #32755 /
#32778 did not reproduce here. Drop the F12 carries when upstream merges #32735 and #32740.
- Inkling (F12, section 5): the nothink entry's chat-structure loop came from
`--reasoning-budget 0` and was fixed in the config on 2026-11-22 (F12). Upstream #32755 / #32778
did not reproduce here; F13 bounds the loop-pass expert copies (more in F13). Drop the F12 carries when upstream merges #32735 and #32740.
- Open after the 2026-11-21 rebase: is the fork's concat fast path (F3) now redundant with upstream's
own tiled concat (#30591, gated in the fork with `ne0 < 32` and the `GGML_VK_CONCAT_FASTPATH` knob)?
Benchmark it. Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no
+32 -12
View File
@@ -348,14 +348,28 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- Last full run (2026-11-22, candidate `3621025e8` = `3bf6b55b3` + the F12 carries, vs reference
- F14 run (2026-11-22, candidate `f3397cded` = `166ec22e2` (docs on `b13c41355`) + F14, server-only change; one
swapdown window, `/root/arms/usw-window.sh`, log `/root/arms/usw-window.log`, about 15 minutes):
the 35B TTFT A/B with and without `LLAMA_USERSTART_MERGE=0` (FORK.md F14, M U U M, production 35B
think command line with the mmproj), `--diskcache` 15 PASS, 0 FAIL, 0 SKIP, and the
`diskcache-window3.sh` 35B legs (all passed). The full `--perf`/`--arbiter` set was not rerun: the
change touches only the server prompt-batch loop. Deploy downtime 33 s (incremental in-place
build).
- Last full run (2026-11-22, candidate `b13c41355` = `3621025e8` + F13 vs reference `3621025e8`, one
swapdown window, `--ops --concat --diskcache --arbiter --perf --moe-exact --moecache`, log
`/root/arms/overnight/fv.log`): 52 PASS, 0 FAIL, 0 SKIP in 41 minutes. Perf medians ref -> cand:
pp2048 286.61 -> 284.94 (-0.6%), d16384 207.93 -> 207.96, d32768 161.93 -> 162.12, tg256@d4096
14.11 -> 14.47; handoff median ref 715 / cand 706.5 ms; peak VRAM ref 5478 / cand 7771 MiB (the
first-arm spike of FORK.md F9, this time in a candidate arm). With the card free, phase 1 and 2
can run as one invocation like this.
- Previous full run (2026-11-22, candidate `3621025e8` = `3bf6b55b3` + the F12 carries, vs reference
`3bf6b55b3`; logs `/root/arms/carry1122-p1.log`, `-p2.log`): phase 1 `--ops --concat --diskcache`
20 PASS, 1 FAIL (`concat/candidate-timing` beside a full card, see above), 1 SKIP (diskcache GPU
leg); phase 2 `--concat --arbiter --diskcache --perf --moe-exact --moecache` 47 PASS, 0 FAIL, 0
SKIP. Perf medians ref -> cand: pp2048 286.97 -> 286.32, d16384 208.31 -> 208.21, d32768 162.25 ->
162.37; handoff median ref 696 / cand 700 ms; peak VRAM ref 7771 / cand 5502 MiB. Plus SCALE
15/15 and `test-flash-attn-indexed-bias` (FORK.md F12). Window about 30 minutes.
- Previous full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
- Earlier full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
9 PASS plus the 35B disk-cache legs (`q35.py`, mmproj loaded). Perf medians ref -> cand: pp2048
259.39 -> 286.61 (+10.5%), d16384 199.26 -> 207.63 (+4.2%), d32768 156.61 -> 157.51 (+0.6%),
tg256@d4096 9.17 -> 8.85 t/s (noisy, not gated). Handoff median: candidate 692 ms, reference
@@ -404,8 +418,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 14 (holds 3bf6b55b3)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-15.txt
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 16 (holds b13c41355)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-17.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3):
@@ -544,11 +558,17 @@ on `reasoning_content` and returns an empty `content`; for one, raise `max_token
whether any falls in the deploy window (2026-10-25: the newest was about 19 h old).
8. Inkling (FORK.md section 5, F12): judge the answer text, not only HTTP 200. The first request
after a fresh load should be a prompt of about 150 tokens (upstream #32778 gives all-'?' output
there); then a long prompt (Pan Tadeusz, 8K and 14K tokens, upstream #32755). Image and tool
requests go to the think entry: the nothink entry returns repeated
`<|message_model|><|content_thinking|><|end_message|>` for them on every build (FORK.md F12).
Check that the env of the running process has `GGML_OP_OFFLOAD_MIN_BATCH=64`
(`tr '\0' '\n' < /proc/<pid>/environ`). The memory breakdown table is printed only at `-lv 4`,
there); then a long prompt (Pan Tadeusz, 8K and 14K tokens, upstream #32755). Send an image
and a tool request to both entries: since the nothink macro dropped `--reasoning-budget 0`
(FORK.md F12) both answer them; repeated `<|message_model|><|content_thinking|><|end_message|>`
from the nothink entry means a budget of 0 is back. Since F13 a scratch server with the
production command line plus `-lv 4` must log `post_load_tensors: N views of host weights`
(production's own log level hides libllama's info lines, so the llama-swap log never shows it),
and the entries no longer set
`GGML_OP_OFFLOAD_MIN_BATCH=64` (F13 measured the 32-63-token window correct without it). The
placement is `-ot` with the loop layers 6..6+k-1 left on the GPU (FORK.md F13 placement guide);
after a change of context or quant, check the fit again (the breakdown must leave room for the
mmproj, about 0.8 GiB). The memory breakdown table is printed only at `-lv 4`,
which production does not set, so it is not in the llama-swap log: read it from a scratch server
started with production's command line plus `-lv 4`. Drivers used on 2026-11-22 (pod only):
`/root/arms/carry1122-post.py` (through llama-swap) and `/root/arms/carry1122-ink-direct.py`
@@ -600,10 +620,10 @@ chain in the exec shell: it can die with the exec session halfway, leaving llama
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
Pointer files 2 to 13 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
Pointer files 2 to 16 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, commit-12 holds
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, and commit-14 holds `3bf6b55b3`, the rollback target
for the current production `3621025e8` (next free: 15). The formats vary
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, commit-14 holds `3bf6b55b3`, commit-15 holds `3621025e8`, and commit-16 holds `b13c41355`, the rollback target
for the current production `f3397cded` (next free: 17; build copy `/root/arms/build-b13c41355`). The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
+3
View File
@@ -2253,6 +2253,7 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
}
if (ml.no_alloc) {
post_load_tensors(ml);
return true;
}
@@ -2283,6 +2284,8 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
load_tensors_hadamard();
post_load_tensors(ml);
return true;
}
+3
View File
@@ -1091,6 +1091,9 @@ struct llama_model {
// optional, runs after the scale pass
virtual void post_arch_tensors(llama_model_loader &) {}
// optional, runs after the weight buffers are allocated and loaded (with no_alloc: after the dummy buffers are set)
virtual void post_load_tensors(llama_model_loader &) {}
protected:
llama_model_params params;
+102
View File
@@ -1,5 +1,12 @@
#include "models.h"
#include <algorithm>
#include <cstdlib>
#include <cstring>
#include <stdexcept>
#include <unordered_map>
#include <vector>
int llama_model_inkling2_loop::layer_phys(int il) const {
return il < n_phys ? il : core_start + (il - n_phys) % (core_end - core_start);
}
@@ -69,6 +76,101 @@ void llama_model_inkling2_loop::post_arch_tensors(llama_model_loader &) {
}
}
// the scheduler makes one device copy of a host weight per graph, alive from its first to its last use, and the expert copy
// callback runs at the first use only: give each later pass its own view, so it gets its own copy with the experts of its own ids
// knob: LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors
void llama_model_inkling2_loop::post_load_tensors(llama_model_loader & ml) {
const char * env = getenv("LLAMA_LOOP_HOST_VIEWS");
if (env != nullptr && strcmp(env, "0") == 0) {
LLAMA_LOG_INFO("%s: LLAMA_LOOP_HOST_VIEWS=0, the loop passes share the host weights\n", __func__);
return;
}
// llama_layer holds only tensor pointers, see llama_context::opt_init
static_assert(sizeof(llama_layer) % sizeof(ggml_tensor *) == 0, "llama_layer must hold only tensor pointers");
const size_t n_ptr = sizeof(llama_layer)/sizeof(ggml_tensor *);
const int n_slot = (int) hparams.n_layer();
const int n_core = (int) (core_end - core_start);
// weights on a device are used in place, only host weights are copied
std::vector<std::vector<ggml_tensor *>> host(n_slot);
size_t n_views = 0;
for (int il = n_phys; il < n_slot; ++il) {
ggml_tensor ** ts = reinterpret_cast<ggml_tensor **>(&layers[il]);
for (size_t i = 0; i < n_ptr; ++i) {
ggml_tensor * t = ts[i];
if (t == nullptr || t->buffer == nullptr || !ggml_backend_buffer_is_host(t->buffer) || (!ml.no_alloc && t->data == nullptr)) {
continue;
}
if (std::find(host[il].begin(), host[il].end(), t) == host[il].end()) {
host[il].push_back(t);
}
}
n_views += host[il].size();
}
if (n_views == 0) {
return;
}
ggml_init_params init_params = {
/*.mem_size =*/ n_views*ggml_tensor_overhead(),
/*.mem_buffer =*/ nullptr,
/*.no_alloc =*/ true,
};
ctx_visits.reset(ggml_init(init_params));
if (!ctx_visits) {
throw std::runtime_error("inkling2_loop: failed to create the context of the loop views");
}
size_t n_bytes = 0;
for (int il = n_phys; il < n_slot; ++il) {
const int pass = 1 + (il - n_phys)/n_core;
std::unordered_map<const ggml_tensor *, ggml_tensor *> view_of;
for (ggml_tensor * t : host[il]) {
ggml_tensor * v = ggml_view_tensor(ctx_visits.get(), t);
ggml_format_name(v, "%s.visit%d", t->name, pass);
if (ml.no_alloc) {
// the dummy buffer has no data, it only tells the scheduler that this is a weight
v->buffer = t->buffer;
} else if (ggml_backend_view_init(v) != GGML_STATUS_SUCCESS) {
throw std::runtime_error(format("inkling2_loop: failed to initialize the view %s", v->name));
}
view_of[t] = v;
n_bytes += ggml_nbytes(t);
// the per-weight maps are keyed by the tensor
if (const auto it = w_scales.find(t); it != w_scales.end()) {
ggml_tensor * s = it->second;
w_scales.emplace(v, s);
}
if (const auto it = prec_policy.prec_acc.find(t); it != prec_policy.prec_acc.end()) {
const ggml_prec prec = it->second;
prec_policy.prec_acc.emplace(v, prec);
}
if (const auto it = prec_policy.prec_src1.find(t); it != prec_policy.prec_src1.end()) {
const ggml_prec prec = it->second;
prec_policy.prec_src1.emplace(v, prec);
}
if (const auto it = hdmd.rot.find(t); it != hdmd.rot.end()) {
const llama_hadamard_transform tr = it->second;
hdmd.rot.emplace(v, tr);
}
}
ggml_tensor ** ts = reinterpret_cast<ggml_tensor **>(&layers[il]);
for (size_t i = 0; i < n_ptr; ++i) {
if (const auto it = view_of.find(ts[i]); it != view_of.end()) {
ts[i] = it->second;
}
}
}
LLAMA_LOG_INFO("%s: %zu views of host weights for the loop passes after the first (%.2f MiB, not copied)\n",
__func__, n_views, n_bytes/1024.0/1024.0);
}
std::unique_ptr<llm_graph_context> llama_model_inkling2_loop::build_arch_graph(const llm_graph_params & params) const {
return std::make_unique<graph>(params);
}
+4
View File
@@ -2559,6 +2559,7 @@ struct llama_model_inkling2_loop : public llama_model_inkling2 {
void load_arch_hparams(llama_model_loader & ml) override;
void load_arch_tensors(llama_model_loader & ml) override;
void post_arch_tensors(llama_model_loader & ml) override;
void post_load_tensors(llama_model_loader & ml) override;
int n_layer_phys() const override { return n_phys; }
int layer_phys(int il) const;
@@ -2573,6 +2574,9 @@ struct llama_model_inkling2_loop : public llama_model_inkling2 {
// indexed by physical layer - core_start
std::vector<loop_layer> loop_layers;
// views of the host weights for the cache slots of the later passes, see post_load_tensors
ggml_context_ptr ctx_visits;
struct graph : public llama_model_inkling2::graph {
graph(const llm_graph_params & params);
};
+14 -1
View File
@@ -4942,6 +4942,19 @@ private:
const int32_t n_decision_first = slot.task->type == SERVER_TASK_TYPE_DECISION ? slot.task->decision.pos_first() : -1;
// [fork F14] do not break at user-message starts when the rest of the prompt
// from here fits in one ubatch: the break turns one ubatch into two, and with
// MoE experts on the host each one of >= 32 tokens uploads the experts again.
// A checkpoint at this batch start (the near-end rule below) and at the end
// minus 4 still exist, so a later prompt that diverges inside the merged range
// re-processes less than one ubatch more. LLAMA_USERSTART_MERGE=0 disables.
static const bool userstart_merge = [] {
const char * env = getenv("LLAMA_USERSTART_MERGE");
return env == nullptr || std::string(env) != "0";
}();
const bool merge_user_starts = userstart_merge && !has_mtmd &&
slot.task->n_tokens() - slot.prompt.n_tokens() <= n_ubatch;
// add prompt tokens for processing in the current batch
while (slot.prompt.n_tokens() < slot.task->n_tokens() && batch.size() < n_batch) {
// get next token to process
@@ -4985,7 +4998,7 @@ private:
slot.prompt.tokens.push_back(cur_tok);
// break at the last user message, or at user messages at least min step past the last checkpoint
if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) {
if (do_checkpoint && !merge_user_starts && spans.is_user_start(slot.prompt.n_tokens())) {
const auto pos = slot.prompt.n_tokens();
const auto & checkpoints = slot.prompt.checkpoints;