From f3397cded01f1867117ec13cae4ab20dbfa7cc7d Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Sun, 22 Nov 2026 06:05:19 +0100 Subject: [PATCH] docs: FORK.md F14, user-message batch-break merge (not deployed) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ --- FORK.md | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/FORK.md b/FORK.md index 6c3fa3a3f..00f761d1c 100644 --- a/FORK.md +++ b/FORK.md @@ -55,6 +55,7 @@ Status at the time of writing (2026-11-22): - F11 Disk prompt cache (save slots on shutdown, restore after a restart) - F12 Temporary upstream carries (open upstream PRs, drop when merged) - F13 Per-pass views of host weights in looped layers (inkling2_loop) + - F14 User-message batch-break merge (one ubatch per short turn) 4. Dead ends 5. Configuration that matters 6. Measurement methodology and traps @@ -139,6 +140,7 @@ Oldest first. "Feature" points at the section below. | d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 | | 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 | | 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 | +| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (on `fable5/userstart-split`, not deployed) | F14 | Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code. Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it); @@ -1397,6 +1399,75 @@ task-model call (arbiter handoff, F4), identical at temperature 0. --- +### F14. User-message batch-break merge (one ubatch per short turn) + +**What and why.** Upstream ends a prompt batch right before a user message so that a context +checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420 +limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state +cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user +message is what a later request uses when it diverges inside that message (the user edits it, or the +client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's +template drops the previous answer's reasoning, which moves the divergence to just after +`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1 +restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation +prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each +ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128 +tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on +the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet), +which splits off the system prompt. Tool results count as user messages (Qwen opens them with +`<|im_start|>user`, the detected delimiter). + +**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop: +`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break. +Commit 510aa6e09. + +**How it works.** When the rest of the prompt from the current batch start fits in one ubatch +(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this +iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch +plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start +(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes +the restored checkpoint at the same position, with the current task id, so the next request still +has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that +diverges inside that message restores the batch-start checkpoint instead and re-processes the +previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts +whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the +whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the +upstream behaviour everywhere. + +**Invariants that must hold.** +- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a + divergence inside the last user message to less than `n_ubatch` tokens. +- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared + by `has_mtmd`); hence the `!has_mtmd` condition. +- It only changes where batches end, never which tokens are processed, so outputs differ from an + unmerged run only by batch-composition rounding. + +**Failure modes and incidents.** None yet; not validated on the pod, not deployed. + +**Rebase hazards.** +- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the + `checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch + start is still checkpointed when the batch ends at prompt end minus 4. + +**Validation plan (pod, not yet run).** +1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the + `created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch + `[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the + last user message. +2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded), + candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn + chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read + `prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens + (the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens. +3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit + the last user message and resend: merged re-processes previous answer + message (bounded by one + ubatch), unmerged only the message; no "forcing full prompt re-processing". +4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line): + restore after a restart, `prompt_n` and output identical to the reference built from the same + commit. + +--- + ## 4. Dead ends Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry