docs: FORK.md F14, user-message batch-break merge (not deployed)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-11-22 10:15:30 +01:00
1 parent 510aa6e090
commit f3397cded0
1 file changed
+71
+71
View File
@@ -55,6 +55,7 @@ Status at the time of writing (2026-11-22):
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
- F14 User-message batch-break merge (one ubatch per short turn)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -139,6 +140,7 @@ Oldest first. "Feature" points at the section below.
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (on `fable5/userstart-split`, not deployed) | F14 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1397,6 +1399,75 @@ task-model call (arbiter handoff, F4), identical at temperature 0.
---
### F14. User-message batch-break merge (one ubatch per short turn)
**What and why.** Upstream ends a prompt batch right before a user message so that a context
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
message is what a later request uses when it diverges inside that message (the user edits it, or the
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
template drops the previous answer's reasoning, which moves the divergence to just after
`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation
prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each
ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
`<|im_start|>user`, the detected delimiter).
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
Commit 510aa6e09.
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
the restored checkpoint at the same position, with the current task id, so the next request still
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
diverges inside that message restores the batch-start checkpoint instead and re-processes the
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
upstream behaviour everywhere.
**Invariants that must hold.**
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
divergence inside the last user message to less than `n_ubatch` tokens.
- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared
by `has_mtmd`); hence the `!has_mtmd` condition.
- It only changes where batches end, never which tokens are processed, so outputs differ from an
unmerged run only by batch-composition rounding.
**Failure modes and incidents.** None yet; not validated on the pod, not deployed.
**Rebase hazards.**
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
`checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch
start is still checkpointed when the batch ends at prompt end minus 4.
**Validation plan (pod, not yet run).**
1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the
`created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch
`[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the
last user message.
2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded),
candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn
chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read
`prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens
(the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens.
3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit
the last user message and resend: merged re-processes previous answer + message (bounded by one
ubatch), unmerged only the message; no "forcing full prompt re-processing".
4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line):
restore after a restart, `prompt_n` and output identical to the reference built from the same
commit.
---
## 4. Dead ends
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry