docs: FORK.md F14, user-message batch-break merge (not deployed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
1 parent
510aa6e090
commit
f3397cded0
1 file changed
+71
@@ -55,6 +55,7 @@ Status at the time of writing (2026-11-22):
|
||||
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
||||
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
||||
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
||||
- F14 User-message batch-break merge (one ubatch per short turn)
|
||||
4. Dead ends
|
||||
5. Configuration that matters
|
||||
6. Measurement methodology and traps
|
||||
@@ -139,6 +140,7 @@ Oldest first. "Feature" points at the section below.
|
||||
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
|
||||
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
||||
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
||||
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (on `fable5/userstart-split`, not deployed) | F14 |
|
||||
|
||||
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
||||
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
||||
@@ -1397,6 +1399,75 @@ task-model call (arbiter handoff, F4), identical at temperature 0.
|
||||
|
||||
---
|
||||
|
||||
### F14. User-message batch-break merge (one ubatch per short turn)
|
||||
|
||||
**What and why.** Upstream ends a prompt batch right before a user message so that a context
|
||||
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
|
||||
limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state
|
||||
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
|
||||
message is what a later request uses when it diverges inside that message (the user edits it, or the
|
||||
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
|
||||
template drops the previous answer's reasoning, which moves the divergence to just after
|
||||
`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
|
||||
restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation
|
||||
prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each
|
||||
ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128
|
||||
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
|
||||
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
|
||||
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
|
||||
`<|im_start|>user`, the detected delimiter).
|
||||
|
||||
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
|
||||
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
|
||||
Commit 510aa6e09.
|
||||
|
||||
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
|
||||
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
|
||||
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
|
||||
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
|
||||
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
|
||||
the restored checkpoint at the same position, with the current task id, so the next request still
|
||||
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
|
||||
diverges inside that message restores the batch-start checkpoint instead and re-processes the
|
||||
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
|
||||
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
|
||||
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
|
||||
upstream behaviour everywhere.
|
||||
|
||||
**Invariants that must hold.**
|
||||
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
|
||||
divergence inside the last user message to less than `n_ubatch` tokens.
|
||||
- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared
|
||||
by `has_mtmd`); hence the `!has_mtmd` condition.
|
||||
- It only changes where batches end, never which tokens are processed, so outputs differ from an
|
||||
unmerged run only by batch-composition rounding.
|
||||
|
||||
**Failure modes and incidents.** None yet; not validated on the pod, not deployed.
|
||||
|
||||
**Rebase hazards.**
|
||||
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
|
||||
`checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch
|
||||
start is still checkpointed when the batch ends at prompt end minus 4.
|
||||
|
||||
**Validation plan (pod, not yet run).**
|
||||
1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the
|
||||
`created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch
|
||||
`[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the
|
||||
last user message.
|
||||
2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded),
|
||||
candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn
|
||||
chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read
|
||||
`prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens
|
||||
(the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens.
|
||||
3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit
|
||||
the last user message and resend: merged re-processes previous answer + message (bounded by one
|
||||
ubatch), unmerged only the message; no "forcing full prompt re-processing".
|
||||
4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line):
|
||||
restore after a restart, `prompt_n` and output identical to the reference built from the same
|
||||
commit.
|
||||
|
||||
---
|
||||
|
||||
## 4. Dead ends
|
||||
|
||||
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
|
||||
|
||||
Reference in new issue
Block a user