Compare commits

Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 2caae7e113 docs: FORK.md/runbook after the f3397cded deploy - F14 measured (35B TTFT A/B, batch layout, edit cost, disk cache), pointer 16
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:48:22 +01:00
LumpiastyandClaude Opus 5.5 f3397cded0 docs: FORK.md F14, user-message batch-break merge (not deployed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
LumpiastyandClaude Opus 5.5 510aa6e090 server: merge user-message batch breaks when the rest of the prompt fits in one ubatch
Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last
user message (and of earlier ones on a fresh prompt) so that a context
checkpoint lands there. On a hybrid/recurrent model with MoE experts on the
host, that turns a typical chat turn (previous answer + new user message, under
one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the
host experts again.

When the rest of the prompt from the current batch start fits in one ubatch,
skip the user-start break. The near-end rule still checkpoints the batch start
and the end-4 rule the prompt end, so a later prompt that diverges inside the
last user message re-processes less than one ubatch more, in the same ubatch.
Prompts longer than one ubatch, and batches after a media chunk, keep the
upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
3 changed files with 172 additions and 14 deletions

No files matched your search

+146 -8
View File
@@ -6,14 +6,14 @@ which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-11-22):
- Production runs `b13c41355` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, and F13, the
per-pass loop views for Inkling), deployed 2026-11-22 08:47 with the Inkling placement change
(section 5, F13). Validated with the full `fork-validate.sh` set in one window (52 PASS, 0 FAIL,
0 SKIP; runbook section 6). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-15.txt` = `3621025e8`
(build copy `/root/arms/build-3621025e8`, config backup `/root/config.yaml.bak-20261122-predeploy`;
restore both, the old build has no F13); next free pointer 16.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12683-b13c41355`) and changes
- Production runs `f3397cded` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, F13, the
per-pass loop views for Inkling, and F14, the user-message batch-break merge), deployed 2026-11-22
10:37 (F14 validated in its own window: 35B TTFT A/B, disk cache 15 PASS and the 35B legs; F14
measured). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-16.txt` = `b13c41355` (build copy
`/root/arms/build-b13c41355`; config unchanged by this deploy); next free pointer 17. The previous
pointer 15 = `3621025e8` needs its config backup `/root/config.yaml.bak-20261122-predeploy` too.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12687-f3397cded`) and changes
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
(section 3, "2026-11-21").
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
@@ -55,6 +55,7 @@ Status at the time of writing (2026-11-22):
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
- F14 User-message batch-break merge (one ubatch per short turn)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -139,6 +140,7 @@ Oldest first. "Feature" points at the section below.
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1397,6 +1399,142 @@ task-model call (arbiter handoff, F4), identical at temperature 0.
---
### F14. User-message batch-break merge (one ubatch per short turn)
Branch `fable5/userstart-split` (rebased onto the F13 production tip `166ec22e2`), fast-forwarded into
the production branch on 2026-11-22 (deploy and results below). Knob: `LLAMA_USERSTART_MERGE=0`.
**What and why.** Upstream ends a prompt batch right before a user message so that a context
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
message is what a later request uses when it diverges inside that message (the user edits it, or the
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
template drops the previous answer's reasoning, which moves the divergence to just after
`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation
prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each
ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
`<|im_start|>user`, the detected delimiter).
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
Commit 510aa6e09.
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
the restored checkpoint at the same position, with the current task id, so the next request still
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
diverges inside that message restores the batch-start checkpoint instead and re-processes the
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
upstream behaviour everywhere.
**Invariants that must hold.**
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
divergence inside the last user message to less than `n_ubatch` tokens.
- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared
by `has_mtmd`); hence the `!has_mtmd` condition.
- It only changes where batches end, never which tokens are processed, so outputs differ from an
unmerged run only by batch-composition rounding.
**Measured (2026-11-22, pod, GPU to itself in one swapdown window, build `f3397cded`; driver
`/root/arms/usw_ab.py`, window script `/root/arms/usw-window.sh`, logs and `res.json` per arm in
`/root/arms/usw-ab/{M1,U1,U2,M2}/`).** Production 35B think entry (`q36t`: `${ctx_256k}
${qwen35_think_args} ${moe_offload_args}`, `-hf` pulls the mmproj, supervisord's environment, scratch
arena and `--cache-disk-path`, plus `-lv 4`), one fresh server per arm, arms M (merged, default) and U
(`LLAMA_USERSTART_MERGE=0`) in the order M U U M. Each chat: system prompt, 6 user turns from
`pt-chunk.txt` (150, 10, 40, 150, 600, 40 tokens), synthetic previous answers of 300, 800, 1200, 500,
1300 tokens, `max_tokens 8`, `temperature 0`, streamed; then the last user message edited in the
middle and resent. TTFT in seconds (mean of the two runs per arm; the single runs in brackets),
`prompt_n` was identical in all four runs on every turn:
| turn | user tokens | prompt_n | M TTFT | U TTFT | M - U |
|---|---|---|---|---|---|
| 1 (fresh prompt) | 150 | 196 | 3.99 (4.43, 3.55) | 6.08 (5.73, 6.42) | -2.1 |
| 2 | 10 | 327 | 4.11 (3.75, 4.47) | 7.66 (6.75, 8.57) | -3.6 |
| 3 | 40 | 869 | 6.34 (6.62, 6.06) | 9.34 (7.26, 11.42) | -3.0 |
| 4 | 150 | 1381 | 7.20 (7.30, 7.10) | 10.95 (12.31, 9.58) | -3.7 |
| 5 | 600 | 1130 | 8.82 (7.47, 10.17) | 9.80 (9.52, 10.07) | -1.0 |
| 6 | 40 | 1370 | 8.11 (8.01, 8.20) | 9.53 (9.33, 9.72) | -1.4 |
| edit of turn 6 | 40 | M 1374, U 66 | 7.94 (8.16, 7.71) | 3.65 (4.09, 3.20) | +4.3 |
- Batch layout (`prompt processing, n_tokens` and checkpoint lines at `-lv 4`): M runs every turn as
one batch `[restore point, end-4)` plus the tail, with checkpoints at the batch start (superseding
the restored one) and at end minus 4; U splits at the last user message (an extra checkpoint
there, for example 1323 between 515 and 1380 on turn 3) and on turn 1 also at the first user
message (system prompt, 23 tokens). Turn N+1 restores the end-minus-4 checkpoint of turn N in both
arms. No `forcing full prompt re-processing` in any run.
- The 10-token turn gained too, against the plan's "no change". U split it into 308 tokens (the
previous answer, on the GPU) and 19 (message, generation prompt and tail), and the 19-token part runs
the experts of all 40 layers on the CPU, which costs seconds rather than nothing at 6 threads
(inferred from the totals, not profiled). The 600-token turn gained
least (its two parts are 508 and 618 tokens, both on the GPU, so the split costs one extra
upload, about 1-2 s against noise of the same size).
- Edit of the last user message (the cost of the trade): M has no checkpoint at the user-message
start, restores the batch start and re-processes previous answer + message (1374 tokens, one
ubatch); U restores the user-start checkpoint and re-processes 66 tokens. +4.3 s per such edit.
Ordinary follow-up turns are far more frequent than edits of the last message in this use; a client
that injects per-request context into the last user message (RAG, web search) pays the same
as an edit on every turn, so set `LLAMA_USERSTART_MERGE=0` for such a client.
- Outputs: identical within each arm across runs; between arms turns 5 and 6 differ after a few
tokens (batch-composition rounding, as the invariant says).
- Disk cache: `fork-validate.sh --diskcache` (in the window, GPU leg included) 15 PASS, 0 FAIL, 0
SKIP. `diskcache-window3.sh` legs with production's 35B command line (`q35.py` copied to
`q35-usw.py` with the candidate build path; log `/root/arms/usw-q35-mm.log`): 18.8k-token chat,
turn 1 83.5 s, save 557 MB in 1.0 s, restart restore and turn 2 `prompt_n` 27 in 3.7 s, output ==
reference; cold save and restore identity (`prompt_n` 4, output == reference); image answered
'Red' and not saved. The saved state after turn 2 holds 4 checkpoints instead of 5 (the user-start
one is gone).
**Production (deployed 2026-11-22 10:37, `f3397cded`, config unchanged).** Deploy script
`/root/arms/dep-usw.sh` (log `dep-usw.log`): incremental in-place build, downtime 33 s. Post-deploy
checks through llama-swap (`/root/arms/post-usw.py`, `ink-usw.py`, `ink2-usw.py`, logs beside them):
`system_fingerprint` `b12687-f3397cded`; 35B 'Paris', then 4B / 35B handoffs twice (correct answers,
`holder` the answering PID each time, 2.9-6.3 s per switch); a 1140-token 35B chat, then a gemma-26B
image ('Red', the 35B exited by itself with `loop stopped, device memory freed` and `disk cache:
saved the prompt of 1157 tokens`), gemma text, 4B text, then the 35B chat plus one user message:
`disk cache: restored 1157 tokens`, `prompt_n` 30 (the suffix only). Inkling nothink: text correct
(it still spends about 40 tokens on reasoning, so a 96-token budget returns an empty content), image
'Red', a 9293-token Pan Tadeusz prompt at 82.0 t/s with the right answer (F13 measured 81.6). Children
bind llama-swap's ports. Log scan clean, `dmesg` clean (no ring timeout, GPU hang or OOM kill), pod
restarts 0 as before.
**Failure modes and incidents.** None in validation or the deploy. Known cost: an edit of the last
user message (or per-request context injected into it) re-processes the previous answer too, +4.3 s
measured above.
**Rebase hazards.**
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
`checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch
start is still checkpointed when the batch ends at prompt end minus 4.
**Validation plan (run 2026-11-22, results under Measured above).**
1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the
`created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch
`[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the
last user message.
2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded),
candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn
chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read
`prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens
(the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens.
3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit
the last user message and resend: merged re-processes previous answer + message (bounded by one
ubatch), unmerged only the message; no "forcing full prompt re-processing".
4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line):
restore after a restart, `prompt_n` and output identical to the reference built from the same
commit.
---
## 4. Dead ends
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
+12 -5
View File
@@ -348,6 +348,13 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- F14 run (2026-11-22, candidate `f3397cded` = `166ec22e2` (docs on `b13c41355`) + F14, server-only change; one
swapdown window, `/root/arms/usw-window.sh`, log `/root/arms/usw-window.log`, about 15 minutes):
the 35B TTFT A/B with and without `LLAMA_USERSTART_MERGE=0` (FORK.md F14, M U U M, production 35B
think command line with the mmproj), `--diskcache` 15 PASS, 0 FAIL, 0 SKIP, and the
`diskcache-window3.sh` 35B legs (all passed). The full `--perf`/`--arbiter` set was not rerun: the
change touches only the server prompt-batch loop. Deploy downtime 33 s (incremental in-place
build).
- Last full run (2026-11-22, candidate `b13c41355` = `3621025e8` + F13 vs reference `3621025e8`, one
swapdown window, `--ops --concat --diskcache --arbiter --perf --moe-exact --moecache`, log
`/root/arms/overnight/fv.log`): 52 PASS, 0 FAIL, 0 SKIP in 41 minutes. Perf medians ref -> cand:
@@ -411,8 +418,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 15 (holds 3621025e8)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-16.txt
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 16 (holds b13c41355)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-17.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3):
@@ -613,10 +620,10 @@ chain in the exec shell: it can die with the exec session halfway, leaving llama
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
Pointer files 2 to 15 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
Pointer files 2 to 16 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, commit-12 holds
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, commit-14 holds `3bf6b55b3`, and commit-15 holds `3621025e8`, the rollback target
for the current production `b13c41355` (next free: 16). The formats vary
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, commit-14 holds `3bf6b55b3`, commit-15 holds `3621025e8`, and commit-16 holds `b13c41355`, the rollback target
for the current production `f3397cded` (next free: 17; build copy `/root/arms/build-b13c41355`). The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
+14 -1
View File
@@ -4942,6 +4942,19 @@ private:
const int32_t n_decision_first = slot.task->type == SERVER_TASK_TYPE_DECISION ? slot.task->decision.pos_first() : -1;
// [fork F14] do not break at user-message starts when the rest of the prompt
// from here fits in one ubatch: the break turns one ubatch into two, and with
// MoE experts on the host each one of >= 32 tokens uploads the experts again.
// A checkpoint at this batch start (the near-end rule below) and at the end
// minus 4 still exist, so a later prompt that diverges inside the merged range
// re-processes less than one ubatch more. LLAMA_USERSTART_MERGE=0 disables.
static const bool userstart_merge = [] {
const char * env = getenv("LLAMA_USERSTART_MERGE");
return env == nullptr || std::string(env) != "0";
}();
const bool merge_user_starts = userstart_merge && !has_mtmd &&
slot.task->n_tokens() - slot.prompt.n_tokens() <= n_ubatch;
// add prompt tokens for processing in the current batch
while (slot.prompt.n_tokens() < slot.task->n_tokens() && batch.size() < n_batch) {
// get next token to process
@@ -4985,7 +4998,7 @@ private:
slot.prompt.tokens.push_back(cur_tok);
// break at the last user message, or at user messages at least min step past the last checkpoint
if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) {
if (do_checkpoint && !merge_user_starts && spans.is_user_start(slot.prompt.n_tokens())) {
const auto pos = slot.prompt.n_tokens();
const auto & checkpoints = slot.prompt.checkpoints;