|
|
@@ -6,14 +6,14 @@ which also holds the operations reference (pod access, files, recovery).
|
|
|
|
|
|
|
|
|
|
|
|
Status at the time of writing (2026-11-22):
|
|
|
|
Status at the time of writing (2026-11-22):
|
|
|
|
|
|
|
|
|
|
|
|
- Production runs `b13c41355` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
|
|
|
- Production runs `f3397cded` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
|
|
|
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, and F13, the
|
|
|
|
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, F13, the
|
|
|
|
per-pass loop views for Inkling), deployed 2026-11-22 08:47 with the Inkling placement change
|
|
|
|
per-pass loop views for Inkling, and F14, the user-message batch-break merge), deployed 2026-11-22
|
|
|
|
(section 5, F13). Validated with the full `fork-validate.sh` set in one window (52 PASS, 0 FAIL,
|
|
|
|
10:37 (F14 validated in its own window: 35B TTFT A/B, disk cache 15 PASS and the 35B legs; F14
|
|
|
|
0 SKIP; runbook section 6). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-15.txt` = `3621025e8`
|
|
|
|
measured). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-16.txt` = `b13c41355` (build copy
|
|
|
|
(build copy `/root/arms/build-3621025e8`, config backup `/root/config.yaml.bak-20261122-predeploy`;
|
|
|
|
`/root/arms/build-b13c41355`; config unchanged by this deploy); next free pointer 17. The previous
|
|
|
|
restore both, the old build has no F13); next free pointer 16.
|
|
|
|
pointer 15 = `3621025e8` needs its config backup `/root/config.yaml.bak-20261122-predeploy` too.
|
|
|
|
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12683-b13c41355`) and changes
|
|
|
|
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12687-f3397cded`) and changes
|
|
|
|
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
|
|
|
|
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
|
|
|
|
(section 3, "2026-11-21").
|
|
|
|
(section 3, "2026-11-21").
|
|
|
|
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
|
|
|
|
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
|
|
|
@@ -55,6 +55,7 @@ Status at the time of writing (2026-11-22):
|
|
|
|
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
|
|
|
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
|
|
|
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
|
|
|
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
|
|
|
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
|
|
|
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
|
|
|
|
|
|
|
- F14 User-message batch-break merge (one ubatch per short turn)
|
|
|
|
4. Dead ends
|
|
|
|
4. Dead ends
|
|
|
|
5. Configuration that matters
|
|
|
|
5. Configuration that matters
|
|
|
|
6. Measurement methodology and traps
|
|
|
|
6. Measurement methodology and traps
|
|
|
@@ -139,6 +140,7 @@ Oldest first. "Feature" points at the section below.
|
|
|
|
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
|
|
|
|
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
|
|
|
|
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
|
|
|
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
|
|
|
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
|
|
|
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
|
|
|
|
|
|
|
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
|
|
|
|
|
|
|
|
|
|
|
|
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
|
|
|
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
|
|
|
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
|
|
|
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
|
|
@@ -1397,6 +1399,142 @@ task-model call (arbiter handoff, F4), identical at temperature 0.
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### F14. User-message batch-break merge (one ubatch per short turn)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Branch `fable5/userstart-split` (rebased onto the F13 production tip `166ec22e2`), fast-forwarded into
|
|
|
|
|
|
|
|
the production branch on 2026-11-22 (deploy and results below). Knob: `LLAMA_USERSTART_MERGE=0`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**What and why.** Upstream ends a prompt batch right before a user message so that a context
|
|
|
|
|
|
|
|
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
|
|
|
|
|
|
|
|
limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state
|
|
|
|
|
|
|
|
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
|
|
|
|
|
|
|
|
message is what a later request uses when it diverges inside that message (the user edits it, or the
|
|
|
|
|
|
|
|
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
|
|
|
|
|
|
|
|
template drops the previous answer's reasoning, which moves the divergence to just after
|
|
|
|
|
|
|
|
`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
|
|
|
|
|
|
|
|
restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation
|
|
|
|
|
|
|
|
prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each
|
|
|
|
|
|
|
|
ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128
|
|
|
|
|
|
|
|
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
|
|
|
|
|
|
|
|
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
|
|
|
|
|
|
|
|
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
|
|
|
|
|
|
|
|
`<|im_start|>user`, the detected delimiter).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
|
|
|
|
|
|
|
|
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
|
|
|
|
|
|
|
|
Commit 510aa6e09.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
|
|
|
|
|
|
|
|
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
|
|
|
|
|
|
|
|
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
|
|
|
|
|
|
|
|
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
|
|
|
|
|
|
|
|
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
|
|
|
|
|
|
|
|
the restored checkpoint at the same position, with the current task id, so the next request still
|
|
|
|
|
|
|
|
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
|
|
|
|
|
|
|
|
diverges inside that message restores the batch-start checkpoint instead and re-processes the
|
|
|
|
|
|
|
|
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
|
|
|
|
|
|
|
|
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
|
|
|
|
|
|
|
|
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
|
|
|
|
|
|
|
|
upstream behaviour everywhere.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Invariants that must hold.**
|
|
|
|
|
|
|
|
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
|
|
|
|
|
|
|
|
divergence inside the last user message to less than `n_ubatch` tokens.
|
|
|
|
|
|
|
|
- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared
|
|
|
|
|
|
|
|
by `has_mtmd`); hence the `!has_mtmd` condition.
|
|
|
|
|
|
|
|
- It only changes where batches end, never which tokens are processed, so outputs differ from an
|
|
|
|
|
|
|
|
unmerged run only by batch-composition rounding.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Measured (2026-11-22, pod, GPU to itself in one swapdown window, build `f3397cded`; driver
|
|
|
|
|
|
|
|
`/root/arms/usw_ab.py`, window script `/root/arms/usw-window.sh`, logs and `res.json` per arm in
|
|
|
|
|
|
|
|
`/root/arms/usw-ab/{M1,U1,U2,M2}/`).** Production 35B think entry (`q36t`: `${ctx_256k}
|
|
|
|
|
|
|
|
${qwen35_think_args} ${moe_offload_args}`, `-hf` pulls the mmproj, supervisord's environment, scratch
|
|
|
|
|
|
|
|
arena and `--cache-disk-path`, plus `-lv 4`), one fresh server per arm, arms M (merged, default) and U
|
|
|
|
|
|
|
|
(`LLAMA_USERSTART_MERGE=0`) in the order M U U M. Each chat: system prompt, 6 user turns from
|
|
|
|
|
|
|
|
`pt-chunk.txt` (150, 10, 40, 150, 600, 40 tokens), synthetic previous answers of 300, 800, 1200, 500,
|
|
|
|
|
|
|
|
1300 tokens, `max_tokens 8`, `temperature 0`, streamed; then the last user message edited in the
|
|
|
|
|
|
|
|
middle and resent. TTFT in seconds (mean of the two runs per arm; the single runs in brackets),
|
|
|
|
|
|
|
|
`prompt_n` was identical in all four runs on every turn:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| turn | user tokens | prompt_n | M TTFT | U TTFT | M - U |
|
|
|
|
|
|
|
|
|---|---|---|---|---|---|
|
|
|
|
|
|
|
|
| 1 (fresh prompt) | 150 | 196 | 3.99 (4.43, 3.55) | 6.08 (5.73, 6.42) | -2.1 |
|
|
|
|
|
|
|
|
| 2 | 10 | 327 | 4.11 (3.75, 4.47) | 7.66 (6.75, 8.57) | -3.6 |
|
|
|
|
|
|
|
|
| 3 | 40 | 869 | 6.34 (6.62, 6.06) | 9.34 (7.26, 11.42) | -3.0 |
|
|
|
|
|
|
|
|
| 4 | 150 | 1381 | 7.20 (7.30, 7.10) | 10.95 (12.31, 9.58) | -3.7 |
|
|
|
|
|
|
|
|
| 5 | 600 | 1130 | 8.82 (7.47, 10.17) | 9.80 (9.52, 10.07) | -1.0 |
|
|
|
|
|
|
|
|
| 6 | 40 | 1370 | 8.11 (8.01, 8.20) | 9.53 (9.33, 9.72) | -1.4 |
|
|
|
|
|
|
|
|
| edit of turn 6 | 40 | M 1374, U 66 | 7.94 (8.16, 7.71) | 3.65 (4.09, 3.20) | +4.3 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Batch layout (`prompt processing, n_tokens` and checkpoint lines at `-lv 4`): M runs every turn as
|
|
|
|
|
|
|
|
one batch `[restore point, end-4)` plus the tail, with checkpoints at the batch start (superseding
|
|
|
|
|
|
|
|
the restored one) and at end minus 4; U splits at the last user message (an extra checkpoint
|
|
|
|
|
|
|
|
there, for example 1323 between 515 and 1380 on turn 3) and on turn 1 also at the first user
|
|
|
|
|
|
|
|
message (system prompt, 23 tokens). Turn N+1 restores the end-minus-4 checkpoint of turn N in both
|
|
|
|
|
|
|
|
arms. No `forcing full prompt re-processing` in any run.
|
|
|
|
|
|
|
|
- The 10-token turn gained too, against the plan's "no change". U split it into 308 tokens (the
|
|
|
|
|
|
|
|
previous answer, on the GPU) and 19 (message, generation prompt and tail), and the 19-token part runs
|
|
|
|
|
|
|
|
the experts of all 40 layers on the CPU, which costs seconds rather than nothing at 6 threads
|
|
|
|
|
|
|
|
(inferred from the totals, not profiled). The 600-token turn gained
|
|
|
|
|
|
|
|
least (its two parts are 508 and 618 tokens, both on the GPU, so the split costs one extra
|
|
|
|
|
|
|
|
upload, about 1-2 s against noise of the same size).
|
|
|
|
|
|
|
|
- Edit of the last user message (the cost of the trade): M has no checkpoint at the user-message
|
|
|
|
|
|
|
|
start, restores the batch start and re-processes previous answer + message (1374 tokens, one
|
|
|
|
|
|
|
|
ubatch); U restores the user-start checkpoint and re-processes 66 tokens. +4.3 s per such edit.
|
|
|
|
|
|
|
|
Ordinary follow-up turns are far more frequent than edits of the last message in this use; a client
|
|
|
|
|
|
|
|
that injects per-request context into the last user message (RAG, web search) pays the same
|
|
|
|
|
|
|
|
as an edit on every turn, so set `LLAMA_USERSTART_MERGE=0` for such a client.
|
|
|
|
|
|
|
|
- Outputs: identical within each arm across runs; between arms turns 5 and 6 differ after a few
|
|
|
|
|
|
|
|
tokens (batch-composition rounding, as the invariant says).
|
|
|
|
|
|
|
|
- Disk cache: `fork-validate.sh --diskcache` (in the window, GPU leg included) 15 PASS, 0 FAIL, 0
|
|
|
|
|
|
|
|
SKIP. `diskcache-window3.sh` legs with production's 35B command line (`q35.py` copied to
|
|
|
|
|
|
|
|
`q35-usw.py` with the candidate build path; log `/root/arms/usw-q35-mm.log`): 18.8k-token chat,
|
|
|
|
|
|
|
|
turn 1 83.5 s, save 557 MB in 1.0 s, restart restore and turn 2 `prompt_n` 27 in 3.7 s, output ==
|
|
|
|
|
|
|
|
reference; cold save and restore identity (`prompt_n` 4, output == reference); image answered
|
|
|
|
|
|
|
|
'Red' and not saved. The saved state after turn 2 holds 4 checkpoints instead of 5 (the user-start
|
|
|
|
|
|
|
|
one is gone).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Production (deployed 2026-11-22 10:37, `f3397cded`, config unchanged).** Deploy script
|
|
|
|
|
|
|
|
`/root/arms/dep-usw.sh` (log `dep-usw.log`): incremental in-place build, downtime 33 s. Post-deploy
|
|
|
|
|
|
|
|
checks through llama-swap (`/root/arms/post-usw.py`, `ink-usw.py`, `ink2-usw.py`, logs beside them):
|
|
|
|
|
|
|
|
`system_fingerprint` `b12687-f3397cded`; 35B 'Paris', then 4B / 35B handoffs twice (correct answers,
|
|
|
|
|
|
|
|
`holder` the answering PID each time, 2.9-6.3 s per switch); a 1140-token 35B chat, then a gemma-26B
|
|
|
|
|
|
|
|
image ('Red', the 35B exited by itself with `loop stopped, device memory freed` and `disk cache:
|
|
|
|
|
|
|
|
saved the prompt of 1157 tokens`), gemma text, 4B text, then the 35B chat plus one user message:
|
|
|
|
|
|
|
|
`disk cache: restored 1157 tokens`, `prompt_n` 30 (the suffix only). Inkling nothink: text correct
|
|
|
|
|
|
|
|
(it still spends about 40 tokens on reasoning, so a 96-token budget returns an empty content), image
|
|
|
|
|
|
|
|
'Red', a 9293-token Pan Tadeusz prompt at 82.0 t/s with the right answer (F13 measured 81.6). Children
|
|
|
|
|
|
|
|
bind llama-swap's ports. Log scan clean, `dmesg` clean (no ring timeout, GPU hang or OOM kill), pod
|
|
|
|
|
|
|
|
restarts 0 as before.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Failure modes and incidents.** None in validation or the deploy. Known cost: an edit of the last
|
|
|
|
|
|
|
|
user message (or per-request context injected into it) re-processes the previous answer too, +4.3 s
|
|
|
|
|
|
|
|
measured above.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Rebase hazards.**
|
|
|
|
|
|
|
|
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
|
|
|
|
|
|
|
|
`checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch
|
|
|
|
|
|
|
|
start is still checkpointed when the batch ends at prompt end minus 4.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Validation plan (run 2026-11-22, results under Measured above).**
|
|
|
|
|
|
|
|
1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the
|
|
|
|
|
|
|
|
`created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch
|
|
|
|
|
|
|
|
`[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the
|
|
|
|
|
|
|
|
last user message.
|
|
|
|
|
|
|
|
2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded),
|
|
|
|
|
|
|
|
candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn
|
|
|
|
|
|
|
|
chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read
|
|
|
|
|
|
|
|
`prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens
|
|
|
|
|
|
|
|
(the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens.
|
|
|
|
|
|
|
|
3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit
|
|
|
|
|
|
|
|
the last user message and resend: merged re-processes previous answer + message (bounded by one
|
|
|
|
|
|
|
|
ubatch), unmerged only the message; no "forcing full prompt re-processing".
|
|
|
|
|
|
|
|
4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line):
|
|
|
|
|
|
|
|
restore after a restart, `prompt_n` and output identical to the reference built from the same
|
|
|
|
|
|
|
|
commit.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
## 4. Dead ends
|
|
|
|
## 4. Dead ends
|
|
|
|
|
|
|
|
|
|
|
|
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
|
|
|
|
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
|
|
|
|