From cb48bf2240268045e0618d2c4dba30ae371c2bd5 Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Sun, 25 Oct 2026 20:57:42 +0100 Subject: [PATCH] docs: FORK.md F11 - 35B results, the two leg-c anomalies explained The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens: 83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save: peak +432 MiB. Leg c, output != reference: c sent turn 2 again, so the server rolled back to the checkpoint at the prompt end and processed 4 tokens, while the reference processed 27 from turn 1. These are different reuse points. A controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a restore, a cold save and a restore) is bit-identical, top-3 logprobs included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh and has not run. Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock -s file sleep) leaked the lock into sleep, so the SIGTERM landed in an ordinary evict_kv release (the full KV shadow, F4). Not the cache; the driver is fixed. Co-Authored-By: Claude Opus 5.5 --- FORK.md | 50 ++++++++++++++++++++++++++++++------- docs/fork-rebase-runbook.md | 3 ++- 2 files changed, 43 insertions(+), 10 deletions(-) diff --git a/FORK.md b/FORK.md index c9ea436a6..4c8863bd9 100644 --- a/FORK.md +++ b/FORK.md @@ -15,7 +15,7 @@ Status at the time of writing (2026-10-25): previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook section 6). Any later commit that touches compiled code needs the same validation before a redeploy. - Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on - top. It is not deployed and has not run on the GPU yet (F11, Validation). + top. It is not deployed; its validation (GPU and 35B included) is in F11. - Commit hashes below refer to that branch. They change on every rebase; find a commit again by its subject line. The commit messages are detailed and are the primary record of each mechanism and measurement; read them with `git show`. @@ -1018,11 +1018,39 @@ MiB free). | 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms | | 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s | -Not run yet: the GPU leg, the 35B, and `--arbiter` on this build. The swapdown window script -`/root/arms/diskcache-window.sh` and the 35B driver `/root/arms/diskcache-test/q35.py` are on the pod -(runbook section 6). Estimate for the 35B: 20 KiB of KV per token (10 attention layers, 2 KV heads, -head_dim 256, f16), so about 600 MiB for 30k tokens, plus the recurrent state and its checkpoints; -about 1-1.5 GB per file, about 1.5-2.5 s more per model switch, against about 150 s of re-prefill. +Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.sh`): +`--diskcache --arbiter` 29 PASS, 0 FAIL, GPU legs included. + +35B (`q35.py`, production 35B flags with `--cache-disk-path` on a scratch dir, arbiter env, an +18760-token pan-tadeusz chat; log `/root/arms/diskcache-test/q35.log`): + +| leg | result | +|---|---| +| turn 1 prefill (reference and candidate) | 18760 tokens in 83.4 s (224.8 t/s), outputs identical | +| warm save on SIGTERM (a) | 556.5 MiB (2 checkpoints), copy 323 ms, write 960 ms; token free in 328 ms (6 ms without the cache), exit 1952 ms (607 ms without), peak RSS +430 MiB | +| restore after the restart (b) | read 860 ms, restore 114 ms; turn 2 prompt_n 27 of 18850 in 3.0 s (the reference that never restarted: 27 in 2.0 s); output identical | +| cold save on SIGTERM (b, KV in the host shadow) | 746.7 MiB (5 checkpoints), copy 310 ms, write 1403 ms, exit 2928 ms, peak RSS +432 MiB: the in-place wrap adds no copy | +| restore of the cold save (c, turn 2 sent again) | read 1189 ms, restore 75 ms, prompt_n 4; output differs from the reference's turn 2 (see below) | + +Two anomalies in leg c, both explained: +- Output != reference: c sent turn 2 again, so the saved list (turn 2 + 63 generated tokens) extends + the new prompt. `update_slots()` then rolls back to the checkpoint at the prompt end minus 4 and + processes 4 tokens, while the reference processed turn 2 from turn 1's state (27 tokens). These are + different reuse points, not the same computation. The controlled test on Qwen3.5-0.8B + (`/root/arms/diskcache-test/cmp.py`) compares the same reuse point. Server X never restarts and + runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced + cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the + GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU + to itself: `/root/arms/diskcache-window2.sh` (about 6 minutes, under swapdown) is ready and not run. + Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the + GPU), not proven on that model. +- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug. + The driver's fake waiter was `flock -s want.lock sleep 120`; killing flock left `sleep` holding + want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right + after its load and again right after its request (log 0.37.50). That ordinary arbiter release + recaptures the full KV allocation as a host shadow, about 2.6 GB at ctx 131072 (F4), and the SIGTERM + arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The + driver now holds the lock in the process it kills, like `fork-validate.sh` does. Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache --cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is @@ -1194,8 +1222,12 @@ Machine-level: Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them. - F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate; its memory effect is verified in production. -- F11 (disk prompt cache) has run CPU only. Not yet run: its GPU leg (a save from VRAM and from a - released state, a restore into VRAM), a 35B measurement, and `--arbiter` on the candidate. The - 35B state size and exit time are estimates. The window script is ready (runbook section 6). +- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the + 35B, only on the 0.8B (CPU and GPU). `/root/arms/diskcache-window2.sh` is ready (about 6 minutes of + downtime). +- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's + host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time + (token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that + shadow while it is cold. - F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB. diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md index 4106603eb..8f743c286 100644 --- a/docs/fork-rebase-runbook.md +++ b/docs/fork-rebase-runbook.md @@ -273,7 +273,8 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f `/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script `/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs - under swapdown; it has not been run yet. + under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window2.sh` runs `cmp.py` on the 35B, + the same-reuse-point identity check (FORK.md F11), not yet run. On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.