docs: FORK.md F11 - 35B results, the two leg-c anomalies explained
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens: 83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save: peak +432 MiB. Leg c, output != reference: c sent turn 2 again, so the server rolled back to the checkpoint at the prompt end and processed 4 tokens, while the reference processed 27 from turn 1. These are different reuse points. A controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a restore, a cold save and a restore) is bit-identical, top-3 logprobs included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh and has not run. Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock -s file sleep) leaked the lock into sleep, so the SIGTERM landed in an ordinary evict_kv release (the full KV shadow, F4). Not the cache; the driver is fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
18691e0090
commit
cb48bf2240
2 files changed
+43
-10
No files matched your search
@@ -15,7 +15,7 @@ Status at the time of writing (2026-10-25):
|
||||
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
|
||||
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
|
||||
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
|
||||
top. It is not deployed and has not run on the GPU yet (F11, Validation).
|
||||
top. It is not deployed; its validation (GPU and 35B included) is in F11.
|
||||
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
|
||||
its subject line. The commit messages are detailed and are the primary record of each mechanism
|
||||
and measurement; read them with `git show`.
|
||||
@@ -1018,11 +1018,39 @@ MiB free).
|
||||
| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms |
|
||||
| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s |
|
||||
|
||||
Not run yet: the GPU leg, the 35B, and `--arbiter` on this build. The swapdown window script
|
||||
`/root/arms/diskcache-window.sh` and the 35B driver `/root/arms/diskcache-test/q35.py` are on the pod
|
||||
(runbook section 6). Estimate for the 35B: 20 KiB of KV per token (10 attention layers, 2 KV heads,
|
||||
head_dim 256, f16), so about 600 MiB for 30k tokens, plus the recurrent state and its checkpoints;
|
||||
about 1-1.5 GB per file, about 1.5-2.5 s more per model switch, against about 150 s of re-prefill.
|
||||
Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.sh`):
|
||||
`--diskcache --arbiter` 29 PASS, 0 FAIL, GPU legs included.
|
||||
|
||||
35B (`q35.py`, production 35B flags with `--cache-disk-path` on a scratch dir, arbiter env, an
|
||||
18760-token pan-tadeusz chat; log `/root/arms/diskcache-test/q35.log`):
|
||||
|
||||
| leg | result |
|
||||
|---|---|
|
||||
| turn 1 prefill (reference and candidate) | 18760 tokens in 83.4 s (224.8 t/s), outputs identical |
|
||||
| warm save on SIGTERM (a) | 556.5 MiB (2 checkpoints), copy 323 ms, write 960 ms; token free in 328 ms (6 ms without the cache), exit 1952 ms (607 ms without), peak RSS +430 MiB |
|
||||
| restore after the restart (b) | read 860 ms, restore 114 ms; turn 2 prompt_n 27 of 18850 in 3.0 s (the reference that never restarted: 27 in 2.0 s); output identical |
|
||||
| cold save on SIGTERM (b, KV in the host shadow) | 746.7 MiB (5 checkpoints), copy 310 ms, write 1403 ms, exit 2928 ms, peak RSS +432 MiB: the in-place wrap adds no copy |
|
||||
| restore of the cold save (c, turn 2 sent again) | read 1189 ms, restore 75 ms, prompt_n 4; output differs from the reference's turn 2 (see below) |
|
||||
|
||||
Two anomalies in leg c, both explained:
|
||||
- Output != reference: c sent turn 2 again, so the saved list (turn 2 + 63 generated tokens) extends
|
||||
the new prompt. `update_slots()` then rolls back to the checkpoint at the prompt end minus 4 and
|
||||
processes 4 tokens, while the reference processed turn 2 from turn 1's state (27 tokens). These are
|
||||
different reuse points, not the same computation. The controlled test on Qwen3.5-0.8B
|
||||
(`/root/arms/diskcache-test/cmp.py`) compares the same reuse point. Server X never restarts and
|
||||
runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced
|
||||
cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the
|
||||
GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU
|
||||
to itself: `/root/arms/diskcache-window2.sh` (about 6 minutes, under swapdown) is ready and not run.
|
||||
Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the
|
||||
GPU), not proven on that model.
|
||||
- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
|
||||
The driver's fake waiter was `flock -s want.lock sleep 120`; killing flock left `sleep` holding
|
||||
want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right
|
||||
after its load and again right after its request (log 0.37.50). That ordinary arbiter release
|
||||
recaptures the full KV allocation as a host shadow, about 2.6 GB at ctx 131072 (F4), and the SIGTERM
|
||||
arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The
|
||||
driver now holds the lock in the process it kills, like `fork-validate.sh` does.
|
||||
|
||||
Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache
|
||||
--cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is
|
||||
@@ -1194,8 +1222,12 @@ Machine-level:
|
||||
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
|
||||
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
|
||||
is an estimate; its memory effect is verified in production.
|
||||
- F11 (disk prompt cache) has run CPU only. Not yet run: its GPU leg (a save from VRAM and from a
|
||||
released state, a restore into VRAM), a 35B measurement, and `--arbiter` on the candidate. The
|
||||
35B state size and exit time are estimates. The window script is ready (runbook section 6).
|
||||
- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the
|
||||
35B, only on the 0.8B (CPU and GPU). `/root/arms/diskcache-window2.sh` is ready (about 6 minutes of
|
||||
downtime).
|
||||
- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's
|
||||
host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time
|
||||
(token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that
|
||||
shadow while it is cold.
|
||||
- F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key
|
||||
holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB.
|
||||
@@ -273,7 +273,8 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
|
||||
`/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the
|
||||
production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script
|
||||
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
|
||||
under swapdown; it has not been run yet.
|
||||
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window2.sh` runs `cmp.py` on the 35B,
|
||||
the same-reuse-point identity check (FORK.md F11), not yet run.
|
||||
|
||||
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
|
||||
|
||||
|
||||
Reference in new issue
Block a user