docs: FORK.md F11 - 35B results, the two leg-c anomalies explained

The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.

Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.

Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 20:57:42 +01:00
1 parent 18691e0090
commit cb48bf2240
2 files changed
+43 -10

No files matched your search

+41 -9
View File
@@ -15,7 +15,7 @@ Status at the time of writing (2026-10-25):
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
top. It is not deployed and has not run on the GPU yet (F11, Validation).
top. It is not deployed; its validation (GPU and 35B included) is in F11.
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line. The commit messages are detailed and are the primary record of each mechanism
and measurement; read them with `git show`.
@@ -1018,11 +1018,39 @@ MiB free).
| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms |
| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s |
Not run yet: the GPU leg, the 35B, and `--arbiter` on this build. The swapdown window script
`/root/arms/diskcache-window.sh` and the 35B driver `/root/arms/diskcache-test/q35.py` are on the pod
(runbook section 6). Estimate for the 35B: 20 KiB of KV per token (10 attention layers, 2 KV heads,
head_dim 256, f16), so about 600 MiB for 30k tokens, plus the recurrent state and its checkpoints;
about 1-1.5 GB per file, about 1.5-2.5 s more per model switch, against about 150 s of re-prefill.
Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.sh`):
`--diskcache --arbiter` 29 PASS, 0 FAIL, GPU legs included.
35B (`q35.py`, production 35B flags with `--cache-disk-path` on a scratch dir, arbiter env, an
18760-token pan-tadeusz chat; log `/root/arms/diskcache-test/q35.log`):
| leg | result |
|---|---|
| turn 1 prefill (reference and candidate) | 18760 tokens in 83.4 s (224.8 t/s), outputs identical |
| warm save on SIGTERM (a) | 556.5 MiB (2 checkpoints), copy 323 ms, write 960 ms; token free in 328 ms (6 ms without the cache), exit 1952 ms (607 ms without), peak RSS +430 MiB |
| restore after the restart (b) | read 860 ms, restore 114 ms; turn 2 prompt_n 27 of 18850 in 3.0 s (the reference that never restarted: 27 in 2.0 s); output identical |
| cold save on SIGTERM (b, KV in the host shadow) | 746.7 MiB (5 checkpoints), copy 310 ms, write 1403 ms, exit 2928 ms, peak RSS +432 MiB: the in-place wrap adds no copy |
| restore of the cold save (c, turn 2 sent again) | read 1189 ms, restore 75 ms, prompt_n 4; output differs from the reference's turn 2 (see below) |
Two anomalies in leg c, both explained:
- Output != reference: c sent turn 2 again, so the saved list (turn 2 + 63 generated tokens) extends
the new prompt. `update_slots()` then rolls back to the checkpoint at the prompt end minus 4 and
processes 4 tokens, while the reference processed turn 2 from turn 1's state (27 tokens). These are
different reuse points, not the same computation. The controlled test on Qwen3.5-0.8B
(`/root/arms/diskcache-test/cmp.py`) compares the same reuse point. Server X never restarts and
runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced
cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the
GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU
to itself: `/root/arms/diskcache-window2.sh` (about 6 minutes, under swapdown) is ready and not run.
Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the
GPU), not proven on that model.
- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
The driver's fake waiter was `flock -s want.lock sleep 120`; killing flock left `sleep` holding
want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right
after its load and again right after its request (log 0.37.50). That ordinary arbiter release
recaptures the full KV allocation as a host shadow, about 2.6 GB at ctx 131072 (F4), and the SIGTERM
arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The
driver now holds the lock in the process it kills, like `fork-validate.sh` does.
Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache
--cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is
@@ -1194,8 +1222,12 @@ Machine-level:
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
is an estimate; its memory effect is verified in production.
- F11 (disk prompt cache) has run CPU only. Not yet run: its GPU leg (a save from VRAM and from a
released state, a restore into VRAM), a 35B measurement, and `--arbiter` on the candidate. The
35B state size and exit time are estimates. The window script is ready (runbook section 6).
- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the
35B, only on the 0.8B (CPU and GPU). `/root/arms/diskcache-window2.sh` is ready (about 6 minutes of
downtime).
- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's
host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time
(token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that
shadow while it is cold.
- F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key
holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB.
+2 -1
View File
@@ -273,7 +273,8 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
`/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the
production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
under swapdown; it has not been run yet.
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window2.sh` runs `cmp.py` on the 35B,
the same-reuse-point identity check (FORK.md F11), not yet run.
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.