diff --git a/FORK.md b/FORK.md index 4c8863bd9..b969764f0 100644 --- a/FORK.md +++ b/FORK.md @@ -15,7 +15,8 @@ Status at the time of writing (2026-10-25): previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook section 6). Any later commit that touches compiled code needs the same validation before a redeploy. - Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on - top. It is not deployed; its validation (GPU and 35B included) is in F11. + top. A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); production stays on + `2fc9f9c35`. The fix is on the branch; the 35B run with the production command line is still due. - Commit hashes below refer to that branch. They change on every rebase; find a commit again by its subject line. The commit messages are detailed and are the primary record of each mechanism and measurement; read them with `git show`. @@ -975,6 +976,16 @@ saved the slot in use on shutdown, which is the main case here. which must outlive them (both are members of the same memory object). **Failure modes and incidents.** +- Production incident (2026-10-25, deploy of cb48bf224 with `--cache-disk-path`, rolled back): the first + request to the 35B aborted in `disk_cache_load()`. `server_tokens::get_tokens()` has + `GGML_ASSERT(!has_mtmd)`, and `has_mtmd` is true whenever an mmproj is loaded, even for text-only + prompts. Production's 35B and gemma-26B always load one (`-hf` pulls it; there is no + `--no-mmproj`). `disk_cache_collect()` and `save()` had the same call, so the shutdown save would + have aborted too. Every test had used `--no-mmproj` (`q35.py` and `--diskcache`), which hid it. + Fixed: the feature takes the token list from `get_text_tokens()`, and a prompt is text-only when + that list is as long as `size()` (`disk_cache_text`). A text-only prompt is cached with an mmproj + loaded. A prompt with media is never saved or restored. When the slot itself holds media, its reuse + estimate is `get_common_prefix()`. Lesson (runbook section 6): test with production's exact command line. - The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself (CPU_REPACK, F4), not in the save. - The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full @@ -1008,9 +1019,12 @@ hybrid, CPU only (`-dev none -ngl 0`), with chat requests and greedy decoding, a that never restarts. Cases: `restore`, `restore-arbiter` and `restore-cold` (turn 1, SIGTERM, restart, turn 2 restored; prompt_n equal to the reference; identical answer), `no-prefix`, `key-ctx`, `key-ctk`, `key-model`, `corrupt` (truncated and byte-flipped), `lru`, `write-error` (RLIMIT_FSIZE, -SIGXFSZ ignored), `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free. First run +SIGXFSZ ignored), `restore-mmproj` (the same restore with the 0.8B's mmproj loaded), +`mmproj-image`, `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free. +`mmproj-image` sends an image (answered "red", not restored), then the text chat (restored, identical), +then an image again, and SIGTERMs with media in the slot (no state copied, exit 0). First run (2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219 -MiB free). +MiB free). After the mmproj fix (beside production, 5.3 GB VRAM free): 15 PASS, 0 FAIL, 0 SKIP. | case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill | |---|---|---|---|---|---|---| @@ -1021,8 +1035,12 @@ MiB free). Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.sh`): `--diskcache --arbiter` 29 PASS, 0 FAIL, GPU legs included. -35B (`q35.py`, production 35B flags with `--cache-disk-path` on a scratch dir, arbiter env, an -18760-token pan-tadeusz chat; log `/root/arms/diskcache-test/q35.log`): +35B (`q35.py`, the production 35B flags but WITH `--no-mmproj`, unlike production, which hid the +incident above; `--cache-disk-path` on a scratch dir, arbiter env, an 18760-token pan-tadeusz chat; +log `/root/arms/diskcache-test/q35.log`). `q35.py` now uses the exact production entry (`-hf`, so +the mmproj loads, nothing extra). That run (`/root/arms/diskcache-window3.sh`, about 8 minutes under +swapdown, which also covers the same-reuse-point identity and an image request on the 35B) is +ready and not run: | leg | result | |---|---| @@ -1041,7 +1059,9 @@ Two anomalies in leg c, both explained: runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU - to itself: `/root/arms/diskcache-window2.sh` (about 6 minutes, under swapdown) is ready and not run. + to itself: `/root/arms/diskcache-window3.sh` (production command line, reference "turn 2 again" against + leg c; about 8 minutes under swapdown) is ready and not run. It supersedes `diskcache-window2.sh`, + whose `cmp.py` runs with `--no-mmproj`. Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the GPU), not proven on that model. - Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug. @@ -1223,8 +1243,9 @@ Machine-level: - F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate; its memory effect is verified in production. - F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the - 35B, only on the 0.8B (CPU and GPU). `/root/arms/diskcache-window2.sh` is ready (about 6 minutes of - downtime). + 35B, only on the 0.8B (CPU and GPU), and no 35B run has used production's command line (mmproj + loaded) yet. `/root/arms/diskcache-window3.sh` covers both (about 8 minutes of downtime). Do not + deploy `--cache-disk-path` before it passes. - A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time (token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md index 8f743c286..b7d5691be 100644 --- a/docs/fork-rebase-runbook.md +++ b/docs/fork-rebase-runbook.md @@ -200,12 +200,27 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c ' harmless with `-DLLAMA_BUILD_UI=OFF`; the listed targets still build. - Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 /root/arms/X.sh' < + X.sh`), then start it alone: `setsid nohup sh /root/arms/X.sh > X.log 2>&1 X && sh -n X && setsid ... &` inside the `kubectl exec` shell. The + chain is a child of that session, and it can die with the session before `setsid` runs. ## 6. Validate with `scripts/fork-validate.sh` Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of the script describes each check. +Rule: any test of a production model must use production's exact command line: the `cmd` of its +`/root/config.yaml` entry with the macros expanded, including the mmproj that `-hf` pulls (production +never passes `--no-mmproj` to the 35B or gemma), and supervisord's `environment=`. Change only +`--port`, the arena and scratch paths. The 2026-10-25 deploy of the disk prompt cache aborted on the +first request: every test had used `--no-mmproj`, which hid a `GGML_ASSERT(!has_mtmd)`, and it was +rolled back (FORK.md F11). + ```sh # phase 1, beside production bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \ @@ -273,8 +288,10 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f `/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script `/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs - under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window2.sh` runs `cmp.py` on the 35B, - the same-reuse-point identity check (FORK.md F11), not yet run. + under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window3.sh` runs `q35.py` + with production's exact 35B command line (mmproj loaded): save/restore, cold save, the + same-reuse-point identity and an image request (FORK.md F11). Not yet run; it supersedes + `diskcache-window2.sh` (`cmp.py` runs with `--no-mmproj`). On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs. @@ -286,6 +303,10 @@ the validated worktree build, so nothing is copied over; compare `grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt` in both trees if in doubt. +Run the stop, build and start (steps 4-7) as one script on the pod, started with `setsid nohup sh +