docs: F11 mmproj incident and fix; runbook - production command line, build targets, detached scripts

FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort
with the production mmproj), the fix, the new mmproj cases (15 PASS), and
that the earlier 35B numbers were measured with --no-mmproj. The 35B run with
production's exact command line (diskcache-window3.sh) is ready, not run, and
a precondition for deploying --cache-disk-path.

Runbook:
- tests must use production's exact model command line, mmproj included;
- the validated build has all three targets;
- the deploy mtime list includes libllama-server-impl.so and
  libllama-common.so (llama-server is a stub);
- deploy and rollback steps run as one setsid-wrapped script on the pod,
  never as a backgrounded cat && sh -n && setsid chain from kubectl exec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 21:38:58 +01:00
1 parent fe57bc93a0
commit adc2aa3ecb
2 files changed
+65 -13

No files matched your search

+29 -8
View File
@@ -15,7 +15,8 @@ Status at the time of writing (2026-10-25):
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
top. It is not deployed; its validation (GPU and 35B included) is in F11.
top. A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); production stays on
`2fc9f9c35`. The fix is on the branch; the 35B run with the production command line is still due.
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line. The commit messages are detailed and are the primary record of each mechanism
and measurement; read them with `git show`.
@@ -975,6 +976,16 @@ saved the slot in use on shutdown, which is the main case here.
which must outlive them (both are members of the same memory object).
**Failure modes and incidents.**
- Production incident (2026-10-25, deploy of cb48bf224 with `--cache-disk-path`, rolled back): the first
request to the 35B aborted in `disk_cache_load()`. `server_tokens::get_tokens()` has
`GGML_ASSERT(!has_mtmd)`, and `has_mtmd` is true whenever an mmproj is loaded, even for text-only
prompts. Production's 35B and gemma-26B always load one (`-hf` pulls it; there is no
`--no-mmproj`). `disk_cache_collect()` and `save()` had the same call, so the shutdown save would
have aborted too. Every test had used `--no-mmproj` (`q35.py` and `--diskcache`), which hid it.
Fixed: the feature takes the token list from `get_text_tokens()`, and a prompt is text-only when
that list is as long as `size()` (`disk_cache_text`). A text-only prompt is cached with an mmproj
loaded. A prompt with media is never saved or restored. When the slot itself holds media, its reuse
estimate is `get_common_prefix()`. Lesson (runbook section 6): test with production's exact command line.
- The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself
(CPU_REPACK, F4), not in the save.
- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full
@@ -1008,9 +1019,12 @@ hybrid, CPU only (`-dev none -ngl 0`), with chat requests and greedy decoding, a
that never restarts. Cases: `restore`, `restore-arbiter` and `restore-cold` (turn 1, SIGTERM, restart,
turn 2 restored; prompt_n equal to the reference; identical answer), `no-prefix`, `key-ctx`,
`key-ctk`, `key-model`, `corrupt` (truncated and byte-flipped), `lru`, `write-error` (RLIMIT_FSIZE,
SIGXFSZ ignored), `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free. First run
SIGXFSZ ignored), `restore-mmproj` (the same restore with the 0.8B's mmproj loaded),
`mmproj-image`, `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free.
`mmproj-image` sends an image (answered "red", not restored), then the text chat (restored, identical),
then an image again, and SIGTERMs with media in the slot (no state copied, exit 0). First run
(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219
MiB free).
MiB free). After the mmproj fix (beside production, 5.3 GB VRAM free): 15 PASS, 0 FAIL, 0 SKIP.
| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill |
|---|---|---|---|---|---|---|
@@ -1021,8 +1035,12 @@ MiB free).
Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.sh`):
`--diskcache --arbiter` 29 PASS, 0 FAIL, GPU legs included.
35B (`q35.py`, production 35B flags with `--cache-disk-path` on a scratch dir, arbiter env, an
18760-token pan-tadeusz chat; log `/root/arms/diskcache-test/q35.log`):
35B (`q35.py`, the production 35B flags but WITH `--no-mmproj`, unlike production, which hid the
incident above; `--cache-disk-path` on a scratch dir, arbiter env, an 18760-token pan-tadeusz chat;
log `/root/arms/diskcache-test/q35.log`). `q35.py` now uses the exact production entry (`-hf`, so
the mmproj loads, nothing extra). That run (`/root/arms/diskcache-window3.sh`, about 8 minutes under
swapdown, which also covers the same-reuse-point identity and an image request on the 35B) is
ready and not run:
| leg | result |
|---|---|
@@ -1041,7 +1059,9 @@ Two anomalies in leg c, both explained:
runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced
cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the
GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU
to itself: `/root/arms/diskcache-window2.sh` (about 6 minutes, under swapdown) is ready and not run.
to itself: `/root/arms/diskcache-window3.sh` (production command line, reference "turn 2 again" against
leg c; about 8 minutes under swapdown) is ready and not run. It supersedes `diskcache-window2.sh`,
whose `cmp.py` runs with `--no-mmproj`.
Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the
GPU), not proven on that model.
- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
@@ -1223,8 +1243,9 @@ Machine-level:
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
is an estimate; its memory effect is verified in production.
- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the
35B, only on the 0.8B (CPU and GPU). `/root/arms/diskcache-window2.sh` is ready (about 6 minutes of
downtime).
35B, only on the 0.8B (CPU and GPU), and no 35B run has used production's command line (mmproj
loaded) yet. `/root/arms/diskcache-window3.sh` covers both (about 8 minutes of downtime). Do not
deploy `--cache-disk-path` before it passes.
- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's
host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time
(token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that
+36 -5
View File
@@ -200,12 +200,27 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '
harmless with `-DLLAMA_BUILD_UI=OFF`; the listed targets still build.
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
holding a `kubectl exec` open for hours.
- The validated build must contain all three targets: `llama-server`, `llama-bench` and
`test-backend-ops` (`--perf` and `--ops` run the last two from `--build`). A build script that
makes only `llama-server` (as the 2026-10-25 `diskcache-build.sh` first did) is not a validated
build.
- Detach only one script. Upload it first (`kubectl exec -i ... sh -c 'cat > /root/arms/X.sh' <
X.sh`), then start it alone: `setsid nohup sh /root/arms/X.sh > X.log 2>&1 </dev/null &`. Do not
background a chain such as `cat > X && sh -n X && setsid ... &` inside the `kubectl exec` shell. The
chain is a child of that session, and it can die with the session before `setsid` runs.
## 6. Validate with `scripts/fork-validate.sh`
Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of
the script describes each check.
Rule: any test of a production model must use production's exact command line: the `cmd` of its
`/root/config.yaml` entry with the macros expanded, including the mmproj that `-hf` pulls (production
never passes `--no-mmproj` to the 35B or gemma), and supervisord's `environment=`. Change only
`--port`, the arena and scratch paths. The 2026-10-25 deploy of the disk prompt cache aborted on the
first request: every test had used `--no-mmproj`, which hid a `GGML_ASSERT(!has_mtmd)`, and it was
rolled back (FORK.md F11).
```sh
# phase 1, beside production
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
@@ -273,8 +288,10 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
`/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the
production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window2.sh` runs `cmp.py` on the 35B,
the same-reuse-point identity check (FORK.md F11), not yet run.
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window3.sh` runs `q35.py`
with production's exact 35B command line (mmproj loaded): save/restore, cold save, the
same-reuse-point identity and an image request (FORK.md F11). Not yet run; it supersedes
`diskcache-window2.sh` (`cmp.py` runs with `--no-mmproj`).
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
@@ -286,6 +303,10 @@ the validated worktree build, so nothing is copied over; compare
`grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt` in both
trees if in doubt.
Run the stop, build and start (steps 4-7) as one script on the pod, started with `setsid nohup sh
<script> > <log> 2>&1 </dev/null &`, never as a backgrounded `cat && sh -n && setsid` chain from the
`kubectl exec` shell (see section 9).
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
```sh
@@ -343,12 +364,15 @@ trees if in doubt.
6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
```sh
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12
ls -lL build/bin/llama-server build/bin/libllama.so build/bin/libmtmd.so \
build/bin/libggml.so build/bin/libggml-base.so build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench test-backend-ops -j 12
ls -lL build/bin/llama-server build/bin/libllama-server-impl.so build/bin/libllama-common.so \
build/bin/libllama.so build/bin/libmtmd.so build/bin/libggml.so build/bin/libggml-base.so \
build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
```
All must have fresh mtimes. Use `-L` (follow the symlinks): `build/bin` keeps every old versioned
`libggml-*.so.0.x` and `libmtmd.so.0.x`, so a plain `ls` lists stale files beside the new ones.
`llama-server` is a 17 KB stub. The server code is in `libllama-server-impl.so`, the common code
(arguments, sampling) in `libllama-common.so`; a server change shows only in their mtimes.
7. **Start and probe.**
```sh
supervisorctl -c /root/supervisord.conf start llama-swap
@@ -438,6 +462,13 @@ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in
production. Keep the candidate worktree for a later redeploy.
If you run the rollback (or the deploy steps 4-7) from a workstation through `kubectl exec`, put the
whole sequence in one script on the pod and start it as `setsid nohup sh /root/arms/rollback.sh >
/root/arms/rollback.log 2>&1 </dev/null &`. Do not background a `cat > X && sh -n X && setsid ...`
chain in the exec shell: it can die with the exec session halfway, leaving llama-swap stopped (section
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
Pointer files 2 to 10 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`, and
commit-10 `6c480f7e7`, the rollback target for the current production `2fc9f9c35`. The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on