docs: F11 mmproj incident and fix; runbook - production command line, build targets, detached scripts
FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort with the production mmproj), the fix, the new mmproj cases (15 PASS), and that the earlier 35B numbers were measured with --no-mmproj. The 35B run with production's exact command line (diskcache-window3.sh) is ready, not run, and a precondition for deploying --cache-disk-path. Runbook: - tests must use production's exact model command line, mmproj included; - the validated build has all three targets; - the deploy mtime list includes libllama-server-impl.so and libllama-common.so (llama-server is a stub); - deploy and rollback steps run as one setsid-wrapped script on the pod, never as a backgrounded cat && sh -n && setsid chain from kubectl exec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
fe57bc93a0
commit
adc2aa3ecb
2 files changed
+65
-13
No files matched your search
@@ -15,7 +15,8 @@ Status at the time of writing (2026-10-25):
|
||||
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
|
||||
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
|
||||
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
|
||||
top. It is not deployed; its validation (GPU and 35B included) is in F11.
|
||||
top. A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); production stays on
|
||||
`2fc9f9c35`. The fix is on the branch; the 35B run with the production command line is still due.
|
||||
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
|
||||
its subject line. The commit messages are detailed and are the primary record of each mechanism
|
||||
and measurement; read them with `git show`.
|
||||
@@ -975,6 +976,16 @@ saved the slot in use on shutdown, which is the main case here.
|
||||
which must outlive them (both are members of the same memory object).
|
||||
|
||||
**Failure modes and incidents.**
|
||||
- Production incident (2026-10-25, deploy of cb48bf224 with `--cache-disk-path`, rolled back): the first
|
||||
request to the 35B aborted in `disk_cache_load()`. `server_tokens::get_tokens()` has
|
||||
`GGML_ASSERT(!has_mtmd)`, and `has_mtmd` is true whenever an mmproj is loaded, even for text-only
|
||||
prompts. Production's 35B and gemma-26B always load one (`-hf` pulls it; there is no
|
||||
`--no-mmproj`). `disk_cache_collect()` and `save()` had the same call, so the shutdown save would
|
||||
have aborted too. Every test had used `--no-mmproj` (`q35.py` and `--diskcache`), which hid it.
|
||||
Fixed: the feature takes the token list from `get_text_tokens()`, and a prompt is text-only when
|
||||
that list is as long as `size()` (`disk_cache_text`). A text-only prompt is cached with an mmproj
|
||||
loaded. A prompt with media is never saved or restored. When the slot itself holds media, its reuse
|
||||
estimate is `get_common_prefix()`. Lesson (runbook section 6): test with production's exact command line.
|
||||
- The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself
|
||||
(CPU_REPACK, F4), not in the save.
|
||||
- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full
|
||||
@@ -1008,9 +1019,12 @@ hybrid, CPU only (`-dev none -ngl 0`), with chat requests and greedy decoding, a
|
||||
that never restarts. Cases: `restore`, `restore-arbiter` and `restore-cold` (turn 1, SIGTERM, restart,
|
||||
turn 2 restored; prompt_n equal to the reference; identical answer), `no-prefix`, `key-ctx`,
|
||||
`key-ctk`, `key-model`, `corrupt` (truncated and byte-flipped), `lru`, `write-error` (RLIMIT_FSIZE,
|
||||
SIGXFSZ ignored), `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free. First run
|
||||
SIGXFSZ ignored), `restore-mmproj` (the same restore with the 0.8B's mmproj loaded),
|
||||
`mmproj-image`, `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free.
|
||||
`mmproj-image` sends an image (answered "red", not restored), then the text chat (restored, identical),
|
||||
then an image again, and SIGTERMs with media in the slot (no state copied, exit 0). First run
|
||||
(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219
|
||||
MiB free).
|
||||
MiB free). After the mmproj fix (beside production, 5.3 GB VRAM free): 15 PASS, 0 FAIL, 0 SKIP.
|
||||
|
||||
| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill |
|
||||
|---|---|---|---|---|---|---|
|
||||
@@ -1021,8 +1035,12 @@ MiB free).
|
||||
Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.sh`):
|
||||
`--diskcache --arbiter` 29 PASS, 0 FAIL, GPU legs included.
|
||||
|
||||
35B (`q35.py`, production 35B flags with `--cache-disk-path` on a scratch dir, arbiter env, an
|
||||
18760-token pan-tadeusz chat; log `/root/arms/diskcache-test/q35.log`):
|
||||
35B (`q35.py`, the production 35B flags but WITH `--no-mmproj`, unlike production, which hid the
|
||||
incident above; `--cache-disk-path` on a scratch dir, arbiter env, an 18760-token pan-tadeusz chat;
|
||||
log `/root/arms/diskcache-test/q35.log`). `q35.py` now uses the exact production entry (`-hf`, so
|
||||
the mmproj loads, nothing extra). That run (`/root/arms/diskcache-window3.sh`, about 8 minutes under
|
||||
swapdown, which also covers the same-reuse-point identity and an image request on the 35B) is
|
||||
ready and not run:
|
||||
|
||||
| leg | result |
|
||||
|---|---|
|
||||
@@ -1041,7 +1059,9 @@ Two anomalies in leg c, both explained:
|
||||
runs turn 1, turn 2, then turn 2 again. Server Z runs turn 2 again after a disk restore, a forced
|
||||
cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the
|
||||
GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU
|
||||
to itself: `/root/arms/diskcache-window2.sh` (about 6 minutes, under swapdown) is ready and not run.
|
||||
to itself: `/root/arms/diskcache-window3.sh` (production command line, reference "turn 2 again" against
|
||||
leg c; about 8 minutes under swapdown) is ready and not run. It supersedes `diskcache-window2.sh`,
|
||||
whose `cmp.py` runs with `--no-mmproj`.
|
||||
Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the
|
||||
GPU), not proven on that model.
|
||||
- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
|
||||
@@ -1223,8 +1243,9 @@ Machine-level:
|
||||
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
|
||||
is an estimate; its memory effect is verified in production.
|
||||
- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the
|
||||
35B, only on the 0.8B (CPU and GPU). `/root/arms/diskcache-window2.sh` is ready (about 6 minutes of
|
||||
downtime).
|
||||
35B, only on the 0.8B (CPU and GPU), and no 35B run has used production's command line (mmproj
|
||||
loaded) yet. `/root/arms/diskcache-window3.sh` covers both (about 8 minutes of downtime). Do not
|
||||
deploy `--cache-disk-path` before it passes.
|
||||
- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's
|
||||
host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time
|
||||
(token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that
|
||||
|
||||
@@ -200,12 +200,27 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '
|
||||
harmless with `-DLLAMA_BUILD_UI=OFF`; the listed targets still build.
|
||||
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
|
||||
holding a `kubectl exec` open for hours.
|
||||
- The validated build must contain all three targets: `llama-server`, `llama-bench` and
|
||||
`test-backend-ops` (`--perf` and `--ops` run the last two from `--build`). A build script that
|
||||
makes only `llama-server` (as the 2026-10-25 `diskcache-build.sh` first did) is not a validated
|
||||
build.
|
||||
- Detach only one script. Upload it first (`kubectl exec -i ... sh -c 'cat > /root/arms/X.sh' <
|
||||
X.sh`), then start it alone: `setsid nohup sh /root/arms/X.sh > X.log 2>&1 </dev/null &`. Do not
|
||||
background a chain such as `cat > X && sh -n X && setsid ... &` inside the `kubectl exec` shell. The
|
||||
chain is a child of that session, and it can die with the session before `setsid` runs.
|
||||
|
||||
## 6. Validate with `scripts/fork-validate.sh`
|
||||
|
||||
Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of
|
||||
the script describes each check.
|
||||
|
||||
Rule: any test of a production model must use production's exact command line: the `cmd` of its
|
||||
`/root/config.yaml` entry with the macros expanded, including the mmproj that `-hf` pulls (production
|
||||
never passes `--no-mmproj` to the 35B or gemma), and supervisord's `environment=`. Change only
|
||||
`--port`, the arena and scratch paths. The 2026-10-25 deploy of the disk prompt cache aborted on the
|
||||
first request: every test had used `--no-mmproj`, which hid a `GGML_ASSERT(!has_mtmd)`, and it was
|
||||
rolled back (FORK.md F11).
|
||||
|
||||
```sh
|
||||
# phase 1, beside production
|
||||
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
|
||||
@@ -273,8 +288,10 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
|
||||
`/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the
|
||||
production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script
|
||||
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
|
||||
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window2.sh` runs `cmp.py` on the 35B,
|
||||
the same-reuse-point identity check (FORK.md F11), not yet run.
|
||||
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window3.sh` runs `q35.py`
|
||||
with production's exact 35B command line (mmproj loaded): save/restore, cold save, the
|
||||
same-reuse-point identity and an image request (FORK.md F11). Not yet run; it supersedes
|
||||
`diskcache-window2.sh` (`cmp.py` runs with `--no-mmproj`).
|
||||
|
||||
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
|
||||
|
||||
@@ -286,6 +303,10 @@ the validated worktree build, so nothing is copied over; compare
|
||||
`grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt` in both
|
||||
trees if in doubt.
|
||||
|
||||
Run the stop, build and start (steps 4-7) as one script on the pod, started with `setsid nohup sh
|
||||
<script> > <log> 2>&1 </dev/null &`, never as a backgrounded `cat && sh -n && setsid` chain from the
|
||||
`kubectl exec` shell (see section 9).
|
||||
|
||||
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
|
||||
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
|
||||
```sh
|
||||
@@ -343,12 +364,15 @@ trees if in doubt.
|
||||
6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
|
||||
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
|
||||
```sh
|
||||
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12
|
||||
ls -lL build/bin/llama-server build/bin/libllama.so build/bin/libmtmd.so \
|
||||
build/bin/libggml.so build/bin/libggml-base.so build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
|
||||
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench test-backend-ops -j 12
|
||||
ls -lL build/bin/llama-server build/bin/libllama-server-impl.so build/bin/libllama-common.so \
|
||||
build/bin/libllama.so build/bin/libmtmd.so build/bin/libggml.so build/bin/libggml-base.so \
|
||||
build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
|
||||
```
|
||||
All must have fresh mtimes. Use `-L` (follow the symlinks): `build/bin` keeps every old versioned
|
||||
`libggml-*.so.0.x` and `libmtmd.so.0.x`, so a plain `ls` lists stale files beside the new ones.
|
||||
`llama-server` is a 17 KB stub. The server code is in `libllama-server-impl.so`, the common code
|
||||
(arguments, sampling) in `libllama-common.so`; a server change shows only in their mtimes.
|
||||
7. **Start and probe.**
|
||||
```sh
|
||||
supervisorctl -c /root/supervisord.conf start llama-swap
|
||||
@@ -438,6 +462,13 @@ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
|
||||
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in
|
||||
production. Keep the candidate worktree for a later redeploy.
|
||||
|
||||
If you run the rollback (or the deploy steps 4-7) from a workstation through `kubectl exec`, put the
|
||||
whole sequence in one script on the pod and start it as `setsid nohup sh /root/arms/rollback.sh >
|
||||
/root/arms/rollback.log 2>&1 </dev/null &`. Do not background a `cat > X && sh -n X && setsid ...`
|
||||
chain in the exec shell: it can die with the exec session halfway, leaving llama-swap stopped (section
|
||||
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
|
||||
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
|
||||
|
||||
Pointer files 2 to 10 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`, and
|
||||
commit-10 `6c480f7e7`, the rollback target for the current production `2fc9f9c35`. The formats vary
|
||||
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
|
||||
|
||||
Reference in new issue
Block a user