docs: runbook/FORK.md after the adc2aa3ec deploy

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Sonnet committed 2026-11-22 20:38:33 +01:00
1 parent 92abb2d89b
commit 86bf601db6
2 files changed
+52 -32

No files matched your search

+27 -21
View File
@@ -4,19 +4,20 @@ Maintainer documentation for this private fork. Read it before you rebase, chang
anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md),
which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-10-25):
Status at the time of writing (2026-11-21):
- Branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772`. The fork commits are
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
(this file among them), so count it rather than trusting a number written here. Section 2 maps
every commit to a feature.
- Production runs `2fc9f9c35` (deployed 2026-10-25 13:01 UTC, `system_fingerprint`
`b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
top. A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); production stays on
`2fc9f9c35`. The fix is on the branch; the 35B run with the production command line is still due.
- Production runs `adc2aa3ec` (system_fingerprint `b11906-adc2aa3ec`; rollback pointer
`/root/DEPLOY-ROLLBACK-commit-12.txt` = `2fc9f9c35`, the previous production; next free pointer 13),
with the disk prompt cache ON (F11). `2fc9f9c35` was validated with the full `fork-validate.sh` set
(37 PASS, 0 FAIL, runbook section 6). Any later commit that touches compiled code needs the same
validation before a redeploy.
- Branch `fable5/disk-prompt-cache` added F11 (disk prompt cache) and a CPU_REPACK release fix (F4).
A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); the fix (adc2aa3ec) passed
the 35B run with the production command line on 2026-11-21 and is deployed.
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line. The commit messages are detailed and are the primary record of each mechanism
and measurement; read them with `git show`.
@@ -1039,8 +1040,11 @@ Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.
incident above; `--cache-disk-path` on a scratch dir, arbiter env, an 18760-token pan-tadeusz chat;
log `/root/arms/diskcache-test/q35.log`). `q35.py` now uses the exact production entry (`-hf`, so
the mmproj loads, nothing extra). That run (`/root/arms/diskcache-window3.sh`, about 8 minutes under
swapdown, which also covers the same-reuse-point identity and an image request on the 35B) is
ready and not run:
swapdown, which also covers the same-reuse-point identity and an image request on the 35B) was run
2026-11-21 on adc2aa3ec (log `/root/arms/diskcache-test/q35-mm.log`) and passed all legs: 18.8k-token
chat, prefill 84.6 s -> 4.7 s for turn 2 after the restart with identical output; save 557 MB in
1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image answered and not saved. The
table below is the earlier `--no-mmproj` run:
| leg | result |
|---|---|
@@ -1060,10 +1064,9 @@ Two anomalies in leg c, both explained:
cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the
GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU
to itself: `/root/arms/diskcache-window3.sh` (production command line, reference "turn 2 again" against
leg c; about 8 minutes under swapdown) is ready and not run. It supersedes `diskcache-window2.sh`,
whose `cmp.py` runs with `--no-mmproj`.
Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the
GPU), not proven on that model.
leg c; about 8 minutes under swapdown). It ran 2026-11-21 and passed (same-reuse-point identity:
identical output), so the difference above is the reuse point. It supersedes
`diskcache-window2.sh`, whose `cmp.py` runs with `--no-mmproj`.
- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
The driver's fake waiter was `flock -s want.lock sleep 120`; killing flock left `sleep` holding
want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right
@@ -1072,9 +1075,12 @@ Two anomalies in leg c, both explained:
arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The
driver now holds the lock in the process it kills, like `fork-validate.sh` does.
Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache
--cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is
created on first start. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
Configuration (applied in production 2026-11-21): `--cache-disk-path /root/prompt-cache
--cache-disk-size 4096` appended to the `moe_offload_args` and `moe_offload_gemma` macros in
`/root/config.yaml` (backup `/root/config.yaml.bak-diskcache-20261121`). The directory is created on
first start. Verified in production: a 35B 4.6k-token chat was saved on eviction (278.6 MiB, copy
126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms), and turn 2 prefilled
only 32 tokens; a gemma image request with the flags on works and is not cached. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
applies. Models only pick up the flags when they restart.
---
@@ -1242,10 +1248,10 @@ Machine-level:
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
is an estimate; its memory effect is verified in production.
- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the
35B, only on the 0.8B (CPU and GPU), and no 35B run has used production's command line (mmproj
loaded) yet. `/root/arms/diskcache-window3.sh` covers both (about 8 minutes of downtime). Do not
deploy `--cache-disk-path` before it passes.
- F11 (disk prompt cache): the same-reuse-point identity check and the production-command-line (mmproj
loaded) 35B run both passed on 2026-11-21 (`/root/arms/diskcache-window3.sh`, F11 Validation), and
the cache is on in production. Remaining gap: only the 35B and gemma were checked in production;
other models gain the flag only if they use those macros.
- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's
host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time
(token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that
+25 -11
View File
@@ -184,6 +184,7 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '
git -C /root/llama.cpp fetch origin "$B"
git -C /root/llama.cpp worktree add "$W" FETCH_HEAD
cd "$W"
export HF_UI_VERSION=b11906 # production's stamp: cat /root/llama.cpp/build/tools/ui/.ui-stamp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF
nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8
'
@@ -196,8 +197,12 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '
- Do not overlap a build with a large benchmark (host OOM). Disk is tight: `/root` had about 13 GB
free (97% full) on 2026-10-25, so check `df -h /root` first. Each build tree is about 700 MB;
remove scratch worktrees you no longer need with `git -C /root/llama.cpp worktree remove <dir>`.
- The build prints `UI: download dist.tar.gz.sha256 ... failed` (no UI assets download). It is
harmless with `-DLLAMA_BUILD_UI=OFF`; the listed targets still build.
- `-DLLAMA_BUILD_UI=OFF` does NOT make the UI asset step harmless. The `llama-ui-assets` step still
runs, and when the build number has no HF tag it falls back to `latest`. On 2026-11-21 `latest`
broke the build (missing `bundle[hash].js`, `.css` and `workbox[hash].js`). Set
`HF_UI_VERSION=<stamp>` for both `cmake -B build` and `cmake --build`, where the stamp is
production's current one in `build/tools/ui/.ui-stamp` (b11906 on 2026-11-21). The validated
worktree's UI assets may then differ from the deployed ones (the code is identical).
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
holding a `kubectl exec` open for hours.
- The validated build must contain all three targets: `llama-server`, `llama-bench` and
@@ -290,7 +295,10 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window3.sh` runs `q35.py`
with production's exact 35B command line (mmproj loaded): save/restore, cold save, the
same-reuse-point identity and an image request (FORK.md F11). Not yet run; it supersedes
same-reuse-point identity and an image request (FORK.md F11). Run 2026-11-21 on adc2aa3ec: all
legs passed (18.8k-token chat, prefill 84.6 s -> turn 2 4.7 s after the restart, identical
output; save 557 MB in 1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image
answered and not saved; log `/root/arms/diskcache-test/q35-mm.log`). It supersedes
`diskcache-window2.sh` (`cmp.py` runs with `--no-mmproj`).
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
@@ -310,8 +318,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 10 (holds 6c480f7e7)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-11.txt
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 12 (holds 2fc9f9c35)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-13.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea):
@@ -364,7 +372,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
```sh
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench test-backend-ops -j 12
cd /root/llama.cpp && HF_UI_VERSION=$(cat build/tools/ui/.ui-stamp) \
cmake --build build --target llama-server llama-bench test-backend-ops -j 12
ls -lL build/bin/llama-server build/bin/libllama-server-impl.so build/bin/libllama-common.so \
build/bin/libllama.so build/bin/libmtmd.so build/bin/libggml.so build/bin/libggml-base.so \
build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
@@ -454,7 +463,7 @@ supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
cd /root/llama.cpp
git checkout -B <previous-production-branch> "$(head -1 /root/DEPLOY-ROLLBACK-commit-N.txt | cut -d' ' -f1)"
cmake --build build --target llama-server llama-bench -j 12
HF_UI_VERSION=$(cat build/tools/ui/.ui-stamp) cmake --build build --target llama-server llama-bench -j 12
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
```
@@ -469,8 +478,9 @@ chain in the exec shell: it can die with the exec session halfway, leaving llama
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
Pointer files 2 to 10 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`, and
commit-10 `6c480f7e7`, the rollback target for the current production `2fc9f9c35`. The formats vary
Pointer files 2 to 12 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, and commit-12 holds
`2fc9f9c35`, the rollback target for the current production `adc2aa3ec` (next free: 13). The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
@@ -532,8 +542,12 @@ Files (all under `/root`, a persistent volume):
- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program).
- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`).
- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers.
- `/root/prompt-cache/` (recommended, not yet configured): disk prompt cache files `*.lpcd` of every
model with `--cache-disk-path` (FORK.md F11). Safe to empty while llama-swap is stopped.
- `/root/prompt-cache/` (configured 2026-11-21, `--cache-disk-path /root/prompt-cache
--cache-disk-size 4096` in the `moe_offload_args` and `moe_offload_gemma` macros; backup
`/root/config.yaml.bak-diskcache-20261121`): disk prompt cache files `*.lpcd` of every model
with the flag (FORK.md F11). Production check: a 35B 4.6k-token chat was saved on eviction (278.6
MiB, copy 126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms) and turn 2
prefilled 32 tokens; a gemma image request works with the flags on and is not cached. Safe to empty while llama-swap is stopped.
- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/`
directory from the old design may remain). Never point a test at it.