docs: runbook/FORK.md after the adc2aa3ec deploy
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
This commit is contained in:
1 parent
92abb2d89b
commit
86bf601db6
2 files changed
+52
-32
No files matched your search
@@ -4,19 +4,20 @@ Maintainer documentation for this private fork. Read it before you rebase, chang
|
||||
anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md),
|
||||
which also holds the operations reference (pod access, files, recovery).
|
||||
|
||||
Status at the time of writing (2026-10-25):
|
||||
Status at the time of writing (2026-11-21):
|
||||
|
||||
- Branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772`. The fork commits are
|
||||
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
|
||||
(this file among them), so count it rather than trusting a number written here. Section 2 maps
|
||||
every commit to a feature.
|
||||
- Production runs `2fc9f9c35` (deployed 2026-10-25 13:01 UTC, `system_fingerprint`
|
||||
`b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the
|
||||
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
|
||||
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
|
||||
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
|
||||
top. A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); production stays on
|
||||
`2fc9f9c35`. The fix is on the branch; the 35B run with the production command line is still due.
|
||||
- Production runs `adc2aa3ec` (system_fingerprint `b11906-adc2aa3ec`; rollback pointer
|
||||
`/root/DEPLOY-ROLLBACK-commit-12.txt` = `2fc9f9c35`, the previous production; next free pointer 13),
|
||||
with the disk prompt cache ON (F11). `2fc9f9c35` was validated with the full `fork-validate.sh` set
|
||||
(37 PASS, 0 FAIL, runbook section 6). Any later commit that touches compiled code needs the same
|
||||
validation before a redeploy.
|
||||
- Branch `fable5/disk-prompt-cache` added F11 (disk prompt cache) and a CPU_REPACK release fix (F4).
|
||||
A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); the fix (adc2aa3ec) passed
|
||||
the 35B run with the production command line on 2026-11-21 and is deployed.
|
||||
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
|
||||
its subject line. The commit messages are detailed and are the primary record of each mechanism
|
||||
and measurement; read them with `git show`.
|
||||
@@ -1039,8 +1040,11 @@ Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window.
|
||||
incident above; `--cache-disk-path` on a scratch dir, arbiter env, an 18760-token pan-tadeusz chat;
|
||||
log `/root/arms/diskcache-test/q35.log`). `q35.py` now uses the exact production entry (`-hf`, so
|
||||
the mmproj loads, nothing extra). That run (`/root/arms/diskcache-window3.sh`, about 8 minutes under
|
||||
swapdown, which also covers the same-reuse-point identity and an image request on the 35B) is
|
||||
ready and not run:
|
||||
swapdown, which also covers the same-reuse-point identity and an image request on the 35B) was run
|
||||
2026-11-21 on adc2aa3ec (log `/root/arms/diskcache-test/q35-mm.log`) and passed all legs: 18.8k-token
|
||||
chat, prefill 84.6 s -> 4.7 s for turn 2 after the restart with identical output; save 557 MB in
|
||||
1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image answered and not saved. The
|
||||
table below is the earlier `--no-mmproj` run:
|
||||
|
||||
| leg | result |
|
||||
|---|---|
|
||||
@@ -1060,10 +1064,9 @@ Two anomalies in leg c, both explained:
|
||||
cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the
|
||||
GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU
|
||||
to itself: `/root/arms/diskcache-window3.sh` (production command line, reference "turn 2 again" against
|
||||
leg c; about 8 minutes under swapdown) is ready and not run. It supersedes `diskcache-window2.sh`,
|
||||
whose `cmp.py` runs with `--no-mmproj`.
|
||||
Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the
|
||||
GPU), not proven on that model.
|
||||
leg c; about 8 minutes under swapdown). It ran 2026-11-21 and passed (same-reuse-point identity:
|
||||
identical output), so the difference above is the reuse point. It supersedes
|
||||
`diskcache-window2.sh`, whose `cmp.py` runs with `--no-mmproj`.
|
||||
- Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug.
|
||||
The driver's fake waiter was `flock -s want.lock sleep 120`; killing flock left `sleep` holding
|
||||
want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right
|
||||
@@ -1072,9 +1075,12 @@ Two anomalies in leg c, both explained:
|
||||
arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The
|
||||
driver now holds the lock in the process it kills, like `fork-validate.sh` does.
|
||||
|
||||
Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache
|
||||
--cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is
|
||||
created on first start. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
|
||||
Configuration (applied in production 2026-11-21): `--cache-disk-path /root/prompt-cache
|
||||
--cache-disk-size 4096` appended to the `moe_offload_args` and `moe_offload_gemma` macros in
|
||||
`/root/config.yaml` (backup `/root/config.yaml.bak-diskcache-20261121`). The directory is created on
|
||||
first start. Verified in production: a 35B 4.6k-token chat was saved on eviction (278.6 MiB, copy
|
||||
126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms), and turn 2 prefilled
|
||||
only 32 tokens; a gemma image request with the flags on works and is not cached. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
|
||||
applies. Models only pick up the flags when they restart.
|
||||
|
||||
---
|
||||
@@ -1242,10 +1248,10 @@ Machine-level:
|
||||
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
|
||||
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
|
||||
is an estimate; its memory effect is verified in production.
|
||||
- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the
|
||||
35B, only on the 0.8B (CPU and GPU), and no 35B run has used production's command line (mmproj
|
||||
loaded) yet. `/root/arms/diskcache-window3.sh` covers both (about 8 minutes of downtime). Do not
|
||||
deploy `--cache-disk-path` before it passes.
|
||||
- F11 (disk prompt cache): the same-reuse-point identity check and the production-command-line (mmproj
|
||||
loaded) 35B run both passed on 2026-11-21 (`/root/arms/diskcache-window3.sh`, F11 Validation), and
|
||||
the cache is on in production. Remaining gap: only the 35B and gemma were checked in production;
|
||||
other models gain the flag only if they use those macros.
|
||||
- A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's
|
||||
host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time
|
||||
(token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that
|
||||
|
||||
+25
-11
@@ -184,6 +184,7 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '
|
||||
git -C /root/llama.cpp fetch origin "$B"
|
||||
git -C /root/llama.cpp worktree add "$W" FETCH_HEAD
|
||||
cd "$W"
|
||||
export HF_UI_VERSION=b11906 # production's stamp: cat /root/llama.cpp/build/tools/ui/.ui-stamp
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF
|
||||
nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8
|
||||
'
|
||||
@@ -196,8 +197,12 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '
|
||||
- Do not overlap a build with a large benchmark (host OOM). Disk is tight: `/root` had about 13 GB
|
||||
free (97% full) on 2026-10-25, so check `df -h /root` first. Each build tree is about 700 MB;
|
||||
remove scratch worktrees you no longer need with `git -C /root/llama.cpp worktree remove <dir>`.
|
||||
- The build prints `UI: download dist.tar.gz.sha256 ... failed` (no UI assets download). It is
|
||||
harmless with `-DLLAMA_BUILD_UI=OFF`; the listed targets still build.
|
||||
- `-DLLAMA_BUILD_UI=OFF` does NOT make the UI asset step harmless. The `llama-ui-assets` step still
|
||||
runs, and when the build number has no HF tag it falls back to `latest`. On 2026-11-21 `latest`
|
||||
broke the build (missing `bundle[hash].js`, `.css` and `workbox[hash].js`). Set
|
||||
`HF_UI_VERSION=<stamp>` for both `cmake -B build` and `cmake --build`, where the stamp is
|
||||
production's current one in `build/tools/ui/.ui-stamp` (b11906 on 2026-11-21). The validated
|
||||
worktree's UI assets may then differ from the deployed ones (the code is identical).
|
||||
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
|
||||
holding a `kubectl exec` open for hours.
|
||||
- The validated build must contain all three targets: `llama-server`, `llama-bench` and
|
||||
@@ -290,7 +295,10 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
|
||||
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
|
||||
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window3.sh` runs `q35.py`
|
||||
with production's exact 35B command line (mmproj loaded): save/restore, cold save, the
|
||||
same-reuse-point identity and an image request (FORK.md F11). Not yet run; it supersedes
|
||||
same-reuse-point identity and an image request (FORK.md F11). Run 2026-11-21 on adc2aa3ec: all
|
||||
legs passed (18.8k-token chat, prefill 84.6 s -> turn 2 4.7 s after the restart, identical
|
||||
output; save 557 MB in 1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image
|
||||
answered and not saved; log `/root/arms/diskcache-test/q35-mm.log`). It supersedes
|
||||
`diskcache-window2.sh` (`cmp.py` runs with `--no-mmproj`).
|
||||
|
||||
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
|
||||
@@ -310,8 +318,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
|
||||
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
|
||||
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
|
||||
```sh
|
||||
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 10 (holds 6c480f7e7)
|
||||
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-11.txt
|
||||
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 12 (holds 2fc9f9c35)
|
||||
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-13.txt
|
||||
git -C /root/llama.cpp status --short # must be clean
|
||||
```
|
||||
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea):
|
||||
@@ -364,7 +372,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
|
||||
6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
|
||||
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
|
||||
```sh
|
||||
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench test-backend-ops -j 12
|
||||
cd /root/llama.cpp && HF_UI_VERSION=$(cat build/tools/ui/.ui-stamp) \
|
||||
cmake --build build --target llama-server llama-bench test-backend-ops -j 12
|
||||
ls -lL build/bin/llama-server build/bin/libllama-server-impl.so build/bin/libllama-common.so \
|
||||
build/bin/libllama.so build/bin/libmtmd.so build/bin/libggml.so build/bin/libggml-base.so \
|
||||
build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
|
||||
@@ -454,7 +463,7 @@ supervisorctl -c /root/supervisord.conf stop llama-swap
|
||||
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
|
||||
cd /root/llama.cpp
|
||||
git checkout -B <previous-production-branch> "$(head -1 /root/DEPLOY-ROLLBACK-commit-N.txt | cut -d' ' -f1)"
|
||||
cmake --build build --target llama-server llama-bench -j 12
|
||||
HF_UI_VERSION=$(cat build/tools/ui/.ui-stamp) cmake --build build --target llama-server llama-bench -j 12
|
||||
supervisorctl -c /root/supervisord.conf start llama-swap
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
|
||||
```
|
||||
@@ -469,8 +478,9 @@ chain in the exec shell: it can die with the exec session halfway, leaving llama
|
||||
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
|
||||
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
|
||||
|
||||
Pointer files 2 to 10 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`, and
|
||||
commit-10 `6c480f7e7`, the rollback target for the current production `2fc9f9c35`. The formats vary
|
||||
Pointer files 2 to 12 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
|
||||
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, and commit-12 holds
|
||||
`2fc9f9c35`, the rollback target for the current production `adc2aa3ec` (next free: 13). The formats vary
|
||||
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
|
||||
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
|
||||
|
||||
@@ -532,8 +542,12 @@ Files (all under `/root`, a persistent volume):
|
||||
- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program).
|
||||
- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`).
|
||||
- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers.
|
||||
- `/root/prompt-cache/` (recommended, not yet configured): disk prompt cache files `*.lpcd` of every
|
||||
model with `--cache-disk-path` (FORK.md F11). Safe to empty while llama-swap is stopped.
|
||||
- `/root/prompt-cache/` (configured 2026-11-21, `--cache-disk-path /root/prompt-cache
|
||||
--cache-disk-size 4096` in the `moe_offload_args` and `moe_offload_gemma` macros; backup
|
||||
`/root/config.yaml.bak-diskcache-20261121`): disk prompt cache files `*.lpcd` of every model
|
||||
with the flag (FORK.md F11). Production check: a 35B 4.6k-token chat was saved on eviction (278.6
|
||||
MiB, copy 126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms) and turn 2
|
||||
prefilled 32 tokens; a gemma image request works with the flags on and is not cached. Safe to empty while llama-swap is stopped.
|
||||
- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/`
|
||||
directory from the old design may remain). Never point a test at it.
|
||||
|
||||
|
||||
Reference in new issue
Block a user