docs: runbook - new fork-validate checks, gdb gone, scratch lost on restart

Phase 1 now covers the VRAM release/restore check and the handoff latency
A/B against --ref (which must be the production build); phase 2 adds
--moe-exact on the 35B and keeps its scratch dir. Explains median gating and
--perf-report, that a pod restart empties /tmp, that the manual checks are
now automated, and that gdb is not in the pod image (lost in the 2026-10-24
restart). States the commit trailer convention and what llama-swap does with
a child that exits vs one that wedges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 02:52:32 +02:00
1 parent 47b892271c
commit d7b3510499
1 file changed
+32 -14
+32 -14
View File
@@ -82,7 +82,8 @@ git cherry-pick <first-fork-commit> # one at a time, in the original order
Rules:
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
trailer the owner asks for.
fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block),
unless the owner asks for something else.
- Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms.
- Fold each fix into the commit it belongs to so every commit builds on its own:
`git commit --fixup=<sha>` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`.
@@ -199,9 +200,10 @@ bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
cat > /root/arms/fv-perf.sh <<'EOF'
Q35=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf \
--moecache /root/.cache/huggingface/hub/<gemma-4-26B-A4B snapshot>/<file>.gguf
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf --moe-exact "$Q35" \
--moecache /root/.cache/huggingface/hub/<gemma-4-26B-A4B snapshot>/<file>.gguf --keep
EOF
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
```
@@ -213,14 +215,26 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`).
- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits,
which proves nothing).
- Phase 2 costs about 27 minutes of downtime and leaves tens of GB of model files in page cache.
- Last full run (2026-10-24, 33 PASS, 0 FAIL): perf candidate vs reference pp2048 -4.1%, -2.2% at d16k,
-0.7% at d32k (gate is -5%), tg256@d4096 -0.2%.
- Manual checks the script does not do: VRAM actually drops when a server goes cold
(`cat /sys/class/drm/card*/device/mem_info_vram_used` before and after); greedy output of the 35B
against the reference if `sched_copy_experts` changed.
- Any arbiter change: A/B the ordinary handoff latency against the current binary, not only the
pathological case. Two deploys were lost by skipping that.
- Phase 1 now also checks that a cold server really frees its VRAM (`v-vram-release`,
`v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs
ordinary handoff latency against `--ref` (`a-handoff-latency`, about 1.5 extra minutes). `--ref`
must be the build currently in production, so both comparisons are against what serves today.
- Phase 2 costs about 27 minutes of downtime for `--perf` plus `--moecache`, and `--moe-exact` adds
three loads of the 35B (estimated 3-5 minutes; not yet timed). It leaves tens of GB of model files
in page cache. `--moe-exact` proves the routing-ids readback skip exact (F1): the candidate's
greedy output equals the reference's and, with the `LLAMA_MOE_IDS_SKIP` knob (any build of this
branch), the candidate's with the skip forced off.
- `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- Last full run (2026-10-24, 33 PASS, 0 FAIL; old script, means of 2 arm means, no `--moe-exact`):
perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, -0.7% at d32k (gate is -5%),
tg256@d4096 -0.2%. The new phase-1 checks first ran on 2026-10-25 (reb1024 against itself beside
production): 16 PASS; VRAM warm +546 / cold +4 / restored +546 MiB; handoff median 1050.5 vs 1082.5 ms.
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
regression gate, not a measurement.
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
@@ -316,8 +330,11 @@ holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged.
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
shows a process holding WRITE on `token.lock` that is not generating. Inspect it with
`gdb -p <pid> -batch -ex 'thread apply all bt'` (gdb is installed in the pod) before killing it.
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with
`gdb -p <pid> -batch -ex 'thread apply all bt'` before killing it. gdb is not in
the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check
`command -v gdb` and reinstall it first if needed.
## 9. Rollback
@@ -406,7 +423,8 @@ Useful commands:
- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`.
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
- The pod has `curl`, `python3`, `gdb`, `flock`; no `jq`.
- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the
last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log).
Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2).
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do