docs: runbook - new fork-validate checks, gdb gone, scratch lost on restart
Phase 1 now covers the VRAM release/restore check and the handoff latency A/B against --ref (which must be the production build); phase 2 adds --moe-exact on the 35B and keeps its scratch dir. Explains median gating and --perf-report, that a pod restart empties /tmp, that the manual checks are now automated, and that gdb is not in the pod image (lost in the 2026-10-24 restart). States the commit trailer convention and what llama-swap does with a child that exits vs one that wedges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
47b892271c
commit
d7b3510499
1 file changed
+32
-14
+32
-14
@@ -82,7 +82,8 @@ git cherry-pick <first-fork-commit> # one at a time, in the original order
|
||||
|
||||
Rules:
|
||||
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
|
||||
trailer the owner asks for.
|
||||
fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block),
|
||||
unless the owner asks for something else.
|
||||
- Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms.
|
||||
- Fold each fix into the commit it belongs to so every commit builds on its own:
|
||||
`git commit --fixup=<sha>` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`.
|
||||
@@ -199,9 +200,10 @@ bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
|
||||
|
||||
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
|
||||
cat > /root/arms/fv-perf.sh <<'EOF'
|
||||
Q35=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
|
||||
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
|
||||
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf \
|
||||
--moecache /root/.cache/huggingface/hub/<gemma-4-26B-A4B snapshot>/<file>.gguf
|
||||
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf --moe-exact "$Q35" \
|
||||
--moecache /root/.cache/huggingface/hub/<gemma-4-26B-A4B snapshot>/<file>.gguf --keep
|
||||
EOF
|
||||
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
|
||||
```
|
||||
@@ -213,14 +215,26 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
|
||||
e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`).
|
||||
- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits,
|
||||
which proves nothing).
|
||||
- Phase 2 costs about 27 minutes of downtime and leaves tens of GB of model files in page cache.
|
||||
- Last full run (2026-10-24, 33 PASS, 0 FAIL): perf candidate vs reference pp2048 -4.1%, -2.2% at d16k,
|
||||
-0.7% at d32k (gate is -5%), tg256@d4096 -0.2%.
|
||||
- Manual checks the script does not do: VRAM actually drops when a server goes cold
|
||||
(`cat /sys/class/drm/card*/device/mem_info_vram_used` before and after); greedy output of the 35B
|
||||
against the reference if `sched_copy_experts` changed.
|
||||
- Any arbiter change: A/B the ordinary handoff latency against the current binary, not only the
|
||||
pathological case. Two deploys were lost by skipping that.
|
||||
- Phase 1 now also checks that a cold server really frees its VRAM (`v-vram-release`,
|
||||
`v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs
|
||||
ordinary handoff latency against `--ref` (`a-handoff-latency`, about 1.5 extra minutes). `--ref`
|
||||
must be the build currently in production, so both comparisons are against what serves today.
|
||||
- Phase 2 costs about 27 minutes of downtime for `--perf` plus `--moecache`, and `--moe-exact` adds
|
||||
three loads of the 35B (estimated 3-5 minutes; not yet timed). It leaves tens of GB of model files
|
||||
in page cache. `--moe-exact` proves the routing-ids readback skip exact (F1): the candidate's
|
||||
greedy output equals the reference's and, with the `LLAMA_MOE_IDS_SKIP` knob (any build of this
|
||||
branch), the candidate's with the skip forced off.
|
||||
- `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in
|
||||
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
|
||||
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
|
||||
scratch dirs were lost that way).
|
||||
- Last full run (2026-10-24, 33 PASS, 0 FAIL; old script, means of 2 arm means, no `--moe-exact`):
|
||||
perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, -0.7% at d32k (gate is -5%),
|
||||
tg256@d4096 -0.2%. The new phase-1 checks first ran on 2026-10-25 (reb1024 against itself beside
|
||||
production): 16 PASS; VRAM warm +546 / cold +4 / restored +546 MiB; handoff median 1050.5 vs 1082.5 ms.
|
||||
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
|
||||
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
|
||||
regression gate, not a measurement.
|
||||
|
||||
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
|
||||
|
||||
@@ -316,8 +330,11 @@ holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE
|
||||
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged.
|
||||
|
||||
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
|
||||
shows a process holding WRITE on `token.lock` that is not generating. Inspect it with
|
||||
`gdb -p <pid> -batch -ex 'thread apply all bt'` (gdb is installed in the pod) before killing it.
|
||||
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
|
||||
that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with
|
||||
`gdb -p <pid> -batch -ex 'thread apply all bt'` before killing it. gdb is not in
|
||||
the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check
|
||||
`command -v gdb` and reinstall it first if needed.
|
||||
|
||||
## 9. Rollback
|
||||
|
||||
@@ -406,7 +423,8 @@ Useful commands:
|
||||
- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`.
|
||||
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
|
||||
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
|
||||
- The pod has `curl`, `python3`, `gdb`, `flock`; no `jq`.
|
||||
- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the
|
||||
last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log).
|
||||
|
||||
Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2).
|
||||
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do
|
||||
|
||||
Reference in new issue
Block a user