diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md index 6baab37b3..108b395cc 100644 --- a/docs/fork-rebase-runbook.md +++ b/docs/fork-rebase-runbook.md @@ -82,7 +82,8 @@ git cherry-pick # one at a time, in the original order Rules: - Cherry-pick in the original order, keep authors and messages. New commits you author get the - trailer the owner asks for. + fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block), + unless the owner asks for something else. - Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms. - Fold each fix into the commit it belongs to so every commit builds on its own: `git commit --fixup=` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`. @@ -199,9 +200,10 @@ bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \ # phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap cat > /root/arms/fv-perf.sh <<'EOF' +Q35=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \ - --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf \ - --moecache /root/.cache/huggingface/hub//.gguf + --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf --moe-exact "$Q35" \ + --moecache /root/.cache/huggingface/hub//.gguf --keep EOF setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 /root/arms/f e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`). - `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits, which proves nothing). -- Phase 2 costs about 27 minutes of downtime and leaves tens of GB of model files in page cache. -- Last full run (2026-10-24, 33 PASS, 0 FAIL): perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, - -0.7% at d32k (gate is -5%), tg256@d4096 -0.2%. -- Manual checks the script does not do: VRAM actually drops when a server goes cold - (`cat /sys/class/drm/card*/device/mem_info_vram_used` before and after); greedy output of the 35B - against the reference if `sched_copy_experts` changed. -- Any arbiter change: A/B the ordinary handoff latency against the current binary, not only the - pathological case. Two deploys were lost by skipping that. +- Phase 1 now also checks that a cold server really frees its VRAM (`v-vram-release`, + `v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs + ordinary handoff latency against `--ref` (`a-handoff-latency`, about 1.5 extra minutes). `--ref` + must be the build currently in production, so both comparisons are against what serves today. +- Phase 2 costs about 27 minutes of downtime for `--perf` plus `--moecache`, and `--moe-exact` adds + three loads of the 35B (estimated 3-5 minutes; not yet timed). It leaves tens of GB of model files + in page cache. `--moe-exact` proves the routing-ids readback skip exact (F1): the candidate's + greedy output equals the reference's and, with the `LLAMA_MOE_IDS_SKIP` knob (any build of this + branch), the candidate's with the skip forced off. +- `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in + `/perf.tsv`, and `fork-validate.sh --perf-report ` re-aggregates them offline. A pod + restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24 + scratch dirs were lost that way). +- Last full run (2026-10-24, 33 PASS, 0 FAIL; old script, means of 2 arm means, no `--moe-exact`): + perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, -0.7% at d32k (gate is -5%), + tg256@d4096 -0.2%. The new phase-1 checks first ran on 2026-10-25 (reb1024 against itself beside + production): 16 PASS; VRAM warm +546 / cold +4 / restored +546 MiB; handoff median 1050.5 vs 1082.5 ms. +- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed + is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a + regression gate, not a measurement. On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs. @@ -316,8 +330,11 @@ holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE 7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged. A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks` -shows a process holding WRITE on `token.lock` that is not generating. Inspect it with -`gdb -p -batch -ex 'thread apply all bt'` (gdb is installed in the pod) before killing it. +shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child +that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with +`gdb -p -batch -ex 'thread apply all bt'` before killing it. gdb is not in +the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check +`command -v gdb` and reinstall it first if needed. ## 9. Rollback @@ -406,7 +423,8 @@ Useful commands: - VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`. - Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock` (WRITE = holder on token.lock, READ = registered waiters on want.lock). -- The pod has `curl`, `python3`, `gdb`, `flock`; no `jq`. +- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the + last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log). Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2). Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do