diff --git a/FORK.md b/FORK.md index d419f7452..82b710e60 100644 --- a/FORK.md +++ b/FORK.md @@ -10,10 +10,10 @@ Status at the time of writing (2026-10-25): `git log --oneline origin/master..HEAD`; that list includes documentation and script commits (this file among them), so count it rather than trusting a number written here. Section 2 maps every commit to a feature. -- Production runs `6c480f7e7`. Everything after it on the branch is validation tooling, env-gated - A/B knobs, comment fixes, documentation and one loader behaviour change (F10, the page-cache - drop); none of it was deployed. The knobs and the comment - fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation. +- Production runs `2fc9f9c35` (deployed 2026-10-25 13:01 UTC, `system_fingerprint` + `b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the + previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook + section 6). Any later commit that touches compiled code needs the same validation before a redeploy. - Commit hashes below refer to that branch. They change on every rebase; find a commit again by its subject line. The commit messages are detailed and are the primary record of each mechanism and measurement; read them with `git show`. @@ -118,7 +118,8 @@ Oldest first. "Feature" points at the section below. | cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 | Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code. -Everything from `e085c347d` down is after the production commit `6c480f7e7`. +Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it); +`6c480f7e7` was the previous production commit. --- @@ -176,8 +177,9 @@ compares 64 greedy tokens after a prompt of about twice that, every expert on th candidate == reference (`identical-ref`), and with the `LLAMA_MOE_IDS_SKIP` knob (f664b0f20; `=0` forces the readback, `=log` reports each skip on stderr) candidate == candidate with the skip forced off (`identical-noskip`, which proves the skip exact even when the reference build carries -it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). Not yet run on the GPU -(added after the 20261024 deploy; only dry-run against stub servers). Also `--ops` (the MUL_MAT_ID +it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). First GPU run +2026-10-25 (35B, 2fc9f9c35 vs 6c480f7e7): prompt 397 tokens, 39 skips logged, 0 with the skip forced +off, output identical in all three runs, 78 s. Also `--ops` (the MUL_MAT_ID kernel, not this callback), `--perf` (prefill regression) and `--moecache` (the cache path through this callback). @@ -648,7 +650,8 @@ first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/serv eagerly (no new process in the 2 s after the kill), and starts a new process on the next request for that model (new PID, health check, HTTP 200). The production log since the last pod restart has no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers - the restart pays a full model load (about 26 s for the 35B with `-lm none`), and under the arbiter + the restart pays a full model load (about 26 s for the 35B with `-lm none`; the first 35B request + after the 2026-10-25 deploy took 34 s including load and answer), and under the arbiter the new process takes the token through `vram_acquire_for_load()` like any load. A process that wedges instead of exiting (the incidents below) is never restarted: `/health` keeps answering. @@ -683,8 +686,9 @@ fixed deadline: keep the order "check `!running`, return null, then `wait_until( **Validation.** `fork-validate.sh --arbiter`: `d-term-in-release` (gdb breakpoint on `llama_context_release_device`, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and the token), `e-term-waiting` (exit 0 within 1.5 s, never took the token), `f-term-streaming` -(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run: -736 ms, 200 ms, 93 ms (token free in 28 ms), 80/157/156 ms. Post-deploy: a forced llama-swap eviction +(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run +(2026-10-25, 2fc9f9c35, `d` by log polling): 750 ms, 225 ms, 126 ms (token free in 23 ms), +108/159/159 ms. Post-deploy: a forced llama-swap eviction must log `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token` and exit by itself. ### F8. Cold-teardown guards @@ -744,14 +748,17 @@ required, so a rebase can be validated by someone with no context. inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline. See the header of the script for what each check proves. -- Which phase needs what: `--ops --concat --arbiter` run beside production (`--arbiter` refuses - below 1.5 GiB free VRAM); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the +- Which phase needs what: `--ops --concat` run beside production; `--arbiter` too only if 1536 MiB + of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs + in the swapdown window, about 2 minutes); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share the card either). **Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24, production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s, -pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. The corpus +pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. On 2026-10-25 +(medians, `6c480f7e7` / `2fc9f9c35`): pp2048 274.5/273.8, d16384 200.5/200.4, d32768 157.6/157.6, +tg256@d4096 16.8/16.7. The corpus harness and older numbers are in `scripts/rx580-bench/README.md`. A fresh run far outside these means something else is wrong (another GPU user, thermal state, a bad build) before it means a regression or a win. @@ -778,11 +785,13 @@ script greps for (`server entering sleeping state`, `transpose fast path, transp markdown layout; the GGUF keys `.expert_count` / `.expert_used_count`; the DRM fdinfo keys `drm-client-id` / `drm-memory-vram`. -**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test (one candidate -arm once showed an unexplained 1.6 GiB spike that did not repeat); the SIGTERM-in-release breakpoint -is at function entry, not after the host-shadow capture; `--moe-exact` and the median-gated `--perf` -have not run on the GPU yet (tested offline only, see the runbook); the `v-vram-release` SKIP path -(production moving during the measurement) has not been exercised. +**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test. The first +candidate arm has now twice shown an unexplained spike of about 1.6 GiB over the other arms +(2026-10-25: ref 5476, cand 7062, cand 5767, ref 5476 MiB); the cause is open (per-test peaks would +locate it), and it stays far below the 95% gate. The SIGTERM-in-release breakpoint is at function +entry, not after the host-shadow capture. The `v-vram-release` SKIP path (production moving during +the measurement) has not been exercised. `--moe-exact` and the median-gated `--perf` first ran on the +GPU on 2026-10-25 (runbook section 6). ### F10. Page-cache drop after a copying load @@ -822,7 +831,9 @@ after the tensor reads. Commit cce0ac1f0. load: the VRAM release/restore (F4) uses host shadows, not the file. - The drop is a hint: a failure must not fail the load. -**Failure modes and incidents.** None yet (not deployed). Costs to know about: +**Failure modes and incidents.** None. Deployed 2026-10-25 (`2fc9f9c35`); on a 35B GPU load in +production the node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB. Costs to know +about: - Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a 5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few seconds on a 26 s load. @@ -858,9 +869,9 @@ Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it count available. Greedy output byte-identical in all arms. `-lm mmap` control: the model file stays fully resident and mapped (2498 MiB, `file_mapped` +2.5 GiB) after load and after a request, and only the copied mmproj is dropped. `-lzm on` control: no drop (the file is mapped), 1252 MiB of copied -tensors stay cached. Not run: the GPU upload path (the per-tensor drop sits after all three copy -paths, so it is covered by construction, but the pinned async upload was not exercised: the card had -only 1.2 GiB free beside production), and a 35B load. +tensors stay cached. The GPU upload path and a 35B load were first exercised by the 2026-10-25 +deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B +request took 34 s including load and answer). --- @@ -1020,14 +1031,9 @@ Machine-level: null buffer would crash. Only the mmproj path handles a failed restore. - Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4). - Router mode can spin at shutdown (F7); not used here. -- The stale arbiter comments (KV "never evicted", encoder "rebuilt by `vram_ensure_warm()`", lazy - wake, `process_mtmd_chunk`) and the "keeping KV cache" sleep log line were fixed in f5c0d8f28, - after the deployed build: production still logs "keeping KV cache" while it evicts the KV. -- The new checks (`--moe-exact`, `a-handoff-latency`, `v-vram-*`, median `--perf`) and the - `LLAMA_MOE_IDS_SKIP` knob were written after the 20261024 deploy; the knob is syntax-checked only - (not built), and only `--arbiter` has run on the GPU. Run the full set at the next rebase. +- The unexplained ~1.6 GiB peak-VRAM spike in the first candidate `--perf` arm, seen twice (F9). - Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context. Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them. -- F10 (page-cache drop) has not run on the GPU upload path or on a 35B load; its load-time cost there - (synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate. +- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model) + is an estimate; its memory effect is verified in production. diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md index b4e586830..30e2ebc2e 100644 --- a/docs/fork-rebase-runbook.md +++ b/docs/fork-rebase-runbook.md @@ -18,6 +18,17 @@ Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a co the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU; 11. Operations reference. +**Redeploying a new tip on the same upstream base (no rebase)**, e.g. new fork commits on the +production branch (2026-10-25: `6c480f7e7` -> `2fc9f9c35`, same branch, same base): +- Skip sections 1-3. Of section 4 only the ASCII check (on `git diff ..`) and a careful + read of that diff apply; `` is `git -C /root/llama.cpp rev-parse HEAD` on the pod. There is + nothing to range-diff. +- Do sections 5-9 as written: build the tip in a scratch worktree, validate it with `--ref` = the + production build, then deploy. +- The branch name stays the same, so deploy step 2's `git checkout -B ` moves the + production branch on the pod to the tip. After that the rollback pointer file is the only record + of the old production commit on the pod, so write it first (deploy step 1). + --- ## 1. Survey upstream @@ -182,9 +193,11 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c ' If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed). - `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes. - Report compile errors verbatim against the list from step 4. -- Do not overlap a build with a large benchmark (host OOM). Disk is limited (about 22 GB free; each - build tree is about 700 MB); remove scratch worktrees you no longer need with - `git -C /root/llama.cpp worktree remove `. +- Do not overlap a build with a large benchmark (host OOM). Disk is tight: `/root` had about 13 GB + free (97% full) on 2026-10-25, so check `df -h /root` first. Each build tree is about 700 MB; + remove scratch worktrees you no longer need with `git -C /root/llama.cpp worktree remove `. +- The build prints `UI: download dist.tar.gz.sha256 ... failed` (no UI assets download). It is + harmless with `-DLLAMA_BUILD_UI=OFF`; the listed targets still build. - Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 = 1.5 GiB VRAM free for --arbiter; it refuses otherwise) +# phase 1, beside production bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \ - --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat --arbiter + --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat +``` +Add `--arbiter` to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus the bytes in +`cat /sys/class/drm/card*/device/mem_info_vram_used`); below that it FAILs at once. With gemma-26B +warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run +`--arbiter` in the phase-2 swapdown window instead (about 2 minutes more downtime), as below. + +**Page cache before and after phase 2.** The `--ref` build may lack F10 (FORK.md), and `--perf` +loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the +eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check +`/proc/meminfo`. + +```sh # phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap cat > /root/arms/fv-perf.sh <<'EOF' -Q35=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf +HUB=/root/.cache/huggingface/hub +Q35=$HUB/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf +G26=$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/snapshots/e6759b29958180a998f62f991e4c741fe9b4d452/gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \ - --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf --moe-exact "$Q35" \ - --moecache /root/.cache/huggingface/hub//.gguf --keep + --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build \ + --arbiter --perf --moe-exact "$Q35" --moecache "$G26" --keep EOF setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 ` (pod-only helper, not in the repo) stops llama-swap, waits for an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints `SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200. @@ -215,23 +245,25 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`). - `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits, which proves nothing). -- Phase 1 now also checks that a cold server really frees its VRAM (`v-vram-release`, +- `--arbiter` also checks that a cold server really frees its VRAM (`v-vram-release`, `v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs - ordinary handoff latency against `--ref` (`a-handoff-latency`, about 1.5 extra minutes). `--ref` - must be the build currently in production, so both comparisons are against what serves today. -- Phase 2 costs about 27 minutes of downtime for `--perf` plus `--moecache`, and `--moe-exact` adds - three loads of the 35B (estimated 3-5 minutes; not yet timed). It leaves tens of GB of model files - in page cache. `--moe-exact` proves the routing-ids readback skip exact (F1): the candidate's - greedy output equals the reference's and, with the `LLAMA_MOE_IDS_SKIP` knob (any build of this - branch), the candidate's with the skip forced off. + ordinary handoff latency against `--ref` (`a-handoff-latency`). `--ref` must be the build currently + in production, so both comparisons are against what serves today. +- Phase 2 downtime, measured 2026-10-25: about 32 minutes in all (`--arbiter` about 2 min, `--perf` + about 28 min, `--moecache` under 30 s, `--moe-exact` 78 s). `--moe-exact` proves the routing-ids + readback skip exact (F1): the candidate's greedy output equals the reference's and, with the + `LLAMA_MOE_IDS_SKIP` knob (any build of this branch), the candidate's with the skip forced off. - `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in `/perf.tsv`, and `fork-validate.sh --perf-report ` re-aggregates them offline. A pod restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24 scratch dirs were lost that way). -- Last full run (2026-10-24, 33 PASS, 0 FAIL; old script, means of 2 arm means, no `--moe-exact`): - perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, -0.7% at d32k (gate is -5%), - tg256@d4096 -0.2%. The new phase-1 checks first ran on 2026-10-25 (reb1024 against itself beside - production): 16 PASS; VRAM warm +546 / cold +4 / restored +546 MiB; handoff median 1050.5 vs 1082.5 ms. +- Last full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs + `/root/arms/dep-2fc9f9c35-fv1.log` and `-fv2.log`): phase 1 `--ops --concat` 8 PASS, phase 2 + `--arbiter --perf --moe-exact --moecache` 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048 + 274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff + median ref 693.5 / cand 678 ms. `--moe-exact`: 39 skips logged, identical to the reference and to + the skip forced off. Peak VRAM per arm ref 5476, cand 7062, cand 5767, ref 5476 MiB: the ~1.6 GiB + spike in the first candidate arm has now been seen twice (open question, FORK.md F9). - Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a regression gate, not a measurement. @@ -240,12 +272,17 @@ On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept w ## 7. Deploy -Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 minutes. +Deploy exactly the validated commit. Downtime: 6 minutes for the 2026-10-24 redeploy. The deploy +rebuilds the production tree in place (step 6) from the same commit and with the same cmake flags as +the validated worktree build, so nothing is copied over; compare +`grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt` in both +trees if in doubt. -1. **Rollback pointer.** Record the current production commit in the next free pointer file: +1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the + full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9): ```sh - ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 9 (holds 5751c977b) - git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-10.txt + ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 10 (holds 6c480f7e7) + git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-11.txt git -C /root/llama.cpp status --short # must be clean ``` 2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea): @@ -254,13 +291,14 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD ``` -3. **Memory.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM victim. With - `-lm none` each load reads the GGUF through the page cache, and a recent benchmark leaves tens of GB - of model files cached. During the first 2026-10-24 deploy the Talos OOM controller killed the whole - pod 17 s into verification (cause likely, not proven, memory pressure). Before stopping llama-swap, - evict the model files' page cache (unprivileged, safe while servers run: `-lm none` servers hold - their weights in anonymous memory, and pages mapped by an mmap model are not dropped) and check - memory: + On a redeploy with the same branch name this moves the production branch to the new commit. +3. **Evict the page cache.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM + victim, and benchmarks, mmap loads and builds without F10 leave tens of GB of model files cached. + During the first 2026-10-24 deploy the Talos OOM controller killed the whole pod 17 s into + verification (cause likely, not proven, memory pressure). Evict while production still runs + (unprivileged and safe: `-lm none` servers hold their weights in anonymous memory, and pages + mapped by an mmap model are not dropped). The same snippet is used before and after validation + phase 2: ```sh python3 - <<'EOF' import glob, os @@ -275,14 +313,11 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 os.close(fd) print('evicted', n, 'files') EOF - grep -E 'MemAvailable|MemFree' /proc/meminfo ``` - A build with F10 (FORK.md) drops the cache of every `-lm none` load by itself, but files cached - by mmap loads, benchmarks or older builds still need this step. - Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by - models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache - from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the - container is the HOST root cgroup; the pod's own cgroup is under + F10 (FORK.md) drops the cache of every `-lm none` load by itself (verified in production on a 35B + GPU load: node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB), but files cached + by mmap loads, benchmarks or older builds still need this step. `/sys/fs/cgroup/memory.stat` + inside the container is the HOST root cgroup; the pod's own cgroup is under `/sys/fs/cgroup/kubepods/besteffort/pod/`. 4. **Stop llama-swap and reap orphans.** ```sh @@ -292,19 +327,29 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 ps -eo pid,args | grep '[l]lama-server' # must be empty ``` `supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1), - one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM. -5. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's + one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM. Keep + this check even when it finds nothing (2026-10-25: no orphans). +5. **Check memory**, now that the running models are gone: `grep -E 'MemAvailable|MemFree' + /proc/meminfo`. Proceed only with MemAvailable >= 25 GB. In 2026-10-24 the eviction took node-wide + file cache from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. +6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's `LD_LIBRARY_PATH` points there, so do not rebuild it under running servers): ```sh cd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12 + ls -lL build/bin/llama-server build/bin/libllama.so build/bin/libmtmd.so \ + build/bin/libggml.so build/bin/libggml-base.so build/bin/libggml-cpu.so build/bin/libggml-vulkan.so ``` - Check that the libraries in `build/bin` have fresh mtimes. -6. **Start and probe.** + All must have fresh mtimes. Use `-L` (follow the symlinks): `build/bin` keeps every old versioned + `libggml-*.so.0.x` and `libmtmd.so.0.x`, so a plain `ls` lists stale files beside the new ones. +7. **Start and probe.** ```sh supervisorctl -c /root/supervisord.conf start llama-swap curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200 ``` - Sample `/proc/meminfo` every few seconds during the first model loads. + Sample `/proc/meminfo` every few seconds during the first model loads. Confirm the running commit + from the first chat response (section 8): `system_fingerprint` is `b-` + (2026-10-25: `b11906-2fc9f9c35`). The first 35B request after that deploy took 34 s including + the model load and the answer. ## 8. Post-deploy checks @@ -314,22 +359,42 @@ All through llama-swap on :8080. A request helper: ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \ -d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; } holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; } +# image request body: a 64x64 solid red binary PPM, no image tools needed (like fork-validate.sh) +img() { python3 - "$1" <<'EOF' +import base64, json, sys +ppm = b'P6\n64 64\n255\n' + b'\xff\x00\x00' * (64 * 64) +url = 'data:image/x-portable-pixmap;base64,' + base64.b64encode(ppm).decode() +print(json.dumps({'model': sys.argv[1], 'max_tokens': 32, 'temperature': 0, 'messages': [{'role': 'user', + 'content': [{'type': 'image_url', 'image_url': {'url': url}}, + {'type': 'text', 'text': 'What is the main colour of this image? One word.'}]}]})) +EOF +} +# img MODEL | curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @- ``` +Use the `-nothink` aliases with these `max_tokens: 32` helpers. A thinking alias spends the 32 tokens +on `reasoning_content` and returns an empty `content`; for one, raise `max_tokens` to >= 512. + 1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers; `holder` shows the 35B's PID (check with `ps -o args -p `). 2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch (2026-10-24: 3.1-5.6 s). -3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF:UD-Q4_K_XL` (a small - solid-colour image, ask for the colour). This also forces an eviction in the llama-swap matrix: the +3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF-nothink:UD-Q4_K_XL` + with the `img` helper; the answer names red. This also forces an eviction in the llama-swap matrix: the evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token`, with no kill escalation. 4. Text to gemma and to the 4B after the eviction; `holder` correct each time. 5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931). 6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for `GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none. -7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged. +7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`). No pod restart and no OOM kill: + - Restarts, from the workstation: `kubectl get pods -n llama | grep supervisord`; RESTARTS of the + Running pod must equal the value you noted before the deploy. A stale `supervisord-...-pbmjq` + in state `Unknown` is also listed; it is an old leftover, ignore it. + - OOM kills, in the pod: `dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'`. The + timestamps are seconds since boot; compare them with the first field of `/proc/uptime` to see + whether any falls in the deploy window (2026-10-25: the newest was about 19 h old). A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks` shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child @@ -344,7 +409,7 @@ the pod image: it was installed by hand and was gone after the 2026-10-24 pod re supervisorctl -c /root/supervisord.conf stop llama-swap ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID cd /root/llama.cpp -git checkout -B "$(cat /root/DEPLOY-ROLLBACK-commit-N.txt)" +git checkout -B "$(head -1 /root/DEPLOY-ROLLBACK-commit-N.txt | cut -d' ' -f1)" cmake --build build --target llama-server llama-bench -j 12 supervisorctl -c /root/supervisord.conf start llama-swap curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models @@ -353,8 +418,10 @@ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in production. Keep the candidate worktree for a later redeploy. -Pointer files 2 to 9 exist; for example commit-7 holds `e70486447`, commit-8 `64543a435`, and commit-9 -`5751c977b`, the rollback target for the current production `6c480f7e7`. +Pointer files 2 to 10 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`, and +commit-10 `6c480f7e7`, the rollback target for the current production `2fc9f9c35`. The formats vary +(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on +line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1. ## 10. Recovering a hung GPU @@ -383,8 +450,9 @@ nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg ``` -Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active -(`/tmp/fan-control.log` shows `power1_cap ... -> 120000000`). The two 2026-10-09 hangs both came at the +Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active: +`cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` reads `120000000` (the +`power1_cap ... -> 120000000` line in `/tmp/fan-control.log` may be missing after a pod restart). The two 2026-10-09 hangs both came at the first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork and a clean-upstream build hung, so they were treated as hardware/power, not code. @@ -425,6 +493,7 @@ Useful commands: - VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`. - Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock` (WRITE = holder on token.lock, READ = registered waiters on want.lock). +- 120 W cap active: `cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` = `120000000`. - The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log). diff --git a/scripts/fork-validate.sh b/scripts/fork-validate.sh index 1e6904d74..c1f4f5324 100644 --- a/scripts/fork-validate.sh +++ b/scripts/fork-validate.sh @@ -29,6 +29,8 @@ # session (fresh server pairs, blocks ref,cand,cand,ref x 16 handoffs) -> FAIL if the # candidate median exceeds the reference median by more than max(25%, 200 ms). Two # production deploys were lost to arbiter changes that only showed up here. +# Needs >= 1536 MiB free VRAM (FAILs otherwise); beside a warm gemma-26B only about +# 1.2 GiB is free, so then run it in the --perf swapdown window (about 2 minutes). # --perf llama-bench, candidate vs reference, alternating ref,cand,cand,ref, on the 35B MoE # -> no prefill regression > 5% at depth 0/16k/32k (FAIL), decode tg256@d4096 # reported; peak VRAM per arm (FAIL if > 95% of the card). Compares MEDIANS of every