docs: FORK.md and runbook after the 2fc9f9c35 deploy

Feedback from the 2026-10-25 deploy, which used only these docs:
- status: production is 2fc9f9c35; F10 deployed and verified on a 35B
  GPU load; --moe-exact and median --perf have run on the GPU; last
  full fork-validate run (37 PASS) and its timings; the first-candidate
  -arm VRAM spike seen twice is an open question.
- runbook: a no-rebase redeploy path; --arbiter beside production only
  if 1536 MiB VRAM is free, else in the swapdown window; full --moecache
  model path; page-cache eviction before and after phase 2; deploy
  steps reordered so the memory check follows stopping llama-swap;
  in-place rebuild and system_fingerprint check; mtime check through the
  versioned symlinks; rollback pointer numbering and one format; disk
  space; nothink alias and a PPM image helper for post-deploy requests;
  power1_cap via sysfs; restart and OOM checks; harmless UI warning.
- fork-validate.sh header: note the --arbiter VRAM requirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 14:12:52 +01:00
1 parent 2fc9f9c351
commit 71d4cbe195
3 files changed
+158 -81

No files matched your search

+36 -30
View File
@@ -10,10 +10,10 @@ Status at the time of writing (2026-10-25):
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
(this file among them), so count it rather than trusting a number written here. Section 2 maps
every commit to a feature.
- Production runs `6c480f7e7`. Everything after it on the branch is validation tooling, env-gated
A/B knobs, comment fixes, documentation and one loader behaviour change (F10, the page-cache
drop); none of it was deployed. The knobs and the comment
fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation.
- Production runs `2fc9f9c35` (deployed 2026-10-25 13:01 UTC, `system_fingerprint`
`b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line. The commit messages are detailed and are the primary record of each mechanism
and measurement; read them with `git show`.
@@ -118,7 +118,8 @@ Oldest first. "Feature" points at the section below.
| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything from `e085c347d` down is after the production commit `6c480f7e7`.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
`6c480f7e7` was the previous production commit.
---
@@ -176,8 +177,9 @@ compares 64 greedy tokens after a prompt of about twice that, every expert on th
candidate == reference (`identical-ref`), and with the `LLAMA_MOE_IDS_SKIP` knob (f664b0f20;
`=0` forces the readback, `=log` reports each skip on stderr) candidate == candidate with the skip
forced off (`identical-noskip`, which proves the skip exact even when the reference build carries
it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). Not yet run on the GPU
(added after the 20261024 deploy; only dry-run against stub servers). Also `--ops` (the MUL_MAT_ID
it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). First GPU run
2026-10-25 (35B, 2fc9f9c35 vs 6c480f7e7): prompt 397 tokens, 39 skips logged, 0 with the skip forced
off, output identical in all three runs, 78 s. Also `--ops` (the MUL_MAT_ID
kernel, not this callback), `--perf` (prefill regression) and `--moecache` (the cache path through
this callback).
@@ -648,7 +650,8 @@ first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/serv
eagerly (no new process in the 2 s after the kill), and starts a new process on the next request
for that model (new PID, health check, HTTP 200). The production log since the last pod restart has
no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers
the restart pays a full model load (about 26 s for the 35B with `-lm none`), and under the arbiter
the restart pays a full model load (about 26 s for the 35B with `-lm none`; the first 35B request
after the 2026-10-25 deploy took 34 s including load and answer), and under the arbiter
the new process takes the token through `vram_acquire_for_load()` like any load. A process that
wedges instead of exiting (the incidents below) is never restarted: `/health` keeps answering.
@@ -683,8 +686,9 @@ fixed deadline: keep the order "check `!running`, return null, then `wait_until(
**Validation.** `fork-validate.sh --arbiter`: `d-term-in-release` (gdb breakpoint on
`llama_context_release_device`, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and
the token), `e-term-waiting` (exit 0 within 1.5 s, never took the token), `f-term-streaming`
(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run:
736 ms, 200 ms, 93 ms (token free in 28 ms), 80/157/156 ms. Post-deploy: a forced llama-swap eviction
(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run
(2026-10-25, 2fc9f9c35, `d` by log polling): 750 ms, 225 ms, 126 ms (token free in 23 ms),
108/159/159 ms. Post-deploy: a forced llama-swap eviction
must log `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token` and exit by itself.
### F8. Cold-teardown guards
@@ -744,14 +748,17 @@ required, so a rebase can be validated by someone with no context.
inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check
flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline.
See the header of the script for what each check proves.
- Which phase needs what: `--ops --concat --arbiter` run beside production (`--arbiter` refuses
below 1.5 GiB free VRAM); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
- Which phase needs what: `--ops --concat` run beside production; `--arbiter` too only if 1536 MiB
of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs
in the swapdown window, about 2 minutes); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share
the card either).
**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24,
production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s,
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. The corpus
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. On 2026-10-25
(medians, `6c480f7e7` / `2fc9f9c35`): pp2048 274.5/273.8, d16384 200.5/200.4, d32768 157.6/157.6,
tg256@d4096 16.8/16.7. The corpus
harness and older numbers are in `scripts/rx580-bench/README.md`. A fresh run far outside these
means something else is wrong (another GPU user, thermal state, a bad build) before it means a
regression or a win.
@@ -778,11 +785,13 @@ script greps for (`server entering sleeping state`, `transpose fast path, transp
markdown layout; the GGUF keys `<arch>.expert_count` / `<arch>.expert_used_count`; the DRM fdinfo
keys `drm-client-id` / `drm-memory-vram`.
**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test (one candidate
arm once showed an unexplained 1.6 GiB spike that did not repeat); the SIGTERM-in-release breakpoint
is at function entry, not after the host-shadow capture; `--moe-exact` and the median-gated `--perf`
have not run on the GPU yet (tested offline only, see the runbook); the `v-vram-release` SKIP path
(production moving during the measurement) has not been exercised.
**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test. The first
candidate arm has now twice shown an unexplained spike of about 1.6 GiB over the other arms
(2026-10-25: ref 5476, cand 7062, cand 5767, ref 5476 MiB); the cause is open (per-test peaks would
locate it), and it stays far below the 95% gate. The SIGTERM-in-release breakpoint is at function
entry, not after the host-shadow capture. The `v-vram-release` SKIP path (production moving during
the measurement) has not been exercised. `--moe-exact` and the median-gated `--perf` first ran on the
GPU on 2026-10-25 (runbook section 6).
### F10. Page-cache drop after a copying load
@@ -822,7 +831,9 @@ after the tensor reads. Commit cce0ac1f0.
load: the VRAM release/restore (F4) uses host shadows, not the file.
- The drop is a hint: a failure must not fail the load.
**Failure modes and incidents.** None yet (not deployed). Costs to know about:
**Failure modes and incidents.** None. Deployed 2026-10-25 (`2fc9f9c35`); on a 35B GPU load in
production the node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB. Costs to know
about:
- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a
5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few
seconds on a 26 s load.
@@ -858,9 +869,9 @@ Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it count
available. Greedy output byte-identical in all arms. `-lm mmap` control: the model file stays fully
resident and mapped (2498 MiB, `file_mapped` +2.5 GiB) after load and after a request, and only the
copied mmproj is dropped. `-lzm on` control: no drop (the file is mapped), 1252 MiB of copied
tensors stay cached. Not run: the GPU upload path (the per-tensor drop sits after all three copy
paths, so it is covered by construction, but the pinned async upload was not exercised: the card had
only 1.2 GiB free beside production), and a 35B load.
tensors stay cached. The GPU upload path and a 35B load were first exercised by the 2026-10-25
deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B
request took 34 s including load and answer).
---
@@ -1020,14 +1031,9 @@ Machine-level:
null buffer would crash. Only the mmproj path handles a failed restore.
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
- Router mode can spin at shutdown (F7); not used here.
- The stale arbiter comments (KV "never evicted", encoder "rebuilt by `vram_ensure_warm()`", lazy
wake, `process_mtmd_chunk`) and the "keeping KV cache" sleep log line were fixed in f5c0d8f28,
after the deployed build: production still logs "keeping KV cache" while it evicts the KV.
- The new checks (`--moe-exact`, `a-handoff-latency`, `v-vram-*`, median `--perf`) and the
`LLAMA_MOE_IDS_SKIP` knob were written after the 20261024 deploy; the knob is syntax-checked only
(not built), and only `--arbiter` has run on the GPU. Run the full set at the next rebase.
- The unexplained ~1.6 GiB peak-VRAM spike in the first candidate `--perf` arm, seen twice (F9).
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
- F10 (page-cache drop) has not run on the GPU upload path or on a 35B load; its load-time cost there
(synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate.
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
is an estimate; its memory effect is verified in production.
+120 -51
View File
@@ -18,6 +18,17 @@ Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a co
the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU;
11. Operations reference.
**Redeploying a new tip on the same upstream base (no rebase)**, e.g. new fork commits on the
production branch (2026-10-25: `6c480f7e7` -> `2fc9f9c35`, same branch, same base):
- Skip sections 1-3. Of section 4 only the ASCII check (on `git diff <prod>..<tip>`) and a careful
read of that diff apply; `<prod>` is `git -C /root/llama.cpp rev-parse HEAD` on the pod. There is
nothing to range-diff.
- Do sections 5-9 as written: build the tip in a scratch worktree, validate it with `--ref` = the
production build, then deploy.
- The branch name stays the same, so deploy step 2's `git checkout -B <same branch> <tip>` moves the
production branch on the pod to the tip. After that the rollback pointer file is the only record
of the old production commit on the pod, so write it first (deploy step 1).
---
## 1. Survey upstream
@@ -182,9 +193,11 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '
If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed).
- `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes.
- Report compile errors verbatim against the list from step 4.
- Do not overlap a build with a large benchmark (host OOM). Disk is limited (about 22 GB free; each
build tree is about 700 MB); remove scratch worktrees you no longer need with
`git -C /root/llama.cpp worktree remove <dir>`.
- Do not overlap a build with a large benchmark (host OOM). Disk is tight: `/root` had about 13 GB
free (97% full) on 2026-10-25, so check `df -h /root` first. Each build tree is about 700 MB;
remove scratch worktrees you no longer need with `git -C /root/llama.cpp worktree remove <dir>`.
- The build prints `UI: download dist.tar.gz.sha256 ... failed` (no UI assets download). It is
harmless with `-DLLAMA_BUILD_UI=OFF`; the listed targets still build.
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
holding a `kubectl exec` open for hours.
@@ -194,20 +207,37 @@ Copy the script from the candidate tree (it is in the repo) and run it in two ph
the script describes each check.
```sh
# phase 1, beside production (needs >= 1.5 GiB VRAM free for --arbiter; it refuses otherwise)
# phase 1, beside production
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat --arbiter
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat
```
Add `--arbiter` to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus the bytes in
`cat /sys/class/drm/card*/device/mem_info_vram_used`); below that it FAILs at once. With gemma-26B
warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run
`--arbiter` in the phase-2 swapdown window instead (about 2 minutes more downtime), as below.
**Page cache before and after phase 2.** The `--ref` build may lack F10 (FORK.md), and `--perf`
loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the
eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check
`/proc/meminfo`.
```sh
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
cat > /root/arms/fv-perf.sh <<'EOF'
Q35=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
HUB=/root/.cache/huggingface/hub
Q35=$HUB/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
G26=$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/snapshots/e6759b29958180a998f62f991e4c741fe9b4d452/gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf --moe-exact "$Q35" \
--moecache /root/.cache/huggingface/hub/<gemma-4-26B-A4B snapshot>/<file>.gguf --keep
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build \
--arbiter --perf --moe-exact "$Q35" --moecache "$G26" --keep
EOF
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
```
- `G26` is the production SC117 gemma-26B, the `refs/main` snapshot (`e6759b29...` on 2026-10-25;
an older snapshot `82c751632...` is also on disk). If it moves, read the snapshot name from
`$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/refs/main`.
- `/root/bench/moe/swapdown.sh <script>` (pod-only helper, not in the repo) stops llama-swap, waits for
an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints
`SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200.
@@ -215,23 +245,25 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`).
- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits,
which proves nothing).
- Phase 1 now also checks that a cold server really frees its VRAM (`v-vram-release`,
- `--arbiter` also checks that a cold server really frees its VRAM (`v-vram-release`,
`v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs
ordinary handoff latency against `--ref` (`a-handoff-latency`, about 1.5 extra minutes). `--ref`
must be the build currently in production, so both comparisons are against what serves today.
- Phase 2 costs about 27 minutes of downtime for `--perf` plus `--moecache`, and `--moe-exact` adds
three loads of the 35B (estimated 3-5 minutes; not yet timed). It leaves tens of GB of model files
in page cache. `--moe-exact` proves the routing-ids readback skip exact (F1): the candidate's
greedy output equals the reference's and, with the `LLAMA_MOE_IDS_SKIP` knob (any build of this
branch), the candidate's with the skip forced off.
ordinary handoff latency against `--ref` (`a-handoff-latency`). `--ref` must be the build currently
in production, so both comparisons are against what serves today.
- Phase 2 downtime, measured 2026-10-25: about 32 minutes in all (`--arbiter` about 2 min, `--perf`
about 28 min, `--moecache` under 30 s, `--moe-exact` 78 s). `--moe-exact` proves the routing-ids
readback skip exact (F1): the candidate's greedy output equals the reference's and, with the
`LLAMA_MOE_IDS_SKIP` knob (any build of this branch), the candidate's with the skip forced off.
- `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- Last full run (2026-10-24, 33 PASS, 0 FAIL; old script, means of 2 arm means, no `--moe-exact`):
perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, -0.7% at d32k (gate is -5%),
tg256@d4096 -0.2%. The new phase-1 checks first ran on 2026-10-25 (reb1024 against itself beside
production): 16 PASS; VRAM warm +546 / cold +4 / restored +546 MiB; handoff median 1050.5 vs 1082.5 ms.
- Last full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs
`/root/arms/dep-2fc9f9c35-fv1.log` and `-fv2.log`): phase 1 `--ops --concat` 8 PASS, phase 2
`--arbiter --perf --moe-exact --moecache` 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048
274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff
median ref 693.5 / cand 678 ms. `--moe-exact`: 39 skips logged, identical to the reference and to
the skip forced off. Peak VRAM per arm ref 5476, cand 7062, cand 5767, ref 5476 MiB: the ~1.6 GiB
spike in the first candidate arm has now been seen twice (open question, FORK.md F9).
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
regression gate, not a measurement.
@@ -240,12 +272,17 @@ On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept w
## 7. Deploy
Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 minutes.
Deploy exactly the validated commit. Downtime: 6 minutes for the 2026-10-24 redeploy. The deploy
rebuilds the production tree in place (step 6) from the same commit and with the same cmake flags as
the validated worktree build, so nothing is copied over; compare
`grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt` in both
trees if in doubt.
1. **Rollback pointer.** Record the current production commit in the next free pointer file:
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 9 (holds 5751c977b)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-10.txt
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 10 (holds 6c480f7e7)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-11.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea):
@@ -254,13 +291,14 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6
git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD
git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD <validated-commit>
```
3. **Memory.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM victim. With
`-lm none` each load reads the GGUF through the page cache, and a recent benchmark leaves tens of GB
of model files cached. During the first 2026-10-24 deploy the Talos OOM controller killed the whole
pod 17 s into verification (cause likely, not proven, memory pressure). Before stopping llama-swap,
evict the model files' page cache (unprivileged, safe while servers run: `-lm none` servers hold
their weights in anonymous memory, and pages mapped by an mmap model are not dropped) and check
memory:
On a redeploy with the same branch name this moves the production branch to the new commit.
3. **Evict the page cache.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM
victim, and benchmarks, mmap loads and builds without F10 leave tens of GB of model files cached.
During the first 2026-10-24 deploy the Talos OOM controller killed the whole pod 17 s into
verification (cause likely, not proven, memory pressure). Evict while production still runs
(unprivileged and safe: `-lm none` servers hold their weights in anonymous memory, and pages
mapped by an mmap model are not dropped). The same snippet is used before and after validation
phase 2:
```sh
python3 - <<'EOF'
import glob, os
@@ -275,14 +313,11 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6
os.close(fd)
print('evicted', n, 'files')
EOF
grep -E 'MemAvailable|MemFree' /proc/meminfo
```
A build with F10 (FORK.md) drops the cache of every `-lm none` load by itself, but files cached
by mmap loads, benchmarks or older builds still need this step.
Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by
models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache
from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the
container is the HOST root cgroup; the pod's own cgroup is under
F10 (FORK.md) drops the cache of every `-lm none` load by itself (verified in production on a 35B
GPU load: node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB), but files cached
by mmap loads, benchmarks or older builds still need this step. `/sys/fs/cgroup/memory.stat`
inside the container is the HOST root cgroup; the pod's own cgroup is under
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/`.
4. **Stop llama-swap and reap orphans.**
```sh
@@ -292,19 +327,29 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6
ps -eo pid,args | grep '[l]lama-server' # must be empty
```
`supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1),
one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM.
5. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM. Keep
this check even when it finds nothing (2026-10-25: no orphans).
5. **Check memory**, now that the running models are gone: `grep -E 'MemAvailable|MemFree'
/proc/meminfo`. Proceed only with MemAvailable >= 25 GB. In 2026-10-24 the eviction took node-wide
file cache from 15.6 to 4.9 GB and MemAvailable to 45.8 GB.
6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
```sh
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12
ls -lL build/bin/llama-server build/bin/libllama.so build/bin/libmtmd.so \
build/bin/libggml.so build/bin/libggml-base.so build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
```
Check that the libraries in `build/bin` have fresh mtimes.
6. **Start and probe.**
All must have fresh mtimes. Use `-L` (follow the symlinks): `build/bin` keeps every old versioned
`libggml-*.so.0.x` and `libmtmd.so.0.x`, so a plain `ls` lists stale files beside the new ones.
7. **Start and probe.**
```sh
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200
```
Sample `/proc/meminfo` every few seconds during the first model loads.
Sample `/proc/meminfo` every few seconds during the first model loads. Confirm the running commit
from the first chat response (section 8): `system_fingerprint` is `b<build>-<short sha>`
(2026-10-25: `b11906-2fc9f9c35`). The first 35B request after that deploy took 34 s including
the model load and the answer.
## 8. Post-deploy checks
@@ -314,22 +359,42 @@ All through llama-swap on :8080. A request helper:
ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; }
holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; }
# image request body: a 64x64 solid red binary PPM, no image tools needed (like fork-validate.sh)
img() { python3 - "$1" <<'EOF'
import base64, json, sys
ppm = b'P6\n64 64\n255\n' + b'\xff\x00\x00' * (64 * 64)
url = 'data:image/x-portable-pixmap;base64,' + base64.b64encode(ppm).decode()
print(json.dumps({'model': sys.argv[1], 'max_tokens': 32, 'temperature': 0, 'messages': [{'role': 'user',
'content': [{'type': 'image_url', 'image_url': {'url': url}},
{'type': 'text', 'text': 'What is the main colour of this image? One word.'}]}]}))
EOF
}
# img MODEL | curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @-
```
Use the `-nothink` aliases with these `max_tokens: 32` helpers. A thinking alias spends the 32 tokens
on `reasoning_content` and returns an empty `content`; for one, raise `max_tokens` to >= 512.
1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers;
`holder` shows the 35B's PID (check with `ps -o args -p <pid>`).
2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is
correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch
(2026-10-24: 3.1-5.6 s).
3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF:UD-Q4_K_XL` (a small
solid-colour image, ask for the colour). This also forces an eviction in the llama-swap matrix: the
3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF-nothink:UD-Q4_K_XL`
with the `img` helper; the answer names red. This also forces an eviction in the llama-swap matrix: the
evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed,
releasing the GPU token`, with no kill escalation.
4. Text to gemma and to the 4B after the eviction; `holder` correct each time.
5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931).
6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for
`GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none.
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged.
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`). No pod restart and no OOM kill:
- Restarts, from the workstation: `kubectl get pods -n llama | grep supervisord`; RESTARTS of the
Running pod must equal the value you noted before the deploy. A stale `supervisord-...-pbmjq`
in state `Unknown` is also listed; it is an old leftover, ignore it.
- OOM kills, in the pod: `dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'`. The
timestamps are seconds since boot; compare them with the first field of `/proc/uptime` to see
whether any falls in the deploy window (2026-10-25: the newest was about 19 h old).
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
@@ -344,7 +409,7 @@ the pod image: it was installed by hand and was gone after the 2026-10-24 pod re
supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
cd /root/llama.cpp
git checkout -B <previous-production-branch> "$(cat /root/DEPLOY-ROLLBACK-commit-N.txt)"
git checkout -B <previous-production-branch> "$(head -1 /root/DEPLOY-ROLLBACK-commit-N.txt | cut -d' ' -f1)"
cmake --build build --target llama-server llama-bench -j 12
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
@@ -353,8 +418,10 @@ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in
production. Keep the candidate worktree for a later redeploy.
Pointer files 2 to 9 exist; for example commit-7 holds `e70486447`, commit-8 `64543a435`, and commit-9
`5751c977b`, the rollback target for the current production `6c480f7e7`.
Pointer files 2 to 10 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`, and
commit-10 `6c480f7e7`, the rollback target for the current production `2fc9f9c35`. The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
## 10. Recovering a hung GPU
@@ -383,8 +450,9 @@ nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices
nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg
```
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active
(`/tmp/fan-control.log` shows `power1_cap ... -> 120000000`). The two 2026-10-09 hangs both came at the
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active:
`cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` reads `120000000` (the
`power1_cap ... -> 120000000` line in `/tmp/fan-control.log` may be missing after a pod restart). The two 2026-10-09 hangs both came at the
first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork
and a clean-upstream build hung, so they were treated as hardware/power, not code.
@@ -425,6 +493,7 @@ Useful commands:
- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`.
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
- 120 W cap active: `cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` = `120000000`.
- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the
last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log).
+2
View File
@@ -29,6 +29,8 @@
# session (fresh server pairs, blocks ref,cand,cand,ref x 16 handoffs) -> FAIL if the
# candidate median exceeds the reference median by more than max(25%, 200 ms). Two
# production deploys were lost to arbiter changes that only showed up here.
# Needs >= 1536 MiB free VRAM (FAILs otherwise); beside a warm gemma-26B only about
# 1.2 GiB is free, so then run it in the --perf swapdown window (about 2 minutes).
# --perf llama-bench, candidate vs reference, alternating ref,cand,cand,ref, on the 35B MoE
# -> no prefill regression > 5% at depth 0/16k/32k (FAIL), decode tg256@d4096
# reported; peak VRAM per arm (FAIL if > 95% of the card). Compares MEDIANS of every