docs: FORK.md and runbook after the 2fc9f9c35 deploy
Feedback from the 2026-10-25 deploy, which used only these docs: - status: production is 2fc9f9c35; F10 deployed and verified on a 35B GPU load; --moe-exact and median --perf have run on the GPU; last full fork-validate run (37 PASS) and its timings; the first-candidate -arm VRAM spike seen twice is an open question. - runbook: a no-rebase redeploy path; --arbiter beside production only if 1536 MiB VRAM is free, else in the swapdown window; full --moecache model path; page-cache eviction before and after phase 2; deploy steps reordered so the memory check follows stopping llama-swap; in-place rebuild and system_fingerprint check; mtime check through the versioned symlinks; rollback pointer numbering and one format; disk space; nothink alias and a PPM image helper for post-deploy requests; power1_cap via sysfs; restart and OOM checks; harmless UI warning. - fork-validate.sh header: note the --arbiter VRAM requirement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2fc9f9c351
commit
71d4cbe195
3 files changed
+158
-81
No files matched your search
@@ -10,10 +10,10 @@ Status at the time of writing (2026-10-25):
|
||||
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
|
||||
(this file among them), so count it rather than trusting a number written here. Section 2 maps
|
||||
every commit to a feature.
|
||||
- Production runs `6c480f7e7`. Everything after it on the branch is validation tooling, env-gated
|
||||
A/B knobs, comment fixes, documentation and one loader behaviour change (F10, the page-cache
|
||||
drop); none of it was deployed. The knobs and the comment
|
||||
fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation.
|
||||
- Production runs `2fc9f9c35` (deployed 2026-10-25 13:01 UTC, `system_fingerprint`
|
||||
`b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the
|
||||
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
|
||||
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
|
||||
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
|
||||
its subject line. The commit messages are detailed and are the primary record of each mechanism
|
||||
and measurement; read them with `git show`.
|
||||
@@ -118,7 +118,8 @@ Oldest first. "Feature" points at the section below.
|
||||
| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 |
|
||||
|
||||
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
||||
Everything from `e085c347d` down is after the production commit `6c480f7e7`.
|
||||
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
||||
`6c480f7e7` was the previous production commit.
|
||||
|
||||
---
|
||||
|
||||
@@ -176,8 +177,9 @@ compares 64 greedy tokens after a prompt of about twice that, every expert on th
|
||||
candidate == reference (`identical-ref`), and with the `LLAMA_MOE_IDS_SKIP` knob (f664b0f20;
|
||||
`=0` forces the readback, `=log` reports each skip on stderr) candidate == candidate with the skip
|
||||
forced off (`identical-noskip`, which proves the skip exact even when the reference build carries
|
||||
it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). Not yet run on the GPU
|
||||
(added after the 20261024 deploy; only dry-run against stub servers). Also `--ops` (the MUL_MAT_ID
|
||||
it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). First GPU run
|
||||
2026-10-25 (35B, 2fc9f9c35 vs 6c480f7e7): prompt 397 tokens, 39 skips logged, 0 with the skip forced
|
||||
off, output identical in all three runs, 78 s. Also `--ops` (the MUL_MAT_ID
|
||||
kernel, not this callback), `--perf` (prefill regression) and `--moecache` (the cache path through
|
||||
this callback).
|
||||
|
||||
@@ -648,7 +650,8 @@ first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/serv
|
||||
eagerly (no new process in the 2 s after the kill), and starts a new process on the next request
|
||||
for that model (new PID, health check, HTTP 200). The production log since the last pod restart has
|
||||
no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers
|
||||
the restart pays a full model load (about 26 s for the 35B with `-lm none`), and under the arbiter
|
||||
the restart pays a full model load (about 26 s for the 35B with `-lm none`; the first 35B request
|
||||
after the 2026-10-25 deploy took 34 s including load and answer), and under the arbiter
|
||||
the new process takes the token through `vram_acquire_for_load()` like any load. A process that
|
||||
wedges instead of exiting (the incidents below) is never restarted: `/health` keeps answering.
|
||||
|
||||
@@ -683,8 +686,9 @@ fixed deadline: keep the order "check `!running`, return null, then `wait_until(
|
||||
**Validation.** `fork-validate.sh --arbiter`: `d-term-in-release` (gdb breakpoint on
|
||||
`llama_context_release_device`, then SIGTERM: exit 0 within 2.5 s, the other server gets HTTP 200 and
|
||||
the token), `e-term-waiting` (exit 0 within 1.5 s, never took the token), `f-term-streaming`
|
||||
(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run:
|
||||
736 ms, 200 ms, 93 ms (token free in 28 ms), 80/157/156 ms. Post-deploy: a forced llama-swap eviction
|
||||
(exit 0, SSE error event, token free within 2.5 s), `h-shutdown-warm`, `h-shutdown-cold`. Last run
|
||||
(2026-10-25, 2fc9f9c35, `d` by log polling): 750 ms, 225 ms, 126 ms (token free in 23 ms),
|
||||
108/159/159 ms. Post-deploy: a forced llama-swap eviction
|
||||
must log `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token` and exit by itself.
|
||||
|
||||
### F8. Cold-teardown guards
|
||||
@@ -744,14 +748,17 @@ required, so a rebase can be validated by someone with no context.
|
||||
inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check
|
||||
flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline.
|
||||
See the header of the script for what each check proves.
|
||||
- Which phase needs what: `--ops --concat --arbiter` run beside production (`--arbiter` refuses
|
||||
below 1.5 GiB free VRAM); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
|
||||
- Which phase needs what: `--ops --concat` run beside production; `--arbiter` too only if 1536 MiB
|
||||
of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs
|
||||
in the swapdown window, about 2 minutes); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
|
||||
first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share
|
||||
the card either).
|
||||
|
||||
**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24,
|
||||
production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s,
|
||||
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. The corpus
|
||||
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. On 2026-10-25
|
||||
(medians, `6c480f7e7` / `2fc9f9c35`): pp2048 274.5/273.8, d16384 200.5/200.4, d32768 157.6/157.6,
|
||||
tg256@d4096 16.8/16.7. The corpus
|
||||
harness and older numbers are in `scripts/rx580-bench/README.md`. A fresh run far outside these
|
||||
means something else is wrong (another GPU user, thermal state, a bad build) before it means a
|
||||
regression or a win.
|
||||
@@ -778,11 +785,13 @@ script greps for (`server entering sleeping state`, `transpose fast path, transp
|
||||
markdown layout; the GGUF keys `<arch>.expert_count` / `<arch>.expert_used_count`; the DRM fdinfo
|
||||
keys `drm-client-id` / `drm-memory-vram`.
|
||||
|
||||
**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test (one candidate
|
||||
arm once showed an unexplained 1.6 GiB spike that did not repeat); the SIGTERM-in-release breakpoint
|
||||
is at function entry, not after the host-shadow capture; `--moe-exact` and the median-gated `--perf`
|
||||
have not run on the GPU yet (tested offline only, see the runbook); the `v-vram-release` SKIP path
|
||||
(production moving during the measurement) has not been exercised.
|
||||
**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test. The first
|
||||
candidate arm has now twice shown an unexplained spike of about 1.6 GiB over the other arms
|
||||
(2026-10-25: ref 5476, cand 7062, cand 5767, ref 5476 MiB); the cause is open (per-test peaks would
|
||||
locate it), and it stays far below the 95% gate. The SIGTERM-in-release breakpoint is at function
|
||||
entry, not after the host-shadow capture. The `v-vram-release` SKIP path (production moving during
|
||||
the measurement) has not been exercised. `--moe-exact` and the median-gated `--perf` first ran on the
|
||||
GPU on 2026-10-25 (runbook section 6).
|
||||
|
||||
### F10. Page-cache drop after a copying load
|
||||
|
||||
@@ -822,7 +831,9 @@ after the tensor reads. Commit cce0ac1f0.
|
||||
load: the VRAM release/restore (F4) uses host shadows, not the file.
|
||||
- The drop is a hint: a failure must not fail the load.
|
||||
|
||||
**Failure modes and incidents.** None yet (not deployed). Costs to know about:
|
||||
**Failure modes and incidents.** None. Deployed 2026-10-25 (`2fc9f9c35`); on a 35B GPU load in
|
||||
production the node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB. Costs to know
|
||||
about:
|
||||
- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a
|
||||
5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few
|
||||
seconds on a 26 s load.
|
||||
@@ -858,9 +869,9 @@ Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it count
|
||||
available. Greedy output byte-identical in all arms. `-lm mmap` control: the model file stays fully
|
||||
resident and mapped (2498 MiB, `file_mapped` +2.5 GiB) after load and after a request, and only the
|
||||
copied mmproj is dropped. `-lzm on` control: no drop (the file is mapped), 1252 MiB of copied
|
||||
tensors stay cached. Not run: the GPU upload path (the per-tensor drop sits after all three copy
|
||||
paths, so it is covered by construction, but the pinned async upload was not exercised: the card had
|
||||
only 1.2 GiB free beside production), and a 35B load.
|
||||
tensors stay cached. The GPU upload path and a 35B load were first exercised by the 2026-10-25
|
||||
deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B
|
||||
request took 34 s including load and answer).
|
||||
|
||||
---
|
||||
|
||||
@@ -1020,14 +1031,9 @@ Machine-level:
|
||||
null buffer would crash. Only the mmproj path handles a failed restore.
|
||||
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
|
||||
- Router mode can spin at shutdown (F7); not used here.
|
||||
- The stale arbiter comments (KV "never evicted", encoder "rebuilt by `vram_ensure_warm()`", lazy
|
||||
wake, `process_mtmd_chunk`) and the "keeping KV cache" sleep log line were fixed in f5c0d8f28,
|
||||
after the deployed build: production still logs "keeping KV cache" while it evicts the KV.
|
||||
- The new checks (`--moe-exact`, `a-handoff-latency`, `v-vram-*`, median `--perf`) and the
|
||||
`LLAMA_MOE_IDS_SKIP` knob were written after the 20261024 deploy; the knob is syntax-checked only
|
||||
(not built), and only `--arbiter` has run on the GPU. Run the full set at the next rebase.
|
||||
- The unexplained ~1.6 GiB peak-VRAM spike in the first candidate `--perf` arm, seen twice (F9).
|
||||
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
|
||||
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
|
||||
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
|
||||
- F10 (page-cache drop) has not run on the GPU upload path or on a 35B load; its load-time cost there
|
||||
(synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate.
|
||||
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
|
||||
is an estimate; its memory effect is verified in production.
|
||||
+120
-51
@@ -18,6 +18,17 @@ Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a co
|
||||
the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU;
|
||||
11. Operations reference.
|
||||
|
||||
**Redeploying a new tip on the same upstream base (no rebase)**, e.g. new fork commits on the
|
||||
production branch (2026-10-25: `6c480f7e7` -> `2fc9f9c35`, same branch, same base):
|
||||
- Skip sections 1-3. Of section 4 only the ASCII check (on `git diff <prod>..<tip>`) and a careful
|
||||
read of that diff apply; `<prod>` is `git -C /root/llama.cpp rev-parse HEAD` on the pod. There is
|
||||
nothing to range-diff.
|
||||
- Do sections 5-9 as written: build the tip in a scratch worktree, validate it with `--ref` = the
|
||||
production build, then deploy.
|
||||
- The branch name stays the same, so deploy step 2's `git checkout -B <same branch> <tip>` moves the
|
||||
production branch on the pod to the tip. After that the rollback pointer file is the only record
|
||||
of the old production commit on the pod, so write it first (deploy step 1).
|
||||
|
||||
---
|
||||
|
||||
## 1. Survey upstream
|
||||
@@ -182,9 +193,11 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '
|
||||
If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed).
|
||||
- `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes.
|
||||
- Report compile errors verbatim against the list from step 4.
|
||||
- Do not overlap a build with a large benchmark (host OOM). Disk is limited (about 22 GB free; each
|
||||
build tree is about 700 MB); remove scratch worktrees you no longer need with
|
||||
`git -C /root/llama.cpp worktree remove <dir>`.
|
||||
- Do not overlap a build with a large benchmark (host OOM). Disk is tight: `/root` had about 13 GB
|
||||
free (97% full) on 2026-10-25, so check `df -h /root` first. Each build tree is about 700 MB;
|
||||
remove scratch worktrees you no longer need with `git -C /root/llama.cpp worktree remove <dir>`.
|
||||
- The build prints `UI: download dist.tar.gz.sha256 ... failed` (no UI assets download). It is
|
||||
harmless with `-DLLAMA_BUILD_UI=OFF`; the listed targets still build.
|
||||
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
|
||||
holding a `kubectl exec` open for hours.
|
||||
|
||||
@@ -194,20 +207,37 @@ Copy the script from the candidate tree (it is in the repo) and run it in two ph
|
||||
the script describes each check.
|
||||
|
||||
```sh
|
||||
# phase 1, beside production (needs >= 1.5 GiB VRAM free for --arbiter; it refuses otherwise)
|
||||
# phase 1, beside production
|
||||
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
|
||||
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat --arbiter
|
||||
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat
|
||||
```
|
||||
|
||||
Add `--arbiter` to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus the bytes in
|
||||
`cat /sys/class/drm/card*/device/mem_info_vram_used`); below that it FAILs at once. With gemma-26B
|
||||
warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run
|
||||
`--arbiter` in the phase-2 swapdown window instead (about 2 minutes more downtime), as below.
|
||||
|
||||
**Page cache before and after phase 2.** The `--ref` build may lack F10 (FORK.md), and `--perf`
|
||||
loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the
|
||||
eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check
|
||||
`/proc/meminfo`.
|
||||
|
||||
```sh
|
||||
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
|
||||
cat > /root/arms/fv-perf.sh <<'EOF'
|
||||
Q35=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
|
||||
HUB=/root/.cache/huggingface/hub
|
||||
Q35=$HUB/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
|
||||
G26=$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/snapshots/e6759b29958180a998f62f991e4c741fe9b4d452/gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf
|
||||
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
|
||||
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf --moe-exact "$Q35" \
|
||||
--moecache /root/.cache/huggingface/hub/<gemma-4-26B-A4B snapshot>/<file>.gguf --keep
|
||||
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build \
|
||||
--arbiter --perf --moe-exact "$Q35" --moecache "$G26" --keep
|
||||
EOF
|
||||
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
|
||||
```
|
||||
|
||||
- `G26` is the production SC117 gemma-26B, the `refs/main` snapshot (`e6759b29...` on 2026-10-25;
|
||||
an older snapshot `82c751632...` is also on disk). If it moves, read the snapshot name from
|
||||
`$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/refs/main`.
|
||||
- `/root/bench/moe/swapdown.sh <script>` (pod-only helper, not in the repo) stops llama-swap, waits for
|
||||
an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints
|
||||
`SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200.
|
||||
@@ -215,23 +245,25 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
|
||||
e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`).
|
||||
- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits,
|
||||
which proves nothing).
|
||||
- Phase 1 now also checks that a cold server really frees its VRAM (`v-vram-release`,
|
||||
- `--arbiter` also checks that a cold server really frees its VRAM (`v-vram-release`,
|
||||
`v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs
|
||||
ordinary handoff latency against `--ref` (`a-handoff-latency`, about 1.5 extra minutes). `--ref`
|
||||
must be the build currently in production, so both comparisons are against what serves today.
|
||||
- Phase 2 costs about 27 minutes of downtime for `--perf` plus `--moecache`, and `--moe-exact` adds
|
||||
three loads of the 35B (estimated 3-5 minutes; not yet timed). It leaves tens of GB of model files
|
||||
in page cache. `--moe-exact` proves the routing-ids readback skip exact (F1): the candidate's
|
||||
greedy output equals the reference's and, with the `LLAMA_MOE_IDS_SKIP` knob (any build of this
|
||||
branch), the candidate's with the skip forced off.
|
||||
ordinary handoff latency against `--ref` (`a-handoff-latency`). `--ref` must be the build currently
|
||||
in production, so both comparisons are against what serves today.
|
||||
- Phase 2 downtime, measured 2026-10-25: about 32 minutes in all (`--arbiter` about 2 min, `--perf`
|
||||
about 28 min, `--moecache` under 30 s, `--moe-exact` 78 s). `--moe-exact` proves the routing-ids
|
||||
readback skip exact (F1): the candidate's greedy output equals the reference's and, with the
|
||||
`LLAMA_MOE_IDS_SKIP` knob (any build of this branch), the candidate's with the skip forced off.
|
||||
- `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in
|
||||
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
|
||||
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
|
||||
scratch dirs were lost that way).
|
||||
- Last full run (2026-10-24, 33 PASS, 0 FAIL; old script, means of 2 arm means, no `--moe-exact`):
|
||||
perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, -0.7% at d32k (gate is -5%),
|
||||
tg256@d4096 -0.2%. The new phase-1 checks first ran on 2026-10-25 (reb1024 against itself beside
|
||||
production): 16 PASS; VRAM warm +546 / cold +4 / restored +546 MiB; handoff median 1050.5 vs 1082.5 ms.
|
||||
- Last full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs
|
||||
`/root/arms/dep-2fc9f9c35-fv1.log` and `-fv2.log`): phase 1 `--ops --concat` 8 PASS, phase 2
|
||||
`--arbiter --perf --moe-exact --moecache` 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048
|
||||
274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff
|
||||
median ref 693.5 / cand 678 ms. `--moe-exact`: 39 skips logged, identical to the reference and to
|
||||
the skip forced off. Peak VRAM per arm ref 5476, cand 7062, cand 5767, ref 5476 MiB: the ~1.6 GiB
|
||||
spike in the first candidate arm has now been seen twice (open question, FORK.md F9).
|
||||
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
|
||||
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
|
||||
regression gate, not a measurement.
|
||||
@@ -240,12 +272,17 @@ On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept w
|
||||
|
||||
## 7. Deploy
|
||||
|
||||
Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 minutes.
|
||||
Deploy exactly the validated commit. Downtime: 6 minutes for the 2026-10-24 redeploy. The deploy
|
||||
rebuilds the production tree in place (step 6) from the same commit and with the same cmake flags as
|
||||
the validated worktree build, so nothing is copied over; compare
|
||||
`grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt` in both
|
||||
trees if in doubt.
|
||||
|
||||
1. **Rollback pointer.** Record the current production commit in the next free pointer file:
|
||||
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
|
||||
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
|
||||
```sh
|
||||
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 9 (holds 5751c977b)
|
||||
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-10.txt
|
||||
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 10 (holds 6c480f7e7)
|
||||
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-11.txt
|
||||
git -C /root/llama.cpp status --short # must be clean
|
||||
```
|
||||
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea):
|
||||
@@ -254,13 +291,14 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6
|
||||
git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD
|
||||
git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD <validated-commit>
|
||||
```
|
||||
3. **Memory.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM victim. With
|
||||
`-lm none` each load reads the GGUF through the page cache, and a recent benchmark leaves tens of GB
|
||||
of model files cached. During the first 2026-10-24 deploy the Talos OOM controller killed the whole
|
||||
pod 17 s into verification (cause likely, not proven, memory pressure). Before stopping llama-swap,
|
||||
evict the model files' page cache (unprivileged, safe while servers run: `-lm none` servers hold
|
||||
their weights in anonymous memory, and pages mapped by an mmap model are not dropped) and check
|
||||
memory:
|
||||
On a redeploy with the same branch name this moves the production branch to the new commit.
|
||||
3. **Evict the page cache.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM
|
||||
victim, and benchmarks, mmap loads and builds without F10 leave tens of GB of model files cached.
|
||||
During the first 2026-10-24 deploy the Talos OOM controller killed the whole pod 17 s into
|
||||
verification (cause likely, not proven, memory pressure). Evict while production still runs
|
||||
(unprivileged and safe: `-lm none` servers hold their weights in anonymous memory, and pages
|
||||
mapped by an mmap model are not dropped). The same snippet is used before and after validation
|
||||
phase 2:
|
||||
```sh
|
||||
python3 - <<'EOF'
|
||||
import glob, os
|
||||
@@ -275,14 +313,11 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6
|
||||
os.close(fd)
|
||||
print('evicted', n, 'files')
|
||||
EOF
|
||||
grep -E 'MemAvailable|MemFree' /proc/meminfo
|
||||
```
|
||||
A build with F10 (FORK.md) drops the cache of every `-lm none` load by itself, but files cached
|
||||
by mmap loads, benchmarks or older builds still need this step.
|
||||
Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by
|
||||
models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache
|
||||
from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the
|
||||
container is the HOST root cgroup; the pod's own cgroup is under
|
||||
F10 (FORK.md) drops the cache of every `-lm none` load by itself (verified in production on a 35B
|
||||
GPU load: node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB), but files cached
|
||||
by mmap loads, benchmarks or older builds still need this step. `/sys/fs/cgroup/memory.stat`
|
||||
inside the container is the HOST root cgroup; the pod's own cgroup is under
|
||||
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/`.
|
||||
4. **Stop llama-swap and reap orphans.**
|
||||
```sh
|
||||
@@ -292,19 +327,29 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6
|
||||
ps -eo pid,args | grep '[l]lama-server' # must be empty
|
||||
```
|
||||
`supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1),
|
||||
one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM.
|
||||
5. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
|
||||
one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM. Keep
|
||||
this check even when it finds nothing (2026-10-25: no orphans).
|
||||
5. **Check memory**, now that the running models are gone: `grep -E 'MemAvailable|MemFree'
|
||||
/proc/meminfo`. Proceed only with MemAvailable >= 25 GB. In 2026-10-24 the eviction took node-wide
|
||||
file cache from 15.6 to 4.9 GB and MemAvailable to 45.8 GB.
|
||||
6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
|
||||
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
|
||||
```sh
|
||||
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12
|
||||
ls -lL build/bin/llama-server build/bin/libllama.so build/bin/libmtmd.so \
|
||||
build/bin/libggml.so build/bin/libggml-base.so build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
|
||||
```
|
||||
Check that the libraries in `build/bin` have fresh mtimes.
|
||||
6. **Start and probe.**
|
||||
All must have fresh mtimes. Use `-L` (follow the symlinks): `build/bin` keeps every old versioned
|
||||
`libggml-*.so.0.x` and `libmtmd.so.0.x`, so a plain `ls` lists stale files beside the new ones.
|
||||
7. **Start and probe.**
|
||||
```sh
|
||||
supervisorctl -c /root/supervisord.conf start llama-swap
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200
|
||||
```
|
||||
Sample `/proc/meminfo` every few seconds during the first model loads.
|
||||
Sample `/proc/meminfo` every few seconds during the first model loads. Confirm the running commit
|
||||
from the first chat response (section 8): `system_fingerprint` is `b<build>-<short sha>`
|
||||
(2026-10-25: `b11906-2fc9f9c35`). The first 35B request after that deploy took 34 s including
|
||||
the model load and the answer.
|
||||
|
||||
## 8. Post-deploy checks
|
||||
|
||||
@@ -314,22 +359,42 @@ All through llama-swap on :8080. A request helper:
|
||||
ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; }
|
||||
holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; }
|
||||
# image request body: a 64x64 solid red binary PPM, no image tools needed (like fork-validate.sh)
|
||||
img() { python3 - "$1" <<'EOF'
|
||||
import base64, json, sys
|
||||
ppm = b'P6\n64 64\n255\n' + b'\xff\x00\x00' * (64 * 64)
|
||||
url = 'data:image/x-portable-pixmap;base64,' + base64.b64encode(ppm).decode()
|
||||
print(json.dumps({'model': sys.argv[1], 'max_tokens': 32, 'temperature': 0, 'messages': [{'role': 'user',
|
||||
'content': [{'type': 'image_url', 'image_url': {'url': url}},
|
||||
{'type': 'text', 'text': 'What is the main colour of this image? One word.'}]}]}))
|
||||
EOF
|
||||
}
|
||||
# img MODEL | curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @-
|
||||
```
|
||||
|
||||
Use the `-nothink` aliases with these `max_tokens: 32` helpers. A thinking alias spends the 32 tokens
|
||||
on `reasoning_content` and returns an empty `content`; for one, raise `max_tokens` to >= 512.
|
||||
|
||||
1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers;
|
||||
`holder` shows the 35B's PID (check with `ps -o args -p <pid>`).
|
||||
2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is
|
||||
correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch
|
||||
(2026-10-24: 3.1-5.6 s).
|
||||
3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF:UD-Q4_K_XL` (a small
|
||||
solid-colour image, ask for the colour). This also forces an eviction in the llama-swap matrix: the
|
||||
3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF-nothink:UD-Q4_K_XL`
|
||||
with the `img` helper; the answer names red. This also forces an eviction in the llama-swap matrix: the
|
||||
evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed,
|
||||
releasing the GPU token`, with no kill escalation.
|
||||
4. Text to gemma and to the 4B after the eviction; `holder` correct each time.
|
||||
5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931).
|
||||
6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for
|
||||
`GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none.
|
||||
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged.
|
||||
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`). No pod restart and no OOM kill:
|
||||
- Restarts, from the workstation: `kubectl get pods -n llama | grep supervisord`; RESTARTS of the
|
||||
Running pod must equal the value you noted before the deploy. A stale `supervisord-...-pbmjq`
|
||||
in state `Unknown` is also listed; it is an old leftover, ignore it.
|
||||
- OOM kills, in the pod: `dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'`. The
|
||||
timestamps are seconds since boot; compare them with the first field of `/proc/uptime` to see
|
||||
whether any falls in the deploy window (2026-10-25: the newest was about 19 h old).
|
||||
|
||||
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
|
||||
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
|
||||
@@ -344,7 +409,7 @@ the pod image: it was installed by hand and was gone after the 2026-10-24 pod re
|
||||
supervisorctl -c /root/supervisord.conf stop llama-swap
|
||||
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
|
||||
cd /root/llama.cpp
|
||||
git checkout -B <previous-production-branch> "$(cat /root/DEPLOY-ROLLBACK-commit-N.txt)"
|
||||
git checkout -B <previous-production-branch> "$(head -1 /root/DEPLOY-ROLLBACK-commit-N.txt | cut -d' ' -f1)"
|
||||
cmake --build build --target llama-server llama-bench -j 12
|
||||
supervisorctl -c /root/supervisord.conf start llama-swap
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
|
||||
@@ -353,8 +418,10 @@ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
|
||||
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in
|
||||
production. Keep the candidate worktree for a later redeploy.
|
||||
|
||||
Pointer files 2 to 9 exist; for example commit-7 holds `e70486447`, commit-8 `64543a435`, and commit-9
|
||||
`5751c977b`, the rollback target for the current production `6c480f7e7`.
|
||||
Pointer files 2 to 10 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`, and
|
||||
commit-10 `6c480f7e7`, the rollback target for the current production `2fc9f9c35`. The formats vary
|
||||
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
|
||||
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
|
||||
|
||||
## 10. Recovering a hung GPU
|
||||
|
||||
@@ -383,8 +450,9 @@ nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices
|
||||
nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg
|
||||
```
|
||||
|
||||
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active
|
||||
(`/tmp/fan-control.log` shows `power1_cap ... -> 120000000`). The two 2026-10-09 hangs both came at the
|
||||
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active:
|
||||
`cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` reads `120000000` (the
|
||||
`power1_cap ... -> 120000000` line in `/tmp/fan-control.log` may be missing after a pod restart). The two 2026-10-09 hangs both came at the
|
||||
first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork
|
||||
and a clean-upstream build hung, so they were treated as hardware/power, not code.
|
||||
|
||||
@@ -425,6 +493,7 @@ Useful commands:
|
||||
- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`.
|
||||
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
|
||||
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
|
||||
- 120 W cap active: `cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` = `120000000`.
|
||||
- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the
|
||||
last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log).
|
||||
|
||||
|
||||
@@ -29,6 +29,8 @@
|
||||
# session (fresh server pairs, blocks ref,cand,cand,ref x 16 handoffs) -> FAIL if the
|
||||
# candidate median exceeds the reference median by more than max(25%, 200 ms). Two
|
||||
# production deploys were lost to arbiter changes that only showed up here.
|
||||
# Needs >= 1536 MiB free VRAM (FAILs otherwise); beside a warm gemma-26B only about
|
||||
# 1.2 GiB is free, so then run it in the --perf swapdown window (about 2 minutes).
|
||||
# --perf llama-bench, candidate vs reference, alternating ref,cand,cand,ref, on the 35B MoE
|
||||
# -> no prefill regression > 5% at depth 0/16k/32k (FAIL), decode tg256@d4096
|
||||
# reported; peak VRAM per arm (FAIL if > 95% of the card). Compares MEDIANS of every
|
||||
|
||||
Reference in new issue
Block a user