Phase 1 now covers the VRAM release/restore check and the handoff latency A/B against --ref (which must be the production build); phase 2 adds --moe-exact on the 35B and keeps its scratch dir. Explains median gating and --perf-report, that a pod restart empties /tmp, that the manual checks are now automated, and that gdb is not in the pod image (lost in the 2026-10-24 restart). States the commit trailer convention and what llama-swap does with a child that exits vs one that wedges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
436 lines
25 KiB
Markdown
436 lines
25 KiB
Markdown
# Fork rebase and deploy runbook (RX 580 fork)
|
|
|
|
Step-by-step procedure for moving the fork onto a new upstream and putting it into production, as it
|
|
has evolved over the 2026-09-09, 2026-10-09 and 2026-10-24 rebases. Read [FORK.md](../FORK.md) first:
|
|
it lists every fork feature with its invariants and rebase hazards, and this runbook refers to it.
|
|
|
|
Ground rules:
|
|
|
|
- **Never build on the workstation.** All builds run on the RX 580 pod. Locally, rely on careful reading.
|
|
- **Never resolve a conflict with `-X theirs` or `-X ours`.** Resolve every hunk by hand.
|
|
- **"Applied cleanly" is not "correct".** Re-read every clean pick against the new upstream.
|
|
- **Never `pkill -f` on the pod.** It matches the `kubectl exec` shell and kills it. Kill by explicit PID.
|
|
- **Never leave the pod not serving.** Anything that stops llama-swap must restart it from a trap.
|
|
- Keep the previous production branch untouched as the safety net. Do not push for the owner; the owner
|
|
reviews and pushes.
|
|
|
|
Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a compiler; 5. Build on
|
|
the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU;
|
|
11. Operations reference.
|
|
|
|
---
|
|
|
|
## 1. Survey upstream
|
|
|
|
Local remotes: `origin` is upstream (github.com/ggerganov/llama.cpp), `gitea` is the fork
|
|
(git@gitea.lumpiasty.xyz:Lumpiasty/llama.cpp.git, SSH push from the workstation). On the pod,
|
|
`origin` is the same gitea repository over HTTPS (fetch only).
|
|
|
|
```sh
|
|
git fetch origin
|
|
OLD_BASE=$(git merge-base origin/master HEAD) # the old upstream base (still an ancestor of the new master)
|
|
OLD_TIP=$(git rev-parse HEAD)
|
|
git log --oneline "$OLD_BASE"..HEAD # the fork commits, oldest last
|
|
git log --oneline "$OLD_BASE"..origin/master | wc -l # size of the upstream delta
|
|
```
|
|
|
|
Then look at upstream changes in the areas FORK.md lists as hazards:
|
|
|
|
```sh
|
|
git log --oneline "$OLD_BASE"..origin/master -- \
|
|
src/llama-context.cpp src/llama-context.h src/llama-model.cpp src/llama-model.h \
|
|
src/llama-kv-cache.cpp src/llama-kv-cache.h src/llama-kv-cache-iswa.cpp \
|
|
src/llama-memory.h src/llama-memory-recurrent.cpp src/llama-memory-hybrid.cpp \
|
|
src/llama-moe-cache.cpp src/llama-moe-cache.h src/models/delta-net-base.cpp \
|
|
ggml/src/ggml-vulkan/ ggml/src/ggml-backend.cpp ggml/src/ggml-backend-impl.h ggml/include/ggml-backend.h \
|
|
tools/server/server-queue.cpp tools/server/server-queue.h tools/server/server-context.cpp tools/server/server.cpp \
|
|
tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/
|
|
```
|
|
|
|
For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past
|
|
decisions:
|
|
- 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional `-ncmoe` (all measured
|
|
losses or no win on this card, FORK.md section 4).
|
|
- 2026-10-09: the routing-ids readback skip had nothing to apply to (upstream #29943 deleted its block
|
|
in `ggml-backend.cpp`); it was re-authored in `llama_context::sched_copy_experts`. Upstream's new MoE
|
|
cache (#29887) needed a new fork commit so the VRAM arbiter releases and invalidates it.
|
|
- Also check CLI changes that break production config or the scripts (upstream #28334 removed
|
|
`--no-mmap`/`--mmap`/`--mlock` and llama-bench `-mmp`; the default `--port` changed to 9931).
|
|
|
|
## 2. Trial rebase in a scratch worktree
|
|
|
|
Measure the conflicts before doing the real thing:
|
|
|
|
```sh
|
|
git worktree add /tmp/trial-rebase origin/master
|
|
cd /tmp/trial-rebase
|
|
git cherry-pick "$OLD_BASE".."$OLD_TIP" # stop at each conflict, note file + hunk, then:
|
|
git cherry-pick --abort # or resolve roughly to see the next one
|
|
cd - && git worktree remove --force /tmp/trial-rebase
|
|
```
|
|
|
|
Write down, per commit: clean / conflict (which file, which hunk, what both sides did). That list
|
|
becomes the brief for the real rebase.
|
|
|
|
## 3. The real rebase
|
|
|
|
```sh
|
|
NEW=fable5/vram-ondemand-rebase-$(date +%Y%m%d)
|
|
git switch -c "$NEW" origin/master
|
|
git cherry-pick <first-fork-commit> # one at a time, in the original order
|
|
```
|
|
|
|
Rules:
|
|
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
|
|
fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block),
|
|
unless the owner asks for something else.
|
|
- Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms.
|
|
- Fold each fix into the commit it belongs to so every commit builds on its own:
|
|
`git commit --fixup=<sha>` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`.
|
|
- Leave the previous production branch untouched.
|
|
|
|
**What `-X theirs` silently broke in a trial (2026-10-09).** It "applied" every commit and produced
|
|
broken code:
|
|
- resurrected the scheduler readback block upstream had deleted from `ggml-backend.cpp`;
|
|
- dropped `!use_sparse` from the GCN `use_mask_opt` condition;
|
|
- dropped upstream's `copy_experts_info` (compile error);
|
|
- dropped upstream's `is_empty()` in the recurrent memory;
|
|
- dropped upstream's MoE-cache memory breakdown block;
|
|
- left a static/extern mismatch with the new `ggml-vulkan-common.h` declarations (compile error).
|
|
|
|
The correct resolutions in that rebase: keep upstream's `n_embd` width check plus our decode
|
|
auto-restore; `copy_experts_info` plus our `kv_device_evicted`; `is_empty()` plus our `dev_shadows`;
|
|
upstream's `if (moe_cache)` breakdown block outside our `if (sched)` guard; make `ggml_vk_concat` and
|
|
`ggml_backend_vk_free` non-static to match the new header while our helpers stay static.
|
|
|
|
**Clean picks that were wrong (2026-10-24).**
|
|
- Concat fast path (3df719e1e): applied cleanly, did not compile. Upstream `733f57a99` renamed
|
|
`pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to `pipeline_cpy_32`/`pipeline_cpy_16` (bitwise copies,
|
|
same push constants).
|
|
- MoE cache release/restore (7ac2014a8): applied cleanly, did not compile (upstream `33d6135cb` changed
|
|
LRU to SLRU, `g.lru` -> `g.slru`) and had a runtime bug: `moe_cache_slru::init()` resets only the
|
|
probation segment, so re-initialising a used cache corrupts its lists. `invalidate()` now assigns a
|
|
fresh `moe_cache_slru()` first.
|
|
- VRAM sharing core (481be8b28): besides its one conflict, upstream's new `load_tensors_hadamard()`
|
|
pushes two more entries into `pimpl->ctxs_bufs`. Without matching `weight_release.emplace_back()`
|
|
calls, `release_device_weights()` reads past the end of `weight_release`. Nothing conflicted.
|
|
- Conflicts that round: `ctx_buf_maps.emplace_back` replaced upstream by `ctx_bufs.emplace_back(ctx,
|
|
std::move(buf_ptrs))` (keep upstream's line, our release-slot block before it); `llama-context.h`
|
|
(keep upstream's `n_reallocs_same/diff` beside our `kv_device_evicted`); `recv_with_timeout`
|
|
(upstream `4e201388c` uses a fixed deadline: keep our order "check `!running`, return null", then
|
|
`wait_until(lock, deadline)`).
|
|
|
|
**Semantic re-checks to do for every rebase, clean or not** (details in FORK.md):
|
|
- F1: `sched_copy_experts` and its `st` struct unchanged in meaning.
|
|
- F2: `use_mask_opt` still contains upstream's terms plus our `>= 256`.
|
|
- F3: `build_conv_state` still emits `ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0)` with
|
|
`qkv_mixed` contiguous `[ch, n_seq_tokens, n_seqs]`; pipeline names; `types_ok` still excludes
|
|
quantized types.
|
|
- F4: every `ctxs_bufs` push in `src/llama-model.cpp` has a `weight_release` push
|
|
(`git grep -n "ctxs_bufs.emplace_back\|weight_release" src/llama-model.cpp`); list memory types
|
|
(`git grep -n "public llama_memory_i" src/`) and note any without a release override; MoE-cache
|
|
internals; `sched_reserve()` still goes through `sched_new()` and re-registers callbacks;
|
|
`free_scratch` still the last `ggml_backend_i` member; Vulkan prealloc fields match `ggml_vk_cleanup`.
|
|
- F6/F7: `server_queue::start_loop` still evaluates `should_sleep()` only from the idle branch, never
|
|
inside `yield_to_queue()`; `handle_sleeping_state`, `server_res_generator`, `load_model` ordering;
|
|
signal setup in `server.cpp`.
|
|
- F8: new iterators over `ctxs_bufs` or users of `sched` on teardown paths.
|
|
- Scripts: flag spellings used by `scripts/rx580-bench/*` and `scripts/fork-validate.sh`, and the log
|
|
lines `fork-validate.sh` greps for (FORK.md F9).
|
|
- Production config: every flag in `/root/config.yaml` still exists in `common/arg.cpp`.
|
|
|
|
## 4. Checks without a compiler
|
|
|
|
Before handing the branch over for a build:
|
|
|
|
```sh
|
|
git diff origin/master..HEAD | grep -n '^[+-]\(<<<<<<<\|=======\|>>>>>>>\)' # no conflict markers
|
|
git diff origin/master..HEAD | grep -nP '^\+.*[^\x00-\x7F]' # ASCII only in added lines
|
|
git range-diff "$OLD_BASE".."$OLD_TIP" origin/master..HEAD # patch vs patch
|
|
```
|
|
|
|
- The range-diff must show only the edits you made on purpose. In 2026-10-24 the whole patch was
|
|
identical to the old one except for 4 deliberate edits.
|
|
- Brace balance per changed file: the change in `{`/`}` balance from new base to new tip should equal
|
|
the change from old base to old tip.
|
|
- Every identifier in added lines must exist on the new master or be defined by the fork. This catches
|
|
a renamed function but not a renamed member (like `lru` -> `slru`); re-read member accesses by hand.
|
|
- Write down every spot where you would normally lean on the compiler and hand that list to whoever
|
|
builds.
|
|
|
|
Then the owner pushes the branch to gitea.
|
|
|
|
## 5. Build on the pod, in a scratch worktree
|
|
|
|
Production keeps serving during this step.
|
|
|
|
```sh
|
|
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
|
|
kubectl -n llama exec -i deploy/supervisord -- sh -c '
|
|
set -e
|
|
B=fable5/vram-ondemand-rebase-YYYYMMDD
|
|
W=/root/arms/rebYYYYMMDD
|
|
git -C /root/llama.cpp fetch origin "$B"
|
|
git -C /root/llama.cpp worktree add "$W" FETCH_HEAD
|
|
cd "$W"
|
|
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF
|
|
nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8
|
|
'
|
|
```
|
|
|
|
- Build only the targets you need; the full `all` target fails on the UI assets (needs npm/network).
|
|
If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed).
|
|
- `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes.
|
|
- Report compile errors verbatim against the list from step 4.
|
|
- Do not overlap a build with a large benchmark (host OOM). Disk is limited (about 22 GB free; each
|
|
build tree is about 700 MB); remove scratch worktrees you no longer need with
|
|
`git -C /root/llama.cpp worktree remove <dir>`.
|
|
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
|
|
holding a `kubectl exec` open for hours.
|
|
|
|
## 6. Validate with `scripts/fork-validate.sh`
|
|
|
|
Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of
|
|
the script describes each check.
|
|
|
|
```sh
|
|
# phase 1, beside production (needs >= 1.5 GiB VRAM free for --arbiter; it refuses otherwise)
|
|
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
|
|
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat --arbiter
|
|
|
|
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
|
|
cat > /root/arms/fv-perf.sh <<'EOF'
|
|
Q35=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
|
|
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
|
|
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf --moe-exact "$Q35" \
|
|
--moecache /root/.cache/huggingface/hub/<gemma-4-26B-A4B snapshot>/<file>.gguf --keep
|
|
EOF
|
|
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
|
|
```
|
|
|
|
- `/root/bench/moe/swapdown.sh <script>` (pod-only helper, not in the repo) stops llama-swap, waits for
|
|
an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints
|
|
`SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200.
|
|
- The `--concat` check needs a build with the `GGML_VK_CONCAT_FASTPATH` knob (in the fork since
|
|
e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`).
|
|
- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits,
|
|
which proves nothing).
|
|
- Phase 1 now also checks that a cold server really frees its VRAM (`v-vram-release`,
|
|
`v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs
|
|
ordinary handoff latency against `--ref` (`a-handoff-latency`, about 1.5 extra minutes). `--ref`
|
|
must be the build currently in production, so both comparisons are against what serves today.
|
|
- Phase 2 costs about 27 minutes of downtime for `--perf` plus `--moecache`, and `--moe-exact` adds
|
|
three loads of the 35B (estimated 3-5 minutes; not yet timed). It leaves tens of GB of model files
|
|
in page cache. `--moe-exact` proves the routing-ids readback skip exact (F1): the candidate's
|
|
greedy output equals the reference's and, with the `LLAMA_MOE_IDS_SKIP` knob (any build of this
|
|
branch), the candidate's with the skip forced off.
|
|
- `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in
|
|
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
|
|
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
|
|
scratch dirs were lost that way).
|
|
- Last full run (2026-10-24, 33 PASS, 0 FAIL; old script, means of 2 arm means, no `--moe-exact`):
|
|
perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, -0.7% at d32k (gate is -5%),
|
|
tg256@d4096 -0.2%. The new phase-1 checks first ran on 2026-10-25 (reb1024 against itself beside
|
|
production): 16 PASS; VRAM warm +546 / cold +4 / restored +546 MiB; handoff median 1050.5 vs 1082.5 ms.
|
|
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
|
|
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
|
|
regression gate, not a measurement.
|
|
|
|
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
|
|
|
|
## 7. Deploy
|
|
|
|
Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 minutes.
|
|
|
|
1. **Rollback pointer.** Record the current production commit in the next free pointer file:
|
|
```sh
|
|
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 9 (holds 5751c977b)
|
|
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-10.txt
|
|
git -C /root/llama.cpp status --short # must be clean
|
|
```
|
|
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea):
|
|
```sh
|
|
cd /root/llama.cpp
|
|
git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD
|
|
git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD <validated-commit>
|
|
```
|
|
3. **Memory.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM victim. With
|
|
`-lm none` each load reads the GGUF through the page cache, and a recent benchmark leaves tens of GB
|
|
of model files cached. During the first 2026-10-24 deploy the Talos OOM controller killed the whole
|
|
pod 17 s into verification (cause likely, not proven, memory pressure). Before stopping llama-swap,
|
|
evict the model files' page cache (unprivileged, safe while servers run: `-lm none` servers hold
|
|
their weights in anonymous memory, and pages mapped by an mmap model are not dropped) and check
|
|
memory:
|
|
```sh
|
|
python3 - <<'EOF'
|
|
import glob, os
|
|
n = 0
|
|
for root in ('/root/.cache/huggingface/hub', '/root/models'):
|
|
for p in glob.glob(root + '/**/*.gguf', recursive=True):
|
|
fd = os.open(p, os.O_RDONLY)
|
|
try:
|
|
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
|
|
n += 1
|
|
finally:
|
|
os.close(fd)
|
|
print('evicted', n, 'files')
|
|
EOF
|
|
grep -E 'MemAvailable|MemFree' /proc/meminfo
|
|
```
|
|
Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by
|
|
models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache
|
|
from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the
|
|
container is the HOST root cgroup; the pod's own cgroup is under
|
|
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/`.
|
|
4. **Stop llama-swap and reap orphans.**
|
|
```sh
|
|
supervisorctl -c /root/supervisord.conf stop llama-swap
|
|
ps -eo pid,ppid,etimes,args | grep '[l]lama-server'
|
|
kill -TERM <pid> <pid> # explicit PIDs only, never pkill -f
|
|
ps -eo pid,args | grep '[l]lama-server' # must be empty
|
|
```
|
|
`supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1),
|
|
one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM.
|
|
5. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
|
|
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
|
|
```sh
|
|
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12
|
|
```
|
|
Check that the libraries in `build/bin` have fresh mtimes.
|
|
6. **Start and probe.**
|
|
```sh
|
|
supervisorctl -c /root/supervisord.conf start llama-swap
|
|
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200
|
|
```
|
|
Sample `/proc/meminfo` every few seconds during the first model loads.
|
|
|
|
## 8. Post-deploy checks
|
|
|
|
All through llama-swap on :8080. A request helper:
|
|
|
|
```sh
|
|
ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
|
|
-d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; }
|
|
holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; }
|
|
```
|
|
|
|
1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers;
|
|
`holder` shows the 35B's PID (check with `ps -o args -p <pid>`).
|
|
2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is
|
|
correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch
|
|
(2026-10-24: 3.1-5.6 s).
|
|
3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF:UD-Q4_K_XL` (a small
|
|
solid-colour image, ask for the colour). This also forces an eviction in the llama-swap matrix: the
|
|
evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed,
|
|
releasing the GPU token`, with no kill escalation.
|
|
4. Text to gemma and to the 4B after the eviction; `holder` correct each time.
|
|
5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931).
|
|
6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for
|
|
`GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none.
|
|
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged.
|
|
|
|
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
|
|
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
|
|
that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with
|
|
`gdb -p <pid> -batch -ex 'thread apply all bt'` before killing it. gdb is not in
|
|
the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check
|
|
`command -v gdb` and reinstall it first if needed.
|
|
|
|
## 9. Rollback
|
|
|
|
```sh
|
|
supervisorctl -c /root/supervisord.conf stop llama-swap
|
|
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
|
|
cd /root/llama.cpp
|
|
git checkout -B <previous-production-branch> "$(cat /root/DEPLOY-ROLLBACK-commit-N.txt)"
|
|
cmake --build build --target llama-server llama-bench -j 12
|
|
supervisorctl -c /root/supervisord.conf start llama-swap
|
|
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
|
|
```
|
|
|
|
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in
|
|
production. Keep the candidate worktree for a later redeploy.
|
|
|
|
Pointer files 2 to 9 exist; for example commit-7 holds `e70486447`, commit-8 `64543a435`, and commit-9
|
|
`5751c977b`, the rollback target for the current production `6c480f7e7`.
|
|
|
|
## 10. Recovering a hung GPU
|
|
|
|
Symptoms: a process that survives SIGKILL (stuck in the driver); `ring sdma1 timeout` or
|
|
`ring comp_... timeout` in the kernel log; after a reboot the supervisord pod sits in FailedMount with
|
|
`/dev/kfd is not a character device` / `/dev/dri is not a directory`, and
|
|
`talosctl get pcidevices` lists no VGA device (the RX 580 and its HDMI audio function are missing).
|
|
A warm reboot does not bring the card back (classic Polaris failed reset).
|
|
|
|
The node (`anapistula-delrosalae`, Talos, bare metal) has a NanoKVM at http://192.168.255.8.
|
|
Credentials: ask the owner. Procedure (owner's instructions, follow exactly):
|
|
|
|
1. Soft shutdown: **one short press** of the power button through the NanoKVM.
|
|
2. **Never use the reset button.** It does not cut GPU power and is useless here.
|
|
3. Wait until the power LED goes off.
|
|
4. Press the power button once more to boot.
|
|
5. Only if the LED is still on 10+ minutes after the short press: one **long press of 8 seconds** to
|
|
force power off, wait for the LED to go off, then a short press to boot. With the GPU off the bus the
|
|
soft shutdown has hung before (Talos services stopped, LED stayed on), and the long press was needed.
|
|
|
|
Talos access from the workstation:
|
|
|
|
```sh
|
|
export TALOSCONFIG=/home/user/Projects/klaster/talos/generated/talosconfig
|
|
nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices
|
|
nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg
|
|
```
|
|
|
|
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active
|
|
(`/tmp/fan-control.log` shows `power1_cap ... -> 120000000`). The two 2026-10-09 hangs both came at the
|
|
first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork
|
|
and a clean-upstream build hung, so they were treated as hardware/power, not code.
|
|
|
|
## 11. Operations reference
|
|
|
|
Pod access:
|
|
|
|
```sh
|
|
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
|
|
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
|
|
kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts
|
|
```
|
|
|
|
Wiring: k8s service `llama:11434` -> pod port 8080 (selector `app=llama-swap`) -> llama-swap `:8080`;
|
|
a Caddy proxy fronts `llama.lumpiasty.xyz`. llama-swap must listen on `:8080`.
|
|
|
|
Files (all under `/root`, a persistent volume):
|
|
- `/root/llama.cpp`: production tree, built in `build/` (shared libraries, so the real code is in
|
|
`build/bin/libllama.so` and `libggml-*.so`). Scratch worktrees go in `/root/arms/`.
|
|
- `/root/supervisord.conf`: supervisord (PID 1). `[program:llama-swap]` runs
|
|
`/root/llama-swap -config /root/config.yaml -listen :8080 -watch-config`; its `environment=` sets
|
|
`PATH` and `LD_LIBRARY_PATH` to `/root/llama.cpp/build/bin` and the arbiter variables (FORK.md section 5).
|
|
- `/root/config.yaml`: llama-swap models, macros (`moe_offload_args`, `moe_offload_gemma`,
|
|
`common_args`), matrix/exclusivity sets, preload. Backups `/root/config.yaml.bak-*`. Edit only with
|
|
the owner's approval.
|
|
- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program).
|
|
- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`).
|
|
- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers.
|
|
- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/`
|
|
directory from the old design may remain). Never point a test at it.
|
|
|
|
Reloads: a `config.yaml` change is picked up by `-watch-config`, but a running model keeps its old
|
|
arguments until it restarts (`supervisorctl -c /root/supervisord.conf restart llama-swap`). An
|
|
`environment=` change needs `supervisorctl -c /root/supervisord.conf reread && supervisorctl -c
|
|
/root/supervisord.conf update`.
|
|
|
|
Useful commands:
|
|
- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`.
|
|
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
|
|
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
|
|
- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the
|
|
last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log).
|
|
|
|
Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2).
|
|
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do
|
|
not transfer: about 2.25x the RX 580's prefill, 13% slower decode, and the GCN-only mask_opt change does
|
|
not apply. Do not change its system config.
|
|
|
|
Not a fork problem: OpenWebUI's stop button does not cancel generation (its Redis task-command listener
|
|
never handles the stop); llama-server and llama-swap both cancel correctly on client disconnect.
|