Files
llama.cpp/docs/fork-rebase-runbook.md
T
LumpiastyandClaude Opus 5.5 263d75cba3 docs: runbook - new fork-validate checks, gdb gone, scratch lost on restart
Phase 1 now covers the VRAM release/restore check and the handoff latency
A/B against --ref (which must be the production build); phase 2 adds
--moe-exact on the 35B and keeps its scratch dir. Explains median gating and
--perf-report, that a pod restart empties /tmp, that the manual checks are
now automated, and that gdb is not in the pod image (lost in the 2026-10-24
restart). States the commit trailer convention and what llama-swap does with
a child that exits vs one that wedges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00

436 lines
25 KiB
Markdown

# Fork rebase and deploy runbook (RX 580 fork)
Step-by-step procedure for moving the fork onto a new upstream and putting it into production, as it
has evolved over the 2026-09-09, 2026-10-09 and 2026-10-24 rebases. Read [FORK.md](../FORK.md) first:
it lists every fork feature with its invariants and rebase hazards, and this runbook refers to it.
Ground rules:
- **Never build on the workstation.** All builds run on the RX 580 pod. Locally, rely on careful reading.
- **Never resolve a conflict with `-X theirs` or `-X ours`.** Resolve every hunk by hand.
- **"Applied cleanly" is not "correct".** Re-read every clean pick against the new upstream.
- **Never `pkill -f` on the pod.** It matches the `kubectl exec` shell and kills it. Kill by explicit PID.
- **Never leave the pod not serving.** Anything that stops llama-swap must restart it from a trap.
- Keep the previous production branch untouched as the safety net. Do not push for the owner; the owner
reviews and pushes.
Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a compiler; 5. Build on
the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU;
11. Operations reference.
---
## 1. Survey upstream
Local remotes: `origin` is upstream (github.com/ggerganov/llama.cpp), `gitea` is the fork
(git@gitea.lumpiasty.xyz:Lumpiasty/llama.cpp.git, SSH push from the workstation). On the pod,
`origin` is the same gitea repository over HTTPS (fetch only).
```sh
git fetch origin
OLD_BASE=$(git merge-base origin/master HEAD) # the old upstream base (still an ancestor of the new master)
OLD_TIP=$(git rev-parse HEAD)
git log --oneline "$OLD_BASE"..HEAD # the fork commits, oldest last
git log --oneline "$OLD_BASE"..origin/master | wc -l # size of the upstream delta
```
Then look at upstream changes in the areas FORK.md lists as hazards:
```sh
git log --oneline "$OLD_BASE"..origin/master -- \
src/llama-context.cpp src/llama-context.h src/llama-model.cpp src/llama-model.h \
src/llama-kv-cache.cpp src/llama-kv-cache.h src/llama-kv-cache-iswa.cpp \
src/llama-memory.h src/llama-memory-recurrent.cpp src/llama-memory-hybrid.cpp \
src/llama-moe-cache.cpp src/llama-moe-cache.h src/models/delta-net-base.cpp \
ggml/src/ggml-vulkan/ ggml/src/ggml-backend.cpp ggml/src/ggml-backend-impl.h ggml/include/ggml-backend.h \
tools/server/server-queue.cpp tools/server/server-queue.h tools/server/server-context.cpp tools/server/server.cpp \
tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/
```
For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past
decisions:
- 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional `-ncmoe` (all measured
losses or no win on this card, FORK.md section 4).
- 2026-10-09: the routing-ids readback skip had nothing to apply to (upstream #29943 deleted its block
in `ggml-backend.cpp`); it was re-authored in `llama_context::sched_copy_experts`. Upstream's new MoE
cache (#29887) needed a new fork commit so the VRAM arbiter releases and invalidates it.
- Also check CLI changes that break production config or the scripts (upstream #28334 removed
`--no-mmap`/`--mmap`/`--mlock` and llama-bench `-mmp`; the default `--port` changed to 9931).
## 2. Trial rebase in a scratch worktree
Measure the conflicts before doing the real thing:
```sh
git worktree add /tmp/trial-rebase origin/master
cd /tmp/trial-rebase
git cherry-pick "$OLD_BASE".."$OLD_TIP" # stop at each conflict, note file + hunk, then:
git cherry-pick --abort # or resolve roughly to see the next one
cd - && git worktree remove --force /tmp/trial-rebase
```
Write down, per commit: clean / conflict (which file, which hunk, what both sides did). That list
becomes the brief for the real rebase.
## 3. The real rebase
```sh
NEW=fable5/vram-ondemand-rebase-$(date +%Y%m%d)
git switch -c "$NEW" origin/master
git cherry-pick <first-fork-commit> # one at a time, in the original order
```
Rules:
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block),
unless the owner asks for something else.
- Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms.
- Fold each fix into the commit it belongs to so every commit builds on its own:
`git commit --fixup=<sha>` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`.
- Leave the previous production branch untouched.
**What `-X theirs` silently broke in a trial (2026-10-09).** It "applied" every commit and produced
broken code:
- resurrected the scheduler readback block upstream had deleted from `ggml-backend.cpp`;
- dropped `!use_sparse` from the GCN `use_mask_opt` condition;
- dropped upstream's `copy_experts_info` (compile error);
- dropped upstream's `is_empty()` in the recurrent memory;
- dropped upstream's MoE-cache memory breakdown block;
- left a static/extern mismatch with the new `ggml-vulkan-common.h` declarations (compile error).
The correct resolutions in that rebase: keep upstream's `n_embd` width check plus our decode
auto-restore; `copy_experts_info` plus our `kv_device_evicted`; `is_empty()` plus our `dev_shadows`;
upstream's `if (moe_cache)` breakdown block outside our `if (sched)` guard; make `ggml_vk_concat` and
`ggml_backend_vk_free` non-static to match the new header while our helpers stay static.
**Clean picks that were wrong (2026-10-24).**
- Concat fast path (3df719e1e): applied cleanly, did not compile. Upstream `733f57a99` renamed
`pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to `pipeline_cpy_32`/`pipeline_cpy_16` (bitwise copies,
same push constants).
- MoE cache release/restore (7ac2014a8): applied cleanly, did not compile (upstream `33d6135cb` changed
LRU to SLRU, `g.lru` -> `g.slru`) and had a runtime bug: `moe_cache_slru::init()` resets only the
probation segment, so re-initialising a used cache corrupts its lists. `invalidate()` now assigns a
fresh `moe_cache_slru()` first.
- VRAM sharing core (481be8b28): besides its one conflict, upstream's new `load_tensors_hadamard()`
pushes two more entries into `pimpl->ctxs_bufs`. Without matching `weight_release.emplace_back()`
calls, `release_device_weights()` reads past the end of `weight_release`. Nothing conflicted.
- Conflicts that round: `ctx_buf_maps.emplace_back` replaced upstream by `ctx_bufs.emplace_back(ctx,
std::move(buf_ptrs))` (keep upstream's line, our release-slot block before it); `llama-context.h`
(keep upstream's `n_reallocs_same/diff` beside our `kv_device_evicted`); `recv_with_timeout`
(upstream `4e201388c` uses a fixed deadline: keep our order "check `!running`, return null", then
`wait_until(lock, deadline)`).
**Semantic re-checks to do for every rebase, clean or not** (details in FORK.md):
- F1: `sched_copy_experts` and its `st` struct unchanged in meaning.
- F2: `use_mask_opt` still contains upstream's terms plus our `>= 256`.
- F3: `build_conv_state` still emits `ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0)` with
`qkv_mixed` contiguous `[ch, n_seq_tokens, n_seqs]`; pipeline names; `types_ok` still excludes
quantized types.
- F4: every `ctxs_bufs` push in `src/llama-model.cpp` has a `weight_release` push
(`git grep -n "ctxs_bufs.emplace_back\|weight_release" src/llama-model.cpp`); list memory types
(`git grep -n "public llama_memory_i" src/`) and note any without a release override; MoE-cache
internals; `sched_reserve()` still goes through `sched_new()` and re-registers callbacks;
`free_scratch` still the last `ggml_backend_i` member; Vulkan prealloc fields match `ggml_vk_cleanup`.
- F6/F7: `server_queue::start_loop` still evaluates `should_sleep()` only from the idle branch, never
inside `yield_to_queue()`; `handle_sleeping_state`, `server_res_generator`, `load_model` ordering;
signal setup in `server.cpp`.
- F8: new iterators over `ctxs_bufs` or users of `sched` on teardown paths.
- Scripts: flag spellings used by `scripts/rx580-bench/*` and `scripts/fork-validate.sh`, and the log
lines `fork-validate.sh` greps for (FORK.md F9).
- Production config: every flag in `/root/config.yaml` still exists in `common/arg.cpp`.
## 4. Checks without a compiler
Before handing the branch over for a build:
```sh
git diff origin/master..HEAD | grep -n '^[+-]\(<<<<<<<\|=======\|>>>>>>>\)' # no conflict markers
git diff origin/master..HEAD | grep -nP '^\+.*[^\x00-\x7F]' # ASCII only in added lines
git range-diff "$OLD_BASE".."$OLD_TIP" origin/master..HEAD # patch vs patch
```
- The range-diff must show only the edits you made on purpose. In 2026-10-24 the whole patch was
identical to the old one except for 4 deliberate edits.
- Brace balance per changed file: the change in `{`/`}` balance from new base to new tip should equal
the change from old base to old tip.
- Every identifier in added lines must exist on the new master or be defined by the fork. This catches
a renamed function but not a renamed member (like `lru` -> `slru`); re-read member accesses by hand.
- Write down every spot where you would normally lean on the compiler and hand that list to whoever
builds.
Then the owner pushes the branch to gitea.
## 5. Build on the pod, in a scratch worktree
Production keeps serving during this step.
```sh
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '
set -e
B=fable5/vram-ondemand-rebase-YYYYMMDD
W=/root/arms/rebYYYYMMDD
git -C /root/llama.cpp fetch origin "$B"
git -C /root/llama.cpp worktree add "$W" FETCH_HEAD
cd "$W"
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF
nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8
'
```
- Build only the targets you need; the full `all` target fails on the UI assets (needs npm/network).
If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed).
- `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes.
- Report compile errors verbatim against the list from step 4.
- Do not overlap a build with a large benchmark (host OOM). Disk is limited (about 22 GB free; each
build tree is about 700 MB); remove scratch worktrees you no longer need with
`git -C /root/llama.cpp worktree remove <dir>`.
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
holding a `kubectl exec` open for hours.
## 6. Validate with `scripts/fork-validate.sh`
Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of
the script describes each check.
```sh
# phase 1, beside production (needs >= 1.5 GiB VRAM free for --arbiter; it refuses otherwise)
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat --arbiter
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
cat > /root/arms/fv-perf.sh <<'EOF'
Q35=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --perf --moe-exact "$Q35" \
--moecache /root/.cache/huggingface/hub/<gemma-4-26B-A4B snapshot>/<file>.gguf --keep
EOF
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
```
- `/root/bench/moe/swapdown.sh <script>` (pod-only helper, not in the repo) stops llama-swap, waits for
an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints
`SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200.
- The `--concat` check needs a build with the `GGML_VK_CONCAT_FASTPATH` knob (in the fork since
e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`).
- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits,
which proves nothing).
- Phase 1 now also checks that a cold server really frees its VRAM (`v-vram-release`,
`v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs
ordinary handoff latency against `--ref` (`a-handoff-latency`, about 1.5 extra minutes). `--ref`
must be the build currently in production, so both comparisons are against what serves today.
- Phase 2 costs about 27 minutes of downtime for `--perf` plus `--moecache`, and `--moe-exact` adds
three loads of the 35B (estimated 3-5 minutes; not yet timed). It leaves tens of GB of model files
in page cache. `--moe-exact` proves the routing-ids readback skip exact (F1): the candidate's
greedy output equals the reference's and, with the `LLAMA_MOE_IDS_SKIP` knob (any build of this
branch), the candidate's with the skip forced off.
- `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- Last full run (2026-10-24, 33 PASS, 0 FAIL; old script, means of 2 arm means, no `--moe-exact`):
perf candidate vs reference pp2048 -4.1%, -2.2% at d16k, -0.7% at d32k (gate is -5%),
tg256@d4096 -0.2%. The new phase-1 checks first ran on 2026-10-25 (reb1024 against itself beside
production): 16 PASS; VRAM warm +546 / cold +4 / restored +546 MiB; handoff median 1050.5 vs 1082.5 ms.
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
regression gate, not a measurement.
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
## 7. Deploy
Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 minutes.
1. **Rollback pointer.** Record the current production commit in the next free pointer file:
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 9 (holds 5751c977b)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-10.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea):
```sh
cd /root/llama.cpp
git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD
git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD <validated-commit>
```
3. **Memory.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM victim. With
`-lm none` each load reads the GGUF through the page cache, and a recent benchmark leaves tens of GB
of model files cached. During the first 2026-10-24 deploy the Talos OOM controller killed the whole
pod 17 s into verification (cause likely, not proven, memory pressure). Before stopping llama-swap,
evict the model files' page cache (unprivileged, safe while servers run: `-lm none` servers hold
their weights in anonymous memory, and pages mapped by an mmap model are not dropped) and check
memory:
```sh
python3 - <<'EOF'
import glob, os
n = 0
for root in ('/root/.cache/huggingface/hub', '/root/models'):
for p in glob.glob(root + '/**/*.gguf', recursive=True):
fd = os.open(p, os.O_RDONLY)
try:
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
n += 1
finally:
os.close(fd)
print('evicted', n, 'files')
EOF
grep -E 'MemAvailable|MemFree' /proc/meminfo
```
Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by
models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache
from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the
container is the HOST root cgroup; the pod's own cgroup is under
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/`.
4. **Stop llama-swap and reap orphans.**
```sh
supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,ppid,etimes,args | grep '[l]lama-server'
kill -TERM <pid> <pid> # explicit PIDs only, never pkill -f
ps -eo pid,args | grep '[l]lama-server' # must be empty
```
`supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1),
one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM.
5. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
```sh
cd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12
```
Check that the libraries in `build/bin` have fresh mtimes.
6. **Start and probe.**
```sh
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200
```
Sample `/proc/meminfo` every few seconds during the first model loads.
## 8. Post-deploy checks
All through llama-swap on :8080. A request helper:
```sh
ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; }
holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; }
```
1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers;
`holder` shows the 35B's PID (check with `ps -o args -p <pid>`).
2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is
correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch
(2026-10-24: 3.1-5.6 s).
3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF:UD-Q4_K_XL` (a small
solid-colour image, ask for the colour). This also forces an eviction in the llama-swap matrix: the
evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed,
releasing the GPU token`, with no kill escalation.
4. Text to gemma and to the 4B after the eviction; `holder` correct each time.
5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931).
6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for
`GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none.
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`); pod restart count and Talos OOM events unchanged.
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with
`gdb -p <pid> -batch -ex 'thread apply all bt'` before killing it. gdb is not in
the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check
`command -v gdb` and reinstall it first if needed.
## 9. Rollback
```sh
supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
cd /root/llama.cpp
git checkout -B <previous-production-branch> "$(cat /root/DEPLOY-ROLLBACK-commit-N.txt)"
cmake --build build --target llama-server llama-bench -j 12
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
```
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in
production. Keep the candidate worktree for a later redeploy.
Pointer files 2 to 9 exist; for example commit-7 holds `e70486447`, commit-8 `64543a435`, and commit-9
`5751c977b`, the rollback target for the current production `6c480f7e7`.
## 10. Recovering a hung GPU
Symptoms: a process that survives SIGKILL (stuck in the driver); `ring sdma1 timeout` or
`ring comp_... timeout` in the kernel log; after a reboot the supervisord pod sits in FailedMount with
`/dev/kfd is not a character device` / `/dev/dri is not a directory`, and
`talosctl get pcidevices` lists no VGA device (the RX 580 and its HDMI audio function are missing).
A warm reboot does not bring the card back (classic Polaris failed reset).
The node (`anapistula-delrosalae`, Talos, bare metal) has a NanoKVM at http://192.168.255.8.
Credentials: ask the owner. Procedure (owner's instructions, follow exactly):
1. Soft shutdown: **one short press** of the power button through the NanoKVM.
2. **Never use the reset button.** It does not cut GPU power and is useless here.
3. Wait until the power LED goes off.
4. Press the power button once more to boot.
5. Only if the LED is still on 10+ minutes after the short press: one **long press of 8 seconds** to
force power off, wait for the LED to go off, then a short press to boot. With the GPU off the bus the
soft shutdown has hung before (Talos services stopped, LED stayed on), and the long press was needed.
Talos access from the workstation:
```sh
export TALOSCONFIG=/home/user/Projects/klaster/talos/generated/talosconfig
nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices
nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg
```
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active
(`/tmp/fan-control.log` shows `power1_cap ... -> 120000000`). The two 2026-10-09 hangs both came at the
first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork
and a clean-upstream build hung, so they were treated as hardware/power, not code.
## 11. Operations reference
Pod access:
```sh
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts
```
Wiring: k8s service `llama:11434` -> pod port 8080 (selector `app=llama-swap`) -> llama-swap `:8080`;
a Caddy proxy fronts `llama.lumpiasty.xyz`. llama-swap must listen on `:8080`.
Files (all under `/root`, a persistent volume):
- `/root/llama.cpp`: production tree, built in `build/` (shared libraries, so the real code is in
`build/bin/libllama.so` and `libggml-*.so`). Scratch worktrees go in `/root/arms/`.
- `/root/supervisord.conf`: supervisord (PID 1). `[program:llama-swap]` runs
`/root/llama-swap -config /root/config.yaml -listen :8080 -watch-config`; its `environment=` sets
`PATH` and `LD_LIBRARY_PATH` to `/root/llama.cpp/build/bin` and the arbiter variables (FORK.md section 5).
- `/root/config.yaml`: llama-swap models, macros (`moe_offload_args`, `moe_offload_gemma`,
`common_args`), matrix/exclusivity sets, preload. Backups `/root/config.yaml.bak-*`. Edit only with
the owner's approval.
- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program).
- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`).
- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers.
- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/`
directory from the old design may remain). Never point a test at it.
Reloads: a `config.yaml` change is picked up by `-watch-config`, but a running model keeps its old
arguments until it restarts (`supervisorctl -c /root/supervisord.conf restart llama-swap`). An
`environment=` change needs `supervisorctl -c /root/supervisord.conf reread && supervisorctl -c
/root/supervisord.conf update`.
Useful commands:
- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`.
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the
last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log).
Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2).
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do
not transfer: about 2.25x the RX 580's prefill, 13% slower decode, and the GCN-only mask_opt change does
not apply. Do not change its system config.
Not a fork problem: OpenWebUI's stop button does not cancel generation (its Redis task-command listener
never handles the stop); llama-server and llama-swap both cancel correctly on client disconnect.