Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
674 lines
43 KiB
Markdown
674 lines
43 KiB
Markdown
# Fork rebase and deploy runbook (RX 580 fork)
|
|
|
|
Step-by-step procedure for moving the fork onto a new upstream and putting it into production, as it
|
|
has evolved over the 2026-09-09, 2026-10-09, 2026-10-24 and 2026-11-21 rebases. Read [FORK.md](../FORK.md) first:
|
|
it lists every fork feature with its invariants and rebase hazards, and this runbook refers to it.
|
|
|
|
Ground rules:
|
|
|
|
- **Never build on the workstation.** All builds run on the RX 580 pod. Locally, rely on careful reading.
|
|
- **Never resolve a conflict with `-X theirs` or `-X ours`.** Resolve every hunk by hand.
|
|
- **"Applied cleanly" is not "correct".** Re-read every clean pick against the new upstream.
|
|
- **Never `pkill -f` on the pod.** It matches the `kubectl exec` shell and kills it. Kill by explicit PID.
|
|
- **Never leave the pod not serving.** Anything that stops llama-swap must restart it from a trap.
|
|
- Keep the previous production branch untouched as the safety net. Do not push for the owner; the owner
|
|
reviews and pushes.
|
|
|
|
Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a compiler; 5. Build on
|
|
the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU;
|
|
11. Operations reference.
|
|
|
|
**Redeploying a new tip on the same upstream base (no rebase)**, e.g. new fork commits on the
|
|
production branch (2026-10-25: `6c480f7e7` -> `2fc9f9c35`, same branch, same base):
|
|
- Skip sections 1-3. Of section 4 only the ASCII check (on `git diff <prod>..<tip>`) and a careful
|
|
read of that diff apply; `<prod>` is `git -C /root/llama.cpp rev-parse HEAD` on the pod. There is
|
|
nothing to range-diff.
|
|
- Do sections 5-9 as written: build the tip in a scratch worktree, validate it with `--ref` = the
|
|
production build, then deploy.
|
|
- The branch name stays the same, so deploy step 2's `git checkout -B <same branch> <tip>` moves the
|
|
production branch on the pod to the tip. After that the rollback pointer file is the only record
|
|
of the old production commit on the pod, so write it first (deploy step 1).
|
|
|
|
---
|
|
|
|
## 1. Survey upstream
|
|
|
|
Local remotes: `origin` is upstream (github.com/ggerganov/llama.cpp), `gitea` is the fork
|
|
(git@gitea.lumpiasty.xyz:Lumpiasty/llama.cpp.git, SSH push from the workstation). On the pod,
|
|
`origin` is the same gitea repository over HTTPS (fetch only).
|
|
|
|
```sh
|
|
git fetch origin
|
|
OLD_BASE=$(git merge-base origin/master HEAD) # the old upstream base (still an ancestor of the new master)
|
|
OLD_TIP=$(git rev-parse HEAD)
|
|
git log --oneline "$OLD_BASE"..HEAD # the fork commits, oldest last
|
|
git log --oneline "$OLD_BASE"..origin/master | wc -l # size of the upstream delta
|
|
```
|
|
|
|
Then look at upstream changes in the areas FORK.md lists as hazards:
|
|
|
|
```sh
|
|
git log --oneline "$OLD_BASE"..origin/master -- \
|
|
src/llama-context.cpp src/llama-context.h src/llama-model.cpp src/llama-model.h \
|
|
src/llama-kv-cache.cpp src/llama-kv-cache.h src/llama-kv-cache-iswa.cpp \
|
|
src/llama-memory.h src/llama-memory-recurrent.cpp src/llama-memory-hybrid.cpp \
|
|
src/llama-moe-cache.cpp src/llama-moe-cache.h src/models/delta-net-base.cpp \
|
|
ggml/src/ggml-vulkan/ ggml/src/ggml-backend.cpp ggml/src/ggml-backend-impl.h ggml/include/ggml-backend.h \
|
|
tools/server/server-queue.cpp tools/server/server-queue.h tools/server/server-context.cpp tools/server/server.cpp \
|
|
tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/
|
|
```
|
|
|
|
**Temporary upstream carries (FORK.md F12)** come first: `git log --oneline --grep '^upstream-carry'
|
|
"$OLD_BASE"..HEAD`. For each: merged into the new upstream -> do not cherry-pick it (check that the
|
|
merged version is the carried head or newer); still open -> re-fetch the PR head and re-carry it if
|
|
it moved; closed -> ask the owner. To carry a PR (2026-11-22: #32735, #32740):
|
|
```sh
|
|
git fetch https://github.com/ggml-org/llama.cpp pull/<N>/head:pr-<N>
|
|
git cherry-pick $(git merge-base pr-<N> HEAD)..pr-<N> # the PR's commits, oldest first; resolve by hand
|
|
# squash them into one commit with the PR author, subject "upstream-carry #<N>: <PR title>", and a
|
|
# body with the PR URL, the carried head and any conflict resolution (git commit-tree on the final tree)
|
|
diff <(git diff $(git merge-base pr-<N> HEAD) pr-<N> | grep '^[+-]') <(git diff HEAD~1 HEAD | grep '^[+-]')
|
|
```
|
|
The last diff shows only the lines you changed while resolving conflicts (for #32740: none, the F2
|
|
line differs only in context). Validate a carry like any compiled change (sections 5-6), plus the
|
|
PR's own tests.
|
|
|
|
For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past
|
|
decisions:
|
|
- 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional `-ncmoe` (all measured
|
|
losses or no win on this card, FORK.md section 4).
|
|
- 2026-10-09: the routing-ids readback skip had nothing to apply to (upstream #29943 deleted its block
|
|
in `ggml-backend.cpp`); it was re-authored in `llama_context::sched_copy_experts`. Upstream's new MoE
|
|
cache (#29887) needed a new fork commit so the VRAM arbiter releases and invalidates it.
|
|
- Also check CLI changes that break production config or the scripts (upstream #28334 removed
|
|
`--no-mmap`/`--mmap`/`--mlock` and llama-bench `-mmp`; the default `--port` changed to 9931).
|
|
|
|
## 2. Trial rebase in a scratch worktree
|
|
|
|
Measure the conflicts before doing the real thing:
|
|
|
|
```sh
|
|
git worktree add /tmp/trial-rebase origin/master
|
|
cd /tmp/trial-rebase
|
|
git cherry-pick "$OLD_BASE".."$OLD_TIP" # stop at each conflict, note file + hunk, then:
|
|
git cherry-pick --abort # or resolve roughly to see the next one
|
|
cd - && git worktree remove --force /tmp/trial-rebase
|
|
```
|
|
|
|
Write down, per commit: clean / conflict (which file, which hunk, what both sides did). That list
|
|
becomes the brief for the real rebase.
|
|
|
|
## 3. The real rebase
|
|
|
|
```sh
|
|
NEW=fable5/vram-ondemand-rebase-$(date +%Y%m%d)
|
|
git switch -c "$NEW" origin/master
|
|
git cherry-pick <first-fork-commit> # one at a time, in the original order
|
|
```
|
|
|
|
Rules:
|
|
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
|
|
fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block),
|
|
unless the owner asks for something else.
|
|
- Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms.
|
|
- Fold each fix into the commit it belongs to so every commit builds on its own:
|
|
`git commit --fixup=<sha>` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`.
|
|
- Leave the previous production branch untouched.
|
|
|
|
**What `-X theirs` silently broke in a trial (2026-10-09).** It "applied" every commit and produced
|
|
broken code:
|
|
- resurrected the scheduler readback block upstream had deleted from `ggml-backend.cpp`;
|
|
- dropped `!use_sparse` from the GCN `use_mask_opt` condition;
|
|
- dropped upstream's `copy_experts_info` (compile error);
|
|
- dropped upstream's `is_empty()` in the recurrent memory;
|
|
- dropped upstream's MoE-cache memory breakdown block;
|
|
- left a static/extern mismatch with the new `ggml-vulkan-common.h` declarations (compile error).
|
|
|
|
The correct resolutions in that rebase: keep upstream's `n_embd` width check plus our decode
|
|
auto-restore; `copy_experts_info` plus our `kv_device_evicted`; `is_empty()` plus our `dev_shadows`;
|
|
upstream's `if (moe_cache)` breakdown block outside our `if (sched)` guard; make `ggml_vk_concat` and
|
|
`ggml_backend_vk_free` non-static to match the new header while our helpers stay static.
|
|
|
|
**Clean picks that were wrong (2026-10-24).**
|
|
- Concat fast path (3df719e1e): applied cleanly, did not compile. Upstream `733f57a99` renamed
|
|
`pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to `pipeline_cpy_32`/`pipeline_cpy_16` (bitwise copies,
|
|
same push constants).
|
|
- MoE cache release/restore (7ac2014a8): applied cleanly, did not compile (upstream `33d6135cb` changed
|
|
LRU to SLRU, `g.lru` -> `g.slru`) and had a runtime bug: `moe_cache_slru::init()` resets only the
|
|
probation segment, so re-initialising a used cache corrupts its lists. `invalidate()` now assigns a
|
|
fresh `moe_cache_slru()` first.
|
|
- VRAM sharing core (481be8b28): besides its one conflict, upstream's new `load_tensors_hadamard()`
|
|
pushes two more entries into `pimpl->ctxs_bufs`. Without matching `weight_release.emplace_back()`
|
|
calls, `release_device_weights()` reads past the end of `weight_release`. Nothing conflicted.
|
|
- Conflicts that round: `ctx_buf_maps.emplace_back` replaced upstream by `ctx_bufs.emplace_back(ctx,
|
|
std::move(buf_ptrs))` (keep upstream's line, our release-slot block before it); `llama-context.h`
|
|
(keep upstream's `n_reallocs_same/diff` beside our `kv_device_evicted`); `recv_with_timeout`
|
|
(upstream `4e201388c` uses a fixed deadline: keep our order "check `!running`, return null", then
|
|
`wait_until(lock, deadline)`).
|
|
|
|
**What the 2026-11-21 rebase (upstream `e6d624b9a`, 757 commits past `3f03b1772`) changed** (nothing
|
|
built or run locally; every item below is by reading only and needs the pod build):
|
|
- Upstream refactors that moved fork code (hand ports, not conflicts of one hunk):
|
|
- `clip.cpp`/`clip.h` are gone (#31865). The encoder release/restore (F5) is now
|
|
`mtmd_model_release_device`/`mtmd_model_restore_device` on `mtmd_model` (`mtmd-model.cpp`), called
|
|
from `mtmd_release_device`/`mtmd_restore_device` under `compute_mutex` after syncing the shared
|
|
scheduler; the page-cache drop of F10 moved to `mtmd_model_loader::load_tensors`.
|
|
- `server_prompt_cache` moved to `server-slot-sched.{h,cpp}` and `get_available_slot()` was replaced
|
|
by `server_slot_sched::select()`. The disk prompt cache (F11) lives there now
|
|
(`server_prompt_disk_cache`), is called from the task dispatch after `update_prompt_cache()`, and
|
|
uses `common_state_data` and `shared_ptr<const common_prompt_checkpoint>`. Only prompts made with
|
|
the server lora config are saved, and a restored prompt gets that config in `prompt.lora`.
|
|
- Upstream's `llama_memory_bufs` replaced the raw `ctxs_bufs` vectors of the KV cache and the recurrent
|
|
memory: the null-buffer guard moved into `llama_memory_bufs::memory_breakdown()`, and the release
|
|
code indexes `ctxs_bufs.bufs[i]`. Those two did not compile after a clean pick.
|
|
- The MoE cache is now a ClassLRU (`moe_cache_lru`, `g.lru`) with `no_alloc` as a constructor
|
|
argument and `cpu_share`. `invalidate()` rebuilds `g.lru`, refills every slot map with
|
|
`g.lru.empty` (not -1), clears the CPU maps and `cpu_experts`.
|
|
- `server.cpp` was split into `server_run_instance` and friends: the self-pipe shutdown thread
|
|
(F7) is started in `server_run_instance` only, just before `server_install_signal_handlers()`.
|
|
- `server_queue` got `n_inflight` (#32133). `should_sleep` checks it first, so a response that is still
|
|
open also blocks the want-lock yield. This is a decision, not an accident: check it with `--arbiter`.
|
|
- Upstream added its own tiled-transpose concat (#30591) with no size gate. The fork path still runs
|
|
first (`ne0 >= 32`); the gate and the `GGML_VK_CONCAT_FASTPATH` knob were added to upstream's selector
|
|
(`ggml_vk_concat_is_transposed`), otherwise decode would take upstream's tiled path again (F3).
|
|
- Memory types with no release override (stay in VRAM when cold, not used in production):
|
|
`llama_kv_cache_dsa`, `_dsa_iswa`, `_dsv4`, `_msa`, `llama_memory_hybrid_iswa`. The KV rotation
|
|
tensors (`attn_rot_bufs`, #31073) are small and are not released either.
|
|
|
|
Open questions from that rebase:
|
|
- Is the fork's own concat fast path (F3) now redundant with upstream's tiled concat (#30591)?
|
|
Benchmark `GGML_VK_CONCAT_FASTPATH=0` against the default on pp and decode before deciding.
|
|
- Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no: handoff median
|
|
candidate 692 ms vs reference 1068 ms (`a-handoff-latency`, 2026-11-21).
|
|
|
|
**Semantic re-checks to do for every rebase, clean or not** (details in FORK.md):
|
|
- F1: `sched_copy_experts` and its `st` struct unchanged in meaning.
|
|
- F2: `use_mask_opt` still contains upstream's terms plus our `>= 256` (and `!bias &&` from #32740, carried as F12 or merged upstream).
|
|
- F3: `build_conv_state` still emits `ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0)` with
|
|
`qkv_mixed` contiguous `[ch, n_seq_tokens, n_seqs]`; pipeline names; `types_ok` still excludes
|
|
quantized types.
|
|
- F4: every `ctxs_bufs` push in `src/llama-model.cpp` has a `weight_release` push
|
|
(`git grep -n "ctxs_bufs.emplace_back\|weight_release" src/llama-model.cpp`); list memory types
|
|
(`git grep -n "public llama_memory_i" src/`) and note any without a release override; MoE-cache
|
|
internals; `sched_reserve()` still goes through `sched_new()` and re-registers callbacks;
|
|
`free_scratch` still the last `ggml_backend_i` member; Vulkan prealloc fields match `ggml_vk_cleanup`.
|
|
- F6/F7: `server_queue::start_loop` still evaluates `should_sleep()` only from the idle branch, never
|
|
inside `yield_to_queue()`; `handle_sleeping_state`, `server_res_generator`, `load_model` ordering;
|
|
signal setup in `server.cpp`.
|
|
- F8: new iterators over `ctxs_bufs` or users of `sched` on teardown paths.
|
|
- Scripts: flag spellings used by `scripts/rx580-bench/*` and `scripts/fork-validate.sh`, and the log
|
|
lines `fork-validate.sh` greps for (FORK.md F9).
|
|
- Production config: every flag in `/root/config.yaml` still exists in `common/arg.cpp`.
|
|
|
|
## 4. Checks without a compiler
|
|
|
|
Before handing the branch over for a build:
|
|
|
|
```sh
|
|
git diff origin/master..HEAD | grep -n '^[+-]\(<<<<<<<\|=======\|>>>>>>>\)' # no conflict markers
|
|
git diff origin/master..HEAD | grep -nP '^\+.*[^\x00-\x7F]' # ASCII only in added lines
|
|
git range-diff "$OLD_BASE".."$OLD_TIP" origin/master..HEAD # patch vs patch
|
|
```
|
|
|
|
- The range-diff must show only the edits you made on purpose. In 2026-10-24 the whole patch was
|
|
identical to the old one except for 4 deliberate edits.
|
|
- Brace balance per changed file: the change in `{`/`}` balance from new base to new tip should equal
|
|
the change from old base to old tip.
|
|
- Every identifier in added lines must exist on the new master or be defined by the fork. This catches
|
|
a renamed function but not a renamed member (like `lru` -> `slru`); re-read member accesses by hand.
|
|
- Write down every spot where you would normally lean on the compiler and hand that list to whoever
|
|
builds.
|
|
|
|
Then the owner pushes the branch to gitea, and the pod fetches it from there (section 5).
|
|
|
|
**If you were told not to push**, do not stop at an unbuilt branch. Unless the task says otherwise, a
|
|
rebase task continues through the build and validation (sections 5 and 6) and reports the results;
|
|
the owner pushes afterwards. Get the branch onto the pod with a git bundle:
|
|
```sh
|
|
# workstation; <base> is a commit the pod already has, e.g. production's HEAD
|
|
git bundle create /tmp/x.bundle <base>..<branch>
|
|
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
|
|
kubectl -n llama exec -i deploy/supervisord -- sh -c 'cat > /root/x.bundle' < /tmp/x.bundle
|
|
kubectl -n llama exec deploy/supervisord -- sh -c 'cd /root/llama.cpp && git fetch /root/x.bundle <branch>:<branch>'
|
|
```
|
|
The branch is then local in `/root/llama.cpp`: in section 5 skip the `git fetch origin` and use
|
|
`git worktree add "$W" <branch>`. Delete `/root/x.bundle` afterwards (disk is tight). Deploy step 2
|
|
likewise uses the local branch instead of fetching from origin.
|
|
|
|
## 5. Build on the pod, in a scratch worktree
|
|
|
|
Production keeps serving during this step.
|
|
|
|
```sh
|
|
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
|
|
kubectl -n llama exec -i deploy/supervisord -- sh -c '
|
|
set -e
|
|
B=fable5/vram-ondemand-rebase-YYYYMMDD
|
|
W=/root/arms/rebYYYYMMDD
|
|
git -C /root/llama.cpp fetch origin "$B"
|
|
git -C /root/llama.cpp worktree add "$W" FETCH_HEAD
|
|
cd "$W"
|
|
export HF_UI_VERSION=b11906 # production's: cut -d'|' -f2 /root/llama.cpp/build/tools/ui/.ui-stamp
|
|
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF
|
|
nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8
|
|
'
|
|
```
|
|
|
|
- Build only the targets you need; the full `all` target fails on the UI assets (needs npm/network).
|
|
If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed).
|
|
- `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes.
|
|
- Report compile errors verbatim against the list from step 4.
|
|
- Do not overlap a build with a large benchmark (host OOM). Disk is tight: `/root` had about 13 GB
|
|
free (97% full) on 2026-10-25, so check `df -h /root` first. Each build tree is about 700 MB;
|
|
remove scratch worktrees you no longer need with `git -C /root/llama.cpp worktree remove <dir>`.
|
|
- `-DLLAMA_BUILD_UI=OFF` does NOT make the UI asset step harmless. The `llama-ui-assets` step still
|
|
runs, and when the build number has no HF tag it falls back to `latest`. On 2026-11-21 `latest`
|
|
broke the build (missing `bundle[hash].js`, `.css` and `workbox[hash].js`). Set
|
|
`HF_UI_VERSION=<stamp>` for both `cmake -B build` and `cmake --build`, where the stamp is
|
|
production's current one in `build/tools/ui/.ui-stamp` (b11906 on 2026-11-21). The file holds
|
|
`ggml-org/llama-ui|b11906` and CMake requires `^[A-Za-z0-9._-]+$`, so take only the tag with
|
|
`cut -d'|' -f2 build/tools/ui/.ui-stamp`; never pass `$(cat ...)`. The validated
|
|
worktree's UI assets may then differ from the deployed ones (the code is identical).
|
|
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
|
|
holding a `kubectl exec` open for hours.
|
|
- The validated build must contain all three targets: `llama-server`, `llama-bench` and
|
|
`test-backend-ops` (`--perf` and `--ops` run the last two from `--build`). A build script that
|
|
makes only `llama-server` (as the 2026-10-25 `diskcache-build.sh` first did) is not a validated
|
|
build.
|
|
- Detach only one script. Upload it first (`kubectl exec -i ... sh -c 'cat > /root/arms/X.sh' <
|
|
X.sh`), then start it alone: `setsid nohup sh /root/arms/X.sh > X.log 2>&1 </dev/null &`. Do not
|
|
background a chain such as `cat > X && sh -n X && setsid ... &` inside the `kubectl exec` shell. The
|
|
chain is a child of that session, and it can die with the session before `setsid` runs.
|
|
|
|
## 6. Validate with `scripts/fork-validate.sh`
|
|
|
|
Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of
|
|
the script describes each check.
|
|
|
|
Rule: any test of a production model must use production's exact command line: the `cmd` of its
|
|
`/root/config.yaml` entry with the macros expanded, including the mmproj that `-hf` pulls (production
|
|
never passes `--no-mmproj` to the 35B or gemma), and supervisord's `environment=`. Change only
|
|
`--port`, the arena and scratch paths. The 2026-10-25 deploy of the disk prompt cache aborted on the
|
|
first request: every test had used `--no-mmproj`, which hid a `GGML_ASSERT(!has_mtmd)`, and it was
|
|
rolled back (FORK.md F11).
|
|
|
|
```sh
|
|
# phase 1, beside production
|
|
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
|
|
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat
|
|
```
|
|
|
|
Add `--arbiter` to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus the bytes in
|
|
`cat /sys/class/drm/card*/device/mem_info_vram_used`); below that it FAILs at once. With gemma-26B
|
|
warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run
|
|
`--arbiter` in the phase-2 swapdown window instead (about 2 minutes more downtime), as below.
|
|
|
|
**Page cache before and after phase 2.** The `--ref` build may lack F10 (FORK.md), and `--perf`
|
|
loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the
|
|
eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check
|
|
`/proc/meminfo`.
|
|
|
|
```sh
|
|
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
|
|
cat > /root/arms/fv-perf.sh <<'EOF'
|
|
HUB=/root/.cache/huggingface/hub
|
|
Q35=$HUB/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
|
|
G26=$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/snapshots/e6759b29958180a998f62f991e4c741fe9b4d452/gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf
|
|
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
|
|
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build \
|
|
--arbiter --perf --moe-exact "$Q35" --moecache "$G26" --keep
|
|
EOF
|
|
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
|
|
```
|
|
|
|
- `G26` is the production SC117 gemma-26B, the `refs/main` snapshot (`e6759b29...` on 2026-10-25;
|
|
an older snapshot `82c751632...` is also on disk). If it moves, read the snapshot name from
|
|
`$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/refs/main`.
|
|
- `/root/bench/moe/swapdown.sh <script>` (pod-only helper, not in the repo) stops llama-swap, waits for
|
|
an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints
|
|
`SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200.
|
|
- The `--concat` check needs a build with the `GGML_VK_CONCAT_FASTPATH` knob (in the fork since
|
|
e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`).
|
|
- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits,
|
|
which proves nothing).
|
|
- `--arbiter` also checks that a cold server really frees its VRAM (`v-vram-release`,
|
|
`v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs
|
|
ordinary handoff latency against `--ref` (`a-handoff-latency`). `--ref` must be the build currently
|
|
in production, so both comparisons are against what serves today.
|
|
- Phase 2 downtime, measured 2026-10-25: about 32 minutes in all (`--arbiter` about 2 min, `--perf`
|
|
about 28 min, `--moecache` under 30 s, `--moe-exact` 78 s). `--moe-exact` proves the routing-ids
|
|
readback skip exact (F1): the candidate's greedy output equals the reference's and, with the
|
|
`LLAMA_MOE_IDS_SKIP` knob (any build of this branch), the candidate's with the skip forced off.
|
|
- `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in
|
|
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
|
|
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
|
|
scratch dirs were lost that way).
|
|
- Last full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
|
|
9 PASS plus the 35B disk-cache legs (`q35.py`, mmproj loaded). Perf medians ref -> cand: pp2048
|
|
259.39 -> 286.61 (+10.5%), d16384 199.26 -> 207.63 (+4.2%), d32768 156.61 -> 157.51 (+0.6%),
|
|
tg256@d4096 9.17 -> 8.85 t/s (noisy, not gated). Handoff median: candidate 692 ms, reference
|
|
1068 ms. Downtime: about 42 minutes of validation and about 7 minutes of deploy.
|
|
- Previous full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs
|
|
`/root/arms/dep-2fc9f9c35-fv1.log` and `-fv2.log`): phase 1 `--ops --concat` 8 PASS, phase 2
|
|
`--arbiter --perf --moe-exact --moecache` 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048
|
|
274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff
|
|
median ref 693.5 / cand 678 ms. `--moe-exact`: 39 skips logged, identical to the reference and to
|
|
the skip forced off. Peak VRAM per arm ref 5476, cand 7062, cand 5767, ref 5476 MiB: the ~1.6 GiB
|
|
spike in the first candidate arm has now been seen twice (open question, FORK.md F9).
|
|
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
|
|
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
|
|
regression gate, not a measurement.
|
|
- Disk prompt cache (FORK.md F11, from `fable5/disk-prompt-cache`): add `--diskcache` to phase 1. It
|
|
is CPU only and runs beside production (about 2 minutes). Its GPU leg SKIPs below 1536 MiB of free
|
|
VRAM; then run `--diskcache` again in the phase-2 window. A 35B measurement driver is on the pod:
|
|
`/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the
|
|
production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script
|
|
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
|
|
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window3.sh` runs `q35.py`
|
|
with production's exact 35B command line (mmproj loaded): save/restore, cold save, the
|
|
same-reuse-point identity and an image request (FORK.md F11). Run 2026-11-21 on adc2aa3ec: all
|
|
legs passed (18.8k-token chat, prefill 84.6 s -> turn 2 4.7 s after the restart, identical
|
|
output; save 557 MB in 1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image
|
|
answered and not saved; log `/root/arms/diskcache-test/q35-mm.log`). It supersedes
|
|
`diskcache-window2.sh` (`cmp.py` runs with `--no-mmproj`).
|
|
`q35.py` and the window scripts live only on the pod (not in the repo) and hard-code a build path.
|
|
Before each use, edit them to point at the candidate build (`/root/arms/rebYYYYMMDD/build`), not
|
|
production's `/root/llama.cpp/build`, or you validate the wrong binary.
|
|
|
|
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
|
|
|
|
## 7. Deploy
|
|
|
|
Deploy exactly the validated commit. Downtime: 6 minutes for the 2026-10-24 redeploy. The deploy
|
|
rebuilds the production tree in place (step 6) from the same commit and with the same cmake flags as
|
|
the validated worktree build, so nothing is copied over; compare
|
|
`grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt` in both
|
|
trees if in doubt.
|
|
|
|
Run the stop, build and start (steps 4-7) as one script on the pod, started with `setsid nohup sh
|
|
<script> > <log> 2>&1 </dev/null &`, never as a backgrounded `cat && sh -n && setsid` chain from the
|
|
`kubectl exec` shell (see section 9).
|
|
|
|
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
|
|
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
|
|
```sh
|
|
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 13 (holds adc2aa3ec)
|
|
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-14.txt
|
|
git -C /root/llama.cpp status --short # must be clean
|
|
```
|
|
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3):
|
|
```sh
|
|
cd /root/llama.cpp
|
|
git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD
|
|
git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD <validated-commit>
|
|
```
|
|
On a redeploy with the same branch name this moves the production branch to the new commit.
|
|
3. **Evict the page cache.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM
|
|
victim, and benchmarks, mmap loads and builds without F10 leave tens of GB of model files cached.
|
|
During the first 2026-10-24 deploy the Talos OOM controller killed the whole pod 17 s into
|
|
verification (cause likely, not proven, memory pressure). Evict while production still runs
|
|
(unprivileged and safe: `-lm none` servers hold their weights in anonymous memory, and pages
|
|
mapped by an mmap model are not dropped). The same snippet is used before and after validation
|
|
phase 2:
|
|
```sh
|
|
python3 - <<'EOF'
|
|
import glob, os
|
|
n = 0
|
|
for root in ('/root/.cache/huggingface/hub', '/root/models'):
|
|
for p in glob.glob(root + '/**/*.gguf', recursive=True):
|
|
fd = os.open(p, os.O_RDONLY)
|
|
try:
|
|
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
|
|
n += 1
|
|
finally:
|
|
os.close(fd)
|
|
print('evicted', n, 'files')
|
|
EOF
|
|
```
|
|
F10 (FORK.md) drops the cache of every `-lm none` load by itself (verified in production on a 35B
|
|
GPU load: node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB), but files cached
|
|
by mmap loads, benchmarks or older builds still need this step. `/sys/fs/cgroup/memory.stat`
|
|
inside the container is the HOST root cgroup; the pod's own cgroup is under
|
|
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/`.
|
|
4. **Back up the current build, then stop llama-swap and reap orphans.** The deploy rebuilds
|
|
`/root/llama.cpp/build` in place, so copy it first (production keeps serving during the copy);
|
|
rollback (section 9) is then a copy, not a rebuild. Check `df -h /root` first (about 850 MB; the
|
|
2026-11-21 deploy made `/root/arms/build-adc2aa3ec`, 851 MB):
|
|
```sh
|
|
cp -a /root/llama.cpp/build /root/arms/build-<oldsha> # <oldsha>: the HEAD recorded in step 1
|
|
```
|
|
Delete the copy once the deploy is confirmed stable. Then:
|
|
```sh
|
|
supervisorctl -c /root/supervisord.conf stop llama-swap
|
|
ps -eo pid,ppid,etimes,args | grep '[l]lama-server'
|
|
kill -TERM <pid> <pid> # explicit PIDs only, never pkill -f
|
|
ps -eo pid,args | grep '[l]lama-server' # must be empty
|
|
```
|
|
`supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1),
|
|
one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM. Keep
|
|
this check even when it finds nothing (2026-10-25: no orphans).
|
|
5. **Check memory**, now that the running models are gone: `grep -E 'MemAvailable|MemFree'
|
|
/proc/meminfo`. Proceed only with MemAvailable >= 25 GB. In 2026-10-24 the eviction took node-wide
|
|
file cache from 15.6 to 4.9 GB and MemAvailable to 45.8 GB.
|
|
6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
|
|
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
|
|
```sh
|
|
cd /root/llama.cpp && HF_UI_VERSION=$(cut -d'|' -f2 build/tools/ui/.ui-stamp) \
|
|
cmake --build build --target llama-server llama-bench test-backend-ops -j 12
|
|
ls -lL build/bin/llama-server build/bin/libllama-server-impl.so build/bin/libllama-common.so \
|
|
build/bin/libllama.so build/bin/libmtmd.so build/bin/libggml.so build/bin/libggml-base.so \
|
|
build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
|
|
```
|
|
All must have fresh mtimes. Use `-L` (follow the symlinks): `build/bin` keeps every old versioned
|
|
`libggml-*.so.0.x` and `libmtmd.so.0.x`, so a plain `ls` lists stale files beside the new ones.
|
|
`llama-server` is a 17 KB stub. The server code is in `libllama-server-impl.so`, the common code
|
|
(arguments, sampling) in `libllama-common.so`; a server change shows only in their mtimes.
|
|
7. **Start and probe.**
|
|
```sh
|
|
supervisorctl -c /root/supervisord.conf start llama-swap
|
|
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200
|
|
```
|
|
Sample `/proc/meminfo` every few seconds during the first model loads. Confirm the running commit
|
|
from the first chat response (section 8): `system_fingerprint` is `b<build number>-<short sha>`
|
|
(the build number changes with each rebase; the sha is the deployed commit). The first 35B request after that deploy took 34 s including
|
|
the model load and the answer.
|
|
8. **Disk prompt cache** (FORK.md F11), only once `--cache-disk-path` is in `/root/config.yaml`:
|
|
- Every deploy empties the cache: the key holds the build commit, so the previous build's files are
|
|
never restored. They stay in the directory as "other files" until the size limit evicts them.
|
|
To free the space at once, `rm /root/prompt-cache/*.lpcd` while llama-swap is stopped (step 4).
|
|
- Check `/root` free space (`df -h /root`; 11-13 GB on 2026-10-25). The cap (`--cache-disk-size`)
|
|
plus a 1 GiB floor bounds the cache, but builds and model downloads share the PVC.
|
|
- Stopping llama-swap takes longer: each large model writes its slot state on SIGTERM, about 1 s
|
|
per GB of state, before it exits (FORK.md F7, F11).
|
|
|
|
## 8. Post-deploy checks
|
|
|
|
All through llama-swap on :8080. A request helper:
|
|
|
|
```sh
|
|
ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
|
|
-d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; }
|
|
holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; }
|
|
# image request body: a 64x64 solid red binary PPM, no image tools needed (like fork-validate.sh)
|
|
img() { python3 - "$1" <<'EOF'
|
|
import base64, json, sys
|
|
ppm = b'P6\n64 64\n255\n' + b'\xff\x00\x00' * (64 * 64)
|
|
url = 'data:image/x-portable-pixmap;base64,' + base64.b64encode(ppm).decode()
|
|
print(json.dumps({'model': sys.argv[1], 'max_tokens': 32, 'temperature': 0, 'messages': [{'role': 'user',
|
|
'content': [{'type': 'image_url', 'image_url': {'url': url}},
|
|
{'type': 'text', 'text': 'What is the main colour of this image? One word.'}]}]}))
|
|
EOF
|
|
}
|
|
# img MODEL | curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @-
|
|
```
|
|
|
|
Use the `-nothink` aliases with these `max_tokens: 32` helpers. A thinking alias spends the 32 tokens
|
|
on `reasoning_content` and returns an empty `content`; for one, raise `max_tokens` to >= 512.
|
|
|
|
1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers;
|
|
`holder` shows the 35B's PID (check with `ps -o args -p <pid>`).
|
|
2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is
|
|
correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch
|
|
(2026-10-24: 3.1-5.6 s).
|
|
3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF-nothink:UD-Q4_K_XL`
|
|
with the `img` helper; the answer names red. This also forces an eviction in the llama-swap matrix: the
|
|
evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed,
|
|
releasing the GPU token`, with no kill escalation.
|
|
4. Text to gemma and to the 4B after the eviction; `holder` correct each time.
|
|
With the disk prompt cache on (F11): send the 35B a chat of at least 256 tokens before step 3.
|
|
Its eviction must log `disk cache: saved the prompt of N tokens`. Then resend the same chat plus one
|
|
new user message to the 35B (earlier messages must stay byte-identical: editing an earlier message
|
|
changes the prefix and legitimately gives no restore): the log shows `disk cache: restored`, and `timings.prompt_n` is
|
|
only the new suffix.
|
|
5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931).
|
|
6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for
|
|
`GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none.
|
|
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`). No pod restart and no OOM kill:
|
|
- Restarts, from the workstation: `kubectl get pods -n llama | grep supervisord`; RESTARTS of the
|
|
Running pod must equal the value you noted before the deploy. A stale `supervisord-...-pbmjq`
|
|
in state `Unknown` is also listed; it is an old leftover, ignore it.
|
|
- OOM kills, in the pod: `dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'`. The
|
|
timestamps are seconds since boot; compare them with the first field of `/proc/uptime` to see
|
|
whether any falls in the deploy window (2026-10-25: the newest was about 19 h old).
|
|
|
|
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
|
|
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
|
|
that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with
|
|
`gdb -p <pid> -batch -ex 'thread apply all bt'` before killing it. gdb is not in
|
|
the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check
|
|
`command -v gdb` and reinstall it first if needed.
|
|
|
|
## 9. Rollback
|
|
|
|
```sh
|
|
supervisorctl -c /root/supervisord.conf stop llama-swap
|
|
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
|
|
cd /root/llama.cpp
|
|
git checkout -B <previous-production-branch> "$(head -1 /root/DEPLOY-ROLLBACK-commit-N.txt | cut -d' ' -f1)"
|
|
HF_UI_VERSION=$(cut -d'|' -f2 build/tools/ui/.ui-stamp) cmake --build build --target llama-server llama-bench -j 12
|
|
supervisorctl -c /root/supervisord.conf start llama-swap
|
|
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
|
|
```
|
|
|
|
This is the fallback when no build backup exists. With the deploy step 4 backup, roll back by copy
|
|
instead (stop, checkout the old commit, replace the build dir, start):
|
|
|
|
```sh
|
|
supervisorctl -c /root/supervisord.conf stop llama-swap
|
|
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
|
|
cd /root/llama.cpp
|
|
git checkout -B <previous-production-branch> <oldsha>
|
|
mv build /root/arms/build-failed-<newsha> # or rm -rf build; mind the disk
|
|
cp -a /root/arms/build-<oldsha> build
|
|
supervisorctl -c /root/supervisord.conf start llama-swap
|
|
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
|
|
```
|
|
|
|
Once the new deploy is confirmed stable, delete the copy (`rm -rf /root/arms/build-<oldsha>`; the
|
|
disk is about 98% full) and any `build-failed-*` directory.
|
|
|
|
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in
|
|
production. Keep the candidate worktree for a later redeploy.
|
|
|
|
If you run the rollback (or the deploy steps 4-7) from a workstation through `kubectl exec`, put the
|
|
whole sequence in one script on the pod and start it as `setsid nohup sh /root/arms/rollback.sh >
|
|
/root/arms/rollback.log 2>&1 </dev/null &`. Do not background a `cat > X && sh -n X && setsid ...`
|
|
chain in the exec shell: it can die with the exec session halfway, leaving llama-swap stopped (section
|
|
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
|
|
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
|
|
|
|
Pointer files 2 to 13 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
|
|
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, and commit-12 holds
|
|
`2fc9f9c35`, and commit-13 holds `adc2aa3ec`, the rollback target for the current production
|
|
`3bf6b55b3` (next free: 14). The formats vary
|
|
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
|
|
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
|
|
|
|
## 10. Recovering a hung GPU
|
|
|
|
Symptoms: a process that survives SIGKILL (stuck in the driver); `ring sdma1 timeout` or
|
|
`ring comp_... timeout` in the kernel log; after a reboot the supervisord pod sits in FailedMount with
|
|
`/dev/kfd is not a character device` / `/dev/dri is not a directory`, and
|
|
`talosctl get pcidevices` lists no VGA device (the RX 580 and its HDMI audio function are missing).
|
|
A warm reboot does not bring the card back (classic Polaris failed reset).
|
|
|
|
The node (`anapistula-delrosalae`, Talos, bare metal) has a NanoKVM at http://192.168.255.8.
|
|
Credentials: ask the owner. Procedure (owner's instructions, follow exactly):
|
|
|
|
1. Soft shutdown: **one short press** of the power button through the NanoKVM.
|
|
2. **Never use the reset button.** It does not cut GPU power and is useless here.
|
|
3. Wait until the power LED goes off.
|
|
4. Press the power button once more to boot.
|
|
5. Only if the LED is still on 10+ minutes after the short press: one **long press of 8 seconds** to
|
|
force power off, wait for the LED to go off, then a short press to boot. With the GPU off the bus the
|
|
soft shutdown has hung before (Talos services stopped, LED stayed on), and the long press was needed.
|
|
|
|
Talos access from the workstation:
|
|
|
|
```sh
|
|
export TALOSCONFIG=/home/user/Projects/klaster/talos/generated/talosconfig
|
|
nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices
|
|
nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg
|
|
```
|
|
|
|
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active:
|
|
`cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` reads `120000000` (the
|
|
`power1_cap ... -> 120000000` line in `/tmp/fan-control.log` may be missing after a pod restart). The two 2026-10-09 hangs both came at the
|
|
first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork
|
|
and a clean-upstream build hung, so they were treated as hardware/power, not code.
|
|
|
|
## 11. Operations reference
|
|
|
|
Pod access:
|
|
|
|
```sh
|
|
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
|
|
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
|
|
kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts
|
|
```
|
|
|
|
Wiring: k8s service `llama:11434` -> pod port 8080 (selector `app=llama-swap`) -> llama-swap `:8080`;
|
|
a Caddy proxy fronts `llama.lumpiasty.xyz`. llama-swap must listen on `:8080`.
|
|
|
|
Files (all under `/root`, a persistent volume):
|
|
- `/root/llama.cpp`: production tree, built in `build/` (shared libraries, so the real code is in
|
|
`build/bin/libllama.so` and `libggml-*.so`). Scratch worktrees go in `/root/arms/`.
|
|
- `/root/supervisord.conf`: supervisord (PID 1). `[program:llama-swap]` runs
|
|
`/root/llama-swap -config /root/config.yaml -listen :8080 -watch-config`; its `environment=` sets
|
|
`PATH` and `LD_LIBRARY_PATH` to `/root/llama.cpp/build/bin` and the arbiter variables (FORK.md section 5).
|
|
- `/root/config.yaml`: llama-swap models, macros (`moe_offload_args`, `moe_offload_gemma`,
|
|
`common_args`), matrix/exclusivity sets, preload. Backups `/root/config.yaml.bak-*`. Edit only with
|
|
the owner's approval.
|
|
- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program).
|
|
- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`).
|
|
- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers.
|
|
- `/root/prompt-cache/` (configured 2026-11-21, `--cache-disk-path /root/prompt-cache
|
|
--cache-disk-size 4096` in the `moe_offload_args` and `moe_offload_gemma` macros; backup
|
|
`/root/config.yaml.bak-diskcache-20261121`): disk prompt cache files `*.lpcd` of every model
|
|
with the flag (FORK.md F11). Production check: a 35B 4.6k-token chat was saved on eviction (278.6
|
|
MiB, copy 126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms) and turn 2
|
|
prefilled 32 tokens; a gemma image request works with the flags on and is not cached. Safe to empty while llama-swap is stopped.
|
|
- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/`
|
|
directory from the old design may remain). Never point a test at it.
|
|
|
|
Reloads: a `config.yaml` change is picked up by `-watch-config`, but a running model keeps its old
|
|
arguments until it restarts (`supervisorctl -c /root/supervisord.conf restart llama-swap`). An
|
|
`environment=` change needs `supervisorctl -c /root/supervisord.conf reread && supervisorctl -c
|
|
/root/supervisord.conf update`.
|
|
|
|
Useful commands:
|
|
- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`.
|
|
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
|
|
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
|
|
- 120 W cap active: `cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` = `120000000`.
|
|
- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the
|
|
last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log).
|
|
|
|
Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2).
|
|
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do
|
|
not transfer: about 2.25x the RX 580's prefill, 13% slower decode, and the GCN-only mask_opt change does
|
|
not apply. Do not change its system config.
|
|
|
|
Not a fork problem: OpenWebUI's stop button does not cancel generation (its Redis task-command listener
|
|
never handles the stop); llama-server and llama-swap both cancel correctly on client disconnect.
|