The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens: 83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save: peak +432 MiB. Leg c, output != reference: c sent turn 2 again, so the server rolled back to the checkpoint at the prompt end and processed 4 tokens, while the reference processed 27 from turn 1. These are different reuse points. A controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a restore, a cold save and a restore) is bit-identical, top-3 logprobs included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh and has not run. Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock -s file sleep) leaked the lock into sleep, so the SIGTERM landed in an ordinary evict_kv release (the full KV shadow, F4). Not the cache; the driver is fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
32 KiB
Fork rebase and deploy runbook (RX 580 fork)
Step-by-step procedure for moving the fork onto a new upstream and putting it into production, as it has evolved over the 2026-09-09, 2026-10-09 and 2026-10-24 rebases. Read FORK.md first: it lists every fork feature with its invariants and rebase hazards, and this runbook refers to it.
Ground rules:
- Never build on the workstation. All builds run on the RX 580 pod. Locally, rely on careful reading.
- Never resolve a conflict with
-X theirsor-X ours. Resolve every hunk by hand. - "Applied cleanly" is not "correct". Re-read every clean pick against the new upstream.
- Never
pkill -fon the pod. It matches thekubectl execshell and kills it. Kill by explicit PID. - Never leave the pod not serving. Anything that stops llama-swap must restart it from a trap.
- Keep the previous production branch untouched as the safety net. Do not push for the owner; the owner reviews and pushes.
Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a compiler; 5. Build on the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU; 11. Operations reference.
Redeploying a new tip on the same upstream base (no rebase), e.g. new fork commits on the
production branch (2026-10-25: 6c480f7e7 -> 2fc9f9c35, same branch, same base):
- Skip sections 1-3. Of section 4 only the ASCII check (on
git diff <prod>..<tip>) and a careful read of that diff apply;<prod>isgit -C /root/llama.cpp rev-parse HEADon the pod. There is nothing to range-diff. - Do sections 5-9 as written: build the tip in a scratch worktree, validate it with
--ref= the production build, then deploy. - The branch name stays the same, so deploy step 2's
git checkout -B <same branch> <tip>moves the production branch on the pod to the tip. After that the rollback pointer file is the only record of the old production commit on the pod, so write it first (deploy step 1).
1. Survey upstream
Local remotes: origin is upstream (github.com/ggerganov/llama.cpp), gitea is the fork
(git@gitea.lumpiasty.xyz:Lumpiasty/llama.cpp.git, SSH push from the workstation). On the pod,
origin is the same gitea repository over HTTPS (fetch only).
git fetch origin
OLD_BASE=$(git merge-base origin/master HEAD) # the old upstream base (still an ancestor of the new master)
OLD_TIP=$(git rev-parse HEAD)
git log --oneline "$OLD_BASE"..HEAD # the fork commits, oldest last
git log --oneline "$OLD_BASE"..origin/master | wc -l # size of the upstream delta
Then look at upstream changes in the areas FORK.md lists as hazards:
git log --oneline "$OLD_BASE"..origin/master -- \
src/llama-context.cpp src/llama-context.h src/llama-model.cpp src/llama-model.h \
src/llama-kv-cache.cpp src/llama-kv-cache.h src/llama-kv-cache-iswa.cpp \
src/llama-memory.h src/llama-memory-recurrent.cpp src/llama-memory-hybrid.cpp \
src/llama-moe-cache.cpp src/llama-moe-cache.h src/models/delta-net-base.cpp \
ggml/src/ggml-vulkan/ ggml/src/ggml-backend.cpp ggml/src/ggml-backend-impl.h ggml/include/ggml-backend.h \
tools/server/server-queue.cpp tools/server/server-queue.h tools/server/server-context.cpp tools/server/server.cpp \
tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/
For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past decisions:
- 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional
-ncmoe(all measured losses or no win on this card, FORK.md section 4). - 2026-10-09: the routing-ids readback skip had nothing to apply to (upstream #29943 deleted its block
in
ggml-backend.cpp); it was re-authored inllama_context::sched_copy_experts. Upstream's new MoE cache (#29887) needed a new fork commit so the VRAM arbiter releases and invalidates it. - Also check CLI changes that break production config or the scripts (upstream #28334 removed
--no-mmap/--mmap/--mlockand llama-bench-mmp; the default--portchanged to 9931).
2. Trial rebase in a scratch worktree
Measure the conflicts before doing the real thing:
git worktree add /tmp/trial-rebase origin/master
cd /tmp/trial-rebase
git cherry-pick "$OLD_BASE".."$OLD_TIP" # stop at each conflict, note file + hunk, then:
git cherry-pick --abort # or resolve roughly to see the next one
cd - && git worktree remove --force /tmp/trial-rebase
Write down, per commit: clean / conflict (which file, which hunk, what both sides did). That list becomes the brief for the real rebase.
3. The real rebase
NEW=fable5/vram-ondemand-rebase-$(date +%Y%m%d)
git switch -c "$NEW" origin/master
git cherry-pick <first-fork-commit> # one at a time, in the original order
Rules:
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
fork's trailer (
Co-Authored-By: Claude ...for agent-written commits, FORK.md status block), unless the owner asks for something else. - Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms.
- Fold each fix into the commit it belongs to so every commit builds on its own:
git commit --fixup=<sha>thenGIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master. - Leave the previous production branch untouched.
What -X theirs silently broke in a trial (2026-10-09). It "applied" every commit and produced
broken code:
- resurrected the scheduler readback block upstream had deleted from
ggml-backend.cpp; - dropped
!use_sparsefrom the GCNuse_mask_optcondition; - dropped upstream's
copy_experts_info(compile error); - dropped upstream's
is_empty()in the recurrent memory; - dropped upstream's MoE-cache memory breakdown block;
- left a static/extern mismatch with the new
ggml-vulkan-common.hdeclarations (compile error).
The correct resolutions in that rebase: keep upstream's n_embd width check plus our decode
auto-restore; copy_experts_info plus our kv_device_evicted; is_empty() plus our dev_shadows;
upstream's if (moe_cache) breakdown block outside our if (sched) guard; make ggml_vk_concat and
ggml_backend_vk_free non-static to match the new header while our helpers stay static.
Clean picks that were wrong (2026-10-24).
- Concat fast path (
3df719e1e): applied cleanly, did not compile. Upstream733f57a99renamedpipeline_cpy_f32_f32/pipeline_cpy_f16_f16topipeline_cpy_32/pipeline_cpy_16(bitwise copies, same push constants). - MoE cache release/restore (
7ac2014a8): applied cleanly, did not compile (upstream33d6135cbchanged LRU to SLRU,g.lru->g.slru) and had a runtime bug:moe_cache_slru::init()resets only the probation segment, so re-initialising a used cache corrupts its lists.invalidate()now assigns a freshmoe_cache_slru()first. - VRAM sharing core (
481be8b28): besides its one conflict, upstream's newload_tensors_hadamard()pushes two more entries intopimpl->ctxs_bufs. Without matchingweight_release.emplace_back()calls,release_device_weights()reads past the end ofweight_release. Nothing conflicted. - Conflicts that round:
ctx_buf_maps.emplace_backreplaced upstream byctx_bufs.emplace_back(ctx, std::move(buf_ptrs))(keep upstream's line, our release-slot block before it);llama-context.h(keep upstream'sn_reallocs_same/diffbeside ourkv_device_evicted);recv_with_timeout(upstream4e201388cuses a fixed deadline: keep our order "check!running, return null", thenwait_until(lock, deadline)).
Semantic re-checks to do for every rebase, clean or not (details in FORK.md):
- F1:
sched_copy_expertsand itsststruct unchanged in meaning. - F2:
use_mask_optstill contains upstream's terms plus our>= 256. - F3:
build_conv_statestill emitsggml_concat(conv_states, ggml_transpose(qkv_mixed), 0)withqkv_mixedcontiguous[ch, n_seq_tokens, n_seqs]; pipeline names;types_okstill excludes quantized types. - F4: every
ctxs_bufspush insrc/llama-model.cpphas aweight_releasepush (git grep -n "ctxs_bufs.emplace_back\|weight_release" src/llama-model.cpp); list memory types (git grep -n "public llama_memory_i" src/) and note any without a release override; MoE-cache internals;sched_reserve()still goes throughsched_new()and re-registers callbacks;free_scratchstill the lastggml_backend_imember; Vulkan prealloc fields matchggml_vk_cleanup. - F6/F7:
server_queue::start_loopstill evaluatesshould_sleep()only from the idle branch, never insideyield_to_queue();handle_sleeping_state,server_res_generator,load_modelordering; signal setup inserver.cpp. - F8: new iterators over
ctxs_bufsor users ofschedon teardown paths. - Scripts: flag spellings used by
scripts/rx580-bench/*andscripts/fork-validate.sh, and the log linesfork-validate.shgreps for (FORK.md F9). - Production config: every flag in
/root/config.yamlstill exists incommon/arg.cpp.
4. Checks without a compiler
Before handing the branch over for a build:
git diff origin/master..HEAD | grep -n '^[+-]\(<<<<<<<\|=======\|>>>>>>>\)' # no conflict markers
git diff origin/master..HEAD | grep -nP '^\+.*[^\x00-\x7F]' # ASCII only in added lines
git range-diff "$OLD_BASE".."$OLD_TIP" origin/master..HEAD # patch vs patch
- The range-diff must show only the edits you made on purpose. In 2026-10-24 the whole patch was identical to the old one except for 4 deliberate edits.
- Brace balance per changed file: the change in
{/}balance from new base to new tip should equal the change from old base to old tip. - Every identifier in added lines must exist on the new master or be defined by the fork. This catches
a renamed function but not a renamed member (like
lru->slru); re-read member accesses by hand. - Write down every spot where you would normally lean on the compiler and hand that list to whoever builds.
Then the owner pushes the branch to gitea.
5. Build on the pod, in a scratch worktree
Production keeps serving during this step.
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '
set -e
B=fable5/vram-ondemand-rebase-YYYYMMDD
W=/root/arms/rebYYYYMMDD
git -C /root/llama.cpp fetch origin "$B"
git -C /root/llama.cpp worktree add "$W" FETCH_HEAD
cd "$W"
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF
nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8
'
- Build only the targets you need; the full
alltarget fails on the UI assets (needs npm/network). If a tree has a staletools/ui/distwith a stubindex.html, remove it (it breaks llama-ui-embed). nice -n 19 -j 8leaves the serving models CPU. A full build takes about 3.5 minutes.- Report compile errors verbatim against the list from step 4.
- Do not overlap a build with a large benchmark (host OOM). Disk is tight:
/roothad about 13 GB free (97% full) on 2026-10-25, so checkdf -h /rootfirst. Each build tree is about 700 MB; remove scratch worktrees you no longer need withgit -C /root/llama.cpp worktree remove <dir>. - The build prints
UI: download dist.tar.gz.sha256 ... failed(no UI assets download). It is harmless with-DLLAMA_BUILD_UI=OFF; the listed targets still build. - Long runs: launch detached (
setsid nohup sh X > X.log 2>&1 </dev/null &) and poll, rather than holding akubectl execopen for hours.
6. Validate with scripts/fork-validate.sh
Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of the script describes each check.
# phase 1, beside production
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat
Add --arbiter to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus the bytes in
cat /sys/class/drm/card*/device/mem_info_vram_used); below that it FAILs at once. With gemma-26B
warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run
--arbiter in the phase-2 swapdown window instead (about 2 minutes more downtime), as below.
Page cache before and after phase 2. The --ref build may lack F10 (FORK.md), and --perf
loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the
eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check
/proc/meminfo.
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
cat > /root/arms/fv-perf.sh <<'EOF'
HUB=/root/.cache/huggingface/hub
Q35=$HUB/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
G26=$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/snapshots/e6759b29958180a998f62f991e4c741fe9b4d452/gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build \
--arbiter --perf --moe-exact "$Q35" --moecache "$G26" --keep
EOF
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
G26is the production SC117 gemma-26B, therefs/mainsnapshot (e6759b29...on 2026-10-25; an older snapshot82c751632...is also on disk). If it moves, read the snapshot name from$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/refs/main./root/bench/moe/swapdown.sh <script>(pod-only helper, not in the repo) stops llama-swap, waits for an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It printsSWAPDOWN_DONEat the end. Afterwards confirm llama-swap is RUNNING and/v1/modelsreturns 200.- The
--concatcheck needs a build with theGGML_VK_CONCAT_FASTPATHknob (in the fork sincee085c347d, so a rebased candidate has it; older candidates need--concat-build DIR). --moecacheneeds a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits, which proves nothing).--arbiteralso checks that a cold server really frees its VRAM (v-vram-release,v-vram-restore; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs ordinary handoff latency against--ref(a-handoff-latency).--refmust be the build currently in production, so both comparisons are against what serves today.- Phase 2 downtime, measured 2026-10-25: about 32 minutes in all (
--arbiterabout 2 min,--perfabout 28 min,--moecacheunder 30 s,--moe-exact78 s).--moe-exactproves the routing-ids readback skip exact (F1): the candidate's greedy output equals the reference's and, with theLLAMA_MOE_IDS_SKIPknob (any build of this branch), the candidate's with the skip forced off. --perfgates on medians of all repetition samples (since2512290f9); the samples are in<scratch>/perf.tsv, andfork-validate.sh --perf-report <file>re-aggregates them offline. A pod restart empties/tmp, so copy a scratch dir you want to keep to/root/arms/(the 2026-10-24 scratch dirs were lost that way).- Last full run (2026-10-25, candidate
2fc9f9c35vs reference6c480f7e7, logs/root/arms/dep-2fc9f9c35-fv1.logand-fv2.log): phase 1--ops --concat8 PASS, phase 2--arbiter --perf --moe-exact --moecache29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048 274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff median ref 693.5 / cand 678 ms.--moe-exact: 39 skips logged, identical to the reference and to the skip forced off. Peak VRAM per arm ref 5476, cand 7062, cand 5767, ref 5476 MiB: the ~1.6 GiB spike in the first candidate arm has now been seen twice (open question, FORK.md F9). - Any arbiter change:
a-handoff-latencyis the ordinary-handoff A/B that two lost deploys showed is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a regression gate, not a measurement. - Disk prompt cache (FORK.md F11, from
fable5/disk-prompt-cache): add--diskcacheto phase 1. It is CPU only and runs beside production (about 2 minutes). Its GPU leg SKIPs below 1536 MiB of free VRAM; then run--diskcacheagain in the phase-2 window. A 35B measurement driver is on the pod:/root/arms/diskcache-test/q35.py [CHARS](scratch port 8095, scratch arena and cache dir, the production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script/root/arms/diskcache-window.sh(page-cache evictions,--diskcache --arbiter,q35.py) runs under swapdown (run 2026-10-25: 29 PASS)./root/arms/diskcache-window2.shrunscmp.pyon the 35B, the same-reuse-point identity check (FORK.md F11), not yet run.
On any FAIL: stop and report; the scratch dir (/tmp/fork-validate.*) is kept with all logs.
7. Deploy
Deploy exactly the validated commit. Downtime: 6 minutes for the 2026-10-24 redeploy. The deploy
rebuilds the production tree in place (step 6) from the same commit and with the same cmake flags as
the validated worktree build, so nothing is copied over; compare
grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt in both
trees if in doubt.
- Rollback pointer. Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 10 (holds 6c480f7e7) git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-11.txt git -C /root/llama.cpp status --short # must be clean - Check out the candidate in the production tree (fetch first; the branch must be on gitea):
On a redeploy with the same branch name this moves the production branch to the new commit.
cd /root/llama.cpp git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD <validated-commit> - Evict the page cache. The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM
victim, and benchmarks, mmap loads and builds without F10 leave tens of GB of model files cached.
During the first 2026-10-24 deploy the Talos OOM controller killed the whole pod 17 s into
verification (cause likely, not proven, memory pressure). Evict while production still runs
(unprivileged and safe:
-lm noneservers hold their weights in anonymous memory, and pages mapped by an mmap model are not dropped). The same snippet is used before and after validation phase 2:F10 (FORK.md) drops the cache of everypython3 - <<'EOF' import glob, os n = 0 for root in ('/root/.cache/huggingface/hub', '/root/models'): for p in glob.glob(root + '/**/*.gguf', recursive=True): fd = os.open(p, os.O_RDONLY) try: os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) n += 1 finally: os.close(fd) print('evicted', n, 'files') EOF-lm noneload by itself (verified in production on a 35B GPU load: node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB), but files cached by mmap loads, benchmarks or older builds still need this step./sys/fs/cgroup/memory.statinside the container is the HOST root cgroup; the pod's own cgroup is under/sys/fs/cgroup/kubepods/besteffort/pod<uid>/. - Stop llama-swap and reap orphans.
supervisorctl -c /root/supervisord.conf stop llama-swap ps -eo pid,ppid,etimes,args | grep '[l]lama-server' kill -TERM <pid> <pid> # explicit PIDs only, never pkill -f ps -eo pid,args | grep '[l]lama-server' # must be emptysupervisorctl stop llama-swaphas left its llama-server children running as orphans (ppid 1), one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM. Keep this check even when it finds nothing (2026-10-25: no orphans). - Check memory, now that the running models are gone:
grep -E 'MemAvailable|MemFree' /proc/meminfo. Proceed only with MemAvailable >= 25 GB. In 2026-10-24 the eviction took node-wide file cache from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. - Build in place, now that nothing runs from
/root/llama.cpp/build/bin(llama-swap'sLD_LIBRARY_PATHpoints there, so do not rebuild it under running servers):All must have fresh mtimes. Usecd /root/llama.cpp && cmake --build build --target llama-server llama-bench -j 12 ls -lL build/bin/llama-server build/bin/libllama.so build/bin/libmtmd.so \ build/bin/libggml.so build/bin/libggml-base.so build/bin/libggml-cpu.so build/bin/libggml-vulkan.so-L(follow the symlinks):build/binkeeps every old versionedlibggml-*.so.0.xandlibmtmd.so.0.x, so a plainlslists stale files beside the new ones. - Start and probe.
Sample
supervisorctl -c /root/supervisord.conf start llama-swap curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200/proc/meminfoevery few seconds during the first model loads. Confirm the running commit from the first chat response (section 8):system_fingerprintisb<build>-<short sha>(2026-10-25:b11906-2fc9f9c35). The first 35B request after that deploy took 34 s including the model load and the answer. - Disk prompt cache (FORK.md F11), only once
--cache-disk-pathis in/root/config.yaml:- Every deploy empties the cache: the key holds the build commit, so the previous build's files are
never restored. They stay in the directory as "other files" until the size limit evicts them.
To free the space at once,
rm /root/prompt-cache/*.lpcdwhile llama-swap is stopped (step 4). - Check
/rootfree space (df -h /root; 11-13 GB on 2026-10-25). The cap (--cache-disk-size) plus a 1 GiB floor bounds the cache, but builds and model downloads share the PVC. - Stopping llama-swap takes longer: each large model writes its slot state on SIGTERM, about 1 s per GB of state, before it exits (FORK.md F7, F11).
- Every deploy empties the cache: the key holds the build commit, so the previous build's files are
never restored. They stay in the directory as "other files" until the size limit evicts them.
To free the space at once,
8. Post-deploy checks
All through llama-swap on :8080. A request helper:
ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; }
holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; }
# image request body: a 64x64 solid red binary PPM, no image tools needed (like fork-validate.sh)
img() { python3 - "$1" <<'EOF'
import base64, json, sys
ppm = b'P6\n64 64\n255\n' + b'\xff\x00\x00' * (64 * 64)
url = 'data:image/x-portable-pixmap;base64,' + base64.b64encode(ppm).decode()
print(json.dumps({'model': sys.argv[1], 'max_tokens': 32, 'temperature': 0, 'messages': [{'role': 'user',
'content': [{'type': 'image_url', 'image_url': {'url': url}},
{'type': 'text', 'text': 'What is the main colour of this image? One word.'}]}]}))
EOF
}
# img MODEL | curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @-
Use the -nothink aliases with these max_tokens: 32 helpers. A thinking alias spends the 32 tokens
on reasoning_content and returns an empty content; for one, raise max_tokens to >= 512.
ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'answers;holdershows the 35B's PID (check withps -o args -p <pid>).- Handoffs: alternate
Qwen3.5-4B-task:Q4_K_Mand the 35B two or three times. Each answer is correct andholdershows the PID of the model that just answered. Expect a few seconds per switch (2026-10-24: 3.1-5.6 s). - Multimodal: one image request to
SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF-nothink:UD-Q4_K_XLwith theimghelper; the answer names red. This also forces an eviction in the llama-swap matrix: the evicted process must exit by itself, loggingVRAM arbiter: loop stopped, device memory freed, releasing the GPU token, with no kill escalation. - Text to gemma and to the 4B after the eviction;
holdercorrect each time. With the disk prompt cache on (F11): send the 35B a chat of at least 256 tokens before step 3. Its eviction must logdisk cache: saved the prompt of N tokens. Then send the same chat plus one new message to the 35B again: the log showsdisk cache: restored, andtimings.prompt_nis only the new suffix. - Ports: the children bind the
--port ${PORT}values llama-swap passes (not upstream's default 9931). - Log scan of the new llama-swap log lines (
curl -s http://127.0.0.1:8080/logs) forGGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST: none. dmesgclean (noring ... timeout, noGPU_HUNG). No pod restart and no OOM kill:- Restarts, from the workstation:
kubectl get pods -n llama | grep supervisord; RESTARTS of the Running pod must equal the value you noted before the deploy. A stalesupervisord-...-pbmjqin stateUnknownis also listed; it is an old leftover, ignore it. - OOM kills, in the pod:
dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'. The timestamps are seconds since boot; compare them with the first field of/proc/uptimeto see whether any falls in the deploy window (2026-10-25: the newest was about 19 h old).
- Restarts, from the workstation:
A wedged arbiter looks like this: requests hang while /health still returns 200, and /proc/locks
shows a process holding WRITE on token.lock that is not generating. llama-swap restarts a child
that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with
gdb -p <pid> -batch -ex 'thread apply all bt' before killing it. gdb is not in
the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check
command -v gdb and reinstall it first if needed.
9. Rollback
supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
cd /root/llama.cpp
git checkout -B <previous-production-branch> "$(head -1 /root/DEPLOY-ROLLBACK-commit-N.txt | cut -d' ' -f1)"
cmake --build build --target llama-server llama-bench -j 12
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in production. Keep the candidate worktree for a later redeploy.
Pointer files 2 to 10 exist; for example commit-8 holds 64543a435, commit-9 5751c977b, and
commit-10 6c480f7e7, the rollback target for the current production 2fc9f9c35. The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
line 2), hence head -1 | cut -d' ' -f1. New pointers: the full SHA alone on line 1.
10. Recovering a hung GPU
Symptoms: a process that survives SIGKILL (stuck in the driver); ring sdma1 timeout or
ring comp_... timeout in the kernel log; after a reboot the supervisord pod sits in FailedMount with
/dev/kfd is not a character device / /dev/dri is not a directory, and
talosctl get pcidevices lists no VGA device (the RX 580 and its HDMI audio function are missing).
A warm reboot does not bring the card back (classic Polaris failed reset).
The node (anapistula-delrosalae, Talos, bare metal) has a NanoKVM at http://192.168.255.8.
Credentials: ask the owner. Procedure (owner's instructions, follow exactly):
- Soft shutdown: one short press of the power button through the NanoKVM.
- Never use the reset button. It does not cut GPU power and is useless here.
- Wait until the power LED goes off.
- Press the power button once more to boot.
- Only if the LED is still on 10+ minutes after the short press: one long press of 8 seconds to force power off, wait for the LED to go off, then a short press to boot. With the GPU off the bus the soft shutdown has hung before (Talos services stopped, LED stayed on), and the long press was needed.
Talos access from the workstation:
export TALOSCONFIG=/home/user/Projects/klaster/talos/generated/talosconfig
nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices
nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, /v1/models 200, and the 120 W cap active:
cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap reads 120000000 (the
power1_cap ... -> 120000000 line in /tmp/fan-control.log may be missing after a pod restart). The two 2026-10-09 hangs both came at the
first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork
and a clean-upstream build hung, so they were treated as hardware/power, not code.
11. Operations reference
Pod access:
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts
Wiring: k8s service llama:11434 -> pod port 8080 (selector app=llama-swap) -> llama-swap :8080;
a Caddy proxy fronts llama.lumpiasty.xyz. llama-swap must listen on :8080.
Files (all under /root, a persistent volume):
/root/llama.cpp: production tree, built inbuild/(shared libraries, so the real code is inbuild/bin/libllama.soandlibggml-*.so). Scratch worktrees go in/root/arms/./root/supervisord.conf: supervisord (PID 1).[program:llama-swap]runs/root/llama-swap -config /root/config.yaml -listen :8080 -watch-config; itsenvironment=setsPATHandLD_LIBRARY_PATHto/root/llama.cpp/build/binand the arbiter variables (FORK.md section 5)./root/config.yaml: llama-swap models, macros (moe_offload_args,moe_offload_gemma,common_args), matrix/exclusivity sets, preload. Backups/root/config.yaml.bak-*. Edit only with the owner's approval./root/fan-control.sh: fan curve and the fixed 120 W cap (separate supervisord program)./root/bench/: benchmark results and pod-side helpers (/root/bench/moe/swapdown.sh)./root/DEPLOY-ROLLBACK-commit-N.txt: rollback pointers./root/prompt-cache/(recommended, not yet configured): disk prompt cache files*.lpcdof every model with--cache-disk-path(FORK.md F11). Safe to empty while llama-swap is stopped./dev/shm/llama-vram/: production arbiter arena (token.lock,want.lock; an inertdoorbell/directory from the old design may remain). Never point a test at it.
Reloads: a config.yaml change is picked up by -watch-config, but a running model keeps its old
arguments until it restarts (supervisorctl -c /root/supervisord.conf restart llama-swap). An
environment= change needs supervisorctl -c /root/supervisord.conf reread && supervisorctl -c /root/supervisord.conf update.
Useful commands:
- VRAM in use:
cat /sys/class/drm/card*/device/mem_info_vram_used. - Token holder and waiters:
/proc/locksfiltered by the inode oftoken.lock/want.lock(WRITE = holder on token.lock, READ = registered waiters on want.lock). - 120 W cap active:
cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap=120000000. - The pod has
curl,python3,flock; nojq, and nogdbunless reinstalled by hand since the last pod restart (fork-validate.shthen runs its SIGTERM-in-release case by polling the log).
Second test box (optional): ssh user@gaming-pc, the owner's NixOS desktop with an RX 6600 (RDNA2).
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do
not transfer: about 2.25x the RX 580's prefill, 13% slower decode, and the GCN-only mask_opt change does
not apply. Do not change its system config.
Not a fork problem: OpenWebUI's stop button does not cancel generation (its Redis task-command listener never handles the stop); llama-server and llama-swap both cancel correctly on client disconnect.