Files
llama.cpp/docs/fork-rebase-runbook.md
T

47 KiB

Fork rebase and deploy runbook (RX 580 fork)

Step-by-step procedure for moving the fork onto a new upstream and putting it into production, as it has evolved over the 2026-09-09, 2026-10-09, 2026-10-24 and 2026-11-21 rebases. Read FORK.md first: it lists every fork feature with its invariants and rebase hazards, and this runbook refers to it.

Ground rules:

  • Never build on the workstation. All builds run on the RX 580 pod. Locally, rely on careful reading.
  • Never resolve a conflict with -X theirs or -X ours. Resolve every hunk by hand.
  • "Applied cleanly" is not "correct". Re-read every clean pick against the new upstream.
  • Never pkill -f on the pod. It matches the kubectl exec shell and kills it. Kill by explicit PID.
  • Never leave the pod not serving. Anything that stops llama-swap must restart it from a trap.
  • Keep the previous production branch untouched as the safety net. Do not push for the owner; the owner reviews and pushes.

Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a compiler; 5. Build on the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU; 11. Operations reference.

Redeploying a new tip on the same upstream base (no rebase), e.g. new fork commits on the production branch (2026-10-25: 6c480f7e7 -> 2fc9f9c35, same branch, same base):

  • Skip sections 1-3. Of section 4 only the ASCII check (on git diff <prod>..<tip>) and a careful read of that diff apply; <prod> is git -C /root/llama.cpp rev-parse HEAD on the pod. There is nothing to range-diff.
  • Do sections 5-9 as written: build the tip in a scratch worktree, validate it with --ref = the production build, then deploy.
  • The branch name stays the same, so deploy step 2's git checkout -B <same branch> <tip> moves the production branch on the pod to the tip. After that the rollback pointer file is the only record of the old production commit on the pod, so write it first (deploy step 1).

1. Survey upstream

Local remotes: origin is upstream (github.com/ggerganov/llama.cpp), gitea is the fork (git@gitea.lumpiasty.xyz:Lumpiasty/llama.cpp.git, SSH push from the workstation). On the pod, origin is the same gitea repository over HTTPS (fetch only).

git fetch origin
OLD_BASE=$(git merge-base origin/master HEAD)       # the old upstream base (still an ancestor of the new master)
OLD_TIP=$(git rev-parse HEAD)
git log --oneline "$OLD_BASE"..HEAD                   # the fork commits, oldest last
git log --oneline "$OLD_BASE"..origin/master | wc -l  # size of the upstream delta

Then look at upstream changes in the areas FORK.md lists as hazards:

git log --oneline "$OLD_BASE"..origin/master -- \
  src/llama-context.cpp src/llama-context.h src/llama-model.cpp src/llama-model.h \
  src/llama-kv-cache.cpp src/llama-kv-cache.h src/llama-kv-cache-iswa.cpp \
  src/llama-memory.h src/llama-memory-recurrent.cpp src/llama-memory-hybrid.cpp \
  src/llama-moe-cache.cpp src/llama-moe-cache.h src/models/delta-net-base.cpp \
  ggml/src/ggml-vulkan/ ggml/src/ggml-backend.cpp ggml/src/ggml-backend-impl.h ggml/include/ggml-backend.h \
  tools/server/server-queue.cpp tools/server/server-queue.h tools/server/server-context.cpp tools/server/server.cpp \
  tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/

Temporary upstream carries (FORK.md F12) come first: git log --oneline --grep '^upstream-carry' "$OLD_BASE"..HEAD. For each: merged into the new upstream -> do not cherry-pick it (check that the merged version is the carried head or newer); still open -> re-fetch the PR head and re-carry it if it moved; closed -> ask the owner. To carry a PR (2026-11-22: #32735, #32740):

git fetch https://github.com/ggml-org/llama.cpp pull/<N>/head:pr-<N>
git cherry-pick $(git merge-base pr-<N> HEAD)..pr-<N>     # the PR's commits, oldest first; resolve by hand
# squash them into one commit with the PR author, subject "upstream-carry #<N>: <PR title>", and a
# body with the PR URL, the carried head and any conflict resolution (git commit-tree on the final tree)
diff <(git diff $(git merge-base pr-<N> HEAD) pr-<N> | grep '^[+-]') <(git diff HEAD~1 HEAD | grep '^[+-]')

The last diff shows only the lines you changed while resolving conflicts (for #32740: none, the F2 line differs only in context). Validate a carry like any compiled change (sections 5-6), plus the PR's own tests.

For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past decisions:

  • 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional -ncmoe (all measured losses or no win on this card, FORK.md section 4).
  • 2026-10-09: the routing-ids readback skip had nothing to apply to (upstream #29943 deleted its block in ggml-backend.cpp); it was re-authored in llama_context::sched_copy_experts. Upstream's new MoE cache (#29887) needed a new fork commit so the VRAM arbiter releases and invalidates it.
  • Also check CLI changes that break production config or the scripts (upstream #28334 removed --no-mmap/--mmap/--mlock and llama-bench -mmp; the default --port changed to 9931).

2. Trial rebase in a scratch worktree

Measure the conflicts before doing the real thing:

git worktree add /tmp/trial-rebase origin/master
cd /tmp/trial-rebase
git cherry-pick "$OLD_BASE".."$OLD_TIP"     # stop at each conflict, note file + hunk, then:
git cherry-pick --abort                   # or resolve roughly to see the next one
cd - && git worktree remove --force /tmp/trial-rebase

Write down, per commit: clean / conflict (which file, which hunk, what both sides did). That list becomes the brief for the real rebase.

3. The real rebase

NEW=fable5/vram-ondemand-rebase-$(date +%Y%m%d)
git switch -c "$NEW" origin/master
git cherry-pick <first-fork-commit>      # one at a time, in the original order

Rules:

  • Cherry-pick in the original order, keep authors and messages. New commits you author get the fork's trailer (Co-Authored-By: Claude ... for agent-written commits, FORK.md status block), unless the owner asks for something else.
  • Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms.
  • Fold each fix into the commit it belongs to so every commit builds on its own: git commit --fixup=<sha> then GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master.
  • Leave the previous production branch untouched.

What -X theirs silently broke in a trial (2026-10-09). It "applied" every commit and produced broken code:

  • resurrected the scheduler readback block upstream had deleted from ggml-backend.cpp;
  • dropped !use_sparse from the GCN use_mask_opt condition;
  • dropped upstream's copy_experts_info (compile error);
  • dropped upstream's is_empty() in the recurrent memory;
  • dropped upstream's MoE-cache memory breakdown block;
  • left a static/extern mismatch with the new ggml-vulkan-common.h declarations (compile error).

The correct resolutions in that rebase: keep upstream's n_embd width check plus our decode auto-restore; copy_experts_info plus our kv_device_evicted; is_empty() plus our dev_shadows; upstream's if (moe_cache) breakdown block outside our if (sched) guard; make ggml_vk_concat and ggml_backend_vk_free non-static to match the new header while our helpers stay static.

Clean picks that were wrong (2026-10-24).

  • Concat fast path (3df719e1e): applied cleanly, did not compile. Upstream 733f57a99 renamed pipeline_cpy_f32_f32/pipeline_cpy_f16_f16 to pipeline_cpy_32/pipeline_cpy_16 (bitwise copies, same push constants).
  • MoE cache release/restore (7ac2014a8): applied cleanly, did not compile (upstream 33d6135cb changed LRU to SLRU, g.lru -> g.slru) and had a runtime bug: moe_cache_slru::init() resets only the probation segment, so re-initialising a used cache corrupts its lists. invalidate() now assigns a fresh moe_cache_slru() first.
  • VRAM sharing core (481be8b28): besides its one conflict, upstream's new load_tensors_hadamard() pushes two more entries into pimpl->ctxs_bufs. Without matching weight_release.emplace_back() calls, release_device_weights() reads past the end of weight_release. Nothing conflicted.
  • Conflicts that round: ctx_buf_maps.emplace_back replaced upstream by ctx_bufs.emplace_back(ctx, std::move(buf_ptrs)) (keep upstream's line, our release-slot block before it); llama-context.h (keep upstream's n_reallocs_same/diff beside our kv_device_evicted); recv_with_timeout (upstream 4e201388c uses a fixed deadline: keep our order "check !running, return null", then wait_until(lock, deadline)).

What the 2026-11-21 rebase (upstream e6d624b9a, 757 commits past 3f03b1772) changed (nothing built or run locally; every item below is by reading only and needs the pod build):

  • Upstream refactors that moved fork code (hand ports, not conflicts of one hunk):
    • clip.cpp/clip.h are gone (#31865). The encoder release/restore (F5) is now mtmd_model_release_device/mtmd_model_restore_device on mtmd_model (mtmd-model.cpp), called from mtmd_release_device/mtmd_restore_device under compute_mutex after syncing the shared scheduler; the page-cache drop of F10 moved to mtmd_model_loader::load_tensors.
    • server_prompt_cache moved to server-slot-sched.{h,cpp} and get_available_slot() was replaced by server_slot_sched::select(). The disk prompt cache (F11) lives there now (server_prompt_disk_cache), is called from the task dispatch after update_prompt_cache(), and uses common_state_data and shared_ptr<const common_prompt_checkpoint>. Only prompts made with the server lora config are saved, and a restored prompt gets that config in prompt.lora.
    • Upstream's llama_memory_bufs replaced the raw ctxs_bufs vectors of the KV cache and the recurrent memory: the null-buffer guard moved into llama_memory_bufs::memory_breakdown(), and the release code indexes ctxs_bufs.bufs[i]. Those two did not compile after a clean pick.
    • The MoE cache is now a ClassLRU (moe_cache_lru, g.lru) with no_alloc as a constructor argument and cpu_share. invalidate() rebuilds g.lru, refills every slot map with g.lru.empty (not -1), clears the CPU maps and cpu_experts.
    • server.cpp was split into server_run_instance and friends: the self-pipe shutdown thread (F7) is started in server_run_instance only, just before server_install_signal_handlers().
    • server_queue got n_inflight (#32133). should_sleep checks it first, so a response that is still open also blocks the want-lock yield. This is a decision, not an accident: check it with --arbiter.
  • Upstream added its own tiled-transpose concat (#30591) with no size gate. The fork path still runs first (ne0 >= 32); the gate and the GGML_VK_CONCAT_FASTPATH knob were added to upstream's selector (ggml_vk_concat_is_transposed), otherwise decode would take upstream's tiled path again (F3).
  • Memory types with no release override (stay in VRAM when cold, not used in production): llama_kv_cache_dsa, _dsa_iswa, _dsv4, _msa, llama_memory_hybrid_iswa. The KV rotation tensors (attn_rot_bufs, #31073) are small and are not released either.

Open questions from that rebase:

  • Is the fork's own concat fast path (F3) now redundant with upstream's tiled concat (#30591)? Benchmark GGML_VK_CONCAT_FASTPATH=0 against the default on pp and decode before deciding.
  • Does n_inflight blocking the want-lock yield delay handoffs? Validation says no: handoff median candidate 692 ms vs reference 1068 ms (a-handoff-latency, 2026-11-21).

Semantic re-checks to do for every rebase, clean or not (details in FORK.md):

  • F1: sched_copy_experts and its st struct unchanged in meaning.
  • F2: use_mask_opt still contains upstream's terms plus our >= 256 (and !bias && from #32740, carried as F12 or merged upstream).
  • F3: build_conv_state still emits ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0) with qkv_mixed contiguous [ch, n_seq_tokens, n_seqs]; pipeline names; types_ok still excludes quantized types.
  • F4: every ctxs_bufs push in src/llama-model.cpp has a weight_release push (git grep -n "ctxs_bufs.emplace_back\|weight_release" src/llama-model.cpp); list memory types (git grep -n "public llama_memory_i" src/) and note any without a release override; MoE-cache internals; sched_reserve() still goes through sched_new() and re-registers callbacks; free_scratch still the last ggml_backend_i member; Vulkan prealloc fields match ggml_vk_cleanup.
  • F6/F7: server_queue::start_loop still evaluates should_sleep() only from the idle branch, never inside yield_to_queue(); handle_sleeping_state, server_res_generator, load_model ordering; signal setup in server.cpp.
  • F8: new iterators over ctxs_bufs or users of sched on teardown paths.
  • Scripts: flag spellings used by scripts/rx580-bench/* and scripts/fork-validate.sh, and the log lines fork-validate.sh greps for (FORK.md F9).
  • Production config: every flag in /root/config.yaml still exists in common/arg.cpp.

4. Checks without a compiler

Before handing the branch over for a build:

git diff origin/master..HEAD | grep -n '^[+-]\(<<<<<<<\|=======\|>>>>>>>\)'   # no conflict markers
git diff origin/master..HEAD | grep -nP '^\+.*[^\x00-\x7F]'                    # ASCII only in added lines
git range-diff "$OLD_BASE".."$OLD_TIP" origin/master..HEAD                     # patch vs patch
  • The range-diff must show only the edits you made on purpose. In 2026-10-24 the whole patch was identical to the old one except for 4 deliberate edits.
  • Brace balance per changed file: the change in {/} balance from new base to new tip should equal the change from old base to old tip.
  • Every identifier in added lines must exist on the new master or be defined by the fork. This catches a renamed function but not a renamed member (like lru -> slru); re-read member accesses by hand.
  • Write down every spot where you would normally lean on the compiler and hand that list to whoever builds.

Then the owner pushes the branch to gitea, and the pod fetches it from there (section 5).

If you were told not to push, do not stop at an unbuilt branch. Unless the task says otherwise, a rebase task continues through the build and validation (sections 5 and 6) and reports the results; the owner pushes afterwards. Get the branch onto the pod with a git bundle:

# workstation; <base> is a commit the pod already has, e.g. production's HEAD
git bundle create /tmp/x.bundle <base>..<branch>
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c 'cat > /root/x.bundle' < /tmp/x.bundle
kubectl -n llama exec deploy/supervisord -- sh -c 'cd /root/llama.cpp && git fetch /root/x.bundle <branch>:<branch>'

The branch is then local in /root/llama.cpp: in section 5 skip the git fetch origin and use git worktree add "$W" <branch>. Delete /root/x.bundle afterwards (disk is tight). Deploy step 2 likewise uses the local branch instead of fetching from origin.

5. Build on the pod, in a scratch worktree

Production keeps serving during this step.

export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '
  set -e
  B=fable5/vram-ondemand-rebase-YYYYMMDD
  W=/root/arms/rebYYYYMMDD
  git -C /root/llama.cpp fetch origin "$B"
  git -C /root/llama.cpp worktree add "$W" FETCH_HEAD
  cd "$W"
  export HF_UI_VERSION=b11906   # production's: cut -d'|' -f2 /root/llama.cpp/build/tools/ui/.ui-stamp
  cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF
  nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8
'
  • Build only the targets you need; the full all target fails on the UI assets (needs npm/network). If a tree has a stale tools/ui/dist with a stub index.html, remove it (it breaks llama-ui-embed).
  • nice -n 19 -j 8 leaves the serving models CPU. A full build takes about 3.5 minutes.
  • Report compile errors verbatim against the list from step 4.
  • Do not overlap a build with a large benchmark (host OOM). Disk is tight: /root had about 13 GB free (97% full) on 2026-10-25, so check df -h /root first. Each build tree is about 700 MB; remove scratch worktrees you no longer need with git -C /root/llama.cpp worktree remove <dir>.
  • -DLLAMA_BUILD_UI=OFF does NOT make the UI asset step harmless. The llama-ui-assets step still runs, and when the build number has no HF tag it falls back to latest. On 2026-11-21 latest broke the build (missing bundle[hash].js, .css and workbox[hash].js). Set HF_UI_VERSION=<stamp> for both cmake -B build and cmake --build, where the stamp is production's current one in build/tools/ui/.ui-stamp (b11906 on 2026-11-21). The file holds ggml-org/llama-ui|b11906 and CMake requires ^[A-Za-z0-9._-]+$, so take only the tag with cut -d'|' -f2 build/tools/ui/.ui-stamp; never pass $(cat ...). The validated worktree's UI assets may then differ from the deployed ones (the code is identical).
  • Long runs: launch detached (setsid nohup sh X > X.log 2>&1 </dev/null &) and poll, rather than holding a kubectl exec open for hours.
  • The validated build must contain all three targets: llama-server, llama-bench and test-backend-ops (--perf and --ops run the last two from --build). A build script that makes only llama-server (as the 2026-10-25 diskcache-build.sh first did) is not a validated build.
  • Detach only one script. Upload it first (kubectl exec -i ... sh -c 'cat > /root/arms/X.sh' < X.sh), then start it alone: setsid nohup sh /root/arms/X.sh > X.log 2>&1 </dev/null &. Do not background a chain such as cat > X && sh -n X && setsid ... & inside the kubectl exec shell. The chain is a child of that session, and it can die with the session before setsid runs.

6. Validate with scripts/fork-validate.sh

Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of the script describes each check.

Rule: any test of a production model must use production's exact command line: the cmd of its /root/config.yaml entry with the macros expanded, including the mmproj that -hf pulls (production never passes --no-mmproj to the 35B or gemma), and supervisord's environment=. Change only --port, the arena and scratch paths. The 2026-10-25 deploy of the disk prompt cache aborted on the first request: every test had used --no-mmproj, which hid a GGML_ASSERT(!has_mtmd), and it was rolled back (FORK.md F11).

# phase 1, beside production
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
  --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat

Add --arbiter to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus the bytes in cat /sys/class/drm/card*/device/mem_info_vram_used); below that it FAILs at once. With gemma-26B warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run --arbiter in the phase-2 swapdown window instead (about 2 minutes more downtime), as below.

The same applies to --concat's candidate-timing gate: with the card nearly full (2026-11-22: 956 MiB free beside a warm Inkling) every concat arm ran about 2x slower and the fast path showed no gain (off/cand 0.9x, FAIL); in the swapdown window the same build passed with 5.1x. A phase-1 candidate-timing FAIL with little free VRAM is not a verdict: rerun --concat in phase 2.

Page cache before and after phase 2. The --ref build may lack F10 (FORK.md), and --perf loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check /proc/meminfo.

# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
cat > /root/arms/fv-perf.sh <<'EOF'
HUB=/root/.cache/huggingface/hub
Q35=$HUB/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
G26=$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/snapshots/e6759b29958180a998f62f991e4c741fe9b4d452/gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
  --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build \
  --arbiter --perf --moe-exact "$Q35" --moecache "$G26" --keep
EOF
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
  • G26 is the production SC117 gemma-26B, the refs/main snapshot (e6759b29... on 2026-10-25; an older snapshot 82c751632... is also on disk). If it moves, read the snapshot name from $HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/refs/main.
  • /root/bench/moe/swapdown.sh <script> (pod-only helper, not in the repo) stops llama-swap, waits for an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints SWAPDOWN_DONE at the end. Afterwards confirm llama-swap is RUNNING and /v1/models returns 200.
  • The --concat check needs a build with the GGML_VK_CONCAT_FASTPATH knob (in the fork since e085c347d, so a rebased candidate has it; older candidates need --concat-build DIR).
  • --moecache needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits, which proves nothing).
  • --arbiter also checks that a cold server really frees its VRAM (v-vram-release, v-vram-restore; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs ordinary handoff latency against --ref (a-handoff-latency). --ref must be the build currently in production, so both comparisons are against what serves today.
  • Phase 2 downtime, measured 2026-10-25: about 32 minutes in all (--arbiter about 2 min, --perf about 28 min, --moecache under 30 s, --moe-exact 78 s). --moe-exact proves the routing-ids readback skip exact (F1): the candidate's greedy output equals the reference's and, with the LLAMA_MOE_IDS_SKIP knob (any build of this branch), the candidate's with the skip forced off.
  • --perf gates on medians of all repetition samples (since 2512290f9); the samples are in <scratch>/perf.tsv, and fork-validate.sh --perf-report <file> re-aggregates them offline. A pod restart empties /tmp, so copy a scratch dir you want to keep to /root/arms/ (the 2026-10-24 scratch dirs were lost that way).
  • F14 run (2026-11-22, candidate f3397cded = 166ec22e2 (docs on b13c41355) + F14, server-only change; one swapdown window, /root/arms/usw-window.sh, log /root/arms/usw-window.log, about 15 minutes): the 35B TTFT A/B with and without LLAMA_USERSTART_MERGE=0 (FORK.md F14, M U U M, production 35B think command line with the mmproj), --diskcache 15 PASS, 0 FAIL, 0 SKIP, and the diskcache-window3.sh 35B legs (all passed). The full --perf/--arbiter set was not rerun: the change touches only the server prompt-batch loop. Deploy downtime 33 s (incremental in-place build).
  • Last full run (2026-11-22, candidate b13c41355 = 3621025e8 + F13 vs reference 3621025e8, one swapdown window, --ops --concat --diskcache --arbiter --perf --moe-exact --moecache, log /root/arms/overnight/fv.log): 52 PASS, 0 FAIL, 0 SKIP in 41 minutes. Perf medians ref -> cand: pp2048 286.61 -> 284.94 (-0.6%), d16384 207.93 -> 207.96, d32768 161.93 -> 162.12, tg256@d4096 14.11 -> 14.47; handoff median ref 715 / cand 706.5 ms; peak VRAM ref 5478 / cand 7771 MiB (the first-arm spike of FORK.md F9, this time in a candidate arm). With the card free, phase 1 and 2 can run as one invocation like this.
  • Previous full run (2026-11-22, candidate 3621025e8 = 3bf6b55b3 + the F12 carries, vs reference 3bf6b55b3; logs /root/arms/carry1122-p1.log, -p2.log): phase 1 --ops --concat --diskcache 20 PASS, 1 FAIL (concat/candidate-timing beside a full card, see above), 1 SKIP (diskcache GPU leg); phase 2 --concat --arbiter --diskcache --perf --moe-exact --moecache 47 PASS, 0 FAIL, 0 SKIP. Perf medians ref -> cand: pp2048 286.97 -> 286.32, d16384 208.31 -> 208.21, d32768 162.25 -> 162.37; handoff median ref 696 / cand 700 ms; peak VRAM ref 7771 / cand 5502 MiB. Plus SCALE 15/15 and test-flash-attn-indexed-bias (FORK.md F12). Window about 30 minutes.
  • Earlier full run (2026-11-21, candidate 3bf6b55b3 vs reference adc2aa3ec): phase 1 39 PASS, phase 2 9 PASS plus the 35B disk-cache legs (q35.py, mmproj loaded). Perf medians ref -> cand: pp2048 259.39 -> 286.61 (+10.5%), d16384 199.26 -> 207.63 (+4.2%), d32768 156.61 -> 157.51 (+0.6%), tg256@d4096 9.17 -> 8.85 t/s (noisy, not gated). Handoff median: candidate 692 ms, reference 1068 ms. Downtime: about 42 minutes of validation and about 7 minutes of deploy.
  • Earlier full run (2026-10-25, candidate 2fc9f9c35 vs reference 6c480f7e7, logs /root/arms/dep-2fc9f9c35-fv1.log and -fv2.log): phase 1 --ops --concat 8 PASS, phase 2 --arbiter --perf --moe-exact --moecache 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048 274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff median ref 693.5 / cand 678 ms. --moe-exact: 39 skips logged, identical to the reference and to the skip forced off. Peak VRAM per arm ref 5476, cand 7062, cand 5767, ref 5476 MiB: the ~1.6 GiB spike in the first candidate arm has now been seen twice (open question, FORK.md F9).
  • Any arbiter change: a-handoff-latency is the ordinary-handoff A/B that two lost deploys showed is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a regression gate, not a measurement.
  • Disk prompt cache (FORK.md F11, from fable5/disk-prompt-cache): add --diskcache to phase 1. It is CPU only and runs beside production (about 2 minutes). Its GPU leg SKIPs below 1536 MiB of free VRAM; then run --diskcache again in the phase-2 window. A 35B measurement driver is on the pod: /root/arms/diskcache-test/q35.py [CHARS] (scratch port 8095, scratch arena and cache dir, the production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script /root/arms/diskcache-window.sh (page-cache evictions, --diskcache --arbiter, q35.py) runs under swapdown (run 2026-10-25: 29 PASS). /root/arms/diskcache-window3.sh runs q35.py with production's exact 35B command line (mmproj loaded): save/restore, cold save, the same-reuse-point identity and an image request (FORK.md F11). Run 2026-11-21 on adc2aa3ec: all legs passed (18.8k-token chat, prefill 84.6 s -> turn 2 4.7 s after the restart, identical output; save 557 MB in 1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image answered and not saved; log /root/arms/diskcache-test/q35-mm.log). It supersedes diskcache-window2.sh (cmp.py runs with --no-mmproj). q35.py and the window scripts live only on the pod (not in the repo) and hard-code a build path. Before each use, edit them to point at the candidate build (/root/arms/rebYYYYMMDD/build), not production's /root/llama.cpp/build, or you validate the wrong binary.

On any FAIL: stop and report; the scratch dir (/tmp/fork-validate.*) is kept with all logs.

7. Deploy

Deploy exactly the validated commit. Downtime: 6 minutes for the 2026-10-24 redeploy. The deploy rebuilds the production tree in place (step 6) from the same commit and with the same cmake flags as the validated worktree build, so nothing is copied over; compare grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt in both trees if in doubt.

Run the stop, build and start (steps 4-7) as one script on the pod, started with `setsid nohup sh

<script> > 2>&1 </dev/null &`, never as a backgrounded `cat && sh -n && setsid` chain from the `kubectl exec` shell (see section 9). 1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9): ```sh ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 16 (holds b13c41355) git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-17.txt git -C /root/llama.cpp status --short # must be clean ``` 2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3): ```sh cd /root/llama.cpp git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD ``` On a redeploy with the same branch name this moves the production branch to the new commit. 3. **Evict the page cache.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM victim, and benchmarks, mmap loads and builds without F10 leave tens of GB of model files cached. During the first 2026-10-24 deploy the Talos OOM controller killed the whole pod 17 s into verification (cause likely, not proven, memory pressure). Evict while production still runs (unprivileged and safe: `-lm none` servers hold their weights in anonymous memory, and pages mapped by an mmap model are not dropped). The same snippet is used before and after validation phase 2: ```sh python3 - <<'EOF' import glob, os n = 0 for root in ('/root/.cache/huggingface/hub', '/root/models'): for p in glob.glob(root + '/**/*.gguf', recursive=True): fd = os.open(p, os.O_RDONLY) try: os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) n += 1 finally: os.close(fd) print('evicted', n, 'files') EOF ``` F10 (FORK.md) drops the cache of every `-lm none` load by itself (verified in production on a 35B GPU load: node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB), but files cached by mmap loads, benchmarks or older builds still need this step. `/sys/fs/cgroup/memory.stat` inside the container is the HOST root cgroup; the pod's own cgroup is under `/sys/fs/cgroup/kubepods/besteffort/pod/`. 4. **Back up the current build, then stop llama-swap and reap orphans.** The deploy rebuilds `/root/llama.cpp/build` in place, so copy it first (production keeps serving during the copy); rollback (section 9) is then a copy, not a rebuild. Check `df -h /root` first (about 850 MB; the 2026-11-21 deploy made `/root/arms/build-adc2aa3ec`, 851 MB): ```sh cp -a /root/llama.cpp/build /root/arms/build- # : the HEAD recorded in step 1 ``` Delete the copy once the deploy is confirmed stable. Then: ```sh supervisorctl -c /root/supervisord.conf stop llama-swap ps -eo pid,ppid,etimes,args | grep '[l]lama-server' kill -TERM # explicit PIDs only, never pkill -f ps -eo pid,args | grep '[l]lama-server' # must be empty ``` `supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1), one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM. Keep this check even when it finds nothing (2026-10-25: no orphans). 5. **Check memory**, now that the running models are gone: `grep -E 'MemAvailable|MemFree' /proc/meminfo`. Proceed only with MemAvailable >= 25 GB. In 2026-10-24 the eviction took node-wide file cache from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. 6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's `LD_LIBRARY_PATH` points there, so do not rebuild it under running servers): ```sh cd /root/llama.cpp && HF_UI_VERSION=$(cut -d'|' -f2 build/tools/ui/.ui-stamp) \ cmake --build build --target llama-server llama-bench test-backend-ops -j 12 ls -lL build/bin/llama-server build/bin/libllama-server-impl.so build/bin/libllama-common.so \ build/bin/libllama.so build/bin/libmtmd.so build/bin/libggml.so build/bin/libggml-base.so \ build/bin/libggml-cpu.so build/bin/libggml-vulkan.so ``` All must have fresh mtimes. Use `-L` (follow the symlinks): `build/bin` keeps every old versioned `libggml-*.so.0.x` and `libmtmd.so.0.x`, so a plain `ls` lists stale files beside the new ones. `llama-server` is a 17 KB stub. The server code is in `libllama-server-impl.so`, the common code (arguments, sampling) in `libllama-common.so`; a server change shows only in their mtimes. 7. **Start and probe.** ```sh supervisorctl -c /root/supervisord.conf start llama-swap curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200 ``` Sample `/proc/meminfo` every few seconds during the first model loads. Confirm the running commit from the first chat response (section 8): `system_fingerprint` is `b-` (the build number changes with each rebase; the sha is the deployed commit). The first 35B request after that deploy took 34 s including the model load and the answer. 8. **Disk prompt cache** (FORK.md F11), only once `--cache-disk-path` is in `/root/config.yaml`: - Every deploy empties the cache: the key holds the build commit, so the previous build's files are never restored. They stay in the directory as "other files" until the size limit evicts them. To free the space at once, `rm /root/prompt-cache/*.lpcd` while llama-swap is stopped (step 4). - Check `/root` free space (`df -h /root`; 11-13 GB on 2026-10-25). The cap (`--cache-disk-size`) plus a 1 GiB floor bounds the cache, but builds and model downloads share the PVC. - Stopping llama-swap takes longer: each large model writes its slot state on SIGTERM, about 1 s per GB of state, before it exits (FORK.md F7, F11). ## 8. Post-deploy checks All through llama-swap on :8080. A request helper: ```sh ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \ -d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; } holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; } # image request body: a 64x64 solid red binary PPM, no image tools needed (like fork-validate.sh) img() { python3 - "$1" <<'EOF' import base64, json, sys ppm = b'P6\n64 64\n255\n' + b'\xff\x00\x00' * (64 * 64) url = 'data:image/x-portable-pixmap;base64,' + base64.b64encode(ppm).decode() print(json.dumps({'model': sys.argv[1], 'max_tokens': 32, 'temperature': 0, 'messages': [{'role': 'user', 'content': [{'type': 'image_url', 'image_url': {'url': url}}, {'type': 'text', 'text': 'What is the main colour of this image? One word.'}]}]})) EOF } # img MODEL | curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @- ``` Use the `-nothink` aliases with these `max_tokens: 32` helpers. A thinking alias spends the 32 tokens on `reasoning_content` and returns an empty `content`; for one, raise `max_tokens` to >= 512. 1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers; `holder` shows the 35B's PID (check with `ps -o args -p `). 2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch (2026-10-24: 3.1-5.6 s). 3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF-nothink:UD-Q4_K_XL` with the `img` helper; the answer names red. This also forces an eviction in the llama-swap matrix: the evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token`, with no kill escalation. 4. Text to gemma and to the 4B after the eviction; `holder` correct each time. With the disk prompt cache on (F11): send the 35B a chat of at least 256 tokens before step 3. Its eviction must log `disk cache: saved the prompt of N tokens`. Then resend the same chat plus one new user message to the 35B (earlier messages must stay byte-identical: editing an earlier message changes the prefix and legitimately gives no restore): the log shows `disk cache: restored`, and `timings.prompt_n` is only the new suffix. 5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931). 6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for `GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none. 7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`). No pod restart and no OOM kill: - Restarts, from the workstation: `kubectl get pods -n llama | grep supervisord`; RESTARTS of the Running pod must equal the value you noted before the deploy. A stale `supervisord-...-pbmjq` in state `Unknown` is also listed; it is an old leftover, ignore it. - OOM kills, in the pod: `dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'`. The timestamps are seconds since boot; compare them with the first field of `/proc/uptime` to see whether any falls in the deploy window (2026-10-25: the newest was about 19 h old). 8. Inkling (FORK.md section 5, F12): judge the answer text, not only HTTP 200. The first request after a fresh load should be a prompt of about 150 tokens (upstream #32778 gives all-'?' output there); then a long prompt (Pan Tadeusz, 8K and 14K tokens, upstream #32755). Send an image and a tool request to both entries: since the nothink macro dropped `--reasoning-budget 0` (FORK.md F12) both answer them; repeated `<|message_model|><|content_thinking|><|end_message|>` from the nothink entry means a budget of 0 is back. Since F13 a scratch server with the production command line plus `-lv 4` must log `post_load_tensors: N views of host weights` (production's own log level hides libllama's info lines, so the llama-swap log never shows it), and the entries no longer set `GGML_OP_OFFLOAD_MIN_BATCH=64` (F13 measured the 32-63-token window correct without it). The placement is `-ot` with the loop layers 6..6+k-1 left on the GPU (FORK.md F13 placement guide); after a change of context or quant, check the fit again (the breakdown must leave room for the mmproj, about 0.8 GiB). The memory breakdown table is printed only at `-lv 4`, which production does not set, so it is not in the llama-swap log: read it from a scratch server started with production's command line plus `-lv 4`. Drivers used on 2026-11-22 (pod only): `/root/arms/carry1122-post.py` (through llama-swap) and `/root/arms/carry1122-ink-direct.py` (scratch server, GPU to itself). A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks` shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with `gdb -p -batch -ex 'thread apply all bt'` before killing it. gdb is not in the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check `command -v gdb` and reinstall it first if needed. ## 9. Rollback ```sh supervisorctl -c /root/supervisord.conf stop llama-swap ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID cd /root/llama.cpp git checkout -B "$(head -1 /root/DEPLOY-ROLLBACK-commit-N.txt | cut -d' ' -f1)" HF_UI_VERSION=$(cut -d'|' -f2 build/tools/ui/.ui-stamp) cmake --build build --target llama-server llama-bench -j 12 supervisorctl -c /root/supervisord.conf start llama-swap curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models ``` This is the fallback when no build backup exists. With the deploy step 4 backup, roll back by copy instead (stop, checkout the old commit, replace the build dir, start): ```sh supervisorctl -c /root/supervisord.conf stop llama-swap ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID cd /root/llama.cpp git checkout -B mv build /root/arms/build-failed- # or rm -rf build; mind the disk cp -a /root/arms/build- build supervisorctl -c /root/supervisord.conf start llama-swap curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models ``` Once the new deploy is confirmed stable, delete the copy (`rm -rf /root/arms/build-`; the disk is about 98% full) and any `build-failed-*` directory. Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in production. Keep the candidate worktree for a later redeploy. If you run the rollback (or the deploy steps 4-7) from a workstation through `kubectl exec`, put the whole sequence in one script on the pod and start it as `setsid nohup sh /root/arms/rollback.sh > /root/arms/rollback.log 2>&1 </dev/null &`. Do not background a `cat > X && sh -n X && setsid ...` chain in the exec shell: it can die with the exec session halfway, leaving llama-swap stopped (section 5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from `/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`. Pointer files 2 to 16 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`, commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, commit-12 holds `2fc9f9c35`, commit-13 holds `adc2aa3ec`, commit-14 holds `3bf6b55b3`, commit-15 holds `3621025e8`, and commit-16 holds `b13c41355`, the rollback target for the current production `f3397cded` (next free: 17; build copy `/root/arms/build-b13c41355`). The formats vary (commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1. ## 10. Recovering a hung GPU Symptoms: a process that survives SIGKILL (stuck in the driver); `ring sdma1 timeout` or `ring comp_... timeout` in the kernel log; after a reboot the supervisord pod sits in FailedMount with `/dev/kfd is not a character device` / `/dev/dri is not a directory`, and `talosctl get pcidevices` lists no VGA device (the RX 580 and its HDMI audio function are missing). A warm reboot does not bring the card back (classic Polaris failed reset). The node (`anapistula-delrosalae`, Talos, bare metal) has a NanoKVM at http://192.168.255.8. Credentials: ask the owner. Procedure (owner's instructions, follow exactly): 1. Soft shutdown: **one short press** of the power button through the NanoKVM. 2. **Never use the reset button.** It does not cut GPU power and is useless here. 3. Wait until the power LED goes off. 4. Press the power button once more to boot. 5. Only if the LED is still on 10+ minutes after the short press: one **long press of 8 seconds** to force power off, wait for the LED to go off, then a short press to boot. With the GPU off the bus the soft shutdown has hung before (Talos services stopped, LED stayed on), and the long press was needed. Talos access from the workstation: ```sh export TALOSCONFIG=/home/user/Projects/klaster/talos/generated/talosconfig nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg ``` Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active: `cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` reads `120000000` (the `power1_cap ... -> 120000000` line in `/tmp/fan-control.log` may be missing after a pod restart). The two 2026-10-09 hangs both came at the first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork and a clean-upstream build hung, so they were treated as hardware/power, not code. ## 11. Operations reference Pod access: ```sh export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig kubectl -n llama exec -i deploy/supervisord -- sh -c '' kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts ``` Wiring: k8s service `llama:11434` -> pod port 8080 (selector `app=llama-swap`) -> llama-swap `:8080`; a Caddy proxy fronts `llama.lumpiasty.xyz`. llama-swap must listen on `:8080`. Files (all under `/root`, a persistent volume): - `/root/llama.cpp`: production tree, built in `build/` (shared libraries, so the real code is in `build/bin/libllama.so` and `libggml-*.so`). Scratch worktrees go in `/root/arms/`. - `/root/supervisord.conf`: supervisord (PID 1). `[program:llama-swap]` runs `/root/llama-swap -config /root/config.yaml -listen :8080 -watch-config`; its `environment=` sets `PATH` and `LD_LIBRARY_PATH` to `/root/llama.cpp/build/bin` and the arbiter variables (FORK.md section 5). - `/root/config.yaml`: llama-swap models, macros (`moe_offload_args`, `moe_offload_gemma`, `common_args`), matrix/exclusivity sets, preload. Backups `/root/config.yaml.bak-*`. Edit only with the owner's approval. - `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program). - `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`). - `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers. - `/root/prompt-cache/` (configured 2026-11-21, `--cache-disk-path /root/prompt-cache --cache-disk-size 4096` in the `moe_offload_args` and `moe_offload_gemma` macros; backup `/root/config.yaml.bak-diskcache-20261121`): disk prompt cache files `*.lpcd` of every model with the flag (FORK.md F11). Production check: a 35B 4.6k-token chat was saved on eviction (278.6 MiB, copy 126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms) and turn 2 prefilled 32 tokens; a gemma image request works with the flags on and is not cached. Safe to empty while llama-swap is stopped. - `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/` directory from the old design may remain). Never point a test at it. Reloads: a `config.yaml` change is picked up by `-watch-config`, but a running model keeps its old arguments until it restarts (`supervisorctl -c /root/supervisord.conf restart llama-swap`). An `environment=` change needs `supervisorctl -c /root/supervisord.conf reread && supervisorctl -c /root/supervisord.conf update`. Useful commands: - VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`. - Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock` (WRITE = holder on token.lock, READ = registered waiters on want.lock). - 120 W cap active: `cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` = `120000000`. - The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log). Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2). Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do not transfer: about 2.25x the RX 580's prefill, 13% slower decode, and the GCN-only mask_opt change does not apply. Do not change its system config. Not a fork problem: OpenWebUI's stop button does not cancel generation (its Redis task-command listener never handles the stop); llama-server and llama-swap both cancel correctly on client disconnect.