# Fork rebase and deploy runbook (RX 580 fork) Step-by-step procedure for moving the fork onto a new upstream and putting it into production, as it has evolved over the 2026-09-09, 2026-10-09, 2026-10-24 and 2026-11-21 rebases. Read [FORK.md](../FORK.md) first: it lists every fork feature with its invariants and rebase hazards, and this runbook refers to it. Ground rules: - **Never build on the workstation.** All builds run on the RX 580 pod. Locally, rely on careful reading. - **Never resolve a conflict with `-X theirs` or `-X ours`.** Resolve every hunk by hand. - **"Applied cleanly" is not "correct".** Re-read every clean pick against the new upstream. - **Never `pkill -f` on the pod.** It matches the `kubectl exec` shell and kills it. Kill by explicit PID. - **Never leave the pod not serving.** Anything that stops llama-swap must restart it from a trap. - Keep the previous production branch untouched as the safety net. Do not push for the owner; the owner reviews and pushes. Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a compiler; 5. Build on the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU; 11. Operations reference. **Redeploying a new tip on the same upstream base (no rebase)**, e.g. new fork commits on the production branch (2026-10-25: `6c480f7e7` -> `2fc9f9c35`, same branch, same base): - Skip sections 1-3. Of section 4 only the ASCII check (on `git diff ..`) and a careful read of that diff apply; `` is `git -C /root/llama.cpp rev-parse HEAD` on the pod. There is nothing to range-diff. - Do sections 5-9 as written: build the tip in a scratch worktree, validate it with `--ref` = the production build, then deploy. - The branch name stays the same, so deploy step 2's `git checkout -B ` moves the production branch on the pod to the tip. After that the rollback pointer file is the only record of the old production commit on the pod, so write it first (deploy step 1). --- ## 1. Survey upstream Local remotes: `origin` is upstream (github.com/ggerganov/llama.cpp), `gitea` is the fork (git@gitea.lumpiasty.xyz:Lumpiasty/llama.cpp.git, SSH push from the workstation). On the pod, `origin` is the same gitea repository over HTTPS (fetch only). ```sh git fetch origin OLD_BASE=$(git merge-base origin/master HEAD) # the old upstream base (still an ancestor of the new master) OLD_TIP=$(git rev-parse HEAD) git log --oneline "$OLD_BASE"..HEAD # the fork commits, oldest last git log --oneline "$OLD_BASE"..origin/master | wc -l # size of the upstream delta ``` Then look at upstream changes in the areas FORK.md lists as hazards: ```sh git log --oneline "$OLD_BASE"..origin/master -- \ src/llama-context.cpp src/llama-context.h src/llama-model.cpp src/llama-model.h \ src/llama-kv-cache.cpp src/llama-kv-cache.h src/llama-kv-cache-iswa.cpp \ src/llama-memory.h src/llama-memory-recurrent.cpp src/llama-memory-hybrid.cpp \ src/llama-moe-cache.cpp src/llama-moe-cache.h src/models/delta-net-base.cpp \ ggml/src/ggml-vulkan/ ggml/src/ggml-backend.cpp ggml/src/ggml-backend-impl.h ggml/include/ggml-backend.h \ tools/server/server-queue.cpp tools/server/server-queue.h tools/server/server-context.cpp tools/server/server.cpp \ tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/ ``` **Temporary upstream carries (FORK.md F12)** come first: `git log --oneline --grep '^upstream-carry' "$OLD_BASE"..HEAD`. For each: merged into the new upstream -> do not cherry-pick it (check that the merged version is the carried head or newer); still open -> re-fetch the PR head and re-carry it if it moved; closed -> ask the owner. To carry a PR (2026-11-22: #32735, #32740): ```sh git fetch https://github.com/ggml-org/llama.cpp pull//head:pr- git cherry-pick $(git merge-base pr- HEAD)..pr- # the PR's commits, oldest first; resolve by hand # squash them into one commit with the PR author, subject "upstream-carry #: ", and a # body with the PR URL, the carried head and any conflict resolution (git commit-tree on the final tree) diff <(git diff $(git merge-base pr- HEAD) pr- | grep '^[+-]') <(git diff HEAD~1 HEAD | grep '^[+-]') ``` The last diff shows only the lines you changed while resolving conflicts (for #32740: none, the F2 line differs only in context). Validate a carry like any compiled change (sections 5-6), plus the PR's own tests. For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past decisions: - 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional `-ncmoe` (all measured losses or no win on this card, FORK.md section 4). - 2026-10-09: the routing-ids readback skip had nothing to apply to (upstream #29943 deleted its block in `ggml-backend.cpp`); it was re-authored in `llama_context::sched_copy_experts`. Upstream's new MoE cache (#29887) needed a new fork commit so the VRAM arbiter releases and invalidates it. - Also check CLI changes that break production config or the scripts (upstream #28334 removed `--no-mmap`/`--mmap`/`--mlock` and llama-bench `-mmp`; the default `--port` changed to 9931). ## 2. Trial rebase in a scratch worktree Measure the conflicts before doing the real thing: ```sh git worktree add /tmp/trial-rebase origin/master cd /tmp/trial-rebase git cherry-pick "$OLD_BASE".."$OLD_TIP" # stop at each conflict, note file + hunk, then: git cherry-pick --abort # or resolve roughly to see the next one cd - && git worktree remove --force /tmp/trial-rebase ``` Write down, per commit: clean / conflict (which file, which hunk, what both sides did). That list becomes the brief for the real rebase. ## 3. The real rebase ```sh NEW=fable5/vram-ondemand-rebase-$(date +%Y%m%d) git switch -c "$NEW" origin/master git cherry-pick # one at a time, in the original order ``` Rules: - Cherry-pick in the original order, keep authors and messages. New commits you author get the fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block), unless the owner asks for something else. - Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms. - Fold each fix into the commit it belongs to so every commit builds on its own: `git commit --fixup=` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`. - Leave the previous production branch untouched. **What `-X theirs` silently broke in a trial (2026-10-09).** It "applied" every commit and produced broken code: - resurrected the scheduler readback block upstream had deleted from `ggml-backend.cpp`; - dropped `!use_sparse` from the GCN `use_mask_opt` condition; - dropped upstream's `copy_experts_info` (compile error); - dropped upstream's `is_empty()` in the recurrent memory; - dropped upstream's MoE-cache memory breakdown block; - left a static/extern mismatch with the new `ggml-vulkan-common.h` declarations (compile error). The correct resolutions in that rebase: keep upstream's `n_embd` width check plus our decode auto-restore; `copy_experts_info` plus our `kv_device_evicted`; `is_empty()` plus our `dev_shadows`; upstream's `if (moe_cache)` breakdown block outside our `if (sched)` guard; make `ggml_vk_concat` and `ggml_backend_vk_free` non-static to match the new header while our helpers stay static. **Clean picks that were wrong (2026-10-24).** - Concat fast path (3df719e1e): applied cleanly, did not compile. Upstream `733f57a99` renamed `pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to `pipeline_cpy_32`/`pipeline_cpy_16` (bitwise copies, same push constants). - MoE cache release/restore (7ac2014a8): applied cleanly, did not compile (upstream `33d6135cb` changed LRU to SLRU, `g.lru` -> `g.slru`) and had a runtime bug: `moe_cache_slru::init()` resets only the probation segment, so re-initialising a used cache corrupts its lists. `invalidate()` now assigns a fresh `moe_cache_slru()` first. - VRAM sharing core (481be8b28): besides its one conflict, upstream's new `load_tensors_hadamard()` pushes two more entries into `pimpl->ctxs_bufs`. Without matching `weight_release.emplace_back()` calls, `release_device_weights()` reads past the end of `weight_release`. Nothing conflicted. - Conflicts that round: `ctx_buf_maps.emplace_back` replaced upstream by `ctx_bufs.emplace_back(ctx, std::move(buf_ptrs))` (keep upstream's line, our release-slot block before it); `llama-context.h` (keep upstream's `n_reallocs_same/diff` beside our `kv_device_evicted`); `recv_with_timeout` (upstream `4e201388c` uses a fixed deadline: keep our order "check `!running`, return null", then `wait_until(lock, deadline)`). **What the 2026-11-21 rebase (upstream `e6d624b9a`, 757 commits past `3f03b1772`) changed** (nothing built or run locally; every item below is by reading only and needs the pod build): - Upstream refactors that moved fork code (hand ports, not conflicts of one hunk): - `clip.cpp`/`clip.h` are gone (#31865). The encoder release/restore (F5) is now `mtmd_model_release_device`/`mtmd_model_restore_device` on `mtmd_model` (`mtmd-model.cpp`), called from `mtmd_release_device`/`mtmd_restore_device` under `compute_mutex` after syncing the shared scheduler; the page-cache drop of F10 moved to `mtmd_model_loader::load_tensors`. - `server_prompt_cache` moved to `server-slot-sched.{h,cpp}` and `get_available_slot()` was replaced by `server_slot_sched::select()`. The disk prompt cache (F11) lives there now (`server_prompt_disk_cache`), is called from the task dispatch after `update_prompt_cache()`, and uses `common_state_data` and `shared_ptr`. Only prompts made with the server lora config are saved, and a restored prompt gets that config in `prompt.lora`. - Upstream's `llama_memory_bufs` replaced the raw `ctxs_bufs` vectors of the KV cache and the recurrent memory: the null-buffer guard moved into `llama_memory_bufs::memory_breakdown()`, and the release code indexes `ctxs_bufs.bufs[i]`. Those two did not compile after a clean pick. - The MoE cache is now a ClassLRU (`moe_cache_lru`, `g.lru`) with `no_alloc` as a constructor argument and `cpu_share`. `invalidate()` rebuilds `g.lru`, refills every slot map with `g.lru.empty` (not -1), clears the CPU maps and `cpu_experts`. - `server.cpp` was split into `server_run_instance` and friends: the self-pipe shutdown thread (F7) is started in `server_run_instance` only, just before `server_install_signal_handlers()`. - `server_queue` got `n_inflight` (#32133). `should_sleep` checks it first, so a response that is still open also blocks the want-lock yield. This is a decision, not an accident: check it with `--arbiter`. - Upstream added its own tiled-transpose concat (#30591) with no size gate. The fork path still runs first (`ne0 >= 32`); the gate and the `GGML_VK_CONCAT_FASTPATH` knob were added to upstream's selector (`ggml_vk_concat_is_transposed`), otherwise decode would take upstream's tiled path again (F3). - Memory types with no release override (stay in VRAM when cold, not used in production): `llama_kv_cache_dsa`, `_dsa_iswa`, `_dsv4`, `_msa`, `llama_memory_hybrid_iswa`. The KV rotation tensors (`attn_rot_bufs`, #31073) are small and are not released either. Open questions from that rebase: - Is the fork's own concat fast path (F3) now redundant with upstream's tiled concat (#30591)? Benchmark `GGML_VK_CONCAT_FASTPATH=0` against the default on pp and decode before deciding. - Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no: handoff median candidate 692 ms vs reference 1068 ms (`a-handoff-latency`, 2026-11-21). **Semantic re-checks to do for every rebase, clean or not** (details in FORK.md): - F1: `sched_copy_experts` and its `st` struct unchanged in meaning. - F2: `use_mask_opt` still contains upstream's terms plus our `>= 256` (and `!bias &&` from #32740, carried as F12 or merged upstream). - F3: `build_conv_state` still emits `ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0)` with `qkv_mixed` contiguous `[ch, n_seq_tokens, n_seqs]`; pipeline names; `types_ok` still excludes quantized types. - F4: every `ctxs_bufs` push in `src/llama-model.cpp` has a `weight_release` push (`git grep -n "ctxs_bufs.emplace_back\|weight_release" src/llama-model.cpp`); list memory types (`git grep -n "public llama_memory_i" src/`) and note any without a release override; MoE-cache internals; `sched_reserve()` still goes through `sched_new()` and re-registers callbacks; `free_scratch` still the last `ggml_backend_i` member; Vulkan prealloc fields match `ggml_vk_cleanup`. - F6/F7: `server_queue::start_loop` still evaluates `should_sleep()` only from the idle branch, never inside `yield_to_queue()`; `handle_sleeping_state`, `server_res_generator`, `load_model` ordering; signal setup in `server.cpp`. - F8: new iterators over `ctxs_bufs` or users of `sched` on teardown paths. - Scripts: flag spellings used by `scripts/rx580-bench/*` and `scripts/fork-validate.sh`, and the log lines `fork-validate.sh` greps for (FORK.md F9). - Production config: every flag in `/root/config.yaml` still exists in `common/arg.cpp`. ## 4. Checks without a compiler Before handing the branch over for a build: ```sh git diff origin/master..HEAD | grep -n '^[+-]\(<<<<<<<\|=======\|>>>>>>>\)' # no conflict markers git diff origin/master..HEAD | grep -nP '^\+.*[^\x00-\x7F]' # ASCII only in added lines git range-diff "$OLD_BASE".."$OLD_TIP" origin/master..HEAD # patch vs patch ``` - The range-diff must show only the edits you made on purpose. In 2026-10-24 the whole patch was identical to the old one except for 4 deliberate edits. - Brace balance per changed file: the change in `{`/`}` balance from new base to new tip should equal the change from old base to old tip. - Every identifier in added lines must exist on the new master or be defined by the fork. This catches a renamed function but not a renamed member (like `lru` -> `slru`); re-read member accesses by hand. - Write down every spot where you would normally lean on the compiler and hand that list to whoever builds. Then the owner pushes the branch to gitea, and the pod fetches it from there (section 5). **If you were told not to push**, do not stop at an unbuilt branch. Unless the task says otherwise, a rebase task continues through the build and validation (sections 5 and 6) and reports the results; the owner pushes afterwards. Get the branch onto the pod with a git bundle: ```sh # workstation; is a commit the pod already has, e.g. production's HEAD git bundle create /tmp/x.bundle .. export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig kubectl -n llama exec -i deploy/supervisord -- sh -c 'cat > /root/x.bundle' < /tmp/x.bundle kubectl -n llama exec deploy/supervisord -- sh -c 'cd /root/llama.cpp && git fetch /root/x.bundle :' ``` The branch is then local in `/root/llama.cpp`: in section 5 skip the `git fetch origin` and use `git worktree add "$W" `. Delete `/root/x.bundle` afterwards (disk is tight). Deploy step 2 likewise uses the local branch instead of fetching from origin. ## 5. Build on the pod, in a scratch worktree Production keeps serving during this step. ```sh export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig kubectl -n llama exec -i deploy/supervisord -- sh -c ' set -e B=fable5/vram-ondemand-rebase-YYYYMMDD W=/root/arms/rebYYYYMMDD git -C /root/llama.cpp fetch origin "$B" git -C /root/llama.cpp worktree add "$W" FETCH_HEAD cd "$W" export HF_UI_VERSION=b11906 # production's: cut -d'|' -f2 /root/llama.cpp/build/tools/ui/.ui-stamp cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8 ' ``` - Build only the targets you need; the full `all` target fails on the UI assets (needs npm/network). If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed). - `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes. - Report compile errors verbatim against the list from step 4. - Do not overlap a build with a large benchmark (host OOM). Disk is tight: `/root` had about 13 GB free (97% full) on 2026-10-25, so check `df -h /root` first. Each build tree is about 700 MB; remove scratch worktrees you no longer need with `git -C /root/llama.cpp worktree remove `. - `-DLLAMA_BUILD_UI=OFF` does NOT make the UI asset step harmless. The `llama-ui-assets` step still runs, and when the build number has no HF tag it falls back to `latest`. On 2026-11-21 `latest` broke the build (missing `bundle[hash].js`, `.css` and `workbox[hash].js`). Set `HF_UI_VERSION=` for both `cmake -B build` and `cmake --build`, where the stamp is production's current one in `build/tools/ui/.ui-stamp` (b11906 on 2026-11-21). The file holds `ggml-org/llama-ui|b11906` and CMake requires `^[A-Za-z0-9._-]+$`, so take only the tag with `cut -d'|' -f2 build/tools/ui/.ui-stamp`; never pass `$(cat ...)`. The validated worktree's UI assets may then differ from the deployed ones (the code is identical). - Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 /root/arms/X.sh' < X.sh`), then start it alone: `setsid nohup sh /root/arms/X.sh > X.log 2>&1 X && sh -n X && setsid ... &` inside the `kubectl exec` shell. The chain is a child of that session, and it can die with the session before `setsid` runs. ## 6. Validate with `scripts/fork-validate.sh` Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of the script describes each check. Rule: any test of a production model must use production's exact command line: the `cmd` of its `/root/config.yaml` entry with the macros expanded, including the mmproj that `-hf` pulls (production never passes `--no-mmproj` to the 35B or gemma), and supervisord's `environment=`. Change only `--port`, the arena and scratch paths. The 2026-10-25 deploy of the disk prompt cache aborted on the first request: every test had used `--no-mmproj`, which hid a `GGML_ASSERT(!has_mtmd)`, and it was rolled back (FORK.md F11). ```sh # phase 1, beside production bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \ --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat ``` Add `--arbiter` to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus the bytes in `cat /sys/class/drm/card*/device/mem_info_vram_used`); below that it FAILs at once. With gemma-26B warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run `--arbiter` in the phase-2 swapdown window instead (about 2 minutes more downtime), as below. **Page cache before and after phase 2.** The `--ref` build may lack F10 (FORK.md), and `--perf` loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check `/proc/meminfo`. ```sh # phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap cat > /root/arms/fv-perf.sh <<'EOF' HUB=/root/.cache/huggingface/hub Q35=$HUB/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf G26=$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/snapshots/e6759b29958180a998f62f991e4c741fe9b4d452/gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \ --build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build \ --arbiter --perf --moe-exact "$Q35" --moecache "$G26" --keep EOF setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 ` (pod-only helper, not in the repo) stops llama-swap, waits for an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints `SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200. - The `--concat` check needs a build with the `GGML_VK_CONCAT_FASTPATH` knob (in the fork since e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`). - `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits, which proves nothing). - `--arbiter` also checks that a cold server really frees its VRAM (`v-vram-release`, `v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs ordinary handoff latency against `--ref` (`a-handoff-latency`). `--ref` must be the build currently in production, so both comparisons are against what serves today. - Phase 2 downtime, measured 2026-10-25: about 32 minutes in all (`--arbiter` about 2 min, `--perf` about 28 min, `--moecache` under 30 s, `--moe-exact` 78 s). `--moe-exact` proves the routing-ids readback skip exact (F1): the candidate's greedy output equals the reference's and, with the `LLAMA_MOE_IDS_SKIP` knob (any build of this branch), the candidate's with the skip forced off. - `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in `/perf.tsv`, and `fork-validate.sh --perf-report ` re-aggregates them offline. A pod restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24 scratch dirs were lost that way). - Last full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2 9 PASS plus the 35B disk-cache legs (`q35.py`, mmproj loaded). Perf medians ref -> cand: pp2048 259.39 -> 286.61 (+10.5%), d16384 199.26 -> 207.63 (+4.2%), d32768 156.61 -> 157.51 (+0.6%), tg256@d4096 9.17 -> 8.85 t/s (noisy, not gated). Handoff median: candidate 692 ms, reference 1068 ms. Downtime: about 42 minutes of validation and about 7 minutes of deploy. - Previous full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs `/root/arms/dep-2fc9f9c35-fv1.log` and `-fv2.log`): phase 1 `--ops --concat` 8 PASS, phase 2 `--arbiter --perf --moe-exact --moecache` 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048 274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff median ref 693.5 / cand 678 ms. `--moe-exact`: 39 skips logged, identical to the reference and to the skip forced off. Peak VRAM per arm ref 5476, cand 7062, cand 5767, ref 5476 MiB: the ~1.6 GiB spike in the first candidate arm has now been seen twice (open question, FORK.md F9). - Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a regression gate, not a measurement. - Disk prompt cache (FORK.md F11, from `fable5/disk-prompt-cache`): add `--diskcache` to phase 1. It is CPU only and runs beside production (about 2 minutes). Its GPU leg SKIPs below 1536 MiB of free VRAM; then run `--diskcache` again in the phase-2 window. A 35B measurement driver is on the pod: `/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script `/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window3.sh` runs `q35.py` with production's exact 35B command line (mmproj loaded): save/restore, cold save, the same-reuse-point identity and an image request (FORK.md F11). Run 2026-11-21 on adc2aa3ec: all legs passed (18.8k-token chat, prefill 84.6 s -> turn 2 4.7 s after the restart, identical output; save 557 MB in 1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image answered and not saved; log `/root/arms/diskcache-test/q35-mm.log`). It supersedes `diskcache-window2.sh` (`cmp.py` runs with `--no-mmproj`). `q35.py` and the window scripts live only on the pod (not in the repo) and hard-code a build path. Before each use, edit them to point at the candidate build (`/root/arms/rebYYYYMMDD/build`), not production's `/root/llama.cpp/build`, or you validate the wrong binary. On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs. ## 7. Deploy Deploy exactly the validated commit. Downtime: 6 minutes for the 2026-10-24 redeploy. The deploy rebuilds the production tree in place (step 6) from the same commit and with the same cmake flags as the validated worktree build, so nothing is copied over; compare `grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt` in both trees if in doubt. Run the stop, build and start (steps 4-7) as one script on the pod, started with `setsid nohup sh