scripts: add the RX 580 benchmark harness

Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ

README.md

@@ -17,6 +17,82 @@
 
 </div>
 
+## This fork - Polaris / GCN tuning for large MoE models
+
+Changes and measurements for running large MoE models with their experts offloaded to system RAM
+(`--n-cpu-moe`) on an old GCN card. The two code changes below are auto-on, need no flag, and are
+**token-identical** to mainline. Everything else here is tuning guidance.
+
+Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16, no fp16, no matrix cores) with
+**Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`:
+
+- **Flash-attention `mask_opt` is enabled for GCN large head sizes (this fork's own change).**
+  Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips
+  fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on
+  a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B
+  (head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical.
+- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp).
+- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short
+  prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that
+  VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`).
+
+**Recommended RX 580 / Polaris serving command** (per model):
+
+```bash
+llama-server -hf <repo>:<quant> -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1
+```
+
+Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high
+for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so
+`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path.
+
+### Very large MoE (experts bigger than the GTT limit)
+
+Measured on the same RX 580 with **Laguna-S-2.1 118B IQ2_M** (48 layers, 256 experts, 10 used,
+experts 30.7 GiB of a 34.7 GiB file, all host-resident at `--n-cpu-moe 48`). The advice above changes
+in this regime:
+
+- **`--no-mmap` stops being an option.** Its pinned host buffer is charged against the amdgpu GTT
+  limit (~31.4 GiB here, about half of system RAM). At 30.7 GiB of experts the model no longer
+  loads, and the allocation spike can OOM the box. Use mmap and accept the staging copy.
+- **The routing-ids readback is pure overhead at prefill batch sizes** and this fork now skips it.
+  To decide which experts to upload, the scheduler read the ids back from the device that had just
+  produced them, forcing a full pipeline flush once per MoE layer per eval. With 2048 tokens x 10
+  experts over 256 experts every expert is used anyway. Skipping it is exact - `mul_mat_id` only
+  reads the rows the ids point at. Auto-on above `4 * n_expert` ids; decode keeps the old path.
+  **pp2048 +4.7% @ 32k depth, +0.9% @ 16k, tg +3%.**
+- **Where the time actually goes** (`GGML_VK_PERF_LOGGER=1`, depth 0, 22.65 s per 2048-token eval,
+  17.18 s of it GPU-busy so ~24% is H2D stall): expert `MUL_MAT_ID` **52%**, attention projections
+  **24%**, `FLASH_ATTN_EXT` **16%**, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s
+  while dense `MUL_MAT` q5_K/q6_K in the same graph reaches 3096-3691 - **the single largest
+  remaining opportunity on this hardware is closing that gap**, not the transfer path.
+- **Interleaved SWA keeps its own small KV cache**, so a sliding-window layer costs the same at any
+  depth (`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`). On this model 36 of 48 layers are O(1) in
+  depth and the entire high-context slowdown comes from the 12 full-attention layers.
+- **`--parallel 1`** is worth setting for a solo large model: the server otherwise auto-selects 4
+  slots, and the SWA cache is sized `n_swa * n_seq_max + n_ubatch`, so 4 slots cost 4096 cells
+  instead of 2560. Measured **223 MiB of VRAM freed** at 64k context.
+
+Dead ends measured on this hardware, recorded so they are not retried:
+
+| Change | Result |
+| --- | ---: |
+| flash-attn `shmem_staging` enabled for GCN | **-6.7% @ 16k, -7.4% @ 32k** |
+| `-b 4096 -ub 4096` (to amortize the fixed per-eval expert upload) | flat (-1%) |
+| `--n-cpu-moe` 44 instead of 48 | +1.5%, but does not fit at 64k ctx |
+| `mask_opt` gate relaxed below head_dim 256 | -18.5% @ 16k |
+
+`shmem_staging` looks like a certain win (without it each rowgroup re-reads the whole K/V block
+through a 16 KiB L1) but the `kvsh` stride of `D/4+1` dwords is 4 mod 32, which costs an 8-way LDS
+bank conflict on wave64 - that `+1` padding is tuned for warp32. `-ub 4096` fails because halving the
+number of expert uploads is exactly cancelled by intra-ubatch attention growing quadratically.
+
+**Serving note that outweighs all of the above.** With a model this large, anything that restarts the
+process is far more expensive than any kernel win: the server's prompt cache is RAM-only with no disk
+backing, so a restart forces a full re-prefill of the conversation. If a model swapper can evict this
+model to run a small helper model (chat-title generation and the like), fix that first - keeping the
+process alive across a swap took a repeat turn from a 21,960 ms prefill down to 225 ms.
+
 ## Quick start
 
 A few options to get `llama.cpp` installed on your machine:

ggml/include/ggml-backend.h

@@ -104,6 +104,11 @@ extern "C" {
     GGML_API enum ggml_status ggml_backend_graph_compute      (ggml_backend_t backend, struct ggml_cgraph * cgraph);
     GGML_API enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph);
 
+    // Free transient/scratch device memory the backend holds outside of any allocated buffer
+    // (compute preallocations, staging buffers). No-op if the backend does not implement it.
+    // The backend remains usable; scratch is reallocated lazily on the next compute.
+    GGML_API void ggml_backend_free_scratch(ggml_backend_t backend);
+
     // NOTE: will be removed, use device version instead
     GGML_API bool ggml_backend_supports_op(ggml_backend_t backend, const struct ggml_tensor * op);
     GGML_API bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft);

ggml/src/ggml-backend-impl.h

@@ -153,6 +153,11 @@ extern "C" {
 
         // (optional) sort/optimize the nodes in the graph
         void                      (*graph_optimize)    (ggml_backend_t backend, struct ggml_cgraph * cgraph, struct ggml_backend_graph_optimize_params * params);
+
+        // (optional) free transient/scratch device memory the backend holds outside of any buffer
+        // (e.g. compute preallocations and staging buffers). The backend stays usable; the scratch
+        // is reallocated lazily on the next compute. Used to shrink an idle model's device footprint.
+        void                      (*free_scratch)      (ggml_backend_t backend);
     };
 
     struct ggml_backend {

ggml/src/ggml-backend.cpp

@@ -431,6 +431,15 @@ void ggml_backend_synchronize(ggml_backend_t backend) {
     backend->iface.synchronize(backend);
 }
 
+void ggml_backend_free_scratch(ggml_backend_t backend) {
+    GGML_ASSERT(backend);
+    if (backend->iface.free_scratch == NULL) {
+        return;
+    }
+
+    backend->iface.free_scratch(backend);
+}
+
 ggml_backend_graph_plan_t ggml_backend_graph_plan_create(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
     GGML_ASSERT(backend);
     GGML_ASSERT(backend->iface.graph_plan_create != NULL);

ggml/src/ggml-vulkan/ggml-vulkan.cpp

@@ -13579,6 +13579,8 @@ static void ggml_vk_concat(ggml_backend_vk_context * ctx, vk_context& subctx, co
         const uint32_t ts = ggml_type_size(s->type);
         return s->nb[1] == ts                                              // dim1 innermost
             && s->nb[0] == (size_t) s->ne[1] * ts                          // consistent 2D transpose
+            && s->ne[0] >= 32                                              // at least one full transpose tile (TILE_DIM); at 1 token the
+                                                                           // tiled path is pure overhead, the generic copy is faster
             && s->ne[2] == 1 && s->ne[3] == 1;
     };
     const uint32_t dst_ts = ggml_type_size(dst->type);
@@ -17184,6 +17186,35 @@ static const char * ggml_backend_vk_name(ggml_backend_t backend) {
     return ctx->name.c_str();
 }
 
+// Free the compute-scratch device buffers (prealloc_* and the transfer staging buffer) without
+// tearing down the backend (pipelines, command pools, fences and device stay alive). These buffers
+// are reallocated lazily by ggml_vk_preallocate_buffers() on the next compute, so this just shrinks
+// an idle model's device footprint. Mirrors the buffer-freeing subset of ggml_vk_cleanup().
+static void ggml_backend_vk_free_scratch(ggml_backend_t backend) {
+    ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
+    VK_LOG_DEBUG("ggml_backend_vk_free_scratch(" << ctx->name << ")");
+
+    // discard any unsubmitted command buffer and wait for in-flight work before freeing
+    ctx->compute_ctx.reset();
+    ggml_vk_synchronize(ctx);
+
+    ggml_vk_destroy_buffer(ctx->prealloc_x);
+    ggml_vk_destroy_buffer(ctx->prealloc_y);
+    ggml_vk_destroy_buffer(ctx->prealloc_split_k);
+    ggml_vk_destroy_buffer(ctx->prealloc_add_rms_partials);
+    ggml_vk_destroy_buffer(ctx->sync_staging);
+
+    ctx->prealloc_y_last_pipeline_used = nullptr;
+    ctx->prealloc_y_last_tensor_used = nullptr;
+    ctx->prealloc_y_last_k_padded = false;
+
+    ctx->prealloc_size_x = 0;
+    ctx->prealloc_size_y = 0;
+    ctx->prealloc_size_split_k = 0;
+    ctx->prealloc_size_add_rms_partials = 0;
+    ctx->prealloc_size_add_rms_partials_offset = 0;
+}
+
 static void ggml_backend_vk_free(ggml_backend_t backend) {
     ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
     VK_LOG_DEBUG("ggml_backend_vk_free(" << ctx->name << ")");
@@ -19122,6 +19153,7 @@ static ggml_backend_i ggml_backend_vk_interface = {
     /* .event_record            = */ ggml_backend_vk_event_record,
     /* .event_wait              = */ ggml_backend_vk_event_wait,
     /* .graph_optimize          = */ ggml_vk_graph_optimize,
+    /* .free_scratch            = */ ggml_backend_vk_free_scratch,
 };
 
 static ggml_guid_t ggml_backend_vk_guid() {

scripts/rx580-bench/README.md

@@ -0,0 +1,200 @@
+# rx580-bench
+
+Prompt-processing benchmark harness for the RX 580 (Polaris) serving pod.
+
+It exists so that no future session has to rediscover how to measure this
+machine. It encodes the production config, fixed corpus slices, the repeat
+and median discipline, and the noise floor.
+
+## What it measures
+
+Two things, deliberately:
+
+1. **Corpus prompt processing** (the number we actually care about). Real
+   Polish prose from `pan-tadeusz.txt` pushed through `llama-server`'s
+   `/completion` endpoint with `n_predict: 1`, `cache_prompt: false`. This
+   mimics agentic tool-result parsing: a big blob of real text arriving cold.
+   Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k
+   and 32k tokens.
+2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random
+   tokens, `pp2048`, `pp8192`, `tg32`. This is what prior work on this box
+   recorded, so it is the continuity metric.
+
+They do not measure the same thing and they do not agree. `llama-bench`
+pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
+growing KV cache, so they fall off with prompt length. Both are useful.
+
+## How to run it
+
+One command from the workstation:
+
+```
+./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean
+```
+
+Interleaved A/B, which is the only valid way to compare two builds:
+
+```
+./scripts/rx580-bench/run.sh \
+  --build   /root/arms/new-clean/build --label   new-clean \
+  --build-b /root/arms/new-fork/build  --label-b new-fork \
+  --rounds 3
+```
+
+Summarize whatever has accumulated:
+
+```
+./scripts/rx580-bench/run.sh --summarize
+```
+
+`run.sh` copies the harness onto the pod and execs it there. The wrapper is
+thin on purpose: all the methodology lives in the pod-side scripts, so it can
+also be driven directly on the pod if kubectl is inconvenient.
+
+The run is detached on the pod and then tailed, so Ctrl-C on the wrapper does
+not orphan a half-finished run. `bench.sh` restarts `llama-swap` from an EXIT
+trap on success, failure and interrupt alike.
+
+### Pod access
+
+```
+export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
+kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
+```
+
+The pod has `curl` and `python3` but **no `jq`**; the harness uses python3 for
+all JSON.
+
+## Files
+
+| file | side | what |
+|---|---|---|
+| `run.sh` | workstation | one-command wrapper; installs and execs the rest |
+| `bench.sh` | pod | orchestrator: stops llama-swap, waits for idle GPU, runs both harnesses, restarts llama-swap via trap |
+| `ppbench.py` | pod | the corpus prompt-processing harness |
+| `lbsweep.sh` | pod | the llama-bench sweep |
+| `summarize.py` | pod | turns `results.txt` into a median/min-max table with delta significance |
+
+Results append to `/root/bench/results.txt` as parseable `RESULT` /
+`SUMMARY` / `LBRESULT` lines. Server logs land in `/root/bench/server-*.log`.
+
+## The config it assumes
+
+Production serving config, matching `/root/config.yaml`:
+
+```
+-t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --no-mmap
+--ctx-size 40960 --no-warmup
+```
+
+Model:
+
+```
+/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
+```
+
+Qwen3.6-35B-A3B, 20.8 GiB, 40 layers, 256 experts of which 8 are used.
+Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16,
+12 CPU cores, 62 GB RAM.
+
+The `llama-bench` equivalent of that config is:
+
+```
+-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -mmp 0 -p 2048,8192 -n 32 -r 3
+```
+
+## The corpus slices are constants, not recalibrated
+
+`pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this
+model's tokenizer it runs **2.6702 chars/token**, and `/completion` reports
+`prompt_n` identical to `/tokenize` (BOS offset 0).
+
+The slices are `corpus[0:NCHARS]` and are baked into `ppbench.py`:
+
+| target tokens | chars | utf-8 bytes |
+|---|---|---|
+| 4096 | 10776 | 11608 |
+| 16384 | 43858 | 47175 |
+| 32768 | 87165 | 93843 |
+
+These hit the target `prompt_n` exactly. Every build is therefore measured on
+byte-identical input. `ppbench.py` warns loudly if the reported `prompt_n`
+ever stops matching the target, which is the signal that the table is stale.
+Only recalibrate (`ppbench.py <build> <label> --calibrate`) if the model file
+or the corpus changes.
+
+## Reading the numbers
+
+* `prompt_n` is the token count actually prefilled. It must equal the target.
+* `prompt_ms` is wall-clock prefill time.
+* `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better.
+* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0.
+* `tg32` is token generation at DEFAULT depth (n_ctx 32). **It is NOT a valid
+  decode metric on this box.** At that size decode is launch/barrier bound, not
+  compute bound, and measures a regime serving never reaches: it read -36%
+  between two upstream commits that are at parity in real use (the cause was
+  the view-alias fix in ggml_vk_graph_optimize). For decode use **tg256 with an
+  explicit depth**, e.g. `-p 0 -n 256 -d 2048`. tg is also NOT monotonic in
+  n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 -
+  so only ever compare same-depth cells.
+
+## Noise floor: deltas under about 5 percent are unproven
+
+This is the single most important thing in this directory.
+
+`llama-bench`'s within-run error bars understate **cross-invocation** variance
+by roughly **14x**. The same config measured `276.7 +/- 0.9` in one invocation
+and `296.4 +/- 0.6` in another, 7 percent apart, with error bars that claimed
+0.3 percent precision.
+
+Therefore:
+
+* Any comparison you want to draw a conclusion from must live **inside a
+  single `llama-bench` invocation**, using comma-separated sweeps.
+* Two different *builds* cannot share an invocation, so run them
+  **interleaved A/B/A/B/A/B across at least 3 rounds** and compare medians.
+  That is what `--build-b` does.
+* State explicitly that any cross-build delta under about 5 percent is
+  unproven. `summarize.py` labels them `unproven` for you.
+
+## Traps
+
+* **`-tb` is not a `llama-bench` flag.** It is a `llama-server` flag. Passing
+  it makes `llama-bench` print usage and exit silently, mid-sweep, which looks
+  exactly like a run that produced nothing. Always check the row count.
+* **A ubatch larger than the prompt never fills.** With `-ub 2048`, prompt
+  lengths must be multiples of 2048 or the sweep measures nothing meaningful.
+* **Nothing else may touch the GPU.** `bench.sh` stops `llama-swap` and
+  refuses to start if any `llama-*` process is still alive.
+* **Never leave the pod not serving.** `bench.sh` restarts `llama-swap` from
+  an EXIT trap. If you bypass the script, restart it by hand:
+  `supervisorctl -c /root/supervisord.conf start llama-swap`.
+* **Do not run `llama-cli` detached with closed stdin** on this box; it loops
+  forever. Use the server harness or `llama-bench -d` for depth timing.
+* `--no-mmap` is deprecated upstream in favour of `--load-mode` / `-lm none`,
+  but still accepted and still what production passes.
+
+## Known-good reference values
+
+If a fresh run disagrees with these by much more than the noise floor,
+something is wrong (something else on the GPU, a thermal problem, a bad
+build) before you believe you found a speedup.
+
+Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config, current upstream
+(`f3f1a8f27`) and the deployed fork, measured 2026-09-08/09:
+
+| metric | clean upstream | fork |
+|---|---|---|
+| corpus pp @ 4k | about 240 t/s | about 255 t/s |
+| corpus pp @ 16k | about 206 t/s | about 225 t/s |
+| corpus pp @ 32k | about 171 t/s | about 195 t/s |
+| llama-bench pp2048 | see results.txt | about 285 t/s |
+| llama-bench tg32 | | not a valid metric, see above |
+
+Older reference points for the same model/config: `pp2048` depth-0 about
+285 t/s. The historical "tg32 about 19 t/s" figure is retained only as trivia -
+do not treat it as a target, and see the tg32 warning above.
+
+Model load with `--no-mmap` takes about 26 s warm, longer cold.
+A full 3-round A/B run of both harnesses takes roughly 100 minutes;
+add about 16 minutes per extra arm per round.

scripts/rx580-bench/bench.sh

@@ -0,0 +1,167 @@
+#!/bin/sh
+#
+# RX 580 prompt-processing benchmark orchestrator. Runs ON THE POD.
+#
+# Stops llama-swap, waits for the GPU to go idle, runs the real-corpus
+# prompt-processing harness and the llama-bench sweep for every arm, appends
+# parseable results, and ALWAYS restarts llama-swap on exit (success, error
+# or Ctrl-C).
+#
+# Usage:
+#   bench.sh --build DIR --label NAME
+#            [--build-b DIR --label-b NAME]   second arm, interleaved
+#            [--arm DIR:LABEL ...]            extra arms, repeatable (N-way)
+#            [--rounds N]                     default 3 multi-arm, 1 otherwise
+#            [--reps N]                       corpus repeats per size, default 3
+#            [--sizes 4096,16384,32768]
+#            [--corpus-only | --lb-only]
+#            [--results PATH]                 default /root/bench/results.txt
+#            [--keep-swap-down]               do not restart llama-swap at exit
+#
+set -e
+
+BENCH_DIR=/root/bench
+SUPCONF=/root/supervisord.conf
+RESULTS="$BENCH_DIR/results.txt"
+MODEL=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
+
+ARMS=""            # space separated DIR|LABEL entries
+BUILD_A=""; LABEL_A=""
+BUILD_B=""; LABEL_B=""
+ROUNDS=""
+REPS=3
+SIZES=4096,16384,32768
+DO_CORPUS=1
+DO_LB=1
+KEEP_SWAP_DOWN=0
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --build)     BUILD_A="$2"; shift 2 ;;
+    --label)     LABEL_A="$2"; shift 2 ;;
+    --build-b)   BUILD_B="$2"; shift 2 ;;
+    --label-b)   LABEL_B="$2"; shift 2 ;;
+    --arm)       ARMS="$ARMS ${2%%:*}|${2#*:}"; shift 2 ;;
+    --rounds)    ROUNDS="$2"; shift 2 ;;
+    --reps)      REPS="$2"; shift 2 ;;
+    --sizes)     SIZES="$2"; shift 2 ;;
+    --results)   RESULTS="$2"; shift 2 ;;
+    --corpus-only) DO_LB=0; shift ;;
+    --lb-only)     DO_CORPUS=0; shift ;;
+    --keep-swap-down) KEEP_SWAP_DOWN=1; shift ;;
+    -h|--help)   sed -n '2,20p' "$0"; exit 0 ;;
+    *) echo "unknown arg: $1" >&2; exit 2 ;;
+  esac
+done
+
+if [ -n "$BUILD_A" ] && [ -z "$LABEL_A" ]; then
+  echo "error: --build needs --label" >&2; exit 2
+fi
+if [ -n "$BUILD_B" ] && [ -z "$LABEL_B" ]; then
+  echo "error: --build-b needs --label-b" >&2; exit 2
+fi
+# --build/--label and --build-b/--label-b are sugar for the first two arms.
+if [ -n "$BUILD_A" ]; then ARMS="$BUILD_A|$LABEL_A $ARMS"; fi
+if [ -n "$BUILD_B" ]; then ARMS="$ARMS $BUILD_B|$LABEL_B"; fi
+ARMS=$(echo $ARMS)
+if [ -z "$ARMS" ]; then
+  echo "error: need at least one arm (--build/--label or --arm DIR:LABEL)" >&2
+  exit 2
+fi
+NARMS=$(echo "$ARMS" | wc -w)
+if [ -z "$ROUNDS" ]; then
+  if [ "$NARMS" -gt 1 ]; then ROUNDS=3; else ROUNDS=1; fi
+fi
+for e in $ARMS; do
+  d=${e%%|*}
+  if [ ! -x "$d/bin/llama-server" ] || [ ! -x "$d/bin/llama-bench" ]; then
+    echo "error: $d lacks bin/llama-server or bin/llama-bench" >&2
+    exit 2
+  fi
+done
+
+cat <<'WARN'
+-------------------------------------------------------------------------
+RX 580 benchmark. Read this before trusting any number.
+
+  * Cross-invocation variance on this box is about 7 percent. llama-bench
+    within-run error bars understate it by roughly 14x. Any cross-build
+    delta under about 5 percent is UNPROVEN noise.
+  * Comparisons you care about must live inside ONE llama-bench invocation
+    (comma-separated sweeps), or be interleaved across at least 3 rounds.
+    That is what multiple arms plus --rounds does.
+  * Nothing else may touch the GPU while this runs. llama-swap is stopped
+    for the duration and restarted on exit, including on error or Ctrl-C.
+  * Traps: -tb is NOT a llama-bench flag. A ubatch larger than -p never
+    fills, so prompt lengths must be multiples of the 2048 ubatch. A bad
+    flag makes llama-bench print usage and exit silently mid-sweep.
+-------------------------------------------------------------------------
+WARN
+
+echo "[bench] $NARMS arm(s), $ROUNDS round(s): $ARMS"
+
+restore_swap() {
+  rc=$?
+  if [ "$KEEP_SWAP_DOWN" -eq 0 ]; then
+    echo ""
+    echo "[bench] restarting llama-swap"
+    supervisorctl -c "$SUPCONF" start llama-swap || true
+    supervisorctl -c "$SUPCONF" status llama-swap || true
+  else
+    echo "[bench] --keep-swap-down: llama-swap left STOPPED"
+  fi
+  exit $rc
+}
+trap restore_swap EXIT INT TERM
+
+echo "[bench] stopping llama-swap"
+supervisorctl -c "$SUPCONF" stop llama-swap || true
+
+wait_gpu_idle() {
+  i=0
+  while [ $i -lt 60 ]; do
+    if ! pgrep -f 'bin/llama-server|bin/llama-bench|bin/llama-cli' >/dev/null 2>&1; then
+      return 0
+    fi
+    sleep 2
+    i=$((i+1))
+  done
+  echo "error: something is still on the GPU:" >&2
+  pgrep -af 'bin/llama-server|bin/llama-bench|bin/llama-cli' >&2
+  exit 3
+}
+
+echo "[bench] waiting for the GPU to go idle"
+wait_gpu_idle
+echo "[bench] GPU idle"
+
+run_corpus() {
+  [ "$DO_CORPUS" -eq 1 ] || return 0
+  echo "===== corpus $2 ====="
+  wait_gpu_idle
+  python3 "$BENCH_DIR/ppbench.py" "$1" "$2" \
+    --sizes "$SIZES" --reps "$REPS" --results "$RESULTS"
+}
+
+run_lb() {
+  [ "$DO_LB" -eq 1 ] || return 0
+  echo "===== llama-bench $2 ====="
+  wait_gpu_idle
+  RESULTS="$RESULTS" MODEL="$MODEL" sh "$BENCH_DIR/lbsweep.sh" "$1" "$2"
+}
+
+# Interleave: every arm is measured once per round, so slow drift in the
+# machine hits all arms roughly equally instead of biasing whichever ran first.
+r=1
+while [ "$r" -le "$ROUNDS" ]; do
+  echo ""
+  echo "########## ROUND $r / $ROUNDS ##########"
+  for e in $ARMS; do run_corpus "${e%%|*}" "${e#*|}"; done
+  for e in $ARMS; do run_lb     "${e%%|*}" "${e#*|}"; done
+  r=$((r+1))
+done
+
+echo ""
+echo "ALL_RUNS_DONE"
+echo "[bench] raw results appended to $RESULTS"
+python3 "$BENCH_DIR/summarize.py" "$RESULTS" || true

scripts/rx580-bench/lbsweep.sh

@@ -0,0 +1,69 @@
+#!/bin/sh
+#
+# llama-bench sweep, the controlled cross-check for the corpus harness.
+# Runs ON THE POD. Normally invoked by bench.sh.
+#
+# Usage: lbsweep.sh <build_dir> <label>
+# Env overrides: RESULTS, MODEL, LB_P, LB_N, LB_R
+#
+# Everything that must be compared lives inside ONE llama-bench invocation
+# (-p takes a comma-separated list), because cross-invocation variance on this
+# box is about 7 percent while the within-run error bars are about 0.3 percent.
+#
+# Flag traps, learned the hard way:
+#   * -tb is NOT a llama-bench flag. It exists on llama-server only.
+#   * -ncmoe is the llama-bench spelling of --n-cpu-moe.
+#   * -mmp 0 is the llama-bench equivalent of the server's --no-mmap.
+#   * A bad flag makes llama-bench print usage and exit silently, which looks
+#     exactly like a sweep that produced no rows. Always check row count.
+#   * A ubatch larger than -p never fills, so keep -p a multiple of -ub.
+#
+set -e
+BUILD_DIR="$1"
+LABEL="$2"
+if [ -z "$BUILD_DIR" ] || [ -z "$LABEL" ]; then
+  echo "usage: $0 <build_dir> <label>" >&2
+  exit 2
+fi
+
+RESULTS="${RESULTS:-/root/bench/results.txt}"
+MODEL="${MODEL:-/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf}"
+LB_P="${LB_P:-2048,8192}"
+LB_N="${LB_N:-32}"
+LB_R="${LB_R:-3}"
+
+STAMP=$(date -u +%Y-%m-%dT%H:%M:%S)
+export LD_LIBRARY_PATH="$BUILD_DIR/bin"
+export LABEL BUILD_DIR STAMP RESULTS
+
+OUT=$("$BUILD_DIR/bin/llama-bench" \
+  -m "$MODEL" \
+  -t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -mmp 0 \
+  -p "$LB_P" -n "$LB_N" -r "$LB_R" -o json 2>/dev/null)
+
+if [ -z "$OUT" ]; then
+  echo "ERROR: llama-bench produced no output for $LABEL (bad flag?)" >&2
+  exit 3
+fi
+
+printf '%s' "$OUT" | python3 -c '
+import json, sys, os
+label = os.environ["LABEL"]
+build = os.environ["BUILD_DIR"]
+stamp = os.environ["STAMP"]
+results = os.environ["RESULTS"]
+rows = json.load(sys.stdin)
+if not rows:
+    sys.stderr.write("ERROR: llama-bench returned zero rows\n")
+    sys.exit(3)
+out = open(results, "a")
+for r in rows:
+    test = "pp%d" % r["n_prompt"] if r["n_prompt"] else "tg%d" % r["n_gen"]
+    line = ("LBRESULT label=%s build=%s test=%s tps=%.2f stddev=%.2f "
+            "samples=%s time=%s"
+            % (label, build, test, r["avg_ts"], r["stddev_ts"],
+               r["samples_ts"], stamp))
+    out.write(line + "\n")
+    print(line)
+out.close()
+'

scripts/rx580-bench/ppbench.py

@@ -0,0 +1,239 @@
+#!/usr/bin/env python3
+"""
+Real-corpus prompt-processing benchmark for llama-server on the RX 580 pod.
+
+Runs ON THE POD. Normally invoked by bench.sh, not directly.
+
+Starts llama-server from a given build directory with the production config,
+waits for /health, POSTs fixed slices of the Pan Tadeusz corpus to /completion,
+records timings.prompt_n / prompt_ms / prompt_per_second, then shuts the server
+down cleanly.
+
+Prompt slices are FIXED CONSTANTS (see SLICES below), calibrated once against
+the Qwen3.6-35B-A3B tokenizer, so that every build is measured on byte-identical
+input. Do not re-calibrate for a normal run; use --calibrate only if the model
+or the corpus changes.
+
+Usage:
+  ppbench.py <build_dir> <label> [--sizes 4096,16384,32768] [--reps 3]
+             [--results PATH] [--calibrate]
+"""
+
+import json
+import os
+import signal
+import subprocess
+import sys
+import time
+import urllib.request
+from statistics import median
+
+BENCH_DIR = "/root/bench"
+CORPUS = os.path.join(BENCH_DIR, "pan-tadeusz.txt")
+DEFAULT_RESULTS = os.path.join(BENCH_DIR, "results.txt")
+
+MODEL = ("/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/"
+         "snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/"
+         "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf")
+
+PORT = 8099
+BASE = "http://127.0.0.1:%d" % PORT
+
+# Production serving config. Keep in sync with /root/config.yaml (llama-swap).
+# Traps encoded here on purpose:
+#   -b / -ub 2048 : a ubatch larger than the prompt never fills, so every
+#                   prompt length below must stay a multiple of 2048.
+#   --no-mmap     : deprecated upstream in favour of --load-mode / -lm none,
+#                   but still accepted; it is what production passes today.
+SERVER_ARGS = ["-t", "6", "-ngl", "99", "--n-cpu-moe", "40",
+               "-b", "2048", "-ub", "2048", "-fa", "1", "--no-mmap",
+               "--ctx-size", "40960", "--no-warmup",
+               "--port", str(PORT), "-m", MODEL]
+
+# Calibrated once on 2026-09-08 against Qwen3.6-35B-A3B-UD-Q4_K_XL:
+# Polish text of pan-tadeusz.txt runs 2.6702 chars/token, and /completion
+# reports prompt_n identical to /tokenize (BOS offset 0).
+# Each slice is corpus[0:NCHARS] read as UTF-8 text; NBYTES is the resulting
+# UTF-8 byte length, recorded so the slice can be reproduced with byte tools.
+# These produce EXACTLY the target prompt_n. Do not edit without recalibrating.
+SLICES = {
+    4096:  {"chars": 10776, "bytes": 11608},
+    16384: {"chars": 43858, "bytes": 47175},
+    32768: {"chars": 87165, "bytes": 93843},
+}
+
+DEFAULT_SIZES = [4096, 16384, 32768]
+
+
+def post(path, payload, timeout=2400):
+    req = urllib.request.Request(
+        BASE + path,
+        data=json.dumps(payload).encode("utf-8"),
+        headers={"Content-Type": "application/json"},
+        method="POST")
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.loads(r.read().decode("utf-8"))
+
+
+def health_ok():
+    try:
+        with urllib.request.urlopen(BASE + "/health", timeout=5) as r:
+            return json.loads(r.read().decode("utf-8")).get("status") == "ok"
+    except Exception:
+        return False
+
+
+def start_server(build_dir, logpath):
+    env = dict(os.environ)
+    env["LD_LIBRARY_PATH"] = os.path.join(build_dir, "bin")
+    binary = os.path.join(build_dir, "bin", "llama-server")
+    if not os.path.exists(binary):
+        raise RuntimeError("no llama-server at %s" % binary)
+    log = open(logpath, "wb")
+    p = subprocess.Popen([binary] + SERVER_ARGS, stdout=log,
+                         stderr=subprocess.STDOUT, env=env,
+                         start_new_session=True)
+    deadline = time.time() + 1200
+    while time.time() < deadline:
+        if p.poll() is not None:
+            raise RuntimeError("llama-server exited early rc=%s, see %s"
+                               % (p.returncode, logpath))
+        if health_ok():
+            return p
+        time.sleep(2)
+    stop_server(p)
+    raise RuntimeError("llama-server not healthy within 1200s, see %s" % logpath)
+
+
+def stop_server(p):
+    if p is None or p.poll() is not None:
+        return
+    try:
+        os.killpg(os.getpgid(p.pid), signal.SIGTERM)
+    except Exception:
+        p.terminate()
+    for _ in range(120):
+        if p.poll() is not None:
+            return
+        time.sleep(1)
+    try:
+        os.killpg(os.getpgid(p.pid), signal.SIGKILL)
+    except Exception:
+        p.kill()
+    p.wait()
+
+
+def n_tokens(text):
+    """Exact token count via /tokenize. Cheap: no prefill."""
+    return len(post("/tokenize", {"content": text}, timeout=300)["tokens"])
+
+
+def calibrate(corpus, targets):
+    """Recompute the SLICES table. Only needed if model or corpus changes."""
+    probe = corpus[:2000]
+    tk = n_tokens(probe)
+    r = post("/completion", {"prompt": probe, "n_predict": 1,
+                             "cache_prompt": False, "temperature": 0})
+    offset = r["timings"]["prompt_n"] - tk
+    cpt = len(probe) / float(tk)
+    sys.stderr.write("calibrate: %.4f chars/token, prompt_n offset %d\n"
+                     % (cpt, offset))
+    for target in targets:
+        want = target - offset
+        lo, hi = 1, len(corpus)
+        guess = min(len(corpus), max(1, int(want * cpt)))
+        best = None
+        for _ in range(60):
+            got = n_tokens(corpus[:guess])
+            if got == want:
+                best = guess
+                break
+            if got < want:
+                lo = guess + 1
+            else:
+                hi = guess - 1
+            if lo > hi:
+                break
+            guess = (lo + hi) // 2
+        if best is None:
+            best = guess
+        nbytes = len(corpus[:best].encode("utf-8"))
+        sys.stderr.write("calibrate: %d tokens -> chars=%d bytes=%d\n"
+                         % (target, best, nbytes))
+
+
+def main():
+    if len(sys.argv) < 3:
+        sys.stderr.write(__doc__)
+        return 2
+    build_dir = os.path.abspath(sys.argv[1])
+    label = sys.argv[2]
+    sizes = list(DEFAULT_SIZES)
+    reps = 3
+    results_path = DEFAULT_RESULTS
+    do_calibrate = "--calibrate" in sys.argv
+    args = sys.argv[3:]
+    for i, a in enumerate(args):
+        if a == "--sizes":
+            sizes = [int(x) for x in args[i + 1].split(",")]
+        elif a == "--reps":
+            reps = int(args[i + 1])
+        elif a == "--results":
+            results_path = args[i + 1]
+
+    corpus = open(CORPUS, encoding="utf-8").read()
+    logpath = os.path.join(BENCH_DIR, "server-%s-%d.log" % (label, int(time.time())))
+
+    p = None
+    try:
+        sys.stderr.write("[%s] starting llama-server from %s\n" % (label, build_dir))
+        t0 = time.time()
+        p = start_server(build_dir, logpath)
+        sys.stderr.write("[%s] healthy after %.1fs\n" % (label, time.time() - t0))
+
+        if do_calibrate:
+            calibrate(corpus, sizes)
+            return 0
+
+        stamp = time.strftime("%Y-%m-%dT%H:%M:%S")
+        out = open(results_path, "a")
+        for target in sizes:
+            if target not in SLICES:
+                sys.stderr.write("no calibrated slice for %d tokens, skipping\n" % target)
+                continue
+            nchars = SLICES[target]["chars"]
+            prompt = corpus[:nchars]
+            rates, mss, ns = [], [], set()
+            for rep in range(1, reps + 1):
+                r = post("/completion", {"prompt": prompt, "n_predict": 1,
+                                         "cache_prompt": False, "temperature": 0})
+                t = r["timings"]
+                rates.append(t["prompt_per_second"])
+                mss.append(t["prompt_ms"])
+                ns.add(t["prompt_n"])
+                line = ("RESULT corpus label=%s build=%s target=%d chars=%d "
+                        "prompt_n=%d rep=%d prompt_ms=%.2f tps=%.2f time=%s"
+                        % (label, build_dir, target, nchars, t["prompt_n"], rep,
+                           t["prompt_ms"], t["prompt_per_second"], stamp))
+                out.write(line + "\n")
+                out.flush()
+                sys.stderr.write(line + "\n")
+            if ns != {target}:
+                sys.stderr.write("WARNING: prompt_n %s != target %d; slice table "
+                                 "is stale, rerun with --calibrate\n" % (sorted(ns), target))
+            s = ("SUMMARY corpus label=%s target=%d prompt_n=%s median_tps=%.2f "
+                 "min_tps=%.2f max_tps=%.2f median_ms=%.1f time=%s"
+                 % (label, target, sorted(ns), median(rates), min(rates),
+                    max(rates), median(mss), stamp))
+            out.write(s + "\n")
+            out.flush()
+            sys.stderr.write(s + "\n")
+        out.close()
+    finally:
+        stop_server(p)
+        sys.stderr.write("[%s] server stopped\n" % label)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/rx580-bench/run.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+#
+# One-command entry point for the RX 580 prompt-processing benchmark.
+#
+# Runs from your workstation. Copies the pod-side harness into the pod and
+# execs it there over kubectl. All the real work happens on the pod; this is
+# a thin wrapper so there is exactly one command to remember.
+#
+# Examples:
+#   ./scripts/rx580-bench/run.sh --build /root/llama.cpp/build --label fork-before
+#   ./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean \
+#                               --build-b /root/arms/new-fork/build --label-b new-fork
+#   ./scripts/rx580-bench/run.sh --summarize
+#   ./scripts/rx580-bench/run.sh --install-only
+#
+# Every argument other than the wrapper-only flags below is passed straight
+# through to bench.sh on the pod. See bench.sh --help.
+#
+set -euo pipefail
+
+KUBECONFIG_PATH="${KUBECONFIG:-/home/user/Projects/klaster/talos/generated/kubeconfig}"
+NS="${RX580_NS:-llama}"
+DEPLOY="${RX580_DEPLOY:-deploy/supervisord}"
+POD_BENCH_DIR=/root/bench
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+export KUBECONFIG="$KUBECONFIG_PATH"
+
+kx() { kubectl -n "$NS" exec -i "$DEPLOY" -- sh -c "$1"; }
+
+install_harness() {
+  echo "[run] installing harness into $POD_BENCH_DIR"
+  kx "mkdir -p $POD_BENCH_DIR"
+  for f in bench.sh ppbench.py lbsweep.sh summarize.py; do
+    kx "cat > $POD_BENCH_DIR/$f" < "$HERE/$f"
+  done
+  kx "chmod +x $POD_BENCH_DIR/bench.sh $POD_BENCH_DIR/lbsweep.sh $POD_BENCH_DIR/ppbench.py $POD_BENCH_DIR/summarize.py"
+  # The corpus is large and rarely changes; only push it if missing.
+  if ! kx "test -s $POD_BENCH_DIR/pan-tadeusz.txt" 2>/dev/null; then
+    if [ -f "$HERE/pan-tadeusz.txt" ]; then
+      echo "[run] uploading corpus"
+      kx "cat > $POD_BENCH_DIR/pan-tadeusz.txt" < "$HERE/pan-tadeusz.txt"
+    else
+      echo "[run] ERROR: $POD_BENCH_DIR/pan-tadeusz.txt missing on the pod and" >&2
+      echo "      no local copy at $HERE/pan-tadeusz.txt to upload." >&2
+      exit 4
+    fi
+  fi
+}
+
+ARGS=()
+INSTALL_ONLY=0
+SUMMARIZE=0
+for a in "$@"; do
+  case "$a" in
+    --install-only) INSTALL_ONLY=1 ;;
+    --summarize)    SUMMARIZE=1 ;;
+    *) ARGS+=("$a") ;;
+  esac
+done
+
+install_harness
+
+if [ "$INSTALL_ONLY" -eq 1 ]; then
+  echo "[run] harness installed; not running anything"
+  exit 0
+fi
+
+if [ "$SUMMARIZE" -eq 1 ]; then
+  kx "python3 $POD_BENCH_DIR/summarize.py ${ARGS[*]:-}"
+  exit 0
+fi
+
+if [ ${#ARGS[@]} -eq 0 ]; then
+  kx "sh $POD_BENCH_DIR/bench.sh --help"
+  exit 2
+fi
+
+# A full 3-round A/B run takes hours; run detached on the pod and tail it, so a
+# dropped kubectl connection cannot orphan a half-finished run with llama-swap
+# still down. bench.sh restarts llama-swap from its own EXIT trap either way.
+STAMP="$(date -u +%Y%m%d-%H%M%S)"
+LOG="$POD_BENCH_DIR/run-$STAMP.log"
+echo "[run] starting detached run on the pod, log: $LOG"
+kx "cd $POD_BENCH_DIR && nohup sh $POD_BENCH_DIR/bench.sh ${ARGS[*]} > $LOG 2>&1 & echo started"
+
+echo "[run] tailing until ALL_RUNS_DONE (safe to Ctrl-C: the pod keeps running)"
+kx "i=0; while [ \$i -lt 100000 ]; do
+      if grep -qE 'ALL_RUNS_DONE|Traceback|error:' $LOG; then break; fi
+      sleep 15; i=\$((i+1));
+    done; cat $LOG"

scripts/rx580-bench/summarize.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""
+Summarize a results.txt produced by bench.sh into a per-label table.
+
+Usage: summarize.py [results.txt]
+
+Prints median and min-max of every metric for every label, and flags
+cross-label deltas against the ~5 percent noise floor of this machine.
+"""
+
+import re
+import sys
+from statistics import median
+
+NOISE_FLOOR_PCT = 5.0
+
+RE_CORPUS = re.compile(
+    r"^RESULT corpus label=(\S+) build=\S+ target=(\d+) chars=\d+ "
+    r"prompt_n=(\d+) rep=\d+ prompt_ms=([\d.]+) tps=([\d.]+)")
+RE_LB = re.compile(
+    r"^LBRESULT label=(\S+) build=\S+ test=(\S+) tps=([\d.]+)")
+
+
+def main():
+    path = sys.argv[1] if len(sys.argv) > 1 else "/root/bench/results.txt"
+    # data[metric][label] = list of tps
+    data = {}
+    order = []
+    labels = []
+    for line in open(path):
+        m = RE_CORPUS.match(line)
+        if m:
+            label, target, _pn, _ms, tps = m.groups()
+            metric = "pp%s-corpus" % target
+        else:
+            m = RE_LB.match(line)
+            if not m:
+                continue
+            label, test, tps = m.groups()
+            metric = "%s-llama-bench" % test
+        if metric not in data:
+            data[metric] = {}
+            order.append(metric)
+        data[metric].setdefault(label, []).append(float(tps))
+        if label not in labels:
+            labels.append(label)
+
+    if not data:
+        print("no parseable results in %s" % path)
+        return 1
+
+    def sort_key(m):
+        n = re.search(r"(\d+)", m)
+        return (0 if "corpus" in m else 1, int(n.group(1)) if n else 0)
+    order.sort(key=sort_key)
+
+    w = max(len(l) for l in labels) + 2
+    head = "metric".ljust(20) + "".join(l.ljust(max(w, 24)) for l in labels)
+    print("")
+    print("t/s, median (min-max), n samples")
+    print(head)
+    print("-" * len(head))
+    for metric in order:
+        row = metric.ljust(20)
+        for label in labels:
+            vals = data[metric].get(label)
+            if not vals:
+                row += "-".ljust(max(w, 24))
+            else:
+                cell = "%.1f (%.1f-%.1f) n=%d" % (
+                    median(vals), min(vals), max(vals), len(vals))
+                row += cell.ljust(max(w, 24))
+        print(row)
+
+    if len(labels) >= 2:
+        base = labels[0]
+        print("")
+        print("deltas vs %s (noise floor %.0f%%, anything under it is UNPROVEN)"
+              % (base, NOISE_FLOOR_PCT))
+        for other in labels[1:]:
+            print("  %s vs %s:" % (other, base))
+            for metric in order:
+                a = data[metric].get(base)
+                b = data[metric].get(other)
+                if not a or not b:
+                    continue
+                ma, mb = median(a), median(b)
+                pct = (mb - ma) / ma * 100.0
+                verdict = "SIGNIFICANT" if abs(pct) >= NOISE_FLOOR_PCT else "unproven"
+                print("    %-20s %+6.1f%%  %s" % (metric, pct, verdict))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

src/llama-context.cpp

@@ -755,11 +755,20 @@ void llama_context::release_device(bool evict_kv) {
     if (evict_kv && memory && !kv_device_evicted) {
         memory->release_device_buffers();
         kv_device_evicted = true;
+        // also free the scheduler and its worst-case compute buffer (hundreds of MiB) so a cold
+        // model holds essentially no VRAM. Rebuilt lazily by sched_reserve() on restore.
+        sched.reset();
+        sched_need_reserve = true;
+        // finally, free each backend's own compute-scratch (Vulkan prealloc/staging buffers, which
+        // are owned by the backend and survive sched.reset()). Reallocated lazily on next compute.
+        for (auto & backend : backends) {
+            ggml_backend_free_scratch(backend.get());
+        }
     }
 }
 
 void llama_context::restore_device() {
-    if (model.weights_resident() && !kv_device_evicted) {
+    if (model.weights_resident() && !kv_device_evicted && sched) {
         return;
     }
     model.restore_device_weights();
@@ -767,6 +776,10 @@ void llama_context::restore_device() {
         memory->restore_device_buffers();
         kv_device_evicted = false;
     }
+    // rebuild the scheduler + compute buffer if they were freed on release (evict_kv mode)
+    if (!sched) {
+        sched_reserve();
+    }
     // make sure all weight/KV uploads have completed before any compute reads them
     if (sched) {
         ggml_backend_sched_synchronize(sched.get());
@@ -3400,17 +3413,21 @@ llama_memory_breakdown llama_context::memory_breakdown() const {
             ret[buft].context += size;
         }
     }
-    if (model.hparams.no_alloc) {
-        for (size_t i = 0; i < backends.size(); ++i) {
-            ggml_backend_t             backend = backends[i].get();
-            ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
-            ret[buft].compute += backend_buf_exp_size[i];
-        }
-    } else {
-        for (const auto & backend_ptr : backends) {
-            ggml_backend_t             backend = backend_ptr.get();
-            ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
-            ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
+    // the scheduler (and its compute buffers) may have been freed while the model is cold
+    // (on-demand VRAM eviction, see release_device); it contributes no compute VRAM then.
+    if (sched) {
+        if (model.hparams.no_alloc) {
+            for (size_t i = 0; i < backends.size(); ++i) {
+                ggml_backend_t             backend = backends[i].get();
+                ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
+                ret[buft].compute += backend_buf_exp_size[i];
+            }
+        } else {
+            for (const auto & backend_ptr : backends) {
+                ggml_backend_t             backend = backend_ptr.get();
+                ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
+                ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
+            }
         }
     }
     return ret;

src/llama-memory-hybrid.cpp

@@ -202,7 +202,7 @@ void llama_memory_hybrid::state_read(llama_io_read_i & io, llama_seq_id seq_id,
 }
 
 void llama_memory_hybrid::release_device_buffers() {
-    // evict the attention KV (grows with context); the recurrent state uses the no-op default
+    // evict both the attention KV (grows with context) and the recurrent/SSM state
     mem_attn->release_device_buffers();
     mem_recr->release_device_buffers();
 }

src/llama-memory-recurrent.cpp

@@ -151,7 +151,9 @@ void llama_memory_recurrent::clear(bool data) {
 
     if (data) {
         for (auto & [_, buf] : ctxs_bufs) {
-            ggml_backend_buffer_clear(buf.get(), 0);
+            if (buf) { // may be null if evicted for on-demand VRAM sharing
+                ggml_backend_buffer_clear(buf.get(), 0);
+            }
         }
     }
 
@@ -420,6 +422,7 @@ void llama_memory_recurrent::set_rs_idx(llama_seq_id seq_id, uint32_t idx) {
 std::map<ggml_backend_buffer_type_t, size_t> llama_memory_recurrent::memory_breakdown() const {
     std::map<ggml_backend_buffer_type_t, size_t> ret;
     for (const auto & [_, buf] : ctxs_bufs) {
+        if (!buf) { continue; } // may be null if evicted for on-demand VRAM sharing
         ret[ggml_backend_buffer_get_type(buf.get())] += ggml_backend_buffer_get_size(buf.get());
     }
     return ret;
@@ -721,12 +724,87 @@ bool llama_memory_recurrent::get_can_shift() const {
 size_t llama_memory_recurrent::total_size() const {
     size_t size = 0;
     for (const auto & [_, buf] : ctxs_bufs) {
-        size += ggml_backend_buffer_get_size(buf.get());
+        if (buf) { // may be null if evicted for on-demand VRAM sharing
+            size += ggml_backend_buffer_get_size(buf.get());
+        }
     }
 
     return size;
 }
 
+void llama_memory_recurrent::release_device_buffers() {
+    // Same mechanism as llama_kv_cache: the recurrent (SSM/conv) state is read-write, so its host
+    // shadow is (re)captured on every release. The caller must have synchronized the backend.
+    if (dev_released) {
+        return;
+    }
+    dev_shadows.assign(ctxs_bufs.size(), device_buffer_shadow{});
+    size_t freed = 0;
+    for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
+        ggml_context * ctx = ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = ctxs_bufs[i].second.get();
+        if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
+            continue;
+        }
+        auto & sh = dev_shadows[i];
+        sh.releasable = true;
+        sh.buft = ggml_backend_buffer_get_type(buf);
+
+        size_t total = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src == nullptr) { total += ggml_nbytes(t); }
+        }
+        sh.data.resize(total);
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_get(t, sh.data.data() + off, 0, n);
+            off += n;
+        }
+
+        freed += ggml_backend_buffer_get_size(buf);
+        ctxs_bufs[i].second.reset();
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            t->buffer = nullptr;
+            t->data   = nullptr;
+        }
+    }
+    dev_released = true;
+    if (freed > 0) {
+        LLAMA_LOG_INFO("%s: released %.2f MiB of recurrent state from device\n", __func__, freed / 1024.0 / 1024.0);
+    }
+}
+
+bool llama_memory_recurrent::restore_device_buffers() {
+    if (!dev_released) {
+        return true;
+    }
+    for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
+        auto & sh = dev_shadows[i];
+        if (!sh.releasable) {
+            continue;
+        }
+        ggml_context * ctx = ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, sh.buft);
+        if (buf == nullptr) {
+            LLAMA_LOG_ERROR("%s: failed to reallocate recurrent device buffer (out of VRAM?)\n", __func__);
+            return false;
+        }
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_set(t, sh.data.data() + off, 0, n);
+            off += n;
+        }
+        ctxs_bufs[i].second.reset(buf);
+    }
+    dev_released = false;
+    dev_shadows.clear();
+    return true;
+}
+
 size_t llama_memory_recurrent::size_r_bytes() const {
     size_t size_r_bytes = 0;
 

src/llama-memory-recurrent.h

@@ -66,6 +66,10 @@ public:
     void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
     void state_read (llama_io_read_i  & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
 
+    // on-demand device (VRAM) residency (see llama_memory_i)
+    void release_device_buffers() override;
+    bool restore_device_buffers() override;
+
     uint32_t head = 0; // the location where the batch will be placed in the cache (see find_slot())
     uint32_t size = 0; // total number of cells, shared across all sequences
     uint32_t used = 0; // used cells (i.e. at least one seq_id)
@@ -123,6 +127,16 @@ private:
     // ggml contexts for the KV cache along with the allocated backend buffers:
     std::vector<std::pair<ggml_context_ptr, ggml_backend_buffer_ptr>> ctxs_bufs;
 
+    // on-demand device eviction (see release_device_buffers): host shadow of each device buffer's
+    // live contents (recaptured on every release since the recurrent state is read-write)
+    struct device_buffer_shadow {
+        ggml_backend_buffer_type_t buft = nullptr;
+        bool releasable = false;
+        std::vector<uint8_t> data;
+    };
+    std::vector<device_buffer_shadow> dev_shadows; // parallel to ctxs_bufs
+    bool dev_released = false;
+
     size_t total_size() const;
 
     size_t size_r_bytes() const;

tools/mtmd/clip.cpp

@@ -159,6 +159,13 @@ struct clip_ctx {
     ggml_backend_t backend_cpu = nullptr;
     ggml_backend_buffer_ptr buf;
 
+    // on-demand device (VRAM) residency: the vision/audio encoder weights are read-only, so a host
+    // shadow is captured once and the device buffer can be freed while the model is cold, then
+    // rebuilt on wake (mirrors llama_model::release_device_weights). See clip_release_device().
+    ggml_backend_buffer_type_t dev_buft = nullptr;
+    std::vector<uint8_t>       dev_shadow;
+    bool                       dev_released = false;
+
 
     int max_nodes = 8192;
     ggml_backend_sched_ptr sched;
@@ -4026,6 +4033,74 @@ void clip_free(clip_ctx * ctx) {
     delete ctx;
 }
 
+void clip_release_device(struct clip_ctx * ctx) {
+    if (ctx == nullptr || ctx->dev_released || ctx->no_alloc) {
+        return;
+    }
+    ggml_backend_buffer_t buf = ctx->buf.get();
+    if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
+        return; // CPU-backed encoder: nothing in VRAM to free
+    }
+    // ensure no encode is in flight before freeing the weights
+    if (ctx->sched) {
+        ggml_backend_sched_synchronize(ctx->sched.get());
+    }
+    ctx->dev_buft = ggml_backend_buffer_get_type(buf);
+
+    ggml_context * cd = ctx->ctx_data.get();
+    // capture the host shadow once (weights are read-only): compact, stable iteration order,
+    // skipping view tensors (which alias a base and are restored implicitly)
+    if (ctx->dev_shadow.empty()) {
+        size_t total = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+            if (t->view_src == nullptr) { total += ggml_nbytes(t); }
+        }
+        ctx->dev_shadow.resize(total);
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_get(t, ctx->dev_shadow.data() + off, 0, n);
+            off += n;
+        }
+    }
+
+    // free the device buffer and clear the now-dangling tensor pointers so restore reallocates cleanly
+    ctx->buf.reset();
+    for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+        t->buffer = nullptr;
+        t->data   = nullptr;
+    }
+    ctx->dev_released = true;
+}
+
+bool clip_restore_device(struct clip_ctx * ctx) {
+    if (ctx == nullptr || !ctx->dev_released) {
+        return true;
+    }
+    ggml_context * cd = ctx->ctx_data.get();
+    ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(cd, ctx->dev_buft);
+    if (buf == nullptr) {
+        LOG_ERR("%s: failed to reallocate encoder device buffer (out of VRAM?)\n", __func__);
+        return false;
+    }
+    ggml_backend_buffer_set_usage(buf, GGML_BACKEND_BUFFER_USAGE_WEIGHTS);
+    size_t off = 0;
+    for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+        if (t->view_src != nullptr) { continue; }
+        const size_t n = ggml_nbytes(t);
+        ggml_backend_tensor_set(t, ctx->dev_shadow.data() + off, 0, n);
+        off += n;
+    }
+    ctx->buf.reset(buf);
+    ctx->dev_released = false;
+    return true;
+}
+
+bool clip_weights_resident(const struct clip_ctx * ctx) {
+    return ctx == nullptr || !ctx->dev_released;
+}
+
 const char * clip_patch_merge_type(const struct clip_ctx * ctx) {
     return ctx->model.hparams.mm_patch_merge_type == PATCH_MERGE_SPATIAL_UNPAD ? "spatial_unpad" : "flat";
 }

tools/mtmd/clip.h

@@ -70,6 +70,12 @@ struct clip_init_result clip_init(const char * fname, struct clip_context_params
 
 void clip_free(struct clip_ctx * ctx);
 
+// on-demand device (VRAM) residency: free/rebuild the encoder weight buffer to shrink an idle
+// (cold) multimodal model's VRAM footprint. No-op for a CPU-backed encoder. See clip.cpp.
+void clip_release_device(struct clip_ctx * ctx);
+bool clip_restore_device(struct clip_ctx * ctx);
+bool clip_weights_resident(const struct clip_ctx * ctx);
+
 // TODO: should be enum, not string
 const char * clip_patch_merge_type(const struct clip_ctx * ctx);
 

tools/mtmd/mtmd.cpp

@@ -1116,6 +1116,24 @@ std::vector<std::vector<const mtmd_bitmap *>> mtmd_group_mergeable_bitmaps(std::
     return output;
 }
 
+void mtmd_release_device(mtmd_context * ctx) {
+    if (ctx == nullptr) {
+        return;
+    }
+    if (ctx->ctx_v) { clip_release_device(ctx->ctx_v); }
+    if (ctx->ctx_a) { clip_release_device(ctx->ctx_a); }
+}
+
+bool mtmd_restore_device(mtmd_context * ctx) {
+    if (ctx == nullptr) {
+        return true;
+    }
+    bool ok = true;
+    if (ctx->ctx_v) { ok = clip_restore_device(ctx->ctx_v) && ok; }
+    if (ctx->ctx_a) { ok = clip_restore_device(ctx->ctx_a) && ok; }
+    return ok;
+}
+
 struct mtmd_tokenizer {
     const mtmd_context * ctx;
 

tools/mtmd/mtmd.h

@@ -136,6 +136,12 @@ MTMD_API mtmd_context * mtmd_init_from_file(const char * mmproj_fname,
 
 MTMD_API void mtmd_free(mtmd_context * ctx);
 
+// on-demand device (VRAM) residency: free / rebuild the vision+audio encoder weight buffers so an
+// idle (cold) multimodal model releases its encoder VRAM and reclaims it on wake. No-op for a
+// CPU-backed encoder. Restore returns false if reallocation failed (out of VRAM).
+MTMD_API void mtmd_release_device(mtmd_context * ctx);
+MTMD_API bool mtmd_restore_device(mtmd_context * ctx);
+
 // whether we need to set non-causal mask before llama_decode
 // if chunk is nullptr, we assume the default case where chunk is an image chunk
 MTMD_API bool mtmd_decode_use_non_causal(const mtmd_context * ctx, const mtmd_input_chunk * chunk);

tools/server/server-context.cpp

@@ -829,7 +829,36 @@ static int process_mtmd_chunk(const server_slot & slot, mtmd::batch_ptr & mbatch
     // TODO @ngxson : move this log line to debug when it become more stable
     SLT_TRC(slot, "encoding mtmd batch from idx = %zu, n_chunks = %d\n", idx, n_added);
 
+    // Bring the vision/audio encoder into VRAM just for this encode. In on-demand mode the
+    // encoder is normally kept in RAM (its VRAM freed for KV / expert cache). If it does not
+    // fit, evict the LLM backbone weights first: they are NOT needed while the encoder runs (the
+    // decode that uses them happens afterwards) and their host shadow is read-only, so this is a
+    // cheap free (no D2H) and leaves the KV cache / prompt cache untouched.
+    static const bool mmproj_ondemand = getenv("LLAMA_MMPROJ_ONDEMAND") != nullptr;
+    bool weights_evicted = false;
+    if (mctx && !mtmd_restore_device(mctx)) {
+        if (mmproj_ondemand) {
+            llama_context_release_device(slot.ctx_tgt, /* evict_kv = */ false); // free backbone, keep KV
+            weights_evicted = true;
+        }
+        if (!mtmd_restore_device(mctx)) {
+            if (weights_evicted) { llama_context_restore_device(slot.ctx_tgt); }
+            SLT_ERR(slot, "%s", "failed to bring the multimodal encoder into VRAM for encoding\n");
+            return -1;
+        }
+    }
+
     res = mtmd_batch_encode(mbatch.get());
+
+    // release the encoder VRAM again (on-demand), then restore the backbone weights for decode.
+    // Order matters: free the encoder BEFORE re-uploading the weights so the peak stays within VRAM.
+    if (mctx && mmproj_ondemand) {
+        mtmd_release_device(mctx);
+    }
+    if (weights_evicted) {
+        llama_context_restore_device(slot.ctx_tgt);
+    }
+
     if (res != 0) {
         SLT_ERR(slot, "failed to encode mtmd batch for chunk idx = %zu, res = %d\n", idx, res);
         return -1;
@@ -1055,6 +1084,11 @@ private:
         if (ctx_dft != nullptr) {
             llama_context_restore_device(ctx_dft);
         }
+        // NOTE: the multimodal (vision/audio) encoder is intentionally NOT restored here. It is
+        // brought into VRAM just-in-time before an image/audio encode (process_mtmd_chunk) and, in
+        // on-demand mode, released again right after - so a warm text-only model holds no encoder
+        // VRAM (that space is free for KV / expert cache). A cold->warm wake for a *text* request
+        // therefore leaves the encoder in RAM; an image request restores it at encode time.
         vram_cold = false;
     }
 
@@ -1079,6 +1113,11 @@ private:
         if (ctx_dft != nullptr) {
             llama_context_release_device(ctx_dft, vram_evict_kv);
         }
+        // also release the multimodal (vision/audio) encoder weights (dead weight while cold);
+        // its read-only host shadow is captured once and rebuilt on wake by vram_ensure_warm()
+        if (mctx != nullptr) {
+            mtmd_release_device(mctx);
+        }
 #if !defined(_WIN32)
         if (vram_flock) {
             flock(vram_lock_fd, LOCK_UN);
@@ -1164,8 +1203,12 @@ private:
                 SRV_INF("%s", "server entering sleeping state (VRAM-only: releasing device weights, keeping KV cache)\n");
                 vram_go_cold();
             } else {
-                SRV_INF("%s", "server exiting sleeping state (VRAM-only: weights restored on next decode)\n");
-                // token is acquired and weights restored by vram_ensure_warm() before decode
+                SRV_INF("%s", "server exiting sleeping state (VRAM-only: restoring device weights/KV)\n");
+                // Restore NOW, on wake, before update_slots runs. update_slots touches the KV cache
+                // (e.g. SWA checkpoint creation reads it via ggml_backend_tensor_get) before the
+                // decode-time vram_ensure_warm(), so with KV eviction the KV must already be resident
+                // here or those reads hit a freed (null) buffer.
+                vram_ensure_warm();
             }
             sleeping = new_state;
             return;
@@ -1394,6 +1437,13 @@ private:
             init_opt.video_params.ffmpeg_bin_dir = params_base.video_ffmpeg_bin_dir.empty()
                                 ? nullptr : params_base.video_ffmpeg_bin_dir.c_str();
 
+            // on-demand encoder: keep the vision/audio encoder weights in RAM (out of VRAM) until an
+            // image/audio actually needs encoding, freeing that VRAM for KV / expert cache on the
+            // common text-only path. process_mtmd_chunk() brings the encoder in just-in-time.
+            if (getenv("LLAMA_MMPROJ_ONDEMAND") != nullptr) {
+                mtmd_release_device(mctx);
+            }
+
             if (params_base.ctx_shift) {
                 params_base.ctx_shift = false;
                 SRV_WRN("%s\n", "ctx_shift is not supported by multimodal, it will be disabled");