llama: skip the compute-buffer size check when the context is cold

release_device(evict_kv) drops the scheduler, so a context evicted by the VRAM
arbiter and then destroyed reaches the destructor with a null sched. The upstream
"compute buffer size matches expectation" loop calls
ggml_backend_sched_get_buffer_size() unconditionally, whose GGML_ASSERT(sched)
then aborts: the server exits 134 instead of 0 on every shutdown taken while
cold, which under llama-swap makes an ordinary stop look like a crashed child and
leaves a ggml backtrace in the log each time. It also skipped the rest of the
destructor, so the cold teardown path had never actually run to completion.

Guard at the call site rather than relaxing the assert - the assert is right, and
every other caller reserves the scheduler first. Same shape as the null guards in
synchronize(), memory_breakdown() and the released-buffer iteration.

The loop is a diagnostic size comparison, already reported by sched_reserve() at
load, so skipping it when cold loses nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ

README.md

@@ -17,6 +17,82 @@
 
 </div>
 
+## This fork - Polaris / GCN tuning for large MoE models
+
+Changes and measurements for running large MoE models with their experts offloaded to system RAM
+(`--n-cpu-moe`) on an old GCN card. The two code changes below are auto-on, need no flag, and are
+**token-identical** to mainline. Everything else here is tuning guidance.
+
+Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16, no fp16, no matrix cores) with
+**Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`:
+
+- **Flash-attention `mask_opt` is enabled for GCN large head sizes (this fork's own change).**
+  Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips
+  fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on
+  a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B
+  (head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical.
+- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp).
+- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short
+  prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that
+  VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`).
+
+**Recommended RX 580 / Polaris serving command** (per model):
+
+```bash
+llama-server -hf <repo>:<quant> -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1
+```
+
+Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high
+for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so
+`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path.
+
+### Very large MoE (experts bigger than the GTT limit)
+
+Measured on the same RX 580 with **Laguna-S-2.1 118B IQ2_M** (48 layers, 256 experts, 10 used,
+experts 30.7 GiB of a 34.7 GiB file, all host-resident at `--n-cpu-moe 48`). The advice above changes
+in this regime:
+
+- **`--no-mmap` stops being an option.** Its pinned host buffer is charged against the amdgpu GTT
+  limit (~31.4 GiB here, about half of system RAM). At 30.7 GiB of experts the model no longer
+  loads, and the allocation spike can OOM the box. Use mmap and accept the staging copy.
+- **The routing-ids readback is pure overhead at prefill batch sizes** and this fork now skips it.
+  To decide which experts to upload, the scheduler read the ids back from the device that had just
+  produced them, forcing a full pipeline flush once per MoE layer per eval. With 2048 tokens x 10
+  experts over 256 experts every expert is used anyway. Skipping it is exact - `mul_mat_id` only
+  reads the rows the ids point at. Auto-on above `4 * n_expert` ids; decode keeps the old path.
+  **pp2048 +4.7% @ 32k depth, +0.9% @ 16k, tg +3%.**
+- **Where the time actually goes** (`GGML_VK_PERF_LOGGER=1`, depth 0, 22.65 s per 2048-token eval,
+  17.18 s of it GPU-busy so ~24% is H2D stall): expert `MUL_MAT_ID` **52%**, attention projections
+  **24%**, `FLASH_ATTN_EXT` **16%**, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s
+  while dense `MUL_MAT` q5_K/q6_K in the same graph reaches 3096-3691 - **the single largest
+  remaining opportunity on this hardware is closing that gap**, not the transfer path.
+- **Interleaved SWA keeps its own small KV cache**, so a sliding-window layer costs the same at any
+  depth (`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`). On this model 36 of 48 layers are O(1) in
+  depth and the entire high-context slowdown comes from the 12 full-attention layers.
+- **`--parallel 1`** is worth setting for a solo large model: the server otherwise auto-selects 4
+  slots, and the SWA cache is sized `n_swa * n_seq_max + n_ubatch`, so 4 slots cost 4096 cells
+  instead of 2560. Measured **223 MiB of VRAM freed** at 64k context.
+
+Dead ends measured on this hardware, recorded so they are not retried:
+
+| Change | Result |
+| --- | ---: |
+| flash-attn `shmem_staging` enabled for GCN | **-6.7% @ 16k, -7.4% @ 32k** |
+| `-b 4096 -ub 4096` (to amortize the fixed per-eval expert upload) | flat (-1%) |
+| `--n-cpu-moe` 44 instead of 48 | +1.5%, but does not fit at 64k ctx |
+| `mask_opt` gate relaxed below head_dim 256 | -18.5% @ 16k |
+
+`shmem_staging` looks like a certain win (without it each rowgroup re-reads the whole K/V block
+through a 16 KiB L1) but the `kvsh` stride of `D/4+1` dwords is 4 mod 32, which costs an 8-way LDS
+bank conflict on wave64 - that `+1` padding is tuned for warp32. `-ub 4096` fails because halving the
+number of expert uploads is exactly cancelled by intra-ubatch attention growing quadratically.
+
+**Serving note that outweighs all of the above.** With a model this large, anything that restarts the
+process is far more expensive than any kernel win: the server's prompt cache is RAM-only with no disk
+backing, so a restart forces a full re-prefill of the conversation. If a model swapper can evict this
+model to run a small helper model (chat-title generation and the like), fix that first - keeping the
+process alive across a swap took a repeat turn from a 21,960 ms prefill down to 225 ms.
+
 ## Quick start
 
 A few options to get `llama.cpp` installed on your machine:

ggml/include/ggml-backend.h

@@ -104,6 +104,11 @@ extern "C" {
     GGML_API enum ggml_status ggml_backend_graph_compute      (ggml_backend_t backend, struct ggml_cgraph * cgraph);
     GGML_API enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph);
 
+    // Free transient/scratch device memory the backend holds outside of any allocated buffer
+    // (compute preallocations, staging buffers). No-op if the backend does not implement it.
+    // The backend remains usable; scratch is reallocated lazily on the next compute.
+    GGML_API void ggml_backend_free_scratch(ggml_backend_t backend);
+
     // NOTE: will be removed, use device version instead
     GGML_API bool ggml_backend_supports_op(ggml_backend_t backend, const struct ggml_tensor * op);
     GGML_API bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft);

ggml/src/ggml-backend-impl.h

@@ -153,6 +153,11 @@ extern "C" {
 
         // (optional) sort/optimize the nodes in the graph
         void                      (*graph_optimize)    (ggml_backend_t backend, struct ggml_cgraph * cgraph, struct ggml_backend_graph_optimize_params * params);
+
+        // (optional) free transient/scratch device memory the backend holds outside of any buffer
+        // (e.g. compute preallocations and staging buffers). The backend stays usable; the scratch
+        // is reallocated lazily on the next compute. Used to shrink an idle model's device footprint.
+        void                      (*free_scratch)      (ggml_backend_t backend);
     };
 
     struct ggml_backend {

ggml/src/ggml-backend.cpp

@@ -431,6 +431,15 @@ void ggml_backend_synchronize(ggml_backend_t backend) {
     backend->iface.synchronize(backend);
 }
 
+void ggml_backend_free_scratch(ggml_backend_t backend) {
+    GGML_ASSERT(backend);
+    if (backend->iface.free_scratch == NULL) {
+        return;
+    }
+
+    backend->iface.free_scratch(backend);
+}
+
 ggml_backend_graph_plan_t ggml_backend_graph_plan_create(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
     GGML_ASSERT(backend);
     GGML_ASSERT(backend->iface.graph_plan_create != NULL);

ggml/src/ggml-vulkan/ggml-vulkan.cpp

@@ -13579,6 +13579,8 @@ static void ggml_vk_concat(ggml_backend_vk_context * ctx, vk_context& subctx, co
         const uint32_t ts = ggml_type_size(s->type);
         return s->nb[1] == ts                                              // dim1 innermost
             && s->nb[0] == (size_t) s->ne[1] * ts                          // consistent 2D transpose
+            && s->ne[0] >= 32                                              // at least one full transpose tile (TILE_DIM); at 1 token the
+                                                                           // tiled path is pure overhead, the generic copy is faster
             && s->ne[2] == 1 && s->ne[3] == 1;
     };
     const uint32_t dst_ts = ggml_type_size(dst->type);
@@ -17184,6 +17186,35 @@ static const char * ggml_backend_vk_name(ggml_backend_t backend) {
     return ctx->name.c_str();
 }
 
+// Free the compute-scratch device buffers (prealloc_* and the transfer staging buffer) without
+// tearing down the backend (pipelines, command pools, fences and device stay alive). These buffers
+// are reallocated lazily by ggml_vk_preallocate_buffers() on the next compute, so this just shrinks
+// an idle model's device footprint. Mirrors the buffer-freeing subset of ggml_vk_cleanup().
+static void ggml_backend_vk_free_scratch(ggml_backend_t backend) {
+    ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
+    VK_LOG_DEBUG("ggml_backend_vk_free_scratch(" << ctx->name << ")");
+
+    // discard any unsubmitted command buffer and wait for in-flight work before freeing
+    ctx->compute_ctx.reset();
+    ggml_vk_synchronize(ctx);
+
+    ggml_vk_destroy_buffer(ctx->prealloc_x);
+    ggml_vk_destroy_buffer(ctx->prealloc_y);
+    ggml_vk_destroy_buffer(ctx->prealloc_split_k);
+    ggml_vk_destroy_buffer(ctx->prealloc_add_rms_partials);
+    ggml_vk_destroy_buffer(ctx->sync_staging);
+
+    ctx->prealloc_y_last_pipeline_used = nullptr;
+    ctx->prealloc_y_last_tensor_used = nullptr;
+    ctx->prealloc_y_last_k_padded = false;
+
+    ctx->prealloc_size_x = 0;
+    ctx->prealloc_size_y = 0;
+    ctx->prealloc_size_split_k = 0;
+    ctx->prealloc_size_add_rms_partials = 0;
+    ctx->prealloc_size_add_rms_partials_offset = 0;
+}
+
 static void ggml_backend_vk_free(ggml_backend_t backend) {
     ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
     VK_LOG_DEBUG("ggml_backend_vk_free(" << ctx->name << ")");
@@ -19122,6 +19153,7 @@ static ggml_backend_i ggml_backend_vk_interface = {
     /* .event_record            = */ ggml_backend_vk_event_record,
     /* .event_wait              = */ ggml_backend_vk_event_wait,
     /* .graph_optimize          = */ ggml_vk_graph_optimize,
+    /* .free_scratch            = */ ggml_backend_vk_free_scratch,
 };
 
 static ggml_guid_t ggml_backend_vk_guid() {

scripts/rx580-bench/README.md

@@ -0,0 +1,200 @@
+# rx580-bench
+
+Prompt-processing benchmark harness for the RX 580 (Polaris) serving pod.
+
+It exists so that no future session has to rediscover how to measure this
+machine. It encodes the production config, fixed corpus slices, the repeat
+and median discipline, and the noise floor.
+
+## What it measures
+
+Two things, deliberately:
+
+1. **Corpus prompt processing** (the number we actually care about). Real
+   Polish prose from `pan-tadeusz.txt` pushed through `llama-server`'s
+   `/completion` endpoint with `n_predict: 1`, `cache_prompt: false`. This
+   mimics agentic tool-result parsing: a big blob of real text arriving cold.
+   Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k
+   and 32k tokens.
+2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random
+   tokens, `pp2048`, `pp8192`, `tg32`. This is what prior work on this box
+   recorded, so it is the continuity metric.
+
+They do not measure the same thing and they do not agree. `llama-bench`
+pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
+growing KV cache, so they fall off with prompt length. Both are useful.
+
+## How to run it
+
+One command from the workstation:
+
+```
+./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean
+```
+
+Interleaved A/B, which is the only valid way to compare two builds:
+
+```
+./scripts/rx580-bench/run.sh \
+  --build   /root/arms/new-clean/build --label   new-clean \
+  --build-b /root/arms/new-fork/build  --label-b new-fork \
+  --rounds 3
+```
+
+Summarize whatever has accumulated:
+
+```
+./scripts/rx580-bench/run.sh --summarize
+```
+
+`run.sh` copies the harness onto the pod and execs it there. The wrapper is
+thin on purpose: all the methodology lives in the pod-side scripts, so it can
+also be driven directly on the pod if kubectl is inconvenient.
+
+The run is detached on the pod and then tailed, so Ctrl-C on the wrapper does
+not orphan a half-finished run. `bench.sh` restarts `llama-swap` from an EXIT
+trap on success, failure and interrupt alike.
+
+### Pod access
+
+```
+export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
+kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
+```
+
+The pod has `curl` and `python3` but **no `jq`**; the harness uses python3 for
+all JSON.
+
+## Files
+
+| file | side | what |
+|---|---|---|
+| `run.sh` | workstation | one-command wrapper; installs and execs the rest |
+| `bench.sh` | pod | orchestrator: stops llama-swap, waits for idle GPU, runs both harnesses, restarts llama-swap via trap |
+| `ppbench.py` | pod | the corpus prompt-processing harness |
+| `lbsweep.sh` | pod | the llama-bench sweep |
+| `summarize.py` | pod | turns `results.txt` into a median/min-max table with delta significance |
+
+Results append to `/root/bench/results.txt` as parseable `RESULT` /
+`SUMMARY` / `LBRESULT` lines. Server logs land in `/root/bench/server-*.log`.
+
+## The config it assumes
+
+Production serving config, matching `/root/config.yaml`:
+
+```
+-t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --no-mmap
+--ctx-size 40960 --no-warmup
+```
+
+Model:
+
+```
+/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
+```
+
+Qwen3.6-35B-A3B, 20.8 GiB, 40 layers, 256 experts of which 8 are used.
+Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16,
+12 CPU cores, 62 GB RAM.
+
+The `llama-bench` equivalent of that config is:
+
+```
+-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -mmp 0 -p 2048,8192 -n 32 -r 3
+```
+
+## The corpus slices are constants, not recalibrated
+
+`pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this
+model's tokenizer it runs **2.6702 chars/token**, and `/completion` reports
+`prompt_n` identical to `/tokenize` (BOS offset 0).
+
+The slices are `corpus[0:NCHARS]` and are baked into `ppbench.py`:
+
+| target tokens | chars | utf-8 bytes |
+|---|---|---|
+| 4096 | 10776 | 11608 |
+| 16384 | 43858 | 47175 |
+| 32768 | 87165 | 93843 |
+
+These hit the target `prompt_n` exactly. Every build is therefore measured on
+byte-identical input. `ppbench.py` warns loudly if the reported `prompt_n`
+ever stops matching the target, which is the signal that the table is stale.
+Only recalibrate (`ppbench.py <build> <label> --calibrate`) if the model file
+or the corpus changes.
+
+## Reading the numbers
+
+* `prompt_n` is the token count actually prefilled. It must equal the target.
+* `prompt_ms` is wall-clock prefill time.
+* `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better.
+* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0.
+* `tg32` is token generation at DEFAULT depth (n_ctx 32). **It is NOT a valid
+  decode metric on this box.** At that size decode is launch/barrier bound, not
+  compute bound, and measures a regime serving never reaches: it read -36%
+  between two upstream commits that are at parity in real use (the cause was
+  the view-alias fix in ggml_vk_graph_optimize). For decode use **tg256 with an
+  explicit depth**, e.g. `-p 0 -n 256 -d 2048`. tg is also NOT monotonic in
+  n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 -
+  so only ever compare same-depth cells.
+
+## Noise floor: deltas under about 5 percent are unproven
+
+This is the single most important thing in this directory.
+
+`llama-bench`'s within-run error bars understate **cross-invocation** variance
+by roughly **14x**. The same config measured `276.7 +/- 0.9` in one invocation
+and `296.4 +/- 0.6` in another, 7 percent apart, with error bars that claimed
+0.3 percent precision.
+
+Therefore:
+
+* Any comparison you want to draw a conclusion from must live **inside a
+  single `llama-bench` invocation**, using comma-separated sweeps.
+* Two different *builds* cannot share an invocation, so run them
+  **interleaved A/B/A/B/A/B across at least 3 rounds** and compare medians.
+  That is what `--build-b` does.
+* State explicitly that any cross-build delta under about 5 percent is
+  unproven. `summarize.py` labels them `unproven` for you.
+
+## Traps
+
+* **`-tb` is not a `llama-bench` flag.** It is a `llama-server` flag. Passing
+  it makes `llama-bench` print usage and exit silently, mid-sweep, which looks
+  exactly like a run that produced nothing. Always check the row count.
+* **A ubatch larger than the prompt never fills.** With `-ub 2048`, prompt
+  lengths must be multiples of 2048 or the sweep measures nothing meaningful.
+* **Nothing else may touch the GPU.** `bench.sh` stops `llama-swap` and
+  refuses to start if any `llama-*` process is still alive.
+* **Never leave the pod not serving.** `bench.sh` restarts `llama-swap` from
+  an EXIT trap. If you bypass the script, restart it by hand:
+  `supervisorctl -c /root/supervisord.conf start llama-swap`.
+* **Do not run `llama-cli` detached with closed stdin** on this box; it loops
+  forever. Use the server harness or `llama-bench -d` for depth timing.
+* `--no-mmap` is deprecated upstream in favour of `--load-mode` / `-lm none`,
+  but still accepted and still what production passes.
+
+## Known-good reference values
+
+If a fresh run disagrees with these by much more than the noise floor,
+something is wrong (something else on the GPU, a thermal problem, a bad
+build) before you believe you found a speedup.
+
+Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config, current upstream
+(`f3f1a8f27`) and the deployed fork, measured 2026-09-08/09:
+
+| metric | clean upstream | fork |
+|---|---|---|
+| corpus pp @ 4k | about 240 t/s | about 255 t/s |
+| corpus pp @ 16k | about 206 t/s | about 225 t/s |
+| corpus pp @ 32k | about 171 t/s | about 195 t/s |
+| llama-bench pp2048 | see results.txt | about 285 t/s |
+| llama-bench tg32 | | not a valid metric, see above |
+
+Older reference points for the same model/config: `pp2048` depth-0 about
+285 t/s. The historical "tg32 about 19 t/s" figure is retained only as trivia -
+do not treat it as a target, and see the tg32 warning above.
+
+Model load with `--no-mmap` takes about 26 s warm, longer cold.
+A full 3-round A/B run of both harnesses takes roughly 100 minutes;
+add about 16 minutes per extra arm per round.

scripts/rx580-bench/bench.sh

@@ -0,0 +1,167 @@
+#!/bin/sh
+#
+# RX 580 prompt-processing benchmark orchestrator. Runs ON THE POD.
+#
+# Stops llama-swap, waits for the GPU to go idle, runs the real-corpus
+# prompt-processing harness and the llama-bench sweep for every arm, appends
+# parseable results, and ALWAYS restarts llama-swap on exit (success, error
+# or Ctrl-C).
+#
+# Usage:
+#   bench.sh --build DIR --label NAME
+#            [--build-b DIR --label-b NAME]   second arm, interleaved
+#            [--arm DIR:LABEL ...]            extra arms, repeatable (N-way)
+#            [--rounds N]                     default 3 multi-arm, 1 otherwise
+#            [--reps N]                       corpus repeats per size, default 3
+#            [--sizes 4096,16384,32768]
+#            [--corpus-only | --lb-only]
+#            [--results PATH]                 default /root/bench/results.txt
+#            [--keep-swap-down]               do not restart llama-swap at exit
+#
+set -e
+
+BENCH_DIR=/root/bench
+SUPCONF=/root/supervisord.conf
+RESULTS="$BENCH_DIR/results.txt"
+MODEL=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
+
+ARMS=""            # space separated DIR|LABEL entries
+BUILD_A=""; LABEL_A=""
+BUILD_B=""; LABEL_B=""
+ROUNDS=""
+REPS=3
+SIZES=4096,16384,32768
+DO_CORPUS=1
+DO_LB=1
+KEEP_SWAP_DOWN=0
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --build)     BUILD_A="$2"; shift 2 ;;
+    --label)     LABEL_A="$2"; shift 2 ;;
+    --build-b)   BUILD_B="$2"; shift 2 ;;
+    --label-b)   LABEL_B="$2"; shift 2 ;;
+    --arm)       ARMS="$ARMS ${2%%:*}|${2#*:}"; shift 2 ;;
+    --rounds)    ROUNDS="$2"; shift 2 ;;
+    --reps)      REPS="$2"; shift 2 ;;
+    --sizes)     SIZES="$2"; shift 2 ;;
+    --results)   RESULTS="$2"; shift 2 ;;
+    --corpus-only) DO_LB=0; shift ;;
+    --lb-only)     DO_CORPUS=0; shift ;;
+    --keep-swap-down) KEEP_SWAP_DOWN=1; shift ;;
+    -h|--help)   sed -n '2,20p' "$0"; exit 0 ;;
+    *) echo "unknown arg: $1" >&2; exit 2 ;;
+  esac
+done
+
+if [ -n "$BUILD_A" ] && [ -z "$LABEL_A" ]; then
+  echo "error: --build needs --label" >&2; exit 2
+fi
+if [ -n "$BUILD_B" ] && [ -z "$LABEL_B" ]; then
+  echo "error: --build-b needs --label-b" >&2; exit 2
+fi
+# --build/--label and --build-b/--label-b are sugar for the first two arms.
+if [ -n "$BUILD_A" ]; then ARMS="$BUILD_A|$LABEL_A $ARMS"; fi
+if [ -n "$BUILD_B" ]; then ARMS="$ARMS $BUILD_B|$LABEL_B"; fi
+ARMS=$(echo $ARMS)
+if [ -z "$ARMS" ]; then
+  echo "error: need at least one arm (--build/--label or --arm DIR:LABEL)" >&2
+  exit 2
+fi
+NARMS=$(echo "$ARMS" | wc -w)
+if [ -z "$ROUNDS" ]; then
+  if [ "$NARMS" -gt 1 ]; then ROUNDS=3; else ROUNDS=1; fi
+fi
+for e in $ARMS; do
+  d=${e%%|*}
+  if [ ! -x "$d/bin/llama-server" ] || [ ! -x "$d/bin/llama-bench" ]; then
+    echo "error: $d lacks bin/llama-server or bin/llama-bench" >&2
+    exit 2
+  fi
+done
+
+cat <<'WARN'
+-------------------------------------------------------------------------
+RX 580 benchmark. Read this before trusting any number.
+
+  * Cross-invocation variance on this box is about 7 percent. llama-bench
+    within-run error bars understate it by roughly 14x. Any cross-build
+    delta under about 5 percent is UNPROVEN noise.
+  * Comparisons you care about must live inside ONE llama-bench invocation
+    (comma-separated sweeps), or be interleaved across at least 3 rounds.
+    That is what multiple arms plus --rounds does.
+  * Nothing else may touch the GPU while this runs. llama-swap is stopped
+    for the duration and restarted on exit, including on error or Ctrl-C.
+  * Traps: -tb is NOT a llama-bench flag. A ubatch larger than -p never
+    fills, so prompt lengths must be multiples of the 2048 ubatch. A bad
+    flag makes llama-bench print usage and exit silently mid-sweep.
+-------------------------------------------------------------------------
+WARN
+
+echo "[bench] $NARMS arm(s), $ROUNDS round(s): $ARMS"
+
+restore_swap() {
+  rc=$?
+  if [ "$KEEP_SWAP_DOWN" -eq 0 ]; then
+    echo ""
+    echo "[bench] restarting llama-swap"
+    supervisorctl -c "$SUPCONF" start llama-swap || true
+    supervisorctl -c "$SUPCONF" status llama-swap || true
+  else
+    echo "[bench] --keep-swap-down: llama-swap left STOPPED"
+  fi
+  exit $rc
+}
+trap restore_swap EXIT INT TERM
+
+echo "[bench] stopping llama-swap"
+supervisorctl -c "$SUPCONF" stop llama-swap || true
+
+wait_gpu_idle() {
+  i=0
+  while [ $i -lt 60 ]; do
+    if ! pgrep -f 'bin/llama-server|bin/llama-bench|bin/llama-cli' >/dev/null 2>&1; then
+      return 0
+    fi
+    sleep 2
+    i=$((i+1))
+  done
+  echo "error: something is still on the GPU:" >&2
+  pgrep -af 'bin/llama-server|bin/llama-bench|bin/llama-cli' >&2
+  exit 3
+}
+
+echo "[bench] waiting for the GPU to go idle"
+wait_gpu_idle
+echo "[bench] GPU idle"
+
+run_corpus() {
+  [ "$DO_CORPUS" -eq 1 ] || return 0
+  echo "===== corpus $2 ====="
+  wait_gpu_idle
+  python3 "$BENCH_DIR/ppbench.py" "$1" "$2" \
+    --sizes "$SIZES" --reps "$REPS" --results "$RESULTS"
+}
+
+run_lb() {
+  [ "$DO_LB" -eq 1 ] || return 0
+  echo "===== llama-bench $2 ====="
+  wait_gpu_idle
+  RESULTS="$RESULTS" MODEL="$MODEL" sh "$BENCH_DIR/lbsweep.sh" "$1" "$2"
+}
+
+# Interleave: every arm is measured once per round, so slow drift in the
+# machine hits all arms roughly equally instead of biasing whichever ran first.
+r=1
+while [ "$r" -le "$ROUNDS" ]; do
+  echo ""
+  echo "########## ROUND $r / $ROUNDS ##########"
+  for e in $ARMS; do run_corpus "${e%%|*}" "${e#*|}"; done
+  for e in $ARMS; do run_lb     "${e%%|*}" "${e#*|}"; done
+  r=$((r+1))
+done
+
+echo ""
+echo "ALL_RUNS_DONE"
+echo "[bench] raw results appended to $RESULTS"
+python3 "$BENCH_DIR/summarize.py" "$RESULTS" || true

scripts/rx580-bench/lbsweep.sh

@@ -0,0 +1,69 @@
+#!/bin/sh
+#
+# llama-bench sweep, the controlled cross-check for the corpus harness.
+# Runs ON THE POD. Normally invoked by bench.sh.
+#
+# Usage: lbsweep.sh <build_dir> <label>
+# Env overrides: RESULTS, MODEL, LB_P, LB_N, LB_R
+#
+# Everything that must be compared lives inside ONE llama-bench invocation
+# (-p takes a comma-separated list), because cross-invocation variance on this
+# box is about 7 percent while the within-run error bars are about 0.3 percent.
+#
+# Flag traps, learned the hard way:
+#   * -tb is NOT a llama-bench flag. It exists on llama-server only.
+#   * -ncmoe is the llama-bench spelling of --n-cpu-moe.
+#   * -mmp 0 is the llama-bench equivalent of the server's --no-mmap.
+#   * A bad flag makes llama-bench print usage and exit silently, which looks
+#     exactly like a sweep that produced no rows. Always check row count.
+#   * A ubatch larger than -p never fills, so keep -p a multiple of -ub.
+#
+set -e
+BUILD_DIR="$1"
+LABEL="$2"
+if [ -z "$BUILD_DIR" ] || [ -z "$LABEL" ]; then
+  echo "usage: $0 <build_dir> <label>" >&2
+  exit 2
+fi
+
+RESULTS="${RESULTS:-/root/bench/results.txt}"
+MODEL="${MODEL:-/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf}"
+LB_P="${LB_P:-2048,8192}"
+LB_N="${LB_N:-32}"
+LB_R="${LB_R:-3}"
+
+STAMP=$(date -u +%Y-%m-%dT%H:%M:%S)
+export LD_LIBRARY_PATH="$BUILD_DIR/bin"
+export LABEL BUILD_DIR STAMP RESULTS
+
+OUT=$("$BUILD_DIR/bin/llama-bench" \
+  -m "$MODEL" \
+  -t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -mmp 0 \
+  -p "$LB_P" -n "$LB_N" -r "$LB_R" -o json 2>/dev/null)
+
+if [ -z "$OUT" ]; then
+  echo "ERROR: llama-bench produced no output for $LABEL (bad flag?)" >&2
+  exit 3
+fi
+
+printf '%s' "$OUT" | python3 -c '
+import json, sys, os
+label = os.environ["LABEL"]
+build = os.environ["BUILD_DIR"]
+stamp = os.environ["STAMP"]
+results = os.environ["RESULTS"]
+rows = json.load(sys.stdin)
+if not rows:
+    sys.stderr.write("ERROR: llama-bench returned zero rows\n")
+    sys.exit(3)
+out = open(results, "a")
+for r in rows:
+    test = "pp%d" % r["n_prompt"] if r["n_prompt"] else "tg%d" % r["n_gen"]
+    line = ("LBRESULT label=%s build=%s test=%s tps=%.2f stddev=%.2f "
+            "samples=%s time=%s"
+            % (label, build, test, r["avg_ts"], r["stddev_ts"],
+               r["samples_ts"], stamp))
+    out.write(line + "\n")
+    print(line)
+out.close()
+'

scripts/rx580-bench/ppbench.py

@@ -0,0 +1,239 @@
+#!/usr/bin/env python3
+"""
+Real-corpus prompt-processing benchmark for llama-server on the RX 580 pod.
+
+Runs ON THE POD. Normally invoked by bench.sh, not directly.
+
+Starts llama-server from a given build directory with the production config,
+waits for /health, POSTs fixed slices of the Pan Tadeusz corpus to /completion,
+records timings.prompt_n / prompt_ms / prompt_per_second, then shuts the server
+down cleanly.
+
+Prompt slices are FIXED CONSTANTS (see SLICES below), calibrated once against
+the Qwen3.6-35B-A3B tokenizer, so that every build is measured on byte-identical
+input. Do not re-calibrate for a normal run; use --calibrate only if the model
+or the corpus changes.
+
+Usage:
+  ppbench.py <build_dir> <label> [--sizes 4096,16384,32768] [--reps 3]
+             [--results PATH] [--calibrate]
+"""
+
+import json
+import os
+import signal
+import subprocess
+import sys
+import time
+import urllib.request
+from statistics import median
+
+BENCH_DIR = "/root/bench"
+CORPUS = os.path.join(BENCH_DIR, "pan-tadeusz.txt")
+DEFAULT_RESULTS = os.path.join(BENCH_DIR, "results.txt")
+
+MODEL = ("/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/"
+         "snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/"
+         "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf")
+
+PORT = 8099
+BASE = "http://127.0.0.1:%d" % PORT
+
+# Production serving config. Keep in sync with /root/config.yaml (llama-swap).
+# Traps encoded here on purpose:
+#   -b / -ub 2048 : a ubatch larger than the prompt never fills, so every
+#                   prompt length below must stay a multiple of 2048.
+#   --no-mmap     : deprecated upstream in favour of --load-mode / -lm none,
+#                   but still accepted; it is what production passes today.
+SERVER_ARGS = ["-t", "6", "-ngl", "99", "--n-cpu-moe", "40",
+               "-b", "2048", "-ub", "2048", "-fa", "1", "--no-mmap",
+               "--ctx-size", "40960", "--no-warmup",
+               "--port", str(PORT), "-m", MODEL]
+
+# Calibrated once on 2026-09-08 against Qwen3.6-35B-A3B-UD-Q4_K_XL:
+# Polish text of pan-tadeusz.txt runs 2.6702 chars/token, and /completion
+# reports prompt_n identical to /tokenize (BOS offset 0).
+# Each slice is corpus[0:NCHARS] read as UTF-8 text; NBYTES is the resulting
+# UTF-8 byte length, recorded so the slice can be reproduced with byte tools.
+# These produce EXACTLY the target prompt_n. Do not edit without recalibrating.
+SLICES = {
+    4096:  {"chars": 10776, "bytes": 11608},
+    16384: {"chars": 43858, "bytes": 47175},
+    32768: {"chars": 87165, "bytes": 93843},
+}
+
+DEFAULT_SIZES = [4096, 16384, 32768]
+
+
+def post(path, payload, timeout=2400):
+    req = urllib.request.Request(
+        BASE + path,
+        data=json.dumps(payload).encode("utf-8"),
+        headers={"Content-Type": "application/json"},
+        method="POST")
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.loads(r.read().decode("utf-8"))
+
+
+def health_ok():
+    try:
+        with urllib.request.urlopen(BASE + "/health", timeout=5) as r:
+            return json.loads(r.read().decode("utf-8")).get("status") == "ok"
+    except Exception:
+        return False
+
+
+def start_server(build_dir, logpath):
+    env = dict(os.environ)
+    env["LD_LIBRARY_PATH"] = os.path.join(build_dir, "bin")
+    binary = os.path.join(build_dir, "bin", "llama-server")
+    if not os.path.exists(binary):
+        raise RuntimeError("no llama-server at %s" % binary)
+    log = open(logpath, "wb")
+    p = subprocess.Popen([binary] + SERVER_ARGS, stdout=log,
+                         stderr=subprocess.STDOUT, env=env,
+                         start_new_session=True)
+    deadline = time.time() + 1200
+    while time.time() < deadline:
+        if p.poll() is not None:
+            raise RuntimeError("llama-server exited early rc=%s, see %s"
+                               % (p.returncode, logpath))
+        if health_ok():
+            return p
+        time.sleep(2)
+    stop_server(p)
+    raise RuntimeError("llama-server not healthy within 1200s, see %s" % logpath)
+
+
+def stop_server(p):
+    if p is None or p.poll() is not None:
+        return
+    try:
+        os.killpg(os.getpgid(p.pid), signal.SIGTERM)
+    except Exception:
+        p.terminate()
+    for _ in range(120):
+        if p.poll() is not None:
+            return
+        time.sleep(1)
+    try:
+        os.killpg(os.getpgid(p.pid), signal.SIGKILL)
+    except Exception:
+        p.kill()
+    p.wait()
+
+
+def n_tokens(text):
+    """Exact token count via /tokenize. Cheap: no prefill."""
+    return len(post("/tokenize", {"content": text}, timeout=300)["tokens"])
+
+
+def calibrate(corpus, targets):
+    """Recompute the SLICES table. Only needed if model or corpus changes."""
+    probe = corpus[:2000]
+    tk = n_tokens(probe)
+    r = post("/completion", {"prompt": probe, "n_predict": 1,
+                             "cache_prompt": False, "temperature": 0})
+    offset = r["timings"]["prompt_n"] - tk
+    cpt = len(probe) / float(tk)
+    sys.stderr.write("calibrate: %.4f chars/token, prompt_n offset %d\n"
+                     % (cpt, offset))
+    for target in targets:
+        want = target - offset
+        lo, hi = 1, len(corpus)
+        guess = min(len(corpus), max(1, int(want * cpt)))
+        best = None
+        for _ in range(60):
+            got = n_tokens(corpus[:guess])
+            if got == want:
+                best = guess
+                break
+            if got < want:
+                lo = guess + 1
+            else:
+                hi = guess - 1
+            if lo > hi:
+                break
+            guess = (lo + hi) // 2
+        if best is None:
+            best = guess
+        nbytes = len(corpus[:best].encode("utf-8"))
+        sys.stderr.write("calibrate: %d tokens -> chars=%d bytes=%d\n"
+                         % (target, best, nbytes))
+
+
+def main():
+    if len(sys.argv) < 3:
+        sys.stderr.write(__doc__)
+        return 2
+    build_dir = os.path.abspath(sys.argv[1])
+    label = sys.argv[2]
+    sizes = list(DEFAULT_SIZES)
+    reps = 3
+    results_path = DEFAULT_RESULTS
+    do_calibrate = "--calibrate" in sys.argv
+    args = sys.argv[3:]
+    for i, a in enumerate(args):
+        if a == "--sizes":
+            sizes = [int(x) for x in args[i + 1].split(",")]
+        elif a == "--reps":
+            reps = int(args[i + 1])
+        elif a == "--results":
+            results_path = args[i + 1]
+
+    corpus = open(CORPUS, encoding="utf-8").read()
+    logpath = os.path.join(BENCH_DIR, "server-%s-%d.log" % (label, int(time.time())))
+
+    p = None
+    try:
+        sys.stderr.write("[%s] starting llama-server from %s\n" % (label, build_dir))
+        t0 = time.time()
+        p = start_server(build_dir, logpath)
+        sys.stderr.write("[%s] healthy after %.1fs\n" % (label, time.time() - t0))
+
+        if do_calibrate:
+            calibrate(corpus, sizes)
+            return 0
+
+        stamp = time.strftime("%Y-%m-%dT%H:%M:%S")
+        out = open(results_path, "a")
+        for target in sizes:
+            if target not in SLICES:
+                sys.stderr.write("no calibrated slice for %d tokens, skipping\n" % target)
+                continue
+            nchars = SLICES[target]["chars"]
+            prompt = corpus[:nchars]
+            rates, mss, ns = [], [], set()
+            for rep in range(1, reps + 1):
+                r = post("/completion", {"prompt": prompt, "n_predict": 1,
+                                         "cache_prompt": False, "temperature": 0})
+                t = r["timings"]
+                rates.append(t["prompt_per_second"])
+                mss.append(t["prompt_ms"])
+                ns.add(t["prompt_n"])
+                line = ("RESULT corpus label=%s build=%s target=%d chars=%d "
+                        "prompt_n=%d rep=%d prompt_ms=%.2f tps=%.2f time=%s"
+                        % (label, build_dir, target, nchars, t["prompt_n"], rep,
+                           t["prompt_ms"], t["prompt_per_second"], stamp))
+                out.write(line + "\n")
+                out.flush()
+                sys.stderr.write(line + "\n")
+            if ns != {target}:
+                sys.stderr.write("WARNING: prompt_n %s != target %d; slice table "
+                                 "is stale, rerun with --calibrate\n" % (sorted(ns), target))
+            s = ("SUMMARY corpus label=%s target=%d prompt_n=%s median_tps=%.2f "
+                 "min_tps=%.2f max_tps=%.2f median_ms=%.1f time=%s"
+                 % (label, target, sorted(ns), median(rates), min(rates),
+                    max(rates), median(mss), stamp))
+            out.write(s + "\n")
+            out.flush()
+            sys.stderr.write(s + "\n")
+        out.close()
+    finally:
+        stop_server(p)
+        sys.stderr.write("[%s] server stopped\n" % label)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/rx580-bench/run.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+#
+# One-command entry point for the RX 580 prompt-processing benchmark.
+#
+# Runs from your workstation. Copies the pod-side harness into the pod and
+# execs it there over kubectl. All the real work happens on the pod; this is
+# a thin wrapper so there is exactly one command to remember.
+#
+# Examples:
+#   ./scripts/rx580-bench/run.sh --build /root/llama.cpp/build --label fork-before
+#   ./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean \
+#                               --build-b /root/arms/new-fork/build --label-b new-fork
+#   ./scripts/rx580-bench/run.sh --summarize
+#   ./scripts/rx580-bench/run.sh --install-only
+#
+# Every argument other than the wrapper-only flags below is passed straight
+# through to bench.sh on the pod. See bench.sh --help.
+#
+set -euo pipefail
+
+KUBECONFIG_PATH="${KUBECONFIG:-/home/user/Projects/klaster/talos/generated/kubeconfig}"
+NS="${RX580_NS:-llama}"
+DEPLOY="${RX580_DEPLOY:-deploy/supervisord}"
+POD_BENCH_DIR=/root/bench
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+export KUBECONFIG="$KUBECONFIG_PATH"
+
+kx() { kubectl -n "$NS" exec -i "$DEPLOY" -- sh -c "$1"; }
+
+install_harness() {
+  echo "[run] installing harness into $POD_BENCH_DIR"
+  kx "mkdir -p $POD_BENCH_DIR"
+  for f in bench.sh ppbench.py lbsweep.sh summarize.py; do
+    kx "cat > $POD_BENCH_DIR/$f" < "$HERE/$f"
+  done
+  kx "chmod +x $POD_BENCH_DIR/bench.sh $POD_BENCH_DIR/lbsweep.sh $POD_BENCH_DIR/ppbench.py $POD_BENCH_DIR/summarize.py"
+  # The corpus is large and rarely changes; only push it if missing.
+  if ! kx "test -s $POD_BENCH_DIR/pan-tadeusz.txt" 2>/dev/null; then
+    if [ -f "$HERE/pan-tadeusz.txt" ]; then
+      echo "[run] uploading corpus"
+      kx "cat > $POD_BENCH_DIR/pan-tadeusz.txt" < "$HERE/pan-tadeusz.txt"
+    else
+      echo "[run] ERROR: $POD_BENCH_DIR/pan-tadeusz.txt missing on the pod and" >&2
+      echo "      no local copy at $HERE/pan-tadeusz.txt to upload." >&2
+      exit 4
+    fi
+  fi
+}
+
+ARGS=()
+INSTALL_ONLY=0
+SUMMARIZE=0
+for a in "$@"; do
+  case "$a" in
+    --install-only) INSTALL_ONLY=1 ;;
+    --summarize)    SUMMARIZE=1 ;;
+    *) ARGS+=("$a") ;;
+  esac
+done
+
+install_harness
+
+if [ "$INSTALL_ONLY" -eq 1 ]; then
+  echo "[run] harness installed; not running anything"
+  exit 0
+fi
+
+if [ "$SUMMARIZE" -eq 1 ]; then
+  kx "python3 $POD_BENCH_DIR/summarize.py ${ARGS[*]:-}"
+  exit 0
+fi
+
+if [ ${#ARGS[@]} -eq 0 ]; then
+  kx "sh $POD_BENCH_DIR/bench.sh --help"
+  exit 2
+fi
+
+# A full 3-round A/B run takes hours; run detached on the pod and tail it, so a
+# dropped kubectl connection cannot orphan a half-finished run with llama-swap
+# still down. bench.sh restarts llama-swap from its own EXIT trap either way.
+STAMP="$(date -u +%Y%m%d-%H%M%S)"
+LOG="$POD_BENCH_DIR/run-$STAMP.log"
+echo "[run] starting detached run on the pod, log: $LOG"
+kx "cd $POD_BENCH_DIR && nohup sh $POD_BENCH_DIR/bench.sh ${ARGS[*]} > $LOG 2>&1 & echo started"
+
+echo "[run] tailing until ALL_RUNS_DONE (safe to Ctrl-C: the pod keeps running)"
+kx "i=0; while [ \$i -lt 100000 ]; do
+      if grep -qE 'ALL_RUNS_DONE|Traceback|error:' $LOG; then break; fi
+      sleep 15; i=\$((i+1));
+    done; cat $LOG"

scripts/rx580-bench/summarize.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""
+Summarize a results.txt produced by bench.sh into a per-label table.
+
+Usage: summarize.py [results.txt]
+
+Prints median and min-max of every metric for every label, and flags
+cross-label deltas against the ~5 percent noise floor of this machine.
+"""
+
+import re
+import sys
+from statistics import median
+
+NOISE_FLOOR_PCT = 5.0
+
+RE_CORPUS = re.compile(
+    r"^RESULT corpus label=(\S+) build=\S+ target=(\d+) chars=\d+ "
+    r"prompt_n=(\d+) rep=\d+ prompt_ms=([\d.]+) tps=([\d.]+)")
+RE_LB = re.compile(
+    r"^LBRESULT label=(\S+) build=\S+ test=(\S+) tps=([\d.]+)")
+
+
+def main():
+    path = sys.argv[1] if len(sys.argv) > 1 else "/root/bench/results.txt"
+    # data[metric][label] = list of tps
+    data = {}
+    order = []
+    labels = []
+    for line in open(path):
+        m = RE_CORPUS.match(line)
+        if m:
+            label, target, _pn, _ms, tps = m.groups()
+            metric = "pp%s-corpus" % target
+        else:
+            m = RE_LB.match(line)
+            if not m:
+                continue
+            label, test, tps = m.groups()
+            metric = "%s-llama-bench" % test
+        if metric not in data:
+            data[metric] = {}
+            order.append(metric)
+        data[metric].setdefault(label, []).append(float(tps))
+        if label not in labels:
+            labels.append(label)
+
+    if not data:
+        print("no parseable results in %s" % path)
+        return 1
+
+    def sort_key(m):
+        n = re.search(r"(\d+)", m)
+        return (0 if "corpus" in m else 1, int(n.group(1)) if n else 0)
+    order.sort(key=sort_key)
+
+    w = max(len(l) for l in labels) + 2
+    head = "metric".ljust(20) + "".join(l.ljust(max(w, 24)) for l in labels)
+    print("")
+    print("t/s, median (min-max), n samples")
+    print(head)
+    print("-" * len(head))
+    for metric in order:
+        row = metric.ljust(20)
+        for label in labels:
+            vals = data[metric].get(label)
+            if not vals:
+                row += "-".ljust(max(w, 24))
+            else:
+                cell = "%.1f (%.1f-%.1f) n=%d" % (
+                    median(vals), min(vals), max(vals), len(vals))
+                row += cell.ljust(max(w, 24))
+        print(row)
+
+    if len(labels) >= 2:
+        base = labels[0]
+        print("")
+        print("deltas vs %s (noise floor %.0f%%, anything under it is UNPROVEN)"
+              % (base, NOISE_FLOOR_PCT))
+        for other in labels[1:]:
+            print("  %s vs %s:" % (other, base))
+            for metric in order:
+                a = data[metric].get(base)
+                b = data[metric].get(other)
+                if not a or not b:
+                    continue
+                ma, mb = median(a), median(b)
+                pct = (mb - ma) / ma * 100.0
+                verdict = "SIGNIFICANT" if abs(pct) >= NOISE_FLOOR_PCT else "unproven"
+                print("    %-20s %+6.1f%%  %s" % (metric, pct, verdict))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

src/llama-context.cpp

@@ -483,7 +483,8 @@ llama_context::~llama_context() {
     synchronize();
 
     // when training, ggml_opt allocates extra buffers through the scheduler, so the sizes no longer match the expectation
-    if (!model.hparams.no_alloc && !opt_ctx) {
+    // the scheduler is also gone if the context is destroyed while cold (on-demand VRAM eviction, see release_device)
+    if (sched && !model.hparams.no_alloc && !opt_ctx) {
         for (size_t i = 0; i < backend_ptrs.size(); ++i) {
             ggml_backend_t             backend = backend_ptrs[i];
             ggml_backend_buffer_type_t buft    = backend_buft[i];
@@ -755,11 +756,20 @@ void llama_context::release_device(bool evict_kv) {
     if (evict_kv && memory && !kv_device_evicted) {
         memory->release_device_buffers();
         kv_device_evicted = true;
+        // also free the scheduler and its worst-case compute buffer (hundreds of MiB) so a cold
+        // model holds essentially no VRAM. Rebuilt lazily by sched_reserve() on restore.
+        sched.reset();
+        sched_need_reserve = true;
+        // finally, free each backend's own compute-scratch (Vulkan prealloc/staging buffers, which
+        // are owned by the backend and survive sched.reset()). Reallocated lazily on next compute.
+        for (auto & backend : backends) {
+            ggml_backend_free_scratch(backend.get());
+        }
     }
 }
 
 void llama_context::restore_device() {
-    if (model.weights_resident() && !kv_device_evicted) {
+    if (model.weights_resident() && !kv_device_evicted && sched) {
         return;
     }
     model.restore_device_weights();
@@ -767,6 +777,10 @@ void llama_context::restore_device() {
         memory->restore_device_buffers();
         kv_device_evicted = false;
     }
+    // rebuild the scheduler + compute buffer if they were freed on release (evict_kv mode)
+    if (!sched) {
+        sched_reserve();
+    }
     // make sure all weight/KV uploads have completed before any compute reads them
     if (sched) {
         ggml_backend_sched_synchronize(sched.get());
@@ -3400,17 +3414,21 @@ llama_memory_breakdown llama_context::memory_breakdown() const {
             ret[buft].context += size;
         }
     }
-    if (model.hparams.no_alloc) {
-        for (size_t i = 0; i < backends.size(); ++i) {
-            ggml_backend_t             backend = backends[i].get();
-            ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
-            ret[buft].compute += backend_buf_exp_size[i];
-        }
-    } else {
-        for (const auto & backend_ptr : backends) {
-            ggml_backend_t             backend = backend_ptr.get();
-            ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
-            ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
+    // the scheduler (and its compute buffers) may have been freed while the model is cold
+    // (on-demand VRAM eviction, see release_device); it contributes no compute VRAM then.
+    if (sched) {
+        if (model.hparams.no_alloc) {
+            for (size_t i = 0; i < backends.size(); ++i) {
+                ggml_backend_t             backend = backends[i].get();
+                ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
+                ret[buft].compute += backend_buf_exp_size[i];
+            }
+        } else {
+            for (const auto & backend_ptr : backends) {
+                ggml_backend_t             backend = backend_ptr.get();
+                ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
+                ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
+            }
         }
     }
     return ret;

src/llama-memory-hybrid.cpp

@@ -202,7 +202,7 @@ void llama_memory_hybrid::state_read(llama_io_read_i & io, llama_seq_id seq_id,
 }
 
 void llama_memory_hybrid::release_device_buffers() {
-    // evict the attention KV (grows with context); the recurrent state uses the no-op default
+    // evict both the attention KV (grows with context) and the recurrent/SSM state
     mem_attn->release_device_buffers();
     mem_recr->release_device_buffers();
 }

src/llama-memory-recurrent.cpp

@@ -151,7 +151,9 @@ void llama_memory_recurrent::clear(bool data) {
 
     if (data) {
         for (auto & [_, buf] : ctxs_bufs) {
-            ggml_backend_buffer_clear(buf.get(), 0);
+            if (buf) { // may be null if evicted for on-demand VRAM sharing
+                ggml_backend_buffer_clear(buf.get(), 0);
+            }
         }
     }
 
@@ -420,6 +422,7 @@ void llama_memory_recurrent::set_rs_idx(llama_seq_id seq_id, uint32_t idx) {
 std::map<ggml_backend_buffer_type_t, size_t> llama_memory_recurrent::memory_breakdown() const {
     std::map<ggml_backend_buffer_type_t, size_t> ret;
     for (const auto & [_, buf] : ctxs_bufs) {
+        if (!buf) { continue; } // may be null if evicted for on-demand VRAM sharing
         ret[ggml_backend_buffer_get_type(buf.get())] += ggml_backend_buffer_get_size(buf.get());
     }
     return ret;
@@ -721,12 +724,87 @@ bool llama_memory_recurrent::get_can_shift() const {
 size_t llama_memory_recurrent::total_size() const {
     size_t size = 0;
     for (const auto & [_, buf] : ctxs_bufs) {
-        size += ggml_backend_buffer_get_size(buf.get());
+        if (buf) { // may be null if evicted for on-demand VRAM sharing
+            size += ggml_backend_buffer_get_size(buf.get());
+        }
     }
 
     return size;
 }
 
+void llama_memory_recurrent::release_device_buffers() {
+    // Same mechanism as llama_kv_cache: the recurrent (SSM/conv) state is read-write, so its host
+    // shadow is (re)captured on every release. The caller must have synchronized the backend.
+    if (dev_released) {
+        return;
+    }
+    dev_shadows.assign(ctxs_bufs.size(), device_buffer_shadow{});
+    size_t freed = 0;
+    for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
+        ggml_context * ctx = ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = ctxs_bufs[i].second.get();
+        if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
+            continue;
+        }
+        auto & sh = dev_shadows[i];
+        sh.releasable = true;
+        sh.buft = ggml_backend_buffer_get_type(buf);
+
+        size_t total = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src == nullptr) { total += ggml_nbytes(t); }
+        }
+        sh.data.resize(total);
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_get(t, sh.data.data() + off, 0, n);
+            off += n;
+        }
+
+        freed += ggml_backend_buffer_get_size(buf);
+        ctxs_bufs[i].second.reset();
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            t->buffer = nullptr;
+            t->data   = nullptr;
+        }
+    }
+    dev_released = true;
+    if (freed > 0) {
+        LLAMA_LOG_INFO("%s: released %.2f MiB of recurrent state from device\n", __func__, freed / 1024.0 / 1024.0);
+    }
+}
+
+bool llama_memory_recurrent::restore_device_buffers() {
+    if (!dev_released) {
+        return true;
+    }
+    for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
+        auto & sh = dev_shadows[i];
+        if (!sh.releasable) {
+            continue;
+        }
+        ggml_context * ctx = ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, sh.buft);
+        if (buf == nullptr) {
+            LLAMA_LOG_ERROR("%s: failed to reallocate recurrent device buffer (out of VRAM?)\n", __func__);
+            return false;
+        }
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_set(t, sh.data.data() + off, 0, n);
+            off += n;
+        }
+        ctxs_bufs[i].second.reset(buf);
+    }
+    dev_released = false;
+    dev_shadows.clear();
+    return true;
+}
+
 size_t llama_memory_recurrent::size_r_bytes() const {
     size_t size_r_bytes = 0;
 

src/llama-memory-recurrent.h

@@ -66,6 +66,10 @@ public:
     void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
     void state_read (llama_io_read_i  & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
 
+    // on-demand device (VRAM) residency (see llama_memory_i)
+    void release_device_buffers() override;
+    bool restore_device_buffers() override;
+
     uint32_t head = 0; // the location where the batch will be placed in the cache (see find_slot())
     uint32_t size = 0; // total number of cells, shared across all sequences
     uint32_t used = 0; // used cells (i.e. at least one seq_id)
@@ -123,6 +127,16 @@ private:
     // ggml contexts for the KV cache along with the allocated backend buffers:
     std::vector<std::pair<ggml_context_ptr, ggml_backend_buffer_ptr>> ctxs_bufs;
 
+    // on-demand device eviction (see release_device_buffers): host shadow of each device buffer's
+    // live contents (recaptured on every release since the recurrent state is read-write)
+    struct device_buffer_shadow {
+        ggml_backend_buffer_type_t buft = nullptr;
+        bool releasable = false;
+        std::vector<uint8_t> data;
+    };
+    std::vector<device_buffer_shadow> dev_shadows; // parallel to ctxs_bufs
+    bool dev_released = false;
+
     size_t total_size() const;
 
     size_t size_r_bytes() const;

tools/mtmd/clip.cpp

@@ -159,6 +159,13 @@ struct clip_ctx {
     ggml_backend_t backend_cpu = nullptr;
     ggml_backend_buffer_ptr buf;
 
+    // on-demand device (VRAM) residency: the vision/audio encoder weights are read-only, so a host
+    // shadow is captured once and the device buffer can be freed while the model is cold, then
+    // rebuilt on wake (mirrors llama_model::release_device_weights). See clip_release_device().
+    ggml_backend_buffer_type_t dev_buft = nullptr;
+    std::vector<uint8_t>       dev_shadow;
+    bool                       dev_released = false;
+
 
     int max_nodes = 8192;
     ggml_backend_sched_ptr sched;
@@ -4026,6 +4033,74 @@ void clip_free(clip_ctx * ctx) {
     delete ctx;
 }
 
+void clip_release_device(struct clip_ctx * ctx) {
+    if (ctx == nullptr || ctx->dev_released || ctx->no_alloc) {
+        return;
+    }
+    ggml_backend_buffer_t buf = ctx->buf.get();
+    if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
+        return; // CPU-backed encoder: nothing in VRAM to free
+    }
+    // ensure no encode is in flight before freeing the weights
+    if (ctx->sched) {
+        ggml_backend_sched_synchronize(ctx->sched.get());
+    }
+    ctx->dev_buft = ggml_backend_buffer_get_type(buf);
+
+    ggml_context * cd = ctx->ctx_data.get();
+    // capture the host shadow once (weights are read-only): compact, stable iteration order,
+    // skipping view tensors (which alias a base and are restored implicitly)
+    if (ctx->dev_shadow.empty()) {
+        size_t total = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+            if (t->view_src == nullptr) { total += ggml_nbytes(t); }
+        }
+        ctx->dev_shadow.resize(total);
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_get(t, ctx->dev_shadow.data() + off, 0, n);
+            off += n;
+        }
+    }
+
+    // free the device buffer and clear the now-dangling tensor pointers so restore reallocates cleanly
+    ctx->buf.reset();
+    for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+        t->buffer = nullptr;
+        t->data   = nullptr;
+    }
+    ctx->dev_released = true;
+}
+
+bool clip_restore_device(struct clip_ctx * ctx) {
+    if (ctx == nullptr || !ctx->dev_released) {
+        return true;
+    }
+    ggml_context * cd = ctx->ctx_data.get();
+    ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(cd, ctx->dev_buft);
+    if (buf == nullptr) {
+        LOG_ERR("%s: failed to reallocate encoder device buffer (out of VRAM?)\n", __func__);
+        return false;
+    }
+    ggml_backend_buffer_set_usage(buf, GGML_BACKEND_BUFFER_USAGE_WEIGHTS);
+    size_t off = 0;
+    for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+        if (t->view_src != nullptr) { continue; }
+        const size_t n = ggml_nbytes(t);
+        ggml_backend_tensor_set(t, ctx->dev_shadow.data() + off, 0, n);
+        off += n;
+    }
+    ctx->buf.reset(buf);
+    ctx->dev_released = false;
+    return true;
+}
+
+bool clip_weights_resident(const struct clip_ctx * ctx) {
+    return ctx == nullptr || !ctx->dev_released;
+}
+
 const char * clip_patch_merge_type(const struct clip_ctx * ctx) {
     return ctx->model.hparams.mm_patch_merge_type == PATCH_MERGE_SPATIAL_UNPAD ? "spatial_unpad" : "flat";
 }

tools/mtmd/clip.h

@@ -70,6 +70,12 @@ struct clip_init_result clip_init(const char * fname, struct clip_context_params
 
 void clip_free(struct clip_ctx * ctx);
 
+// on-demand device (VRAM) residency: free/rebuild the encoder weight buffer to shrink an idle
+// (cold) multimodal model's VRAM footprint. No-op for a CPU-backed encoder. See clip.cpp.
+void clip_release_device(struct clip_ctx * ctx);
+bool clip_restore_device(struct clip_ctx * ctx);
+bool clip_weights_resident(const struct clip_ctx * ctx);
+
 // TODO: should be enum, not string
 const char * clip_patch_merge_type(const struct clip_ctx * ctx);
 

tools/mtmd/mtmd.cpp

@@ -1116,6 +1116,24 @@ std::vector<std::vector<const mtmd_bitmap *>> mtmd_group_mergeable_bitmaps(std::
     return output;
 }
 
+void mtmd_release_device(mtmd_context * ctx) {
+    if (ctx == nullptr) {
+        return;
+    }
+    if (ctx->ctx_v) { clip_release_device(ctx->ctx_v); }
+    if (ctx->ctx_a) { clip_release_device(ctx->ctx_a); }
+}
+
+bool mtmd_restore_device(mtmd_context * ctx) {
+    if (ctx == nullptr) {
+        return true;
+    }
+    bool ok = true;
+    if (ctx->ctx_v) { ok = clip_restore_device(ctx->ctx_v) && ok; }
+    if (ctx->ctx_a) { ok = clip_restore_device(ctx->ctx_a) && ok; }
+    return ok;
+}
+
 struct mtmd_tokenizer {
     const mtmd_context * ctx;
 

tools/mtmd/mtmd.h

@@ -136,6 +136,12 @@ MTMD_API mtmd_context * mtmd_init_from_file(const char * mmproj_fname,
 
 MTMD_API void mtmd_free(mtmd_context * ctx);
 
+// on-demand device (VRAM) residency: free / rebuild the vision+audio encoder weight buffers so an
+// idle (cold) multimodal model releases its encoder VRAM and reclaims it on wake. No-op for a
+// CPU-backed encoder. Restore returns false if reallocation failed (out of VRAM).
+MTMD_API void mtmd_release_device(mtmd_context * ctx);
+MTMD_API bool mtmd_restore_device(mtmd_context * ctx);
+
 // whether we need to set non-causal mask before llama_decode
 // if chunk is nullptr, we assume the default case where chunk is an image chunk
 MTMD_API bool mtmd_decode_use_non_causal(const mtmd_context * ctx, const mtmd_input_chunk * chunk);

tools/server/server-context.cpp

@@ -38,14 +38,14 @@
 #include <windows.h>
 #endif
 
-// POSIX file locking + inotify doorbell for the cross-process VRAM arbiter (see vram_share_* below)
+// POSIX file locking for the cross-process VRAM arbiter (see vram_share_* below)
 #if !defined(_WIN32)
 #include <sys/file.h>
 #include <sys/stat.h>
-#include <sys/inotify.h>
 #include <fcntl.h>
 #include <unistd.h>
 #include <poll.h>
+#include <cerrno>
 #endif
 
 constexpr int HTTP_POLLING_SECONDS = 1;
@@ -829,7 +829,36 @@ static int process_mtmd_chunk(const server_slot & slot, mtmd::batch_ptr & mbatch
     // TODO @ngxson : move this log line to debug when it become more stable
     SLT_TRC(slot, "encoding mtmd batch from idx = %zu, n_chunks = %d\n", idx, n_added);
 
+    // Bring the vision/audio encoder into VRAM just for this encode. In on-demand mode the
+    // encoder is normally kept in RAM (its VRAM freed for KV / expert cache). If it does not
+    // fit, evict the LLM backbone weights first: they are NOT needed while the encoder runs (the
+    // decode that uses them happens afterwards) and their host shadow is read-only, so this is a
+    // cheap free (no D2H) and leaves the KV cache / prompt cache untouched.
+    static const bool mmproj_ondemand = getenv("LLAMA_MMPROJ_ONDEMAND") != nullptr;
+    bool weights_evicted = false;
+    if (mctx && !mtmd_restore_device(mctx)) {
+        if (mmproj_ondemand) {
+            llama_context_release_device(slot.ctx_tgt, /* evict_kv = */ false); // free backbone, keep KV
+            weights_evicted = true;
+        }
+        if (!mtmd_restore_device(mctx)) {
+            if (weights_evicted) { llama_context_restore_device(slot.ctx_tgt); }
+            SLT_ERR(slot, "%s", "failed to bring the multimodal encoder into VRAM for encoding\n");
+            return -1;
+        }
+    }
+
     res = mtmd_batch_encode(mbatch.get());
+
+    // release the encoder VRAM again (on-demand), then restore the backbone weights for decode.
+    // Order matters: free the encoder BEFORE re-uploading the weights so the peak stays within VRAM.
+    if (mctx && mmproj_ondemand) {
+        mtmd_release_device(mctx);
+    }
+    if (weights_evicted) {
+        llama_context_restore_device(slot.ctx_tgt);
+    }
+
     if (res != 0) {
         SLT_ERR(slot, "failed to encode mtmd batch for chunk idx = %zu, res = %d\n", idx, res);
         return -1;
@@ -869,7 +898,7 @@ public:
     }
 
     ~server_context_impl() {
-        // stop the VRAM warden thread (and release the token) before tearing anything down
+        // release the VRAM token before tearing anything down
         vram_share_shutdown();
         if (!sleeping) {
             // destroy() is already called when entering sleeping state
@@ -910,18 +939,20 @@ private:
     bool vram_only    = false;   // release-mode sleep active (LLAMA_SLEEP_VRAM_ONLY)
     bool vram_evict_kv = false;  // also evict the KV cache to host on release (LLAMA_SLEEP_EVICT_KV)
     bool vram_flock   = false;   // cross-process flock coordination available
-    std::atomic<bool> vram_cold{true}; // weights currently released (read by warden thread)
+    std::atomic<bool> vram_cold{true}; // weights currently released
     int  vram_lock_fd = -1;      // fd for <arena>/token.lock
 
-    // inotify "doorbell": a waiter that wants the VRAM token touches <arena>/doorbell/<pid>, which
-    // wakes the current holder's warden thread so it releases immediately instead of only on idle.
-    std::string       vram_doorbell_dir;
-    std::string       vram_pid_str;
-    int               vram_inotify_fd = -1;
-    std::thread       vram_warden;
-    std::atomic<bool> vram_warden_run{false};
+    // "want" lock: a process that waits for the token holds <arena>/want.lock shared, and drops it
+    // as soon as it owns the token. The holder probes that lock with a non-blocking exclusive
+    // flock, which fails only while somebody waits. This is level-triggered state owned by the
+    // kernel, so there is no event to miss and nothing goes stale if a waiter dies.
+    // The probe needs its own fd: flock() treats two open file descriptions of one file
+    // independently, so probing on the waiter fd would convert our own lock instead of conflicting.
+    int  vram_want_fd       = -1;    // waiter side, LOCK_SH while waiting
+    int  vram_want_probe_fd = -1;    // probe side, LOCK_EX | LOCK_NB only
+    bool vram_want_held     = false; // only touched by the waiting thread
 
-    // open the shared arena (flock token + doorbell dir). Idempotent; sets vram_only/vram_flock.
+    // open the shared arena (token.lock + want.lock). Idempotent; sets vram_only/vram_flock.
     void vram_arena_open() {
         if (getenv("LLAMA_SLEEP_VRAM_ONLY") == nullptr) {
             return;
@@ -939,11 +970,14 @@ private:
         vram_lock_fd = open(lock_path.c_str(), O_RDWR | O_CREAT, 0666);
         if (vram_lock_fd >= 0) {
             vram_flock = true;
-            vram_pid_str = std::to_string(getpid());
-            vram_doorbell_dir = arena + "/doorbell";
-            mkdir(vram_doorbell_dir.c_str(), 0777);
-            SRV_INF("VRAM arbiter: cross-process GPU sharing via %s (doorbell %s)\n",
-                    lock_path.c_str(), vram_doorbell_dir.c_str());
+            const std::string want_path = arena + "/want.lock";
+            vram_want_fd       = open(want_path.c_str(), O_RDWR | O_CREAT, 0666);
+            vram_want_probe_fd = open(want_path.c_str(), O_RDWR | O_CREAT, 0666);
+            if (vram_want_fd < 0 || vram_want_probe_fd < 0) {
+                SRV_WRN("VRAM arbiter: cannot open %s, the holder will only release on idle\n", want_path.c_str());
+            }
+            SRV_INF("VRAM arbiter: cross-process GPU sharing via %s (want %s)\n",
+                    lock_path.c_str(), want_path.c_str());
         } else {
             SRV_WRN("VRAM arbiter: cannot open %s, running VRAM-only sleep without cross-process lock\n", lock_path.c_str());
         }
@@ -959,18 +993,15 @@ private:
         if (!vram_only) {
             return;
         }
-#if !defined(_WIN32)
-        if (vram_flock) {
-            vram_ring_doorbell();         // nudge whoever is resident to release
-            flock(vram_lock_fd, LOCK_EX); // block until the GPU is free
+        if (!vram_take_token()) {
+            SRV_WRN("%s", "VRAM arbiter: loading weights without the GPU token\n");
         }
-#endif
         vram_cold = false; // we hold the token; weights will be resident after the upload
     }
 
-    // Start the doorbell warden. Called from init() after the (coordinated) load. The model stays
-    // warm (holding the token acquired in vram_acquire_for_load) and serves its first request
-    // without a re-warm; the warden releases it when another model rings the doorbell.
+    // Called from init() after the (coordinated) load. The model stays warm (holding the token
+    // acquired in vram_acquire_for_load) and serves its first request without a re-warm; the queue
+    // releases it once another model registers on want.lock.
     void vram_share_init() {
         vram_arena_open();
         if (!vram_only) {
@@ -978,64 +1009,92 @@ private:
         }
         vram_cold = false; // resident and holding the token after the coordinated load
 #if !defined(_WIN32)
-        if (vram_flock && vram_inotify_fd < 0) {
-            vram_inotify_fd = inotify_init1(IN_NONBLOCK);
-            if (vram_inotify_fd >= 0) {
-                inotify_add_watch(vram_inotify_fd, vram_doorbell_dir.c_str(), IN_CLOSE_WRITE);
-                vram_warden_run = true;
-                vram_warden = std::thread([this]{ vram_warden_loop(); });
-            }
+        if (vram_flock && vram_want_probe_fd >= 0) {
+            queue_tasks.on_should_yield([this]{ return vram_should_yield(); });
         }
 #endif
     }
 
-    // warden thread: block on the doorbell; when another process rings (wants the token) and we
-    // currently hold it, ask the loop to yield (release) at its next idle point. Only sets a flag on
-    // the queue - it never posts a task and never touches the GPU. The flag is consumed by
-    // should_sleep() on the start_loop() thread, which is only reached after callback_update_slots()
-    // has returned, so a release can never happen underneath an in-flight decode - not even via the
-    // worker thread of yield_to_queue(), which declines everything except read-only tasks.
-    void vram_warden_loop() {
+    // true while another process waits for the token and we still hold it. Called by should_sleep()
+    // on the start_loop() thread, under mutex_tasks, and only when the queue is idle - so a release
+    // can never happen underneath an in-flight decode, and there is no event to latch: the answer
+    // comes from live kernel state at the moment the loop asks.
+    bool vram_should_yield() {
 #if !defined(_WIN32)
-        char buf[4096];
-        while (vram_warden_run.load()) {
-            struct pollfd pfd { vram_inotify_fd, POLLIN, 0 };
-            int pr = poll(&pfd, 1, 500); // 500ms so we periodically re-check the run flag
-            if (pr <= 0) {
-                continue;
-            }
-            ssize_t n = read(vram_inotify_fd, buf, sizeof(buf));
-            if (n <= 0) {
-                continue;
-            }
-            bool foreign_ring = false;
-            for (char * p = buf; p < buf + n; ) {
-                struct inotify_event * ev = (struct inotify_event *) p;
-                if (ev->len > 0 && vram_pid_str != ev->name) {
-                    foreign_ring = true; // someone else wants the GPU
-                }
-                p += sizeof(struct inotify_event) + ev->len;
-            }
-            if (foreign_ring && !vram_cold) {
-                queue_tasks.request_yield();
-            }
+        if (!vram_flock || vram_cold || vram_want_probe_fd < 0) {
+            return false; // we hold no token to give away
         }
+        // never blocks: LOCK_NB only, and we drop the probe lock at once if we win it
+        if (flock(vram_want_probe_fd, LOCK_EX | LOCK_NB) == 0) {
+            flock(vram_want_probe_fd, LOCK_UN);
+            return false; // nobody wants the GPU
+        }
+        return errno == EWOULDBLOCK || errno == EAGAIN;
+#else
+        return false;
 #endif
     }
 
-    // ring the doorbell so the current token holder releases promptly
-    void vram_ring_doorbell() {
+    // register as a waiter, so the current token holder sees us and yields. Shared, so several
+    // waiters coexist. The kernel drops it if we die, so it can never strand the holder.
+    void vram_want_acquire() {
 #if !defined(_WIN32)
-        if (!vram_flock || vram_doorbell_dir.empty()) {
+        if (vram_want_fd < 0 || vram_want_held) {
             return;
         }
-        const std::string f = vram_doorbell_dir + "/" + vram_pid_str;
-        int fd = open(f.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0666);
-        if (fd >= 0) {
-            ssize_t w = write(fd, "1", 1);
-            (void) w;
-            close(fd);
+        while (flock(vram_want_fd, LOCK_SH) != 0) {
+            if (errno != EINTR) { // our signal handlers run without SA_RESTART
+                SRV_WRN("VRAM arbiter: cannot take want.lock (errno %d)\n", errno);
+                return;
+            }
         }
+        vram_want_held = true;
+#endif
+    }
+
+    // stop asking for the token. Must run on every path out of vram_take_token(), or the holder
+    // yields forever.
+    void vram_want_release() {
+#if !defined(_WIN32)
+        if (vram_want_fd < 0 || !vram_want_held) {
+            return;
+        }
+        flock(vram_want_fd, LOCK_UN);
+        vram_want_held = false;
+#endif
+    }
+
+    // take the shared VRAM token: register on want.lock, then wait for token.lock. want.lock is
+    // level-triggered state, so the holder still learns of us if it is loading or restoring when we
+    // start to wait. No timeout: the holder can be mid-generation, and decoding without the token
+    // risks an OOM that kills both processes.
+    bool vram_take_token() {
+#if !defined(_WIN32)
+        if (!vram_flock) {
+            return true; // no cross-process lock: there is no token to take
+        }
+        vram_want_acquire();
+        bool logged = false;
+        while (flock(vram_lock_fd, LOCK_EX | LOCK_NB) != 0) {
+            if (errno == EWOULDBLOCK || errno == EAGAIN) {
+                if (!logged) {
+                    SRV_INF("%s", "VRAM arbiter: waiting for the GPU token\n");
+                    logged = true;
+                }
+                flock(vram_lock_fd, LOCK_EX); // blocking: the kernel grants it the moment the holder unlocks
+                continue; // re-check with LOCK_NB, which also succeeds on a lock we already own
+            }
+            if (errno == EINTR) {
+                continue; // our signal handlers run without SA_RESTART
+            }
+            SRV_WRN("VRAM arbiter: flock failed (errno %d)\n", errno);
+            vram_want_release();
+            return false;
+        }
+        vram_want_release();
+        return true;
+#else
+        return true;
 #endif
     }
 
@@ -1045,27 +1104,30 @@ private:
         if (!vram_only || !vram_cold) {
             return;
         }
-#if !defined(_WIN32)
-        if (vram_flock) {
-            vram_ring_doorbell();         // nudge the current holder to release
-            flock(vram_lock_fd, LOCK_EX); // blocks until the current holder goes cold
+        if (!vram_take_token()) {
+            SRV_WRN("%s", "VRAM arbiter: restoring weights without the GPU token\n");
         }
-#endif
         llama_context_restore_device(ctx_tgt);
         if (ctx_dft != nullptr) {
             llama_context_restore_device(ctx_dft);
         }
+        // NOTE: the multimodal (vision/audio) encoder is intentionally NOT restored here. It is
+        // brought into VRAM just-in-time before an image/audio encode (process_mtmd_chunk) and, in
+        // on-demand mode, released again right after - so a warm text-only model holds no encoder
+        // VRAM (that space is free for KV / expert cache). A cold->warm wake for a *text* request
+        // therefore leaves the encoder in RAM; an image request restores it at encode time.
+        // NOTE: keep this AFTER the restore, so the loop cannot give away the token while we still
+        // bring the weights up. We dropped want.lock when we took the token, so the next probe
+        // reports only the waiters that are still there.
         vram_cold = false;
     }
 
     void vram_share_shutdown() {
 #if !defined(_WIN32)
-        vram_warden_run = false;
-        if (vram_warden.joinable()) {
-            vram_warden.join();
-        }
-        if (vram_inotify_fd >= 0) { close(vram_inotify_fd); vram_inotify_fd = -1; }
-        if (vram_lock_fd    >= 0) { flock(vram_lock_fd, LOCK_UN); close(vram_lock_fd); vram_lock_fd = -1; }
+        vram_want_release();
+        if (vram_want_probe_fd >= 0) { close(vram_want_probe_fd); vram_want_probe_fd = -1; }
+        if (vram_want_fd       >= 0) { close(vram_want_fd);       vram_want_fd       = -1; }
+        if (vram_lock_fd       >= 0) { flock(vram_lock_fd, LOCK_UN); close(vram_lock_fd); vram_lock_fd = -1; }
 #endif
     }
 
@@ -1079,6 +1141,11 @@ private:
         if (ctx_dft != nullptr) {
             llama_context_release_device(ctx_dft, vram_evict_kv);
         }
+        // also release the multimodal (vision/audio) encoder weights (dead weight while cold);
+        // its read-only host shadow is captured once and rebuilt on wake by vram_ensure_warm()
+        if (mctx != nullptr) {
+            mtmd_release_device(mctx);
+        }
 #if !defined(_WIN32)
         if (vram_flock) {
             flock(vram_lock_fd, LOCK_UN);
@@ -1164,8 +1231,12 @@ private:
                 SRV_INF("%s", "server entering sleeping state (VRAM-only: releasing device weights, keeping KV cache)\n");
                 vram_go_cold();
             } else {
-                SRV_INF("%s", "server exiting sleeping state (VRAM-only: weights restored on next decode)\n");
-                // token is acquired and weights restored by vram_ensure_warm() before decode
+                SRV_INF("%s", "server exiting sleeping state (VRAM-only: restoring device weights/KV)\n");
+                // Restore NOW, on wake, before update_slots runs. update_slots touches the KV cache
+                // (e.g. SWA checkpoint creation reads it via ggml_backend_tensor_get) before the
+                // decode-time vram_ensure_warm(), so with KV eviction the KV must already be resident
+                // here or those reads hit a freed (null) buffer.
+                vram_ensure_warm();
             }
             sleeping = new_state;
             return;
@@ -1394,6 +1465,13 @@ private:
             init_opt.video_params.ffmpeg_bin_dir = params_base.video_ffmpeg_bin_dir.empty()
                                 ? nullptr : params_base.video_ffmpeg_bin_dir.c_str();
 
+            // on-demand encoder: keep the vision/audio encoder weights in RAM (out of VRAM) until an
+            // image/audio actually needs encoding, freeing that VRAM for KV / expert cache on the
+            // common text-only path. process_mtmd_chunk() brings the encoder in just-in-time.
+            if (getenv("LLAMA_MMPROJ_ONDEMAND") != nullptr) {
+                mtmd_release_device(mctx);
+            }
+
             if (params_base.ctx_shift) {
                 params_base.ctx_shift = false;
                 SRV_WRN("%s\n", "ctx_shift is not supported by multimodal, it will be disabled");
@@ -1638,9 +1716,9 @@ private:
             handle_sleeping_state(sleeping);
         });
 
-        // VRAM arbiter: start the doorbell warden. The load was coordinated (vram_acquire_for_load
-        // grabbed the token before uploading), so we stay warm holding the token and serve the first
-        // request without a re-warm; the warden releases us when another model rings the doorbell.
+        // VRAM arbiter: arm the yield probe. The load was coordinated (vram_acquire_for_load
+        // grabbed the token before uploading), so we stay warm holding the token and serve the
+        // first request without a re-warm; the loop releases it when another model waits for it.
         vram_share_init();
 
         metrics.init();
@@ -4464,9 +4542,9 @@ struct server_res_generator : server_res_spipe {
     server_res_generator(server_queue & queue_tasks, server_response & queue_results, int sleep_idle_seconds, bool bypass_sleep = false)
             : rd(queue_tasks, queue_results, HTTP_POLLING_SECONDS) {
         // fast path in case sleeping is disabled. Note: the VRAM arbiter (LLAMA_SLEEP_VRAM_ONLY)
-        // can put the server to sleep via the cross-process doorbell even when idle-sleep is
-        // disabled (sleep_idle_seconds < 0), so in that case requests must still wake it - otherwise
-        // a doorbell-slept server would hang, never returning from a request.
+        // can put the server to sleep to hand over the GPU token even when idle-sleep is disabled
+        // (sleep_idle_seconds < 0), so in that case requests must still wake it - otherwise an
+        // arbiter-slept server would hang, never returning from a request.
         static const bool vram_arbiter = getenv("LLAMA_SLEEP_VRAM_ONLY") != nullptr;
         bypass_sleep |= (sleep_idle_seconds < 0 && !vram_arbiter);
         if (!bypass_sleep) {

tools/server/server-queue.cpp

@@ -114,27 +114,19 @@ void server_queue::pop_deferred_task(int id_slot) {
 
 void server_queue::wait_until_no_sleep() {
     std::unique_lock<std::mutex> lock(mutex_tasks);
-    if (!sleeping) {
-        return;
-    } else {
+    // re-ask on every wake: the loop clears req_stop_sleeping on the way into sleep, so a loop that
+    // goes back to sleep before we run would strand us here forever
+    while (sleeping) {
         if (!req_stop_sleeping) {
             QUE_DBG("%s", "requesting to stop sleeping\n");
             req_stop_sleeping = true;
-            condition_tasks.notify_one(); // only main thread is waiting on this
+            condition_tasks.notify_all(); // other threads may wait on this too
         }
         QUE_DBG("%s", "waiting until no sleep\n");
-        condition_tasks.wait(lock, [&]{
-            return !sleeping;
-        });
+        condition_tasks.wait(lock);
     }
 }
 
-void server_queue::request_yield() {
-    std::unique_lock<std::mutex> lock(mutex_tasks);
-    yield_requested = true;
-    condition_tasks.notify_all();
-}
-
 void server_queue::terminate() {
     std::unique_lock<std::mutex> lock(mutex_tasks);
     running = false;
@@ -292,11 +284,17 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
     worker.yielding = false;
     worker.thread = std::thread([this]() { worker_loop(); });
 
-    constexpr auto max_wait_time = std::chrono::seconds(1);
+    // the arbiter predicate is read from live state, so this timeout bounds how fast we notice a
+    // process waiting for the VRAM token
+    const auto max_wait_time = should_yield_cb ? std::chrono::milliseconds(10) : std::chrono::milliseconds(1000);
+    // after a wake, keep the VRAM for at least this long: the request that woke us is not in the
+    // queue yet, so yielding at once would only send it through another wake
+    constexpr int64_t yield_grace_ms = 100;
+    int64_t time_last_wake = 0;
     auto should_sleep = [&]() -> bool {
         // caller must hold mutex_tasks
-        if (yield_requested) {
-            return true; // another process rang the VRAM doorbell - release now
+        if (should_yield_cb && ggml_time_ms() - time_last_wake >= yield_grace_ms && should_yield_cb()) {
+            return true; // another process waits for the VRAM token - release now
         }
         if (idle_sleep_ms < 0) {
             return false;
@@ -336,15 +334,16 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
             if (should_sleep()) {
                 QUE_INF("%s", "entering sleeping state\n");
                 sleeping = true;
-                yield_requested = false; // consumed
                 // Call order cb0 -> cb1 -> cb{N}
                 for (auto & cb : callback_sleeping_state) {
                     cb(true);
                 }
                 req_stop_sleeping = false;
-                // wait until we are requested to exit sleeping state
+                // wait until we are requested to exit sleeping state, or a task arrives: post() only
+                // notifies, so a task queued right after wait_until_no_sleep() saw us awake must be
+                // able to wake us by itself, else it waits here for an unrelated request
                 condition_tasks.wait(lock, [&]{
-                    return (!running || req_stop_sleeping);
+                    return (!running || req_stop_sleeping || !queue_tasks.empty());
                 });
                 if (!running) { // may changed during sleep
                     break; // terminate
@@ -357,12 +356,13 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
                 }
                 sleeping = false;
                 time_last_task = ggml_time_ms();
+                time_last_wake = time_last_task;
                 condition_tasks.notify_all(); // notify wait_until_no_sleep()
                 break; // process new tasks
             } else {
-                // wait for new tasks, a VRAM yield request, or timeout for checking sleeping condition
+                // wait for new tasks, or timeout for checking sleeping condition
                 bool res = condition_tasks.wait_for(lock, max_wait_time, [&]{
-                    return (!queue_tasks.empty() || !running || yield_requested);
+                    return (!queue_tasks.empty() || !running);
                 });
                 if (res && !queue_tasks.empty()) {
                     break; // new task arrived or terminate
@@ -370,7 +370,7 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
                 if (!running) {
                     break;
                 }
-                // otherwise (timeout or yield request), loop again to re-check should_sleep
+                // otherwise (timeout), loop again to re-check should_sleep
             }
         }
     }

tools/server/server-queue.h

@@ -18,7 +18,6 @@ private:
     bool running  = false;
     bool sleeping = false;
     bool req_stop_sleeping = false;
-    bool yield_requested = false; // set by request_yield() when another process wants the VRAM token
     int64_t time_last_task = 0;
 
     // queues
@@ -46,6 +45,7 @@ private:
     std::function<bool(server_task &&, bool)> callback_new_task;
     std::function<void(void)>                 callback_update_slots;
     std::vector<std::function<void(bool)>>    callback_sleeping_state;
+    std::function<bool()>                     should_yield_cb;
 
 public:
     ~server_queue() { worker_stop(); }
@@ -70,11 +70,6 @@ public:
     // returns immediately if not sleeping
     void wait_until_no_sleep();
 
-    // request that the loop go to sleep (release VRAM) as soon as it is idle - called from the
-    // VRAM-arbiter warden thread when another process rings the doorbell for the GPU token.
-    // Thread-safe; wakes the loop so it releases promptly instead of at the next idle poll.
-    void request_yield();
-
     bool is_sleeping() {
         std::unique_lock<std::mutex> lock(mutex_tasks);
         return sleeping;
@@ -133,6 +128,14 @@ public:
         callback_update_slots = std::move(callback);
     }
 
+    // Register a predicate asking the loop to sleep (release VRAM) as soon as it is idle - used by
+    // the VRAM arbiter to check whether another process waits for the GPU token.
+    // Called on the start_loop() thread while holding mutex_tasks, so it must not block or post
+    // tasks. While it is set, the idle wait polls faster, so the loop reacts without a notify.
+    void on_should_yield(std::function<bool()> callback) {
+        should_yield_cb = std::move(callback);
+    }
+
     // Register callback for sleeping state change; multiple callbacks are allowed
     // for example: register order cb0, cb1, cb2
     // entering sleep: queue.sleeping = true --> cb0(true) --> cb1(true) --> cb2(true)