server: on-demand mmproj - free encoder VRAM on the text path

The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.

When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.

Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.

Assisted-by: Claude

README.md

@@ -12,6 +12,77 @@
 
 LLM inference in C/C++
 
+## ⚡ This fork — Fable's MoE-offload prefill optimizations
+
+Two **opt-in** optimizations for large MoE models whose experts are offloaded to system RAM
+(`--n-cpu-moe`), found and implemented by Fable. Both are **off by default**, toggled via
+environment variables, and produce **token-identical** output to mainline.
+
+| Env var | What it does |
+| --- | --- |
+| `GGML_CUDA_REGISTER_HOST=1` | Page-locks (pins) the mmap'd CPU expert weights so host->device copies go straight over DMA instead of through the driver's hidden bounce buffer (~6-7 -> ~20 GB/s). Works on CUDA and Vulkan (also honored as `GGML_VK_REGISTER_HOST`). Note: it is a presence check, so `=0` still enables it. |
+| `GGML_SCHED_PREFETCH_EXPERTS=1` | Prefetches each layer's experts on a second stream, so the weight uploads overlap compute instead of stalling the GPU. **CUDA only** - on the Vulkan backend the second backend instance shares one device queue, giving no overlap, so this regresses (see Vulkan note below). Leave it off on Vulkan. |
+
+### Benchmark
+
+Measured on an **RTX 3060 12GB** with **Qwen3.6-35B-A3B** (`--n-cpu-moe 26`), prompt-processing at 2048 (`MODEL` = path to your `.gguf`):
+
+```bash
+# baseline (patches off):
+./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 26 -p 2048 -n 0 -r 5 -b 2048 -ub 2048
+
+# patched (both optimizations on):
+GGML_CUDA_REGISTER_HOST=1 GGML_SCHED_PREFETCH_EXPERTS=1 \
+./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 26 -p 2048 -n 0 -r 5 -b 2048 -ub 2048
+```
+
+Result: **~1143 → ~1880 t/s** prefill (**+64%**) — same GPU, same settings, token-identical.
+
+Branches: [`fable5/host-register`](https://github.com/thecodacus/llama.cpp/tree/fable5/host-register) (pinning only) · [`fable5/prefetch-experts`](https://github.com/thecodacus/llama.cpp/tree/fable5/prefetch-experts) (both — this branch).
+
+### Vulkan (older AMD, e.g. RX 580 / Polaris)
+
+On the Vulkan backend the CUDA-oriented flags above behave differently, and this fork adds a
+Polaris-specific flash-attention fix. Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16,
+no fp16, no matrix cores) with **Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`:
+
+- **Flash-attention `mask_opt` is now enabled for GCN large head sizes (this fork's own change).**
+  Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips
+  fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on
+  a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B
+  (head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical.
+- **For a long-running server, load with `--no-mmap`, not pinning.** `GGML_CUDA_REGISTER_HOST=1`
+  (pinning) gives ~+17% in an isolated `llama-bench` run, but in a server the RADV host-pointer
+  import fails and the fallback pre-stage buffer allocation fails for large / co-resident models, so
+  it silently reverts to slow staging (and can trip warnings/OOM). `--no-mmap` (weights in RAM) is
+  both faster and clean there. Pinning is still fine for one-off `llama-bench` numbers.
+- **`GGML_SCHED_PREFETCH_EXPERTS=1` regresses - do not use it** on Vulkan (its second backend shares
+  one device queue, so uploads never overlap compute).
+- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp).
+- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short
+  prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that
+  VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`).
+
+Prefill throughput (isolated `llama-bench`, pinned unless noted):
+
+| Config | pp2048 (t/s) |
+| --- | ---: |
+| baseline, unpinned, `ncmoe 40` | ~252 |
+| pinned, `ncmoe 40` | ~294 |
+| pinned, `ncmoe 28` | ~308 |
+| server default (`--no-mmap`, `ncmoe 40`) | ~285 |
+| + `mask_opt`, @ 32k context | **+12%** |
+
+**Recommended RX 580 / Polaris serving command** (per model):
+
+```bash
+llama-server -hf <repo>:<quant> --no-mmap -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1
+```
+
+Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high
+for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so
+`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path.
+
 ## Recent API changes
 
 - [Changelog for `libllama` API](https://github.com/ggml-org/llama.cpp/issues/9289)

common/arg.cpp

@@ -2575,15 +2575,17 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
     ).set_env("LLAMA_ARG_CPU_MOE"));
     add_opt(common_arg(
         {"-ncmoe", "--n-cpu-moe"}, "N",
-        "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU",
-        [](common_params & params, int value) {
-            if (value < 0) {
+        "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU; "
+        "fractional N offloads part of the boundary layer at tensor granularity",
+        [](common_params & params, const std::string & value) {
+            const double n = std::stod(value);
+            if (n < 0) {
                 throw std::invalid_argument("invalid value");
             }
-            for (int i = 0; i < value; ++i) {
+            for (const std::string & re : llm_ffn_exps_cpu_block_regexes(n)) {
                 // keep strings alive and avoid leaking memory by storing them in a static vector
                 static std::list<std::string> buft_overrides;
-                buft_overrides.push_back(llm_ffn_exps_block_regex(i));
+                buft_overrides.push_back(re);
                 params.tensor_buft_overrides.push_back({buft_overrides.back().c_str(), ggml_backend_cpu_buffer_type()});
             }
         }

common/common.h

@@ -1080,6 +1080,28 @@ inline llama_model_tensor_buft_override llm_ffn_exps_cpu_override() {
     return { LLM_FFN_EXPS_REGEX, ggml_backend_cpu_buffer_type() };
 }
 
+// ATSInfer-style tensor-granularity static placement of MoE expert weights.
+// Offloads the expert weights of the first floor(n) layers to the CPU, plus a
+// subset of the boundary layer's three expert tensors for the fractional part.
+// Expert tensors are dropped from the GPU in ascending performance-density
+// order (gate, then up), keeping the higher-value down_proj resident longest.
+inline std::vector<std::string> llm_ffn_exps_cpu_block_regexes(double n_cpu_moe) {
+    std::vector<std::string> regexes;
+    const int n_full = n_cpu_moe > 0 ? (int) n_cpu_moe : 0;
+    for (int i = 0; i < n_full; ++i) {
+        regexes.push_back(llm_ffn_exps_block_regex(i));
+    }
+    const int k = (int) ((n_cpu_moe - n_full) * 3.0 + 0.5);
+    if (k >= 3) {
+        regexes.push_back(llm_ffn_exps_block_regex(n_full));
+    } else if (k == 2) {
+        regexes.push_back(string_format("blk\\.%d\\.ffn_(gate|up)_(ch|)exps", n_full));
+    } else if (k == 1) {
+        regexes.push_back(string_format("blk\\.%d\\.ffn_gate_(ch|)exps", n_full));
+    }
+    return regexes;
+}
+
 //
 // training utils
 //

ggml/include/ggml-backend.h

@@ -104,6 +104,11 @@ extern "C" {
     GGML_API enum ggml_status ggml_backend_graph_compute      (ggml_backend_t backend, struct ggml_cgraph * cgraph);
     GGML_API enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph);
 
+    // Free transient/scratch device memory the backend holds outside of any allocated buffer
+    // (compute preallocations, staging buffers). No-op if the backend does not implement it.
+    // The backend remains usable; scratch is reallocated lazily on the next compute.
+    GGML_API void ggml_backend_free_scratch(ggml_backend_t backend);
+
     // NOTE: will be removed, use device version instead
     GGML_API bool ggml_backend_supports_op(ggml_backend_t backend, const struct ggml_tensor * op);
     GGML_API bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft);

ggml/src/ggml-backend-impl.h

@@ -137,6 +137,11 @@ extern "C" {
 
         // (optional) sort/optimize the nodes in the graph
         void                      (*graph_optimize)    (ggml_backend_t backend, struct ggml_cgraph * cgraph);
+
+        // (optional) free transient/scratch device memory the backend holds outside of any buffer
+        // (e.g. compute preallocations and staging buffers). The backend stays usable; the scratch
+        // is reallocated lazily on the next compute. Used to shrink an idle model's device footprint.
+        void                      (*free_scratch)       (ggml_backend_t backend);
     };
 
     struct ggml_backend {

ggml/src/ggml-backend.cpp

@@ -420,6 +420,15 @@ void ggml_backend_synchronize(ggml_backend_t backend) {
     backend->iface.synchronize(backend);
 }
 
+void ggml_backend_free_scratch(ggml_backend_t backend) {
+    GGML_ASSERT(backend);
+    if (backend->iface.free_scratch == NULL) {
+        return;
+    }
+
+    backend->iface.free_scratch(backend);
+}
+
 ggml_backend_graph_plan_t ggml_backend_graph_plan_create(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
     GGML_ASSERT(backend);
     GGML_ASSERT(backend->iface.graph_plan_create != NULL);
@@ -761,6 +770,10 @@ static bool ggml_is_view_op(enum ggml_op op) {
 #define GGML_SCHED_MAX_COPIES 4
 #endif
 
+#ifndef GGML_SCHED_MAX_PREFETCH_SLOTS
+#define GGML_SCHED_MAX_PREFETCH_SLOTS 8
+#endif
+
 struct ggml_backend_sched_split {
     int backend_id;
     int i_start;
@@ -818,6 +831,19 @@ struct ggml_backend_sched {
 
     bool op_offload;
 
+    // full-tensor prefetch of offloaded MUL_MAT_ID weights (GGML_SCHED_PREFETCH_EXPERTS)
+    // with a large batch virtually every expert is used, so the routing ids are not worth
+    // waiting for; uploads run through a second backend instance on the same device so
+    // they overlap compute, alternating between two staging slots
+    bool prefetch_experts;
+    ggml_backend_t prefetch_backend;
+    int prefetch_n_slots;
+    ggml_backend_buffer_t prefetch_slots[GGML_SCHED_MAX_PREFETCH_SLOTS];
+    ggml_backend_event_t prefetch_ready[GGML_SCHED_MAX_PREFETCH_SLOTS];
+    ggml_backend_event_t prefetch_free[GGML_SCHED_MAX_PREFETCH_SLOTS];
+    bool prefetch_used[GGML_SCHED_MAX_PREFETCH_SLOTS];
+    int prefetch_cur;
+
     int debug;
 
     // used for debugging graph reallocations [GGML_SCHED_DEBUG_REALLOC]
@@ -1538,6 +1564,94 @@ static bool ggml_backend_sched_alloc_splits(ggml_backend_sched_t sched) {
     return true;
 }
 
+static void ggml_backend_sched_prefetch_disable(ggml_backend_sched_t sched, ggml_backend_t split_backend) {
+    sched->prefetch_experts = false;
+    if (sched->prefetch_backend) {
+        ggml_backend_synchronize(split_backend);
+        ggml_backend_synchronize(sched->prefetch_backend);
+    }
+    for (int i = 0; i < sched->prefetch_n_slots; i++) {
+        ggml_backend_buffer_free(sched->prefetch_slots[i]);
+        sched->prefetch_slots[i] = NULL;
+        sched->prefetch_used[i] = false;
+    }
+}
+
+// slots are sized once for the largest offloaded expert tensor in the current graph so
+// that they never need to grow mid-eval
+static size_t ggml_backend_sched_prefetch_max_size(ggml_backend_sched_t sched) {
+    size_t max_size = 0;
+    for (int split_id = 0; split_id < sched->n_splits; split_id++) {
+        struct ggml_backend_sched_split * split = &sched->splits[split_id];
+        if (split->graph.n_nodes == 0 || split->graph.nodes[0]->op != GGML_OP_MUL_MAT_ID) {
+            continue;
+        }
+        for (int input_id = 0; input_id < split->n_inputs; input_id++) {
+            const ggml_tensor * input = split->inputs[input_id];
+            if (input->buffer &&
+                ggml_backend_buffer_get_usage(input->buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS &&
+                ggml_backend_buffer_is_host(input->buffer)) {
+                max_size = std::max(max_size, ggml_nbytes(input));
+            }
+        }
+    }
+    return max_size;
+}
+
+static bool ggml_backend_sched_prefetch_init(ggml_backend_sched_t sched, ggml_backend_t split_backend, size_t size) {
+    if (sched->prefetch_backend == NULL) {
+        ggml_backend_dev_t dev = split_backend->device;
+        ggml_backend_dev_props props;
+        ggml_backend_dev_get_props(dev, &props);
+        if (!props.caps.async || !props.caps.events) {
+            sched->prefetch_experts = false;
+            return false;
+        }
+        sched->prefetch_backend = ggml_backend_dev_init(dev, NULL);
+        if (sched->prefetch_backend == NULL) {
+            sched->prefetch_experts = false;
+            return false;
+        }
+        for (int i = 0; i < sched->prefetch_n_slots; i++) {
+            sched->prefetch_ready[i] = ggml_backend_event_new(dev);
+            sched->prefetch_free[i]  = ggml_backend_event_new(dev);
+            if (sched->prefetch_ready[i] == NULL || sched->prefetch_free[i] == NULL) {
+                sched->prefetch_experts = false;
+                return false;
+            }
+        }
+    }
+
+    size = std::max(size, ggml_backend_sched_prefetch_max_size(sched));
+
+    ggml_backend_buffer_type_t buft = ggml_backend_get_default_buffer_type(split_backend);
+    for (int i = 0; i < sched->prefetch_n_slots; i++) {
+        if (sched->prefetch_slots[i] == NULL || ggml_backend_buffer_get_size(sched->prefetch_slots[i]) < size) {
+            // allocate before freeing so a failure leaves the old slot intact
+            ggml_backend_buffer_t new_buf = ggml_backend_buft_alloc_buffer(buft, size);
+            if (new_buf == NULL) {
+                // overlap needs at least 2 slots, otherwise run with what fits
+                if (i >= 2 && sched->prefetch_slots[0] != NULL &&
+                    ggml_backend_buffer_get_size(sched->prefetch_slots[0]) >= size) {
+                    sched->prefetch_n_slots = i;
+                    sched->prefetch_cur = 0;
+                    return true;
+                }
+                ggml_backend_sched_prefetch_disable(sched, split_backend);
+                return false;
+            }
+            if (sched->prefetch_slots[i] != NULL) {
+                ggml_backend_synchronize(split_backend);
+                ggml_backend_synchronize(sched->prefetch_backend);
+                ggml_backend_buffer_free(sched->prefetch_slots[i]);
+            }
+            sched->prefetch_slots[i] = new_buf;
+            sched->prefetch_used[i] = false;
+        }
+    }
+    return true;
+}
+
 static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t sched) {
     GGML_ASSERT(sched);
     struct ggml_backend_sched_split * splits = sched->splits;
@@ -1550,6 +1664,10 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
         struct ggml_backend_sched_split * split = &splits[split_id];
         int split_backend_id = split->backend_id;
         ggml_backend_t split_backend = sched->backends[split_backend_id];
+        int split_prefetch_slot = -1;
+        ggml_tensor * prefetch_input_cpy = NULL;
+        ggml_backend_buffer_t prefetch_saved_buffer = NULL;
+        void * prefetch_saved_data = NULL;
 
         // copy the input tensors to the split backend
         for (int input_id = 0; input_id < split->n_inputs; input_id++) {
@@ -1566,6 +1684,41 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
                 }
                 ggml_backend_tensor_copy(input, input_cpy);
             } else {
+                // with a large batch virtually every expert is used, so instead of waiting
+                // for the routing ids, upload the full tensor through the prefetch backend
+                // and let the copy overlap compute of the previous split
+                if (sched->prefetch_experts && !sched->callback_eval && split_prefetch_slot == -1 && split->graph.n_nodes > 0) {
+                    ggml_tensor * node = split->graph.nodes[0];
+                    if (ggml_backend_buffer_get_usage(input->buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS &&
+                        ggml_backend_buffer_is_host(input->buffer) &&
+                        node->op == GGML_OP_MUL_MAT_ID && node->src[0] == input_cpy) {
+                        const ggml_tensor * ids = node->src[2];
+                        const int64_t n_expert = input->ne[2];
+                        if (ids->ne[0]*ids->ne[1] >= 2*n_expert &&
+                            ggml_backend_sched_prefetch_init(sched, split_backend, ggml_nbytes(input))) {
+                            const int slot = sched->prefetch_cur;
+                            sched->prefetch_cur = (sched->prefetch_cur + 1) % sched->prefetch_n_slots;
+                            // wait for the previous user of this slot to finish computing
+                            if (sched->prefetch_used[slot]) {
+                                ggml_backend_event_wait(sched->prefetch_backend, sched->prefetch_free[slot]);
+                            }
+                            // point the staging copy at the slot only for the duration of
+                            // this split, so a fallback to the regular path on a later
+                            // eval can never see a dangling slot pointer
+                            prefetch_input_cpy    = input_cpy;
+                            prefetch_saved_buffer = input_cpy->buffer;
+                            prefetch_saved_data   = input_cpy->data;
+                            input_cpy->buffer = sched->prefetch_slots[slot];
+                            input_cpy->data   = ggml_backend_buffer_get_base(sched->prefetch_slots[slot]);
+                            ggml_backend_tensor_set_async(sched->prefetch_backend, input_cpy, input->data, 0, ggml_nbytes(input));
+                            ggml_backend_event_record(sched->prefetch_ready[slot], sched->prefetch_backend);
+                            ggml_backend_event_wait(split_backend, sched->prefetch_ready[slot]);
+                            split_prefetch_slot = slot;
+                            continue;
+                        }
+                    }
+                }
+
                 // wait for the split backend to finish using the input before overwriting it
                 if (sched->events[split_backend_id][sched->cur_copy] != NULL) {
                     ggml_backend_event_wait(split_backend, sched->events[split_backend_id][sched->cur_copy]);
@@ -1676,6 +1829,13 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
 
         if (!sched->callback_eval) {
             enum ggml_status ec = ggml_backend_graph_compute_async(split_backend, &split->graph);
+            if (split_prefetch_slot != -1) {
+                // the kernels have captured the slot address at launch, safe to restore
+                ggml_backend_event_record(sched->prefetch_free[split_prefetch_slot], split_backend);
+                sched->prefetch_used[split_prefetch_slot] = true;
+                prefetch_input_cpy->buffer = prefetch_saved_buffer;
+                prefetch_input_cpy->data   = prefetch_saved_data;
+            }
             if (ec != GGML_STATUS_SUCCESS) {
                 return ec;
             }
@@ -1788,6 +1948,15 @@ ggml_backend_sched_t ggml_backend_sched_new(
     sched->galloc = ggml_gallocr_new_n(sched->bufts, n_backends);
     sched->op_offload = op_offload;
 
+    // GGML_SCHED_PREFETCH_EXPERTS=1 enables the default slot count, higher values set it
+    // directly; more slots let uploads run further ahead of compute at the cost of one
+    // max-sized expert tensor of device memory per slot
+    const char * GGML_SCHED_PREFETCH_EXPERTS = getenv("GGML_SCHED_PREFETCH_EXPERTS");
+    const int prefetch_n_slots = GGML_SCHED_PREFETCH_EXPERTS ? atoi(GGML_SCHED_PREFETCH_EXPERTS) : 0;
+    sched->prefetch_experts = op_offload && prefetch_n_slots > 0;
+    // default of 3 covers the gate/up/down expert tensors of one MoE layer
+    sched->prefetch_n_slots = prefetch_n_slots <= 1 ? 3 : std::min(prefetch_n_slots, GGML_SCHED_MAX_PREFETCH_SLOTS);
+
     ggml_backend_sched_reset(sched);
 
     return sched;
@@ -1802,6 +1971,16 @@ void ggml_backend_sched_free(ggml_backend_sched_t sched) {
             ggml_backend_event_free(sched->events[b][c]);
         }
     }
+    if (sched->prefetch_backend) {
+        ggml_backend_synchronize(sched->prefetch_backend);
+        // the slot count may have been reduced after a failed allocation, free everything
+        for (int i = 0; i < GGML_SCHED_MAX_PREFETCH_SLOTS; i++) {
+            ggml_backend_event_free(sched->prefetch_ready[i]);
+            ggml_backend_event_free(sched->prefetch_free[i]);
+            ggml_backend_buffer_free(sched->prefetch_slots[i]);
+        }
+        ggml_backend_free(sched->prefetch_backend);
+    }
     ggml_gallocr_free(sched->galloc);
     ggml_free(sched->ctx);
     ggml_hash_set_free(&sched->hash_set);
@@ -1906,6 +2085,9 @@ void ggml_backend_sched_synchronize(ggml_backend_sched_t sched) {
     for (int i = 0; i < sched->n_backends; i++) {
         ggml_backend_synchronize(sched->backends[i]);
     }
+    if (sched->prefetch_backend) {
+        ggml_backend_synchronize(sched->prefetch_backend);
+    }
     if (!sched->is_alloc) {
         // if the graph is not already allocated, always use copy 0 after a synchronization
         // this ensures that during generation the same copy is used every time,

ggml/src/ggml-vulkan/ggml-vulkan.cpp

@@ -3283,6 +3283,7 @@ static vk_buffer ggml_vk_create_buffer(vk_device& device, size_t size, const std
             import_info.setPNext(&mem_flags_info);
             buf->device_memory = device->device.allocateMemory({ size, memory_type_idx, &import_info });
         } catch (const vk::SystemError& e) {
+            GGML_LOG_WARN("ggml_vulkan: host pointer memory import failed (%s)\n", e.what());
         }
     } else {
         for (auto it = req_flags_list.begin(); it != req_flags_list.end(); it++) {
@@ -8128,25 +8129,32 @@ static bool ggml_vk_buffer_write_2d_async(vk_context subctx, vk_buffer& dst, siz
     ggml_vk_host_get(dst->device, src, buf, buf_offset);
 
     if (buf != nullptr) {
-        // Memory is pinned, use as staging buffer
-        std::vector<vk::BufferCopy> slices(1);
-        if (width == spitch && width == dpitch) {
-            // Only do single write if stride is equal
-            slices[0].srcOffset = buf_offset;
-            slices[0].dstOffset = offset;
-            slices[0].size = width * height;
-        } else {
-            slices.resize(height);
-            for (size_t i = 0; i < height; i++) {
-                slices[i].srcOffset = buf_offset + i * spitch;
-                slices[i].dstOffset = offset + i * dpitch;
-                slices[i].size = width;
+        // extent of the read in pinned source memory; guard against tensors that
+        // straddle a pinned-chunk boundary (they fall back to staging below)
+        size_t src_extent = (width == spitch) ? (size_t) width * height
+                                              : (height > 0 ? (height - 1) * spitch + width : 0);
+        if (buf_offset + src_extent <= buf->size) {
+            // Memory is pinned, use as staging buffer
+            std::vector<vk::BufferCopy> slices(1);
+            if (width == spitch && width == dpitch) {
+                // Only do single write if stride is equal
+                slices[0].srcOffset = buf_offset;
+                slices[0].dstOffset = offset;
+                slices[0].size = width * height;
+            } else {
+                slices.resize(height);
+                for (size_t i = 0; i < height; i++) {
+                    slices[i].srcOffset = buf_offset + i * spitch;
+                    slices[i].dstOffset = offset + i * dpitch;
+                    slices[i].size = width;
+                }
             }
-        }
 
-        ggml_vk_sync_buffers(nullptr, subctx);
-        subctx->s->buffer->buf.copyBuffer(buf->buffer, dst->buffer, slices);
-        return true;
+            ggml_vk_sync_buffers(nullptr, subctx);
+            subctx->s->buffer->buf.copyBuffer(buf->buffer, dst->buffer, slices);
+            return true;
+        }
+        // straddles a chunk boundary: fall through to staging
     }
     VK_LOG_DEBUG("STAGING");
 
@@ -10584,8 +10592,10 @@ static void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx
     }
 
     // Only use mask opt when the mask is fairly large. This hasn't been tuned extensively.
+    // GCN with large head size (>= 256) benefits in high-context prefill (skipping the
+    // per-block mask add on fully-visible blocks dominates), so enable it there too.
     bool use_mask_opt = mask && nem1 >= 32 && nem0 * nem1 > 32768 && nem0 >= tuning_params.block_cols * 16
-                        && (ctx->device->architecture != vk_device_architecture::AMD_GCN || HSK > 256 || HSV > 256);
+                        && (ctx->device->architecture != vk_device_architecture::AMD_GCN || HSK >= 256 || HSV >= 256);
     vk_fa_pipeline_state fa_pipeline_state = get_fa_pipeline_state(ctx->device, tuning_params, HSK, HSV, aligned, f32acc,
                                                                    mask != nullptr, use_mask_opt, logit_softcap != 0, k->type, v->type);
 
@@ -12512,9 +12522,108 @@ static void ggml_vk_opt_step_sgd(ggml_backend_vk_context * ctx, vk_context& subc
     ggml_vk_op_f32<vk_op_push_constants>(ctx, subctx, src0, src1, src2, nullptr, dst, GGML_OP_OPT_STEP_SGD, { (uint32_t)n, 0, 0.0f, 0.0f, 0.0f, 0.0f });
 }
 
+// Fast path for concat along dim 0 where one source is stored "transposed"
+// (nb[1] == type_size, dim1 innermost) and the other source + dst are
+// contiguous along dim 0. The generic concat shader reads the transposed
+// source with a catastrophic uncoalesced stride; here we instead copy the
+// contiguous source with copy.comp and transpose the other source into the
+// matching dst sub-region with the tiled copy_transpose shader (shared-memory
+// transpose, coalesced read+write). Mirrors the precedent in
+// ggml_vk_cpy_to_contiguous: direct dispatch with custom push constants.
+static void ggml_vk_concat_transpose_fastpath(ggml_backend_vk_context * ctx, vk_context& subctx,
+                                              const ggml_tensor * ctg, const ggml_tensor * trp,
+                                              ggml_tensor * dst, uint32_t off_ctg, uint32_t off_trp) {
+    const uint32_t ts = ggml_type_size(dst->type);
+
+    vk_pipeline pipeline_cpy = (ts == 4) ? ctx->device->pipeline_cpy_f32_f32
+                                          : ctx->device->pipeline_cpy_f16_f16;
+    vk_pipeline pipeline_trp = (ts == 4) ? ctx->device->pipeline_cpy_transpose_32
+                                          : ctx->device->pipeline_cpy_transpose_16;
+
+    ggml_pipeline_request_descriptor_sets(ctx, pipeline_cpy, 1);
+    ggml_pipeline_request_descriptor_sets(ctx, pipeline_trp, 1);
+
+    vk_subbuffer ctg_buf = ggml_vk_tensor_subbuffer(ctx, ctg, true);
+    vk_subbuffer trp_buf = ggml_vk_tensor_subbuffer(ctx, trp, true);
+    vk_subbuffer dst_buf = ggml_vk_tensor_subbuffer(ctx, dst, true);
+
+    const uint32_t a_misalign_ctg = get_misalign_bytes(ctx, ctg) / ts;
+    const uint32_t a_misalign_trp = get_misalign_bytes(ctx, trp) / ts;
+    const uint32_t d_misalign     = get_misalign_bytes(ctx, dst) / ts;
+
+    // Dispatch A: contiguous copy of `ctg` into dst[off_ctg : off_ctg + ctg->ne[0], :]
+    if (ctg->ne[0] > 0) {
+        const uint32_t ne_ctg = (uint32_t) ggml_nelements(ctg);
+        vk_op_unary_push_constants pc = vk_op_unary_push_constants_init(ctg, dst, ne_ctg);
+        pc.ne10 = (uint32_t) ctg->ne[0];                 // only ctg's columns in dst
+        pc.misalign_offsets = (a_misalign_ctg << 16) | (d_misalign + off_ctg);
+        init_pushconst_fastdiv(pc);
+        std::array<uint32_t, 3> el = ne_ctg > 262144 ? std::array<uint32_t,3>{512, 512, CEIL_DIV(ne_ctg, 262144)}
+                                      : ne_ctg > 512  ? std::array<uint32_t,3>{512, CEIL_DIV(ne_ctg, 512), 1}
+                                                      : std::array<uint32_t,3>{ne_ctg, 1, 1};
+        ggml_vk_dispatch_pipeline(ctx, subctx, pipeline_cpy, { ctg_buf, dst_buf }, pc, el);
+    }
+
+    // Dispatch B: tiled transpose of `trp` into dst[off_trp : off_trp + trp->ne[0], :]
+    if (trp->ne[0] > 0) {
+        vk_op_unary_push_constants pc = vk_op_unary_push_constants_init(trp, dst, ggml_nelements(trp));
+        pc.ne10 = (uint32_t) trp->ne[0];                 // dst bound = trp columns (NOT dst->ne[0])
+        pc.misalign_offsets = (a_misalign_trp << 16) | (d_misalign + off_trp);
+        init_pushconst_fastdiv(pc);
+        std::array<uint32_t, 3> el = {
+            (uint32_t) CEIL_DIV(trp->ne[0], 32),
+            (uint32_t) CEIL_DIV(trp->ne[1], 32),
+            (uint32_t) (trp->ne[2] * trp->ne[3]),
+        };
+        el[0] = std::min(el[0], (uint32_t) ctx->device->properties.limits.maxComputeWorkGroupCount[0]);
+        el[1] = std::min(el[1], (uint32_t) ctx->device->properties.limits.maxComputeWorkGroupCount[1]);
+        el[2] = std::min(el[2], (uint32_t) ctx->device->properties.limits.maxComputeWorkGroupCount[2]);
+        ggml_vk_dispatch_pipeline(ctx, subctx, pipeline_trp, { trp_buf, dst_buf }, pc, el);
+    }
+
+    ggml_vk_sync_buffers(ctx, subctx);
+}
+
 static void ggml_vk_concat(ggml_backend_vk_context * ctx, vk_context& subctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) {
     int * op_params = (int *)dst->op_params;
 
+    // Fast path: concat along dim 0 with one source "transposed" (nb[1]==type_size,
+    // dim1 innermost) and the other source + dst contiguous along dim 0. The generic
+    // shader reads the transposed source uncoalesced (~5.6ms on RX 580 vs ~130us for
+    // a coalesced copy); here we instead use a tiled shared-memory transpose.
+    auto src_transposed_2d = [&](const ggml_tensor * s) {
+        const uint32_t ts = ggml_type_size(s->type);
+        return s->nb[1] == ts                                              // dim1 innermost
+            && s->nb[0] == s->ne[1] * ts                                   // consistent 2D transpose
+            && s->ne[2] == 1 && s->ne[3] == 1;
+    };
+    const uint32_t dst_ts = ggml_type_size(dst->type);
+    const bool dim0_ok   = (op_params[0] == 0) && (dst->nb[0] == dst_ts);
+    const bool types_ok  = (dst_ts == 4 || dst_ts == 2)
+                        && (src0->type == src1->type && src0->type == dst->type)
+                        && (ggml_blck_size(dst->type) == 1);
+    const bool shapes_ok = (src0->ne[1] == src1->ne[1] && src1->ne[1] == dst->ne[1])
+                        && (src0->ne[2] == src1->ne[2] && src0->ne[2] == dst->ne[2])
+                        && (src0->ne[3] == src1->ne[3] && src0->ne[3] == dst->ne[3])
+                        && (src0->ne[0] + src1->ne[0] == dst->ne[0]);
+    // doffset is 16-bit for unary shaders; guard against overflow
+    const bool off_fits  = ((get_misalign_bytes(ctx, dst)/dst_ts + dst->ne[0])) < 0xFFFFu;
+
+    const bool s1_trans = dim0_ok && types_ok && shapes_ok && off_fits
+                        && src_transposed_2d(src1) && ggml_is_contiguous(src0);
+    const bool s0_trans = dim0_ok && types_ok && shapes_ok && off_fits
+                        && src_transposed_2d(src0) && ggml_is_contiguous(src1);
+
+    if (s1_trans || s0_trans) {
+        const ggml_tensor * ctg = s1_trans ? src0 : src1;     // contiguous source
+        const ggml_tensor * trp = s1_trans ? src1 : src0;     // transposed source
+        // dst offsets (in elements) where each source region begins
+        const uint32_t off_ctg = s1_trans ? 0u                       : (uint32_t) src1->ne[0];
+        const uint32_t off_trp = s1_trans ? (uint32_t) src0->ne[0]   : 0u;
+        ggml_vk_concat_transpose_fastpath(ctx, subctx, ctg, trp, dst, off_ctg, off_trp);
+        return;
+    }
+
     const uint32_t src0_type_size = ggml_type_size(src0->type);
     const uint32_t src1_type_size = ggml_type_size(src1->type);
     const uint32_t dst_type_size = ggml_type_size(dst->type);
@@ -15734,6 +15843,35 @@ static const char * ggml_backend_vk_name(ggml_backend_t backend) {
     return ctx->name.c_str();
 }
 
+// Free the compute-scratch device buffers (prealloc_* and the transfer staging buffer) without
+// tearing down the backend (pipelines, command pools, fences and device stay alive). These buffers
+// are reallocated lazily by ggml_vk_preallocate_buffers() on the next compute, so this just shrinks
+// an idle model's device footprint. Mirrors the buffer-freeing subset of ggml_vk_cleanup().
+static void ggml_backend_vk_free_scratch(ggml_backend_t backend) {
+    ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
+    VK_LOG_DEBUG("ggml_backend_vk_free_scratch(" << ctx->name << ")");
+
+    // discard any unsubmitted command buffer and wait for in-flight work before freeing
+    ctx->compute_ctx.reset();
+    ggml_vk_synchronize(ctx);
+
+    ggml_vk_destroy_buffer(ctx->prealloc_x);
+    ggml_vk_destroy_buffer(ctx->prealloc_y);
+    ggml_vk_destroy_buffer(ctx->prealloc_split_k);
+    ggml_vk_destroy_buffer(ctx->prealloc_add_rms_partials);
+    ggml_vk_destroy_buffer(ctx->sync_staging);
+
+    ctx->prealloc_y_last_pipeline_used = nullptr;
+    ctx->prealloc_y_last_tensor_used = nullptr;
+    ctx->prealloc_y_last_decode_vector_staging = false;
+
+    ctx->prealloc_size_x = 0;
+    ctx->prealloc_size_y = 0;
+    ctx->prealloc_size_split_k = 0;
+    ctx->prealloc_size_add_rms_partials = 0;
+    ctx->prealloc_size_add_rms_partials_offset = 0;
+}
+
 static void ggml_backend_vk_free(ggml_backend_t backend) {
     ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
     VK_LOG_DEBUG("ggml_backend_vk_free(" << ctx->name << ")");
@@ -16578,7 +16716,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
         std::fill(ctx->query_nodes.begin(), ctx->query_nodes.end(), nullptr);
         std::fill(ctx->query_node_idx.begin(), ctx->query_node_idx.end(), 0);
 
-        GGML_ASSERT(ctx->compute_ctx.expired());
+        // compute_ctx may hold async host uploads recorded before the graph; append the timestamp after them
         compute_ctx = ggml_vk_get_compute_ctx(ctx);
         ctx->query_idx = 0;
         compute_ctx->s->buffer->buf.writeTimestamp(vk::PipelineStageFlagBits::eAllCommands, ctx->query_pool, ctx->query_idx++);
@@ -17269,6 +17407,7 @@ static ggml_backend_i ggml_backend_vk_interface = {
     /* .event_record            = */ ggml_backend_vk_event_record,
     /* .event_wait              = */ ggml_backend_vk_event_wait,
     /* .graph_optimize          = */ ggml_vk_graph_optimize,
+    /* .free_scratch            = */ ggml_backend_vk_free_scratch,
 };
 
 static ggml_guid_t ggml_backend_vk_guid() {
@@ -18243,11 +18382,112 @@ static ggml_backend_dev_t ggml_backend_vk_reg_get_device(ggml_backend_reg_t reg,
     return devices[device];
 }
 
+// Import an mmap-backed host region as a Vulkan pinned buffer via
+// VK_EXT_external_memory_host so H2D uploads DMA straight from system RAM
+// instead of bouncing through the staging buffer + host memcpy. Mirrors the
+// GGML_CUDA_REGISTER_HOST path; populates device->pinned_memory, which the
+// existing pinned fast path in ggml_vk_buffer_write_2d_async looks up.
+//
+// A single Vulkan buffer cannot cover a whole multi-GB mmap (it is capped at
+// device->max_buffer_size), so the region is imported in page-aligned chunks.
+// ggml_vk_host_get resolves a tensor pointer to the chunk that contains it,
+// and ggml_vk_buffer_write_2d_async falls back to staging for any tensor that
+// straddles a chunk boundary, so correctness is preserved.
+static bool ggml_backend_vk_register_host_buffer(void * buffer, size_t size) {
+    if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr && getenv("GGML_VK_REGISTER_HOST") == nullptr) {
+        return false;
+    }
+    if (size == 0) {
+        return false;
+    }
+
+    bool success = false;
+    for (size_t i = 0; i < GGML_VK_MAX_DEVICES; i++) {
+        vk_device& device = vk_instance.devices[i];
+        if (!device || !device->external_memory_host || device->max_buffer_size == 0) {
+            continue;
+        }
+
+        const size_t align = device->min_imported_host_pointer_alignment;
+        size_t chunk = device->max_buffer_size & ~(align - 1);
+        if (chunk == 0) {
+            continue;
+        }
+
+        uint8_t * p = static_cast<uint8_t *>(buffer);
+        size_t remaining = size;
+        bool dev_success = false;
+        bool import_ok = true;   // flips to false once VK_EXT_external_memory_host import fails
+        while (remaining > 0) {
+            size_t cur = std::min(remaining, chunk);
+            vk_buffer buf;
+            if (import_ok) {
+                buf = ggml_vk_buffer_from_host_ptr(device, p, cur);
+            }
+            if (!buf || !buf->buffer) {
+                // Fallback for drivers that can't import file-backed mmap pages (e.g. RADV):
+                // make a one-time copy of the region into a host-visible Vulkan buffer. The GPU
+                // then DMAs straight from it every eval at full PCIe bandwidth, instead of paying
+                // a slow single-threaded pageable memcpy into the staging buffer each time.
+                import_ok = false;
+                buf = ggml_vk_create_buffer_check(device, cur,
+                    vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent | vk::MemoryPropertyFlagBits::eHostCached,
+                    vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent);
+                if (buf && buf->buffer && buf->ptr) {
+                    memcpy(buf->ptr, p, cur);
+                } else {
+                    break;
+                }
+            }
+            {
+                std::lock_guard<std::shared_mutex> guard(device->pinned_memory_mutex);
+                device->pinned_memory.emplace_back(p, cur, buf);
+            }
+            dev_success = true;
+            p += cur;
+            remaining -= cur;
+        }
+        if (dev_success) {
+            success = true;
+        }
+    }
+    return success;
+}
+
+static void ggml_backend_vk_unregister_host_buffer(void * buffer) {
+    for (size_t i = 0; i < GGML_VK_MAX_DEVICES; i++) {
+        vk_device& device = vk_instance.devices[i];
+        if (!device) {
+            continue;
+        }
+        std::lock_guard<std::shared_mutex> guard(device->pinned_memory_mutex);
+        for (auto it = device->pinned_memory.begin(); it != device->pinned_memory.end(); ++it) {
+            if (std::get<0>(*it) == buffer) {
+                vk_buffer buf = std::get<2>(*it);
+                device->pinned_memory.erase(it);
+                ggml_vk_destroy_buffer(buf);
+                break;
+            }
+        }
+    }
+}
+
+static void * ggml_backend_vk_reg_get_proc_address(ggml_backend_reg_t reg, const char * name) {
+    GGML_UNUSED(reg);
+    if (strcmp(name, "ggml_backend_register_host_buffer") == 0) {
+        return (void *) ggml_backend_vk_register_host_buffer;
+    }
+    if (strcmp(name, "ggml_backend_unregister_host_buffer") == 0) {
+        return (void *) ggml_backend_vk_unregister_host_buffer;
+    }
+    return nullptr;
+}
+
 static const struct ggml_backend_reg_i ggml_backend_vk_reg_i = {
     /* .get_name         = */ ggml_backend_vk_reg_get_name,
     /* .get_device_count = */ ggml_backend_vk_reg_get_device_count,
     /* .get_device       = */ ggml_backend_vk_reg_get_device,
-    /* .get_proc_address = */ NULL,
+    /* .get_proc_address = */ ggml_backend_vk_reg_get_proc_address,
 };
 
 ggml_backend_reg_t ggml_backend_vk_reg() {

include/llama.h

@@ -557,6 +557,16 @@ extern "C" {
 
     LLAMA_API const struct llama_model * llama_get_model   (const struct llama_context * ctx);
     LLAMA_API           llama_memory_t   llama_get_memory  (const struct llama_context * ctx);
+
+    // On-demand device (VRAM) residency: free the model's GPU weight buffers (keeping a host
+    // shadow so no reload is needed) to hand VRAM to another model, and rebuild them on demand.
+    // Any subsequent llama_decode auto-restores. If evict_kv is true, the KV cache device buffers
+    // are also evicted to a host shadow (for when the KV would not fit alongside the other model),
+    // at the cost of a D2H/H2D copy of the live KV each cycle; otherwise the KV stays resident.
+    // Intended for time-sharing a single GPU between several always-loaded models.
+    LLAMA_API void llama_context_release_device(struct llama_context * ctx, bool evict_kv);
+    LLAMA_API void llama_context_restore_device(struct llama_context * ctx);
+    LLAMA_API bool llama_context_weights_resident(const struct llama_context * ctx);
     LLAMA_API  enum llama_pooling_type   llama_pooling_type(const struct llama_context * ctx); // TODO: rename to llama_get_pooling_type
 
     LLAMA_API const struct llama_vocab * llama_model_get_vocab(const struct llama_model * model);

src/llama-context.cpp

@@ -728,6 +728,47 @@ void llama_context::synchronize() {
     t_compute_start_us = 0;
 }
 
+void llama_context::release_device(bool evict_kv) {
+    if (!model.weights_resident() && !(evict_kv && !kv_device_evicted)) {
+        return;
+    }
+    // ensure no compute is in flight before freeing the device buffers
+    synchronize();
+    model.release_device_weights();
+    if (evict_kv && memory && !kv_device_evicted) {
+        memory->release_device_buffers();
+        kv_device_evicted = true;
+        // also free the scheduler and its worst-case compute buffer (hundreds of MiB) so a cold
+        // model holds essentially no VRAM. Rebuilt lazily by sched_reserve() on restore.
+        sched.reset();
+        sched_need_reserve = true;
+        // finally, free each backend's own compute-scratch (Vulkan prealloc/staging buffers, which
+        // are owned by the backend and survive sched.reset()). Reallocated lazily on next compute.
+        for (auto & backend : backends) {
+            ggml_backend_free_scratch(backend.get());
+        }
+    }
+}
+
+void llama_context::restore_device() {
+    if (model.weights_resident() && !kv_device_evicted && sched) {
+        return;
+    }
+    model.restore_device_weights();
+    if (kv_device_evicted && memory) {
+        memory->restore_device_buffers();
+        kv_device_evicted = false;
+    }
+    // rebuild the scheduler + compute buffer if they were freed on release (evict_kv mode)
+    if (!sched) {
+        sched_reserve();
+    }
+    // make sure all weight/KV uploads have completed before any compute reads them
+    if (sched) {
+        ggml_backend_sched_synchronize(sched.get());
+    }
+}
+
 const llama_model & llama_context::get_model() const {
     return model;
 }
@@ -1705,6 +1746,12 @@ int llama_context::decode(const llama_batch & batch_inp) {
         return -1;
     }
 
+    // on-demand VRAM: if the weights were released while this model was idle, bring them back
+    // to the device before building the compute graph
+    if (!model.weights_resident()) {
+        restore_device();
+    }
+
     const auto & vocab   = model.vocab;
     const auto & hparams = model.hparams;
 
@@ -3223,17 +3270,21 @@ llama_memory_breakdown llama_context::memory_breakdown() const {
             ret[buft].context += size;
         }
     }
-    if (model.hparams.no_alloc) {
-        for (size_t i = 0; i < backends.size(); ++i) {
-            ggml_backend_t             backend = backends[i].get();
-            ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
-            ret[buft].compute += backend_buf_exp_size[i];
-        }
-    } else {
-        for (const auto & backend_ptr : backends) {
-            ggml_backend_t             backend = backend_ptr.get();
-            ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
-            ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
+    // the scheduler (and its compute buffers) may have been freed while the model is cold
+    // (on-demand VRAM eviction, see release_device); it contributes no compute VRAM then.
+    if (sched) {
+        if (model.hparams.no_alloc) {
+            for (size_t i = 0; i < backends.size(); ++i) {
+                ggml_backend_t             backend = backends[i].get();
+                ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
+                ret[buft].compute += backend_buf_exp_size[i];
+            }
+        } else {
+            for (const auto & backend_ptr : backends) {
+                ggml_backend_t             backend = backend_ptr.get();
+                ggml_backend_buffer_type_t buft    = ggml_backend_sched_get_buffer_type(sched.get(), backend);
+                ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
+            }
         }
     }
     return ret;
@@ -3674,6 +3725,18 @@ void llama_synchronize(llama_context * ctx) {
     ctx->synchronize();
 }
 
+void llama_context_release_device(llama_context * ctx, bool evict_kv) {
+    ctx->release_device(evict_kv);
+}
+
+void llama_context_restore_device(llama_context * ctx) {
+    ctx->restore_device();
+}
+
+bool llama_context_weights_resident(const llama_context * ctx) {
+    return ctx->get_model().weights_resident();
+}
+
 float * llama_get_logits(llama_context * ctx) {
     ctx->synchronize();
 

src/llama-context.h

@@ -57,6 +57,14 @@ struct llama_context {
 
     void synchronize();
 
+    // on-demand device (VRAM) residency: free / rebuild the model's GPU weight buffers so that
+    // VRAM can be time-shared with another model. release_device() synchronizes first; decode()
+    // auto-restores when it finds the weights have been released. If evict_kv is set, the KV cache
+    // device buffers are also evicted to a host shadow (for when the KV would not fit alongside the
+    // other model) - this adds a D2H/H2D copy of the live KV on each cycle.
+    void release_device(bool evict_kv = false);
+    void restore_device();
+
     const llama_model   & get_model()   const;
     const llama_cparams & get_cparams() const;
 
@@ -345,6 +353,9 @@ private:
 
     bool sched_need_reserve = true;
 
+    // on-demand device residency: true while the KV cache device buffers have been evicted to host
+    bool kv_device_evicted = false;
+
     ggml_backend_t backend_cpu = nullptr;
     std::vector<ggml_backend_ptr> backends;
 

src/llama-kv-cache-iswa.cpp

@@ -272,6 +272,17 @@ void llama_kv_cache_iswa::state_read(llama_io_read_i & io, llama_seq_id seq_id,
     kv_swa->state_read(io, seq_id, flags);
 }
 
+void llama_kv_cache_iswa::release_device_buffers() {
+    kv_base->release_device_buffers();
+    kv_swa->release_device_buffers();
+}
+
+bool llama_kv_cache_iswa::restore_device_buffers() {
+    bool ok = kv_base->restore_device_buffers();
+    ok = kv_swa->restore_device_buffers() && ok;
+    return ok;
+}
+
 llama_kv_cache * llama_kv_cache_iswa::get_base() const {
     return kv_base.get();
 }

src/llama-kv-cache-iswa.h

@@ -83,6 +83,9 @@ public:
     void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
     void state_read (llama_io_read_i  & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
 
+    void release_device_buffers() override;
+    bool restore_device_buffers() override;
+
     //
     // llama_kv_cache_iswa specific API
     //

src/llama-kv-cache.cpp

@@ -371,7 +371,9 @@ void llama_kv_cache::clear(bool data) {
 
     if (data) {
         for (auto & [_, buf] : ctxs_bufs) {
-            ggml_backend_buffer_clear(buf.get(), 0);
+            if (buf) { // may be null if evicted for on-demand VRAM sharing
+                ggml_backend_buffer_clear(buf.get(), 0);
+            }
         }
     }
 }
@@ -681,6 +683,9 @@ llama_pos llama_kv_cache::seq_pos_max(llama_seq_id seq_id) const {
 std::map<ggml_backend_buffer_type_t, size_t> llama_kv_cache::memory_breakdown() const {
     std::map<ggml_backend_buffer_type_t, size_t> ret;
     for (const auto & [ctx, buf] : ctxs_bufs) {
+        if (!buf) { // may be null if evicted for on-demand VRAM sharing
+            continue;
+        }
         ggml_backend_buffer_type_t buft = ggml_backend_buffer_get_type(buf.get());
 
         if (hparams.no_alloc) {
@@ -1801,12 +1806,88 @@ size_t llama_kv_cache::total_size() const {
     size_t size = 0;
 
     for (const auto & [_, buf] : ctxs_bufs) {
-        size += ggml_backend_buffer_get_size(buf.get());
+        if (buf) { // may be null if evicted for on-demand VRAM sharing
+            size += ggml_backend_buffer_get_size(buf.get());
+        }
     }
 
     return size;
 }
 
+void llama_kv_cache::release_device_buffers() {
+    // NOTE: the caller must have synchronized the backend so nothing references these buffers.
+    // The KV cache is read-write, so its host shadow is (re)captured fresh on every release.
+    if (dev_released) {
+        return;
+    }
+    dev_shadows.assign(ctxs_bufs.size(), device_buffer_shadow{});
+    size_t freed = 0;
+    for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
+        ggml_context * ctx = ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = ctxs_bufs[i].second.get();
+        if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
+            continue; // only real device (VRAM) buffers are evictable
+        }
+        auto & sh = dev_shadows[i];
+        sh.releasable = true;
+        sh.buft = ggml_backend_buffer_get_type(buf);
+
+        // capture live contents compactly in stable iteration order (skip views, which alias a base)
+        size_t total = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src == nullptr) { total += ggml_nbytes(t); }
+        }
+        sh.data.resize(total);
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_get(t, sh.data.data() + off, 0, n);
+            off += n;
+        }
+
+        freed += ggml_backend_buffer_get_size(buf);
+        ctxs_bufs[i].second.reset(); // free the device buffer
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            t->buffer = nullptr;
+            t->data   = nullptr;
+        }
+    }
+    dev_released = true;
+    if (freed > 0) {
+        LLAMA_LOG_INFO("%s: released %.2f MiB of KV cache from device\n", __func__, freed / 1024.0 / 1024.0);
+    }
+}
+
+bool llama_kv_cache::restore_device_buffers() {
+    if (!dev_released) {
+        return true;
+    }
+    for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
+        auto & sh = dev_shadows[i];
+        if (!sh.releasable) {
+            continue;
+        }
+        ggml_context * ctx = ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, sh.buft);
+        if (buf == nullptr) {
+            LLAMA_LOG_ERROR("%s: failed to reallocate KV cache device buffer (out of VRAM?)\n", __func__);
+            return false;
+        }
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_set(t, sh.data.data() + off, 0, n);
+            off += n;
+        }
+        ctxs_bufs[i].second.reset(buf);
+    }
+    dev_released = false;
+    dev_shadows.clear(); // recaptured on next release
+    return true;
+}
+
 size_t llama_kv_cache::size_k_bytes() const {
     size_t size_k_bytes = 0;
 

src/llama-kv-cache.h

@@ -149,6 +149,10 @@ public:
     void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
     void state_read (llama_io_read_i  & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
 
+    // on-demand device (VRAM) residency (see llama_memory_i)
+    void release_device_buffers() override;
+    bool restore_device_buffers() override;
+
     //
     // llama_kv_cache specific API
     //
@@ -265,6 +269,16 @@ private:
     // ggml contexts for the KV cache along with the allocated backend buffers:
     std::vector<std::pair<ggml_context_ptr, ggml_backend_buffer_ptr>> ctxs_bufs;
 
+    // on-demand device eviction: host shadow of each device buffer's live contents (re-captured on
+    // every release since the KV is read-write), plus the buffer type needed to reallocate it.
+    struct device_buffer_shadow {
+        ggml_backend_buffer_type_t buft = nullptr;
+        bool releasable = false;      // device (VRAM) buffer that was freed
+        std::vector<uint8_t> data;    // captured tensor bytes (compact, iteration order)
+    };
+    std::vector<device_buffer_shadow> dev_shadows; // parallel to ctxs_bufs
+    bool dev_released = false;
+
     // the current index from where we start searching for a free slot in the ring buffer of KV cells (see find_slot())
     // note: this is not part of the KV state and it's only used to speed-up the find_slot() method
     std::vector<uint32_t> v_heads;

src/llama-memory-hybrid.cpp

@@ -201,6 +201,18 @@ void llama_memory_hybrid::state_read(llama_io_read_i & io, llama_seq_id seq_id,
     mem_recr->state_read(io, seq_id, flags);
 }
 
+void llama_memory_hybrid::release_device_buffers() {
+    // evict both the attention KV (grows with context) and the recurrent/SSM state
+    mem_attn->release_device_buffers();
+    mem_recr->release_device_buffers();
+}
+
+bool llama_memory_hybrid::restore_device_buffers() {
+    bool ok = mem_attn->restore_device_buffers();
+    ok = mem_recr->restore_device_buffers() && ok;
+    return ok;
+}
+
 llama_kv_cache * llama_memory_hybrid::get_mem_attn() const {
     return mem_attn.get();
 }

src/llama-memory-hybrid.h

@@ -76,6 +76,9 @@ public:
     void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
     void state_read (llama_io_read_i  & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0)       override;
 
+    void release_device_buffers() override;
+    bool restore_device_buffers() override;
+
     //
     // llama_memory_hybrid specific API
     //

src/llama-memory-recurrent.cpp

@@ -140,7 +140,9 @@ void llama_memory_recurrent::clear(bool data) {
 
     if (data) {
         for (auto & [_, buf] : ctxs_bufs) {
-            ggml_backend_buffer_clear(buf.get(), 0);
+            if (buf) { // may be null if evicted for on-demand VRAM sharing
+                ggml_backend_buffer_clear(buf.get(), 0);
+            }
         }
     }
 
@@ -399,6 +401,7 @@ void llama_memory_recurrent::set_rs_idx(llama_seq_id seq_id, uint32_t idx) {
 std::map<ggml_backend_buffer_type_t, size_t> llama_memory_recurrent::memory_breakdown() const {
     std::map<ggml_backend_buffer_type_t, size_t> ret;
     for (const auto & [_, buf] : ctxs_bufs) {
+        if (!buf) { continue; } // may be null if evicted for on-demand VRAM sharing
         ret[ggml_backend_buffer_get_type(buf.get())] += ggml_backend_buffer_get_size(buf.get());
     }
     return ret;
@@ -700,12 +703,87 @@ bool llama_memory_recurrent::get_can_shift() const {
 size_t llama_memory_recurrent::total_size() const {
     size_t size = 0;
     for (const auto & [_, buf] : ctxs_bufs) {
-        size += ggml_backend_buffer_get_size(buf.get());
+        if (buf) { // may be null if evicted for on-demand VRAM sharing
+            size += ggml_backend_buffer_get_size(buf.get());
+        }
     }
 
     return size;
 }
 
+void llama_memory_recurrent::release_device_buffers() {
+    // Same mechanism as llama_kv_cache: the recurrent (SSM/conv) state is read-write, so its host
+    // shadow is (re)captured on every release. The caller must have synchronized the backend.
+    if (dev_released) {
+        return;
+    }
+    dev_shadows.assign(ctxs_bufs.size(), device_buffer_shadow{});
+    size_t freed = 0;
+    for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
+        ggml_context * ctx = ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = ctxs_bufs[i].second.get();
+        if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
+            continue;
+        }
+        auto & sh = dev_shadows[i];
+        sh.releasable = true;
+        sh.buft = ggml_backend_buffer_get_type(buf);
+
+        size_t total = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src == nullptr) { total += ggml_nbytes(t); }
+        }
+        sh.data.resize(total);
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_get(t, sh.data.data() + off, 0, n);
+            off += n;
+        }
+
+        freed += ggml_backend_buffer_get_size(buf);
+        ctxs_bufs[i].second.reset();
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            t->buffer = nullptr;
+            t->data   = nullptr;
+        }
+    }
+    dev_released = true;
+    if (freed > 0) {
+        LLAMA_LOG_INFO("%s: released %.2f MiB of recurrent state from device\n", __func__, freed / 1024.0 / 1024.0);
+    }
+}
+
+bool llama_memory_recurrent::restore_device_buffers() {
+    if (!dev_released) {
+        return true;
+    }
+    for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
+        auto & sh = dev_shadows[i];
+        if (!sh.releasable) {
+            continue;
+        }
+        ggml_context * ctx = ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, sh.buft);
+        if (buf == nullptr) {
+            LLAMA_LOG_ERROR("%s: failed to reallocate recurrent device buffer (out of VRAM?)\n", __func__);
+            return false;
+        }
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_set(t, sh.data.data() + off, 0, n);
+            off += n;
+        }
+        ctxs_bufs[i].second.reset(buf);
+    }
+    dev_released = false;
+    dev_shadows.clear();
+    return true;
+}
+
 size_t llama_memory_recurrent::size_r_bytes() const {
     size_t size_r_bytes = 0;
 

src/llama-memory-recurrent.h

@@ -66,6 +66,10 @@ public:
     void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
     void state_read (llama_io_read_i  & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
 
+    // on-demand device (VRAM) residency (see llama_memory_i)
+    void release_device_buffers() override;
+    bool restore_device_buffers() override;
+
     uint32_t head = 0; // the location where the batch will be placed in the cache (see find_slot())
     uint32_t size = 0; // total number of cells, shared across all sequences
     uint32_t used = 0; // used cells (i.e. at least one seq_id)
@@ -121,6 +125,16 @@ private:
     // ggml contexts for the KV cache along with the allocated backend buffers:
     std::vector<std::pair<ggml_context_ptr, ggml_backend_buffer_ptr>> ctxs_bufs;
 
+    // on-demand device eviction (see release_device_buffers): host shadow of each device buffer's
+    // live contents (recaptured on every release since the recurrent state is read-write)
+    struct device_buffer_shadow {
+        ggml_backend_buffer_type_t buft = nullptr;
+        bool releasable = false;
+        std::vector<uint8_t> data;
+    };
+    std::vector<device_buffer_shadow> dev_shadows; // parallel to ctxs_bufs
+    bool dev_released = false;
+
     size_t total_size() const;
 
     size_t size_r_bytes() const;

src/llama-memory.h

@@ -124,6 +124,16 @@ struct llama_memory_i {
 
     virtual void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const = 0;
     virtual void state_read (llama_io_read_i  & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) = 0;
+
+    //
+    // on-demand device (VRAM) residency
+    //
+
+    // Free this memory's device (VRAM) buffers to a host shadow and rebuild them on demand, to
+    // hand VRAM to another model while keeping the cached data (no re-prefill). The caller must
+    // have synchronized the backend first. Default: no-op (the memory stays resident).
+    virtual void release_device_buffers() {}
+    virtual bool restore_device_buffers() { return true; }
 };
 
 using llama_memory_ptr = std::unique_ptr<llama_memory_i>;

src/llama-mmap.cpp

@@ -618,13 +618,47 @@ struct llama_mmap::impl {
 };
 
 llama_mmap::llama_mmap(struct llama_file * file, size_t prefetch, bool numa) : pimpl(std::make_unique<impl>(file, prefetch, numa)) {}
-llama_mmap::~llama_mmap() = default;
+
+llama_mmap::~llama_mmap() {
+    // unpin before the pages are unmapped by the impl destructor
+    if (host_reg_addr && host_unreg_fn) {
+        host_unreg_fn(host_reg_addr);
+    }
+}
 
 size_t llama_mmap::size() const { return pimpl->size; }
 void * llama_mmap::addr() const { return pimpl->addr; }
 
 void llama_mmap::unmap_fragment(size_t first, size_t last) { pimpl->unmap_fragment(first, last); }
 
+size_t llama_mmap::register_host(size_t first, size_t last, bool (*reg_fn)(void *, size_t), void (*unreg_fn)(void *)) {
+#ifdef _POSIX_MAPPED_FILES
+    if (host_reg_addr || !reg_fn || !unreg_fn || last <= first) {
+        return 0;
+    }
+
+    // expand outward to the page boundaries retained by unmap_fragment
+    const size_t page_size = sysconf(_SC_PAGESIZE);
+    first = first & ~(page_size - 1);
+    last  = (last + page_size - 1) & ~(page_size - 1);
+
+    void * reg_addr = (uint8_t *) pimpl->addr + first;
+    if (!reg_fn(reg_addr, last - first)) {
+        return 0;
+    }
+
+    host_reg_addr  = reg_addr;
+    host_unreg_fn  = unreg_fn;
+    return last - first;
+#else
+    GGML_UNUSED(first);
+    GGML_UNUSED(last);
+    GGML_UNUSED(reg_fn);
+    GGML_UNUSED(unreg_fn);
+    return 0;
+#endif
+}
+
 #if defined(_POSIX_MEMLOCK_RANGE) || defined(_WIN32)
 const bool llama_mmap::SUPPORTED  = true;
 #else

src/llama-mmap.h

@@ -50,11 +50,19 @@ struct llama_mmap {
 
     void unmap_fragment(size_t first, size_t last);
 
+    // pin the pages backing [first, last) with a backend allocator for faster H2D copies,
+    // unpinned in the destructor before the pages are unmapped
+    // returns the number of bytes registered, 0 on failure
+    size_t register_host(size_t first, size_t last, bool (*reg_fn)(void *, size_t), void (*unreg_fn)(void *));
+
     static const bool SUPPORTED;
 
 private:
     struct impl;
     std::unique_ptr<impl> pimpl;
+
+    void * host_reg_addr = nullptr;
+    void (*host_unreg_fn)(void *) = nullptr;
 };
 
 struct llama_mlock {

src/llama-model-loader.cpp

@@ -1677,6 +1677,15 @@ bool llama_model_loader::load_all_data(
     if (size_done >= size_data) {
         // unmap offloaded tensors and metadata
         if (use_mmap) {
+            // pin the pages backing the weights kept in system memory for faster H2D copies
+            bool (*reg_fn)(void *, size_t) = nullptr;
+            void (*unreg_fn)(void *) = nullptr;
+            for (size_t i = 0; i < ggml_backend_dev_count() && !reg_fn; i++) {
+                ggml_backend_reg_t reg = ggml_backend_dev_backend_reg(ggml_backend_dev_get(i));
+                reg_fn   = (bool (*)(void *, size_t)) ggml_backend_reg_get_proc_address(reg, "ggml_backend_register_host_buffer");
+                unreg_fn = (void (*)(void *))         ggml_backend_reg_get_proc_address(reg, "ggml_backend_unregister_host_buffer");
+            }
+
             for (uint32_t idx = 0; idx < mappings.size(); idx++) {
                 const auto & mmap_used = mmaps_used.at(idx);
                 auto & mapping = mappings.at(idx);
@@ -1684,6 +1693,13 @@ bool llama_model_loader::load_all_data(
                 if (mmap_used.second != 0) {
                     mapping->unmap_fragment(mmap_used.second, mapping->size());
                 }
+                if (mmap_used.second > mmap_used.first) {
+                    size_t n_registered = mapping->register_host(mmap_used.first, mmap_used.second, reg_fn, unreg_fn);
+                    if (n_registered > 0) {
+                        LLAMA_LOG_INFO("%s: pinned %.2f MiB of mapped model memory for faster H2D transfers\n",
+                                __func__, n_registered / 1024.0 / 1024.0);
+                    }
+                }
             }
         }
         if (progress_callback) {

src/llama-model.cpp

@@ -1015,6 +1015,16 @@ struct llama_model::impl {
     // contexts where the model tensors metadata is stored as well as the corresponding buffers:
     std::vector<std::pair<ggml_context_ptr, std::vector<ggml_backend_buffer_ptr>>> ctxs_bufs;
 
+    // on-demand device (VRAM) weight residency: per-ctxs_bufs slot describing whether the
+    // device weight buffer can be freed and rebuilt from a host shadow (see release/restore_device_weights)
+    struct weight_release_slot {
+        ggml_backend_buffer_type_t buft = nullptr; // buffer type to reallocate on restore
+        bool releasable = false;                   // single-buffer alloc-path device (VRAM) buffer
+        bool resident   = true;                    // currently allocated on device
+        std::vector<uint8_t> shadow;               // host copy, captured lazily on first release
+    };
+    std::vector<weight_release_slot> weight_release; // parallel to ctxs_bufs
+
     buft_list_t cpu_buft_list;
     std::map<ggml_backend_dev_t, buft_list_t> gpu_buft_list;
 
@@ -1608,6 +1618,23 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
 
         pimpl->ctxs_bufs.emplace_back(std::move(ctx_ptr), std::move(bufs));
 
+        // record on-demand release metadata (parallel to ctxs_bufs). Only the single-buffer
+        // alloc path on a device (non-host) buffer can be freed and rebuilt from a host shadow;
+        // the mmap buffer_from_host_ptr path and host (CPU) buffers are left resident.
+        {
+            llama_model::impl::weight_release_slot slot;
+            const bool mmap_host_ptr_path = ml.use_mmap && use_mmap_buffer && buffer_from_host_ptr_supported && is_default_buft;
+            auto & last_bufs = pimpl->ctxs_bufs.back().second;
+            if (!mmap_host_ptr_path && last_bufs.size() == 1) {
+                ggml_backend_buffer_t b = last_bufs[0].get();
+                if (b != nullptr && !ggml_backend_buffer_is_host(b)) {
+                    slot.releasable = true;
+                    slot.buft = buft;
+                }
+            }
+            pimpl->weight_release.push_back(std::move(slot));
+        }
+
         ctx_buf_maps.emplace_back(ctx, buf_map);
     }
 
@@ -1662,6 +1689,97 @@ ggml_tensor * llama_model_base::create_tensor(llama_model_loader & ml, const LLM
         tn, ne, flags);
 }
 
+void llama_model::release_device_weights() const {
+    // NOTE: the caller is responsible for synchronizing the backend scheduler first, so that no
+    // compute is in flight referencing these buffers when they are freed.
+    size_t freed = 0;
+    for (size_t i = 0; i < pimpl->ctxs_bufs.size(); ++i) {
+        auto & slot = pimpl->weight_release[i];
+        if (!slot.releasable || !slot.resident) {
+            continue;
+        }
+        ggml_context * ctx = pimpl->ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = pimpl->ctxs_bufs[i].second[0].get();
+        const size_t sz = ggml_backend_buffer_get_size(buf);
+
+        // lazily capture a host shadow of the weights (read-only, so captured only once).
+        // Store tensor bytes compactly in stable iteration order (independent of buffer layout /
+        // alignment padding) so restore does not depend on the re-allocated offsets matching.
+        // View tensors alias their base, so they are skipped (restored implicitly via their base).
+        if (slot.shadow.empty()) {
+            size_t total = 0;
+            for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+                if (t->view_src != nullptr) continue;
+                total += ggml_nbytes(t);
+            }
+            slot.shadow.resize(total);
+            size_t off = 0;
+            for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+                if (t->view_src != nullptr) continue;
+                const size_t n = ggml_nbytes(t);
+                ggml_backend_tensor_get(t, slot.shadow.data() + off, 0, n);
+                off += n;
+            }
+        }
+
+        // free the device (VRAM) buffer and clear the now-dangling tensor pointers so that
+        // ggml_backend_alloc_ctx_tensors_from_buft reallocates them cleanly on restore
+        pimpl->ctxs_bufs[i].second[0].reset();
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            t->buffer = nullptr;
+            t->data   = nullptr;
+        }
+        slot.resident = false;
+        freed += sz;
+    }
+    if (freed > 0) {
+        LLAMA_LOG_INFO("%s: released %.2f MiB of device weights\n", __func__, freed / 1024.0 / 1024.0);
+    }
+}
+
+bool llama_model::restore_device_weights() const {
+    size_t restored = 0;
+    for (size_t i = 0; i < pimpl->ctxs_bufs.size(); ++i) {
+        auto & slot = pimpl->weight_release[i];
+        if (!slot.releasable || slot.resident) {
+            continue;
+        }
+        ggml_context * ctx = pimpl->ctxs_bufs[i].first.get();
+        ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, slot.buft);
+        if (buf == nullptr) {
+            LLAMA_LOG_ERROR("%s: failed to reallocate device weight buffer (out of VRAM?)\n", __func__);
+            return false;
+        }
+        ggml_backend_buffer_set_usage(buf, GGML_BACKEND_BUFFER_USAGE_WEIGHTS);
+
+        // re-upload weights from the compact host shadow, using the same stable iteration order
+        // and view-skipping as the capture above
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
+            if (t->view_src != nullptr) continue;
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_set(t, slot.shadow.data() + off, 0, n);
+            off += n;
+        }
+        pimpl->ctxs_bufs[i].second[0].reset(buf);
+        slot.resident = true;
+        restored += ggml_backend_buffer_get_size(buf);
+    }
+    if (restored > 0) {
+        LLAMA_LOG_INFO("%s: restored %.2f MiB of device weights\n", __func__, restored / 1024.0 / 1024.0);
+    }
+    return true;
+}
+
+bool llama_model::weights_resident() const {
+    for (const auto & slot : pimpl->weight_release) {
+        if (slot.releasable && !slot.resident) {
+            return false;
+        }
+    }
+    return true;
+}
+
 std::string llama_model::arch_name() const {
     return llm_arch_name(arch);
 }
@@ -1714,6 +1832,11 @@ std::map<ggml_backend_buffer_type_t, size_t> llama_model::memory_breakdown() con
             ret[buft] += ggml_backend_alloc_ctx_tensors_from_buft_size(ctx.get(), buft);
         } else {
             for (const auto & buf : bufs) {
+                // buf may be null if the device weights were released for on-demand VRAM sharing
+                // (see release_device_weights); skip it in the breakdown
+                if (!buf) {
+                    continue;
+                }
                 // GGML_ASSERT(ggml_backend_buffer_get_base(buf.get()) != nullptr); // multi_buffer does not have a defined base
                 ret[ggml_backend_buffer_get_type(buf.get())] += ggml_backend_buffer_get_size(buf.get());
             }

src/llama-model.h

@@ -663,6 +663,14 @@ struct llama_model {
 
     const struct ggml_tensor * get_tensor(const char * name) const;
 
+    // on-demand device (VRAM) weight residency: free the GPU weight buffers (keeping a host
+    // shadow) and rebuild them on demand. Used to time-share VRAM between models without
+    // reloading or losing the KV cache. release_device_weights() requires the caller to have
+    // synchronized the backend scheduler first.
+    void release_device_weights() const;
+    bool restore_device_weights() const;
+    bool weights_resident() const;
+
     float get_rope_freq_base (const llama_cparams & cparams, int il) const;
     float get_rope_freq_scale(const llama_cparams & cparams, int il) const;
 

tools/mtmd/clip.cpp

@@ -158,6 +158,13 @@ struct clip_ctx {
     ggml_backend_t backend_cpu = nullptr;
     ggml_backend_buffer_ptr buf;
 
+    // on-demand device (VRAM) residency: the vision/audio encoder weights are read-only, so a host
+    // shadow is captured once and the device buffer can be freed while the model is cold, then
+    // rebuilt on wake (mirrors llama_model::release_device_weights). See clip_release_device().
+    ggml_backend_buffer_type_t dev_buft = nullptr;
+    std::vector<uint8_t>       dev_shadow;
+    bool                       dev_released = false;
+
 
     int max_nodes = 8192;
     ggml_backend_sched_ptr sched;
@@ -3241,6 +3248,74 @@ void clip_free(clip_ctx * ctx) {
     delete ctx;
 }
 
+void clip_release_device(struct clip_ctx * ctx) {
+    if (ctx == nullptr || ctx->dev_released) {
+        return;
+    }
+    ggml_backend_buffer_t buf = ctx->buf.get();
+    if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
+        return; // CPU-backed encoder: nothing in VRAM to free
+    }
+    // ensure no encode is in flight before freeing the weights
+    if (ctx->sched) {
+        ggml_backend_sched_synchronize(ctx->sched.get());
+    }
+    ctx->dev_buft = ggml_backend_buffer_get_type(buf);
+
+    ggml_context * cd = ctx->ctx_data.get();
+    // capture the host shadow once (weights are read-only): compact, stable iteration order,
+    // skipping view tensors (which alias a base and are restored implicitly)
+    if (ctx->dev_shadow.empty()) {
+        size_t total = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+            if (t->view_src == nullptr) { total += ggml_nbytes(t); }
+        }
+        ctx->dev_shadow.resize(total);
+        size_t off = 0;
+        for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+            if (t->view_src != nullptr) { continue; }
+            const size_t n = ggml_nbytes(t);
+            ggml_backend_tensor_get(t, ctx->dev_shadow.data() + off, 0, n);
+            off += n;
+        }
+    }
+
+    // free the device buffer and clear the now-dangling tensor pointers so restore reallocates cleanly
+    ctx->buf.reset();
+    for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+        t->buffer = nullptr;
+        t->data   = nullptr;
+    }
+    ctx->dev_released = true;
+}
+
+bool clip_restore_device(struct clip_ctx * ctx) {
+    if (ctx == nullptr || !ctx->dev_released) {
+        return true;
+    }
+    ggml_context * cd = ctx->ctx_data.get();
+    ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(cd, ctx->dev_buft);
+    if (buf == nullptr) {
+        LOG_ERR("%s: failed to reallocate encoder device buffer (out of VRAM?)\n", __func__);
+        return false;
+    }
+    ggml_backend_buffer_set_usage(buf, GGML_BACKEND_BUFFER_USAGE_WEIGHTS);
+    size_t off = 0;
+    for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
+        if (t->view_src != nullptr) { continue; }
+        const size_t n = ggml_nbytes(t);
+        ggml_backend_tensor_set(t, ctx->dev_shadow.data() + off, 0, n);
+        off += n;
+    }
+    ctx->buf.reset(buf);
+    ctx->dev_released = false;
+    return true;
+}
+
+bool clip_weights_resident(const struct clip_ctx * ctx) {
+    return ctx == nullptr || !ctx->dev_released;
+}
+
 const char * clip_patch_merge_type(const struct clip_ctx * ctx) {
     return ctx->model.hparams.mm_patch_merge_type == PATCH_MERGE_SPATIAL_UNPAD ? "spatial_unpad" : "flat";
 }

tools/mtmd/clip.h

@@ -67,6 +67,12 @@ struct clip_init_result clip_init(const char * fname, struct clip_context_params
 
 void clip_free(struct clip_ctx * ctx);
 
+// on-demand device (VRAM) residency: free/rebuild the encoder weight buffer to shrink an idle
+// (cold) multimodal model's VRAM footprint. No-op for a CPU-backed encoder. See clip.cpp.
+void clip_release_device(struct clip_ctx * ctx);
+bool clip_restore_device(struct clip_ctx * ctx);
+bool clip_weights_resident(const struct clip_ctx * ctx);
+
 // TODO: should be enum, not string
 const char * clip_patch_merge_type(const struct clip_ctx * ctx);
 

tools/mtmd/mtmd.cpp

@@ -806,6 +806,24 @@ void mtmd_free(mtmd_context * ctx) {
     delete ctx;
 }
 
+void mtmd_release_device(mtmd_context * ctx) {
+    if (ctx == nullptr) {
+        return;
+    }
+    if (ctx->ctx_v) { clip_release_device(ctx->ctx_v); }
+    if (ctx->ctx_a) { clip_release_device(ctx->ctx_a); }
+}
+
+bool mtmd_restore_device(mtmd_context * ctx) {
+    if (ctx == nullptr) {
+        return true;
+    }
+    bool ok = true;
+    if (ctx->ctx_v) { ok = clip_restore_device(ctx->ctx_v) && ok; }
+    if (ctx->ctx_a) { ok = clip_restore_device(ctx->ctx_a) && ok; }
+    return ok;
+}
+
 struct mtmd_tokenizer {
     mtmd_context * ctx;
 

tools/mtmd/mtmd.h

@@ -127,6 +127,12 @@ MTMD_API mtmd_context * mtmd_init_from_file(const char * mmproj_fname,
 
 MTMD_API void mtmd_free(mtmd_context * ctx);
 
+// on-demand device (VRAM) residency: free / rebuild the vision+audio encoder weight buffers so an
+// idle (cold) multimodal model releases its encoder VRAM and reclaims it on wake. No-op for a
+// CPU-backed encoder. Restore returns false if reallocation failed (out of VRAM).
+MTMD_API void mtmd_release_device(mtmd_context * ctx);
+MTMD_API bool mtmd_restore_device(mtmd_context * ctx);
+
 // whether we need to set non-causal mask before llama_decode
 // if chunk is nullptr, we assume the default case where chunk is an image chunk
 MTMD_API bool mtmd_decode_use_non_causal(const mtmd_context * ctx, const mtmd_input_chunk * chunk);

tools/server/server-context.cpp

@@ -25,6 +25,8 @@
 #include <filesystem>
 #include <utility>
 #include <fstream>
+#include <thread>
+#include <atomic>
 
 // fix problem with std::min and std::max
 #if defined(_WIN32)
@@ -35,6 +37,16 @@
 #include <windows.h>
 #endif
 
+// POSIX file locking + inotify doorbell for the cross-process VRAM arbiter (see vram_share_* below)
+#if !defined(_WIN32)
+#include <sys/file.h>
+#include <sys/stat.h>
+#include <sys/inotify.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <poll.h>
+#endif
+
 using json = nlohmann::ordered_json;
 
 constexpr int HTTP_POLLING_SECONDS = 1;
@@ -768,7 +780,36 @@ struct server_slot {
         // TODO @ngxson : move this log line to debug when it become more stable
         SLT_TRC(*this, "encoding mtmd batch from idx = %zu, n_chunks = %d\n", idx, n_added);
 
+        // Bring the vision/audio encoder into VRAM just for this encode. In on-demand mode the
+        // encoder is normally kept in RAM (its VRAM freed for KV / expert cache). If it does not
+        // fit, evict the LLM backbone weights first: they are NOT needed while the encoder runs (the
+        // decode that uses them happens afterwards) and their host shadow is read-only, so this is a
+        // cheap free (no D2H) and leaves the KV cache / prompt cache untouched.
+        static const bool mmproj_ondemand = getenv("LLAMA_MMPROJ_ONDEMAND") != nullptr;
+        bool weights_evicted = false;
+        if (mctx && !mtmd_restore_device(mctx)) {
+            if (mmproj_ondemand) {
+                llama_context_release_device(ctx_tgt, /* evict_kv = */ false); // free backbone, keep KV
+                weights_evicted = true;
+            }
+            if (!mtmd_restore_device(mctx)) {
+                if (weights_evicted) { llama_context_restore_device(ctx_tgt); }
+                SLT_ERR(*this, "%s", "failed to bring the multimodal encoder into VRAM for encoding\n");
+                return -1;
+            }
+        }
+
         res = mtmd_batch_encode(mbatch.get());
+
+        // release the encoder VRAM again (on-demand), then restore the backbone weights for decode.
+        // Order matters: free the encoder BEFORE re-uploading the weights so the peak stays within VRAM.
+        if (mctx && mmproj_ondemand) {
+            mtmd_release_device(mctx);
+        }
+        if (weights_evicted) {
+            llama_context_restore_device(ctx_tgt);
+        }
+
         if (res != 0) {
             SLT_ERR(*this, "failed to encode mtmd batch for chunk idx = %zu, res = %d\n", idx, res);
             return -1;
@@ -871,6 +912,8 @@ public:
     }
 
     ~server_context_impl() {
+        // stop the VRAM warden thread (and release the token) before tearing anything down
+        vram_share_shutdown();
         if (!sleeping) {
             // destroy() is already called when entering sleeping state
             // we don't call it again here to avoid double free
@@ -894,6 +937,198 @@ private:
     llama_model   * model_dft = nullptr;
     llama_context * ctx_dft   = nullptr;
 
+    // Cross-process VRAM arbiter: when LLAMA_SLEEP_VRAM_ONLY is set, several always-loaded
+    // llama-server processes time-share one GPU. A single flock() on <arena>/token.lock is the
+    // baton: "resident (weights in VRAM) iff I hold the token". A model warms up (locks + restores
+    // weights) right before it decodes, and goes cold (releases weights + unlocks) when it idles,
+    // so only one model holds VRAM at a time and the KV cache is never evicted (no re-prefill).
+    bool vram_only    = false;   // release-mode sleep active (LLAMA_SLEEP_VRAM_ONLY)
+    bool vram_evict_kv = false;  // also evict the KV cache to host on release (LLAMA_SLEEP_EVICT_KV)
+    bool vram_flock   = false;   // cross-process flock coordination available
+    std::atomic<bool> vram_cold{true}; // weights currently released (read by warden thread)
+    int  vram_lock_fd = -1;      // fd for <arena>/token.lock
+
+    // inotify "doorbell": a waiter that wants the VRAM token touches <arena>/doorbell/<pid>, which
+    // wakes the current holder's warden thread so it releases immediately instead of only on idle.
+    std::string       vram_doorbell_dir;
+    std::string       vram_pid_str;
+    int               vram_inotify_fd = -1;
+    std::thread       vram_warden;
+    std::atomic<bool> vram_warden_run{false};
+
+    // open the shared arena (flock token + doorbell dir). Idempotent; sets vram_only/vram_flock.
+    void vram_arena_open() {
+        if (getenv("LLAMA_SLEEP_VRAM_ONLY") == nullptr) {
+            return;
+        }
+        vram_only = true;
+        vram_evict_kv = getenv("LLAMA_SLEEP_EVICT_KV") != nullptr;
+#if !defined(_WIN32)
+        if (vram_lock_fd >= 0) {
+            return; // already open
+        }
+        const char * arena_env = getenv("LLAMA_VRAM_ARENA");
+        const std::string arena = arena_env ? arena_env : "/dev/shm/llama-vram";
+        mkdir(arena.c_str(), 0777);
+        const std::string lock_path = arena + "/token.lock";
+        vram_lock_fd = open(lock_path.c_str(), O_RDWR | O_CREAT, 0666);
+        if (vram_lock_fd >= 0) {
+            vram_flock = true;
+            vram_pid_str = std::to_string(getpid());
+            vram_doorbell_dir = arena + "/doorbell";
+            mkdir(vram_doorbell_dir.c_str(), 0777);
+            SRV_INF("VRAM arbiter: cross-process GPU sharing via %s (doorbell %s)\n",
+                    lock_path.c_str(), vram_doorbell_dir.c_str());
+        } else {
+            SRV_WRN("VRAM arbiter: cannot open %s, running VRAM-only sleep without cross-process lock\n", lock_path.c_str());
+        }
+#endif
+    }
+
+    // Acquire the VRAM token BEFORE uploading this model's weights, so any model currently resident
+    // on the GPU releases first and the load uploads into free VRAM instead of racing it (which
+    // could OOM, e.g. the 4B task model holding VRAM while a large model loads). Blocks until free.
+    // Called from load_model() right before common_init_from_params().
+    void vram_acquire_for_load() {
+        vram_arena_open();
+        if (!vram_only) {
+            return;
+        }
+#if !defined(_WIN32)
+        if (vram_flock) {
+            vram_ring_doorbell();         // nudge whoever is resident to release
+            flock(vram_lock_fd, LOCK_EX); // block until the GPU is free
+        }
+#endif
+        vram_cold = false; // we hold the token; weights will be resident after the upload
+    }
+
+    // Start the doorbell warden. Called from init() after the (coordinated) load. The model stays
+    // warm (holding the token acquired in vram_acquire_for_load) and serves its first request
+    // without a re-warm; the warden releases it when another model rings the doorbell.
+    void vram_share_init() {
+        vram_arena_open();
+        if (!vram_only) {
+            return;
+        }
+        vram_cold = false; // resident and holding the token after the coordinated load
+#if !defined(_WIN32)
+        if (vram_flock && vram_inotify_fd < 0) {
+            vram_inotify_fd = inotify_init1(IN_NONBLOCK);
+            if (vram_inotify_fd >= 0) {
+                inotify_add_watch(vram_inotify_fd, vram_doorbell_dir.c_str(), IN_CLOSE_WRITE);
+                vram_warden_run = true;
+                vram_warden = std::thread([this]{ vram_warden_loop(); });
+            }
+        }
+#endif
+    }
+
+    // warden thread: block on the doorbell; when another process rings (wants the token) and we
+    // currently hold it, ask the loop to yield (release) at its next idle point. Only touches the
+    // thread-safe queue - never the GPU - so it cannot race a decode.
+    void vram_warden_loop() {
+#if !defined(_WIN32)
+        char buf[4096];
+        while (vram_warden_run.load()) {
+            struct pollfd pfd { vram_inotify_fd, POLLIN, 0 };
+            int pr = poll(&pfd, 1, 500); // 500ms so we periodically re-check the run flag
+            if (pr <= 0) {
+                continue;
+            }
+            ssize_t n = read(vram_inotify_fd, buf, sizeof(buf));
+            if (n <= 0) {
+                continue;
+            }
+            bool foreign_ring = false;
+            for (char * p = buf; p < buf + n; ) {
+                struct inotify_event * ev = (struct inotify_event *) p;
+                if (ev->len > 0 && vram_pid_str != ev->name) {
+                    foreign_ring = true; // someone else wants the GPU
+                }
+                p += sizeof(struct inotify_event) + ev->len;
+            }
+            if (foreign_ring && !vram_cold) {
+                queue_tasks.request_yield();
+            }
+        }
+#endif
+    }
+
+    // ring the doorbell so the current token holder releases promptly
+    void vram_ring_doorbell() {
+#if !defined(_WIN32)
+        if (!vram_flock || vram_doorbell_dir.empty()) {
+            return;
+        }
+        const std::string f = vram_doorbell_dir + "/" + vram_pid_str;
+        int fd = open(f.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0666);
+        if (fd >= 0) {
+            ssize_t w = write(fd, "1", 1);
+            (void) w;
+            close(fd);
+        }
+#endif
+    }
+
+    // acquire the VRAM token (blocking) and bring weights back to the device. Called on the loop
+    // thread right before a decode, so it never races compute.
+    void vram_ensure_warm() {
+        if (!vram_only || !vram_cold) {
+            return;
+        }
+#if !defined(_WIN32)
+        if (vram_flock) {
+            vram_ring_doorbell();         // nudge the current holder to release
+            flock(vram_lock_fd, LOCK_EX); // blocks until the current holder goes cold
+        }
+#endif
+        llama_context_restore_device(ctx_tgt);
+        if (ctx_dft != nullptr) {
+            llama_context_restore_device(ctx_dft);
+        }
+        // NOTE: the multimodal (vision/audio) encoder is intentionally NOT restored here. It is
+        // brought into VRAM just-in-time before an image/audio encode (process_mtmd_chunk) and, in
+        // on-demand mode, released again right after - so a warm text-only model holds no encoder
+        // VRAM (that space is free for KV / expert cache). A cold->warm wake for a *text* request
+        // therefore leaves the encoder in RAM; an image request restores it at encode time.
+        vram_cold = false;
+    }
+
+    void vram_share_shutdown() {
+#if !defined(_WIN32)
+        vram_warden_run = false;
+        if (vram_warden.joinable()) {
+            vram_warden.join();
+        }
+        if (vram_inotify_fd >= 0) { close(vram_inotify_fd); vram_inotify_fd = -1; }
+        if (vram_lock_fd    >= 0) { flock(vram_lock_fd, LOCK_UN); close(vram_lock_fd); vram_lock_fd = -1; }
+#endif
+    }
+
+    // release the device weights (keeping KV + host shadow) and drop the VRAM token so another
+    // model can warm up. Called on the loop thread when the server goes idle.
+    void vram_go_cold() {
+        if (!vram_only || vram_cold) {
+            return;
+        }
+        llama_context_release_device(ctx_tgt, vram_evict_kv);
+        if (ctx_dft != nullptr) {
+            llama_context_release_device(ctx_dft, vram_evict_kv);
+        }
+        // also release the multimodal (vision/audio) encoder weights (dead weight while cold);
+        // its read-only host shadow is captured once and rebuilt on wake by vram_ensure_warm()
+        if (mctx != nullptr) {
+            mtmd_release_device(mctx);
+        }
+#if !defined(_WIN32)
+        if (vram_flock) {
+            flock(vram_lock_fd, LOCK_UN);
+        }
+#endif
+        vram_cold = true;
+    }
+
     common_speculative_init_result_ptr spec_init;
 
     common_context_seq_rm_type ctx_tgt_seq_rm_type = COMMON_CONTEXT_SEQ_RM_TYPE_NO;
@@ -951,6 +1186,29 @@ private:
 
     void handle_sleeping_state(bool new_state) {
         GGML_ASSERT(sleeping != new_state);
+
+        // Lightweight VRAM-only sleep: instead of a full unload/reload (which frees RAM and the
+        // KV cache and pays a full reload on wake), just release the model's device (VRAM) weight
+        // buffers to a host shadow (keeping the context, KV cache and host weights) and drop the
+        // shared VRAM token. Waking is handled lazily by vram_ensure_warm() right before the next
+        // decode (which re-acquires the token first), so a single GPU is time-shared between models
+        // with no re-prefill.
+        if (vram_only && ctx_tgt != nullptr) {
+            if (new_state) {
+                SRV_INF("%s", "server entering sleeping state (VRAM-only: releasing device weights, keeping KV cache)\n");
+                vram_go_cold();
+            } else {
+                SRV_INF("%s", "server exiting sleeping state (VRAM-only: restoring device weights/KV)\n");
+                // Restore NOW, on wake, before update_slots runs. update_slots touches the KV cache
+                // (e.g. SWA checkpoint creation reads it via ggml_backend_tensor_get) before the
+                // decode-time vram_ensure_warm(), so with KV eviction the KV must already be resident
+                // here or those reads hit a freed (null) buffer.
+                vram_ensure_warm();
+            }
+            sleeping = new_state;
+            return;
+        }
+
         if (new_state) {
             SRV_INF("%s", "server is entering sleeping state\n");
             destroy();
@@ -1142,6 +1400,11 @@ private:
             params_base.load_progress_callback_user_data = &load_progress_text;
         }
 
+        // VRAM arbiter: acquire the GPU token before uploading weights, so any resident model (e.g.
+        // the warm 4B task model) releases first and this load uploads into free VRAM instead of
+        // racing it and OOM-ing. No-op unless LLAMA_SLEEP_VRAM_ONLY is set.
+        vram_acquire_for_load();
+
         llama_init = common_init_from_params(params_base);
 
         model_tgt = llama_init->model();
@@ -1212,6 +1475,13 @@ private:
             }
             SRV_INF("loaded multimodal model, '%s'\n", mmproj_path.c_str());
 
+            // on-demand encoder: keep the vision/audio encoder weights in RAM (out of VRAM) until an
+            // image/audio actually needs encoding, freeing that VRAM for KV / expert cache on the
+            // common text-only path. process_mtmd_chunk() brings the encoder in just-in-time.
+            if (getenv("LLAMA_MMPROJ_ONDEMAND") != nullptr) {
+                mtmd_release_device(mctx);
+            }
+
             if (params_base.ctx_shift) {
                 params_base.ctx_shift = false;
                 SRV_WRN("%s\n", "ctx_shift is not supported by multimodal, it will be disabled");
@@ -1409,6 +1679,11 @@ private:
             handle_sleeping_state(sleeping);
         });
 
+        // VRAM arbiter: start the doorbell warden. The load was coordinated (vram_acquire_for_load
+        // grabbed the token before uploading), so we stay warm holding the token and serve the first
+        // request without a re-warm; the warden releases us when another model rings the doorbell.
+        vram_share_init();
+
         metrics.init();
 
         if (params_base.cache_idle_slots) {
@@ -3589,6 +3864,10 @@ private:
             n_empty_consecutive = 0;
         }
 
+        // VRAM arbiter: make sure our weights are resident (acquiring the shared GPU token first)
+        // before we decode. Runs on the loop thread, so it never races an in-flight decode.
+        vram_ensure_warm();
+
         const int ret = llama_decode(ctx_tgt, batch_view);
 
         metrics.on_decoded(slots);
@@ -4010,8 +4289,12 @@ struct server_res_generator : server_res_spipe {
     server_response_reader rd;
     server_res_generator(server_queue & queue_tasks, server_response & queue_results, int sleep_idle_seconds, bool bypass_sleep = false)
             : rd(queue_tasks, queue_results, HTTP_POLLING_SECONDS) {
-        // fast path in case sleeping is disabled
-        bypass_sleep |= sleep_idle_seconds < 0;
+        // fast path in case sleeping is disabled. Note: the VRAM arbiter (LLAMA_SLEEP_VRAM_ONLY)
+        // can put the server to sleep via the cross-process doorbell even when idle-sleep is
+        // disabled (sleep_idle_seconds < 0), so in that case requests must still wake it - otherwise
+        // a doorbell-slept server would hang, never returning from a request.
+        static const bool vram_arbiter = getenv("LLAMA_SLEEP_VRAM_ONLY") != nullptr;
+        bypass_sleep |= (sleep_idle_seconds < 0 && !vram_arbiter);
         if (!bypass_sleep) {
             queue_tasks.wait_until_no_sleep();
         }

tools/server/server-queue.cpp

@@ -116,6 +116,12 @@ void server_queue::wait_until_no_sleep() {
     }
 }
 
+void server_queue::request_yield() {
+    std::unique_lock<std::mutex> lock(mutex_tasks);
+    yield_requested = true;
+    condition_tasks.notify_all();
+}
+
 void server_queue::terminate() {
     std::unique_lock<std::mutex> lock(mutex_tasks);
     running = false;
@@ -129,6 +135,9 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
     constexpr auto max_wait_time = std::chrono::seconds(1);
     auto should_sleep = [&]() -> bool {
         // caller must hold mutex_tasks
+        if (yield_requested) {
+            return true; // another process rang the VRAM doorbell - release now
+        }
         if (idle_sleep_ms < 0) {
             return false;
         }
@@ -178,6 +187,7 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
             if (should_sleep()) {
                 QUE_INF("%s", "entering sleeping state\n");
                 sleeping = true;
+                yield_requested = false; // consumed
                 callback_sleeping_state(true);
                 req_stop_sleeping = false;
                 // wait until we are requested to exit sleeping state
@@ -195,13 +205,17 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
                 condition_tasks.notify_all(); // notify wait_until_no_sleep()
                 break; // process new tasks
             } else {
-                // wait for new tasks or timeout for checking sleeping condition
+                // wait for new tasks, a VRAM yield request, or timeout for checking sleeping condition
                 bool res = condition_tasks.wait_for(lock, max_wait_time, [&]{
-                    return (!queue_tasks.empty() || !running);
+                    return (!queue_tasks.empty() || !running || yield_requested);
                 });
-                if (res) {
+                if (res && !queue_tasks.empty()) {
                     break; // new task arrived or terminate
                 }
+                if (!running) {
+                    break;
+                }
+                // otherwise (timeout or yield request), loop again to re-check should_sleep
                 // otherwise, loop again to check sleeping condition
             }
         }

tools/server/server-queue.h

@@ -16,6 +16,7 @@ private:
     bool running  = false;
     bool sleeping = false;
     bool req_stop_sleeping = false;
+    bool yield_requested = false; // set by request_yield() when another process wants the VRAM token
     int64_t time_last_task = 0;
 
     // queues
@@ -51,6 +52,11 @@ public:
     // returns immediately if not sleeping
     void wait_until_no_sleep();
 
+    // request that the loop go to sleep (release VRAM) as soon as it is idle - called from the
+    // VRAM-arbiter warden thread when another process rings the doorbell for the GPU token.
+    // Thread-safe; wakes the loop so it releases promptly instead of at the next idle poll.
+    void request_yield();
+
     bool is_sleeping() {
         std::unique_lock<std::mutex> lock(mutex_tasks);
         return sleeping;