Author SHA1 Message Date
LumpiastyandClaude Opus 5 f2bb4c7f32 llama: skip the compute-buffer size check when the context is cold
release_device(evict_kv) drops the scheduler, so a context evicted by the VRAM
arbiter and then destroyed reaches the destructor with a null sched. The upstream
"compute buffer size matches expectation" loop calls
ggml_backend_sched_get_buffer_size() unconditionally, whose GGML_ASSERT(sched)
then aborts: the server exits 134 instead of 0 on every shutdown taken while
cold, which under llama-swap makes an ordinary stop look like a crashed child and
leaves a ggml backtrace in the log each time. It also skipped the rest of the
destructor, so the cold teardown path had never actually run to completion.

Guard at the call site rather than relaxing the assert - the assert is right, and
every other caller reserves the scheduler first. Same shape as the null guards in
synchronize(), memory_breakdown() and the released-buffer iteration.

The loop is a diagnostic size comparison, already reported by sched_reserve() at
load, so skipping it when cold loses nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-09-11 03:26:15 +02:00
LumpiastyandClaude Opus 5 48f0221c98 server: replace the VRAM doorbell with a want-lock
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the
holder's warden turned the inotify event into a queue flag. A ring carried no
notion of "still wanted", so it was lost whenever the holder was not already warm
and listening - during its model load (the watch does not exist yet, and the
kernel does not queue events for a watch that is not there) or inside
restore_device - and any ring that survived past the point its sender had been
satisfied caused a spurious release. Lost rings wedged the waiter permanently,
because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder
had no other reason to release. The wait blocks under mutex_tasks, so the whole
server stopped answering while /health still returned 200.

Express the request as kernel state instead. A waiter holds <arena>/want.lock
shared while it waits and drops it once it owns the token; the sleep decision
probes that lock non-blocking and releases the GPU while anyone is waiting. The
probe needs its own fd - flock treats two open file descriptions of one file
independently, so probing on the waiter's fd would convert our own lock rather
than conflict with it. Nothing can be missed, nothing goes stale, and a waiter
that dies is cleaned up by the kernel.

Probe from should_sleep() on the loop thread rather than from a warden thread.
Routing it through a flag is what made the first attempts fail: start_loop()
holds mutex_tasks from should_sleep() through the callbacks to the wait, so a
warden's request_yield() blocks on that mutex and is admitted only after the flag
has been consumed, latching a release for the next wake. Reading live state where
the decision is made has no edge to latch, and drops the warden's poll latency.

Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but
the loop clears req_stop_sleeping on the way in, so a loop that slept again
before the waiter ran stranded it forever - re-ask on every wake. And a task
queued after the waiter saw us awake could not wake us by itself, so sleep now
also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the
request that woke us is not queued yet, and yielding at once only sends it round
again.

Measured on the RX 580 pod, two servers contending, 60 alternating handoffs:
0 stranded, median 0.309 s against the doorbell's 0.314 s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-09-10 23:36:23 +02:00
LumpiastyandClaude Opus 5 f9a5c231ed scripts: add the RX 580 benchmark harness
Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-09-10 17:06:01 +02:00
LumpiastyandClaude Opus 5 c233ce9b51 ggml-vulkan: skip the concat transpose fast path below one tile
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a
loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of
40 layers) build_conv_state emits a transposed concat per recurrent layer, so
the fast path runs 30 times per decoded token.

At one token the transposed source is a single column, so there is no
uncoalesced stride left to fix, but the path still costs two dispatches and an
unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes
the command stream. Measured -17% tg256 on RX 580; the barrier, not the
half-empty transpose dispatch, is nearly all of it.

Require the transposed source to be at least one TILE_DIM wide. That source is
qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the
fast path, decode and small speculative drafts take the generic one.

pp8192 +11% retained, tg256 back to parity, generation byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-09-10 17:06:01 +02:00
Lumpiasty f9d41c711f docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends 2026-09-09 00:41:59 +02:00
Lumpiasty 04e9aec9bd server: on-demand mmproj - free encoder VRAM on the text path
The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.

When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.

Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.

Assisted-by: Claude
2026-09-09 00:40:56 +02:00
Lumpiasty 6564834854 mtmd: release/restore the encoder weights from VRAM on demand
Add clip_release_device/clip_restore_device (and mtmd_release_device/
mtmd_restore_device wrappers over the vision + audio contexts) that free
the multimodal encoder's device weight buffer to a read-only host shadow
and rebuild it on demand, using the same shadow/free/reallocate pattern
as llama_model weights. No-op for a CPU-backed encoder. This lets the
server drop the ~hundreds-of-MiB vision encoder from VRAM when it is not
encoding an image.

Assisted-by: Claude
2026-09-09 00:39:21 +02:00
Lumpiasty c49d19cc64 llama: free backend compute scratch on cold; guard memory_breakdown
On device release with KV eviction, also call ggml_backend_free_scratch()
on each backend so a cold model drops its Vulkan compute preallocations
(reallocated lazily on the next compute via restore).

Also guard llama_context::memory_breakdown() against a freed scheduler:
evict_kv release resets sched to null, so a cold model that is then torn
down (e.g. terminated by the process manager) hit GGML_ASSERT(sched) in
ggml_backend_sched_get_buffer_type. Skip the compute-buffer accounting
when sched is null.

Assisted-by: Claude
2026-09-09 00:38:39 +02:00
Lumpiasty 06a29462c1 ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.

Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).

Assisted-by: Claude
2026-09-09 00:38:35 +02:00
Lumpiasty ec691cd027 llama: evict recurrent/SSM state on device release
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left
resident when a model's device buffers were released for on-demand VRAM
sharing - llama_memory_recurrent::release_device_buffers() was a no-op
default. Implement it (and restore_device_buffers) with the same
capture-host-shadow / free / reallocate pattern as llama_kv_cache, so
llama_memory_hybrid now evicts both its attention KV and its recurrent
state. The state is read-write, so its shadow is recaptured on every
release.

Assisted-by: Claude
2026-09-09 00:36:41 +02:00
Lumpiasty 0d68afc4c1 server: also free the compute-graph scheduler on KV eviction
Under LLAMA_SLEEP_EVICT_KV a cold model still held the scheduler's worst-case
compute buffer (hundreds of MiB, e.g. ~700 MiB for gemma-26B) plus the resident
experts. release_device(evict_kv) now also frees the sched (sched.reset() +
sched_need_reserve), and restore_device() rebuilds it via sched_reserve(), so a
fully-evicted model holds essentially no VRAM (gemma cold: 7147 -> 51 MiB).
Trades a heavier re-warm (weights+KV H2D + sched reserve) for the freed VRAM;
weights-only mode is unchanged (fast switch, keeps KV+compute).

Assisted-by: Claude
2026-09-09 00:36:40 +02:00
Lumpiasty 4597050be9 server: restore weights/KV on wake, not at decode, so KV eviction is safe
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.

Assisted-by: Claude
2026-09-09 00:36:40 +02:00
Lumpiasty 857fbbfb39 llama: guard buffer iteration against released device buffers
release_device_weights()/release_device_buffers() leave a null buffer in
ctxs_bufs while the device memory is released for on-demand VRAM sharing.
The memory-breakdown / total_size / clear paths iterated these and called
ggml_backend_buffer_get_size()/get_type()/clear() on the null buffer, tripping
GGML_ASSERT(buffer) and aborting on shutdown of a cold model. Skip null buffers
in llama_model::memory_breakdown and llama_kv_cache::{memory_breakdown,
total_size,clear}.

Assisted-by: Claude
2026-09-09 00:36:40 +02:00
Lumpiasty 35d35d5c31 server: coordinate model load with the VRAM arbiter to avoid load-time OOM
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.

- add vram_arena_open() (idempotent flock/doorbell setup) and
  vram_acquire_for_load(), called from load_model() before
  common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
  request without a re-warm (drop the init-time go_cold cold-start).

Assisted-by: Claude
2026-09-09 00:36:40 +02:00
Lumpiasty ad657136a2 server: keep the sleep-wake path active when the VRAM arbiter is enabled
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.

Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.

Assisted-by: Claude
2026-09-09 00:36:40 +02:00
Lumpiasty 811b4fab77 server: optionally evict the KV cache too (Phase 2 of VRAM sharing)
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.

- llama_memory_i: add release_device_buffers()/restore_device_buffers()
  (default no-op). Implemented in llama_kv_cache (D2H shadow of the live
  ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
  llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
  buffers when requested; restore_device() rebuilds them. Public API
  llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
  since the KV shadow adds a D2H/H2D copy of the live cache each cycle.

Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.

Assisted-by: Claude
2026-09-09 00:36:40 +02:00
Lumpiasty 811eb55e12 server: on-demand VRAM sharing to time-share one GPU between models
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.

- llama-model: release_device_weights()/restore_device_weights() capture a
  compact host shadow (stable iteration order, view-skipping) and free then
  realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
  restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
  (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
  enforces "resident iff holds token"; an inotify doorbell forces the holder
  to release on contention. The warden thread only touches the task queue, so
  releases run on the loop thread and never race a decode.

Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).

Assisted-by: Claude
2026-09-09 00:36:34 +02:00
Lumpiasty 786b4077a4 ggml-vulkan: tiled transpose fast-path for concat with transposed source
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).

Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).

Assisted-by: opencode
2026-09-09 00:32:14 +02:00
LumpiastyandClaude Opus 5 6d483cc3c1 vulkan: record why flash-attn shmem staging stays off for GCN
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.

Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.

Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-09 00:21:39 +02:00
Lumpiasty c2d0492791 ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.

Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.

Assisted-by: Claude
2026-09-09 00:21:39 +02:00
LumpiastyandClaude Opus 5 dbc449afd0 sched: skip the MoE routing-ids readback when the batch uses every expert
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the scheduler reads the routing ids back to the host. The ids are produced
on the same device we are about to upload to, so the readback forces a full
pipeline flush - 47 of them per eval on a 48-layer MoE.

Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
bitset is all ones and the copy is a single whole-tensor range anyway.

Skip the readback when the batch guarantees that. Uploading an expert that no id
selects cannot change the result, since mul_mat_id only reads the rows the ids
point at, so this stays exact. Decode is unaffected: n_ids there is the number of
experts per token, far below the threshold, so it keeps the bitset path.

Measured on an RX 580 (Vulkan, Polaris) running Laguna-S-2.1 118B IQ2_M with all
experts host-resident: pp2048 +4.7% at depth 32k, +0.9% at 16k, tg32 +3%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-09 00:21:39 +02:00
32 changed files with 2035 additions and 41 deletions

No files matched your search

+76
View File
@@ -17,6 +17,82 @@
</div>
## This fork - Polaris / GCN tuning for large MoE models
Changes and measurements for running large MoE models with their experts offloaded to system RAM
(`--n-cpu-moe`) on an old GCN card. The two code changes below are auto-on, need no flag, and are
**token-identical** to mainline. Everything else here is tuning guidance.
Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16, no fp16, no matrix cores) with
**Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`:
- **Flash-attention `mask_opt` is enabled for GCN large head sizes (this fork's own change).**
Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips
fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on
a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B
(head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical.
- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp).
- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short
prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that
VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`).
**Recommended RX 580 / Polaris serving command** (per model):
```bash
llama-server -hf <repo>:<quant> -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1
```
Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high
for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so
`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path.
### Very large MoE (experts bigger than the GTT limit)
Measured on the same RX 580 with **Laguna-S-2.1 118B IQ2_M** (48 layers, 256 experts, 10 used,
experts 30.7 GiB of a 34.7 GiB file, all host-resident at `--n-cpu-moe 48`). The advice above changes
in this regime:
- **`--no-mmap` stops being an option.** Its pinned host buffer is charged against the amdgpu GTT
limit (~31.4 GiB here, about half of system RAM). At 30.7 GiB of experts the model no longer
loads, and the allocation spike can OOM the box. Use mmap and accept the staging copy.
- **The routing-ids readback is pure overhead at prefill batch sizes** and this fork now skips it.
To decide which experts to upload, the scheduler read the ids back from the device that had just
produced them, forcing a full pipeline flush once per MoE layer per eval. With 2048 tokens x 10
experts over 256 experts every expert is used anyway. Skipping it is exact - `mul_mat_id` only
reads the rows the ids point at. Auto-on above `4 * n_expert` ids; decode keeps the old path.
**pp2048 +4.7% @ 32k depth, +0.9% @ 16k, tg +3%.**
- **Where the time actually goes** (`GGML_VK_PERF_LOGGER=1`, depth 0, 22.65 s per 2048-token eval,
17.18 s of it GPU-busy so ~24% is H2D stall): expert `MUL_MAT_ID` **52%**, attention projections
**24%**, `FLASH_ATTN_EXT` **16%**, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s
while dense `MUL_MAT` q5_K/q6_K in the same graph reaches 3096-3691 - **the single largest
remaining opportunity on this hardware is closing that gap**, not the transfer path.
- **Interleaved SWA keeps its own small KV cache**, so a sliding-window layer costs the same at any
depth (`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`). On this model 36 of 48 layers are O(1) in
depth and the entire high-context slowdown comes from the 12 full-attention layers.
- **`--parallel 1`** is worth setting for a solo large model: the server otherwise auto-selects 4
slots, and the SWA cache is sized `n_swa * n_seq_max + n_ubatch`, so 4 slots cost 4096 cells
instead of 2560. Measured **223 MiB of VRAM freed** at 64k context.
Dead ends measured on this hardware, recorded so they are not retried:
| Change | Result |
| --- | ---: |
| flash-attn `shmem_staging` enabled for GCN | **-6.7% @ 16k, -7.4% @ 32k** |
| `-b 4096 -ub 4096` (to amortize the fixed per-eval expert upload) | flat (-1%) |
| `--n-cpu-moe` 44 instead of 48 | +1.5%, but does not fit at 64k ctx |
| `mask_opt` gate relaxed below head_dim 256 | -18.5% @ 16k |
`shmem_staging` looks like a certain win (without it each rowgroup re-reads the whole K/V block
through a 16 KiB L1) but the `kvsh` stride of `D/4+1` dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - that `+1` padding is tuned for warp32. `-ub 4096` fails because halving the
number of expert uploads is exactly cancelled by intra-ubatch attention growing quadratically.
**Serving note that outweighs all of the above.** With a model this large, anything that restarts the
process is far more expensive than any kernel win: the server's prompt cache is RAM-only with no disk
backing, so a restart forces a full re-prefill of the conversation. If a model swapper can evict this
model to run a small helper model (chat-title generation and the like), fix that first - keeping the
process alive across a swap took a repeat turn from a 21,960 ms prefill down to 225 ms.
## Quick start
A few options to get `llama.cpp` installed on your machine:
+5
View File
@@ -104,6 +104,11 @@ extern "C" {
GGML_API enum ggml_status ggml_backend_graph_compute (ggml_backend_t backend, struct ggml_cgraph * cgraph);
GGML_API enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph);
// Free transient/scratch device memory the backend holds outside of any allocated buffer
// (compute preallocations, staging buffers). No-op if the backend does not implement it.
// The backend remains usable; scratch is reallocated lazily on the next compute.
GGML_API void ggml_backend_free_scratch(ggml_backend_t backend);
// NOTE: will be removed, use device version instead
GGML_API bool ggml_backend_supports_op(ggml_backend_t backend, const struct ggml_tensor * op);
GGML_API bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft);
+5
View File
@@ -153,6 +153,11 @@ extern "C" {
// (optional) sort/optimize the nodes in the graph
void (*graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph, struct ggml_backend_graph_optimize_params * params);
// (optional) free transient/scratch device memory the backend holds outside of any buffer
// (e.g. compute preallocations and staging buffers). The backend stays usable; the scratch
// is reallocated lazily on the next compute. Used to shrink an idle model's device footprint.
void (*free_scratch) (ggml_backend_t backend);
};
struct ggml_backend {
+32 -8
View File
@@ -431,6 +431,15 @@ void ggml_backend_synchronize(ggml_backend_t backend) {
backend->iface.synchronize(backend);
}
void ggml_backend_free_scratch(ggml_backend_t backend) {
GGML_ASSERT(backend);
if (backend->iface.free_scratch == NULL) {
return;
}
backend->iface.free_scratch(backend);
}
ggml_backend_graph_plan_t ggml_backend_graph_plan_create(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
GGML_ASSERT(backend);
GGML_ASSERT(backend->iface.graph_plan_create != NULL);
@@ -1716,18 +1725,33 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
}
if (ids_tensor != prev_ids_tensor) {
ids.resize(ggml_nbytes(ids_tensor) / sizeof(int32_t));
ggml_backend_tensor_get_async(ids_backend, ids_tensor, ids.data(), 0, ggml_nbytes(ids_tensor));
ggml_backend_synchronize(ids_backend);
// Reading the ids back forces a full pipeline flush, since the ids are
// produced on the same device we are about to upload to. Once the batch
// draws enough experts, essentially all of them come back used and the
// readback has bought nothing, so skip it and upload the whole tensor as one
// range. Uploading an expert that no id selects cannot change the result:
// mul_mat_id only reads the rows the ids point at, so this stays exact.
const bool all_experts_used = ids_tensor->ne[0] * ids_tensor->ne[1] >= 4 * n_expert;
// find the used experts
used_ids.clear();
used_ids.resize(ggml_bitset_size(n_expert));
for (int64_t i1 = 0; i1 < ids_tensor->ne[1]; i1++) {
for (int64_t i0 = 0; i0 < ids_tensor->ne[0]; i0++) {
int32_t id = ids[i1 * ids_tensor->nb[1]/sizeof(int32_t) + i0 * ids_tensor->nb[0]/sizeof(int32_t)];
GGML_ASSERT(id >= 0 && id < n_expert);
ggml_bitset_set(used_ids.data(), id);
if (all_experts_used) {
for (int32_t i = 0; i < n_expert; i++) {
ggml_bitset_set(used_ids.data(), i);
}
} else {
ids.resize(ggml_nbytes(ids_tensor) / sizeof(int32_t));
ggml_backend_tensor_get_async(ids_backend, ids_tensor, ids.data(), 0, ggml_nbytes(ids_tensor));
ggml_backend_synchronize(ids_backend);
for (int64_t i1 = 0; i1 < ids_tensor->ne[1]; i1++) {
for (int64_t i0 = 0; i0 < ids_tensor->ne[0]; i0++) {
int32_t id = ids[i1 * ids_tensor->nb[1]/sizeof(int32_t) + i0 * ids_tensor->nb[0]/sizeof(int32_t)];
GGML_ASSERT(id >= 0 && id < n_expert);
ggml_bitset_set(used_ids.data(), id);
}
}
}
+139 -1
View File
@@ -3977,6 +3977,10 @@ static vk_fa_tuning_params get_fa_tuning_params_scalar(const vk_device& device,
result.d_split = std::min(std::min(result.subgroup_size, 8u), D_lsb / 4);
// Staging K/V through shared memory is a loss on AMD GCN: measured -6.7% at depth 16k and
// -7.4% at 32k on Polaris (head size 128). The kvsh stride of D/4+1 dwords is 4 mod 32, which
// costs an 8-way LDS bank conflict on wave64, and the extra shared memory cuts into occupancy
// that this shader is already short of. Keep it to NVIDIA.
result.shmem_staging = (device->vendor_id == VK_VENDOR_ID_NVIDIA && hsk < 256 && hsv < 256) ? 1 : 0;
if (!reduce_block_rows && !ggml_vk_flash_attn_scalar_shmem_support(device, result, hsk, hsv, f32acc, k_type, v_type)) {
@@ -11341,8 +11345,10 @@ static void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx
}
// Only use mask opt when the mask is fairly large. This hasn't been tuned extensively.
// GCN with large head size (>= 256) benefits in high-context prefill (skipping the
// per-block mask add on fully-visible blocks dominates), so enable it there too.
bool use_mask_opt = mask && nem1 >= 32 && nem0 * nem1 > 32768 && nem0 >= tuning_params.block_cols * 16
&& (ctx->device->architecture != vk_device_architecture::AMD_GCN || HSK > 256 || HSV > 256);
&& (ctx->device->architecture != vk_device_architecture::AMD_GCN || HSK >= 256 || HSV >= 256);
vk_fa_pipeline_state fa_pipeline_state = get_fa_pipeline_state(ctx->device, tuning_params, HSK, HSV, aligned, f32acc,
mask != nullptr, use_mask_opt, logit_softcap != 0, k_type_eff, v_type_eff);
@@ -13500,9 +13506,111 @@ static void ggml_vk_opt_step_sgd(ggml_backend_vk_context * ctx, vk_context& subc
ggml_vk_op_f32<vk_op_push_constants>(ctx, subctx, src0, src1, src2, nullptr, dst, GGML_OP_OPT_STEP_SGD, { (uint32_t)n, 0, 0.0f, 0.0f, 0.0f, 0.0f });
}
// Fast path for concat along dim 0 where one source is stored "transposed"
// (nb[1] == type_size, dim1 innermost) and the other source + dst are
// contiguous along dim 0. The generic concat shader reads the transposed
// source with a catastrophic uncoalesced stride; here we instead copy the
// contiguous source with copy.comp and transpose the other source into the
// matching dst sub-region with the tiled copy_transpose shader (shared-memory
// transpose, coalesced read+write). Mirrors the precedent in
// ggml_vk_cpy_to_contiguous: direct dispatch with custom push constants.
static void ggml_vk_concat_transpose_fastpath(ggml_backend_vk_context * ctx, vk_context& subctx,
const ggml_tensor * ctg, const ggml_tensor * trp,
ggml_tensor * dst, uint32_t off_ctg, uint32_t off_trp) {
const uint32_t ts = ggml_type_size(dst->type);
vk_pipeline pipeline_cpy = (ts == 4) ? ctx->device->pipeline_cpy_f32_f32
: ctx->device->pipeline_cpy_f16_f16;
vk_pipeline pipeline_trp = (ts == 4) ? ctx->device->pipeline_cpy_transpose_32
: ctx->device->pipeline_cpy_transpose_16;
ggml_pipeline_request_descriptor_sets(ctx, pipeline_cpy, 1);
ggml_pipeline_request_descriptor_sets(ctx, pipeline_trp, 1);
vk_subbuffer ctg_buf = ggml_vk_tensor_subbuffer(ctx, ctg, true);
vk_subbuffer trp_buf = ggml_vk_tensor_subbuffer(ctx, trp, true);
vk_subbuffer dst_buf = ggml_vk_tensor_subbuffer(ctx, dst, true);
const uint32_t a_misalign_ctg = get_misalign_bytes(ctx, ctg) / ts;
const uint32_t a_misalign_trp = get_misalign_bytes(ctx, trp) / ts;
const uint32_t d_misalign = get_misalign_bytes(ctx, dst) / ts;
// Dispatch A: contiguous copy of `ctg` into dst[off_ctg : off_ctg + ctg->ne[0], :]
if (ctg->ne[0] > 0) {
const uint32_t ne_ctg = (uint32_t) ggml_nelements(ctg);
vk_op_unary_push_constants pc = vk_op_unary_push_constants_init(ctg, dst, ne_ctg);
pc.ne10 = (uint32_t) ctg->ne[0]; // only ctg's columns in dst
pc.misalign_offsets = (a_misalign_ctg << 16) | (d_misalign + off_ctg);
init_pushconst_fastdiv(pc);
std::array<uint32_t, 3> el = ne_ctg > 262144 ? std::array<uint32_t,3>{512, 512, CEIL_DIV(ne_ctg, 262144)}
: ne_ctg > 512 ? std::array<uint32_t,3>{512, CEIL_DIV(ne_ctg, 512), 1}
: std::array<uint32_t,3>{ne_ctg, 1, 1};
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline_cpy, { ctg_buf, dst_buf }, pc, el);
}
// Dispatch B: tiled transpose of `trp` into dst[off_trp : off_trp + trp->ne[0], :]
if (trp->ne[0] > 0) {
vk_op_unary_push_constants pc = vk_op_unary_push_constants_init(trp, dst, ggml_nelements(trp));
pc.ne10 = (uint32_t) trp->ne[0]; // dst bound = trp columns (NOT dst->ne[0])
pc.misalign_offsets = (a_misalign_trp << 16) | (d_misalign + off_trp);
init_pushconst_fastdiv(pc);
std::array<uint32_t, 3> el = {
(uint32_t) CEIL_DIV(trp->ne[0], 32),
(uint32_t) CEIL_DIV(trp->ne[1], 32),
(uint32_t) (trp->ne[2] * trp->ne[3]),
};
el[0] = std::min(el[0], (uint32_t) ctx->device->properties.limits.maxComputeWorkGroupCount[0]);
el[1] = std::min(el[1], (uint32_t) ctx->device->properties.limits.maxComputeWorkGroupCount[1]);
el[2] = std::min(el[2], (uint32_t) ctx->device->properties.limits.maxComputeWorkGroupCount[2]);
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline_trp, { trp_buf, dst_buf }, pc, el);
}
ggml_vk_sync_buffers(ctx, subctx);
}
static void ggml_vk_concat(ggml_backend_vk_context * ctx, vk_context& subctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) {
int * op_params = (int *)dst->op_params;
// Fast path: concat along dim 0 with one source "transposed" (nb[1]==type_size,
// dim1 innermost) and the other source + dst contiguous along dim 0. The generic
// shader reads the transposed source uncoalesced (~5.6ms on RX 580 vs ~130us for
// a coalesced copy); here we instead use a tiled shared-memory transpose.
auto src_transposed_2d = [&](const ggml_tensor * s) {
const uint32_t ts = ggml_type_size(s->type);
return s->nb[1] == ts // dim1 innermost
&& s->nb[0] == (size_t) s->ne[1] * ts // consistent 2D transpose
&& s->ne[0] >= 32 // at least one full transpose tile (TILE_DIM); at 1 token the
// tiled path is pure overhead, the generic copy is faster
&& s->ne[2] == 1 && s->ne[3] == 1;
};
const uint32_t dst_ts = ggml_type_size(dst->type);
const bool dim0_ok = (op_params[0] == 0) && (dst->nb[0] == dst_ts);
// the dispatches below are raw f32/f16 element copies, so quantized and integer
// types stay on the generic path
const bool types_ok = (dst->type == GGML_TYPE_F32 || dst->type == GGML_TYPE_F16)
&& (src0->type == src1->type && src0->type == dst->type);
const bool shapes_ok = (src0->ne[1] == src1->ne[1] && src1->ne[1] == dst->ne[1])
&& (src0->ne[2] == src1->ne[2] && src0->ne[2] == dst->ne[2])
&& (src0->ne[3] == src1->ne[3] && src0->ne[3] == dst->ne[3])
&& (src0->ne[0] + src1->ne[0] == dst->ne[0]);
// doffset is 16-bit for unary shaders; guard against overflow
const bool off_fits = ((get_misalign_bytes(ctx, dst)/dst_ts + dst->ne[0])) < 0xFFFFu;
const bool s1_trans = dim0_ok && types_ok && shapes_ok && off_fits
&& src_transposed_2d(src1) && ggml_is_contiguous(src0);
const bool s0_trans = dim0_ok && types_ok && shapes_ok && off_fits
&& src_transposed_2d(src0) && ggml_is_contiguous(src1);
if (s1_trans || s0_trans) {
const ggml_tensor * ctg = s1_trans ? src0 : src1; // contiguous source
const ggml_tensor * trp = s1_trans ? src1 : src0; // transposed source
// dst offsets (in elements) where each source region begins
const uint32_t off_ctg = s1_trans ? 0u : (uint32_t) src1->ne[0];
const uint32_t off_trp = s1_trans ? (uint32_t) src0->ne[0] : 0u;
ggml_vk_concat_transpose_fastpath(ctx, subctx, ctg, trp, dst, off_ctg, off_trp);
return;
}
const uint32_t unit_size = ggml_vk_concat_unit_size(dst->type);
const uint32_t units_per_block = ggml_type_size(dst->type) / unit_size;
const uint32_t block_size = ggml_blck_size(dst->type);
@@ -17078,6 +17186,35 @@ static const char * ggml_backend_vk_name(ggml_backend_t backend) {
return ctx->name.c_str();
}
// Free the compute-scratch device buffers (prealloc_* and the transfer staging buffer) without
// tearing down the backend (pipelines, command pools, fences and device stay alive). These buffers
// are reallocated lazily by ggml_vk_preallocate_buffers() on the next compute, so this just shrinks
// an idle model's device footprint. Mirrors the buffer-freeing subset of ggml_vk_cleanup().
static void ggml_backend_vk_free_scratch(ggml_backend_t backend) {
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
VK_LOG_DEBUG("ggml_backend_vk_free_scratch(" << ctx->name << ")");
// discard any unsubmitted command buffer and wait for in-flight work before freeing
ctx->compute_ctx.reset();
ggml_vk_synchronize(ctx);
ggml_vk_destroy_buffer(ctx->prealloc_x);
ggml_vk_destroy_buffer(ctx->prealloc_y);
ggml_vk_destroy_buffer(ctx->prealloc_split_k);
ggml_vk_destroy_buffer(ctx->prealloc_add_rms_partials);
ggml_vk_destroy_buffer(ctx->sync_staging);
ctx->prealloc_y_last_pipeline_used = nullptr;
ctx->prealloc_y_last_tensor_used = nullptr;
ctx->prealloc_y_last_k_padded = false;
ctx->prealloc_size_x = 0;
ctx->prealloc_size_y = 0;
ctx->prealloc_size_split_k = 0;
ctx->prealloc_size_add_rms_partials = 0;
ctx->prealloc_size_add_rms_partials_offset = 0;
}
static void ggml_backend_vk_free(ggml_backend_t backend) {
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
VK_LOG_DEBUG("ggml_backend_vk_free(" << ctx->name << ")");
@@ -19016,6 +19153,7 @@ static ggml_backend_i ggml_backend_vk_interface = {
/* .event_record = */ ggml_backend_vk_event_record,
/* .event_wait = */ ggml_backend_vk_event_wait,
/* .graph_optimize = */ ggml_vk_graph_optimize,
/* .free_scratch = */ ggml_backend_vk_free_scratch,
};
static ggml_guid_t ggml_backend_vk_guid() {
+10
View File
@@ -580,6 +580,16 @@ extern "C" {
LLAMA_API const struct llama_model * llama_get_model (const struct llama_context * ctx);
LLAMA_API llama_memory_t llama_get_memory (const struct llama_context * ctx);
// On-demand device (VRAM) residency: free the model's GPU weight buffers (keeping a host
// shadow so no reload is needed) to hand VRAM to another model, and rebuild them on demand.
// Any subsequent llama_decode auto-restores. If evict_kv is true, the KV cache device buffers
// are also evicted to a host shadow (for when the KV would not fit alongside the other model),
// at the cost of a D2H/H2D copy of the live KV each cycle; otherwise the KV stays resident.
// Intended for time-sharing a single GPU between several always-loaded models.
LLAMA_API void llama_context_release_device(struct llama_context * ctx, bool evict_kv);
LLAMA_API void llama_context_restore_device(struct llama_context * ctx);
LLAMA_API bool llama_context_weights_resident(const struct llama_context * ctx);
LLAMA_API enum llama_pooling_type llama_pooling_type(const struct llama_context * ctx); // TODO: rename to llama_get_pooling_type
LLAMA_API const struct llama_vocab * llama_model_get_vocab(const struct llama_model * model);
+200
View File
@@ -0,0 +1,200 @@
# rx580-bench
Prompt-processing benchmark harness for the RX 580 (Polaris) serving pod.
It exists so that no future session has to rediscover how to measure this
machine. It encodes the production config, fixed corpus slices, the repeat
and median discipline, and the noise floor.
## What it measures
Two things, deliberately:
1. **Corpus prompt processing** (the number we actually care about). Real
Polish prose from `pan-tadeusz.txt` pushed through `llama-server`'s
`/completion` endpoint with `n_predict: 1`, `cache_prompt: false`. This
mimics agentic tool-result parsing: a big blob of real text arriving cold.
Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k
and 32k tokens.
2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random
tokens, `pp2048`, `pp8192`, `tg32`. This is what prior work on this box
recorded, so it is the continuity metric.
They do not measure the same thing and they do not agree. `llama-bench`
pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
growing KV cache, so they fall off with prompt length. Both are useful.
## How to run it
One command from the workstation:
```
./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean
```
Interleaved A/B, which is the only valid way to compare two builds:
```
./scripts/rx580-bench/run.sh \
--build /root/arms/new-clean/build --label new-clean \
--build-b /root/arms/new-fork/build --label-b new-fork \
--rounds 3
```
Summarize whatever has accumulated:
```
./scripts/rx580-bench/run.sh --summarize
```
`run.sh` copies the harness onto the pod and execs it there. The wrapper is
thin on purpose: all the methodology lives in the pod-side scripts, so it can
also be driven directly on the pod if kubectl is inconvenient.
The run is detached on the pod and then tailed, so Ctrl-C on the wrapper does
not orphan a half-finished run. `bench.sh` restarts `llama-swap` from an EXIT
trap on success, failure and interrupt alike.
### Pod access
```
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
```
The pod has `curl` and `python3` but **no `jq`**; the harness uses python3 for
all JSON.
## Files
| file | side | what |
|---|---|---|
| `run.sh` | workstation | one-command wrapper; installs and execs the rest |
| `bench.sh` | pod | orchestrator: stops llama-swap, waits for idle GPU, runs both harnesses, restarts llama-swap via trap |
| `ppbench.py` | pod | the corpus prompt-processing harness |
| `lbsweep.sh` | pod | the llama-bench sweep |
| `summarize.py` | pod | turns `results.txt` into a median/min-max table with delta significance |
Results append to `/root/bench/results.txt` as parseable `RESULT` /
`SUMMARY` / `LBRESULT` lines. Server logs land in `/root/bench/server-*.log`.
## The config it assumes
Production serving config, matching `/root/config.yaml`:
```
-t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --no-mmap
--ctx-size 40960 --no-warmup
```
Model:
```
/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
```
Qwen3.6-35B-A3B, 20.8 GiB, 40 layers, 256 experts of which 8 are used.
Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16,
12 CPU cores, 62 GB RAM.
The `llama-bench` equivalent of that config is:
```
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -mmp 0 -p 2048,8192 -n 32 -r 3
```
## The corpus slices are constants, not recalibrated
`pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this
model's tokenizer it runs **2.6702 chars/token**, and `/completion` reports
`prompt_n` identical to `/tokenize` (BOS offset 0).
The slices are `corpus[0:NCHARS]` and are baked into `ppbench.py`:
| target tokens | chars | utf-8 bytes |
|---|---|---|
| 4096 | 10776 | 11608 |
| 16384 | 43858 | 47175 |
| 32768 | 87165 | 93843 |
These hit the target `prompt_n` exactly. Every build is therefore measured on
byte-identical input. `ppbench.py` warns loudly if the reported `prompt_n`
ever stops matching the target, which is the signal that the table is stale.
Only recalibrate (`ppbench.py <build> <label> --calibrate`) if the model file
or the corpus changes.
## Reading the numbers
* `prompt_n` is the token count actually prefilled. It must equal the target.
* `prompt_ms` is wall-clock prefill time.
* `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better.
* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0.
* `tg32` is token generation at DEFAULT depth (n_ctx 32). **It is NOT a valid
decode metric on this box.** At that size decode is launch/barrier bound, not
compute bound, and measures a regime serving never reaches: it read -36%
between two upstream commits that are at parity in real use (the cause was
the view-alias fix in ggml_vk_graph_optimize). For decode use **tg256 with an
explicit depth**, e.g. `-p 0 -n 256 -d 2048`. tg is also NOT monotonic in
n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 -
so only ever compare same-depth cells.
## Noise floor: deltas under about 5 percent are unproven
This is the single most important thing in this directory.
`llama-bench`'s within-run error bars understate **cross-invocation** variance
by roughly **14x**. The same config measured `276.7 +/- 0.9` in one invocation
and `296.4 +/- 0.6` in another, 7 percent apart, with error bars that claimed
0.3 percent precision.
Therefore:
* Any comparison you want to draw a conclusion from must live **inside a
single `llama-bench` invocation**, using comma-separated sweeps.
* Two different *builds* cannot share an invocation, so run them
**interleaved A/B/A/B/A/B across at least 3 rounds** and compare medians.
That is what `--build-b` does.
* State explicitly that any cross-build delta under about 5 percent is
unproven. `summarize.py` labels them `unproven` for you.
## Traps
* **`-tb` is not a `llama-bench` flag.** It is a `llama-server` flag. Passing
it makes `llama-bench` print usage and exit silently, mid-sweep, which looks
exactly like a run that produced nothing. Always check the row count.
* **A ubatch larger than the prompt never fills.** With `-ub 2048`, prompt
lengths must be multiples of 2048 or the sweep measures nothing meaningful.
* **Nothing else may touch the GPU.** `bench.sh` stops `llama-swap` and
refuses to start if any `llama-*` process is still alive.
* **Never leave the pod not serving.** `bench.sh` restarts `llama-swap` from
an EXIT trap. If you bypass the script, restart it by hand:
`supervisorctl -c /root/supervisord.conf start llama-swap`.
* **Do not run `llama-cli` detached with closed stdin** on this box; it loops
forever. Use the server harness or `llama-bench -d` for depth timing.
* `--no-mmap` is deprecated upstream in favour of `--load-mode` / `-lm none`,
but still accepted and still what production passes.
## Known-good reference values
If a fresh run disagrees with these by much more than the noise floor,
something is wrong (something else on the GPU, a thermal problem, a bad
build) before you believe you found a speedup.
Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config, current upstream
(`f3f1a8f27`) and the deployed fork, measured 2026-09-08/09:
| metric | clean upstream | fork |
|---|---|---|
| corpus pp @ 4k | about 240 t/s | about 255 t/s |
| corpus pp @ 16k | about 206 t/s | about 225 t/s |
| corpus pp @ 32k | about 171 t/s | about 195 t/s |
| llama-bench pp2048 | see results.txt | about 285 t/s |
| llama-bench tg32 | | not a valid metric, see above |
Older reference points for the same model/config: `pp2048` depth-0 about
285 t/s. The historical "tg32 about 19 t/s" figure is retained only as trivia -
do not treat it as a target, and see the tg32 warning above.
Model load with `--no-mmap` takes about 26 s warm, longer cold.
A full 3-round A/B run of both harnesses takes roughly 100 minutes;
add about 16 minutes per extra arm per round.
+167
View File
@@ -0,0 +1,167 @@
#!/bin/sh
#
# RX 580 prompt-processing benchmark orchestrator. Runs ON THE POD.
#
# Stops llama-swap, waits for the GPU to go idle, runs the real-corpus
# prompt-processing harness and the llama-bench sweep for every arm, appends
# parseable results, and ALWAYS restarts llama-swap on exit (success, error
# or Ctrl-C).
#
# Usage:
# bench.sh --build DIR --label NAME
# [--build-b DIR --label-b NAME] second arm, interleaved
# [--arm DIR:LABEL ...] extra arms, repeatable (N-way)
# [--rounds N] default 3 multi-arm, 1 otherwise
# [--reps N] corpus repeats per size, default 3
# [--sizes 4096,16384,32768]
# [--corpus-only | --lb-only]
# [--results PATH] default /root/bench/results.txt
# [--keep-swap-down] do not restart llama-swap at exit
#
set -e
BENCH_DIR=/root/bench
SUPCONF=/root/supervisord.conf
RESULTS="$BENCH_DIR/results.txt"
MODEL=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
ARMS="" # space separated DIR|LABEL entries
BUILD_A=""; LABEL_A=""
BUILD_B=""; LABEL_B=""
ROUNDS=""
REPS=3
SIZES=4096,16384,32768
DO_CORPUS=1
DO_LB=1
KEEP_SWAP_DOWN=0
while [ $# -gt 0 ]; do
case "$1" in
--build) BUILD_A="$2"; shift 2 ;;
--label) LABEL_A="$2"; shift 2 ;;
--build-b) BUILD_B="$2"; shift 2 ;;
--label-b) LABEL_B="$2"; shift 2 ;;
--arm) ARMS="$ARMS ${2%%:*}|${2#*:}"; shift 2 ;;
--rounds) ROUNDS="$2"; shift 2 ;;
--reps) REPS="$2"; shift 2 ;;
--sizes) SIZES="$2"; shift 2 ;;
--results) RESULTS="$2"; shift 2 ;;
--corpus-only) DO_LB=0; shift ;;
--lb-only) DO_CORPUS=0; shift ;;
--keep-swap-down) KEEP_SWAP_DOWN=1; shift ;;
-h|--help) sed -n '2,20p' "$0"; exit 0 ;;
*) echo "unknown arg: $1" >&2; exit 2 ;;
esac
done
if [ -n "$BUILD_A" ] && [ -z "$LABEL_A" ]; then
echo "error: --build needs --label" >&2; exit 2
fi
if [ -n "$BUILD_B" ] && [ -z "$LABEL_B" ]; then
echo "error: --build-b needs --label-b" >&2; exit 2
fi
# --build/--label and --build-b/--label-b are sugar for the first two arms.
if [ -n "$BUILD_A" ]; then ARMS="$BUILD_A|$LABEL_A $ARMS"; fi
if [ -n "$BUILD_B" ]; then ARMS="$ARMS $BUILD_B|$LABEL_B"; fi
ARMS=$(echo $ARMS)
if [ -z "$ARMS" ]; then
echo "error: need at least one arm (--build/--label or --arm DIR:LABEL)" >&2
exit 2
fi
NARMS=$(echo "$ARMS" | wc -w)
if [ -z "$ROUNDS" ]; then
if [ "$NARMS" -gt 1 ]; then ROUNDS=3; else ROUNDS=1; fi
fi
for e in $ARMS; do
d=${e%%|*}
if [ ! -x "$d/bin/llama-server" ] || [ ! -x "$d/bin/llama-bench" ]; then
echo "error: $d lacks bin/llama-server or bin/llama-bench" >&2
exit 2
fi
done
cat <<'WARN'
-------------------------------------------------------------------------
RX 580 benchmark. Read this before trusting any number.
* Cross-invocation variance on this box is about 7 percent. llama-bench
within-run error bars understate it by roughly 14x. Any cross-build
delta under about 5 percent is UNPROVEN noise.
* Comparisons you care about must live inside ONE llama-bench invocation
(comma-separated sweeps), or be interleaved across at least 3 rounds.
That is what multiple arms plus --rounds does.
* Nothing else may touch the GPU while this runs. llama-swap is stopped
for the duration and restarted on exit, including on error or Ctrl-C.
* Traps: -tb is NOT a llama-bench flag. A ubatch larger than -p never
fills, so prompt lengths must be multiples of the 2048 ubatch. A bad
flag makes llama-bench print usage and exit silently mid-sweep.
-------------------------------------------------------------------------
WARN
echo "[bench] $NARMS arm(s), $ROUNDS round(s): $ARMS"
restore_swap() {
rc=$?
if [ "$KEEP_SWAP_DOWN" -eq 0 ]; then
echo ""
echo "[bench] restarting llama-swap"
supervisorctl -c "$SUPCONF" start llama-swap || true
supervisorctl -c "$SUPCONF" status llama-swap || true
else
echo "[bench] --keep-swap-down: llama-swap left STOPPED"
fi
exit $rc
}
trap restore_swap EXIT INT TERM
echo "[bench] stopping llama-swap"
supervisorctl -c "$SUPCONF" stop llama-swap || true
wait_gpu_idle() {
i=0
while [ $i -lt 60 ]; do
if ! pgrep -f 'bin/llama-server|bin/llama-bench|bin/llama-cli' >/dev/null 2>&1; then
return 0
fi
sleep 2
i=$((i+1))
done
echo "error: something is still on the GPU:" >&2
pgrep -af 'bin/llama-server|bin/llama-bench|bin/llama-cli' >&2
exit 3
}
echo "[bench] waiting for the GPU to go idle"
wait_gpu_idle
echo "[bench] GPU idle"
run_corpus() {
[ "$DO_CORPUS" -eq 1 ] || return 0
echo "===== corpus $2 ====="
wait_gpu_idle
python3 "$BENCH_DIR/ppbench.py" "$1" "$2" \
--sizes "$SIZES" --reps "$REPS" --results "$RESULTS"
}
run_lb() {
[ "$DO_LB" -eq 1 ] || return 0
echo "===== llama-bench $2 ====="
wait_gpu_idle
RESULTS="$RESULTS" MODEL="$MODEL" sh "$BENCH_DIR/lbsweep.sh" "$1" "$2"
}
# Interleave: every arm is measured once per round, so slow drift in the
# machine hits all arms roughly equally instead of biasing whichever ran first.
r=1
while [ "$r" -le "$ROUNDS" ]; do
echo ""
echo "########## ROUND $r / $ROUNDS ##########"
for e in $ARMS; do run_corpus "${e%%|*}" "${e#*|}"; done
for e in $ARMS; do run_lb "${e%%|*}" "${e#*|}"; done
r=$((r+1))
done
echo ""
echo "ALL_RUNS_DONE"
echo "[bench] raw results appended to $RESULTS"
python3 "$BENCH_DIR/summarize.py" "$RESULTS" || true
+69
View File
@@ -0,0 +1,69 @@
#!/bin/sh
#
# llama-bench sweep, the controlled cross-check for the corpus harness.
# Runs ON THE POD. Normally invoked by bench.sh.
#
# Usage: lbsweep.sh <build_dir> <label>
# Env overrides: RESULTS, MODEL, LB_P, LB_N, LB_R
#
# Everything that must be compared lives inside ONE llama-bench invocation
# (-p takes a comma-separated list), because cross-invocation variance on this
# box is about 7 percent while the within-run error bars are about 0.3 percent.
#
# Flag traps, learned the hard way:
# * -tb is NOT a llama-bench flag. It exists on llama-server only.
# * -ncmoe is the llama-bench spelling of --n-cpu-moe.
# * -mmp 0 is the llama-bench equivalent of the server's --no-mmap.
# * A bad flag makes llama-bench print usage and exit silently, which looks
# exactly like a sweep that produced no rows. Always check row count.
# * A ubatch larger than -p never fills, so keep -p a multiple of -ub.
#
set -e
BUILD_DIR="$1"
LABEL="$2"
if [ -z "$BUILD_DIR" ] || [ -z "$LABEL" ]; then
echo "usage: $0 <build_dir> <label>" >&2
exit 2
fi
RESULTS="${RESULTS:-/root/bench/results.txt}"
MODEL="${MODEL:-/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf}"
LB_P="${LB_P:-2048,8192}"
LB_N="${LB_N:-32}"
LB_R="${LB_R:-3}"
STAMP=$(date -u +%Y-%m-%dT%H:%M:%S)
export LD_LIBRARY_PATH="$BUILD_DIR/bin"
export LABEL BUILD_DIR STAMP RESULTS
OUT=$("$BUILD_DIR/bin/llama-bench" \
-m "$MODEL" \
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -mmp 0 \
-p "$LB_P" -n "$LB_N" -r "$LB_R" -o json 2>/dev/null)
if [ -z "$OUT" ]; then
echo "ERROR: llama-bench produced no output for $LABEL (bad flag?)" >&2
exit 3
fi
printf '%s' "$OUT" | python3 -c '
import json, sys, os
label = os.environ["LABEL"]
build = os.environ["BUILD_DIR"]
stamp = os.environ["STAMP"]
results = os.environ["RESULTS"]
rows = json.load(sys.stdin)
if not rows:
sys.stderr.write("ERROR: llama-bench returned zero rows\n")
sys.exit(3)
out = open(results, "a")
for r in rows:
test = "pp%d" % r["n_prompt"] if r["n_prompt"] else "tg%d" % r["n_gen"]
line = ("LBRESULT label=%s build=%s test=%s tps=%.2f stddev=%.2f "
"samples=%s time=%s"
% (label, build, test, r["avg_ts"], r["stddev_ts"],
r["samples_ts"], stamp))
out.write(line + "\n")
print(line)
out.close()
'
+239
View File
@@ -0,0 +1,239 @@
#!/usr/bin/env python3
"""
Real-corpus prompt-processing benchmark for llama-server on the RX 580 pod.
Runs ON THE POD. Normally invoked by bench.sh, not directly.
Starts llama-server from a given build directory with the production config,
waits for /health, POSTs fixed slices of the Pan Tadeusz corpus to /completion,
records timings.prompt_n / prompt_ms / prompt_per_second, then shuts the server
down cleanly.
Prompt slices are FIXED CONSTANTS (see SLICES below), calibrated once against
the Qwen3.6-35B-A3B tokenizer, so that every build is measured on byte-identical
input. Do not re-calibrate for a normal run; use --calibrate only if the model
or the corpus changes.
Usage:
ppbench.py <build_dir> <label> [--sizes 4096,16384,32768] [--reps 3]
[--results PATH] [--calibrate]
"""
import json
import os
import signal
import subprocess
import sys
import time
import urllib.request
from statistics import median
BENCH_DIR = "/root/bench"
CORPUS = os.path.join(BENCH_DIR, "pan-tadeusz.txt")
DEFAULT_RESULTS = os.path.join(BENCH_DIR, "results.txt")
MODEL = ("/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/"
"snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/"
"Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf")
PORT = 8099
BASE = "http://127.0.0.1:%d" % PORT
# Production serving config. Keep in sync with /root/config.yaml (llama-swap).
# Traps encoded here on purpose:
# -b / -ub 2048 : a ubatch larger than the prompt never fills, so every
# prompt length below must stay a multiple of 2048.
# --no-mmap : deprecated upstream in favour of --load-mode / -lm none,
# but still accepted; it is what production passes today.
SERVER_ARGS = ["-t", "6", "-ngl", "99", "--n-cpu-moe", "40",
"-b", "2048", "-ub", "2048", "-fa", "1", "--no-mmap",
"--ctx-size", "40960", "--no-warmup",
"--port", str(PORT), "-m", MODEL]
# Calibrated once on 2026-09-08 against Qwen3.6-35B-A3B-UD-Q4_K_XL:
# Polish text of pan-tadeusz.txt runs 2.6702 chars/token, and /completion
# reports prompt_n identical to /tokenize (BOS offset 0).
# Each slice is corpus[0:NCHARS] read as UTF-8 text; NBYTES is the resulting
# UTF-8 byte length, recorded so the slice can be reproduced with byte tools.
# These produce EXACTLY the target prompt_n. Do not edit without recalibrating.
SLICES = {
4096: {"chars": 10776, "bytes": 11608},
16384: {"chars": 43858, "bytes": 47175},
32768: {"chars": 87165, "bytes": 93843},
}
DEFAULT_SIZES = [4096, 16384, 32768]
def post(path, payload, timeout=2400):
req = urllib.request.Request(
BASE + path,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST")
with urllib.request.urlopen(req, timeout=timeout) as r:
return json.loads(r.read().decode("utf-8"))
def health_ok():
try:
with urllib.request.urlopen(BASE + "/health", timeout=5) as r:
return json.loads(r.read().decode("utf-8")).get("status") == "ok"
except Exception:
return False
def start_server(build_dir, logpath):
env = dict(os.environ)
env["LD_LIBRARY_PATH"] = os.path.join(build_dir, "bin")
binary = os.path.join(build_dir, "bin", "llama-server")
if not os.path.exists(binary):
raise RuntimeError("no llama-server at %s" % binary)
log = open(logpath, "wb")
p = subprocess.Popen([binary] + SERVER_ARGS, stdout=log,
stderr=subprocess.STDOUT, env=env,
start_new_session=True)
deadline = time.time() + 1200
while time.time() < deadline:
if p.poll() is not None:
raise RuntimeError("llama-server exited early rc=%s, see %s"
% (p.returncode, logpath))
if health_ok():
return p
time.sleep(2)
stop_server(p)
raise RuntimeError("llama-server not healthy within 1200s, see %s" % logpath)
def stop_server(p):
if p is None or p.poll() is not None:
return
try:
os.killpg(os.getpgid(p.pid), signal.SIGTERM)
except Exception:
p.terminate()
for _ in range(120):
if p.poll() is not None:
return
time.sleep(1)
try:
os.killpg(os.getpgid(p.pid), signal.SIGKILL)
except Exception:
p.kill()
p.wait()
def n_tokens(text):
"""Exact token count via /tokenize. Cheap: no prefill."""
return len(post("/tokenize", {"content": text}, timeout=300)["tokens"])
def calibrate(corpus, targets):
"""Recompute the SLICES table. Only needed if model or corpus changes."""
probe = corpus[:2000]
tk = n_tokens(probe)
r = post("/completion", {"prompt": probe, "n_predict": 1,
"cache_prompt": False, "temperature": 0})
offset = r["timings"]["prompt_n"] - tk
cpt = len(probe) / float(tk)
sys.stderr.write("calibrate: %.4f chars/token, prompt_n offset %d\n"
% (cpt, offset))
for target in targets:
want = target - offset
lo, hi = 1, len(corpus)
guess = min(len(corpus), max(1, int(want * cpt)))
best = None
for _ in range(60):
got = n_tokens(corpus[:guess])
if got == want:
best = guess
break
if got < want:
lo = guess + 1
else:
hi = guess - 1
if lo > hi:
break
guess = (lo + hi) // 2
if best is None:
best = guess
nbytes = len(corpus[:best].encode("utf-8"))
sys.stderr.write("calibrate: %d tokens -> chars=%d bytes=%d\n"
% (target, best, nbytes))
def main():
if len(sys.argv) < 3:
sys.stderr.write(__doc__)
return 2
build_dir = os.path.abspath(sys.argv[1])
label = sys.argv[2]
sizes = list(DEFAULT_SIZES)
reps = 3
results_path = DEFAULT_RESULTS
do_calibrate = "--calibrate" in sys.argv
args = sys.argv[3:]
for i, a in enumerate(args):
if a == "--sizes":
sizes = [int(x) for x in args[i + 1].split(",")]
elif a == "--reps":
reps = int(args[i + 1])
elif a == "--results":
results_path = args[i + 1]
corpus = open(CORPUS, encoding="utf-8").read()
logpath = os.path.join(BENCH_DIR, "server-%s-%d.log" % (label, int(time.time())))
p = None
try:
sys.stderr.write("[%s] starting llama-server from %s\n" % (label, build_dir))
t0 = time.time()
p = start_server(build_dir, logpath)
sys.stderr.write("[%s] healthy after %.1fs\n" % (label, time.time() - t0))
if do_calibrate:
calibrate(corpus, sizes)
return 0
stamp = time.strftime("%Y-%m-%dT%H:%M:%S")
out = open(results_path, "a")
for target in sizes:
if target not in SLICES:
sys.stderr.write("no calibrated slice for %d tokens, skipping\n" % target)
continue
nchars = SLICES[target]["chars"]
prompt = corpus[:nchars]
rates, mss, ns = [], [], set()
for rep in range(1, reps + 1):
r = post("/completion", {"prompt": prompt, "n_predict": 1,
"cache_prompt": False, "temperature": 0})
t = r["timings"]
rates.append(t["prompt_per_second"])
mss.append(t["prompt_ms"])
ns.add(t["prompt_n"])
line = ("RESULT corpus label=%s build=%s target=%d chars=%d "
"prompt_n=%d rep=%d prompt_ms=%.2f tps=%.2f time=%s"
% (label, build_dir, target, nchars, t["prompt_n"], rep,
t["prompt_ms"], t["prompt_per_second"], stamp))
out.write(line + "\n")
out.flush()
sys.stderr.write(line + "\n")
if ns != {target}:
sys.stderr.write("WARNING: prompt_n %s != target %d; slice table "
"is stale, rerun with --calibrate\n" % (sorted(ns), target))
s = ("SUMMARY corpus label=%s target=%d prompt_n=%s median_tps=%.2f "
"min_tps=%.2f max_tps=%.2f median_ms=%.1f time=%s"
% (label, target, sorted(ns), median(rates), min(rates),
max(rates), median(mss), stamp))
out.write(s + "\n")
out.flush()
sys.stderr.write(s + "\n")
out.close()
finally:
stop_server(p)
sys.stderr.write("[%s] server stopped\n" % label)
return 0
if __name__ == "__main__":
sys.exit(main())
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env bash
#
# One-command entry point for the RX 580 prompt-processing benchmark.
#
# Runs from your workstation. Copies the pod-side harness into the pod and
# execs it there over kubectl. All the real work happens on the pod; this is
# a thin wrapper so there is exactly one command to remember.
#
# Examples:
# ./scripts/rx580-bench/run.sh --build /root/llama.cpp/build --label fork-before
# ./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean \
# --build-b /root/arms/new-fork/build --label-b new-fork
# ./scripts/rx580-bench/run.sh --summarize
# ./scripts/rx580-bench/run.sh --install-only
#
# Every argument other than the wrapper-only flags below is passed straight
# through to bench.sh on the pod. See bench.sh --help.
#
set -euo pipefail
KUBECONFIG_PATH="${KUBECONFIG:-/home/user/Projects/klaster/talos/generated/kubeconfig}"
NS="${RX580_NS:-llama}"
DEPLOY="${RX580_DEPLOY:-deploy/supervisord}"
POD_BENCH_DIR=/root/bench
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export KUBECONFIG="$KUBECONFIG_PATH"
kx() { kubectl -n "$NS" exec -i "$DEPLOY" -- sh -c "$1"; }
install_harness() {
echo "[run] installing harness into $POD_BENCH_DIR"
kx "mkdir -p $POD_BENCH_DIR"
for f in bench.sh ppbench.py lbsweep.sh summarize.py; do
kx "cat > $POD_BENCH_DIR/$f" < "$HERE/$f"
done
kx "chmod +x $POD_BENCH_DIR/bench.sh $POD_BENCH_DIR/lbsweep.sh $POD_BENCH_DIR/ppbench.py $POD_BENCH_DIR/summarize.py"
# The corpus is large and rarely changes; only push it if missing.
if ! kx "test -s $POD_BENCH_DIR/pan-tadeusz.txt" 2>/dev/null; then
if [ -f "$HERE/pan-tadeusz.txt" ]; then
echo "[run] uploading corpus"
kx "cat > $POD_BENCH_DIR/pan-tadeusz.txt" < "$HERE/pan-tadeusz.txt"
else
echo "[run] ERROR: $POD_BENCH_DIR/pan-tadeusz.txt missing on the pod and" >&2
echo " no local copy at $HERE/pan-tadeusz.txt to upload." >&2
exit 4
fi
fi
}
ARGS=()
INSTALL_ONLY=0
SUMMARIZE=0
for a in "$@"; do
case "$a" in
--install-only) INSTALL_ONLY=1 ;;
--summarize) SUMMARIZE=1 ;;
*) ARGS+=("$a") ;;
esac
done
install_harness
if [ "$INSTALL_ONLY" -eq 1 ]; then
echo "[run] harness installed; not running anything"
exit 0
fi
if [ "$SUMMARIZE" -eq 1 ]; then
kx "python3 $POD_BENCH_DIR/summarize.py ${ARGS[*]:-}"
exit 0
fi
if [ ${#ARGS[@]} -eq 0 ]; then
kx "sh $POD_BENCH_DIR/bench.sh --help"
exit 2
fi
# A full 3-round A/B run takes hours; run detached on the pod and tail it, so a
# dropped kubectl connection cannot orphan a half-finished run with llama-swap
# still down. bench.sh restarts llama-swap from its own EXIT trap either way.
STAMP="$(date -u +%Y%m%d-%H%M%S)"
LOG="$POD_BENCH_DIR/run-$STAMP.log"
echo "[run] starting detached run on the pod, log: $LOG"
kx "cd $POD_BENCH_DIR && nohup sh $POD_BENCH_DIR/bench.sh ${ARGS[*]} > $LOG 2>&1 & echo started"
echo "[run] tailing until ALL_RUNS_DONE (safe to Ctrl-C: the pod keeps running)"
kx "i=0; while [ \$i -lt 100000 ]; do
if grep -qE 'ALL_RUNS_DONE|Traceback|error:' $LOG; then break; fi
sleep 15; i=\$((i+1));
done; cat $LOG"
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""
Summarize a results.txt produced by bench.sh into a per-label table.
Usage: summarize.py [results.txt]
Prints median and min-max of every metric for every label, and flags
cross-label deltas against the ~5 percent noise floor of this machine.
"""
import re
import sys
from statistics import median
NOISE_FLOOR_PCT = 5.0
RE_CORPUS = re.compile(
r"^RESULT corpus label=(\S+) build=\S+ target=(\d+) chars=\d+ "
r"prompt_n=(\d+) rep=\d+ prompt_ms=([\d.]+) tps=([\d.]+)")
RE_LB = re.compile(
r"^LBRESULT label=(\S+) build=\S+ test=(\S+) tps=([\d.]+)")
def main():
path = sys.argv[1] if len(sys.argv) > 1 else "/root/bench/results.txt"
# data[metric][label] = list of tps
data = {}
order = []
labels = []
for line in open(path):
m = RE_CORPUS.match(line)
if m:
label, target, _pn, _ms, tps = m.groups()
metric = "pp%s-corpus" % target
else:
m = RE_LB.match(line)
if not m:
continue
label, test, tps = m.groups()
metric = "%s-llama-bench" % test
if metric not in data:
data[metric] = {}
order.append(metric)
data[metric].setdefault(label, []).append(float(tps))
if label not in labels:
labels.append(label)
if not data:
print("no parseable results in %s" % path)
return 1
def sort_key(m):
n = re.search(r"(\d+)", m)
return (0 if "corpus" in m else 1, int(n.group(1)) if n else 0)
order.sort(key=sort_key)
w = max(len(l) for l in labels) + 2
head = "metric".ljust(20) + "".join(l.ljust(max(w, 24)) for l in labels)
print("")
print("t/s, median (min-max), n samples")
print(head)
print("-" * len(head))
for metric in order:
row = metric.ljust(20)
for label in labels:
vals = data[metric].get(label)
if not vals:
row += "-".ljust(max(w, 24))
else:
cell = "%.1f (%.1f-%.1f) n=%d" % (
median(vals), min(vals), max(vals), len(vals))
row += cell.ljust(max(w, 24))
print(row)
if len(labels) >= 2:
base = labels[0]
print("")
print("deltas vs %s (noise floor %.0f%%, anything under it is UNPROVEN)"
% (base, NOISE_FLOOR_PCT))
for other in labels[1:]:
print(" %s vs %s:" % (other, base))
for metric in order:
a = data[metric].get(base)
b = data[metric].get(other)
if not a or not b:
continue
ma, mb = median(a), median(b)
pct = (mb - ma) / ma * 100.0
verdict = "SIGNIFICANT" if abs(pct) >= NOISE_FLOOR_PCT else "unproven"
print(" %-20s %+6.1f%% %s" % (metric, pct, verdict))
return 0
if __name__ == "__main__":
sys.exit(main())
+76 -12
View File
@@ -483,7 +483,8 @@ llama_context::~llama_context() {
synchronize();
// when training, ggml_opt allocates extra buffers through the scheduler, so the sizes no longer match the expectation
if (!model.hparams.no_alloc && !opt_ctx) {
// the scheduler is also gone if the context is destroyed while cold (on-demand VRAM eviction, see release_device)
if (sched && !model.hparams.no_alloc && !opt_ctx) {
for (size_t i = 0; i < backend_ptrs.size(); ++i) {
ggml_backend_t backend = backend_ptrs[i];
ggml_backend_buffer_type_t buft = backend_buft[i];
@@ -745,6 +746,47 @@ void llama_context::synchronize() {
t_compute_start_us = 0;
}
void llama_context::release_device(bool evict_kv) {
if (!model.weights_resident() && !(evict_kv && !kv_device_evicted)) {
return;
}
// ensure no compute is in flight before freeing the device buffers
synchronize();
model.release_device_weights();
if (evict_kv && memory && !kv_device_evicted) {
memory->release_device_buffers();
kv_device_evicted = true;
// also free the scheduler and its worst-case compute buffer (hundreds of MiB) so a cold
// model holds essentially no VRAM. Rebuilt lazily by sched_reserve() on restore.
sched.reset();
sched_need_reserve = true;
// finally, free each backend's own compute-scratch (Vulkan prealloc/staging buffers, which
// are owned by the backend and survive sched.reset()). Reallocated lazily on next compute.
for (auto & backend : backends) {
ggml_backend_free_scratch(backend.get());
}
}
}
void llama_context::restore_device() {
if (model.weights_resident() && !kv_device_evicted && sched) {
return;
}
model.restore_device_weights();
if (kv_device_evicted && memory) {
memory->restore_device_buffers();
kv_device_evicted = false;
}
// rebuild the scheduler + compute buffer if they were freed on release (evict_kv mode)
if (!sched) {
sched_reserve();
}
// make sure all weight/KV uploads have completed before any compute reads them
if (sched) {
ggml_backend_sched_synchronize(sched.get());
}
}
const llama_model & llama_context::get_model() const {
return model;
}
@@ -1656,6 +1698,12 @@ int llama_context::decode(const llama_batch & batch_inp) {
return -1;
}
// on-demand VRAM: if the weights were released while this model was idle, bring them back
// to the device before building the compute graph
if (!model.weights_resident()) {
restore_device();
}
const auto & vocab = model.vocab;
const auto & hparams = model.hparams;
@@ -3366,17 +3414,21 @@ llama_memory_breakdown llama_context::memory_breakdown() const {
ret[buft].context += size;
}
}
if (model.hparams.no_alloc) {
for (size_t i = 0; i < backends.size(); ++i) {
ggml_backend_t backend = backends[i].get();
ggml_backend_buffer_type_t buft = ggml_backend_sched_get_buffer_type(sched.get(), backend);
ret[buft].compute += backend_buf_exp_size[i];
}
} else {
for (const auto & backend_ptr : backends) {
ggml_backend_t backend = backend_ptr.get();
ggml_backend_buffer_type_t buft = ggml_backend_sched_get_buffer_type(sched.get(), backend);
ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
// the scheduler (and its compute buffers) may have been freed while the model is cold
// (on-demand VRAM eviction, see release_device); it contributes no compute VRAM then.
if (sched) {
if (model.hparams.no_alloc) {
for (size_t i = 0; i < backends.size(); ++i) {
ggml_backend_t backend = backends[i].get();
ggml_backend_buffer_type_t buft = ggml_backend_sched_get_buffer_type(sched.get(), backend);
ret[buft].compute += backend_buf_exp_size[i];
}
} else {
for (const auto & backend_ptr : backends) {
ggml_backend_t backend = backend_ptr.get();
ggml_backend_buffer_type_t buft = ggml_backend_sched_get_buffer_type(sched.get(), backend);
ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
}
}
}
return ret;
@@ -3850,6 +3902,18 @@ void llama_synchronize(llama_context * ctx) {
ctx->synchronize();
}
void llama_context_release_device(llama_context * ctx, bool evict_kv) {
ctx->release_device(evict_kv);
}
void llama_context_restore_device(llama_context * ctx) {
ctx->restore_device();
}
bool llama_context_weights_resident(const llama_context * ctx) {
return ctx->get_model().weights_resident();
}
float * llama_get_logits(llama_context * ctx) {
ctx->synchronize();
+11
View File
@@ -57,6 +57,14 @@ struct llama_context {
void synchronize();
// on-demand device (VRAM) residency: free / rebuild the model's GPU weight buffers so that
// VRAM can be time-shared with another model. release_device() synchronizes first; decode()
// auto-restores when it finds the weights have been released. If evict_kv is set, the KV cache
// device buffers are also evicted to a host shadow (for when the KV would not fit alongside the
// other model) - this adds a D2H/H2D copy of the live KV on each cycle.
void release_device(bool evict_kv = false);
void restore_device();
const llama_model & get_model() const;
const llama_cparams & get_cparams() const;
@@ -345,6 +353,9 @@ private:
bool sched_need_reserve = true;
// on-demand device residency: true while the KV cache device buffers have been evicted to host
bool kv_device_evicted = false;
ggml_backend_t backend_cpu = nullptr;
std::vector<ggml_backend_ptr> backends;
+11
View File
@@ -272,6 +272,17 @@ void llama_kv_cache_iswa::state_read(llama_io_read_i & io, llama_seq_id seq_id,
kv_swa->state_read(io, seq_id, flags);
}
void llama_kv_cache_iswa::release_device_buffers() {
kv_base->release_device_buffers();
kv_swa->release_device_buffers();
}
bool llama_kv_cache_iswa::restore_device_buffers() {
bool ok = kv_base->restore_device_buffers();
ok = kv_swa->restore_device_buffers() && ok;
return ok;
}
llama_kv_cache * llama_kv_cache_iswa::get_base() const {
return kv_base.get();
}
+3
View File
@@ -83,6 +83,9 @@ public:
void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
void state_read (llama_io_read_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
void release_device_buffers() override;
bool restore_device_buffers() override;
//
// llama_kv_cache_iswa specific API
//
+83 -2
View File
@@ -374,7 +374,9 @@ void llama_kv_cache::clear(bool data) {
if (data) {
for (auto & [_, buf] : ctxs_bufs) {
ggml_backend_buffer_clear(buf.get(), 0);
if (buf) { // may be null if evicted for on-demand VRAM sharing
ggml_backend_buffer_clear(buf.get(), 0);
}
}
}
}
@@ -685,6 +687,9 @@ llama_pos llama_kv_cache::seq_pos_max(llama_seq_id seq_id) const {
std::map<ggml_backend_buffer_type_t, size_t> llama_kv_cache::memory_breakdown() const {
std::map<ggml_backend_buffer_type_t, size_t> ret;
for (const auto & [ctx, buf] : ctxs_bufs) {
if (!buf) { // may be null if evicted for on-demand VRAM sharing
continue;
}
ggml_backend_buffer_type_t buft = ggml_backend_buffer_get_type(buf.get());
if (hparams.no_alloc) {
@@ -1893,12 +1898,88 @@ size_t llama_kv_cache::total_size() const {
size_t size = 0;
for (const auto & [_, buf] : ctxs_bufs) {
size += ggml_backend_buffer_get_size(buf.get());
if (buf) { // may be null if evicted for on-demand VRAM sharing
size += ggml_backend_buffer_get_size(buf.get());
}
}
return size;
}
void llama_kv_cache::release_device_buffers() {
// NOTE: the caller must have synchronized the backend so nothing references these buffers.
// The KV cache is read-write, so its host shadow is (re)captured fresh on every release.
if (dev_released) {
return;
}
dev_shadows.assign(ctxs_bufs.size(), device_buffer_shadow{});
size_t freed = 0;
for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
ggml_context * ctx = ctxs_bufs[i].first.get();
ggml_backend_buffer_t buf = ctxs_bufs[i].second.get();
if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
continue; // only real device (VRAM) buffers are evictable
}
auto & sh = dev_shadows[i];
sh.releasable = true;
sh.buft = ggml_backend_buffer_get_type(buf);
// capture live contents compactly in stable iteration order (skip views, which alias a base)
size_t total = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src == nullptr) { total += ggml_nbytes(t); }
}
sh.data.resize(total);
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) { continue; }
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_get(t, sh.data.data() + off, 0, n);
off += n;
}
freed += ggml_backend_buffer_get_size(buf);
ctxs_bufs[i].second.reset(); // free the device buffer
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
t->buffer = nullptr;
t->data = nullptr;
}
}
dev_released = true;
if (freed > 0) {
LLAMA_LOG_INFO("%s: released %.2f MiB of KV cache from device\n", __func__, freed / 1024.0 / 1024.0);
}
}
bool llama_kv_cache::restore_device_buffers() {
if (!dev_released) {
return true;
}
for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
auto & sh = dev_shadows[i];
if (!sh.releasable) {
continue;
}
ggml_context * ctx = ctxs_bufs[i].first.get();
ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, sh.buft);
if (buf == nullptr) {
LLAMA_LOG_ERROR("%s: failed to reallocate KV cache device buffer (out of VRAM?)\n", __func__);
return false;
}
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) { continue; }
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_set(t, sh.data.data() + off, 0, n);
off += n;
}
ctxs_bufs[i].second.reset(buf);
}
dev_released = false;
dev_shadows.clear(); // recaptured on next release
return true;
}
size_t llama_kv_cache::size_k_bytes() const {
size_t size_k_bytes = 0;
+14
View File
@@ -151,6 +151,10 @@ public:
void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
void state_read (llama_io_read_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
// on-demand device (VRAM) residency (see llama_memory_i)
void release_device_buffers() override;
bool restore_device_buffers() override;
//
// llama_kv_cache specific API
//
@@ -291,6 +295,16 @@ private:
// ggml contexts for the KV cache along with the allocated backend buffers:
std::vector<std::pair<ggml_context_ptr, ggml_backend_buffer_ptr>> ctxs_bufs;
// on-demand device eviction: host shadow of each device buffer's live contents (re-captured on
// every release since the KV is read-write), plus the buffer type needed to reallocate it.
struct device_buffer_shadow {
ggml_backend_buffer_type_t buft = nullptr;
bool releasable = false; // device (VRAM) buffer that was freed
std::vector<uint8_t> data; // captured tensor bytes (compact, iteration order)
};
std::vector<device_buffer_shadow> dev_shadows; // parallel to ctxs_bufs
bool dev_released = false;
// the current index from where we start searching for a free slot in the ring buffer of KV cells (see find_slot())
// note: this is not part of the KV state and it's only used to speed-up the find_slot() method
std::vector<uint32_t> v_heads;
+12
View File
@@ -201,6 +201,18 @@ void llama_memory_hybrid::state_read(llama_io_read_i & io, llama_seq_id seq_id,
mem_recr->state_read(io, seq_id, flags);
}
void llama_memory_hybrid::release_device_buffers() {
// evict both the attention KV (grows with context) and the recurrent/SSM state
mem_attn->release_device_buffers();
mem_recr->release_device_buffers();
}
bool llama_memory_hybrid::restore_device_buffers() {
bool ok = mem_attn->restore_device_buffers();
ok = mem_recr->restore_device_buffers() && ok;
return ok;
}
llama_kv_cache * llama_memory_hybrid::get_mem_attn() const {
return mem_attn.get();
}
+3
View File
@@ -76,6 +76,9 @@ public:
void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
void state_read (llama_io_read_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
void release_device_buffers() override;
bool restore_device_buffers() override;
//
// llama_memory_hybrid specific API
//
+80 -2
View File
@@ -151,7 +151,9 @@ void llama_memory_recurrent::clear(bool data) {
if (data) {
for (auto & [_, buf] : ctxs_bufs) {
ggml_backend_buffer_clear(buf.get(), 0);
if (buf) { // may be null if evicted for on-demand VRAM sharing
ggml_backend_buffer_clear(buf.get(), 0);
}
}
}
@@ -420,6 +422,7 @@ void llama_memory_recurrent::set_rs_idx(llama_seq_id seq_id, uint32_t idx) {
std::map<ggml_backend_buffer_type_t, size_t> llama_memory_recurrent::memory_breakdown() const {
std::map<ggml_backend_buffer_type_t, size_t> ret;
for (const auto & [_, buf] : ctxs_bufs) {
if (!buf) { continue; } // may be null if evicted for on-demand VRAM sharing
ret[ggml_backend_buffer_get_type(buf.get())] += ggml_backend_buffer_get_size(buf.get());
}
return ret;
@@ -721,12 +724,87 @@ bool llama_memory_recurrent::get_can_shift() const {
size_t llama_memory_recurrent::total_size() const {
size_t size = 0;
for (const auto & [_, buf] : ctxs_bufs) {
size += ggml_backend_buffer_get_size(buf.get());
if (buf) { // may be null if evicted for on-demand VRAM sharing
size += ggml_backend_buffer_get_size(buf.get());
}
}
return size;
}
void llama_memory_recurrent::release_device_buffers() {
// Same mechanism as llama_kv_cache: the recurrent (SSM/conv) state is read-write, so its host
// shadow is (re)captured on every release. The caller must have synchronized the backend.
if (dev_released) {
return;
}
dev_shadows.assign(ctxs_bufs.size(), device_buffer_shadow{});
size_t freed = 0;
for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
ggml_context * ctx = ctxs_bufs[i].first.get();
ggml_backend_buffer_t buf = ctxs_bufs[i].second.get();
if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
continue;
}
auto & sh = dev_shadows[i];
sh.releasable = true;
sh.buft = ggml_backend_buffer_get_type(buf);
size_t total = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src == nullptr) { total += ggml_nbytes(t); }
}
sh.data.resize(total);
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) { continue; }
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_get(t, sh.data.data() + off, 0, n);
off += n;
}
freed += ggml_backend_buffer_get_size(buf);
ctxs_bufs[i].second.reset();
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
t->buffer = nullptr;
t->data = nullptr;
}
}
dev_released = true;
if (freed > 0) {
LLAMA_LOG_INFO("%s: released %.2f MiB of recurrent state from device\n", __func__, freed / 1024.0 / 1024.0);
}
}
bool llama_memory_recurrent::restore_device_buffers() {
if (!dev_released) {
return true;
}
for (size_t i = 0; i < ctxs_bufs.size(); ++i) {
auto & sh = dev_shadows[i];
if (!sh.releasable) {
continue;
}
ggml_context * ctx = ctxs_bufs[i].first.get();
ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, sh.buft);
if (buf == nullptr) {
LLAMA_LOG_ERROR("%s: failed to reallocate recurrent device buffer (out of VRAM?)\n", __func__);
return false;
}
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) { continue; }
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_set(t, sh.data.data() + off, 0, n);
off += n;
}
ctxs_bufs[i].second.reset(buf);
}
dev_released = false;
dev_shadows.clear();
return true;
}
size_t llama_memory_recurrent::size_r_bytes() const {
size_t size_r_bytes = 0;
+14
View File
@@ -66,6 +66,10 @@ public:
void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
void state_read (llama_io_read_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
// on-demand device (VRAM) residency (see llama_memory_i)
void release_device_buffers() override;
bool restore_device_buffers() override;
uint32_t head = 0; // the location where the batch will be placed in the cache (see find_slot())
uint32_t size = 0; // total number of cells, shared across all sequences
uint32_t used = 0; // used cells (i.e. at least one seq_id)
@@ -123,6 +127,16 @@ private:
// ggml contexts for the KV cache along with the allocated backend buffers:
std::vector<std::pair<ggml_context_ptr, ggml_backend_buffer_ptr>> ctxs_bufs;
// on-demand device eviction (see release_device_buffers): host shadow of each device buffer's
// live contents (recaptured on every release since the recurrent state is read-write)
struct device_buffer_shadow {
ggml_backend_buffer_type_t buft = nullptr;
bool releasable = false;
std::vector<uint8_t> data;
};
std::vector<device_buffer_shadow> dev_shadows; // parallel to ctxs_bufs
bool dev_released = false;
size_t total_size() const;
size_t size_r_bytes() const;
+10
View File
@@ -124,6 +124,16 @@ struct llama_memory_i {
virtual void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const = 0;
virtual void state_read (llama_io_read_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) = 0;
//
// on-demand device (VRAM) residency
//
// Free this memory's device (VRAM) buffers to a host shadow and rebuild them on demand, to
// hand VRAM to another model while keeping the cached data (no re-prefill). The caller must
// have synchronized the backend first. Default: no-op (the memory stays resident).
virtual void release_device_buffers() {}
virtual bool restore_device_buffers() { return true; }
};
using llama_memory_ptr = std::unique_ptr<llama_memory_i>;
+125 -1
View File
@@ -1160,6 +1160,16 @@ struct llama_model::impl {
// contexts where the model tensors metadata is stored as well as the corresponding buffers:
std::vector<std::pair<ggml_context_ptr, std::vector<ggml_backend_buffer_ptr>>> ctxs_bufs;
// on-demand device (VRAM) weight residency: per-ctxs_bufs slot describing whether the
// device weight buffer can be freed and rebuilt from a host shadow (see release/restore_device_weights)
struct weight_release_slot {
ggml_backend_buffer_type_t buft = nullptr; // buffer type to reallocate on restore
bool releasable = false; // single-buffer alloc-path device (VRAM) buffer
bool resident = true; // currently allocated on device
std::vector<uint8_t> shadow; // host copy, captured lazily on first release
};
std::vector<weight_release_slot> weight_release; // parallel to ctxs_bufs
buft_list_t cpu_buft_list;
std::map<ggml_backend_dev_t, buft_list_t> gpu_buft_list;
@@ -1742,8 +1752,9 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
// a lazy context is mapped whatever the load mode, but the memory-fit pass maps nothing
const bool is_lazy_mapped = ctx_key.lazy && !ml.no_alloc;
const bool host_ptr_path = (ml.use_mmap || is_lazy_mapped) && use_mmap_buffer && buffer_from_host_ptr_supported && is_default_buft;
if ((ml.use_mmap || is_lazy_mapped) && use_mmap_buffer && buffer_from_host_ptr_supported && is_default_buft) {
if (host_ptr_path) {
GGML_ASSERT(!ml.no_alloc);
for (uint32_t idx = 0; idx < ml.files.size(); idx++) {
// only the mmap region containing the tensors in the model is mapped to the backend buffer
@@ -1797,6 +1808,23 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
pimpl->ctxs_bufs.emplace_back(std::move(ctx_ptr), std::move(bufs));
// record on-demand release metadata (parallel to ctxs_bufs). Only the single-buffer
// alloc path on a device (non-host) buffer can be freed and rebuilt from a host shadow;
// the host_ptr path (mmap / lazy), the no_alloc dummy buffers and host (CPU) buffers are
// left resident.
{
llama_model::impl::weight_release_slot slot;
auto & last_bufs = pimpl->ctxs_bufs.back().second;
if (!host_ptr_path && !ml.no_alloc && last_bufs.size() == 1) {
ggml_backend_buffer_t b = last_bufs[0].get();
if (b != nullptr && !ggml_backend_buffer_is_host(b)) {
slot.releasable = true;
slot.buft = buft;
}
}
pimpl->weight_release.push_back(std::move(slot));
}
ctx_buf_maps.emplace_back(ctx, buf_map);
}
@@ -1859,6 +1887,97 @@ ggml_tensor * llama_model_base::create_tensor(llama_model_loader & ml, const LLM
tn, ne, flags);
}
void llama_model::release_device_weights() const {
// NOTE: the caller is responsible for synchronizing the backend scheduler first, so that no
// compute is in flight referencing these buffers when they are freed.
size_t freed = 0;
for (size_t i = 0; i < pimpl->ctxs_bufs.size(); ++i) {
auto & slot = pimpl->weight_release[i];
if (!slot.releasable || !slot.resident) {
continue;
}
ggml_context * ctx = pimpl->ctxs_bufs[i].first.get();
ggml_backend_buffer_t buf = pimpl->ctxs_bufs[i].second[0].get();
const size_t sz = ggml_backend_buffer_get_size(buf);
// lazily capture a host shadow of the weights (read-only, so captured only once).
// Store tensor bytes compactly in stable iteration order (independent of buffer layout /
// alignment padding) so restore does not depend on the re-allocated offsets matching.
// View tensors alias their base, so they are skipped (restored implicitly via their base).
if (slot.shadow.empty()) {
size_t total = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) continue;
total += ggml_nbytes(t);
}
slot.shadow.resize(total);
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) continue;
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_get(t, slot.shadow.data() + off, 0, n);
off += n;
}
}
// free the device (VRAM) buffer and clear the now-dangling tensor pointers so that
// ggml_backend_alloc_ctx_tensors_from_buft reallocates them cleanly on restore
pimpl->ctxs_bufs[i].second[0].reset();
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
t->buffer = nullptr;
t->data = nullptr;
}
slot.resident = false;
freed += sz;
}
if (freed > 0) {
LLAMA_LOG_INFO("%s: released %.2f MiB of device weights\n", __func__, freed / 1024.0 / 1024.0);
}
}
bool llama_model::restore_device_weights() const {
size_t restored = 0;
for (size_t i = 0; i < pimpl->ctxs_bufs.size(); ++i) {
auto & slot = pimpl->weight_release[i];
if (!slot.releasable || slot.resident) {
continue;
}
ggml_context * ctx = pimpl->ctxs_bufs[i].first.get();
ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, slot.buft);
if (buf == nullptr) {
LLAMA_LOG_ERROR("%s: failed to reallocate device weight buffer (out of VRAM?)\n", __func__);
return false;
}
ggml_backend_buffer_set_usage(buf, GGML_BACKEND_BUFFER_USAGE_WEIGHTS);
// re-upload weights from the compact host shadow, using the same stable iteration order
// and view-skipping as the capture above
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) continue;
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_set(t, slot.shadow.data() + off, 0, n);
off += n;
}
pimpl->ctxs_bufs[i].second[0].reset(buf);
slot.resident = true;
restored += ggml_backend_buffer_get_size(buf);
}
if (restored > 0) {
LLAMA_LOG_INFO("%s: restored %.2f MiB of device weights\n", __func__, restored / 1024.0 / 1024.0);
}
return true;
}
bool llama_model::weights_resident() const {
for (const auto & slot : pimpl->weight_release) {
if (slot.releasable && !slot.resident) {
return false;
}
}
return true;
}
std::string llama_model::arch_name() const {
return llm_arch_name(arch);
}
@@ -1911,6 +2030,11 @@ std::map<ggml_backend_buffer_type_t, size_t> llama_model::memory_breakdown() con
ret[buft] += ggml_backend_alloc_ctx_tensors_from_buft_size(ctx.get(), buft);
} else {
for (const auto & buf : bufs) {
// buf may be null if the device weights were released for on-demand VRAM sharing
// (see release_device_weights); skip it in the breakdown
if (!buf) {
continue;
}
// GGML_ASSERT(ggml_backend_buffer_get_base(buf.get()) != nullptr); // multi_buffer does not have a defined base
ret[ggml_backend_buffer_get_type(buf.get())] += ggml_backend_buffer_get_size(buf.get());
}
+8
View File
@@ -746,6 +746,14 @@ struct llama_model {
const struct ggml_tensor * get_tensor(const char * name) const;
// on-demand device (VRAM) weight residency: free the GPU weight buffers (keeping a host
// shadow) and rebuild them on demand. Used to time-share VRAM between models without
// reloading or losing the KV cache. release_device_weights() requires the caller to have
// synchronized the backend scheduler first.
void release_device_weights() const;
bool restore_device_weights() const;
bool weights_resident() const;
float get_rope_freq_base (const llama_cparams & cparams, int il) const;
float get_rope_freq_scale(const llama_cparams & cparams, int il) const;
+75
View File
@@ -159,6 +159,13 @@ struct clip_ctx {
ggml_backend_t backend_cpu = nullptr;
ggml_backend_buffer_ptr buf;
// on-demand device (VRAM) residency: the vision/audio encoder weights are read-only, so a host
// shadow is captured once and the device buffer can be freed while the model is cold, then
// rebuilt on wake (mirrors llama_model::release_device_weights). See clip_release_device().
ggml_backend_buffer_type_t dev_buft = nullptr;
std::vector<uint8_t> dev_shadow;
bool dev_released = false;
int max_nodes = 8192;
ggml_backend_sched_ptr sched;
@@ -4026,6 +4033,74 @@ void clip_free(clip_ctx * ctx) {
delete ctx;
}
void clip_release_device(struct clip_ctx * ctx) {
if (ctx == nullptr || ctx->dev_released || ctx->no_alloc) {
return;
}
ggml_backend_buffer_t buf = ctx->buf.get();
if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
return; // CPU-backed encoder: nothing in VRAM to free
}
// ensure no encode is in flight before freeing the weights
if (ctx->sched) {
ggml_backend_sched_synchronize(ctx->sched.get());
}
ctx->dev_buft = ggml_backend_buffer_get_type(buf);
ggml_context * cd = ctx->ctx_data.get();
// capture the host shadow once (weights are read-only): compact, stable iteration order,
// skipping view tensors (which alias a base and are restored implicitly)
if (ctx->dev_shadow.empty()) {
size_t total = 0;
for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
if (t->view_src == nullptr) { total += ggml_nbytes(t); }
}
ctx->dev_shadow.resize(total);
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
if (t->view_src != nullptr) { continue; }
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_get(t, ctx->dev_shadow.data() + off, 0, n);
off += n;
}
}
// free the device buffer and clear the now-dangling tensor pointers so restore reallocates cleanly
ctx->buf.reset();
for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
t->buffer = nullptr;
t->data = nullptr;
}
ctx->dev_released = true;
}
bool clip_restore_device(struct clip_ctx * ctx) {
if (ctx == nullptr || !ctx->dev_released) {
return true;
}
ggml_context * cd = ctx->ctx_data.get();
ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(cd, ctx->dev_buft);
if (buf == nullptr) {
LOG_ERR("%s: failed to reallocate encoder device buffer (out of VRAM?)\n", __func__);
return false;
}
ggml_backend_buffer_set_usage(buf, GGML_BACKEND_BUFFER_USAGE_WEIGHTS);
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(cd); t != nullptr; t = ggml_get_next_tensor(cd, t)) {
if (t->view_src != nullptr) { continue; }
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_set(t, ctx->dev_shadow.data() + off, 0, n);
off += n;
}
ctx->buf.reset(buf);
ctx->dev_released = false;
return true;
}
bool clip_weights_resident(const struct clip_ctx * ctx) {
return ctx == nullptr || !ctx->dev_released;
}
const char * clip_patch_merge_type(const struct clip_ctx * ctx) {
return ctx->model.hparams.mm_patch_merge_type == PATCH_MERGE_SPATIAL_UNPAD ? "spatial_unpad" : "flat";
}
+6
View File
@@ -70,6 +70,12 @@ struct clip_init_result clip_init(const char * fname, struct clip_context_params
void clip_free(struct clip_ctx * ctx);
// on-demand device (VRAM) residency: free/rebuild the encoder weight buffer to shrink an idle
// (cold) multimodal model's VRAM footprint. No-op for a CPU-backed encoder. See clip.cpp.
void clip_release_device(struct clip_ctx * ctx);
bool clip_restore_device(struct clip_ctx * ctx);
bool clip_weights_resident(const struct clip_ctx * ctx);
// TODO: should be enum, not string
const char * clip_patch_merge_type(const struct clip_ctx * ctx);
+18
View File
@@ -1116,6 +1116,24 @@ std::vector<std::vector<const mtmd_bitmap *>> mtmd_group_mergeable_bitmaps(std::
return output;
}
void mtmd_release_device(mtmd_context * ctx) {
if (ctx == nullptr) {
return;
}
if (ctx->ctx_v) { clip_release_device(ctx->ctx_v); }
if (ctx->ctx_a) { clip_release_device(ctx->ctx_a); }
}
bool mtmd_restore_device(mtmd_context * ctx) {
if (ctx == nullptr) {
return true;
}
bool ok = true;
if (ctx->ctx_v) { ok = clip_restore_device(ctx->ctx_v) && ok; }
if (ctx->ctx_a) { ok = clip_restore_device(ctx->ctx_a) && ok; }
return ok;
}
struct mtmd_tokenizer {
const mtmd_context * ctx;
+6
View File
@@ -136,6 +136,12 @@ MTMD_API mtmd_context * mtmd_init_from_file(const char * mmproj_fname,
MTMD_API void mtmd_free(mtmd_context * ctx);
// on-demand device (VRAM) residency: free / rebuild the vision+audio encoder weight buffers so an
// idle (cold) multimodal model releases its encoder VRAM and reclaims it on wake. No-op for a
// CPU-backed encoder. Restore returns false if reallocation failed (out of VRAM).
MTMD_API void mtmd_release_device(mtmd_context * ctx);
MTMD_API bool mtmd_restore_device(mtmd_context * ctx);
// whether we need to set non-causal mask before llama_decode
// if chunk is nullptr, we assume the default case where chunk is an image chunk
MTMD_API bool mtmd_decode_use_non_causal(const mtmd_context * ctx, const mtmd_input_chunk * chunk);
+317 -2
View File
@@ -26,6 +26,8 @@
#include <random>
#include <utility>
#include <fstream>
#include <thread>
#include <atomic>
// fix problem with std::min and std::max
#if defined(_WIN32)
@@ -36,6 +38,16 @@
#include <windows.h>
#endif
// POSIX file locking for the cross-process VRAM arbiter (see vram_share_* below)
#if !defined(_WIN32)
#include <sys/file.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <poll.h>
#include <cerrno>
#endif
constexpr int HTTP_POLLING_SECONDS = 1;
static common_speculative_output_limits server_output_limits(const common_params & params) {
@@ -817,7 +829,36 @@ static int process_mtmd_chunk(const server_slot & slot, mtmd::batch_ptr & mbatch
// TODO @ngxson : move this log line to debug when it become more stable
SLT_TRC(slot, "encoding mtmd batch from idx = %zu, n_chunks = %d\n", idx, n_added);
// Bring the vision/audio encoder into VRAM just for this encode. In on-demand mode the
// encoder is normally kept in RAM (its VRAM freed for KV / expert cache). If it does not
// fit, evict the LLM backbone weights first: they are NOT needed while the encoder runs (the
// decode that uses them happens afterwards) and their host shadow is read-only, so this is a
// cheap free (no D2H) and leaves the KV cache / prompt cache untouched.
static const bool mmproj_ondemand = getenv("LLAMA_MMPROJ_ONDEMAND") != nullptr;
bool weights_evicted = false;
if (mctx && !mtmd_restore_device(mctx)) {
if (mmproj_ondemand) {
llama_context_release_device(slot.ctx_tgt, /* evict_kv = */ false); // free backbone, keep KV
weights_evicted = true;
}
if (!mtmd_restore_device(mctx)) {
if (weights_evicted) { llama_context_restore_device(slot.ctx_tgt); }
SLT_ERR(slot, "%s", "failed to bring the multimodal encoder into VRAM for encoding\n");
return -1;
}
}
res = mtmd_batch_encode(mbatch.get());
// release the encoder VRAM again (on-demand), then restore the backbone weights for decode.
// Order matters: free the encoder BEFORE re-uploading the weights so the peak stays within VRAM.
if (mctx && mmproj_ondemand) {
mtmd_release_device(mctx);
}
if (weights_evicted) {
llama_context_restore_device(slot.ctx_tgt);
}
if (res != 0) {
SLT_ERR(slot, "failed to encode mtmd batch for chunk idx = %zu, res = %d\n", idx, res);
return -1;
@@ -857,6 +898,8 @@ public:
}
~server_context_impl() {
// release the VRAM token before tearing anything down
vram_share_shutdown();
if (!sleeping) {
// destroy() is already called when entering sleeping state
// we don't call it again here to avoid double free
@@ -888,6 +931,229 @@ private:
llama_model * model_dft = nullptr;
llama_context * ctx_dft = nullptr;
// Cross-process VRAM arbiter: when LLAMA_SLEEP_VRAM_ONLY is set, several always-loaded
// llama-server processes time-share one GPU. A single flock() on <arena>/token.lock is the
// baton: "resident (weights in VRAM) iff I hold the token". A model warms up (locks + restores
// weights) right before it decodes, and goes cold (releases weights + unlocks) when it idles,
// so only one model holds VRAM at a time and the KV cache is never evicted (no re-prefill).
bool vram_only = false; // release-mode sleep active (LLAMA_SLEEP_VRAM_ONLY)
bool vram_evict_kv = false; // also evict the KV cache to host on release (LLAMA_SLEEP_EVICT_KV)
bool vram_flock = false; // cross-process flock coordination available
std::atomic<bool> vram_cold{true}; // weights currently released
int vram_lock_fd = -1; // fd for <arena>/token.lock
// "want" lock: a process that waits for the token holds <arena>/want.lock shared, and drops it
// as soon as it owns the token. The holder probes that lock with a non-blocking exclusive
// flock, which fails only while somebody waits. This is level-triggered state owned by the
// kernel, so there is no event to miss and nothing goes stale if a waiter dies.
// The probe needs its own fd: flock() treats two open file descriptions of one file
// independently, so probing on the waiter fd would convert our own lock instead of conflicting.
int vram_want_fd = -1; // waiter side, LOCK_SH while waiting
int vram_want_probe_fd = -1; // probe side, LOCK_EX | LOCK_NB only
bool vram_want_held = false; // only touched by the waiting thread
// open the shared arena (token.lock + want.lock). Idempotent; sets vram_only/vram_flock.
void vram_arena_open() {
if (getenv("LLAMA_SLEEP_VRAM_ONLY") == nullptr) {
return;
}
vram_only = true;
vram_evict_kv = getenv("LLAMA_SLEEP_EVICT_KV") != nullptr;
#if !defined(_WIN32)
if (vram_lock_fd >= 0) {
return; // already open
}
const char * arena_env = getenv("LLAMA_VRAM_ARENA");
const std::string arena = arena_env ? arena_env : "/dev/shm/llama-vram";
mkdir(arena.c_str(), 0777);
const std::string lock_path = arena + "/token.lock";
vram_lock_fd = open(lock_path.c_str(), O_RDWR | O_CREAT, 0666);
if (vram_lock_fd >= 0) {
vram_flock = true;
const std::string want_path = arena + "/want.lock";
vram_want_fd = open(want_path.c_str(), O_RDWR | O_CREAT, 0666);
vram_want_probe_fd = open(want_path.c_str(), O_RDWR | O_CREAT, 0666);
if (vram_want_fd < 0 || vram_want_probe_fd < 0) {
SRV_WRN("VRAM arbiter: cannot open %s, the holder will only release on idle\n", want_path.c_str());
}
SRV_INF("VRAM arbiter: cross-process GPU sharing via %s (want %s)\n",
lock_path.c_str(), want_path.c_str());
} else {
SRV_WRN("VRAM arbiter: cannot open %s, running VRAM-only sleep without cross-process lock\n", lock_path.c_str());
}
#endif
}
// Acquire the VRAM token BEFORE uploading this model's weights, so any model currently resident
// on the GPU releases first and the load uploads into free VRAM instead of racing it (which
// could OOM, e.g. the 4B task model holding VRAM while a large model loads). Blocks until free.
// Called from load_model() right before common_init_from_params().
void vram_acquire_for_load() {
vram_arena_open();
if (!vram_only) {
return;
}
if (!vram_take_token()) {
SRV_WRN("%s", "VRAM arbiter: loading weights without the GPU token\n");
}
vram_cold = false; // we hold the token; weights will be resident after the upload
}
// Called from init() after the (coordinated) load. The model stays warm (holding the token
// acquired in vram_acquire_for_load) and serves its first request without a re-warm; the queue
// releases it once another model registers on want.lock.
void vram_share_init() {
vram_arena_open();
if (!vram_only) {
return;
}
vram_cold = false; // resident and holding the token after the coordinated load
#if !defined(_WIN32)
if (vram_flock && vram_want_probe_fd >= 0) {
queue_tasks.on_should_yield([this]{ return vram_should_yield(); });
}
#endif
}
// true while another process waits for the token and we still hold it. Called by should_sleep()
// on the start_loop() thread, under mutex_tasks, and only when the queue is idle - so a release
// can never happen underneath an in-flight decode, and there is no event to latch: the answer
// comes from live kernel state at the moment the loop asks.
bool vram_should_yield() {
#if !defined(_WIN32)
if (!vram_flock || vram_cold || vram_want_probe_fd < 0) {
return false; // we hold no token to give away
}
// never blocks: LOCK_NB only, and we drop the probe lock at once if we win it
if (flock(vram_want_probe_fd, LOCK_EX | LOCK_NB) == 0) {
flock(vram_want_probe_fd, LOCK_UN);
return false; // nobody wants the GPU
}
return errno == EWOULDBLOCK || errno == EAGAIN;
#else
return false;
#endif
}
// register as a waiter, so the current token holder sees us and yields. Shared, so several
// waiters coexist. The kernel drops it if we die, so it can never strand the holder.
void vram_want_acquire() {
#if !defined(_WIN32)
if (vram_want_fd < 0 || vram_want_held) {
return;
}
while (flock(vram_want_fd, LOCK_SH) != 0) {
if (errno != EINTR) { // our signal handlers run without SA_RESTART
SRV_WRN("VRAM arbiter: cannot take want.lock (errno %d)\n", errno);
return;
}
}
vram_want_held = true;
#endif
}
// stop asking for the token. Must run on every path out of vram_take_token(), or the holder
// yields forever.
void vram_want_release() {
#if !defined(_WIN32)
if (vram_want_fd < 0 || !vram_want_held) {
return;
}
flock(vram_want_fd, LOCK_UN);
vram_want_held = false;
#endif
}
// take the shared VRAM token: register on want.lock, then wait for token.lock. want.lock is
// level-triggered state, so the holder still learns of us if it is loading or restoring when we
// start to wait. No timeout: the holder can be mid-generation, and decoding without the token
// risks an OOM that kills both processes.
bool vram_take_token() {
#if !defined(_WIN32)
if (!vram_flock) {
return true; // no cross-process lock: there is no token to take
}
vram_want_acquire();
bool logged = false;
while (flock(vram_lock_fd, LOCK_EX | LOCK_NB) != 0) {
if (errno == EWOULDBLOCK || errno == EAGAIN) {
if (!logged) {
SRV_INF("%s", "VRAM arbiter: waiting for the GPU token\n");
logged = true;
}
flock(vram_lock_fd, LOCK_EX); // blocking: the kernel grants it the moment the holder unlocks
continue; // re-check with LOCK_NB, which also succeeds on a lock we already own
}
if (errno == EINTR) {
continue; // our signal handlers run without SA_RESTART
}
SRV_WRN("VRAM arbiter: flock failed (errno %d)\n", errno);
vram_want_release();
return false;
}
vram_want_release();
return true;
#else
return true;
#endif
}
// acquire the VRAM token (blocking) and bring weights back to the device. Called on the loop
// thread right before a decode, so it never races compute.
void vram_ensure_warm() {
if (!vram_only || !vram_cold) {
return;
}
if (!vram_take_token()) {
SRV_WRN("%s", "VRAM arbiter: restoring weights without the GPU token\n");
}
llama_context_restore_device(ctx_tgt);
if (ctx_dft != nullptr) {
llama_context_restore_device(ctx_dft);
}
// NOTE: the multimodal (vision/audio) encoder is intentionally NOT restored here. It is
// brought into VRAM just-in-time before an image/audio encode (process_mtmd_chunk) and, in
// on-demand mode, released again right after - so a warm text-only model holds no encoder
// VRAM (that space is free for KV / expert cache). A cold->warm wake for a *text* request
// therefore leaves the encoder in RAM; an image request restores it at encode time.
// NOTE: keep this AFTER the restore, so the loop cannot give away the token while we still
// bring the weights up. We dropped want.lock when we took the token, so the next probe
// reports only the waiters that are still there.
vram_cold = false;
}
void vram_share_shutdown() {
#if !defined(_WIN32)
vram_want_release();
if (vram_want_probe_fd >= 0) { close(vram_want_probe_fd); vram_want_probe_fd = -1; }
if (vram_want_fd >= 0) { close(vram_want_fd); vram_want_fd = -1; }
if (vram_lock_fd >= 0) { flock(vram_lock_fd, LOCK_UN); close(vram_lock_fd); vram_lock_fd = -1; }
#endif
}
// release the device weights (keeping KV + host shadow) and drop the VRAM token so another
// model can warm up. Called on the loop thread when the server goes idle.
void vram_go_cold() {
if (!vram_only || vram_cold) {
return;
}
llama_context_release_device(ctx_tgt, vram_evict_kv);
if (ctx_dft != nullptr) {
llama_context_release_device(ctx_dft, vram_evict_kv);
}
// also release the multimodal (vision/audio) encoder weights (dead weight while cold);
// its read-only host shadow is captured once and rebuilt on wake by vram_ensure_warm()
if (mctx != nullptr) {
mtmd_release_device(mctx);
}
#if !defined(_WIN32)
if (vram_flock) {
flock(vram_lock_fd, LOCK_UN);
}
#endif
vram_cold = true;
}
common_speculative_init_result_ptr spec_init;
common_context_seq_rm_type ctx_tgt_seq_rm_type = COMMON_CONTEXT_SEQ_RM_TYPE_NO;
@@ -953,6 +1219,29 @@ private:
void handle_sleeping_state(bool new_state) {
GGML_ASSERT(sleeping != new_state);
// Lightweight VRAM-only sleep: instead of a full unload/reload (which frees RAM and the
// KV cache and pays a full reload on wake), just release the model's device (VRAM) weight
// buffers to a host shadow (keeping the context, KV cache and host weights) and drop the
// shared VRAM token. Waking is handled lazily by vram_ensure_warm() right before the next
// decode (which re-acquires the token first), so a single GPU is time-shared between models
// with no re-prefill.
if (vram_only && ctx_tgt != nullptr) {
if (new_state) {
SRV_INF("%s", "server entering sleeping state (VRAM-only: releasing device weights, keeping KV cache)\n");
vram_go_cold();
} else {
SRV_INF("%s", "server exiting sleeping state (VRAM-only: restoring device weights/KV)\n");
// Restore NOW, on wake, before update_slots runs. update_slots touches the KV cache
// (e.g. SWA checkpoint creation reads it via ggml_backend_tensor_get) before the
// decode-time vram_ensure_warm(), so with KV eviction the KV must already be resident
// here or those reads hit a freed (null) buffer.
vram_ensure_warm();
}
sleeping = new_state;
return;
}
if (new_state) {
if (callback_state) {
callback_state(SERVER_STATE_SLEEPING, {});
@@ -1096,6 +1385,11 @@ private:
params_base.load_progress_callback_user_data = &load_progress_text;
}
// VRAM arbiter: acquire the GPU token before uploading weights, so any resident model (e.g.
// the warm 4B task model) releases first and this load uploads into free VRAM instead of
// racing it and OOM-ing. No-op unless LLAMA_SLEEP_VRAM_ONLY is set.
vram_acquire_for_load();
llama_init = common_init_from_params(params_base);
model_tgt = llama_init->model();
@@ -1171,6 +1465,13 @@ private:
init_opt.video_params.ffmpeg_bin_dir = params_base.video_ffmpeg_bin_dir.empty()
? nullptr : params_base.video_ffmpeg_bin_dir.c_str();
// on-demand encoder: keep the vision/audio encoder weights in RAM (out of VRAM) until an
// image/audio actually needs encoding, freeing that VRAM for KV / expert cache on the
// common text-only path. process_mtmd_chunk() brings the encoder in just-in-time.
if (getenv("LLAMA_MMPROJ_ONDEMAND") != nullptr) {
mtmd_release_device(mctx);
}
if (params_base.ctx_shift) {
params_base.ctx_shift = false;
SRV_WRN("%s\n", "ctx_shift is not supported by multimodal, it will be disabled");
@@ -1415,6 +1716,11 @@ private:
handle_sleeping_state(sleeping);
});
// VRAM arbiter: arm the yield probe. The load was coordinated (vram_acquire_for_load
// grabbed the token before uploading), so we stay warm holding the token and serve the
// first request without a re-warm; the loop releases it when another model waits for it.
vram_share_init();
metrics.init();
if (params_base.cache_idle_slots) {
@@ -3675,6 +3981,11 @@ private:
has_output |= batch.tokens[i].output;
}
// VRAM arbiter: make sure our weights are resident (acquiring the shared GPU token first)
// before we decode. Runs on the loop thread before entering yield_to_queue(), so it never
// races an in-flight decode.
vram_ensure_warm();
// yield to the queue, so we can still handle metrics tasks while decoding
// note: the sync is done here too, so that the wait is also covered by the yield
int ret = 0;
@@ -4230,8 +4541,12 @@ struct server_res_generator : server_res_spipe {
server_response_reader rd;
server_res_generator(server_queue & queue_tasks, server_response & queue_results, int sleep_idle_seconds, bool bypass_sleep = false)
: rd(queue_tasks, queue_results, HTTP_POLLING_SECONDS) {
// fast path in case sleeping is disabled
bypass_sleep |= sleep_idle_seconds < 0;
// fast path in case sleeping is disabled. Note: the VRAM arbiter (LLAMA_SLEEP_VRAM_ONLY)
// can put the server to sleep to hand over the GPU token even when idle-sleep is disabled
// (sleep_idle_seconds < 0), so in that case requests must still wake it - otherwise an
// arbiter-slept server would hang, never returning from a request.
static const bool vram_arbiter = getenv("LLAMA_SLEEP_VRAM_ONLY") != nullptr;
bypass_sleep |= (sleep_idle_seconds < 0 && !vram_arbiter);
if (!bypass_sleep) {
queue_tasks.wait_until_no_sleep();
}
+26 -13
View File
@@ -114,18 +114,16 @@ void server_queue::pop_deferred_task(int id_slot) {
void server_queue::wait_until_no_sleep() {
std::unique_lock<std::mutex> lock(mutex_tasks);
if (!sleeping) {
return;
} else {
// re-ask on every wake: the loop clears req_stop_sleeping on the way into sleep, so a loop that
// goes back to sleep before we run would strand us here forever
while (sleeping) {
if (!req_stop_sleeping) {
QUE_DBG("%s", "requesting to stop sleeping\n");
req_stop_sleeping = true;
condition_tasks.notify_one(); // only main thread is waiting on this
condition_tasks.notify_all(); // other threads may wait on this too
}
QUE_DBG("%s", "waiting until no sleep\n");
condition_tasks.wait(lock, [&]{
return !sleeping;
});
condition_tasks.wait(lock);
}
}
@@ -286,9 +284,18 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
worker.yielding = false;
worker.thread = std::thread([this]() { worker_loop(); });
constexpr auto max_wait_time = std::chrono::seconds(1);
// the arbiter predicate is read from live state, so this timeout bounds how fast we notice a
// process waiting for the VRAM token
const auto max_wait_time = should_yield_cb ? std::chrono::milliseconds(10) : std::chrono::milliseconds(1000);
// after a wake, keep the VRAM for at least this long: the request that woke us is not in the
// queue yet, so yielding at once would only send it through another wake
constexpr int64_t yield_grace_ms = 100;
int64_t time_last_wake = 0;
auto should_sleep = [&]() -> bool {
// caller must hold mutex_tasks
if (should_yield_cb && ggml_time_ms() - time_last_wake >= yield_grace_ms && should_yield_cb()) {
return true; // another process waits for the VRAM token - release now
}
if (idle_sleep_ms < 0) {
return false;
}
@@ -332,9 +339,11 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
cb(true);
}
req_stop_sleeping = false;
// wait until we are requested to exit sleeping state
// wait until we are requested to exit sleeping state, or a task arrives: post() only
// notifies, so a task queued right after wait_until_no_sleep() saw us awake must be
// able to wake us by itself, else it waits here for an unrelated request
condition_tasks.wait(lock, [&]{
return (!running || req_stop_sleeping);
return (!running || req_stop_sleeping || !queue_tasks.empty());
});
if (!running) { // may changed during sleep
break; // terminate
@@ -347,17 +356,21 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
}
sleeping = false;
time_last_task = ggml_time_ms();
time_last_wake = time_last_task;
condition_tasks.notify_all(); // notify wait_until_no_sleep()
break; // process new tasks
} else {
// wait for new tasks or timeout for checking sleeping condition
// wait for new tasks, or timeout for checking sleeping condition
bool res = condition_tasks.wait_for(lock, max_wait_time, [&]{
return (!queue_tasks.empty() || !running);
});
if (res) {
if (res && !queue_tasks.empty()) {
break; // new task arrived or terminate
}
// otherwise, loop again to check sleeping condition
if (!running) {
break;
}
// otherwise (timeout), loop again to re-check should_sleep
}
}
}
+9
View File
@@ -45,6 +45,7 @@ private:
std::function<bool(server_task &&, bool)> callback_new_task;
std::function<void(void)> callback_update_slots;
std::vector<std::function<void(bool)>> callback_sleeping_state;
std::function<bool()> should_yield_cb;
public:
~server_queue() { worker_stop(); }
@@ -127,6 +128,14 @@ public:
callback_update_slots = std::move(callback);
}
// Register a predicate asking the loop to sleep (release VRAM) as soon as it is idle - used by
// the VRAM arbiter to check whether another process waits for the GPU token.
// Called on the start_loop() thread while holding mutex_tasks, so it must not block or post
// tasks. While it is set, the idle wait polls faster, so the loop reacts without a notify.
void on_should_yield(std::function<bool()> callback) {
should_yield_cb = std::move(callback);
}
// Register callback for sleeping state change; multiple callbacks are allowed
// for example: register order cb0, cb1, cb2
// entering sleep: queue.sleeping = true --> cb0(true) --> cb1(true) --> cb2(true)