Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

server: optionally evict the KV cache too (Phase 2 of VRAM sharing)

Browse Source 
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.

- llama_memory_i: add release_device_buffers()/restore_device_buffers()
  (default no-op). Implemented in llama_kv_cache (D2H shadow of the live
  ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
  llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
  buffers when requested; restore_device() rebuilds them. Public API
  llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
  since the KV shadow adds a D2H/H2D copy of the live cache each cycle.

Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.

Assisted-by: Claude
This commit is contained in:
Lumpiasty
2026-07-25 02:02:18 +02:00
parent be7f3b3172
commit 7398c40eda
11 changed files with 156 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/llama-context.cpp
+14 -6
View File
@@ -728,21 +728,29 @@ void llama_context::synchronize() {
t_compute_start_us = 0;
}
void llama_context::release_device() {
if (!model.weights_resident()) {
void llama_context::release_device(bool evict_kv) {
if (!model.weights_resident() && !(evict_kv && !kv_device_evicted)) {
return;
}
// ensure no compute is in flight before freeing the device buffers
synchronize();
model.release_device_weights();
if (evict_kv && memory && !kv_device_evicted) {
memory->release_device_buffers();
kv_device_evicted = true;
}
}
void llama_context::restore_device() {
if (model.weights_resident()) {
if (model.weights_resident() && !kv_device_evicted) {
return;
}
model.restore_device_weights();
// make sure all weight uploads have completed before any compute reads them
if (kv_device_evicted && memory) {
memory->restore_device_buffers();
kv_device_evicted = false;
}
// make sure all weight/KV uploads have completed before any compute reads them
if (sched) {
ggml_backend_sched_synchronize(sched.get());
}
@@ -3700,8 +3708,8 @@ void llama_synchronize(llama_context * ctx) {
ctx->synchronize();
}
void llama_context_release_device(llama_context * ctx) {
ctx->release_device();
void llama_context_release_device(llama_context * ctx, bool evict_kv) {
ctx->release_device(evict_kv);
}
void llama_context_restore_device(llama_context * ctx) {