server: on-demand VRAM sharing to time-share one GPU between models

Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.

- llama-model: release_device_weights()/restore_device_weights() capture a
  compact host shadow (stable iteration order, view-skipping) and free then
  realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
  restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
  (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
  enforces "resident iff holds token"; an inotify doorbell forces the holder
  to release on contention. The warden thread only touches the task queue, so
  releases run on the loop thread and never race a decode.

Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).

Assisted-by: Claude
This commit is contained in:
2026-07-24 22:32:29 +02:00
parent 2b94398ed7
commit be7f3b3172
8 changed files with 388 additions and 3 deletions
+6
View File
@@ -16,6 +16,7 @@ private:
bool running = false;
bool sleeping = false;
bool req_stop_sleeping = false;
bool yield_requested = false; // set by request_yield() when another process wants the VRAM token
int64_t time_last_task = 0;
// queues
@@ -51,6 +52,11 @@ public:
// returns immediately if not sleeping
void wait_until_no_sleep();
// request that the loop go to sleep (release VRAM) as soon as it is idle - called from the
// VRAM-arbiter warden thread when another process rings the doorbell for the GPU token.
// Thread-safe; wakes the loop so it releases promptly instead of at the next idle poll.
void request_yield();
bool is_sleeping() {
std::unique_lock<std::mutex> lock(mutex_tasks);
return sleeping;