server: on-demand VRAM sharing to time-share one GPU between models
Add release/restore of a model's GPU weight buffers (keeping a host shadow and the KV cache) so several always-loaded llama-server processes can time-share a single GPU without reloading or losing the prompt cache. - llama-model: release_device_weights()/restore_device_weights() capture a compact host shadow (stable iteration order, view-skipping) and free then realloc the device weight buffers; weights_resident() query. - llama-context: release_device()/restore_device() wrappers; decode() auto- restores; public C API llama_context_release_device/restore_device. - server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA enforces "resident iff holds token"; an inotify doorbell forces the holder to release on contention. The warden thread only touches the task queue, so releases run on the loop thread and never race a decode. Validated on RX 580 (Vulkan): two models share 8GB, never both resident, correct output under contention, KV cache preserved (no re-prefill). Assisted-by: Claude
This commit is contained in:
@@ -16,6 +16,7 @@ private:
|
||||
bool running = false;
|
||||
bool sleeping = false;
|
||||
bool req_stop_sleeping = false;
|
||||
bool yield_requested = false; // set by request_yield() when another process wants the VRAM token
|
||||
int64_t time_last_task = 0;
|
||||
|
||||
// queues
|
||||
@@ -51,6 +52,11 @@ public:
|
||||
// returns immediately if not sleeping
|
||||
void wait_until_no_sleep();
|
||||
|
||||
// request that the loop go to sleep (release VRAM) as soon as it is idle - called from the
|
||||
// VRAM-arbiter warden thread when another process rings the doorbell for the GPU token.
|
||||
// Thread-safe; wakes the loop so it releases promptly instead of at the next idle poll.
|
||||
void request_yield();
|
||||
|
||||
bool is_sleeping() {
|
||||
std::unique_lock<std::mutex> lock(mutex_tasks);
|
||||
return sleeping;
|
||||
|
||||
Reference in New Issue
Block a user