server: on-demand VRAM sharing to time-share one GPU between models
Add release/restore of a model's GPU weight buffers (keeping a host shadow and the KV cache) so several always-loaded llama-server processes can time-share a single GPU without reloading or losing the prompt cache. - llama-model: release_device_weights()/restore_device_weights() capture a compact host shadow (stable iteration order, view-skipping) and free then realloc the device weight buffers; weights_resident() query. - llama-context: release_device()/restore_device() wrappers; decode() auto- restores; public C API llama_context_release_device/restore_device. - server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA enforces "resident iff holds token"; an inotify doorbell forces the holder to release on contention. The warden thread only touches the task queue, so releases run on the loop thread and never race a decode. Validated on RX 580 (Vulkan): two models share 8GB, never both resident, correct output under contention, KV cache preserved (no re-prefill). Assisted-by: Claude
This commit is contained in:
1 parent
cfda297cbb
commit
56a1010bbc
8 files changed
+400
-6
No files matched your search
@@ -18,6 +18,7 @@ private:
|
||||
bool running = false;
|
||||
bool sleeping = false;
|
||||
bool req_stop_sleeping = false;
|
||||
bool yield_requested = false; // set by request_yield() when another process wants the VRAM token
|
||||
int64_t time_last_task = 0;
|
||||
int n_inflight = 0; // requests that keep the server awake, see wait_until_no_sleep()
|
||||
|
||||
@@ -74,6 +75,11 @@ public:
|
||||
// let the server sleep again once the caller no longer uses the server state
|
||||
void release_inflight();
|
||||
|
||||
// request that the loop go to sleep (release VRAM) as soon as it is idle - called from the
|
||||
// VRAM-arbiter warden thread when another process rings the doorbell for the GPU token.
|
||||
// Thread-safe; wakes the loop so it releases promptly instead of at the next idle poll.
|
||||
void request_yield();
|
||||
|
||||
bool is_sleeping() {
|
||||
std::unique_lock<std::mutex> lock(mutex_tasks);
|
||||
return sleeping;
|
||||
|
||||
Reference in new issue
Block a user