server: on-demand VRAM sharing to time-share one GPU between models
Add release/restore of a model's GPU weight buffers (keeping a host shadow and the KV cache) so several always-loaded llama-server processes can time-share a single GPU without reloading or losing the prompt cache. - llama-model: release_device_weights()/restore_device_weights() capture a compact host shadow (stable iteration order, view-skipping) and free then realloc the device weight buffers; weights_resident() query. - llama-context: release_device()/restore_device() wrappers; decode() auto- restores; public C API llama_context_release_device/restore_device. - server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA enforces "resident iff holds token"; an inotify doorbell forces the holder to release on contention. The warden thread only touches the task queue, so releases run on the loop thread and never race a decode. Validated on RX 580 (Vulkan): two models share 8GB, never both resident, correct output under contention, KV cache preserved (no re-prefill). Assisted-by: Claude
This commit is contained in:
@@ -57,6 +57,12 @@ struct llama_context {
|
||||
|
||||
void synchronize();
|
||||
|
||||
// on-demand device (VRAM) residency: free / rebuild the model's GPU weight buffers so that
|
||||
// VRAM can be time-shared with another model. release_device() synchronizes first; decode()
|
||||
// auto-restores when it finds the weights have been released.
|
||||
void release_device();
|
||||
void restore_device();
|
||||
|
||||
const llama_model & get_model() const;
|
||||
const llama_cparams & get_cparams() const;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user