server: restore weights/KV on wake, not at decode, so KV eviction is safe

With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.

Assisted-by: Claude
This commit is contained in:
Lumpiasty committed 2026-10-24 18:33:19 +02:00
1 parent 8d80f0673f
commit 2feb6aeb27
1 file changed
+6 -2
+6 -2
View File
@@ -1346,8 +1346,12 @@ private:
SRV_INF("%s", "server entering sleeping state (VRAM-only: releasing device weights, keeping KV cache)\n");
vram_go_cold();
} else {
SRV_INF("%s", "server exiting sleeping state (VRAM-only: weights restored on next decode)\n");
// token is acquired and weights restored by vram_ensure_warm() before decode
SRV_INF("%s", "server exiting sleeping state (VRAM-only: restoring device weights/KV)\n");
// Restore NOW, on wake, before update_slots runs. update_slots touches the KV cache
// (e.g. SWA checkpoint creation reads it via ggml_backend_tensor_get) before the
// decode-time vram_ensure_warm(), so with KV eviction the KV must already be resident
// here or those reads hit a freed (null) buffer.
vram_ensure_warm();
}
sleeping = new_state;
return;