server: restore weights/KV on wake, not at decode, so KV eviction is safe

With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.

Assisted-by: Claude
This commit is contained in:
2026-07-26 00:07:01 +02:00
parent deee33503b
commit 9573505011
+6 -2
View File
@@ -1159,8 +1159,12 @@ private:
SRV_INF("%s", "server entering sleeping state (VRAM-only: releasing device weights, keeping KV cache)\n"); SRV_INF("%s", "server entering sleeping state (VRAM-only: releasing device weights, keeping KV cache)\n");
vram_go_cold(); vram_go_cold();
} else { } else {
SRV_INF("%s", "server exiting sleeping state (VRAM-only: weights restored on next decode)\n"); SRV_INF("%s", "server exiting sleeping state (VRAM-only: restoring device weights/KV)\n");
// token is acquired and weights restored by vram_ensure_warm() before decode // Restore NOW, on wake, before update_slots runs. update_slots touches the KV cache
// (e.g. SWA checkpoint creation reads it via ggml_backend_tensor_get) before the
// decode-time vram_ensure_warm(), so with KV eviction the KV must already be resident
// here or those reads hit a freed (null) buffer.
vram_ensure_warm();
} }
sleeping = new_state; sleeping = new_state;
return; return;