llama: evict recurrent/SSM state on device release
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left resident when a model's device buffers were released for on-demand VRAM sharing - llama_memory_recurrent::release_device_buffers() was a no-op default. Implement it (and restore_device_buffers) with the same capture-host-shadow / free / reallocate pattern as llama_kv_cache, so llama_memory_hybrid now evicts both its attention KV and its recurrent state. The state is read-write, so its shadow is recaptured on every release. Assisted-by: Claude
This commit is contained in:
@@ -202,7 +202,7 @@ void llama_memory_hybrid::state_read(llama_io_read_i & io, llama_seq_id seq_id,
|
||||
}
|
||||
|
||||
void llama_memory_hybrid::release_device_buffers() {
|
||||
// evict the attention KV (grows with context); the recurrent state uses the no-op default
|
||||
// evict both the attention KV (grows with context) and the recurrent/SSM state
|
||||
mem_attn->release_device_buffers();
|
||||
mem_recr->release_device_buffers();
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user