llama: do not release CPU extra buffers (CPU_REPACK) on device release
A model with weights on the CPU device (-dev none / -ngl 0) segfaulted in
llama_model::release_device_weights() the first time the VRAM arbiter put
it to sleep: CPU_REPACK buffers are not host buffers, so load_tensors marked
them releasable, and capturing the shadow called their NULL get_tensor.
Reproduced with the production build 2fc9f9c35 (Qwen3.5-0.8B, -dev none
-ngl 0, LLAMA_SLEEP_VRAM_ONLY=1 LLAMA_SLEEP_EVICT_KV=1, force a yield with a
waiter on want.lock): exit 139 at the release, before any SIGTERM. The
faulting call (rip 0) returned into release_device_weights.
Only buffers on a non-CPU device are releasable now. These buffers are in RAM
already, so there is nothing to free. Production is not affected: every
arbiter model there has its weights on the GPU, and a releasable CPU_REPACK
buffer would have crashed it. Found by the --diskcache check, which runs
its arbiter cases CPU only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
71d4cbe195
commit
f24c3582c2
1 file changed
+3
-1
+3
-1
@@ -2026,7 +2026,9 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
|
||||
auto & last_bufs = pimpl->ctxs_bufs.back().second;
|
||||
if (!host_ptr_path && !ml.no_alloc && last_bufs.size() == 1) {
|
||||
ggml_backend_buffer_t b = last_bufs[0].get();
|
||||
if (b != nullptr && !ggml_backend_buffer_is_host(b)) {
|
||||
// CPU extra buffers (CPU_REPACK) are not host buffers, but they are in RAM and have no get_tensor
|
||||
ggml_backend_dev_t buft_dev = ggml_backend_buft_get_device(buft);
|
||||
if (b != nullptr && !ggml_backend_buffer_is_host(b) && (buft_dev == nullptr || ggml_backend_dev_type(buft_dev) != GGML_BACKEND_DEVICE_TYPE_CPU)) {
|
||||
slot.releasable = true;
|
||||
slot.buft = buft;
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user