llama: read a released KV/recurrent state from its host shadow

After llama_context_release_device(ctx, true) the KV cache and the recurrent
state live only in host shadows, and their tensors have no buffer, so
llama_state_seq_get_data cannot read them. The disk prompt cache must save
a cold server's slots at exit, and a shutting-down server must not restore
to the device (it may not hold the GPU token, F7).

llama_context_restore_memory_host() makes the state readable without the
device. restore_device_buffers(to_host = true) wraps each shadow as a
CPU_Mapped buffer (ggml_backend_cpu_buffer_from_ptr) and points the tensors
into it: there is no copy and no allocation. The first version copied into a
CPU buffer instead, which needs the full KV allocation a second time, about
2.7 GB for the 35B at ctx 131072; the node has had an OOM-controller kill
before. The buffer base is rounded down to the 32-byte alignment that
cpu_buffer_from_ptr asserts (a large std::vector is only 16-byte aligned);
those leading bytes are never accessed. With to_host the shadows are kept,
because the buffers point into them.

It sets device_discarded, so restore_device() aborts afterwards. Exit only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 18:55:57 +01:00
1 parent f24c3582c2
commit dd851fd0c9
13 files changed
+72 -15

No files matched your search

+20
View File
@@ -3,6 +3,7 @@
#include "llama-ext.h"
#include "llama-io.h"
#include <cstdint>
#include <set>
#include <vector>
@@ -90,3 +91,22 @@ std::vector<ggml_backend_buffer_t> llama_memory_get_buffers(const llama_memory_t
return std::vector<ggml_backend_buffer_t>(writer.buffers.begin(), writer.buffers.end());
}
ggml_backend_buffer_t llama_memory_wrap_host(ggml_context * ctx, std::vector<uint8_t> & data) {
// the buffer base must be aligned to 32 bytes; the bytes before data.data() are never accessed
uint8_t * base = (uint8_t *) ((uintptr_t) data.data() & ~(uintptr_t) 31);
ggml_backend_buffer_t buf = ggml_backend_cpu_buffer_from_ptr(base, data.size() + (data.data() - base));
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src == nullptr) {
ggml_backend_tensor_alloc(buf, t, data.data() + off);
off += ggml_nbytes(t);
}
}
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) {
ggml_backend_view_init(t);
}
}
return buf;
}