server: disk prompt cache, save slots on shutdown and restore after a restart

llama-swap SIGTERMs a model when the owner switches models (only one large
model fits in RAM), and switching back re-prefills the whole conversation:
about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR,
each slot with a prompt is saved to DIR on shutdown. After the restart, a
request whose prompt starts with a saved prompt is restored from the file,
so only the suffix is processed. Off by default.

Save, in server_context::start_loop() after the loop returns:
- disk_cache_collect() copies the state of each slot with >= 256 text tokens
  to host memory (llama_state_seq_get_data_ext, flags NONE) while the device
  buffers still exist. A cold server reads its released state through
  llama_context_restore_memory_host(), with no device use and no extra copy.
- vram_share_exit() then frees the VRAM and releases the GPU token as before,
  so only the D2H copy delays the token release, never the disk IO.
- disk_cache_flush() writes the files. A slot is skipped if its KV does not
  cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not
  decoded) or if the same token list is already on disk.

File (server_prompt_disk_cache, server-task.cpp):
- A 48-byte header (magic, version, key hash, sizes, payload hash), then a
  payload of 8-byte padded sections: tokens, checkpoint table, state, the
  checkpoint states. The hash is FNV-1a over the 64-bit words of the payload.
- Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read
  is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked
  the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file.
- The index is built at startup from the headers. A bad magic, a size that
  does not match the header, or an inconsistent table means the file is
  removed. A hash mismatch at load also removes it, and nothing is restored.
  Other keys and other versions are kept. A .tmp older than 1 h is removed.
- --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the
  least recently used *.lpcd files of every model in DIR are removed until the
  new file fits; a restore updates the file mtime. Entries the new prompt
  extends are dropped after the write and do not count against the limit.
  A write also needs its size + 1 GiB free on the file system. A write error
  or a full disk is logged; the file is not saved and the exit stays clean.

Key: the build commit, the model path + size + mtime, LoRA and control
vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the
K/V types, the flash-attn type, and the RoPE/YaRN parameters.

Restore, in get_available_slot() after the RAM cache:
- Only while warm (holding the GPU token), with cache_prompt on and no media
  tokens. The file must reuse at least 256 more tokens than the slot holds.
- Reusable means one of two things. Either the saved token list is an exact
  prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint
  at or below the common prefix can be used. A plain-KV model can be cut back
  to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the
  template rewrites the previous assistant turn (the reasoning is dropped), so
  the saved list is almost never a prefix of the next turn. The checkpoints
  are exact snapshots, and update_slots() rolls back to one exactly as it does
  for the RAM cache. They are restored with id_task -1, so one that fails to
  load means a re-prefill, not an abort.
- The slot's current prompt goes to the RAM cache first. A set_data failure
  clears the slot and removes the file.

Not supported with speculative decoding (warning, disabled): the draft state
is not saved. Only the slots are saved, not the RAM cache entries.

Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention):
- 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms,
  exit 314 ms (125 ms without the cache). After the restart: read 166 ms,
  restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full
  prefill. The answer is identical to a server that never restarted.
- 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS
  +162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms,
  turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill.
- The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node
  overlay) stalled once: 5.8 s for 64.5 MiB.
llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a process that is still there after 30 s. So the exit
save adds to the model switch time: about 1 s per GB on the PVC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 18:55:57 +01:00
1 parent dd851fd0c9
commit 2874dcb107
5 files changed
+674

No files matched your search

+171
View File
@@ -30,6 +30,7 @@
#include <memory>
#include <filesystem>
#include <random>
#include <sstream>
#include <utility>
#include <fstream>
#include <thread>
@@ -1039,6 +1040,9 @@ static bool add_mtmd_chunk(const server_slot & slot, const mtmd_input_chunk * ch
return true;
}
// smaller prompts are not saved, and a smaller gain is not worth a file read
static constexpr size_t DISK_CACHE_MIN_TOKENS = 256;
//
// server_context_impl (private implementation)
//
@@ -1382,6 +1386,152 @@ private:
vram_cold = true;
}
// the model file and every parameter that changes the state layout or the KV values
std::string disk_cache_key() const {
std::ostringstream key;
const auto add_file = [&](const std::string & path) {
std::error_code ec;
const auto size = std::filesystem::file_size(path, ec);
const auto mtime = std::filesystem::last_write_time(path, ec).time_since_epoch().count();
key << path << " " << size << " " << (long long) mtime << "\n";
};
key << "commit " << llama_commit() << "\n";
add_file(params_base.model.path);
for (const auto & la : params_base.lora_adapters) {
add_file(la.path);
key << la.scale << "\n";
}
for (const auto & cv : params_base.control_vectors) {
add_file(cv.fname);
key << cv.strength << "\n";
}
key << llama_n_ctx(ctx_tgt) << " " << llama_n_ctx_seq(ctx_tgt) << " " << llama_n_seq_max(ctx_tgt) << " " << llama_n_ubatch(ctx_tgt) << "\n";
key << params_base.kv_unified << " " << params_base.swa_full << " " << ggml_type_name(params_base.cache_type_k) << " "
<< ggml_type_name(params_base.cache_type_v) << " " << llama_flash_attn_type_name(params_base.flash_attn_type) << "\n";
key << std::hexfloat << (int) params_base.rope_scaling_type << " " << params_base.rope_freq_base << " " << params_base.rope_freq_scale << " "
<< params_base.yarn_ext_factor << " " << params_base.yarn_attn_factor << " " << params_base.yarn_beta_fast << " "
<< params_base.yarn_beta_slow << " " << params_base.yarn_orig_ctx << "\n";
return key.str();
}
// on shutdown, the loop has stopped: copy the state of each slot with a prompt to host memory
// while it is still readable. disk_cache_flush() writes them once the GPU token is released.
void disk_cache_collect() {
if (!disk_cache || ctx_tgt == nullptr) {
return;
}
std::vector<server_slot *> todo;
for (auto & slot : slots) {
const llama_tokens & tokens = slot.prompt.tokens.get_tokens();
if (tokens.size() < DISK_CACHE_MIN_TOKENS || std::find(tokens.begin(), tokens.end(), LLAMA_TOKEN_NULL) != tokens.end()) {
continue;
}
// the last batch is not decoded if the shutdown came during a token wait
if (llama_memory_seq_pos_max(llama_get_memory(ctx_tgt), slot.id) + 1 != (llama_pos) tokens.size()) {
SLT_WRN(slot, "disk cache: the state does not match the %zu prompt tokens, not saved\n", tokens.size());
continue;
}
if (disk_cache->has(tokens)) {
SLT_INF(slot, "disk cache: the prompt of %zu tokens is already saved\n", tokens.size());
continue;
}
todo.push_back(&slot);
}
if (todo.empty()) {
return;
}
// cold: the state is in a host shadow (LLAMA_SLEEP_EVICT_KV) or still in VRAM, read it without the device
if (vram_only && vram_cold && !llama_context_restore_memory_host(ctx_tgt)) {
SRV_WRN("%s", "disk cache: cannot read the released state, not saved\n");
return;
}
for (auto * pslot : todo) {
auto & slot = *pslot;
const int64_t t_start = ggml_time_us();
server_prompt_cache_state st;
try {
st.data.main.resize(llama_state_seq_get_size_ext(ctx_tgt, slot.id, LLAMA_STATE_SEQ_FLAGS_NONE));
} catch (const std::bad_alloc &) {
SLT_WRN(slot, "%s", "disk cache: out of memory, not saved\n");
continue;
}
if (st.data.main.empty() || llama_state_seq_get_data_ext(ctx_tgt, st.data.main.data(), st.data.main.size(), slot.id, LLAMA_STATE_SEQ_FLAGS_NONE) != st.data.main.size()) {
SLT_WRN(slot, "%s", "disk cache: failed to copy the state, not saved\n");
continue;
}
st.prompt = std::move(slot.prompt);
SLT_INF(slot, "disk cache: copied the state of %d tokens (%.1f MiB, %zu checkpoints) in %.0f ms\n",
st.prompt.n_tokens(), st.size() / (1024.0 * 1024.0), st.prompt.checkpoints.size(), (ggml_time_us() - t_start) / 1000.0);
disk_pending.push_back(std::move(st));
}
}
void disk_cache_flush() {
for (const auto & st : disk_pending) {
const int64_t t_start = ggml_time_us();
if (disk_cache->save(st.prompt, st.data.main)) {
SRV_INF("disk cache: saved the prompt of %d tokens (%.1f MiB) in %.0f ms\n",
st.prompt.n_tokens(), st.size() / (1024.0 * 1024.0), (ggml_time_us() - t_start) / 1000.0);
}
}
disk_pending.clear();
}
// replace the slot prompt with a disk cache entry that reuses at least DISK_CACHE_MIN_TOKENS more of the new prompt
void disk_cache_load(server_slot & slot, const server_tokens & tokens_new) {
if (vram_only && vram_cold) {
return; // restore only while holding the GPU
}
const llama_tokens & tokens = tokens_new.get_tokens();
if (std::find(tokens.begin(), tokens.end(), LLAMA_TOKEN_NULL) != tokens.end()) {
return;
}
// a recurrent or SWA state cannot be cut back to an arbitrary prefix (see update_slots)
const bool can_trim = ctx_tgt_seq_rm_type == COMMON_CONTEXT_SEQ_RM_TYPE_PART && n_swa == 0;
std::vector<int64_t> ckpt_n_tokens;
for (const auto & c : slot.prompt.checkpoints) {
ckpt_n_tokens.push_back(c.n_tokens);
}
const size_t n_have = server_prompt_disk_cache::n_reuse(slot.prompt.tokens.get_tokens(), ckpt_n_tokens, tokens, can_trim);
const int i = disk_cache->find(tokens, n_have, DISK_CACHE_MIN_TOKENS, can_trim);
if (i < 0) {
return;
}
const std::string path = disk_cache->entries[i].path;
const size_t n_reuse = server_prompt_disk_cache::n_reuse(disk_cache->entries[i].tokens, disk_cache->entries[i].ckpt_n_tokens, tokens, can_trim);
const int64_t t_start = ggml_time_us();
server_prompt prompt;
std::vector<uint8_t> data;
if (!disk_cache->load(i, prompt, data, mctx != nullptr)) {
return;
}
const int64_t t_read = ggml_time_us();
// keep the current slot prompt in the RAM cache, as get_available_slot() does
if (prompt_cache && slot.prompt_save(*prompt_cache)) {
prompt_cache->update();
}
slot.prompt_clear();
if (llama_state_seq_set_data_ext(ctx_tgt, data.data(), data.size(), slot.id, LLAMA_STATE_SEQ_FLAGS_NONE) != data.size()) {
SLT_WRN(slot, "disk cache: failed to restore the state from '%s', removing it\n", path.c_str());
slot.prompt_clear();
disk_cache->remove(i);
return;
}
while (prompt.checkpoints.size() > (size_t) std::max(0, params_base.n_ctx_checkpoints)) {
prompt.checkpoints.pop_front();
}
size_t size = data.size();
for (const auto & c : prompt.checkpoints) {
size += c.size();
}
slot.prompt = std::move(prompt);
SLT_INF(slot, "disk cache: restored %d tokens (%.1f MiB, %zu checkpoints) from '%s', read %.0f ms, restore %.0f ms, %zu of %zu prompt tokens reusable (had %zu)\n",
slot.prompt.n_tokens(), size / (1024.0 * 1024.0), slot.prompt.checkpoints.size(), path.c_str(),
(t_read - t_start) / 1000.0, (ggml_time_us() - t_read) / 1000.0, n_reuse, tokens.size(), n_have);
}
common_speculative_init_result_ptr spec_init;
common_context_seq_rm_type ctx_tgt_seq_rm_type = COMMON_CONTEXT_SEQ_RM_TYPE_NO;
@@ -1408,6 +1558,10 @@ private:
std::unique_ptr<server_prompt_cache> prompt_cache;
// prompt states saved to disk on shutdown and restored after a restart (--cache-disk-path)
std::unique_ptr<server_prompt_disk_cache> disk_cache;
std::vector<server_prompt_cache_state> disk_pending; // copied before the device buffers are freed, written after
server_metrics metrics;
// queued prompt stats - llama_decode() is async, so the timing is only valid after a sync
@@ -1936,8 +2090,18 @@ private:
} else {
SRV_TRC("%s", "prompt cache is disabled - use `--cache-ram N` to enable it\n");
}
SRV_TRC("%s", "for more info see https://github.com/ggml-org/llama.cpp/pull/16391\n");
disk_cache.reset();
if (!params_base.cache_disk_path.empty()) {
if (spec) {
SRV_WRN("%s", "disk cache: not supported with speculative decoding, disabled\n");
} else {
disk_cache = std::make_unique<server_prompt_disk_cache>(params_base.cache_disk_path, params_base.cache_disk_mib, disk_cache_key());
}
}
if (params_base.n_ctx_checkpoints > 0) {
SRV_TRC("context checkpoints enabled, max = %d, min spacing = %d\n",
params_base.n_ctx_checkpoints, params_base.checkpoint_min_step);
@@ -2231,6 +2395,10 @@ private:
}
}
if (ret && disk_cache && !ret->is_processing() && task.params.cache_prompt && !task.is_stateless()) {
disk_cache_load(*ret, task.tokens);
}
return ret;
}
@@ -5454,7 +5622,10 @@ bool server_context::load_model(common_params & params) {
void server_context::start_loop() {
auto & params = impl->params_base;
impl->queue_tasks.start_loop(params.sleep_idle_seconds * 1000);
// copy the slot states while the device buffers exist, write them after the GPU token is released
impl->disk_cache_collect();
impl->vram_share_exit();
impl->disk_cache_flush();
}
void server_context::terminate() {