From 2874dcb107b64e3c8445640d46ca9443cc1016a3 Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Sun, 25 Oct 2026 14:28:11 +0100 Subject: [PATCH] server: disk prompt cache, save slots on shutdown and restore after a restart llama-swap SIGTERMs a model when the owner switches models (only one large model fits in RAM), and switching back re-prefills the whole conversation: about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR, each slot with a prompt is saved to DIR on shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the file, so only the suffix is processed. Off by default. Save, in server_context::start_loop() after the loop returns: - disk_cache_collect() copies the state of each slot with >= 256 text tokens to host memory (llama_state_seq_get_data_ext, flags NONE) while the device buffers still exist. A cold server reads its released state through llama_context_restore_memory_host(), with no device use and no extra copy. - vram_share_exit() then frees the VRAM and releases the GPU token as before, so only the D2H copy delays the token release, never the disk IO. - disk_cache_flush() writes the files. A slot is skipped if its KV does not cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not decoded) or if the same token list is already on disk. File (server_prompt_disk_cache, server-task.cpp): - A 48-byte header (magic, version, key hash, sizes, payload hash), then a payload of 8-byte padded sections: tokens, checkpoint table, state, the checkpoint states. The hash is FNV-1a over the 64-bit words of the payload. - Plain fwrite to .tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file. - The index is built at startup from the headers. A bad magic, a size that does not match the header, or an inconsistent table means the file is removed. A hash mismatch at load also removes it, and nothing is restored. Other keys and other versions are kept. A .tmp older than 1 h is removed. - --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the least recently used *.lpcd files of every model in DIR are removed until the new file fits; a restore updates the file mtime. Entries the new prompt extends are dropped after the write and do not count against the limit. A write also needs its size + 1 GiB free on the file system. A write error or a full disk is logged; the file is not saved and the exit stays clean. Key: the build commit, the model path + size + mtime, LoRA and control vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the K/V types, the flash-attn type, and the RoPE/YaRN parameters. Restore, in get_available_slot() after the RAM cache: - Only while warm (holding the GPU token), with cache_prompt on and no media tokens. The file must reuse at least 256 more tokens than the slot holds. - Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint at or below the common prefix can be used. A plain-KV model can be cut back to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is almost never a prefix of the next turn. The checkpoints are exact snapshots, and update_slots() rolls back to one exactly as it does for the RAM cache. They are restored with id_task -1, so one that fails to load means a re-prefill, not an abort. - The slot's current prompt goes to the RAM cache first. A set_data failure clears the slot and removes the file. Not supported with speculative decoding (warning, disabled): the draft state is not saved. Only the slots are saved, not the RAM cache entries. Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention): - 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms, exit 314 ms (125 ms without the cache). After the restart: read 166 ms, restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full prefill. The answer is identical to a server that never restarted. - 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS +162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms, turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill. - The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node overlay) stalled once: 5.8 s for 64.5 MiB. llama-swap v239 waits for the old process to exit before it starts the next model, and SIGKILLs a process that is still there after 30 s. So the exit save adds to the model switch time: about 1 s per GB on the PVC. Co-Authored-By: Claude Opus 5.5 --- common/arg.cpp | 17 ++ common/common.h | 2 + tools/server/server-context.cpp | 171 ++++++++++++ tools/server/server-task.cpp | 448 ++++++++++++++++++++++++++++++++ tools/server/server-task.h | 36 +++ 5 files changed, 674 insertions(+) diff --git a/common/arg.cpp b/common/arg.cpp index 9706b3ac5..76a1e60db 100644 --- a/common/arg.cpp +++ b/common/arg.cpp @@ -1732,6 +1732,23 @@ common_params_context common_params_parser_init(common_params & params, llama_ex params.cache_ram_explicit = true; } ).set_env("LLAMA_ARG_CACHE_RAM").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI})); + add_opt(common_arg( + {"--cache-disk-path"}, "PATH", + "save the state of each slot with a prompt to this directory on shutdown, and restore it for a prompt that starts with it after a restart (default: disabled)", + [](common_params & params, const std::string & value) { + params.cache_disk_path = value; + } + ).set_env("LLAMA_ARG_CACHE_DISK_PATH").set_examples({LLAMA_EXAMPLE_SERVER})); + add_opt(common_arg( + {"--cache-disk-size"}, "N", + string_format("size limit of the --cache-disk-path directory in MiB, the least recently used files are removed (default: %d, 0 - no limit)", params.cache_disk_mib), + [](common_params & params, int value) { + if (value < 0) { + throw std::invalid_argument("cache-disk-size must be non-negative"); + } + params.cache_disk_mib = value; + } + ).set_env("LLAMA_ARG_CACHE_DISK_SIZE").set_examples({LLAMA_EXAMPLE_SERVER})); add_opt(common_arg( {"-kvu", "--kv-unified"}, {"-no-kvu", "--no-kv-unified"}, diff --git a/common/common.h b/common/common.h index 70d6a671f..e856f9330 100644 --- a/common/common.h +++ b/common/common.h @@ -641,6 +641,8 @@ struct common_params { int32_t checkpoint_min_step = 8192; // minimum spacing between context checkpoints int32_t cache_ram_mib = 8192; // -1 = no limit, 0 - disable, 1 = 1 MiB, etc. bool cache_ram_explicit = false; // set by --cache-ram, decision models disable the cache otherwise + std::string cache_disk_path; // directory of the disk prompt cache, empty = disabled + int32_t cache_disk_mib = 4096; // size limit of cache_disk_path in MiB, 0 = no limit // llama-connect params bool server_connect = false; diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp index 5eed6ffcd..e259ef6e4 100644 --- a/tools/server/server-context.cpp +++ b/tools/server/server-context.cpp @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -1039,6 +1040,9 @@ static bool add_mtmd_chunk(const server_slot & slot, const mtmd_input_chunk * ch return true; } +// smaller prompts are not saved, and a smaller gain is not worth a file read +static constexpr size_t DISK_CACHE_MIN_TOKENS = 256; + // // server_context_impl (private implementation) // @@ -1382,6 +1386,152 @@ private: vram_cold = true; } + // the model file and every parameter that changes the state layout or the KV values + std::string disk_cache_key() const { + std::ostringstream key; + const auto add_file = [&](const std::string & path) { + std::error_code ec; + const auto size = std::filesystem::file_size(path, ec); + const auto mtime = std::filesystem::last_write_time(path, ec).time_since_epoch().count(); + key << path << " " << size << " " << (long long) mtime << "\n"; + }; + key << "commit " << llama_commit() << "\n"; + add_file(params_base.model.path); + for (const auto & la : params_base.lora_adapters) { + add_file(la.path); + key << la.scale << "\n"; + } + for (const auto & cv : params_base.control_vectors) { + add_file(cv.fname); + key << cv.strength << "\n"; + } + key << llama_n_ctx(ctx_tgt) << " " << llama_n_ctx_seq(ctx_tgt) << " " << llama_n_seq_max(ctx_tgt) << " " << llama_n_ubatch(ctx_tgt) << "\n"; + key << params_base.kv_unified << " " << params_base.swa_full << " " << ggml_type_name(params_base.cache_type_k) << " " + << ggml_type_name(params_base.cache_type_v) << " " << llama_flash_attn_type_name(params_base.flash_attn_type) << "\n"; + key << std::hexfloat << (int) params_base.rope_scaling_type << " " << params_base.rope_freq_base << " " << params_base.rope_freq_scale << " " + << params_base.yarn_ext_factor << " " << params_base.yarn_attn_factor << " " << params_base.yarn_beta_fast << " " + << params_base.yarn_beta_slow << " " << params_base.yarn_orig_ctx << "\n"; + return key.str(); + } + + // on shutdown, the loop has stopped: copy the state of each slot with a prompt to host memory + // while it is still readable. disk_cache_flush() writes them once the GPU token is released. + void disk_cache_collect() { + if (!disk_cache || ctx_tgt == nullptr) { + return; + } + std::vector todo; + for (auto & slot : slots) { + const llama_tokens & tokens = slot.prompt.tokens.get_tokens(); + if (tokens.size() < DISK_CACHE_MIN_TOKENS || std::find(tokens.begin(), tokens.end(), LLAMA_TOKEN_NULL) != tokens.end()) { + continue; + } + // the last batch is not decoded if the shutdown came during a token wait + if (llama_memory_seq_pos_max(llama_get_memory(ctx_tgt), slot.id) + 1 != (llama_pos) tokens.size()) { + SLT_WRN(slot, "disk cache: the state does not match the %zu prompt tokens, not saved\n", tokens.size()); + continue; + } + if (disk_cache->has(tokens)) { + SLT_INF(slot, "disk cache: the prompt of %zu tokens is already saved\n", tokens.size()); + continue; + } + todo.push_back(&slot); + } + if (todo.empty()) { + return; + } + // cold: the state is in a host shadow (LLAMA_SLEEP_EVICT_KV) or still in VRAM, read it without the device + if (vram_only && vram_cold && !llama_context_restore_memory_host(ctx_tgt)) { + SRV_WRN("%s", "disk cache: cannot read the released state, not saved\n"); + return; + } + for (auto * pslot : todo) { + auto & slot = *pslot; + const int64_t t_start = ggml_time_us(); + server_prompt_cache_state st; + try { + st.data.main.resize(llama_state_seq_get_size_ext(ctx_tgt, slot.id, LLAMA_STATE_SEQ_FLAGS_NONE)); + } catch (const std::bad_alloc &) { + SLT_WRN(slot, "%s", "disk cache: out of memory, not saved\n"); + continue; + } + if (st.data.main.empty() || llama_state_seq_get_data_ext(ctx_tgt, st.data.main.data(), st.data.main.size(), slot.id, LLAMA_STATE_SEQ_FLAGS_NONE) != st.data.main.size()) { + SLT_WRN(slot, "%s", "disk cache: failed to copy the state, not saved\n"); + continue; + } + st.prompt = std::move(slot.prompt); + SLT_INF(slot, "disk cache: copied the state of %d tokens (%.1f MiB, %zu checkpoints) in %.0f ms\n", + st.prompt.n_tokens(), st.size() / (1024.0 * 1024.0), st.prompt.checkpoints.size(), (ggml_time_us() - t_start) / 1000.0); + disk_pending.push_back(std::move(st)); + } + } + + void disk_cache_flush() { + for (const auto & st : disk_pending) { + const int64_t t_start = ggml_time_us(); + if (disk_cache->save(st.prompt, st.data.main)) { + SRV_INF("disk cache: saved the prompt of %d tokens (%.1f MiB) in %.0f ms\n", + st.prompt.n_tokens(), st.size() / (1024.0 * 1024.0), (ggml_time_us() - t_start) / 1000.0); + } + } + disk_pending.clear(); + } + + // replace the slot prompt with a disk cache entry that reuses at least DISK_CACHE_MIN_TOKENS more of the new prompt + void disk_cache_load(server_slot & slot, const server_tokens & tokens_new) { + if (vram_only && vram_cold) { + return; // restore only while holding the GPU + } + const llama_tokens & tokens = tokens_new.get_tokens(); + if (std::find(tokens.begin(), tokens.end(), LLAMA_TOKEN_NULL) != tokens.end()) { + return; + } + // a recurrent or SWA state cannot be cut back to an arbitrary prefix (see update_slots) + const bool can_trim = ctx_tgt_seq_rm_type == COMMON_CONTEXT_SEQ_RM_TYPE_PART && n_swa == 0; + std::vector ckpt_n_tokens; + for (const auto & c : slot.prompt.checkpoints) { + ckpt_n_tokens.push_back(c.n_tokens); + } + const size_t n_have = server_prompt_disk_cache::n_reuse(slot.prompt.tokens.get_tokens(), ckpt_n_tokens, tokens, can_trim); + const int i = disk_cache->find(tokens, n_have, DISK_CACHE_MIN_TOKENS, can_trim); + if (i < 0) { + return; + } + const std::string path = disk_cache->entries[i].path; + const size_t n_reuse = server_prompt_disk_cache::n_reuse(disk_cache->entries[i].tokens, disk_cache->entries[i].ckpt_n_tokens, tokens, can_trim); + + const int64_t t_start = ggml_time_us(); + server_prompt prompt; + std::vector data; + if (!disk_cache->load(i, prompt, data, mctx != nullptr)) { + return; + } + const int64_t t_read = ggml_time_us(); + + // keep the current slot prompt in the RAM cache, as get_available_slot() does + if (prompt_cache && slot.prompt_save(*prompt_cache)) { + prompt_cache->update(); + } + slot.prompt_clear(); + if (llama_state_seq_set_data_ext(ctx_tgt, data.data(), data.size(), slot.id, LLAMA_STATE_SEQ_FLAGS_NONE) != data.size()) { + SLT_WRN(slot, "disk cache: failed to restore the state from '%s', removing it\n", path.c_str()); + slot.prompt_clear(); + disk_cache->remove(i); + return; + } + while (prompt.checkpoints.size() > (size_t) std::max(0, params_base.n_ctx_checkpoints)) { + prompt.checkpoints.pop_front(); + } + size_t size = data.size(); + for (const auto & c : prompt.checkpoints) { + size += c.size(); + } + slot.prompt = std::move(prompt); + SLT_INF(slot, "disk cache: restored %d tokens (%.1f MiB, %zu checkpoints) from '%s', read %.0f ms, restore %.0f ms, %zu of %zu prompt tokens reusable (had %zu)\n", + slot.prompt.n_tokens(), size / (1024.0 * 1024.0), slot.prompt.checkpoints.size(), path.c_str(), + (t_read - t_start) / 1000.0, (ggml_time_us() - t_read) / 1000.0, n_reuse, tokens.size(), n_have); + } + common_speculative_init_result_ptr spec_init; common_context_seq_rm_type ctx_tgt_seq_rm_type = COMMON_CONTEXT_SEQ_RM_TYPE_NO; @@ -1408,6 +1558,10 @@ private: std::unique_ptr prompt_cache; + // prompt states saved to disk on shutdown and restored after a restart (--cache-disk-path) + std::unique_ptr disk_cache; + std::vector disk_pending; // copied before the device buffers are freed, written after + server_metrics metrics; // queued prompt stats - llama_decode() is async, so the timing is only valid after a sync @@ -1936,8 +2090,18 @@ private: } else { SRV_TRC("%s", "prompt cache is disabled - use `--cache-ram N` to enable it\n"); } + SRV_TRC("%s", "for more info see https://github.com/ggml-org/llama.cpp/pull/16391\n"); + disk_cache.reset(); + if (!params_base.cache_disk_path.empty()) { + if (spec) { + SRV_WRN("%s", "disk cache: not supported with speculative decoding, disabled\n"); + } else { + disk_cache = std::make_unique(params_base.cache_disk_path, params_base.cache_disk_mib, disk_cache_key()); + } + } + if (params_base.n_ctx_checkpoints > 0) { SRV_TRC("context checkpoints enabled, max = %d, min spacing = %d\n", params_base.n_ctx_checkpoints, params_base.checkpoint_min_step); @@ -2231,6 +2395,10 @@ private: } } + if (ret && disk_cache && !ret->is_processing() && task.params.cache_prompt && !task.is_stateless()) { + disk_cache_load(*ret, task.tokens); + } + return ret; } @@ -5454,7 +5622,10 @@ bool server_context::load_model(common_params & params) { void server_context::start_loop() { auto & params = impl->params_base; impl->queue_tasks.start_loop(params.sleep_idle_seconds * 1000); + // copy the slot states while the device buffers exist, write them after the GPU token is released + impl->disk_cache_collect(); impl->vram_share_exit(); + impl->disk_cache_flush(); } void server_context::terminate() { diff --git a/tools/server/server-task.cpp b/tools/server/server-task.cpp index d7ac2a1dc..96253bb17 100644 --- a/tools/server/server-task.cpp +++ b/tools/server/server-task.cpp @@ -10,8 +10,19 @@ #include "speculative.h" #include "server-common.h" +#include +#include +#include +#include +#include +#include #include +#if defined(__linux__) +#include +#include +#endif + // // task_params // @@ -1934,3 +1945,440 @@ void server_prompt_cache::update() { (const void *)&state, state.prompt.n_tokens(), state.prompt.checkpoints.size(), state.size() / (1024.0 * 1024.0)); } } + +// +// server_prompt_disk_cache +// + +// file = header + payload; the payload is a list of sections, each padded to 8 bytes: +// tokens, checkpoint table, state size, state, the state of each checkpoint +// hash_payload is FNV-1a over the 64-bit words of the payload, so a truncated or corrupt file is never used +static constexpr uint32_t DISK_CACHE_MAGIC = 0x4443504c; // "LPCD" +static constexpr uint32_t DISK_CACHE_VERSION = 1; +static constexpr const char * DISK_CACHE_EXT = ".lpcd"; + +// keep this much space free on the file system +static constexpr uint64_t DISK_CACHE_MIN_FREE = 1024ull*1024ull*1024ull; + +struct disk_cache_header { + uint32_t magic; + uint32_t version; + uint64_t key_hash; + uint64_t n_tokens; + uint64_t n_ckpt; + uint64_t size_payload; + uint64_t hash_payload; +}; + +struct disk_cache_ckpt { + int64_t n_tokens; + llama_pos pos_min; + llama_pos pos_max; + uint64_t size; +}; + +static_assert(sizeof(disk_cache_header) == 48 && sizeof(disk_cache_ckpt) == 24, "disk cache layout"); + +static uint64_t disk_cache_pad(uint64_t n) { + return (8 - n % 8) % 8; +} + +static uint64_t disk_cache_hash_bytes(const void * data, size_t n) { + uint64_t h = 0xcbf29ce484222325ull; + for (size_t i = 0; i < n; i++) { + h = (h ^ ((const uint8_t *) data)[i]) * 0x100000001b3ull; + } + return h; +} + +static uint64_t disk_cache_payload_size(uint64_t n_tokens, const std::vector & ckpts, uint64_t n_state) { + const auto padded = [](uint64_t n) { return n + disk_cache_pad(n); }; + uint64_t res = padded(n_tokens*sizeof(llama_token)) + padded(ckpts.size()*sizeof(disk_cache_ckpt)) + sizeof(uint64_t) + padded(n_state); + for (const auto & c : ckpts) { + res += padded(c.size); + } + return res; +} + +// sequential section IO with the running payload hash +struct disk_cache_io { + FILE * f = nullptr; + uint64_t hash = 0xcbf29ce484222325ull; + bool ok = true; + + ~disk_cache_io() { + if (f) { + fclose(f); + } + } + + void add_hash(const uint8_t * p, size_t size, const uint8_t * pad) { + const size_t n_full = size - size % 8; + for (size_t i = 0; i < n_full; i += 8) { + uint64_t w; + memcpy(&w, p + i, 8); + hash = (hash ^ w) * 0x100000001b3ull; + } + if (size % 8) { + uint64_t w; + memcpy(&w, p + n_full, size % 8); + memcpy((uint8_t *) &w + size % 8, pad, disk_cache_pad(size)); + hash = (hash ^ w) * 0x100000001b3ull; + } + } + + void write(const void * src, size_t size) { + static const uint8_t zero[8] = {}; + const size_t n_pad = disk_cache_pad(size); + ok = ok && fwrite(src, 1, size, f) == size && fwrite(zero, 1, n_pad, f) == n_pad; + add_hash((const uint8_t *) src, size, zero); + } + + void read(void * dst, size_t size) { + uint8_t pad[8]; + const size_t n_pad = disk_cache_pad(size); + ok = ok && fread(dst, 1, size, f) == size && fread(pad, 1, n_pad, f) == n_pad; + if (ok) { + add_hash((const uint8_t *) dst, size, pad); + } + } + + // the data is in our buffers, the cached pages are waste (see llama_file::drop_cache) + void drop_cache() { +#if defined(__linux__) + posix_fadvise(fileno(f), 0, 0, POSIX_FADV_DONTNEED); +#endif + } +}; + +// open a file and read everything up to the state data: 1 = entry of this key, 0 = other key or version, -1 = bad file +static int disk_cache_open(const std::string & path, uint64_t key_hash, disk_cache_io & io, disk_cache_header & hdr, + llama_tokens & tokens, std::vector & ckpts, uint64_t & n_state) { + std::error_code ec; + const uint64_t size = std::filesystem::file_size(path, ec); + io.f = ec ? nullptr : fopen(path.c_str(), "rb"); + if (io.f == nullptr || fread(&hdr, 1, sizeof(hdr), io.f) != sizeof(hdr) || hdr.magic != DISK_CACHE_MAGIC) { + return -1; + } + if (hdr.version != DISK_CACHE_VERSION) { + return 0; + } + if (size != sizeof(hdr) + hdr.size_payload || hdr.n_tokens > size/sizeof(llama_token) || hdr.n_ckpt > size/sizeof(disk_cache_ckpt)) { + return -1; + } + if (hdr.key_hash != key_hash) { + return 0; + } + tokens.resize(hdr.n_tokens); + ckpts.resize(hdr.n_ckpt); + io.read(tokens.data(), tokens.size()*sizeof(llama_token)); + io.read(ckpts.data(), ckpts.size()*sizeof(disk_cache_ckpt)); + io.read(&n_state, sizeof(n_state)); + if (!io.ok || n_state > size) { + return -1; + } + for (const auto & c : ckpts) { + if (c.size > size) { + return -1; + } + } + return disk_cache_payload_size(hdr.n_tokens, ckpts, n_state) == hdr.size_payload ? 1 : -1; +} + +static void disk_cache_touch(const std::string & path) { + std::error_code ec; + std::filesystem::last_write_time(path, std::filesystem::file_time_type::clock::now(), ec); +} + +server_prompt_disk_cache::server_prompt_disk_cache(const std::string & dir, int32_t limit_size_mib, const std::string & key) : dir(dir) { + key_hash = disk_cache_hash_bytes(key.data(), key.size()); + limit_size = 1024ull*1024ull*std::max(0, limit_size_mib); + + size_t n_other = 0; + size_t n_bad = 0; + try { + std::filesystem::create_directories(dir); + const auto now = std::filesystem::file_time_type::clock::now(); + for (const auto & de : std::filesystem::directory_iterator(dir)) { + const std::string path = de.path().string(); + if (!de.is_regular_file()) { + continue; + } + // left by a writer that was killed + if (path.find(std::string(DISK_CACHE_EXT) + ".tmp") != std::string::npos) { + if (now - de.last_write_time() > std::chrono::hours(1)) { + std::filesystem::remove(de.path()); + } + continue; + } + if (de.path().extension() != DISK_CACHE_EXT) { + continue; + } + disk_cache_io io; + disk_cache_header hdr; + server_prompt_disk_entry e; + std::vector ckpts; + uint64_t n_state = 0; + const int res = disk_cache_open(path, key_hash, io, hdr, e.tokens, ckpts, n_state); + if (res < 0) { + SRV_WRN("disk cache: '%s' is truncated or corrupt, removing it\n", path.c_str()); + std::filesystem::remove(de.path()); + n_bad++; + continue; + } + if (res == 0) { + n_other++; + continue; + } + e.path = path; + for (const auto & c : ckpts) { + e.ckpt_n_tokens.push_back(c.n_tokens); + } + entries.push_back(std::move(e)); + } + } catch (const std::exception & err) { + SRV_WRN("disk cache: error reading '%s': %s\n", dir.c_str(), err.what()); + } + + SRV_INF("disk cache: '%s', %zu prompts of this model, %zu other files, %zu bad files removed, limit %.0f MiB\n", + dir.c_str(), entries.size(), n_other, n_bad, limit_size / (1024.0 * 1024.0)); +} + +size_t server_prompt_disk_cache::n_reuse(const llama_tokens & tokens, const std::vector & ckpt_n_tokens, const llama_tokens & tokens_new, bool can_trim) { + size_t lcp = 0; + while (lcp < tokens.size() && lcp < tokens_new.size() && tokens[lcp] == tokens_new[lcp]) { + lcp++; + } + if (lcp == tokens.size() || can_trim) { + return lcp; + } + size_t res = 0; + for (const int64_t n : ckpt_n_tokens) { + if (n <= (int64_t) lcp) { + res = std::max(res, (size_t) n); + } + } + return res; +} + +int server_prompt_disk_cache::find(const llama_tokens & tokens_new, size_t n_have, size_t n_min, bool can_trim) const { + int best = -1; + size_t n_best = 0; + for (size_t i = 0; i < entries.size(); i++) { + const size_t n = n_reuse(entries[i].tokens, entries[i].ckpt_n_tokens, tokens_new, can_trim); + if (n >= n_have + n_min && n > n_best) { + best = (int) i; + n_best = n; + } + } + return best; +} + +bool server_prompt_disk_cache::has(const llama_tokens & tokens) { + for (const auto & e : entries) { + std::error_code ec; + if (e.tokens == tokens && std::filesystem::exists(e.path, ec)) { + disk_cache_touch(e.path); + return true; + } + } + return false; +} + +void server_prompt_disk_cache::remove(int i) { + std::error_code ec; + std::filesystem::remove(entries[i].path, ec); + entries.erase(entries.begin() + i); +} + +bool server_prompt_disk_cache::load(int i, server_prompt & prompt, std::vector & data, bool has_mtmd) { + const std::string path = entries[i].path; + + disk_cache_io io; + disk_cache_header hdr; + llama_tokens tokens; + std::vector ckpts; + uint64_t n_state = 0; + std::list checkpoints; + + bool ok = disk_cache_open(path, key_hash, io, hdr, tokens, ckpts, n_state) == 1 && tokens == entries[i].tokens; + if (ok) { + try { + data.resize(n_state); + io.read(data.data(), data.size()); + for (const auto & c : ckpts) { + auto & cur = checkpoints.emplace_back(); + cur.id_task = -1; // not created by a task: may fail to load, then the prompt is re-processed + cur.n_tokens = c.n_tokens; + cur.pos_min = c.pos_min; + cur.pos_max = c.pos_max; + cur.data_tgt.resize(c.size); + io.read(cur.data_tgt.data(), cur.data_tgt.size()); + } + } catch (const std::bad_alloc &) { + SRV_WRN("disk cache: out of memory reading '%s'\n", path.c_str()); + data.clear(); + return false; + } + ok = io.ok && io.hash == hdr.hash_payload; + io.drop_cache(); + } + if (!ok) { + SRV_WRN("disk cache: '%s' is missing, truncated or corrupt, removing it\n", path.c_str()); + data.clear(); + remove(i); + return false; + } + + disk_cache_touch(path); + + prompt.tokens = server_tokens(tokens, has_mtmd); + prompt.checkpoints = std::move(checkpoints); + + return true; +} + +bool server_prompt_disk_cache::save(const server_prompt & prompt, const std::vector & data) { + const llama_tokens & tokens = prompt.tokens.get_tokens(); + + std::vector ckpts; + for (const auto & c : prompt.checkpoints) { + ckpts.push_back({ c.n_tokens, c.pos_min, c.pos_max, c.data_tgt.size() }); + } + + disk_cache_header hdr = { + /*.magic =*/ DISK_CACHE_MAGIC, + /*.version =*/ DISK_CACHE_VERSION, + /*.key_hash =*/ key_hash, + /*.n_tokens =*/ tokens.size(), + /*.n_ckpt =*/ ckpts.size(), + /*.size_payload =*/ disk_cache_payload_size(tokens.size(), ckpts, data.size()), + /*.hash_payload =*/ 0, + }; + const uint64_t size_file = sizeof(hdr) + hdr.size_payload; + + char name[64]; + snprintf(name, sizeof(name), "%016llx-%016llx", (unsigned long long) key_hash, + (unsigned long long) disk_cache_hash_bytes(tokens.data(), tokens.size()*sizeof(llama_token))); + const std::string path = (std::filesystem::path(dir) / name).string() + DISK_CACHE_EXT; + const std::string path_tmp = path + ".tmp"; + + // entries contained in the new one are obsolete: removed once it is written, not counted for the limit + const auto is_obsolete = [&](const server_prompt_disk_entry & e) { + return e.path != path && e.tokens.size() <= tokens.size() && std::equal(e.tokens.begin(), e.tokens.end(), tokens.begin()); + }; + std::vector obsolete; + for (const auto & e : entries) { + if (is_obsolete(e)) { + obsolete.push_back(e.path); + } + } + + if (limit_size > 0 && size_file > limit_size) { + SRV_WRN("disk cache: prompt state of %.1f MiB exceeds the limit of %.1f MiB, not saved\n", size_file / (1024.0 * 1024.0), limit_size / (1024.0 * 1024.0)); + return false; + } + + try { + // evict the least recently used files of every model in dir + struct file_info { + std::filesystem::path path; + uint64_t size; + std::filesystem::file_time_type time; + }; + std::vector files; + uint64_t size_total = 0; + for (const auto & de : std::filesystem::directory_iterator(dir)) { + if (de.is_regular_file() && de.path().extension() == DISK_CACHE_EXT && de.path().string() != path && + std::find(obsolete.begin(), obsolete.end(), de.path().string()) == obsolete.end()) { + files.push_back({ de.path(), de.file_size(), de.last_write_time() }); + size_total += files.back().size; + } + } + std::sort(files.begin(), files.end(), [](const file_info & a, const file_info & b) { return a.time < b.time; }); + for (size_t i = 0; limit_size > 0 && size_total + size_file > limit_size && i < files.size(); i++) { + SRV_INF("disk cache: size limit, removing '%s' (%.1f MiB)\n", files[i].path.string().c_str(), files[i].size / (1024.0 * 1024.0)); + std::filesystem::remove(files[i].path); + size_total -= files[i].size; + for (size_t j = 0; j < entries.size(); j++) { + if (entries[j].path == files[i].path.string()) { + entries.erase(entries.begin() + j); + break; + } + } + } + const uint64_t avail = std::filesystem::space(dir).available; + if (avail < size_file + DISK_CACHE_MIN_FREE) { + SRV_WRN("disk cache: only %.1f MiB free in '%s', prompt state of %.1f MiB not saved\n", avail / (1024.0 * 1024.0), dir.c_str(), size_file / (1024.0 * 1024.0)); + return false; + } + } catch (const std::exception & err) { + SRV_WRN("disk cache: error reading '%s': %s\n", dir.c_str(), err.what()); + return false; + } + + bool ok; + { + disk_cache_io io; + io.f = fopen(path_tmp.c_str(), "wb"); + ok = io.f != nullptr && fwrite(&hdr, 1, sizeof(hdr), io.f) == sizeof(hdr); + if (ok) { + const uint64_t n_state = data.size(); + io.write(tokens.data(), tokens.size()*sizeof(llama_token)); + io.write(ckpts.data(), ckpts.size()*sizeof(disk_cache_ckpt)); + io.write(&n_state, sizeof(n_state)); + io.write(data.data(), data.size()); + for (const auto & c : prompt.checkpoints) { + io.write(c.data_tgt.data(), c.data_tgt.size()); + } + hdr.hash_payload = io.hash; + ok = io.ok && fseek(io.f, 0, SEEK_SET) == 0 && fwrite(&hdr, 1, sizeof(hdr), io.f) == sizeof(hdr) && fflush(io.f) == 0; +#if defined(__linux__) + // write back now, so the pages can be dropped and the rename never exposes missing data + ok = ok && fdatasync(fileno(io.f)) == 0; + io.drop_cache(); +#endif + } + if (io.f != nullptr && fclose(io.f) != 0) { + ok = false; + } + io.f = nullptr; + if (!ok) { + SRV_WRN("disk cache: failed to write '%s': %s, not saved\n", path_tmp.c_str(), strerror(errno)); + } + } + std::error_code ec; + if (ok) { + std::filesystem::rename(path_tmp, path, ec); + ok = !ec; + if (!ok) { + SRV_WRN("disk cache: failed to rename '%s': %s, not saved\n", path_tmp.c_str(), ec.message().c_str()); + } + } + if (!ok) { + std::filesystem::remove(path_tmp, ec); + return false; + } + + for (size_t i = 0; i < entries.size(); ) { + const auto & e = entries[i]; + if (e.path == path) { + entries.erase(entries.begin() + i); + } else if (is_obsolete(e)) { + SRV_INF("disk cache: removing '%s', a prefix of the new prompt\n", e.path.c_str()); + remove((int) i); + } else { + i++; + } + } + + server_prompt_disk_entry e; + e.path = path; + e.tokens = tokens; + for (const auto & c : ckpts) { + e.ckpt_n_tokens.push_back(c.n_tokens); + } + entries.push_back(std::move(e)); + + return true; +} diff --git a/tools/server/server-task.h b/tools/server/server-task.h index d6b6f4b27..3c17b2556 100644 --- a/tools/server/server-task.h +++ b/tools/server/server-task.h @@ -719,6 +719,42 @@ struct server_prompt_cache { void update(); }; +// prompt states saved as files, to reuse a prompt after a server restart +struct server_prompt_disk_entry { + std::string path; + llama_tokens tokens; + std::vector ckpt_n_tokens; +}; + +struct server_prompt_disk_cache { + // key: model file and every parameter that changes the state layout, see the server context + server_prompt_disk_cache(const std::string & dir, int32_t limit_size_mib, const std::string & key); + + std::string dir; + uint64_t key_hash; + size_t limit_size; + + // entries of this key only, read from the file headers at startup + std::vector entries; + + // tokens of tokens_new that a state of `tokens` with checkpoints at `ckpt_n_tokens` can reuse + // a recurrent or SWA state is reusable only up to an exact prefix or a checkpoint + static size_t n_reuse(const llama_tokens & tokens, const std::vector & ckpt_n_tokens, const llama_tokens & tokens_new, bool can_trim); + + // index of the entry that reuses the most of tokens_new, and at least n_min more than n_have; -1 if none + int find(const llama_tokens & tokens_new, size_t n_have, size_t n_min, bool can_trim) const; + + bool has(const llama_tokens & tokens); + + // read and verify entry i; a bad file is removed + bool load(int i, server_prompt & prompt, std::vector & data, bool has_mtmd); + + // write atomically, after evicting the oldest files in dir over the size limit + bool save(const server_prompt & prompt, const std::vector & data); + + void remove(int i); +}; + // used exclusively by router mode struct server_task_result_router : server_task_result { json data;