server: disk prompt cache, save slots on shutdown and restore after a restart
llama-swap SIGTERMs a model when the owner switches models (only one large model fits in RAM), and switching back re-prefills the whole conversation: about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR, each slot with a prompt is saved to DIR on shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the file, so only the suffix is processed. Off by default. Save, in server_context::start_loop() after the loop returns: - disk_cache_collect() copies the state of each slot with >= 256 text tokens to host memory (llama_state_seq_get_data_ext, flags NONE) while the device buffers still exist. A cold server reads its released state through llama_context_restore_memory_host(), with no device use and no extra copy. - vram_share_exit() then frees the VRAM and releases the GPU token as before, so only the D2H copy delays the token release, never the disk IO. - disk_cache_flush() writes the files. A slot is skipped if its KV does not cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not decoded) or if the same token list is already on disk. File (server_prompt_disk_cache, server-task.cpp): - A 48-byte header (magic, version, key hash, sizes, payload hash), then a payload of 8-byte padded sections: tokens, checkpoint table, state, the checkpoint states. The hash is FNV-1a over the 64-bit words of the payload. - Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file. - The index is built at startup from the headers. A bad magic, a size that does not match the header, or an inconsistent table means the file is removed. A hash mismatch at load also removes it, and nothing is restored. Other keys and other versions are kept. A .tmp older than 1 h is removed. - --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the least recently used *.lpcd files of every model in DIR are removed until the new file fits; a restore updates the file mtime. Entries the new prompt extends are dropped after the write and do not count against the limit. A write also needs its size + 1 GiB free on the file system. A write error or a full disk is logged; the file is not saved and the exit stays clean. Key: the build commit, the model path + size + mtime, LoRA and control vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the K/V types, the flash-attn type, and the RoPE/YaRN parameters. Restore, in get_available_slot() after the RAM cache: - Only while warm (holding the GPU token), with cache_prompt on and no media tokens. The file must reuse at least 256 more tokens than the slot holds. - Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint at or below the common prefix can be used. A plain-KV model can be cut back to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is almost never a prefix of the next turn. The checkpoints are exact snapshots, and update_slots() rolls back to one exactly as it does for the RAM cache. They are restored with id_task -1, so one that fails to load means a re-prefill, not an abort. - The slot's current prompt goes to the RAM cache first. A set_data failure clears the slot and removes the file. Not supported with speculative decoding (warning, disabled): the draft state is not saved. Only the slots are saved, not the RAM cache entries. Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention): - 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms, exit 314 ms (125 ms without the cache). After the restart: read 166 ms, restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full prefill. The answer is identical to a server that never restarted. - 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS +162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms, turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill. - The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node overlay) stalled once: 5.8 s for 64.5 MiB. llama-swap v239 waits for the old process to exit before it starts the next model, and SIGKILLs a process that is still there after 30 s. So the exit save adds to the model switch time: about 1 s per GB on the PVC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
aa46b82321
commit
f1f1f32f3f
5 files changed
+680
No files matched your search
@@ -1704,6 +1704,23 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
|
||||
params.cache_ram_explicit = true;
|
||||
}
|
||||
).set_env("LLAMA_ARG_CACHE_RAM").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}));
|
||||
add_opt(common_arg(
|
||||
{"--cache-disk-path"}, "PATH",
|
||||
"save the state of each slot with a prompt to this directory on shutdown, and restore it for a prompt that starts with it after a restart (default: disabled)",
|
||||
[](common_params & params, const std::string & value) {
|
||||
params.cache_disk_path = value;
|
||||
}
|
||||
).set_env("LLAMA_ARG_CACHE_DISK_PATH").set_examples({LLAMA_EXAMPLE_SERVER}));
|
||||
add_opt(common_arg(
|
||||
{"--cache-disk-size"}, "N",
|
||||
string_format("size limit of the --cache-disk-path directory in MiB, the least recently used files are removed (default: %d, 0 - no limit)", params.cache_disk_mib),
|
||||
[](common_params & params, int value) {
|
||||
if (value < 0) {
|
||||
throw std::invalid_argument("cache-disk-size must be non-negative");
|
||||
}
|
||||
params.cache_disk_mib = value;
|
||||
}
|
||||
).set_env("LLAMA_ARG_CACHE_DISK_SIZE").set_examples({LLAMA_EXAMPLE_SERVER}));
|
||||
add_opt(common_arg(
|
||||
{"-kvu", "--kv-unified"},
|
||||
{"-no-kvu", "--no-kv-unified"},
|
||||
|
||||
Reference in new issue
Block a user