llama-swap SIGTERMs a model when the owner switches models (only one large model fits in RAM), and switching back re-prefills the whole conversation: about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR, each slot with a prompt is saved to DIR on shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the file, so only the suffix is processed. Off by default. Save, in server_context::start_loop() after the loop returns: - disk_cache_collect() copies the state of each slot with >= 256 text tokens to host memory (llama_state_seq_get_data_ext, flags NONE) while the device buffers still exist. A cold server reads its released state through llama_context_restore_memory_host(), with no device use and no extra copy. - vram_share_exit() then frees the VRAM and releases the GPU token as before, so only the D2H copy delays the token release, never the disk IO. - disk_cache_flush() writes the files. A slot is skipped if its KV does not cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not decoded) or if the same token list is already on disk. File (server_prompt_disk_cache, server-task.cpp): - A 48-byte header (magic, version, key hash, sizes, payload hash), then a payload of 8-byte padded sections: tokens, checkpoint table, state, the checkpoint states. The hash is FNV-1a over the 64-bit words of the payload. - Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file. - The index is built at startup from the headers. A bad magic, a size that does not match the header, or an inconsistent table means the file is removed. A hash mismatch at load also removes it, and nothing is restored. Other keys and other versions are kept. A .tmp older than 1 h is removed. - --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the least recently used *.lpcd files of every model in DIR are removed until the new file fits; a restore updates the file mtime. Entries the new prompt extends are dropped after the write and do not count against the limit. A write also needs its size + 1 GiB free on the file system. A write error or a full disk is logged; the file is not saved and the exit stays clean. Key: the build commit, the model path + size + mtime, LoRA and control vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the K/V types, the flash-attn type, and the RoPE/YaRN parameters. Restore, in get_available_slot() after the RAM cache: - Only while warm (holding the GPU token), with cache_prompt on and no media tokens. The file must reuse at least 256 more tokens than the slot holds. - Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint at or below the common prefix can be used. A plain-KV model can be cut back to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is almost never a prefix of the next turn. The checkpoints are exact snapshots, and update_slots() rolls back to one exactly as it does for the RAM cache. They are restored with id_task -1, so one that fails to load means a re-prefill, not an abort. - The slot's current prompt goes to the RAM cache first. A set_data failure clears the slot and removes the file. Not supported with speculative decoding (warning, disabled): the draft state is not saved. Only the slots are saved, not the RAM cache entries. Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention): - 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms, exit 314 ms (125 ms without the cache). After the restart: read 166 ms, restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full prefill. The answer is identical to a server that never restarted. - 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS +162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms, turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill. - The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node overlay) stalled once: 5.8 s for 64.5 MiB. llama-swap v239 waits for the old process to exit before it starts the next model, and SIGKILLs a process that is still there after 30 s. So the exit save adds to the model switch time: about 1 s per GB on the PVC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
171 lines
5.6 KiB
C++
171 lines
5.6 KiB
C++
#pragma once
|
|
|
|
#include "server-common.h"
|
|
#include "server-task.h"
|
|
|
|
#include <list>
|
|
#include <memory>
|
|
#include <vector>
|
|
|
|
// which prompt goes to which slot: slot prompt state, prompt cache and slot selection
|
|
|
|
// what a cached prompt can give to a new prompt
|
|
struct server_prompt_reuse {
|
|
int32_t n_past = 0; // tokens to keep
|
|
llama_pos pos_next = 0;
|
|
|
|
// restore this checkpoint first, nullptr when the live state is used as it is
|
|
std::shared_ptr<const common_prompt_checkpoint> ckpt;
|
|
};
|
|
|
|
struct server_prompt {
|
|
server_tokens tokens;
|
|
|
|
// a checkpoint does not change after it is created, so the prompt cache shares it with the slot instead of a copy
|
|
std::list<std::shared_ptr<const common_prompt_checkpoint>> checkpoints;
|
|
|
|
// the lora config that the tokens were processed with, it follows the state to the prompt cache and back
|
|
// TODO: slot files do not store it
|
|
std::vector<common_adapter_lora_info> lora;
|
|
|
|
void clear() {
|
|
tokens.clear();
|
|
checkpoints.clear();
|
|
}
|
|
|
|
int n_tokens() const {
|
|
return tokens.size();
|
|
}
|
|
|
|
server_prompt clone() const {
|
|
return server_prompt {
|
|
tokens.clone(),
|
|
checkpoints,
|
|
lora,
|
|
};
|
|
}
|
|
|
|
// n_prefix: tokens shared with the new prompt, must be > 0
|
|
// has_new_tokens: the new prompt is longer than n_prefix
|
|
// n_swa: 0 if a state must end right before the next token (no SWA, or recurrent)
|
|
// pos_min: min position of the live state
|
|
// n_past == 0 without a checkpoint means that nothing can be reused
|
|
server_prompt_reuse get_reuse(int32_t n_prefix, bool has_new_tokens, int32_t n_swa, llama_pos pos_min) const;
|
|
};
|
|
|
|
struct server_prompt_data {
|
|
common_state_data main;
|
|
common_state_data drft;
|
|
|
|
size_t size() const {
|
|
return main.size() + drft.size();
|
|
}
|
|
};
|
|
|
|
struct server_prompt_cache_state {
|
|
server_prompt prompt;
|
|
server_prompt_data data;
|
|
|
|
size_t size() const {
|
|
size_t res = data.size();
|
|
|
|
for (const auto & ckpt : prompt.checkpoints) {
|
|
res += ckpt->size();
|
|
}
|
|
|
|
return res;
|
|
}
|
|
};
|
|
|
|
struct server_prompt_cache {
|
|
server_prompt_cache(int32_t limit_size_mib, size_t limit_tokens) {
|
|
this->limit_size = 1024ull*1024ull*(limit_size_mib < 0 ? 0 : limit_size_mib);
|
|
this->limit_tokens = limit_tokens;
|
|
}
|
|
|
|
std::list<server_prompt_cache_state> states;
|
|
|
|
// in bytes, 0 = no limit
|
|
size_t limit_size = 0;
|
|
|
|
// in tokens, 0 = no limit
|
|
size_t limit_tokens = 0;
|
|
|
|
size_t size() const;
|
|
|
|
size_t n_tokens() const;
|
|
|
|
server_prompt_cache_state * alloc(const server_prompt & prompt, size_t state_size_main, size_t state_size_drft);
|
|
|
|
// lora_new: the lora config of the new prompt, the states made with another config are skipped
|
|
bool load(server_prompt & prompt, const server_tokens & tokens_new, const std::vector<common_adapter_lora_info> & lora_new, llama_context * ctx_tgt, llama_context * ctx_dft, int32_t id_slot);
|
|
|
|
void update();
|
|
};
|
|
|
|
// prompt states saved as files, to reuse a prompt after a server restart
|
|
struct server_prompt_disk_entry {
|
|
std::string path;
|
|
llama_tokens tokens;
|
|
std::vector<int64_t> ckpt_n_tokens;
|
|
};
|
|
|
|
struct server_prompt_disk_cache {
|
|
// key: model file and every parameter that changes the state layout, see the server context
|
|
server_prompt_disk_cache(const std::string & dir, int32_t limit_size_mib, const std::string & key);
|
|
|
|
std::string dir;
|
|
uint64_t key_hash;
|
|
size_t limit_size;
|
|
|
|
// entries of this key only, read from the file headers at startup
|
|
std::vector<server_prompt_disk_entry> entries;
|
|
|
|
// tokens of tokens_new that a state of `tokens` with checkpoints at `ckpt_n_tokens` can reuse
|
|
// a recurrent or SWA state is reusable only up to an exact prefix or a checkpoint
|
|
static size_t n_reuse(const llama_tokens & tokens, const std::vector<int64_t> & ckpt_n_tokens, const llama_tokens & tokens_new, bool can_trim);
|
|
|
|
// index of the entry that reuses the most of tokens_new, and at least n_min more than n_have; -1 if none
|
|
int find(const llama_tokens & tokens_new, size_t n_have, size_t n_min, bool can_trim) const;
|
|
|
|
bool has(const llama_tokens & tokens);
|
|
|
|
// read and verify entry i; a bad file is removed
|
|
bool load(int i, server_prompt & prompt, common_state_data & data, bool has_mtmd);
|
|
|
|
// write atomically, after evicting the oldest files in dir over the size limit
|
|
bool save(const server_prompt & prompt, const common_state_data & data);
|
|
|
|
void remove(int i);
|
|
};
|
|
|
|
// what the scheduler needs to know about a slot
|
|
struct server_slot_info {
|
|
int id = -1;
|
|
const server_task * task = nullptr; // only for logging
|
|
|
|
bool is_processing = false;
|
|
int64_t t_last_used = -1;
|
|
|
|
const server_prompt * prompt = nullptr;
|
|
};
|
|
|
|
struct server_slot_plan {
|
|
// -1 if no slot is available; a busy slot if the task asks for it by id, the caller defers the task
|
|
int id_slot = -1;
|
|
|
|
// save the slot state to the prompt cache, then load the best match for the task
|
|
bool update_cache = false;
|
|
};
|
|
|
|
// decides only, the caller applies the plan
|
|
struct server_slot_sched {
|
|
float slot_prompt_similarity = 0.0f;
|
|
|
|
// idle slots that lose the least by being reused: empty slots first, then the least recently used ones
|
|
std::vector<int> get_free_slots(const std::vector<server_slot_info> & slots, size_t n_slots_needed, int exclude_id_slot) const;
|
|
|
|
// lora: the lora config of the task
|
|
server_slot_plan select(const std::vector<server_slot_info> & slots, const server_task & task, const std::vector<common_adapter_lora_info> & lora) const;
|
|
};
|