server: disk prompt cache, save slots on shutdown and restore after a restart
llama-swap SIGTERMs a model when the owner switches models (only one large model fits in RAM), and switching back re-prefills the whole conversation: about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR, each slot with a prompt is saved to DIR on shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the file, so only the suffix is processed. Off by default. Save, in server_context::start_loop() after the loop returns: - disk_cache_collect() copies the state of each slot with >= 256 text tokens to host memory (llama_state_seq_get_data_ext, flags NONE) while the device buffers still exist. A cold server reads its released state through llama_context_restore_memory_host(), with no device use and no extra copy. - vram_share_exit() then frees the VRAM and releases the GPU token as before, so only the D2H copy delays the token release, never the disk IO. - disk_cache_flush() writes the files. A slot is skipped if its KV does not cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not decoded) or if the same token list is already on disk. File (server_prompt_disk_cache, server-task.cpp): - A 48-byte header (magic, version, key hash, sizes, payload hash), then a payload of 8-byte padded sections: tokens, checkpoint table, state, the checkpoint states. The hash is FNV-1a over the 64-bit words of the payload. - Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file. - The index is built at startup from the headers. A bad magic, a size that does not match the header, or an inconsistent table means the file is removed. A hash mismatch at load also removes it, and nothing is restored. Other keys and other versions are kept. A .tmp older than 1 h is removed. - --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the least recently used *.lpcd files of every model in DIR are removed until the new file fits; a restore updates the file mtime. Entries the new prompt extends are dropped after the write and do not count against the limit. A write also needs its size + 1 GiB free on the file system. A write error or a full disk is logged; the file is not saved and the exit stays clean. Key: the build commit, the model path + size + mtime, LoRA and control vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the K/V types, the flash-attn type, and the RoPE/YaRN parameters. Restore, in get_available_slot() after the RAM cache: - Only while warm (holding the GPU token), with cache_prompt on and no media tokens. The file must reuse at least 256 more tokens than the slot holds. - Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint at or below the common prefix can be used. A plain-KV model can be cut back to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is almost never a prefix of the next turn. The checkpoints are exact snapshots, and update_slots() rolls back to one exactly as it does for the RAM cache. They are restored with id_task -1, so one that fails to load means a re-prefill, not an abort. - The slot's current prompt goes to the RAM cache first. A set_data failure clears the slot and removes the file. Not supported with speculative decoding (warning, disabled): the draft state is not saved. Only the slots are saved, not the RAM cache entries. Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention): - 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms, exit 314 ms (125 ms without the cache). After the restart: read 166 ms, restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full prefill. The answer is identical to a server that never restarted. - 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS +162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms, turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill. - The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node overlay) stalled once: 5.8 s for 64.5 MiB. llama-swap v239 waits for the old process to exit before it starts the next model, and SIGKILLs a process that is still there after 30 s. So the exit save adds to the model switch time: about 1 s per GB on the PVC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
dd851fd0c9
commit
2874dcb107
5 files changed
+674
No files matched your search
@@ -10,8 +10,19 @@
|
||||
#include "speculative.h"
|
||||
#include "server-common.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cerrno>
|
||||
#include <chrono>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <filesystem>
|
||||
#include <sstream>
|
||||
|
||||
#if defined(__linux__)
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
#endif
|
||||
|
||||
//
|
||||
// task_params
|
||||
//
|
||||
@@ -1934,3 +1945,440 @@ void server_prompt_cache::update() {
|
||||
(const void *)&state, state.prompt.n_tokens(), state.prompt.checkpoints.size(), state.size() / (1024.0 * 1024.0));
|
||||
}
|
||||
}
|
||||
|
||||
//
|
||||
// server_prompt_disk_cache
|
||||
//
|
||||
|
||||
// file = header + payload; the payload is a list of sections, each padded to 8 bytes:
|
||||
// tokens, checkpoint table, state size, state, the state of each checkpoint
|
||||
// hash_payload is FNV-1a over the 64-bit words of the payload, so a truncated or corrupt file is never used
|
||||
static constexpr uint32_t DISK_CACHE_MAGIC = 0x4443504c; // "LPCD"
|
||||
static constexpr uint32_t DISK_CACHE_VERSION = 1;
|
||||
static constexpr const char * DISK_CACHE_EXT = ".lpcd";
|
||||
|
||||
// keep this much space free on the file system
|
||||
static constexpr uint64_t DISK_CACHE_MIN_FREE = 1024ull*1024ull*1024ull;
|
||||
|
||||
struct disk_cache_header {
|
||||
uint32_t magic;
|
||||
uint32_t version;
|
||||
uint64_t key_hash;
|
||||
uint64_t n_tokens;
|
||||
uint64_t n_ckpt;
|
||||
uint64_t size_payload;
|
||||
uint64_t hash_payload;
|
||||
};
|
||||
|
||||
struct disk_cache_ckpt {
|
||||
int64_t n_tokens;
|
||||
llama_pos pos_min;
|
||||
llama_pos pos_max;
|
||||
uint64_t size;
|
||||
};
|
||||
|
||||
static_assert(sizeof(disk_cache_header) == 48 && sizeof(disk_cache_ckpt) == 24, "disk cache layout");
|
||||
|
||||
static uint64_t disk_cache_pad(uint64_t n) {
|
||||
return (8 - n % 8) % 8;
|
||||
}
|
||||
|
||||
static uint64_t disk_cache_hash_bytes(const void * data, size_t n) {
|
||||
uint64_t h = 0xcbf29ce484222325ull;
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
h = (h ^ ((const uint8_t *) data)[i]) * 0x100000001b3ull;
|
||||
}
|
||||
return h;
|
||||
}
|
||||
|
||||
static uint64_t disk_cache_payload_size(uint64_t n_tokens, const std::vector<disk_cache_ckpt> & ckpts, uint64_t n_state) {
|
||||
const auto padded = [](uint64_t n) { return n + disk_cache_pad(n); };
|
||||
uint64_t res = padded(n_tokens*sizeof(llama_token)) + padded(ckpts.size()*sizeof(disk_cache_ckpt)) + sizeof(uint64_t) + padded(n_state);
|
||||
for (const auto & c : ckpts) {
|
||||
res += padded(c.size);
|
||||
}
|
||||
return res;
|
||||
}
|
||||
|
||||
// sequential section IO with the running payload hash
|
||||
struct disk_cache_io {
|
||||
FILE * f = nullptr;
|
||||
uint64_t hash = 0xcbf29ce484222325ull;
|
||||
bool ok = true;
|
||||
|
||||
~disk_cache_io() {
|
||||
if (f) {
|
||||
fclose(f);
|
||||
}
|
||||
}
|
||||
|
||||
void add_hash(const uint8_t * p, size_t size, const uint8_t * pad) {
|
||||
const size_t n_full = size - size % 8;
|
||||
for (size_t i = 0; i < n_full; i += 8) {
|
||||
uint64_t w;
|
||||
memcpy(&w, p + i, 8);
|
||||
hash = (hash ^ w) * 0x100000001b3ull;
|
||||
}
|
||||
if (size % 8) {
|
||||
uint64_t w;
|
||||
memcpy(&w, p + n_full, size % 8);
|
||||
memcpy((uint8_t *) &w + size % 8, pad, disk_cache_pad(size));
|
||||
hash = (hash ^ w) * 0x100000001b3ull;
|
||||
}
|
||||
}
|
||||
|
||||
void write(const void * src, size_t size) {
|
||||
static const uint8_t zero[8] = {};
|
||||
const size_t n_pad = disk_cache_pad(size);
|
||||
ok = ok && fwrite(src, 1, size, f) == size && fwrite(zero, 1, n_pad, f) == n_pad;
|
||||
add_hash((const uint8_t *) src, size, zero);
|
||||
}
|
||||
|
||||
void read(void * dst, size_t size) {
|
||||
uint8_t pad[8];
|
||||
const size_t n_pad = disk_cache_pad(size);
|
||||
ok = ok && fread(dst, 1, size, f) == size && fread(pad, 1, n_pad, f) == n_pad;
|
||||
if (ok) {
|
||||
add_hash((const uint8_t *) dst, size, pad);
|
||||
}
|
||||
}
|
||||
|
||||
// the data is in our buffers, the cached pages are waste (see llama_file::drop_cache)
|
||||
void drop_cache() {
|
||||
#if defined(__linux__)
|
||||
posix_fadvise(fileno(f), 0, 0, POSIX_FADV_DONTNEED);
|
||||
#endif
|
||||
}
|
||||
};
|
||||
|
||||
// open a file and read everything up to the state data: 1 = entry of this key, 0 = other key or version, -1 = bad file
|
||||
static int disk_cache_open(const std::string & path, uint64_t key_hash, disk_cache_io & io, disk_cache_header & hdr,
|
||||
llama_tokens & tokens, std::vector<disk_cache_ckpt> & ckpts, uint64_t & n_state) {
|
||||
std::error_code ec;
|
||||
const uint64_t size = std::filesystem::file_size(path, ec);
|
||||
io.f = ec ? nullptr : fopen(path.c_str(), "rb");
|
||||
if (io.f == nullptr || fread(&hdr, 1, sizeof(hdr), io.f) != sizeof(hdr) || hdr.magic != DISK_CACHE_MAGIC) {
|
||||
return -1;
|
||||
}
|
||||
if (hdr.version != DISK_CACHE_VERSION) {
|
||||
return 0;
|
||||
}
|
||||
if (size != sizeof(hdr) + hdr.size_payload || hdr.n_tokens > size/sizeof(llama_token) || hdr.n_ckpt > size/sizeof(disk_cache_ckpt)) {
|
||||
return -1;
|
||||
}
|
||||
if (hdr.key_hash != key_hash) {
|
||||
return 0;
|
||||
}
|
||||
tokens.resize(hdr.n_tokens);
|
||||
ckpts.resize(hdr.n_ckpt);
|
||||
io.read(tokens.data(), tokens.size()*sizeof(llama_token));
|
||||
io.read(ckpts.data(), ckpts.size()*sizeof(disk_cache_ckpt));
|
||||
io.read(&n_state, sizeof(n_state));
|
||||
if (!io.ok || n_state > size) {
|
||||
return -1;
|
||||
}
|
||||
for (const auto & c : ckpts) {
|
||||
if (c.size > size) {
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
return disk_cache_payload_size(hdr.n_tokens, ckpts, n_state) == hdr.size_payload ? 1 : -1;
|
||||
}
|
||||
|
||||
static void disk_cache_touch(const std::string & path) {
|
||||
std::error_code ec;
|
||||
std::filesystem::last_write_time(path, std::filesystem::file_time_type::clock::now(), ec);
|
||||
}
|
||||
|
||||
server_prompt_disk_cache::server_prompt_disk_cache(const std::string & dir, int32_t limit_size_mib, const std::string & key) : dir(dir) {
|
||||
key_hash = disk_cache_hash_bytes(key.data(), key.size());
|
||||
limit_size = 1024ull*1024ull*std::max(0, limit_size_mib);
|
||||
|
||||
size_t n_other = 0;
|
||||
size_t n_bad = 0;
|
||||
try {
|
||||
std::filesystem::create_directories(dir);
|
||||
const auto now = std::filesystem::file_time_type::clock::now();
|
||||
for (const auto & de : std::filesystem::directory_iterator(dir)) {
|
||||
const std::string path = de.path().string();
|
||||
if (!de.is_regular_file()) {
|
||||
continue;
|
||||
}
|
||||
// left by a writer that was killed
|
||||
if (path.find(std::string(DISK_CACHE_EXT) + ".tmp") != std::string::npos) {
|
||||
if (now - de.last_write_time() > std::chrono::hours(1)) {
|
||||
std::filesystem::remove(de.path());
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (de.path().extension() != DISK_CACHE_EXT) {
|
||||
continue;
|
||||
}
|
||||
disk_cache_io io;
|
||||
disk_cache_header hdr;
|
||||
server_prompt_disk_entry e;
|
||||
std::vector<disk_cache_ckpt> ckpts;
|
||||
uint64_t n_state = 0;
|
||||
const int res = disk_cache_open(path, key_hash, io, hdr, e.tokens, ckpts, n_state);
|
||||
if (res < 0) {
|
||||
SRV_WRN("disk cache: '%s' is truncated or corrupt, removing it\n", path.c_str());
|
||||
std::filesystem::remove(de.path());
|
||||
n_bad++;
|
||||
continue;
|
||||
}
|
||||
if (res == 0) {
|
||||
n_other++;
|
||||
continue;
|
||||
}
|
||||
e.path = path;
|
||||
for (const auto & c : ckpts) {
|
||||
e.ckpt_n_tokens.push_back(c.n_tokens);
|
||||
}
|
||||
entries.push_back(std::move(e));
|
||||
}
|
||||
} catch (const std::exception & err) {
|
||||
SRV_WRN("disk cache: error reading '%s': %s\n", dir.c_str(), err.what());
|
||||
}
|
||||
|
||||
SRV_INF("disk cache: '%s', %zu prompts of this model, %zu other files, %zu bad files removed, limit %.0f MiB\n",
|
||||
dir.c_str(), entries.size(), n_other, n_bad, limit_size / (1024.0 * 1024.0));
|
||||
}
|
||||
|
||||
size_t server_prompt_disk_cache::n_reuse(const llama_tokens & tokens, const std::vector<int64_t> & ckpt_n_tokens, const llama_tokens & tokens_new, bool can_trim) {
|
||||
size_t lcp = 0;
|
||||
while (lcp < tokens.size() && lcp < tokens_new.size() && tokens[lcp] == tokens_new[lcp]) {
|
||||
lcp++;
|
||||
}
|
||||
if (lcp == tokens.size() || can_trim) {
|
||||
return lcp;
|
||||
}
|
||||
size_t res = 0;
|
||||
for (const int64_t n : ckpt_n_tokens) {
|
||||
if (n <= (int64_t) lcp) {
|
||||
res = std::max(res, (size_t) n);
|
||||
}
|
||||
}
|
||||
return res;
|
||||
}
|
||||
|
||||
int server_prompt_disk_cache::find(const llama_tokens & tokens_new, size_t n_have, size_t n_min, bool can_trim) const {
|
||||
int best = -1;
|
||||
size_t n_best = 0;
|
||||
for (size_t i = 0; i < entries.size(); i++) {
|
||||
const size_t n = n_reuse(entries[i].tokens, entries[i].ckpt_n_tokens, tokens_new, can_trim);
|
||||
if (n >= n_have + n_min && n > n_best) {
|
||||
best = (int) i;
|
||||
n_best = n;
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
bool server_prompt_disk_cache::has(const llama_tokens & tokens) {
|
||||
for (const auto & e : entries) {
|
||||
std::error_code ec;
|
||||
if (e.tokens == tokens && std::filesystem::exists(e.path, ec)) {
|
||||
disk_cache_touch(e.path);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
void server_prompt_disk_cache::remove(int i) {
|
||||
std::error_code ec;
|
||||
std::filesystem::remove(entries[i].path, ec);
|
||||
entries.erase(entries.begin() + i);
|
||||
}
|
||||
|
||||
bool server_prompt_disk_cache::load(int i, server_prompt & prompt, std::vector<uint8_t> & data, bool has_mtmd) {
|
||||
const std::string path = entries[i].path;
|
||||
|
||||
disk_cache_io io;
|
||||
disk_cache_header hdr;
|
||||
llama_tokens tokens;
|
||||
std::vector<disk_cache_ckpt> ckpts;
|
||||
uint64_t n_state = 0;
|
||||
std::list<common_prompt_checkpoint> checkpoints;
|
||||
|
||||
bool ok = disk_cache_open(path, key_hash, io, hdr, tokens, ckpts, n_state) == 1 && tokens == entries[i].tokens;
|
||||
if (ok) {
|
||||
try {
|
||||
data.resize(n_state);
|
||||
io.read(data.data(), data.size());
|
||||
for (const auto & c : ckpts) {
|
||||
auto & cur = checkpoints.emplace_back();
|
||||
cur.id_task = -1; // not created by a task: may fail to load, then the prompt is re-processed
|
||||
cur.n_tokens = c.n_tokens;
|
||||
cur.pos_min = c.pos_min;
|
||||
cur.pos_max = c.pos_max;
|
||||
cur.data_tgt.resize(c.size);
|
||||
io.read(cur.data_tgt.data(), cur.data_tgt.size());
|
||||
}
|
||||
} catch (const std::bad_alloc &) {
|
||||
SRV_WRN("disk cache: out of memory reading '%s'\n", path.c_str());
|
||||
data.clear();
|
||||
return false;
|
||||
}
|
||||
ok = io.ok && io.hash == hdr.hash_payload;
|
||||
io.drop_cache();
|
||||
}
|
||||
if (!ok) {
|
||||
SRV_WRN("disk cache: '%s' is missing, truncated or corrupt, removing it\n", path.c_str());
|
||||
data.clear();
|
||||
remove(i);
|
||||
return false;
|
||||
}
|
||||
|
||||
disk_cache_touch(path);
|
||||
|
||||
prompt.tokens = server_tokens(tokens, has_mtmd);
|
||||
prompt.checkpoints = std::move(checkpoints);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
bool server_prompt_disk_cache::save(const server_prompt & prompt, const std::vector<uint8_t> & data) {
|
||||
const llama_tokens & tokens = prompt.tokens.get_tokens();
|
||||
|
||||
std::vector<disk_cache_ckpt> ckpts;
|
||||
for (const auto & c : prompt.checkpoints) {
|
||||
ckpts.push_back({ c.n_tokens, c.pos_min, c.pos_max, c.data_tgt.size() });
|
||||
}
|
||||
|
||||
disk_cache_header hdr = {
|
||||
/*.magic =*/ DISK_CACHE_MAGIC,
|
||||
/*.version =*/ DISK_CACHE_VERSION,
|
||||
/*.key_hash =*/ key_hash,
|
||||
/*.n_tokens =*/ tokens.size(),
|
||||
/*.n_ckpt =*/ ckpts.size(),
|
||||
/*.size_payload =*/ disk_cache_payload_size(tokens.size(), ckpts, data.size()),
|
||||
/*.hash_payload =*/ 0,
|
||||
};
|
||||
const uint64_t size_file = sizeof(hdr) + hdr.size_payload;
|
||||
|
||||
char name[64];
|
||||
snprintf(name, sizeof(name), "%016llx-%016llx", (unsigned long long) key_hash,
|
||||
(unsigned long long) disk_cache_hash_bytes(tokens.data(), tokens.size()*sizeof(llama_token)));
|
||||
const std::string path = (std::filesystem::path(dir) / name).string() + DISK_CACHE_EXT;
|
||||
const std::string path_tmp = path + ".tmp";
|
||||
|
||||
// entries contained in the new one are obsolete: removed once it is written, not counted for the limit
|
||||
const auto is_obsolete = [&](const server_prompt_disk_entry & e) {
|
||||
return e.path != path && e.tokens.size() <= tokens.size() && std::equal(e.tokens.begin(), e.tokens.end(), tokens.begin());
|
||||
};
|
||||
std::vector<std::string> obsolete;
|
||||
for (const auto & e : entries) {
|
||||
if (is_obsolete(e)) {
|
||||
obsolete.push_back(e.path);
|
||||
}
|
||||
}
|
||||
|
||||
if (limit_size > 0 && size_file > limit_size) {
|
||||
SRV_WRN("disk cache: prompt state of %.1f MiB exceeds the limit of %.1f MiB, not saved\n", size_file / (1024.0 * 1024.0), limit_size / (1024.0 * 1024.0));
|
||||
return false;
|
||||
}
|
||||
|
||||
try {
|
||||
// evict the least recently used files of every model in dir
|
||||
struct file_info {
|
||||
std::filesystem::path path;
|
||||
uint64_t size;
|
||||
std::filesystem::file_time_type time;
|
||||
};
|
||||
std::vector<file_info> files;
|
||||
uint64_t size_total = 0;
|
||||
for (const auto & de : std::filesystem::directory_iterator(dir)) {
|
||||
if (de.is_regular_file() && de.path().extension() == DISK_CACHE_EXT && de.path().string() != path &&
|
||||
std::find(obsolete.begin(), obsolete.end(), de.path().string()) == obsolete.end()) {
|
||||
files.push_back({ de.path(), de.file_size(), de.last_write_time() });
|
||||
size_total += files.back().size;
|
||||
}
|
||||
}
|
||||
std::sort(files.begin(), files.end(), [](const file_info & a, const file_info & b) { return a.time < b.time; });
|
||||
for (size_t i = 0; limit_size > 0 && size_total + size_file > limit_size && i < files.size(); i++) {
|
||||
SRV_INF("disk cache: size limit, removing '%s' (%.1f MiB)\n", files[i].path.string().c_str(), files[i].size / (1024.0 * 1024.0));
|
||||
std::filesystem::remove(files[i].path);
|
||||
size_total -= files[i].size;
|
||||
for (size_t j = 0; j < entries.size(); j++) {
|
||||
if (entries[j].path == files[i].path.string()) {
|
||||
entries.erase(entries.begin() + j);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
const uint64_t avail = std::filesystem::space(dir).available;
|
||||
if (avail < size_file + DISK_CACHE_MIN_FREE) {
|
||||
SRV_WRN("disk cache: only %.1f MiB free in '%s', prompt state of %.1f MiB not saved\n", avail / (1024.0 * 1024.0), dir.c_str(), size_file / (1024.0 * 1024.0));
|
||||
return false;
|
||||
}
|
||||
} catch (const std::exception & err) {
|
||||
SRV_WRN("disk cache: error reading '%s': %s\n", dir.c_str(), err.what());
|
||||
return false;
|
||||
}
|
||||
|
||||
bool ok;
|
||||
{
|
||||
disk_cache_io io;
|
||||
io.f = fopen(path_tmp.c_str(), "wb");
|
||||
ok = io.f != nullptr && fwrite(&hdr, 1, sizeof(hdr), io.f) == sizeof(hdr);
|
||||
if (ok) {
|
||||
const uint64_t n_state = data.size();
|
||||
io.write(tokens.data(), tokens.size()*sizeof(llama_token));
|
||||
io.write(ckpts.data(), ckpts.size()*sizeof(disk_cache_ckpt));
|
||||
io.write(&n_state, sizeof(n_state));
|
||||
io.write(data.data(), data.size());
|
||||
for (const auto & c : prompt.checkpoints) {
|
||||
io.write(c.data_tgt.data(), c.data_tgt.size());
|
||||
}
|
||||
hdr.hash_payload = io.hash;
|
||||
ok = io.ok && fseek(io.f, 0, SEEK_SET) == 0 && fwrite(&hdr, 1, sizeof(hdr), io.f) == sizeof(hdr) && fflush(io.f) == 0;
|
||||
#if defined(__linux__)
|
||||
// write back now, so the pages can be dropped and the rename never exposes missing data
|
||||
ok = ok && fdatasync(fileno(io.f)) == 0;
|
||||
io.drop_cache();
|
||||
#endif
|
||||
}
|
||||
if (io.f != nullptr && fclose(io.f) != 0) {
|
||||
ok = false;
|
||||
}
|
||||
io.f = nullptr;
|
||||
if (!ok) {
|
||||
SRV_WRN("disk cache: failed to write '%s': %s, not saved\n", path_tmp.c_str(), strerror(errno));
|
||||
}
|
||||
}
|
||||
std::error_code ec;
|
||||
if (ok) {
|
||||
std::filesystem::rename(path_tmp, path, ec);
|
||||
ok = !ec;
|
||||
if (!ok) {
|
||||
SRV_WRN("disk cache: failed to rename '%s': %s, not saved\n", path_tmp.c_str(), ec.message().c_str());
|
||||
}
|
||||
}
|
||||
if (!ok) {
|
||||
std::filesystem::remove(path_tmp, ec);
|
||||
return false;
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < entries.size(); ) {
|
||||
const auto & e = entries[i];
|
||||
if (e.path == path) {
|
||||
entries.erase(entries.begin() + i);
|
||||
} else if (is_obsolete(e)) {
|
||||
SRV_INF("disk cache: removing '%s', a prefix of the new prompt\n", e.path.c_str());
|
||||
remove((int) i);
|
||||
} else {
|
||||
i++;
|
||||
}
|
||||
}
|
||||
|
||||
server_prompt_disk_entry e;
|
||||
e.path = path;
|
||||
e.tokens = tokens;
|
||||
for (const auto & c : ckpts) {
|
||||
e.ckpt_n_tokens.push_back(c.n_tokens);
|
||||
}
|
||||
entries.push_back(std::move(e));
|
||||
|
||||
return true;
|
||||
}
|
||||
Reference in new issue
Block a user