server: disk prompt cache - text-only prompts with an mmproj loaded
The deploy of cb48bf224 with --cache-disk-path was rolled back: the first
request to the production 35B aborted in disk_cache_load().
server_tokens::get_tokens() has GGML_ASSERT(!has_mtmd), and has_mtmd is true
whenever an mmproj is loaded, even for a text-only prompt. Production's 35B
and gemma-26B always load one (-hf pulls it). disk_cache_collect() and
server_prompt_disk_cache::save() made the same call, so the shutdown save
would have aborted too. Every test had used --no-mmproj, which hid it.
disk_cache_text() takes the list from get_text_tokens(). A prompt is
text-only when that list is as long as size(), because media positions hold
LLAMA_TOKEN_NULL. So a text-only prompt is saved and restored with an mmproj
loaded, which is the main case (OpenWebUI chats on the 35B and gemma), and a
prompt with media is never saved or restored. When the slot itself holds
media, its reuse estimate is server_tokens::get_common_prefix(). These four
calls were the only get_tokens()/mtmd-unsafe accesses in the feature.
fork-validate --diskcache with the new mmproj cases, beside production:
15 PASS, 0 FAIL, GPU legs included. With the 0.8B's mmproj loaded, a text
chat is restored: prompt_n 58 of 596, output identical to the reference.
An image request is answered "red" with no restore, the text chat after it
is restored, and a SIGTERM with media in the slot copies no state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
cb48bf2240
commit
d9ca3278bf
2 files changed
+15
-6
No files matched your search
@@ -2239,7 +2239,8 @@ bool server_prompt_disk_cache::load(int i, server_prompt & prompt, std::vector<u
|
||||
}
|
||||
|
||||
bool server_prompt_disk_cache::save(const server_prompt & prompt, const std::vector<uint8_t> & data) {
|
||||
const llama_tokens & tokens = prompt.tokens.get_tokens();
|
||||
// text only, see disk_cache_collect
|
||||
const llama_tokens tokens = prompt.tokens.get_text_tokens();
|
||||
|
||||
std::vector<disk_cache_ckpt> ckpts;
|
||||
for (const auto & c : prompt.checkpoints) {
|
||||
|
||||
Reference in new issue
Block a user