server: disk prompt cache - text-only prompts with an mmproj loaded

The deploy of cb48bf224 with --cache-disk-path was rolled back: the first
request to the production 35B aborted in disk_cache_load().
server_tokens::get_tokens() has GGML_ASSERT(!has_mtmd), and has_mtmd is true
whenever an mmproj is loaded, even for a text-only prompt. Production's 35B
and gemma-26B always load one (-hf pulls it). disk_cache_collect() and
server_prompt_disk_cache::save() made the same call, so the shutdown save
would have aborted too. Every test had used --no-mmproj, which hid it.

disk_cache_text() takes the list from get_text_tokens(). A prompt is
text-only when that list is as long as size(), because media positions hold
LLAMA_TOKEN_NULL. So a text-only prompt is saved and restored with an mmproj
loaded, which is the main case (OpenWebUI chats on the 35B and gemma), and a
prompt with media is never saved or restored. When the slot itself holds
media, its reuse estimate is server_tokens::get_common_prefix(). These four
calls were the only get_tokens()/mtmd-unsafe accesses in the feature.

fork-validate --diskcache with the new mmproj cases, beside production:
15 PASS, 0 FAIL, GPU legs included. With the 0.8B's mmproj loaded, a text
chat is restored: prompt_n 58 of 596, output identical to the reference.
An image request is answered "red" with no restore, the text chat after it
is restored, and a SIGTERM with media in the slot copies no state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 21:38:58 +01:00
1 parent cb48bf2240
commit d9ca3278bf
2 files changed
+15 -6

No files matched your search

+2 -1
View File
@@ -2239,7 +2239,8 @@ bool server_prompt_disk_cache::load(int i, server_prompt & prompt, std::vector<u
}
bool server_prompt_disk_cache::save(const server_prompt & prompt, const std::vector<uint8_t> & data) {
const llama_tokens & tokens = prompt.tokens.get_tokens();
// text only, see disk_cache_collect
const llama_tokens tokens = prompt.tokens.get_text_tokens();
std::vector<disk_cache_ckpt> ckpts;
for (const auto & c : prompt.checkpoints) {