server: merge user-message batch breaks when the rest of the prompt fits in one ubatch

Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last
user message (and of earlier ones on a fresh prompt) so that a context
checkpoint lands there. On a hybrid/recurrent model with MoE experts on the
host, that turns a typical chat turn (previous answer + new user message, under
one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the
host experts again.

When the rest of the prompt from the current batch start fits in one ubatch,
skip the user-start break. The near-end rule still checkpoints the batch start
and the end-4 rule the prompt end, so a later prompt that diverges inside the
last user message re-processes less than one ubatch more, in the same ubatch.
Prompts longer than one ubatch, and batches after a media chunk, keep the
upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-11-22 10:15:30 +01:00
1 parent 166ec22e2b
commit 510aa6e090
1 file changed
+14 -1
+14 -1
View File
@@ -4942,6 +4942,19 @@ private:
const int32_t n_decision_first = slot.task->type == SERVER_TASK_TYPE_DECISION ? slot.task->decision.pos_first() : -1;
// [fork F14] do not break at user-message starts when the rest of the prompt
// from here fits in one ubatch: the break turns one ubatch into two, and with
// MoE experts on the host each one of >= 32 tokens uploads the experts again.
// A checkpoint at this batch start (the near-end rule below) and at the end
// minus 4 still exist, so a later prompt that diverges inside the merged range
// re-processes less than one ubatch more. LLAMA_USERSTART_MERGE=0 disables.
static const bool userstart_merge = [] {
const char * env = getenv("LLAMA_USERSTART_MERGE");
return env == nullptr || std::string(env) != "0";
}();
const bool merge_user_starts = userstart_merge && !has_mtmd &&
slot.task->n_tokens() - slot.prompt.n_tokens() <= n_ubatch;
// add prompt tokens for processing in the current batch
while (slot.prompt.n_tokens() < slot.task->n_tokens() && batch.size() < n_batch) {
// get next token to process
@@ -4985,7 +4998,7 @@ private:
slot.prompt.tokens.push_back(cur_tok);
// break at the last user message, or at user messages at least min step past the last checkpoint
if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) {
if (do_checkpoint && !merge_user_starts && spans.is_user_start(slot.prompt.n_tokens())) {
const auto pos = slot.prompt.n_tokens();
const auto & checkpoints = slot.prompt.checkpoints;