server: merge user-message batch breaks when the rest of the prompt fits in one ubatch
Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last user message (and of earlier ones on a fresh prompt) so that a context checkpoint lands there. On a hybrid/recurrent model with MoE experts on the host, that turns a typical chat turn (previous answer + new user message, under one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the host experts again. When the rest of the prompt from the current batch start fits in one ubatch, skip the user-start break. The near-end rule still checkpoints the batch start and the end-4 rule the prompt end, so a later prompt that diverges inside the last user message re-processes less than one ubatch more, in the same ubatch. Prompts longer than one ubatch, and batches after a media chunk, keep the upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
1 parent
166ec22e2b
commit
510aa6e090
1 file changed
+14
-1
@@ -4942,6 +4942,19 @@ private:
|
||||
|
||||
const int32_t n_decision_first = slot.task->type == SERVER_TASK_TYPE_DECISION ? slot.task->decision.pos_first() : -1;
|
||||
|
||||
// [fork F14] do not break at user-message starts when the rest of the prompt
|
||||
// from here fits in one ubatch: the break turns one ubatch into two, and with
|
||||
// MoE experts on the host each one of >= 32 tokens uploads the experts again.
|
||||
// A checkpoint at this batch start (the near-end rule below) and at the end
|
||||
// minus 4 still exist, so a later prompt that diverges inside the merged range
|
||||
// re-processes less than one ubatch more. LLAMA_USERSTART_MERGE=0 disables.
|
||||
static const bool userstart_merge = [] {
|
||||
const char * env = getenv("LLAMA_USERSTART_MERGE");
|
||||
return env == nullptr || std::string(env) != "0";
|
||||
}();
|
||||
const bool merge_user_starts = userstart_merge && !has_mtmd &&
|
||||
slot.task->n_tokens() - slot.prompt.n_tokens() <= n_ubatch;
|
||||
|
||||
// add prompt tokens for processing in the current batch
|
||||
while (slot.prompt.n_tokens() < slot.task->n_tokens() && batch.size() < n_batch) {
|
||||
// get next token to process
|
||||
@@ -4985,7 +4998,7 @@ private:
|
||||
slot.prompt.tokens.push_back(cur_tok);
|
||||
|
||||
// break at the last user message, or at user messages at least min step past the last checkpoint
|
||||
if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) {
|
||||
if (do_checkpoint && !merge_user_starts && spans.is_user_start(slot.prompt.n_tokens())) {
|
||||
const auto pos = slot.prompt.n_tokens();
|
||||
const auto & checkpoints = slot.prompt.checkpoints;
|
||||
|
||||
|
||||
Reference in new issue
Block a user