server : allow RANK pooling batch splitting for causal LLM rerankers (ie. Qwen3 and Qwen3-VL) (#28876)
* server : allow splitting RANK pooling for causal LLM rerankers Rerank models fall into two categories: bidirectional cross-encoders (BERT, etc.) that require all tokens in a single physical batch, and causal LLMs repurposed as rerankers (Qwen3, Qwen3-VL) that can use chunked prefill like any other decoder. Previously the server rejected all RANK-pooling inputs larger than n_ubatch, and the graph builder hardcoded QWEN3/QWEN3VL arch checks to determine last-token pooling. This broke long-document and multimodal reranking for causal models. Fix: expose llama_get_causal_attn(ctx) so the server can check the effective runtime attention type (reflecting any --attention override or set_causal_attn call). Also expose llama_model_is_causal(model) for querying the static architectural property from GGUF metadata. can_split() now permits chunked prefill for RANK pooling when the context is causal. The graph builder's inline arch check is replaced with the same cparams.causal_attn predicate, removing the duplication. Assisted-by: Opencode/Qwen3.8-27B * remove unused llama_model_is_causal, fix whitespace Assisted-by: opencode --------- Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
This commit is contained in:
1 parent
a97cce86a8
commit
4da6337767
5 files changed
+32
-8
No files matched your search
@@ -1259,6 +1259,10 @@ void llama_context::set_causal_attn(bool value) {
|
||||
sched_need_reserve = true;
|
||||
}
|
||||
|
||||
bool llama_context::get_causal_attn() const {
|
||||
return cparams.causal_attn;
|
||||
}
|
||||
|
||||
void llama_context::set_warmup(bool value) {
|
||||
LLAMA_LOG_DEBUG("%s: value = %d\n", __func__, value);
|
||||
|
||||
@@ -3933,6 +3937,10 @@ void llama_set_causal_attn(llama_context * ctx, bool causal_attn) {
|
||||
ctx->set_causal_attn(causal_attn);
|
||||
}
|
||||
|
||||
bool llama_get_causal_attn(const llama_context * ctx) {
|
||||
return ctx->get_causal_attn();
|
||||
}
|
||||
|
||||
void llama_set_warmup(llama_context * ctx, bool warmup) {
|
||||
ctx->set_warmup(warmup);
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user