graph: skip the LM head for pooled embeddings (#30445)
* graph: skip the LM head for pooled embeddings A context that extracts pooled embeddings outputs one vector per sequence, but every model still expanded its LM head, so the graph computed n_vocab logits per token and the context kept a host buffer for them. The models now only declare t_logits, and set_outputs() expands it unless the context extracts embeddings with a pooling type other than NONE. Since the toggle changes the graph shape for a pooled context, set_embeddings() schedules a re-reserve in that case. * graph: drop the redundant t_embd expand Address review from @CISC: with embeddings on, build_pooling() already expands its output for every pooling type, and a decoder graph without embeddings does not read t_embd, so set_outputs() only expands t_logits. * graph: add build_logits() for the LM head output Address review from @ggerganov: the LM head is expanded by a dedicated build_logits(), called from llama_model::build_graph() like build_pooling(), and set_outputs() only marks the outputs again. * graph: call build_logits() before build_sampling() Address review from @ggerganov: build_sampling() reads t_logits, so the LM head is built first. A pooled context then skips the sampling layers too, since it has no logits to sample from. * nit
This commit is contained in:
1 parent
a17e94d229
commit
cb55ffe6e2
136 files changed
+27
-282
No files matched your search
@@ -4063,6 +4063,18 @@ void llm_graph_context::build_pooling(
|
||||
ggml_build_forward_expand(gf, cur);
|
||||
}
|
||||
|
||||
void llm_graph_context::build_logits() const {
|
||||
// a pooled embeddings context outputs one vector per sequence, so its graph skips the LM head
|
||||
if (cparams.embeddings && pooling_type != LLAMA_POOLING_TYPE_NONE) {
|
||||
res->t_logits = nullptr;
|
||||
}
|
||||
|
||||
// the models declare t_logits without expanding it, so the LM head is computed only when it is expanded here
|
||||
if (res->t_logits != nullptr) {
|
||||
ggml_build_forward_expand(gf, res->t_logits);
|
||||
}
|
||||
}
|
||||
|
||||
void llm_graph_context::build_sampling() const {
|
||||
if (samplers.empty() || !res->t_logits) {
|
||||
return;
|
||||
|
||||
Reference in new issue
Block a user