Commit Graph
7 Commits
Author SHA1 Message Date
Pascal cb55ffe6e2 graph: skip the LM head for pooled embeddings (#30445)
* graph: skip the LM head for pooled embeddings

A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.

The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.

* graph: drop the redundant t_embd expand

Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.

* graph: add build_logits() for the LM head output

Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.

* graph: call build_logits() before build_sampling()

Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.

* nit
2026-10-13 16:16:14 +02:00
Xuan-Son Nguyen 781dbc5ac9 spec: properly handle mtmd input for mtp (#30257)
* spec: properly handle mtmd input for mtp

* nits
2026-10-10 11:22:38 +02:00
Sigbjørn Skjæret 97e4ca7358 models : fix incorrect uses of get_key_or_arr (#28868) 2026-09-14 14:05:37 +03:00
Sigbjørn Skjæret 73159c3039 model : fix gemma4-assistant (#28183) 2026-09-01 19:58:44 +03:00
Sigbjørn Skjæret 9d817213a0 model : load hparams.n_layer_nextn before n_layer() calls (#28159)
* load hparams.n_layer_nextn before n_layer() calls

* remove duplicate loads
2026-09-01 13:55:45 +02:00
Max Krasnyansky 7d2b45b4f7 mtp: support for gemma-4 E2B and E4B assistants (#24282)
* models: update converter to support smaller assistants

* models: add masked_embd tensors to gemma4-assist arch

* gemma-4: remove temp debug for conversion

* gemma-4-mtp: filter out masked_embedding tensors during conversion
2026-06-08 13:48:52 -07:00
Aman Gupta 04eb4c446d llama : add Gemma4 MTP (#23398) 2026-06-07 20:50:54 +08:00