Commit Graph
5 Commits
Author SHA1 Message Date
Xuan-Son Nguyen 6023692b1e llama: fix kv shifting, add tests, add per-model build_rope (#31683)
* llama: fix kv shifting and add tests

* llama: add graph build_rope (#31693)

* llama: add graph build_rope

* improve

* migrate new models

* update review skill

* fix ci

* add --kv-shift to CI
2026-11-05 21:10:36 +01:00
Xuan-Son Nguyen e0f96a1a12 llama: refactoring: move model to class member of llm_graph_context (#31733) 2026-11-05 15:41:03 +01:00
Pascal cb55ffe6e2 graph: skip the LM head for pooled embeddings (#30445)
* graph: skip the LM head for pooled embeddings

A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.

The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.

* graph: drop the redundant t_embd expand

Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.

* graph: add build_logits() for the LM head output

Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.

* graph: call build_logits() before build_sampling()

Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.

* nit
2026-10-13 16:16:14 +02:00
zqlandadmin d7a2074112 models : support nanbeige4.2-3B (#27730)
Co-authored-by: admin <lizongqiang@kanzhun.com>
2026-08-27 07:55:31 +03:00
zqlandroot b77d646751 model: Add support for Nanbeige4.2 (#25994)
* support nanbeige4.2 model

* fix

* fix flake8 Lint check

* fix loop bound check and drop redundant head_dim

---------

Co-authored-by: root <lizongqiang@kanzhun.com>
2026-07-27 17:04:18 +02:00