Xuan-Son Nguyen
19bc41dbb4
model: apply .scale tensors automatically + fix correctness ( #32669 )
...
* model: apply .scale tensors automatically
* rm build_moe_ffn "_s" overload
* use llama_prec_policy
* fix ci
* build_lora_mm_id: rm w_s
2026-11-21 16:30:01 +01:00
Xuan-Son Nguyen
6023692b1e
llama: fix kv shifting, add tests, add per-model build_rope ( #31683 )
...
* llama: fix kv shifting and add tests
* llama: add graph build_rope (#31693 )
* llama: add graph build_rope
* improve
* migrate new models
* update review skill
* fix ci
* add --kv-shift to CI
2026-11-05 21:10:36 +01:00
Xuan-Son Nguyen
e0f96a1a12
llama: refactoring: move model to class member of llm_graph_context ( #31733 )
2026-11-05 15:41:03 +01:00
Xuan-Son Nguyen
2851b3bcbf
mtmd: add mtmd_audio_stream + support voxtral-realtime ( #30629 )
...
* mtmd: add mtmd_audio_stream + support voxtral-realtime
* address review
2026-10-16 01:46:42 +02:00
Pascal
cb55ffe6e2
graph: skip the LM head for pooled embeddings ( #30445 )
...
* graph: skip the LM head for pooled embeddings
A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.
The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.
* graph: drop the redundant t_embd expand
Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.
* graph: add build_logits() for the LM head output
Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.
* graph: call build_logits() before build_sampling()
Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.
* nit
2026-10-13 16:16:14 +02:00
Georgi Gerganov and Sigbjørn Skjæret
7acb4e8cd2
hparams : refactor hparams.n_layer ( #24060 )
...
* hparams : refactor hparams.n_layer
* cont : remove `n_layer_kv()`, use n_layer_all instead
* cont : type consistency
* pi : update SYSTEM.md
* models : fix Step3.5 MTP
* cont : remove duplicate switch cases
* cont : explicitly set `false` to extra layers for `is_swa` and `is_recr`
* cont : fix nextn layer count handling
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-06-05 11:09:36 +03:00
Michael Wand
6fe90deffa
models : Attach Mistral3 NVFP4 weight scales ( #23629 )
2026-05-26 07:59:59 +03:00
ynankani
42928bc14d
model : NvFP4 quantized LM head support ( #23046 )
...
* NvFP4 quantized LM head support
Signed-off-by: ynankani <ynankani@nvidia.com >
* Address review commnets
Signed-off-by: ynankani <ynankani@nvidia.com >
* Add assert for NvFp4 lm head and tied embeddings
Signed-off-by: ynankani <ynankani@nvidia.com >
* Address review commnets
Signed-off-by: ynankani <ynankani@nvidia.com >
* Create output_s tensor only when LM head NvFp4
Signed-off-by: ynankani <ynankani@nvidia.com >
---------
Signed-off-by: ynankani <ynankani@nvidia.com >
2026-05-16 11:09:27 +02:00
Xuan-Son Nguyen
994118a183
model: move load_hparams and load_tensors to per-model definition ( #22004 )
...
* git-friendly migration
* add build_graph
* nits
* exclude old code from build
* wip
* add llm_arch_model_i
* prepare downstream functions
* nits
* nits
* wip
* wip
* add back create_tensor_qkv
* fix files missing include
* enforce one llm_build per arch
* cmake: use glob
* missing model params
* nits
* wip
* wip (2)
* wip (3)
* test-llama-archs is happy
* improve switch case
* move more stuff into llm_arch_model_i
* fix downstream code
* nits
* nits (2)
* fix order
* llama_model_base
* LLAMA_LOAD_LOCALS
* small fix
* fix build errors
* auto
* rm migration script and ifdef
2026-05-04 12:36:59 +02:00
Sigbjørn Skjæret
4f02d47339
model : refactor bias tensor variable names ( #22079 )
...
* refactor bias tensor variable names
* use create_tensor_qkv for jina-bert-v2
2026-04-18 20:12:00 +02:00
PikaPikachu
9db77a020c
model : refactor QKV into common build_qkv and create_tensor_qkv helpers ( #21245 )
...
* model : refactor QKV into common build_qkv and create_tensor_qkv helpers
* model : extend build_qkv to bert/mpt/dbrx/olmo/lfm2/nemotron-h/granite-hybrid/gemma3n-iswa/t5-dec and fix wqkv_s
2026-04-16 17:41:34 +02:00
Sigbjørn Skjæret
f772f6e434
model : support NVFP4 tensors for Gemma4 ( #21971 )
...
* support nvfp4 tensors for Gemma4
* add wo_s to build_attn
* add wo_s to build_attn
* fix glm4
2026-04-16 16:51:47 +02:00
Xuan-Son Nguyen
59db9a357d
llama: dynamic head_dim and n_rot for SWA ( #20301 )
...
* llama: dynamic head_dim and n_rot for SWA
* also add gguf_writer wrappers
* fix build
* build_rope_shift arg reorder
2026-03-09 22:22:39 +01:00
Sigbjørn Skjæret
35bee031e1
graph : remove redundant scale_w parameter ( #20235 )
2026-03-08 18:58:28 +01:00
cd3c118908
model: support Ministral3 ( #17644 )
...
* conversion script
* support ministral 3
* maybe this is better?
* add TODO for rope_yarn_log_mul
* better ppl (tested on 14B-Instruct)
* Add Ministral3 support to Mistral format
* improve arch handling
* add sizes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* nits
---------
Co-authored-by: Julien Denize <julien.denize@mistral.ai >
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-01 12:26:52 +01:00