Commit Graph
18 Commits
Author SHA1 Message Date
jjperry69andGabe Goodhart 0c6dca2fae models : Attach Granite NVFP4 weight scales (#31449)
* models : Attach Granite NVFP4 weight scales

* feat: Pass nvfp4 scales through build_ffn for all granite architectures

Branch: granite-nvfp4-moe-scales
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

---------

Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>
2026-11-09 13:04:06 +01:00
Xuan-Son Nguyen 6023692b1e llama: fix kv shifting, add tests, add per-model build_rope (#31683)
* llama: fix kv shifting and add tests

* llama: add graph build_rope (#31693)

* llama: add graph build_rope

* improve

* migrate new models

* update review skill

* fix ci

* add --kv-shift to CI
2026-11-05 21:10:36 +01:00
Xuan-Son Nguyen e0f96a1a12 llama: refactoring: move model to class member of llm_graph_context (#31733) 2026-11-05 15:41:03 +01:00
Gabe Goodhart bf039c8496 models : Record layer inputs for all granite models (#30957)
Branch: GraniteSpeculatorSupport
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-22 10:02:41 +03:00
Pascal cb55ffe6e2 graph: skip the LM head for pooled embeddings (#30445)
* graph: skip the LM head for pooled embeddings

A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.

The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.

* graph: drop the redundant t_embd expand

Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.

* graph: add build_logits() for the LM head output

Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.

* graph: call build_logits() before build_sampling()

Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.

* nit
2026-10-13 16:16:14 +02:00
Aaron Teo d7e86430a7 model: fix all granite family parameter counts (#28643)
* model: fix all granite family parameter counts

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* model: fix additional include, add missing `A` prefix for active experts

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* model: fix code alignment, rm unused 40 block case

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-09-10 09:49:56 +03:00
Gabe GoodhartandSigbjørn Skjæret 7221e24f57 model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505)
* feat(convert): Add conversion for GraniteSWAForCausalLM

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(llama): Add granite_swa support

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(conversion): Add conversion infra for rope_pattern array

NOTE: There is other work also targeting this, so this may be
removed depending on merge order.

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix(conversion): Fix SWA pattern logic and support for non-rope layers

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(conversion): Add support for GraniteMoeSWA

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Add llama_hparams::has_rope and arch constants

NOTE: This shadows the work done for Granite Speech
https://github.com/ggml-org/llama.cpp/pull/25107

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Add support for per-layer rope determination

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: Fix failing flake8 for extra newlines

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver

Branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Load MoE params as optional

Branch: GraniteSWAForCausalLM
AI-usage: draft (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Handle MoE params in conversion

branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: Remove unnecessary newline

AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Remove unnecessary tensor additions to GRANITE architecture

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Correctly handle naming for ffn gate inp

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Always default hparams.rope_pattern to 1s

This isn't strictly necessary, but it will allow other models to rely on
hparams.has_rope(il) without needting to prepopulate.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Move to has_rope for all granite model architectures

Now that we have a proper hparam for this, it's better to use it and not
require a hacky fallback in the hparam method itself.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: No hacky rope_finetuned fallback in has_rope

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Fully remove rope hparam filling in granitemoe

There are no granitemoe models that use NoPE (it's not actually used in the
layer building below), so this was just dead code.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Save out rope_pattern in model-saver

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Set hparams.rope_finetuned for round trip

Since the value is _read_ from rope_finetuned, we need to persist it when
the model is saved with the saver.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Code review cleanup

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* refactor: Keep gate/up fused for MoE path

Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Skip GRANITE_SWA in model saver

https://github.com/ggml-org/llama.cpp/pull/25505#discussion_r3773175651

Keeping is_swa_impl in the saver can break other models.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* add sliding window pattern for model in test

* style: Fix indentation

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Fix \r\n

Thanks Claude!

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Keep shared expert fused

Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: More indentation fixes

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-08-19 16:53:31 +02:00
Georgi GerganovandSigbjørn Skjæret 7acb4e8cd2 hparams : refactor hparams.n_layer (#24060)
* hparams : refactor hparams.n_layer

* cont : remove `n_layer_kv()`, use n_layer_all instead

* cont : type consistency

* pi : update SYSTEM.md

* models : fix Step3.5 MTP

* cont : remove duplicate switch cases

* cont : explicitly set `false` to extra layers for `is_swa` and `is_recr`

* cont : fix nextn layer count handling

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
2026-06-05 11:09:36 +03:00
Georgi Gerganov 06938ac129 tests : add support for qwen3 SSM archs (#24031)
* tests : add support for qwen3 SSM archs

* arch : add LLM_KV_ATTENTION_RECURRENT_LAYERS

* cont : naming + TODOs
2026-06-03 10:15:27 +03:00
ynankani 42928bc14d model : NvFP4 quantized LM head support (#23046)
* NvFP4 quantized LM head support

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review commnets

Signed-off-by: ynankani <ynankani@nvidia.com>

* Add assert for NvFp4 lm head and tied embeddings

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review commnets

Signed-off-by: ynankani <ynankani@nvidia.com>

* Create output_s tensor only when LM head NvFp4

Signed-off-by: ynankani <ynankani@nvidia.com>

---------

Signed-off-by: ynankani <ynankani@nvidia.com>
2026-05-16 11:09:27 +02:00
Xuan-Son Nguyen 994118a183 model: move load_hparams and load_tensors to per-model definition (#22004)
* git-friendly migration

* add build_graph

* nits

* exclude old code from build

* wip

* add llm_arch_model_i

* prepare downstream functions

* nits

* nits

* wip

* wip

* add back create_tensor_qkv

* fix files missing include

* enforce one llm_build per arch

* cmake: use glob

* missing model params

* nits

* wip

* wip (2)

* wip (3)

* test-llama-archs is happy

* improve switch case

* move more stuff into llm_arch_model_i

* fix downstream code

* nits

* nits (2)

* fix order

* llama_model_base

* LLAMA_LOAD_LOCALS

* small fix

* fix build errors

* auto

* rm migration script and ifdef
2026-05-04 12:36:59 +02:00
Sigbjørn Skjæret 4f02d47339 model : refactor bias tensor variable names (#22079)
* refactor bias tensor variable names

* use create_tensor_qkv for jina-bert-v2
2026-04-18 20:12:00 +02:00
PikaPikachu 9db77a020c model : refactor QKV into common build_qkv and create_tensor_qkv helpers (#21245)
* model : refactor QKV into common build_qkv and create_tensor_qkv helpers

* model : extend build_qkv to bert/mpt/dbrx/olmo/lfm2/nemotron-h/granite-hybrid/gemma3n-iswa/t5-dec and fix wqkv_s
2026-04-16 17:41:34 +02:00
Sigbjørn Skjæret f772f6e434 model : support NVFP4 tensors for Gemma4 (#21971)
* support nvfp4 tensors for Gemma4

* add wo_s to build_attn

* add wo_s to build_attn

* fix glm4
2026-04-16 16:51:47 +02:00
Xuan-Son Nguyen 59db9a357d llama: dynamic head_dim and n_rot for SWA (#20301)
* llama: dynamic head_dim and n_rot for SWA

* also add gguf_writer wrappers

* fix build

* build_rope_shift arg reorder
2026-03-09 22:22:39 +01:00
Sigbjørn Skjæret 35bee031e1 graph : remove redundant scale_w parameter (#20235) 2026-03-08 18:58:28 +01:00
Georgi Gerganov cc45f2ada6 models : deduplicate delta-net graphs for Qwen family (#19597)
* models : add llm_build_delta_net_base

* cont : keep qwen35 and qwen35moe graphs intact

* cont : add comments
2026-02-16 14:35:04 +02:00
Piotr Wilkin (ilintar)andSigbjørn Skjæret bea04522ff refactor : llama-model.cpp (#16252)
* Sqashed: llama-model.cpp refactoring

* Fix formatting of attn / ffn / ffn_moe calls

* Fix import regression / unify spacing in models.h

* totally DID NOT miss those!

* Add missing qwen3vl(moe) models

* Add missing new .cpp files to build

* Remove extra semicolons

* Editor checker

* Update src/models/models.h

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
2025-10-31 23:40:23 +01:00