Commit Graph
274 Commits
Author SHA1 Message Date
Hrishith Thadicherla a11f57ba93 model : fix DFlash output head sharing (#30111)
* llama : fix DFlash output head sharing

Assisted-by: Codex

* dflash : read tied output weights from GGUF metadata

Assisted-by: Codex

* llama : share tied word embedding metadata

Assisted-by: Codex

* llama : remove DFlash embedding head fallback

Assisted-by: Codex
2026-10-08 17:42:28 +02:00
bosh 37ac634566 model : support classifier_activation for rerankers (#29692)
* model : support classifier_activation for rerankers

Assisted-by: Claude Opus 5.5

* model : map classifier gelu to gelu_erf and accept tanh

Assisted-by: Claude Opus 5.5

* model : default act_cls to tanh, ModernBERT falls back to gelu_erf

Assisted-by: Claude Opus 5.5
2026-10-08 09:29:20 +03:00
Tarek Dakhran a657f7e981 model : add LiquidAI/d1-omni-600M decision model (#30114)
* model : add LiquidAI/d1-omni-600M decision model

Assisted-by: Claude Opus 5.5

* mtmd : keep conformer GLU sigmoid on CUDA

Assisted-by: Claude Opus 5.5

* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal

Assisted-by: Claude Opus 5.5

* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files

Assisted-by: Claude Opus 5.5
2026-10-08 02:07:49 +02:00
Pascal 448147d42a llama: share the nextn tensor flags between models (#30097)
* llama: share the nextn tensor flags between models

Follow-up of the TODO in glm5-next: move the trunk-only and MTP-only
detection that each model copied into a nextn_flags helper of
llama_model_base. It probes the first trunk layer and the first NextN
layer, and adds TENSOR_SKIP when MTP is not loaded. qwen4exp probes
hc_attn_norm since it has no attn_norm.

deepseek4, nemotron-h, qwen35, qwen35moe, qwen3next and qwen4exp now
also accept a trunk-only file, like the other models.

* llama: avoid capturing structured bindings in the nextn flags

Lambdas that capture structured bindings need C++20, and GCC 15
rejects them under -Werror, so the models read the trunk and MTP
flags into plain variables.
2026-10-07 16:30:00 +02:00
b9acf138a1 feat: add GLM5Next MTP, optimize (#29928)
* llama : add GLM5-Next NextN (MTP) graph

Build the GLM5-Next multi-token-prediction head as graph_mtp: the NextN block
embeds enorm(tok)+hnorm(h) through eh_proj, runs one plain DSA layer and the
shared lm_head, reusing the trunk's builders through the no_build tag ctor.
llama_memory_recurrent also tolerates a partial seq_rm when the context holds
no recurrent layers, which is what the MTP draft context needs.

Assisted-by: Claude

* llama : glm5-next: skip dead compute in headless NextN forwards

A NextN forward with no output rows (the MTP catch-up and the draft-context
prefill) persists only through its cache writes, so the headless graph keeps
the MLA latent, indexer key|gate and pooled-key writes and drops the query
path, the indexer selection, the attention body, the FFN and the LM head. The
4-token catch-up falls from 6.9 ms to 0.33 ms of kernels; the greedy output
hashes and the draft acceptance are unchanged.

Assisted-by: Claude

* llama : glm5-next: fix NextN extraction contracts and shared-tail rollback

Three fixes from the architectural review. The headless graph prune now also
requires that no unmasked nextn extraction is live, because that mode reads
n_tokens hidden rows regardless of the logits flags. Masked extraction
publishes the hidden rows gathered by the output ids, so a batch whose output
flags are not a prefix exports the right rows. A partial recurrent rollback
whose tail cell is shared with another sequence is now rejected instead of
silently moving that sequence's tail.

Assisted-by: Claude

* llama : glm5-next: tidy comments in the MTP changes

Assisted-by: Claude

* llama : glm5-next: crop the MTP graph to the output rows instead of pruning it

Replace the headless NextN prune with the crop pattern the other MTP
graphs use: gather the attention output and the block input at the
output ids before the position-wise FFN and the shared head. A NextN
forward with no output rows (the MTP catch-up and the draft-context
prefill) then runs the FFN and the head over zero rows. The 4-token
catch-up falls from 6.9 ms to 2.9 ms of kernels; greedy output hashes
are unchanged.

Assisted-by: Claude

* glm5-next: use the nextn crop helpers in the MTP graph

Replace the local crop condition and the masked select of t_h_nextn
with crop_before_nextn and crop_after_nextn, so the MTP graph narrows
its rows the same way as the main graph and the other models.

Describe the shared cell and empty filter branches of the recurrent
partial rollback.

* glm5-next: load MTP-only and trunk-only GGUF files

Make the trunk tensors optional when the file only holds the NextN
layer, and the NextN tensors optional when the file only holds the
trunk, so the split MTP GGUF loads as a draft model.

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-07 13:16:53 +02:00
4625240437 model : add K2 Horizon dense and MoVA support (#29535)
* model: K2 Horizon gguf conversion code

* model: loading hparams and tensors in k2-horizon.cpp

* model: K2 Horizon compute graph

* model: K2 Horizon compute graph adjustment and registering tokenizers

* model: K2 Horizon chat template and accomodate safetensors naming

* unicode : add the K2-Horizon pre-tokenizer splitter

The K2-Horizon regex had no arm in unicode_regex_split_custom and fell through to the
general std::regex fallback, which fails two ways.

On MSVC std::regex rejects \p{...}, so no K2-Horizon GGUF loads on Windows at all:
llama-quantize, llama-imatrix and llama-perplexity all abort with
regex_error(error_escape) before a token is produced.

Where the fallback does compile it is still wrong. unicode_regex_split collapses each
codepoint to a single byte naming its Unicode category before matching, and U+200C/U+200D
are category Control, which has no entry in k_ucat_cpt, so both become the 0xD0 fallback
byte. The literal ‌ and ‍ alternatives in K2's regex can then never match and
every ZWNJ or ZWJ ends a letter run.

The splitter is the existing llama3 one with a single rule widened, since K2's regex
differs from llama3's only in that a letter run also takes marks, ZWNJ and ZWJ.

tests/test-unicode.cpp gains a case for this: it fails before the change with
[Amy] [ZWNJ khaham] and passes after with the run intact.

* tests: expand K2 Horizon unicode splitter coverage

* unicode: handle K2 Horizon case folding and empty input

Assisted-by: Codex

* jinja : support sequence indices in selectattr and rejectattr

Assisted-by: Codex

* model : add K2 Horizon dense and MoVA support

Includes the K2 Horizon implementation from ifm-ai/llama.cpp with converter, tensor-parallel and model save/reload fixes.

Assisted-by: Codex

* chat : support K2 Horizon reasoning and tool calls

Assisted-by: Codex

* conversion: remove obsolete K2 Aurora alias

Assisted-by: Codex

* k2-horizon: enforce response schemas and load YaRN betas

Constrain final JSON after reasoning, accept flexible JSON tool envelopes,
enforce XML dialects, and handle repeated or alternate thinking markers.
Load YaRN beta metadata instead of retaining the default values.

Add schema, streaming, continuation, and model reload regressions. Validate
CUDA and CPU builds and 0.9B, 4B, and MoVA conversation/tool round trips.

Assisted-by: Codex

* renaming template fixture

* adressing cisc follows ups

* desloppify the parser / adress aldehir comments

* clean test-chat

* remove fallback : model trained mostly on high anyway

* fix k2 attn_v_exp tn splitting and metal fusion baseline

* k2-horizon : forward expand views before sums

* k2-horizon: copy embds before group norm to fix TP

* disable tesnor parallelism

---------

Co-authored-by: Ryandito Diandaru <ryandito.diandaru@mbzuai.ac.ae>
Co-authored-by: WestWaters <mario.papaleo2013@gmail.com>
Co-authored-by: Natani L. Mayday <71436458+TaskPuppyNatani@users.noreply.github.com>
Co-authored-by: West <100190545+WestWaters@users.noreply.github.com>
Co-authored-by: aaryamonvikram <aaryamonvikram@gmail.com>
Co-authored-by: aaryamonvikram <96529820+aaryamonvikram@users.noreply.github.com>
2026-10-06 18:42:24 +02:00
Pascal 3109914090 llama: remove the gather path of the glm5-next sparse attention (#30042)
The gather path attended over the selected latents with a plain
matmul and softmax. It only ran with n_ubatch <= 16, and the flash
attention backends now skip the masked rows through n_kv_max, so
the scatter path covers every case.

Drop the gather flag, the gathered attention branch and
gather_mla_rows. set_input_kpool always maps padding to the n_kv
sentinel, and the slot mask becomes sel_mask since only the
scatter reads it.
2026-10-06 18:31:13 +02:00
Xuan-Son Nguyen 4fbc76dec5 model: support embeddinggemma2 (text+vision+audio) (#30054) 2026-10-06 18:20:31 +02:00
Georgi Gerganov f0c41e0168 models : consolidate nextn row cropping into shared helpers (#30017)
* mimo2 : always emit h_nextn

the other nextn-capable models set it unconditionally

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

* models : consolidate nextn row cropping into shared helpers

- replace the duplicated crop conditions and the per-model flags (narrow_early,
  crop_before_ffn, crop_last_layer, emit_h_nextn) with two helpers on llm_graph_context:
  crop_before_nextn() / crop_after_nextn()
- models that only tested embeddings_nextn_masked now share the same condition, so they
  crop the last layer before the nextn capture whenever extraction is off
- t_h_nextn is now set unconditionally in mimo2, qwen4exp and deepseek4 (as in the other
  nextn-capable models); host-side reads stay gated by cparams.embeddings_nextn

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
2026-10-06 14:08:54 +03:00
Pascal 43fe9c6428 llama: fix k-pool scatter data race on shared sequences (#29994)
* llama: re-pool each shared k-pool rep once

With shared cells every pool is re-pooled, and since the pooled keys
are always scattered, the pools a seq_cp shares between sequences
wrote the same rep row from several scatter entries, a data race on
the CPU backend. Mark each rep once: the sharing sequences read the
same row through pool_cells.

* llama: assert whole-sequence seq_cp in the hybrid idx memory

The recurrent state is always copied whole whatever the range, and a
k-pool cell shared by a partial copy could carry two pool groupings
with a single pooled row. Every caller copies whole sequences, so
reject partial ranges instead of supporting them.

* llama: drop the k-pool cache_safe mode

With whole-sequence seq_cp, sequences sharing cells share their pools
too, so the pooled row of a shared rep is valid for all of them. Mark
each rep once in every ubatch instead of re-pooling everything while
cells are shared, which removes the sharing scan and the stale-all
workarounds in seq_rm, state_read and state_drop. seq_cp now only
stales the destination.
2026-10-06 07:41:57 +02:00
Xuan-Son Nguyen 9871df5911 server: support vision input for Clef (#29969)
* server: support vision input for Clef

* move input_attn_causal to private

* extend old server_batch::embd

* server_batch::token::pos to multi dim

* nits

* fix abort

* fix img tokens cap

* fix yield_to_queue mutate data
2026-10-05 14:02:40 +02:00
Georgi GerganovandPascal c173a53bdf llama : fix unexpected graph reallocation in the k-pool models (#29958)
* llama : fix unexpected graph reallocation in the k-pool models

Both k-pool models built a graph shape that depends on state the
full-context reserve cannot know:

- qwen4exp branched on inp->cache_safe, which turns false as soon as
  llama_memory_seq_cp shares cells (e.g. batched-bench -pps): the QSA
  layers swapped scatter+gather for fill+concat and dropped the
  new_pool_rep leaf, so the decode graph had 12 fewer nodes than the
  reserved one
- glm5-next branched on gather = n_tokens <= 16 && n_kv > n_sel, so the
  TG decode built the gather shape (7564 nodes) while the last reserve,
  the PP one, had the dense shape (7762 nodes)

Either mismatch forces a decode-time re-reserve that drops the
worst-case sizing and bakes in the current state, so the next state
growth (n_pool, n_kv, n_new) needs more room at an unchanged graph size
and aborts under GGML_SCHED_DEBUG_REALLOC=1. Reproduce with, e.g.:

  GGML_SCHED_DEBUG_REALLOC=1 ./bin/llama-batched-bench \
    -hf ggml-org/GLM-5.3-Flash-GGUF:Q2_K -npp 2500 -ntg 32 -npl 1,2 \
    -c 32768 -pps -kvu

Always scatter+gather the pooled keys, and pick gather from context
constants only: n_ubatch bounds every ubatch, top_k + kpool - 1 bounds
n_sel. Every graph of a context then shares one shape, which the
reserve covers, and the dense path measured faster than the gather path
at 2.5k and 16k context.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

* llama : drop the unused k-pool cache_safe graph API

The k-pool graphs no longer branch on cache_safe, so nothing reads
get_kpool_cache_safe() or the conditional new_pool_rep any more: both
models always pass the scatter target, which set_input_kpool now
requires instead of merely preferring.

Also drop the cache_safe copy in kpool_build_sizes(), a sizes-only
helper. The layout and state flag itself stays, it still decides which
pools a layout with shared cells must re-pool.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

* tests : add a shared-seq graph reserve regression test

Decode a prompt into seq 0, share its cells with seq 1 via
llama_memory_seq_cp (what llama-batched-bench does for -pps), then keep
decoding both sequences. For the k-pool models sharing clears
cache_safe, which changes the graph topology while the pools keep
growing, so a scheduler that re-reserves with the current state
instead of the worst-case one aborts under GGML_SCHED_DEBUG_REALLOC=1.
The test registration sets that flag, and the test aborts on both
k-pool models before 2220411ec1.

kimi-linear and minimax-01 are skipped: they reserve the final pp graph
with n_seqs = 1 (see [TAG_RESERVE_DIAG_DECAY] in llama-context.cpp), so
every multi-seq graph has a different layout and re-reserves by design.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

* cont : add TODOs

* cont : fix comment

* cuda: match the moe weighted reduction on empty ubatches

ggml_cuda_match_moe_weighted_reduction rejected tensors with zero
rows. A ubatch without outputs shrinks the last layer to zero rows
through inp_out_ids, so graph_optimize dropped its alloc dep there and
the scheduler graph lost one node compared to the reserved one. The
scheduler then re-reserved at the size of that ubatch, and the next
ubatch with the same node count but larger tensors aborted under
GGML_SCHED_DEBUG_REALLOC=1.

The compute loop already skips empty nodes before trying any fusion,
so the guard only made the alloc deps depend on the row count.

* tests: build the rollback test only where internal symbols link

The shared-seq case calls llm_arch_from_string, which libllama does
not export through LLAMA_API, so linking test-recurrent-state-rollback
fails on Windows with shared libraries. Its build now sits in the
NOT WIN32 OR NOT BUILD_SHARED_LIBS block, next to test-llama-archs and
the test registration it already lives under.

* tests: skip archs by name in the shared-seq reserve test

The skip of kimi-linear and minimax-01 went through llm_arch_from_string,
which libllama does not export through LLAMA_API, so the test could not
link on Windows with shared libraries. It now compares the
general.architecture string directly, and the test builds on every
platform again.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-05 11:36:25 +03:00
Xuan-Son Nguyen 0bb496dbd3 llama: support both embd + raw tokens in batch (#29622)
* llama: support both embd + raw tokens in batch

* add to test-llama-archs

* also check case llm_arch_supports_mixed_batch = false

* constant graph topology

* have dedicated input for mixed case

* rm set_tensor_backend

* is_embd --> type

* consolidate m-rope pos handling into one place

* nits
2026-10-05 01:35:49 +02:00
PascalandRuben Ortlam 889edf43dd qwen4exp : halve the indexer score memory (#29825)
* qwen4exp : halve the indexer score memory

The indexer scored all heads in one product and rectified a copy of it,
so two [n_pool, n_idx_h, n_tokens] f32 tensors were live at once, the
largest buffers of the graph at long context. Each head now gets its
own product, rectified and summed in place into one [n_pool, n_tokens]
score.

* qwen4exp: let the allocator reuse the indexer score buffers

Address review from CISC: use plain ggml_add and ggml_relu in the
indexer head loop. The graph allocator already runs them in place when
their source has no other consumer, so the _inplace variants are not
needed. The compute buffer and the speed are unchanged.

* cuda: support 4 heads in the lightning indexer

Dispatch 4 heads to the vector kernel, too few for a wmma tile, and
accept them in supports_op. test-backend-ops covers 4 heads.

* metal: take the lightning indexer head count as a function constant

The kernel reads the head count from a function constant and zero fills
the last head tile, so any head count runs and 64 heads is unchanged.

* qwen4exp: compute the indexer score with the lightning indexer

Address review from am17an: the unweighted sum of the rectified head
scores scaled by 1/sqrt(head_dim) is the lightning indexer with every
head weight set to that scale, so the indexer calls
ggml_lightning_indexer on the pooled keys with an f16 pool mask. The
keys are read once for all heads and no per head score is
materialized.

* vulkan: tile the lightning indexer over keys and tokens

A workgroup scores 64 keys against 8 tokens: the keys are staged once
in shared memory, the queries one head at a time, and each invocation
owns one key for two tokens, so no dot product needs a cross invocation
reduction. The subgroup variant and the flat dispatch are gone, the grid
is keys x tokens x streams.

* vectorize vulkan loads and use fp16 dot product

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-10-03 07:19:00 +02:00
Xuan-Son NguyenandSigbjørn Skjæret 99b95488ca model: add support for clef decision model (text-only) (#29831)
* init support for clef (text only)

* more static graph

* clean up

* nits

* nits 2

* Update gguf-py/gguf/constants.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-03 02:50:48 +02:00
Xuan-Son Nguyen a4cb4c61fd llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) (#29818)
* init conversion

* convert: ok

* model loaded

* add server code

* improve conversion script

* support shared prompt prefix

* add docs, imorove UX a bit

* add vision support

* add openjev tiny model for testing

* add dev docs

* support lev & kev

* clean up

* fix lev noul

* fix py lint

* nits docs

* clarify about not supporting date_facts
2026-10-02 11:56:04 +02:00
Georgi Gerganov 4e2713c162 qwen4exp : optimize mask constructions (#29824)
* qwen4exp : optimize mask constructions

* cont : apply the same change for GLM5-next
2026-10-02 11:08:50 +03:00
Aman Gupta 159c651f57 qwen4exp: fix tests (#29819) 2026-10-02 09:12:25 +08:00
Aman GuptaandGeorgi Gerganov c061df1983 Qwen4Exp: add MTP (#29761)
* Qwen4Exp: add MTP

* remove has_state member, check via ctx_bufs being non-empty

* consistent naming + less verbose comments

* cont : clean-up recurrent memory

* cont : clean-up comments

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-01 14:13:27 +03:00
Aman Gupta 66e0c17ee1 llama: fix qwen4exp (#29751)
* llama: fix qwen4exp

* qwen4exp: keep kq_mask input the same shape
2026-10-01 14:13:27 +03:00
Kevin Hopper 10f340d1a2 model : re-enable -sm tensor for qwen4exp (#28569)
#27941 disabled -sm tensor for qwen4exp because test-llama-archs asserted on the
Meta device once the fixture carried a PLE layer:
GGML_ASSERT(ggml_backend_buffer_is_meta(tensor->buffer)) at ggml-backend-meta.cpp:476.

With host-resident embeddings the PLE gather is a CPU node and hc_init (the REPEAT
that fans the embedding out to the hc streams) was first reached through layer 0's
PLE path, after that gather. ggml_backend_sched_split_graph pass 2 expands a device
assignment upwards only until it meets a CPU node, so the REPEAT stayed on the CPU
and the later reshape of hc_init inside the meta split viewed a host-resident node.

Expanding hc_init right after it is built puts the REPEAT directly before the first
device node, where pass 2 assigns it; the embedding reshape stays in the CPU split
and is copied in as a split input, as in deepseek4.
2026-10-01 08:16:49 +03:00
Pascal 05af0d2b13 glm5-next: give dead indexer slots unique scatter rows (#29745)
The sparse indexer mask is built with a set_rows scatter. Padded pools,
absent sequences and missing tail cells all pointed to the same n_kv
sentinel row, and invisible pools picked by top_k to fill the selection
overlap the tail cells of the token, so several CPU threads wrote the
same element (ThreadSanitizer data race in the sanitize CI).

Allocate the slot mask for both selection paths and route every dead
slot to its own dump row n_kv + slot. Live slots address disjoint cells,
so the scatter indices of a token are unique.
2026-09-30 17:10:48 +02:00
Georgi GerganovandSigbjørn Skjæret 22bdcc4cdd mimo : support dflash (convert + feature extraction) (#29650)
* convert : update to support dflash

* cont : fix

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-09-30 17:04:35 +03:00
185103dcf5 llama: llama_prefetch_rows (#29599)
* llama: llama_prefetch_rows

* llama: support row prefetch on Windows

Apply the Windows port contributed by @praneshgo unchanged.

Source: https://github.com/ggml-org/llama.cpp/pull/29599#issuecomment-5887721014

* avoid exposing llama-mmap in model code, route via llama-impl

* add windows check, only prefetch in lazy mode

* cont : clean-up

* cont : fix build

* cont : clarify padding token for gemma4

---------

Co-authored-by: Pranesh Gonegandla <pranesh.iitp@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-09-30 20:27:18 +08:00
649dcb1036 add GLM-5.3-Flash (GLM5-Next) support (#27773)
* Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx

* Add initial MTP support

* Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements.

* Review driven changes, remove env vars, protect tensors

* Strip MTP for initial PR

* Clean up after mtp strip

* Clean up after mtp strip

* Update speculative.cpp

* Update llama-context.h

* Clean up after mtp strip

* Fix tokenizer ignore merges

* Improve quantization protection selection

* Refactor mhc helpers, graph base

* Lint Fixes

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Skip glm5-next in model saver, fix CRLF

* Skip glm5-next in sweep

* Remove T4 fallback

* Review cleanup

* Review suggestions

* Defer separate MTP gguf handling to MTP PR, drop filter

* Repad n_head_kv

* kpool init apply

* Order by descending score

* Drop guard

* read kpool from hparams, clarify kpool cache flags, remove kpool_build_state(nullptr)

* Add glm5-next support to model saver and add arch test fixture

* Review cleanup

* Kpool pooled caching clarify

* Add multi stream support

* Finish Rebase

* Sparse FA fir DSA prefill

* Const

* Update llama-model.cpp to fix rebase error

* gguf-py : merge tensor map entries for HC tensors

* model : use build_gdn_l2_norm in GLM5_NEXT implementation

* chore : remove trailing whitespace

* model : use new OP precision setting API in GLM5_NEXT implementation

* mtmd : use ggml_swiglu_clamp in GLM5V and apply the image token limit

The two clamps around swiglu_split are what ggml_swiglu_clamp already does,
so the clamp bounds collapse back to one value. GLM5V also never called
set_limit_image_tokens(), so --image-max-tokens had no effect.

Assisted-by: Claude Opus 5
(cherry picked from commit 46d18e12d422be4cc04a70e4a9a9e0168bb3d5b7)

* llama : keep the GLM5-Next k-pool layout across ubatches

The layout was rebuilt from a full cell scan on every ubatch. Pools are fixed
by the positions relative to the sequence's first one, so the layout now lives
on the memory and a ubatch only appends to it.

A sequence edit no longer stales every pooled key either, only the ones at or
after the edited position, which makes a tail seq_rm free. The pooling subgraph
is built unconditionally so the graph shape no longer changes every kpool
tokens, and the pool axis is folded into rows before soft_max, which otherwise
exceeds the CUDA gridDim.y limit past n_kv 262144.

Assisted-by: Claude Opus 5
(cherry picked from commit 5d1c40b93e17fddbf73b785efe43e0d02ccb3977)

* model : write the GLM5-Next recurrent rollback checkpoints

The conv state and the delta net state were only written to the live row, so a
rollback restored whatever the checkpoint rows happened to hold. Take the same
route as kimi-k3: build_recurrent_attn for the state, and write all K_rs conv
groups. That also drops a state view that assumed contiguous rows.

Enroll the arch in test-recurrent-state-rollback, which catches this under its
garbage-filled cache pass.

Assisted-by: Claude Opus 5
(cherry picked from commit 5ace37e86d5d448e83ef5dde5632c748185b18cd)

* llama: fix PR #27773 test-save-load-state restore failure

Clear the attention and indexer cache data after a failed hybrid state restore so restored NaNs cannot affect a later sequence.

Assisted-by: Codex

* llama: fix PR #27773 gpu-rocm graph reallocation

Reserve the full GLM5-Next pool capacity and dirty pool count. The gpu-rocm Test step aborts when n_new grows while the graph node count stays fixed; CUDA, Vulkan, Metal, and WebGPU checks report the same error.

Assisted-by: Codex

* llama : fix GLM5-Next k-pool layout staleness after edits and shared teardown

Two defects in the cross-ubatch k-pool layout added by the k-pool commit:

1. Wrong results. An edited sequence only rebuilt its pool layout when its cell
   count changed, so if the first ubatch after an edit added back exactly as many
   cells as were removed, the stale position-to-cell list survived. With a unified
   cache and more than one sequence, where another sequence takes the freed cells,
   the reused layout points at the wrong cells (CPU: large logit drift, CUDA: NaN).
   Rebuild whenever the sequence is stale, not only on a size mismatch.

2. Slowdown. "shared" mode was assumed to end only with an edit that forces a
   rebuild, but sharing also ends when the other sequence is removed. The survivor
   kept shared = true, pinning cache_safe off and re-pooling every pool on every
   ubatch (server trigger: n>1 completions with -kvu, via the seq_cp in
   copy_state_to). In seq_rm, if the layout has shared cells, stale every sequence
   so one rebuild re-derives sharing and cache_safe returns to 1.

Assisted-by: Claude Opus 5

* llama : fix build_attn_mha stream stride for non-contiguous q

build_attn_mha split the batch into streams with a stream stride of
q->nb[3]/n_stream. That only equals one stream's span, (ne[2]/n_stream)*nb[2],
when q is contiguous. GLM5-Next is nope-only, so it does not concat a rope part
and passes the permuted q_absorbed straight in, where nb[3] != ne[2]*nb[2]; the
stride was then n_head times too large and every stream s >= 1 read another
head's queries. Split-KV (-np N without --kv-unified) multi-stream prefill was
wrong for every stream past the first. Unified KV and decode were unaffected
(n_stream == 1, and decode takes the gather path). Other MLA models concat rope
so q is contiguous and the computed value is unchanged for them.

Compute the stride from the token dimension, which is identical for a
contiguous q.

Assisted-by: Claude Opus 5

* llama : re-derive GLM5-Next k-pool sharing on state_read/state_drop

The shared-cell teardown added to seq_rm (stale every sequence when the layout
has shared cells, so a survivor does not keep shared = true and pin cache_safe
off) was missing from the other paths that can free shared cells: state_read
and state_drop staled only the one sequence. Apply the same re-derivation there
and correct the comment that claimed sharing ends only via an edit or seq_rm.

Assisted-by: Claude Opus 5

* quant : drop duplicate GLM5-Next hc_ filter

The hc_ name filter was listed twice in the GLM5_NEXT protection block.

Assisted-by: Claude Opus 5

* glm5-next: scope K-pool cache access to indexed operations

* glm5-next: keep K-pool access in hybrid index memory

* glm5-next: keep mHC graph builders model-local

* glm5-next: mark only touched pools per ubatch

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
2026-09-30 14:20:32 +08:00
bosh 48de2a1bcb model : support classifier_pooling for rerankers (#29627)
* model : support classifier_pooling for ModernBERT rerankers

Assisted-by: Claude Opus 5.5

* model : read classifier pooling type in load_hparams

Write classifier.pooling_type from _try_set_pooling_type whenever the
config has classifier_pooling, and read it in
llama_model_base::load_hparams. ModernBERT falls back to mean when it
is unspecified.

Assisted-by: Claude Opus 5.5

* conversion : only accept cls and mean for classifier_pooling

Assisted-by: Claude Opus 5.5

* model : rename classifier_pooling_type to pooling_type_cls

Assisted-by: Claude Opus 5.5
2026-09-29 22:33:05 +02:00
Pascal 57b557cb95 models: pad on the left with ggml_pad_ext (#29567)
* models: pad on the left with ggml_pad_ext

The Parakeet, LFM2-Audio, Granite Speech and Gemma 4 audio encoders
build a left padding as a right pad followed by a roll, and DFlash2
concatenates a zero filled block in front of the previous tokens.
ggml_pad_ext does both in one node now that every backend supports a
left padding. The Gemma 4 audio embeddings are bit identical.

* models: skip the DFlash2 taps that only read padding

A tap at or past block_size shifts every row out of the block, so its
term is zero. The loop runs min(kernel_size, block_size) taps.
2026-09-28 20:56:15 +02:00
Sihan YuandGeorgi Gerganov ed7ac35e1e context : do not re-reserve the scheduler when toggling causal_attn (#28751)
* context : do not re-reserve the scheduler when toggling causal_attn

`llama_context::set_causal_attn()` marks the scheduler to do a full re-reserve on every change of the flag. For vision inputs, this flag is flipped twice around each non-causal image chunk for Gemma models, resulting in two expensive `sched_reserve()` passes per image. This is especially slow for multi-image or video inputs.

The cost of a re-reserve scales with context and ubatch configurations, so larger settings pay more per image (see table below).

The re-reserve is unnecessary in this case because `causal_attn` only changes the values written to KQ mask, not tensor shapes or any other buffer sizes.

Note: `causal_attn` is a graph reuse key (`llm_graph_params` via `cparams`), so a new graph is built regardless of `sched_need_reserve`, so this doesn't change the graph rebuilding behaviour.

llama-server with gemma-4-26B-A4B Q4_0 + BF16 mmproj, 130-token images,
cache_prompt=false, prompt_ms median of 3 (before -> after):

| images | config | H200 before -> after | RTX 4090 before -> after |
|-|-|-|-|
| 1 | `-c 8192 -ub 512` | 134 -> 105 ms (1.27×) | 201 -> 119 ms (1.69×) |
| 24 | `-c 8192 -ub 512` | 2278 -> 1562 ms (1.46×) | 3559 -> 1748 ms (2.04×) |
| 24 | `-c 32768 -ub 2048` | 5379 -> 1584 ms (3.40×) | 13377 -> 1759 ms (7.61×) |

Generated output remains identical before and after.

* qwen4exp : make the indexer bias shape independent of causal_attn

The block/cell bias path was selected on cparams.causal_attn, so the
causal and non-causal graphs differed in tensor shapes and ops. With the
re-reserve removed (previous commit), a runtime flip resulted in
reallocating the compute buffers, which would fail under
GGML_SCHED_NO_REALLOC.

This commit selects the block path from the mask shape only, independent
of causal_attn. causal_attn is instead passed to set_input_qsa.
causal_attn is fixed per graph as it's part of the reuse key. Causal
values are unchanged. Non-causal values now follow the reference rule,
where every visible block competes on score and only unpooled cells are
always selected.

* context : state the causal_attn shape rule in the comment

* cont : add TODOs

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-09-28 11:58:51 +03:00
Georgi Gerganov 85ca3b52c3 hrm : fix layer placement of z_l_init weight (#29512) 2026-09-27 10:16:23 +03:00
Aman Gupta f805c57a2d llama : fix tensor split for fused qkv with uneven K/V head sizes (#29294)
* llama : fix tensor split for fused qkv with uneven K/V head sizes

Assisted-by: Qwen3.8-27B

* fix v granularity

* convert: fix mtp conversion

* convert: add support for mtp flags

* fix loader
2026-09-25 11:15:33 +08:00
Tobyandaetherbird f830688e91 model : add Ling 3.0 VL support (#29151)
* model : fold Ling 3.0 VL into the BailingMoeV3 architecture

Assisted-by: Scout

* model : keep shared NORM rope list intact when gating bailingmoe3 on mrope sections

---------

Co-authored-by: aetherbird <aetherbird@users.noreply.github.com>
2026-09-24 08:57:31 +02:00
Hrishith Thadicherla 633733d0ae model : support Gemma4 DSpark draft backbone (#29226)
* dspark: add Gemma 4 draft support

Add GGUF conversion and runtime support for full-attention and SWA Gemma 4
DSpark drafts, including tied output weights and boolean backbone metadata.

Assisted-by: Codex

* dflash: infer Gemma draft features from metadata
2026-09-23 13:34:09 +03:00
wendadawen 828fdf282e spec : support DFlash for HunyuanOCR (#28890)
* model : add DFlash layer-input taps for HunyuanVL

DFlash speculative decoding needs the target graph to expose the residual
stream entering each layer (res->t_layer_inp[il]) - the draft model reads
those tensors to build its cross-context. Qwen3 and the other DFlash-capable
targets register them, but the Hunyuan graphs do not, so serving a DFlash
draft against a HunyuanOCR target aborts during the first graph build:

  GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null")

Register the tensor at the top of the layer loop, mirroring qwen3. The
layer input is the residual stream entering layer il, i.e. the output of
layer il-1, which is what the draft's target_layers metadata refers to
(the converter writes target_layer_ids+1). hunyuan-dense.cpp reuses this
graph, so it is covered as well; hunyuan-moe has a separate graph and is
untouched.

The vector is only read when a speculative implementation enables those
layer ids, so there is no behaviour change without a draft model.

Tested with tencent/HunyuanOCR 1.5 and its DFlash draft: image requests now
run, draft acceptance is ~0.5 and the OCR output is byte-identical to the
non-speculative run.

Co-authored-by: wendadawen <wendadawen@qq.com>

* convert : fix DFlash draft conversion against HunYuan targets

Converting a DFlash draft with a HunYuan target failed in two ways.

1. DFlashModel.set_vocab() reuses the target class' vocab handling by
   calling it unbound with the draft instance, but HunYuanModel.set_vocab()
   called self._fix_special_tokens(), a method that only exists on
   HunYuanModel, so the conversion always aborted with

     AttributeError: 'DFlashModel' object has no attribute '_fix_special_tokens'

   Make the vocab helpers module-level functions taking the model
   explicitly, so they do not depend on the instance being a HunYuanModel.
   They have no other callers, so the two id lookups are folded into
   _fix_special_tokens().

2. The delegated call runs with self.dir_model pointed at the target but
   keeps the draft's self.hparams, so config lookups inside the target's
   vocab code (the pad_token_id < 0 guard, eod_token_id) read the draft's
   config instead of the target's. That aborts on targets with
   pad_token_id = -1 (e.g. the HunyuanOCR v1.0 checkpoint) and otherwise
   writes special token ids that disagree with the target.

   Add _vocab_hparams(): it returns the target's config (with text_config
   merged to the root, as TextModel does) when the model is a draft
   converted with --target-model-dir, and the model's own hparams
   otherwise, so a normal conversion is unaffected.

Tested: converting tencent/HunyuanOCR/dflash succeeds with both the 1.5 and
the v1.0 target; converting the base model without --target-model-dir
produces a byte-identical GGUF to before.

Co-authored-by: wendadawen <wendadawen@qq.com>

* convert : fix DFlash draft vocab against HunYuan targets

Switch hparams to the target config for the duration of the borrowed
set_vocab(), matching the existing dir_model swap, instead of teaching
HunYuanModel::set_vocab about draft models.

* convert : fix HunYuan special token ids for DFlash drafts

* convert : use load_hparams for HunYuan special token ids
2026-09-22 15:04:52 +02:00
Georgi Gerganov 161755f29e test-llama-archs : make tensor data stdev configurable and improve help (#29133)
* test-llama-archs : make tensor data stdev configurable

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* test-llama-archs : expand usage and add examples

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* test-llama-archs : fail on unknown args and log usage

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* test-llama-archs : add test run summary

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* test-llama-archs : initialize Mamba ssm_a negative

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* test-recurrent-state-rollback : report NMSE for logits mismatches

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* test-recurrent-state-rollback : use NMSE for rollback logits checks

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* tests : disable invalid test

* cont : adjust nmse_eps

* tests : zero DSA indexer score projection in synthetic fixtures

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* cont : add support for `--arch` regex

* cont : alternative top-k stability

* cont : indentation

* cont : fix top-k value

* cont : consistent logs
2026-09-21 13:57:36 +03:00
Aman Gupta 3cf03257f2 CUDA: enable sparse fa for qwen4 (#28770) 2026-09-20 16:08:11 +08:00
Andrei 4260903678 fix(mamba) : make time-step projection input contiguous (#28832)
* mamba : make time-step projection input contiguous

Assisted-by: ChatGPT

* mamba : skip contiguous copy after normalization

Assisted-by: ChatGPT
2026-09-20 07:58:14 +03:00
Pascal 542348a35c Model-Saver: Write the SWA pattern, 15 more architectures roundtrip (#29042)
* llama: read the SWA pattern as a period or a per-layer array

Add llama_model_base::load_swa_pattern(), which reads
sliding_window_pattern either as one flag per layer or as a period
expanded by set_swa_pattern(), and use it in every loader that reads
the key as a period.

These loaders silently ignored an array and applied their default
period, although the converters of olmo2, gemma3n and exaone4 write
arrays. The published GGUFs match the defaults, so their outputs do
not change. The loaders that already accepted both forms lose their
duplicated scalar-then-array block, and use their declared default
period when the key is absent.

* model-saver: write the SWA pattern and the MLA SWA geometry

Write sliding_window_pattern as one flag per layer, nextn layers
included, for every model using SWA. The array is never collapsed to
a scalar, since the loaders read a scalar as a period.

Also write the MLA key/value lengths and KV LoRA rank of the SWA
layers, required by dots3note.

This enables the saver for plamo3, gemma3, cohere2, cohere2moe,
olmo2, exaone-moe, afmoe, mimo2, spark2_5, muse-glimmer, mellum,
laguna, granite_swa, dots3note and maple, all passing the bit-exact
roundtrip of test-llama-archs.
2026-09-18 15:20:03 +02:00
Kartik Gulia 7076180486 model : extend Nemotron MTP support (#29018)
* first fix

* removed unnecessary declarations
2026-09-17 13:52:36 +02:00
Kartik Gulia 7ceed8737f models : allow Nemotron-H models to only define layer_norm_epsilon (#28989)
* allows nemotron models to get by with just defining layer_norm_epsilon

* made changes to load_arch_hparams instead
2026-09-16 15:24:47 +02:00
GeorgeandSigbjørn Skjæret 7d6f5d02bb model : add support for HrmTextForCausalLM (DFM Mimir 1B) (#27625)
* model : add support for HrmTextForCausalLM (DFM Mimir 1B)

HRM-Text runs two transformer stacks (low, high) in an alternating cycle over the same token stream. The low-cycle state z_l starts from a learned [n_embd] tensor and is broadcast over positions.

- conversion: new writer for the fused gqkv projection (order gate,q,k,v) remapped to llama.cpp q/k/v plus a separate sigmoid gate tensor
- loader: block_count = lps * h_cycles * (l_cycles + 1) cache slots aliasing 2*lps physical blocks via struct copies
- graph: looped build with sigmoid-gated attention, SwiGLU FFN and parameterless RMS norms; learned embedding_scale applied in build_inp_embd
- saver: pointer-deduplicated layer loop (looped archs alias tensors)
- tests: hrm_text fixture (lps 1, h 2, l 3) in test-llama-archs

Limitations:
causal attention only - the upstream prefix-LM mode is not implemented (the prefix_lm GGUF key round-trips unused).
The KV cache holds one entry per pass: 128 layers for Mimir 1B, i.e. 4x a same-width 32-layer model - about 3072 MiB at ctx 4096 in F16 (halves with q8_0 KV + FA).
Every token runs all 128 block passes, so decode cost is roughly 4x a dense model of equal width (2.65 t/s BF16, 8-thread desktop CPU).

Verified against the HF reference: identical argmax at 334/334 positions across 20 prompts (BF16 GGUF vs FP32 golden).
q8_0 requant: 95.8% top-1, all remaining misses inside the HF top-5 (accumulated error over 128 sequential blocks).

AI usage disclosure: YES
Used GLM-5.3 for the majority of code AI-generated under my direction, all gates verified locally.
All in all I could say that I have written less than 20% of the code and most of the heavy lifting has been done by the model. As such, this should be considered experimental.

* Update conversion/hrm_text.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-arch.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* convert : add gguf_writer methods for hrm_text metadata

replace raw add_uint32/add_bool calls with dedicated GGUFWriter methods, following the add_embedding_scale pattern

Assisted-by: GLM-5.3

* convert : map regular hrm_text tensors via tensor_mapping

delegate unfused checkpoints to the base tensor mapping; training-style attn. names are renamed to self_attn. so the patterns match

Assisted-by: GLM-5.3

* model : format hrm-text build_* calls as in other models

one argument group per line, matching sibling model files

Assisted-by: GLM-5.3

* llama : move hrm z_l_init table entries out of the nemotron group

place the name and tensor-info entries with the other global input tensors

Assisted-by: GLM-5.3

* convert : slim down hrm_text comments

Assisted-by: GLM-5.3

* convert : build hrm_text block tensor names from the {bid} template

The tensor map holds concrete per-block names, so format the template
with the computed layer index before handing it to super().

* llama : name hrm metadata keys in their own hrm. namespace

The four keys are arch-independent, unlike the arch-substituted
Keys.LLM entries, so group them under Keys.HRM (like Keys.Split) and
rename the llm_kv entries to LLM_KV_HRM_*. Only our own GGUFs carry
the old hrm_text.* keys; they are regenerated.

* Update src/llama-model-saver.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* llama : keep hrm metadata keys arch-substituted

Per review: the GGUF keys stay "{arch}.h_cycles" style, so the Python
members drop the LLM_KV_HRM_ prefix and keep arch templates; C++ keeps
the LLM_KV_HRM_* enums. GGUF output is unchanged - existing files and
HF uploads stay valid.

* Update gguf-py/gguf/constants.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-arch.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-arch.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* convert : rename hrm writer methods to add_hrm_*

Generic names like add_h_cycles/add_prefix_lm are too broad on the
shared GGUFWriter; prefix them with hrm_ like the metadata keys.

* model : fix meta-split lookup for archs with aliased cache slots

Cache tensors of archs that alias physical blocks across looped slots
(hrm_text, nanbeige with num_loops > 1) can reference block indices
without weight tensor names. Take the output projection from the layer
array instead of asserting; all other lookups are unchanged.

* model : replicate hrm_text tensors on meta devices instead of splitting

The aliased cache slots rotate split states differently from their
physical weights, so the meta-split execution invariants (set_rows
requires the cache state to match the token indices) cannot hold for
any device count. Replicate all hrm_text tensors on every meta device
instead; single-device and non-meta paths are unchanged.

Assisted-by: Claude Sonnet

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-09-16 15:18:45 +02:00
Aman Gupta 37b53fd454 qwen4exp: add hc ops (#28901) 2026-09-16 16:00:01 +08:00
Chris Peterson 9e71716247 models : move build_arch_graph() after graph() template specialization (#28934)
Move build_arch_graph()'s function definitions after the graph<true>
and graph<false> template specializations have been explicitly defined.
2026-09-15 12:33:26 +03:00
Aman Gupta 41abbfd599 qwen4exp: enable rms_norm + mul fusion (#28896)
* qwen4exp: enable rms_norm + mul fusion

* use TENSOR_ALLOW_RESHAPE
2026-09-14 22:16:30 +08:00
Sigbjørn Skjæret 97e4ca7358 models : fix incorrect uses of get_key_or_arr (#28868) 2026-09-14 14:05:37 +03:00
Sigbjørn Skjæret 1aca1f9fcd models : fix mimo2 swa pattern load (#28865) 2026-09-14 14:05:17 +03:00
Alex 3d10bcd197 llama: add Maple 20B-A1B ternary MoE architecture (CPU) (#27000)
* gguf-py: add Maple tensor constants

Add MODEL_ARCH.MAPLE, its "maple" name, and the tensor list for the
Maple 20B-A1B ternary MoE architecture: token embeddings, output,
attention with Q/K RMS norms, and per-expert FFN tensors.

* convert: add Maple HF->GGUF converter

Register MapleForCausalLM in the HF architecture map and add the
converter for the Maple 20B-A1B ternary MoE model: 24 layers, 256
experts with 8 active, sliding-window attention (SWA-512) interleaved
with global attention at a 3:1 ratio, partial rotary factor 0.5, and
per-expert weight stacking into merged 3D tensors.

* llama: add Maple architecture (20B-A1B ternary MoE)

Add the Maple 20B-A1B ternary MoE architecture: 24 layers, 256
experts with 8 active, sliding-window attention (SWA-512) interleaved
with global attention at a 3:1 ratio, and ternary TQ1_0/TQ2_0
quantization support.

- register LLM_ARCH_MAPLE between MAMBA2 and JAMBA
- implement llama_model_maple: Q/K RMS norms after projection (GEMMA4
  style), rope applied only on SWA layers (nope_on_global_attention),
  ISWA KV cache, and MoE FFN with swiglu gate clamp at +7 (DEEPSEEK4
  style)
- mark MAPLE as unsupported by the model saver (roundtrip skipped)

* tests: mark Maple as MoE-mandatory

Maple is always-MoE: the model throws when n_expert == 0, so the
test harness must only run the MoE config for LLM_ARCH_MAPLE.

* maple: apply review feedback (n_ff_exp_arr, get_arr, rope params)

- load_arch_hparams: use n_ff_exp_arr + n_ff_exp() accessor (upstream
  changed these from a scalar member during the rebase)
- sliding_window_pattern: get_arr, the pattern is mandatory for this arch
- partial_rotary_factor: read only from rope_parameters (base.py mirrors
  the top-level key automatically)
- document why TOKEN_EMBD/OUTPUT are forced to F16 (they are the two
  dense tensors in Maple, and the reference GGUFs ship them as F16)
- add @ModelBase.example("deepgrove/maple-preview")

* tests: add Maple to the SWA pattern array list

get_arr for maple.attention.sliding_window_pattern requires an array, but
the harness only emitted a per-layer array for the arches in its list, so
test-llama-archs -a maple failed to load the model.

Assisted-by: DeepSeek Harness

* maple: move swiglu_clamp_exp to the converter

The loader prefilled 7.0 and read the key optionally. The converter now
writes it and the loader reads it as required, because llama-graph.cpp
skips the clamp when the limit is 0 and an optional read would silently
run unclamped. The test harness provides the key for the same reason.

Also drops tensor_force_quant: base.py already forces FFN_GATE_INP to F32
and TOKEN_EMBD/OUTPUT to F16 for ternary file types.

Assisted-by: DeepSeek Harness

* convert: fix the LazyBase func signature in the Maple converter

ty flagged the stack() closure: it takes no argument, while LazyBase is
annotated with func: Callable[[Any], Any]. Pass the tensor list through
args instead of closing over it, the same way kimi_k3 does, so the
callable shape matches.

Assisted-by: DeepSeek Harness
2026-09-14 14:04:05 +03:00
Yaniss Amazouz e49d2c2760 models : guard the expert FFN size fallback in nemotron-h against a zero divisor (#28779)
The NextN/MTP tail loop derives the expert FFN size as n_ff/n_expert_used
when expert_feed_forward_length gives nothing for the layer. Both values come
from per-layer arrays that legitimately hold 0 on layers that are not MoE, so
a checkpoint whose predict layers hold 0 in both divides by zero and dies with
SIGFPE at load time, with no error message. Report the malformed metadata
instead.
2026-09-13 19:20:50 +02:00
Daniel Bevenius 3bcfeb700f cmake : add PCH and unity build to improve build times (#28091)
* scripts : add initial profiling script (wip)

* src : add precompile headers (PCH) for models.h

* common : add common.h as PCH

* ggml : add PCH for ggml-impl.h

* mtmd : use PCH for models.h

* scripts : add script to build with Server/Tools/Tests

* server : add PCH for common.h

* docs: add profiling progress notes (wip)

* ggml : add exclude for GCC + SVE on ARM

Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33393906061/job/99493756214?pr=28091

* ggml : attempt to fix use of std::hardware_destructive_inference_size

Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33396221677/job/99501265689?pr=28091

* squash! ggml : attempt to fix use of std::hardware_destructive_inference_size

Add a version check for GCC 12 to conditionally apply the `-Winterference-size`
pragma.

* editorconfig : exclude profiling reports dir

This directory will not be included in the merge later and this commit
can be ignore at that point. Just fixing to keep CI happy.

* ggml : skip PCH for gcc on non-x86 architectures

* tests : add PCH for peg-parser/tests.h

There are 7 peg-parser tests that can share one PCH instead of then each
parsing the full tests.h.

* common : add PCH for chat.h

* docs : update linux build profiling full results

Just updating after a number of PCH additions. These are not exact
figures and will vary a bit from run to run, but they give a general idea
of the performance impact of PCH.

* cmake : introduce unity build for models

This commit introduces a unity build for the models to improve
compilation time.

The improvements were roughly the following:
```console
+------------------------+-----+------------+------------+------------+
| Build                  | TUs | Frontend   | Backend    | Total      |
+------------------------+-----+------------+------------+------------+
| Full,    master        | 396 |   811.0 s  |   692.2 s  | 1,503.2 s  |
| Full,    with PCH      | 405 |   380.0 s  |   664.7 s  | 1,044.7 s  |
| Full,    with PCH + UB | 264 |   357.7 s  |   635.7 s  |   993.4 s  |
+------------------------+-----+------------+------------+------------+

TU   = Translation Unit.
Full = includes Server, Tools, and Tests.
PCH  = precompiled headers.
UB   = unity build for models.
```

* docs : update linux profiling table with unitiy build results

* docs : update mac profiling results to include unity build [no ci]

* docs: remove profiling reports

* scripts : merge build profile scripts into one script

I was lazy before and just copied the first script to enable Tests,
Server, and Tools. This now merges them into a single script.

* Revert "editorconfig : exclude profiling reports dir" [no ci]

This reverts commit 2922a12118a0730d2f7632bcba265b44a0856c59.

* src : rename ggml_view_2d_slice to gemma3n_view_2d_slice

This is to be consistent with the rename in gemma4.cpp which was
required to avoid a name clash.

* cmake : add build profile script for windows [no ci]

This commit adds a port of the scripts/build-profile.sh script to
windows powershell.

This was developed on Windows on ARM but should work on X64 as well but
needs to be tested there as well.
2026-09-11 13:01:29 +02:00
Georgi Gerganov a2878d30df metal : single-source fusion table + fusion debug rework (#28164)
* metal : rework fusion patterns into a single table

All fusable op patterns for the Metal backend are now declared once in a
fusion table (ggml-metal-fuse.cpp) and consumed by both the graph optimizer
(ggml_metal_fuse_max, packing) and the op encoders (ggml_metal_fuse_next,
compute). The two phases share the same pattern table plus ggml_can_fuse_subgraph_ext
for the structural checks, and differ only in the mode used for the pattern
check (STRUCTURAL at optimize time, since tensors are not allocated yet, and
FULL at compute time, including Metal buffer placement). This also protects the
snake activation (MUL + SIN + SQR + MUL + ADD) from being reordered during graph
optimization, which was previously unprotected.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : fix absolute output indices in fusion patterns

ggml_can_fuse_subgraph_ext expects the outputs array to contain absolute graph
node indices (it indexes cgraph->nodes[outputs[i]]), but the fusion table query
was passing a relative index (n_ops - 1). As a result the last node of every
pattern was not recognized as an output and was subjected to the elidable
use-count check, which failed for essentially all fusions. This silently
disabled the norm/MUL fusion and caused a ~5% token-generation regression.

Pass the absolute graph index of the last node instead.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : fuse gated_delta_net with cache cpy

Add GGML_METAL_FUSE_GDN_CACHE to the fusion table: when the gated_delta_net
kernel is followed by a cpy that scatters its recurrent state snapshots into
the KV cache, the kernel writes the snapshots straight into the cache buffer
and the trailing cpy is elided.

The gdn output has other consumers (the attn scores view), so unlike the
elision-chain patterns this is not a simple chain: a 'raw' flag on the fusion
pattern skips the generic chain/shape and ggml_can_fuse_subgraph_ext checks,
making the pattern-specific check callback the sole validator. Packing
(ggml_metal_fuse_max) now matches on the same view-transparent node sequence
that the compute phase uses, so the gdn + cache cpy group is packed along with
any intermediate views and stays adjacent through the reorder.

The fused cpy is a view consumer of the gdn (it writes the cache directly),
so its mem-range is skipped in the encoder; the skip is restricted to CPY
nodes consuming the previous fused node through a view so other fusions are
unaffected.

Add test_gated_delta_net_cache_fusion and register 5 cases.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : drop is_view_consumer mem-range skip

The is_view_consumer skip was carried over from the upstream gated_delta_net
cache-fusion draft, but it is not needed: keeping the elided cpy's mem-range in
the concurrency tracker only ever adds a (conservative) memory barrier at the
fusion point. It can never remove a barrier, so it cannot introduce a race. The
worst case is one spurious barrier per gdn+cache-cpy fusion, which is within
run-to-run noise on Qwen3.5-0.8B Q8_0.

Dropping the check keeps the mem-range loop uniform for all fused groups.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : rename gated_delta_net fused state output args

Rename the fused cache-write kernel argument to match the rest of the kargs:
state_out_stride -> nb_out (and widen it to uint64_t), and the local buffer id
bid_state_out -> bid_out.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : rename raw fusion flag to unsafe

raw did not convey that the flag opts a fusion pattern out of the generic
elision-chain safety net (ggml_can_fuse_subgraph_ext + chain/shape checks).
rename it to 'unsafe' to make explicit that the pattern's check callback is the
sole validator and must re-establish the safety guarantees itself.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : tidy fusion pattern checks and table

- const-correct ggml_metal_fuse_outputs buffer
- annotate unused check-callback parameters
- drop a redundant size_t cast
- align the ops/table initializers and add blank-line separation

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : add generic fusion stats via ad-hoc proc-address API

Add a device-owned fusion context that lets a test tool count how many
times each fusion pattern fires and toggle fusion. It is exposed through
the ad-hoc ggml_backend_reg_get_proc_address mechanism with generic names
so the testing tool is backend-agnostic:

- ggml_backend_fusion_stats_init: start collecting fusion stats; when a
  context is created afterwards it registers the labels/counters and
  encodes single-threaded (n_cb == 0) so the counters are race-free
- ggml_backend_fusion_stats_reset / _get_stats / _set_enabled

The context lives on the metal device (not on the last backend context),
so counters accumulate across contexts and reads are always consistent.
The enable/disable toggle is initialized from GGML_METAL_FUSION_DISABLE
and can be overridden by the test through set_enabled. Labels are
synthesized from the fuse table via ggml_metal_fuse_label (e.g.
"GATED_DELTA_NET+CPY").

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : add fusion count regression test with per-backend baseline

test-fusion runs every dummy model generated by test-llama-archs on a
single backend (single-threaded encoding, n_cb == 0) with fusion enabled
and disabled, and for each mode (prefill / decode) reports the per-fusion
counters and the NMSE between the fused and unfused logits, plus the NMSE
against a CPU reference.

A fusion pattern that silently stops matching (or fires when it should
not) is caught as a regression by comparing the counters against a
committed per-backend TSV baseline:

- --record writes the golden baseline, --check (default) validates it
- the unfused run doubles as a control: its counters must be all-zero
- NMSE is skipped when it is NaN or the arch is already broken on the
  device (e.g. plamo2 on Metal), so the count check is the hard gate
- baseline counts depend only on graph structure, not weights (verified
  stable across weight seeds)
- the fusion stats API is resolved through the ad-hoc get_proc_address
  mechanism with generic names; a backend that does not export it makes
  the test fail with an error

The committed MTL0.tsv baseline covers 110 dummy archs (298 rows).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : rename fusion api helpers to match stats_init signature

Align the test with the ad-hoc fusion stats API: fusion_stats_init no
longer takes an enable bool (stats are turned on by calling it), so the
proc-address wrappers and typedefs are renamed to the api_* convention.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : rename backend to device in fusion test CLI

The fusion test operates on a compute device (e.g. MTL0), not a backend,
so rename the --backend argument to --device and the backend_name
variable to device_name. Keep "backend" where it refers to the ggml
backend interface (the ad-hoc proc-address mechanism).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : add --model and --help to fusion test

--model FILE runs the fusion regression test over a single model file
instead of enumerating a --models DIR. --models and --model are mutually
exclusive. Also add a --help/-h option that prints the usage.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : use backend base name for fusion baseline output

The fusion test is invoked with a specific device name (e.g. MTL0), but
its output - the recorded baseline and the header it writes - should be
named after the backend base name (e.g. MTL, via ggml_backend_reg_name),
since the counters depend on the backend, not on the specific device
index. Rename the committed baseline to MTL.tsv.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : run fusion test from ci instead of ctest

The fusion test needs Metal and generates a lot of dummy models, so it
does not belong in the generic ctest suite. Move it to ci/run.sh as
gg_run_test_fusion, gated on GG_BUILD_METAL like
gg_run_test_llama_archs_tensor_split: it generates the dummy models with
test-llama-archs -o and then validates the fusion counts against the
committed baseline. test-fusion.cpp is still built (llama_build) but no
longer registered as a ctest.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : align fusion baseline TSV columns

Pad the TSV fields to fixed widths so the columns line up regardless of
the variable arch and fusion-label lengths, and trim each field on parse
so the padded file is still accepted. Regenerate the committed MTL.tsv
baseline in the padded format (data unchanged, verified identical modulo
padding).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : widen label column and align fusion TSV header

Give the label column more room (28 chars) and fix the column header
widths so they match the data rows (moe/mode/label), keeping the header
aligned with the values. Regenerate the MTL.tsv baseline in the new
format (data unchanged).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : switch fusion baseline from TSV to CSV

Use comma-separated values like the rest of the project, keeping the
padded, aligned columns. Split on ',' and trim on parse. Rename the
committed baseline to MTL.csv (data unchanged, verified identical modulo
padding/separator). Update the ci/run.sh check path accordingly.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* cont : rebase + update MTL stats

* tests : avoid graph reallocations for some archs

* metal : tidy fusion debugging context and op init

- simplify the shared fusion debugging context comments
- shorten the ggml_metal_fusion struct comment
- align the ggml_metal_fuse struct fields and comments
- move the fusion parameter of ggml_metal_op_init right after dev

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : dedup fusion baseline into any mode

prefill and decode always produce the same per-graph fusion count, so
store a single row per label with mode = "any" and the per-graph count
instead of two rows. this halves the baseline size and keeps the check
stable.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* ci : move fusion model generation to a separate step

the dummy models generated by test-llama-archs are reused by other tests,
so generate them once in their own step instead of inside test_fusion.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : bump nmse thold

* models : fix plamo2 graph

* tests : remove "skip" logic from test-fusion

* tests : set qwen3tts dummy vocab to codec head size

the dummy qwen3tts model used a vocab of 4096 while the codec head is
3072, so the graph padded the output with -inf which made the NMSE in
test-fusion produce NaN. use the exact codec head size instead so the
padding is not generated at all.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : regen fusion baseline

reflect the plamo2 graph fix, which changed its fusion pattern split
(RMS_NORM+MUL 11->10, RMS_NORM+MUL+ADD 3->4; same total).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* ci : skip dummy model generation on OpenVINO

test-llama-archs does not build on the OpenVINO platform, so do not try
to generate the dummy models there.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* cont : minor

* tests : enable test-llama-archs on windows

* cont : disable on windows + workaround

* metal : naming nits

* test-fusion : add instructions to update baseline

* context : fix Kimi-K3 graph reserve

* fusion : update MTL

* cont : fix naming

* metal : rework fusion info storage

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : align fusion info API

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : use opaque fusion handle in ad-hoc API

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* ci : move fusion test to dedicated workflow

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* cont : run only on ggml changes

* cont : simplify

* fusion : remove multi-output stuff for now

* ci : fix typo
2026-09-11 12:41:54 +03:00
Iggy Jackson d344123fe2 models: clean up some dead switch branches in old models (#28669)
Some of these if statements were copypastaed in a former refactor and
never cleaned up to remove the cases that could never happen anymore. The
only thing that's shared between these relatives anymore is
llama_model_bert::graph::graph, so the rest of the code doesn't need the
conditionals.
2026-09-10 16:09:35 +03:00