Commit Graph
354 Commits
Author SHA1 Message Date
Georgi Gerganov ccd360f060 llama : remove non-fused GDN and ubatch-dependent graph nodes (#32078)
- remove build_delta_net_chunking and build_delta_net_autoregressive
- always use the fused gated delta net
- make graph_max_nodes independent of the ubatch size
- drop the fused GDN probes and fused node tags

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-11 15:16:16 +02:00
Alexey Kopytko 940a30ee3b model : stop renaming the recurrent cache tensors in plamo2 (#31984) 2026-11-10 17:42:30 +09:00
Aman Gupta a8fad21e99 mtp: read the target hidden states from a device buffer (#31927) 2026-11-09 20:52:32 +08:00
jjperry69andGabe Goodhart 0c6dca2fae models : Attach Granite NVFP4 weight scales (#31449)
* models : Attach Granite NVFP4 weight scales

* feat: Pass nvfp4 scales through build_ffn for all granite architectures

Branch: granite-nvfp4-moe-scales
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

---------

Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>
2026-11-09 13:04:06 +01:00
Xuan-Son Nguyen 7d7f028753 mtmd: refactoring: remove clip.h, consolidate code into mtmd (#31865) 2026-11-08 15:49:06 +01:00
PascalandGeorgi Gerganov d6be065572 llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (#31851)
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls

Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.

* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends

Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.

* tests: cover GGML_PREC_BF16 in the flash attention tests

The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.

* llama: request GGML_PREC_BF16 for the KQ multiplication

* cont : add TODO in clip.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-08 08:37:46 +01:00
usrlocalben bcb0152876 model: add Kimi-K3.1-Pro text model (#31779)
* model: add Kimi-K3.1-Pro text model

K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.

Indexer:
  - the selection is an additive mask over the whole cache, not a gather of
    the top-k rows, so it reuses the existing sparse flash attention path;
    off that path the mask stays correct but still costs a kq over the cache
  - the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
    because dsa_config declares no eps of its own

Graph and cache:
  - new arch with four indexer tensors per DSA layer, and a hybrid memory
    that keeps a K-only MLA cache for those layers, recurrent state for the
    KDA layers, and the indexer key cache
  - the graph node budget shares the K3 entry

Conversion:
  - KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
  - dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
    plus the indexer LayerNorm eps; the layer list is 1-indexed like
    linear_attn_config's, and the converter asserts that it covers exactly
    the full-attention layers
  - the routed experts stay MXFP4 and are repacked as in K3, not dequantized
  - the vision tower and mm_projector are skipped: text only for now
  - the arch key follows the checkpoint's own model_type ("kimi_k31") and the
    loader registers KimiK31ForConditionalGeneration; the release name carries
    a -Pro suffix that nothing in the config or the tensor names uses

Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.

Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.

* converter: avoid clutter in the k3 base

* narrow comment to match model

* update metal fusion expectations

* restore

* sync with PR 31733 refactoring
2026-11-06 23:42:46 +01:00
Xuan-Son Nguyen 6023692b1e llama: fix kv shifting, add tests, add per-model build_rope (#31683)
* llama: fix kv shifting and add tests

* llama: add graph build_rope (#31693)

* llama: add graph build_rope

* improve

* migrate new models

* update review skill

* fix ci

* add --kv-shift to CI
2026-11-05 21:10:36 +01:00
Sigbjørn Skjæret e8df20d238 models : recursive fix (#31749) 2026-11-05 20:06:55 +01:00
fecdc81eec model : add Ripple (ricursiveai) (#31692)
* feat : added ricursiveai ripple support

* fix : mandatiry SWA, scale through helper, comments back

* fix : metal

* model : make ricursive RoPE/NoPE explicit and validate the layer pattern

Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.

Assisted-by: Claude Opus 5.5

* model : make ricursive swiglu_clamp per-layer

Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.

Assisted-by: Claude Opus 5.5

* model : use the ricursive per-tensor scales and RoPE parameters

Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.

Assisted-by: Claude Opus 5.5

* convert : drop the ricursive chat_template.jinja override

The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.

Assisted-by: Claude Opus 5.5

* fix : remove NVFP4 checkpoint quantization block

* match #31733 refactor

---------

Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 19:55:15 +01:00
45b6604b20 model: Add support for the xing4_0 (#31589)
* model: add support for Xing4.0

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* fix: rename model file to xing4-0.cpp to match CI expectations

* model: rename xing4_0.cpp to xing4-0.cpp

* lint: fix E128 continuation indent in conversion/xing.py

* tests: add xing4_0 fusion baseline for Metal

* reduce comments

---------

Co-authored-by: shixq7 <shixq7@chinatelecom.cn>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 17:47:27 +01:00
Xuan-Son Nguyen e0f96a1a12 llama: refactoring: move model to class member of llm_graph_context (#31733) 2026-11-05 15:41:03 +01:00
Pascal 040ce199a1 graph: store the MTP conv snapshots of every recurrent model in one set_rows (#31660)
* graph: store the conv state of every rollback arch in one set_rows

The rollback slots of the causal conv history were written with one
copy per slot in qwen4exp, glm5-next, kimi-k3, bailingmoe3, lfm and
mamba2, while delta_net already wrote them with a single set_rows.
build_conv_state_store now does this for all of them: one copy without
rollback, otherwise one set_rows of the overlapping windows into the
min(n, n_rs_seq + 1) slots a rollback can read.

The slot order {n_rs_seq, ..., 0} is computed once per graph and each
conv layer takes a view of its tail. Its size does not depend on the
ubatch, so the reserved graph covers every ubatch shape, and the graph
input that held it is gone.

A model that is not causal never rolls back, so the context clamps
n_rs_seq to 0 for it, like for an arch without rollback support.

* nits: compute the conv slot order in the graph and keep the n_head name of bailingmoe3

* graph: build the conv slot order in each conv layer

The slot order {K-1, ..., 0} of the conv state store was one tensor
shared by every conv layer of the graph, built on the device of the
first one, so each layer on another device copied the view of its
tail. Each conv layer now builds its own order next to its set_rows,
and llm_graph_input_rs no longer holds it.
2026-11-05 09:28:03 +01:00
Daniel Bevenius 330f0189c9 qwen2 : add nvfp4 scales to build_ffn (#31657)
This commit adds nvfp4 scales to qwen2's build_ffn call which are
currently passed as NULL.

The motivation for this is that a model converted to NVFP4 will
currently crash because the scales are loaded but never applied in
the FFN:
```console
llama-completion: llama.cpp/src/llama-sampler.cpp:1218:
void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*):
Assertion `found' failed.
```
The attention already applies its scales but ffn is missing them.
2026-11-05 06:29:50 +01:00
Pascal bbb02c93e9 ggml: bound views by the bytes their strides reach (#31584)
* ggml: bound views by the bytes their strides reach

A view was checked against its source with the size of a contiguous
tensor of the same shape, computed before its strides were set. Views
whose rows overlap failed that check although they stay inside their
source, and views with wider strides passed it while reaching past
the end. The view functions now pass their strides to the tensor
constructor, which checks the bytes they actually reach, like
ggml_nbytes. The conv windows of delta_net are created with all their
rows at once instead of being widened after the check.

* ggml: assert that a strided view starts from contiguous elements

The view functions set nb[0] to the type size, so the source must have
contiguous elements along dim 0 for the view to read them.
2026-11-04 09:47:11 +02:00
b97627967c model, mtmd: add chatterbox TTS support (#30993)
* add chatterbox TTS support

Co-authored-by: Pascal <admin@serveurperso.com>

* add built-in voice support

* rm redundant code

* clean up

* clean up 2

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* improve

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-03 23:47:11 +01:00
6cb3e672e8 gemma4: graph optimizations (massive gains on pp for E2B and E4B) (#31356)
* Optimizing Gemma4 graph for prompt processing

TODO:
	- Currently complete microbatch needs to contain tokens which does not require the logits; need to optimize this further
	- There are a few more operations that are performed after the kv cache is calculated, need to safely optimize them.

* Not filtering the rows when they are unmasked

* Cropping out the attention mask too.

* gemma-4: improved pruning, mask cropping and PLE scaling

Prune before the first KV-shared layer with the single-stream, causal, non-embedding safety gates.
Use cropped masks only during shared-layer attention.
Preserve requested layer-input embeddings in the KV prefix.
Remove the PLE contiguous materialization and fold equivalent PLE scaling into RMSNorm parameters/weights.

* spec-mtp: Enable masked h_nextn extraction only when the draft context shares the target KV cache

This lets Gemma 4 prune unused prompt rows before its shared layers so that we keep the full perf P.P gains with MTP.

Avoid extracting target hidden states for output-free prompt chunks in that shared-MTP path.
When a batch has a selected final token, extract only that row unless the whole verification batch is selected.

Keep ordinary MTP and classic speculative decoding behavior unchanged: they still use unmasked h_nextn because the draft
context must catch up on all target rows.

speculative-simple needs an update to select the final prompt row only for shared-KV MTP,
so the optimization works there without changing other speculative modes.
And it looks like the server already selects its final prompt row for first-token sampling.

* spec-mtp: make sure to not drop first draft round

* gemma4: enable pruning based on static cparams.kv_unified/n_seq_max=1

* gemma4-mtp: simplify prunning logic and avoid graph branches/re-allocs

* gemma4: update pruning gates to use only n_seq_max==1, kv_unified can now be used for multi-seq

* spec-simple: revert spec-simple changes for now

* gemma4: remove causal_attn from pruning gate (last source of reallocs)

* cont : adapt to llm_attn_mask changes

* cont : add TODO

---------

Co-authored-by: Meet Amin <meetamin378@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-03 20:30:37 +02:00
Giulianno 9309bf1bcf model : Add support for Metis-1.6 models (#31134)
Assisted-by: Claude
2026-11-03 16:24:45 +01:00
Tarek Dakhran d1044bc827 model : enable NVFP4 LFM3 ggufs (#31608) 2026-11-03 15:36:44 +01:00
Pascal 749db7f034 qwen4exp: write the conv state tail without an intermediate cont (#31443)
build_conv_state_at makes a contiguous copy of the conv input tail
before writing it into the recurrent state, so every GDN layer issues
two device copies per generated token where one is enough. On
Qwen3.8-Flash-Next (36 GDN layers) that is 72 copies per token, now 36.
Speculative decoding adds one more per rollback slot. ggml_cpy reads
the strided view directly, as the shared build_conv_state already does,
so the extra copy is dropped.
2026-11-03 13:47:23 +02:00
Tarek Dakhran f9448cddc4 model : record layer inputs in LFM3 for spec dec (#31598) 2026-11-03 13:29:10 +02:00
Aman GuptaandPascal b74298c109 delta_net_base: refactor conv state (#30878)
* delta_net: access conv state via views

* fix sm tensor

* delta_net: fill the conv slot index on the host

The rollback slot index of the conv windows is a constant per graph
shape. It is now an I32 input of the recurrent memory filled on the
host, instead of an arange and an argsort computed on the device, and
the meta backend no longer needs a mirrored split for ARANGE.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-03 08:28:35 +01:00
Georgi Gerganov ca8daad199 llama : drop the redundant cnv from llm_attn_mask (#31558)
The dspark block mask is now derived at input-fill time: llm_attn_mask::block_uniform
fills every token of a sequence with the mask row of its first token, so the
graph-level reshape/repeat in dflash is gone and cnv always aliases raw.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-03 08:11:06 +02:00
Aman Gupta 2cbca389f5 dflash: copy hidden features via device copies (#31506) 2026-11-03 09:02:32 +08:00
Maphist0andSigbjørn Skjæret e4b6c8ca6a model: add Clover support (#31306)
* model : add AR-only Clover MoE support

Add GGUF conversion, model loading, and causal hybrid attention for Clover, reusing shared attention, recurrent-state, MoE, and LoRA helpers.

* Update tests/test-llama-archs.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* convert : filter Clover tensors using filter_tensors

* clover : use the standard ssm_a tensor name

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-02 18:37:27 +01:00
Aman GuptaandGeorgi Gerganov 1b6935b227 llama : flash_attn_ext_rows - more efficient unified kv for multi-seq (#29510)
* llama : use llm_attn_mask and kv_rows for unified attention

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* cont : optimize `get_seq_cells`

* llama : build the kv_rows lists from per-sequence cell bitmaps

Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.

Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).

* cont : make get_kv_rows_p_min private

* cont : minor metal

* llama : use kv_rows for every ubatch of a unified cache

A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.

Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.

* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-02 18:13:33 +02:00
Niklas Wenzel 0138937e3f llama : fix rotation of attention tensors in layers with shared KV (#31406)
* llama : fix rotation of layers that reuse another layer's KV cache

* llama : use KV source rotation only for attn_* tensors
2026-10-31 23:30:49 +01:00
Anthony Maltsev ddf48f85c9 reshaping so mm on 2d instead of 3d tensor for lfm sc (#30831) 2026-10-31 20:34:57 +01:00
Lorenzo Bona 344846b4fc gemma3n, gemma4 : create token_embd before the tied output (#31394)
The duplicated output looks up token_embd in its buffer context, so
token_embd must exist first. With the old order a load where input and
output share a buffer type (CPU-only, -ngl 0) held the table twice:
gemma-4-E2B-it Q4_K_M, no mmap, CPU model buffer 3163.73 -> 2947.73 MiB.
2026-10-30 09:36:27 +01:00
Anton IandPascal fb5997274e kv-cache : keep rotation constants on the GPU (#31073)
* kv-cache : keep rotation constants on the GPU

Assisted-by: Codex

* tests : verify resident KV rotation buffers

Assisted-by: Codex

* kv-cache : document remaining multi-device rotation work

Assisted-by: Codex

* kv-cache : support per-layer rotation buffers

Assisted-by: Codex

* kv-cache : fix hybrid SWA rotation inputs and mixed placement check

Assisted-by: Codex

* models : use per-layer rotation in mistral-large4 and dflash

* tests: drop the rotation buffer tests

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-30 10:15:06 +02:00
Oliver SimonsandSigbjørn Skjæret 68d20b884c Quant: OCP FP8 E4M3 support (#27512)
* WIP OCP FP8 E4M3 support

* Fix python type-checker

* Materialize input_scales also for FP8 when converting

* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4

1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
   dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
   weights are reshaped

* Allow remaining FP8 tensors to be dequantized during conversion

* Allow tensor-strategy for FP8-only compressed-tensor ckpts

* Fix and test gguf-py behavior for FP8_E4M3

* Disallow QKV-fusion for FP8 temporarily

* Fix conversion of fused qkvz & FP8 for Qwen3-Next

* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws

* Emit warnings when not fusing FP8 during conversion

* More FP8 conversion fixes

* Add supported for strided copies to/from FP8 in CPU

* remove unneeded scratch buffer from FP8 cpu cpy/dup path

* add explanatory comments and TODOs

* Add dedicated encoder and decoder tests for FP8

* Whitelist ggml_op_get_rows in metal instead of blacklisting

required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list

https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512

* Fix more conversion bugs

* Defer weight materialization during conversion

* Blacklist fp8 in metal backend instead of white-list

* Adjust indents

* Fix lint + execute tests

* Add fast CPU kernel for F8_E4M3

* Conversion fixes

* Fix remaining gaps

* Enable tied weights with shared scales

* Make mandatory-F32 rules apply to FP8 during conversion

* Fix python type checks

https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512

* Fix kaleidai unit-test failures

https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150

* Relax numeric precision constraints for pre-check

* Leave fast-FP8 CPU on RISC-V for follow-up

We already exclude arm from the fast-path

* Allow output scale tensors to be located in any gguf shard

* Move test_conversion_fp8.py test execution to behavior-check.yaml

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it

* Install all requirements also on Windows for behavior-check

Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512

* Unmap lazy tensors before clean-up in Windows

Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512

* LF line endings

https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512

* Extract FP8 tests from llama-arch to its own test-fixture

See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221

* Update gguf-py/gguf/scripts/gguf_convert_endian.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-29 18:14:37 +01:00
Xuan-Son Nguyen 78feec062d model: mistral le chonk (#31376)
* llama: extend DSA and sdpark to new model

* fix ci
2026-10-29 15:43:33 +01:00
Giuseppe Scrivano abb76ab380 model : Support Limite 1B - Violetto (#29433)
* gguf-py : add Limite arch

Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>

* convert : add Limite support

Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>

* llama : add Limite support

Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>

---------

Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
2026-10-29 10:13:41 +02:00
Tarek DakhranandXuan Son Nguyen d05706b4f3 model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B (#31275)
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B

Assisted-by: Claude Opus 5.5

* convert : keep the speech task templates out of the d1 GGUFs

Assisted-by: Claude Opus 5.5

* small refactoring: no_batching

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 23:36:32 +01:00
19cce69199 model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B (#31252)
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* tests : set the SWA rope base of the LFM3 test model

Assisted-by: Claude Opus 5.5

* llama : enable recurrent state rollback for LFM3

Assisted-by: Claude Opus 5.5

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>

* gguf-py : align the lfm3vl mapping comments

Assisted-by: Claude Opus 5.5

* llama : fix CRLF line endings from the review suggestions

Assisted-by: Claude Opus 5.5

* tests : add the LFM3 Metal fusion baseline

Assisted-by: Claude Opus 5.5

* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
2026-10-27 17:06:40 +01:00
Aman GuptaandGeorgi Gerganov 27b7a8a009 llama : build the k-pool sparse selection independently of the kq_mask shape (#31243)
* llama : build the k-pool sparse selection independently of the kq_mask shape

qwen4exp (QSA) and glm5-next pick cache cells by index and no longer read the kq_mask values, but they still shaped the selection like the dense kq_mask. Build the selection as [n_kv, n_tokens] over the cache cells instead, and split it into the streams of the K view right before the attention.

This lets the dense mask change its layout (e.g. per-sequence row lists) without touching the sparse layers.

* improve comment

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* llama : drop the unused build_fused_top_k_mask args, review nits

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-27 21:57:45 +08:00
Alexey Kopytko d78f40d9d3 glm5-next : QSA without kq_mask (+ DSA) (#31009)
* glm5-next : QSA without kq_mask

* DSA: dsa top-k mask change tested with layers 0,3 of DeepSeek-V3.2
2026-10-27 18:54:36 +09:00
Georgi Gerganov 4b851fbee8 models : replace ggml_tensor** out-params with output structs (#31142)
The hyper-connection and DSA graph builders wrote their extra outputs
(post / comb gates, carried top-k selections, inject gates) through
ggml_tensor** out-params. Return them in a small struct next to the
primary output instead.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-26 16:19:47 +02:00
Aman GuptaandGeorgi Gerganov a074215352 deepseekv4.1: follow-up review fixes (#31137)
* kv-cache : use ggml_backend_tensor_copy to refill the DSV4 K mirrors

* llama : move the Engram pad id to llama_hparams and the token map to llama_model

The saver no longer needs the deepseek41 model class.

* deepseek41 : drop the F32 src1 request on the HC mixes matmul

* ggml : add softplus TODOs in the backends that still use log(1 + exp(x))

* convert : make the DSV4/DSV4.1 modify_tensors generators, drop the redundant vision filter

* deepseek41 : clean up comments

* deepseek41 : drop the F32 src1 requests on the router and indexer matmuls

* convert : use format_tensor_name directly in DeepseekV4Model

* s/comp_kv_src/comp_il_src/g

* cont : fix TODOs for log1p

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-26 15:52:02 +08:00
Aman Gupta 2280f0fd16 qwen: merge tensors on convert (#31028)
* qwen: merge tensors on convert

* convert: use generator functions

* qwen4exp: remove conts and pre-scale
2026-10-25 15:23:22 +02:00
Aman GuptaandDaniel Han 74bca39500 model: DSV4.1 flash (#31086)
* model : add DeepSeek-V4.1-Flash (deepseek41)

* deepseek41 : remove MXFP8, convert the Engram table to BF16

* deepseek41 : compute the compressed-KV read layers once in the cache

* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le

* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather

* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer

* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle

* deepseek41 : keep il last in build_hc_mixes

* deepseek41 : sort the top-k rows before the sparse gather

* tests : add deepseek41 to the Metal fusion baseline

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-10-25 09:43:51 +08:00
Satyam Thakurandorigin-labs-ai 73adf89e38 minimax-m3 : forward quant scales to the graph (fixes #31063) (#31097)
The generic scale pass in llama-model.cpp loads the per-expert
NVFP4 scale2 tensors (ffn_{up,gate,down}_exps_s) and the per-tensor
MXFP8 scales for the dense FFN and the shared expert, but
minimax-m3.cpp never forwarded any of them to the graph.

build_moe_ffn() defaults these to nullptr, so build_lora_mm_id()
skipped its w_s branch entirely and the routed-expert weights came
out global_scale times too large, swamping the residual (reported
G ~= 15123 for M3.1 Flash) and producing token soup.

Forward all scale tensors at the three call sites, matching the
pattern used by flint and the other NVFP4-capable MoE models.

Co-authored-by: origin-labs-ai <origin-labs-ai@users.noreply.github.com>
2026-10-24 16:09:02 +02:00
Alexey Kopytko 5c31142682 qwen4exp : build the QSA selection without the kq_mask (#30965)
* qwen4exp : build the QSA selection without the kq_mask

* clean

* lang

* qwen4exp : reshape the fused QSA selection to the kq_mask streams

ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while
build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream].
With several streams the shape assert failed at graph reserve.

* test the QSA optimization

* reword

* update build_fused_top_k_mask instead of handwiring

* simplify

* keep il at the end for consistency
2026-10-23 19:49:08 +02:00
Martin Kleinberger 19ab5a5e65 clef : include <algorithm> for std::find and std::copy (#30981)
[no ci]
2026-10-22 18:14:47 +02:00
Georgi Gerganov 5c19a5a756 glm5 : optimize inputs (#30932)
* glm5 : avoid sel_mask_2d input

* cont : avoid new_pool_flat input
2026-10-22 16:59:29 +03:00
Georgi Gerganov d9ee007a06 model : fix qwen4exp layer inputs for DFlash drafters (#30927)
qwen4exp published the hc-wide residual [n_embd, hc, n_tokens] as
t_layer_inp, but the drafter reads layer inputs as [n_embd, n_tokens]
rows: llama_context sizes each embd_layer_inp buffer as n_embd * n_batch
and the DFlash encoder input is n_extract * n_embd wide. Evaluating a
DFlash draft over a qwen4exp target therefore aborts on the first request
in llama_context::extract_layer_inputs.

Collapse the streams with a plain mean before exposing them, as deepseek4
already does for its hyper-connection residual, and build the mean only
for the layers a drafter actually requested.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-22 16:49:38 +03:00
martin-cimminoandMartin Cimmino 0693dbfb91 nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type (#23343)
* nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type

* gguf-py : write Metaspace prepend_scheme as GGUF metadata

* vocab : parse metaspace prepend_scheme into enum, rename key

* vocab : use metaspace prepend_scheme in detokenize, add TODOs for other tokenizers

* vocab : use constant for metaspace escaped space

---------

Co-authored-by: Martin Cimmino <martin.cimmino@domyn.com>
2026-10-22 13:34:47 +02:00
Gabe Goodhart bf039c8496 models : Record layer inputs for all granite models (#30957)
Branch: GraniteSpeculatorSupport
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-22 10:02:41 +03:00
Hrishith Thadicherla 4912ac24f1 dflash : load optional FC input scale (#30945)
Assisted-by: Codex
2026-10-22 08:24:12 +03:00
alexey-rsiandSigbjørn Skjæret a37cc46104 models : add Flint model (FlintForCausalLM) (#30928)
* Add Flint model (FlintForCausalLM). Assisted by Codex

* cleanup

* one more

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-21 23:26:22 +02:00