Commit Graph
525 Commits
Author SHA1 Message Date
Xuan-Son Nguyen fc504f91d8 mtmd: support higgs v3 STT and vibevoice-ASR (#32329)
* mtmd: support higgs v3 and vibevoice-ASR

* use proper mapping

* address review comment

* nits

* rm hacky get_audio_length, add mtmd_helper_get_media_meta
2026-11-15 21:25:34 +01:00
Xuan-Son Nguyen 7ae3c6b231 mtmd: add zonos3-mini TTS model (#32020)
* zonos3 init

* add audio gen pipeline

* fix string_format

* add default voice as gguf metadata

* better ref

* support voice clone

* add delivery params

* optimize speed

* nits

* nits 2

* add text normalization

* nits
2026-11-15 19:44:44 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5c008e0008 feat: add Inkling 2 support (text + vision) (#32217)
* convert : add Inkling2 text GGUF conversion

Assisted-by: Codex

* llama : implement Inkling2 text inference

Assisted-by: Codex

* tests : cover Inkling2 text support

Assisted-by: Codex

* conversion : dequantize packed Inkling2 experts in row chunks

Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.

Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.

Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.

Assisted-by: Codex

* convert : add Inkling2 vision mmproj support

Assisted-by: Codex

* mtmd : implement Inkling2 still-image encoder

Assisted-by: Codex

* inkling2 : scope image loading

Assisted-by: Codex

* inkling2 : enable flash attention with folded bias and sinks

Assisted-by: Codex

* cuda : route per-head flash attention masks through vec

Assisted-by: Codex

* mtmd : support Inkling2 temporal frame means

Assisted-by: Codex

* style : restore shared image comment

Assisted-by: Codex

* tests : enlarge Inkling2 context fixture

Assisted-by: Codex

* inkling2 : load per-layer expert feed-forward widths

Assisted-by: Codex

* tests : add Inkling2 Metal fusion baselines

Assisted-by: Codex

* inkling2 : group and align registry entries

Assisted-by: Fable

* convert : regenerate the Inkling2 tokenizer hash with the update script

Assisted-by: Fable

* conversion : revert the print_registered_models change

Assisted-by: Fable

* conversion : sort the Inkling2 model map entries

Assisted-by: Fable

* convert : map Inkling2 vision checkpoint names in the tensor table

Assisted-by: Fable

* convert : map Inkling2 text checkpoint names in the tensor table

Assisted-by: Fable

* tests : document the Inkling2 packed expert conversion test

Assisted-by: Fable

* tests : cover Inkling2 checkpoint and transformers tensor names

Assisted-by: Fable

* llama : place Inkling2 fields and list entries by file convention

Assisted-by: Fable

* gguf-py : align Inkling2 tensor mapping comments

Assisted-by: Fable

* mtmd : place Inkling2 entries by file convention

Assisted-by: Fable

* tests : place Inkling2 cases after the existing ones

Assisted-by: Fable

* cuda : index per-head flash attention masks in the tile and mma kernels

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : project the relative bias before the gather, drop arch branches

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* convert : require Inkling2 hparams, add inkling2 pre-tokenizer

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* mtmd : cap Inkling2 image size, drop video frame averaging

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : fix CI build, Meta device assert and Windows conversion test

Assisted-by: Claude

* tests : use the default sliding window for the Inkling2 test model

Assisted-by: Claude

* simplify mtmd part

* give _v arch for vision

* remove alpha channel handling, it should be a follow up

* explicit rule about audio + vision models

* add audio support

* rm audio

* rm gguf mtmd rope base

* address review comment

* nits

* do not normalize embd from mtmd

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-15 00:51:31 +01:00
Oliver Simons 3ba1ce0e28 qwen3.5: merge tensors on convert also (#32102)
* qwen3.5: merge tensors on convert also

* Remove conversion test for qwen3.5
2026-11-12 15:54:58 +01:00
Sigbjørn Skjæret 582eab7186 ci : bump ty to 0.0.100 (#32030)
* bump ty to 0.0.100

* rename rules

* type fixes

* type fix

* proper type fix
2026-11-10 21:54:14 +01:00
Xuan-Son Nguyen 351025ec53 mtmd: add kimi K3.1 vision (#31980)
* mtmd: support kimi k3.1 vision

* fix preproc (both k2.5 and k3.1) and projector mlp

* small renaming
2026-11-10 11:28:47 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5f490d7bfe feat: external PLE file loading (#31185)
* model : allow loading external PLE files

Assisted-by: Codex

* convert : add separate PLE exports

Assisted-by: Codex

* tests : fix PLE export test lint

Assisted-by: Codex

* model : make --ple an alias of --shard, allow multiple shard locations

Reuse the multi-shard loading path for external PLE files instead of a
separate option.

* --ple is an alias of --shard: both take a comma separated list and may
  be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
  auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
  exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
  with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
  common_model_load() uses llama_model_load_from_splits for two or more
  paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
  memory measurement load every shard

Assisted-by: GPT-6-Astra

* rm -isnm

* make sure to dedup paths

* rm test_convert_ple

* address comments

* allow ple shard to override tensor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-09 19:41:01 +01:00
Xuan-Son Nguyen 7d7f028753 mtmd: refactoring: remove clip.h, consolidate code into mtmd (#31865) 2026-11-08 15:49:06 +01:00
Xuan-Son NguyenandPascal 9f0e5098a5 mtmd: support MageVL-Cowork model (#31819)
* mtmd: support MageVL-Cowork model

* mtmd: keep magevl rope_freqs at model level and check its shape

The rope factors are shared by every layer, so they live in clip_model
instead of a pointer copied into each clip_layer. The loader now checks
that v.rope_freqs is F32 with d_head elements and that d_head splits
into the 4:6:6 pairs the graph expects.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-07 19:36:44 +01:00
usrlocalben bcb0152876 model: add Kimi-K3.1-Pro text model (#31779)
* model: add Kimi-K3.1-Pro text model

K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.

Indexer:
  - the selection is an additive mask over the whole cache, not a gather of
    the top-k rows, so it reuses the existing sparse flash attention path;
    off that path the mask stays correct but still costs a kq over the cache
  - the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
    because dsa_config declares no eps of its own

Graph and cache:
  - new arch with four indexer tensors per DSA layer, and a hybrid memory
    that keeps a K-only MLA cache for those layers, recurrent state for the
    KDA layers, and the indexer key cache
  - the graph node budget shares the K3 entry

Conversion:
  - KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
  - dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
    plus the indexer LayerNorm eps; the layer list is 1-indexed like
    linear_attn_config's, and the converter asserts that it covers exactly
    the full-attention layers
  - the routed experts stay MXFP4 and are repacked as in K3, not dequantized
  - the vision tower and mm_projector are skipped: text only for now
  - the arch key follows the checkpoint's own model_type ("kimi_k31") and the
    loader registers KimiK31ForConditionalGeneration; the release name carries
    a -Pro suffix that nothing in the config or the tensor names uses

Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.

Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.

* converter: avoid clutter in the k3 base

* narrow comment to match model

* update metal fusion expectations

* restore

* sync with PR 31733 refactoring
2026-11-06 23:42:46 +01:00
Xuan-Son Nguyen b1c902f026 llama, mtmd: (chatterbox) add use_classifier_free_guidance as gguf metadata (#31698)
* llama, mtmd: add use_cfg as gguf metadata

* use_classifier_free_guidance
2026-11-05 20:09:33 +01:00
fecdc81eec model : add Ripple (ricursiveai) (#31692)
* feat : added ricursiveai ripple support

* fix : mandatiry SWA, scale through helper, comments back

* fix : metal

* model : make ricursive RoPE/NoPE explicit and validate the layer pattern

Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.

Assisted-by: Claude Opus 5.5

* model : make ricursive swiglu_clamp per-layer

Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.

Assisted-by: Claude Opus 5.5

* model : use the ricursive per-tensor scales and RoPE parameters

Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.

Assisted-by: Claude Opus 5.5

* convert : drop the ricursive chat_template.jinja override

The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.

Assisted-by: Claude Opus 5.5

* fix : remove NVFP4 checkpoint quantization block

* match #31733 refactor

---------

Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 19:55:15 +01:00
45b6604b20 model: Add support for the xing4_0 (#31589)
* model: add support for Xing4.0

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* fix: rename model file to xing4-0.cpp to match CI expectations

* model: rename xing4_0.cpp to xing4-0.cpp

* lint: fix E128 continuation indent in conversion/xing.py

* tests: add xing4_0 fusion baseline for Metal

* reduce comments

---------

Co-authored-by: shixq7 <shixq7@chinatelecom.cn>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 17:47:27 +01:00
Sigbjørn Skjæret cc78da97c0 gguf-py : external chat templates have precedence over tokenizer_config (#31717) 2026-11-05 13:04:48 +01:00
Xuan-Son NguyenandPascal 5851a8e505 model, mtmd: drama-3 tts support (#31613)
* model, mtmd: drama-3 tts support

* clean up

* mtmd: drama3 helper stops at the codebook tower limit

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-04 15:15:09 +01:00
Aaron Thomas 034e2535a9 gguf-py : validate every tokenizer merge pair, not only the first (#31631)
* gguf-py : validate every tokenizer merge pair, not only the first

The transformers 4.45 pair format was detected by checking merges[0],
then every element was converted unchecked: a non-pair element raised
a bare TypeError, and pairs with the wrong arity or non-string parts
were silently mangled into invalid merges.

Validate each element (2-string list) up front and raise the existing
"Unknown tokenizer merges format" error for anything else.

Assisted-by: Devin

* gguf-py : remove test_vocab.py per review
2026-11-04 14:42:39 +01:00
b97627967c model, mtmd: add chatterbox TTS support (#30993)
* add chatterbox TTS support

Co-authored-by: Pascal <admin@serveurperso.com>

* add built-in voice support

* rm redundant code

* clean up

* clean up 2

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* improve

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-03 23:47:11 +01:00
Giulianno 9309bf1bcf model : Add support for Metis-1.6 models (#31134)
Assisted-by: Claude
2026-11-03 16:24:45 +01:00
Prabhsimran Singh 28b812d6cb mtmd : add Nemotron 3.5 Transcribe support (#31471)
* mtmd : add Nemotron 3.5 Transcribe support

Assisted-by: Claude

* convert : fix vocab of draft models for Nemotron-H targets

Assisted-by: Claude

* convert : reject non-BF16 Transcribe checkpoints

Assisted-by: Claude

* mtmd : make speaker FIFO predictions local

Assisted-by: Claude

* mtmd : normalize the mel in encode_windows

Assisted-by: Claude

* convert : map the generated speaker kernel

Assisted-by: Claude

* mtmd : validate Nemotron Transcribe mmproj shapes

Assisted-by: Claude
2026-11-03 14:27:05 +01:00
Maphist0andSigbjørn Skjæret e4b6c8ca6a model: add Clover support (#31306)
* model : add AR-only Clover MoE support

Add GGUF conversion, model loading, and causal hybrid attention for Clover, reusing shared attention, recurrent-state, MoE, and LoRA helpers.

* Update tests/test-llama-archs.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* convert : filter Clover tensors using filter_tensors

* clover : use the standard ssm_a tensor name

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-02 18:37:27 +01:00
Oliver SimonsandSigbjørn Skjæret 68d20b884c Quant: OCP FP8 E4M3 support (#27512)
* WIP OCP FP8 E4M3 support

* Fix python type-checker

* Materialize input_scales also for FP8 when converting

* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4

1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
   dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
   weights are reshaped

* Allow remaining FP8 tensors to be dequantized during conversion

* Allow tensor-strategy for FP8-only compressed-tensor ckpts

* Fix and test gguf-py behavior for FP8_E4M3

* Disallow QKV-fusion for FP8 temporarily

* Fix conversion of fused qkvz & FP8 for Qwen3-Next

* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws

* Emit warnings when not fusing FP8 during conversion

* More FP8 conversion fixes

* Add supported for strided copies to/from FP8 in CPU

* remove unneeded scratch buffer from FP8 cpu cpy/dup path

* add explanatory comments and TODOs

* Add dedicated encoder and decoder tests for FP8

* Whitelist ggml_op_get_rows in metal instead of blacklisting

required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list

https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512

* Fix more conversion bugs

* Defer weight materialization during conversion

* Blacklist fp8 in metal backend instead of white-list

* Adjust indents

* Fix lint + execute tests

* Add fast CPU kernel for F8_E4M3

* Conversion fixes

* Fix remaining gaps

* Enable tied weights with shared scales

* Make mandatory-F32 rules apply to FP8 during conversion

* Fix python type checks

https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512

* Fix kaleidai unit-test failures

https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150

* Relax numeric precision constraints for pre-check

* Leave fast-FP8 CPU on RISC-V for follow-up

We already exclude arm from the fast-path

* Allow output scale tensors to be located in any gguf shard

* Move test_conversion_fp8.py test execution to behavior-check.yaml

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it

* Install all requirements also on Windows for behavior-check

Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512

* Unmap lazy tensors before clean-up in Windows

Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512

* LF line endings

https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512

* Extract FP8 tests from llama-arch to its own test-fixture

See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221

* Update gguf-py/gguf/scripts/gguf_convert_endian.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-29 18:14:37 +01:00
Xuan-Son Nguyen 78feec062d model: mistral le chonk (#31376)
* llama: extend DSA and sdpark to new model

* fix ci
2026-10-29 15:43:33 +01:00
Giuseppe Scrivano abb76ab380 model : Support Limite 1B - Violetto (#29433)
* gguf-py : add Limite arch

Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>

* convert : add Limite support

Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>

* llama : add Limite support

Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>

---------

Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
2026-10-29 10:13:41 +02:00
Xuan-Son NguyenandSigbjørn Skjæret 28cb47c7c8 gguf-py: fix np_roundf dropping the sign (#31310)
* gguf-py: fix np_roundf dropping the sign

* Update requirements/requirements-convert_legacy_llama.txt

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* requirements.txt

* Update tools/server/tests/requirements.txt

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-28 14:03:59 +01:00
Tarek DakhranandXuan Son Nguyen d05706b4f3 model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B (#31275)
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B

Assisted-by: Claude Opus 5.5

* convert : keep the speech task templates out of the d1 GGUFs

Assisted-by: Claude Opus 5.5

* small refactoring: no_batching

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 23:36:32 +01:00
Tarek DakhranandXuan Son Nguyen 19ff817fb2 model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B (#31257)
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* convert : read the LFM3-Audio config.json as released

A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).

Assisted-by: Claude Opus 5.5

* mtmd : disable flash attention for the LFM3-Audio depthformer

Assisted-by: Claude Opus 5.5

* tts : add a voice option for LFM3-Audio and LFM3-Omni

The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.

Assisted-by: Claude Opus 5.5

* clean up template handling

* convert : detect LFM3-Audio and LFM3-Omni by model_type

Assisted-by: Claude Opus 5.5

* move banned tokens --> logits bias --> mtmd_helper

* leave mtmd-cli unchanged

* no more template_asr_system

* left over

* llama : keep the LFM3-Audio encoder convs and codec unquantized

Assisted-by: Claude Opus 5.5

* common, mtmd : shorten comments

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 18:47:14 +01:00
19cce69199 model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B (#31252)
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* tests : set the SWA rope base of the LFM3 test model

Assisted-by: Claude Opus 5.5

* llama : enable recurrent state rollback for LFM3

Assisted-by: Claude Opus 5.5

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>

* gguf-py : align the lfm3vl mapping comments

Assisted-by: Claude Opus 5.5

* llama : fix CRLF line endings from the review suggestions

Assisted-by: Claude Opus 5.5

* tests : add the LFM3 Metal fusion baseline

Assisted-by: Claude Opus 5.5

* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
2026-10-27 17:06:40 +01:00
tc-mbandXuan Son Nguyen 7eed6b7685 model: Add MiniCPM-OCR-Jev (#30980)
* model : add MiniCPM-OCR-Jev support

The model is a Qwen3.5-VL trunk with a value head read at the last token: it
scores an OCR text against an image rendered from it, it does not generate.

Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone.
The class scores are the pooled output of the sequence, read with last pooling,
so the conversion turns score.weight into cls.output and sets
embedding_length_out. The checkpoint has no LM head, so text generation is
rejected instead of using the output weights duplicated from the token
embeddings.

The task is fixed, so its prompt is entirely in the systemone template, the
same way h2o-lightning keeps its default there. The processor of the checkpoint
allows up to 16384 merged tokens per image, more than the Qwen-VL default of
mtmd, so the mmproj keeps the image budget of the preprocessor config. The
checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as
other models do.

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix code

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* use gguf meta classifier_output_labels

* remove redudant code changes

---------

Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 11:49:12 +01:00
Aman Gupta 2280f0fd16 qwen: merge tensors on convert (#31028)
* qwen: merge tensors on convert

* convert: use generator functions

* qwen4exp: remove conts and pre-scale
2026-10-25 15:23:22 +02:00
Aman GuptaandDaniel Han 74bca39500 model: DSV4.1 flash (#31086)
* model : add DeepSeek-V4.1-Flash (deepseek41)

* deepseek41 : remove MXFP8, convert the Engram table to BF16

* deepseek41 : compute the compressed-KV read layers once in the cache

* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le

* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather

* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer

* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle

* deepseek41 : keep il last in build_hc_mixes

* deepseek41 : sort the top-k rows before the sparse gather

* tests : add deepseek41 to the Metal fusion baseline

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-10-25 09:43:51 +08:00
Hans 9ae73e68f5 mtmd : add Soprano TTS support (#28607)
Add Soprano-80M and Soprano-1.1-80M (https://huggingface.co/ekwek/Soprano-1.1-80M)
as an audio generation model: a Qwen3 backbone plus a Vocos decoder that runs
as a gen-only mmproj.

- convert: detect Soprano checkpoints and export the backbone as qwen3 and the
  decoder.pth Vocos weights as the mmproj
- vocab: add the soprano pre-tokenizer (lowercase, whitespace collapse,
  per-digit split, [UNK] fallback)
- mtmd: add the Vocos graph and ISTFT, allow loading a gen-only mmproj for
  Soprano and include the gen-audio context in mtmd_get_memory_usage
- mtmd-helper: add the Soprano pipeline; the decoder runs over the whole
  utterance, so stream mode sends the audio in one chunk after flush()
- tts: document Soprano usage

Assisted-by: Codex
Assisted-by: Claude Opus 5.5
2026-10-24 01:09:50 +02:00
tc-mb a08ab75ae7 mtmd : add MiniCPM-OCR 1.0 support (#30979)
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
2026-10-22 19:22:15 +02:00
martin-cimminoandMartin Cimmino 0693dbfb91 nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type (#23343)
* nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type

* gguf-py : write Metaspace prepend_scheme as GGUF metadata

* vocab : parse metaspace prepend_scheme into enum, rename key

* vocab : use metaspace prepend_scheme in detokenize, add TODOs for other tokenizers

* vocab : use constant for metaspace escaped space

---------

Co-authored-by: Martin Cimmino <martin.cimmino@domyn.com>
2026-10-22 13:34:47 +02:00
tkran2 d88040c704 vocab : add clean_spaces GGUF key (#30940) 2026-10-22 09:58:08 +02:00
alexey-rsiandSigbjørn Skjæret a37cc46104 models : add Flint model (FlintForCausalLM) (#30928)
* Add Flint model (FlintForCausalLM). Assisted by Codex

* cleanup

* one more

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-21 23:26:22 +02:00
Maphist0andSigbjørn Skjæret 7775ffb7e4 model: Add support for SenseNova U1 model (#28919)
* model : add SenseNova U1 understanding support

Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.

- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
  tensors and concatenating temporal and spatial Q/K normalization
  weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
  metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.

Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.

* conversion : treat one-expert SenseNova U1 configs as dense

* mtmd : add SenseNova U1 vision support

Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.

The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.

* mtmd : fix SenseNova U1 vision token layout

* model : clean up SenseNova U1 RoPE metadata

Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.

Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].

* mtmd : simplify SenseNova U1 projector integration

Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.

Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.

* mtmd : fix SenseNova U1 image positions and attention

* fix spacing

* mtmd : restore generic test input chunks

* mtmd : restore SenseNova U1 integration after upstream merge

* Update conversion/sensenova_u1.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/sensenova-u1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* model : remove SenseNova U1 BF16 activation casts

Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.

Assisted-by: Codex

* Update src/models/sensenova-u1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-21 21:54:16 +02:00
7e3eeee048 model : add H2O-Lightning-4B decision model (#30361)
* model : add H2O-Lightning-4B decision model

Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b
(Qwen3.5-4B with a merged decision LoRA).

None of the existing types match this model. It answers with " A"
rather than "A", uses up to 255 labels, and was trained on Python
json.dumps text, so floats like 12.0 must render as 12.0.

The converter writes the decision metadata and the systemone template
from the model's serve_config.json and refuses a config that doesn't
match. Tests skip unless the tiny test model is available locally.

Assisted-by: Claude Opus 5.5

* rm redundant code

* put labels to gguf

---------

Co-authored-by: Jonathan C. McKinney <jon.mckinney@h2o.ai>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-20 18:54:27 +02:00
Tarek DakhranandMichael DM 36140dbee2 mtmd : read LFM2 tiling params from GGUF metadata (#30840)
* mtmd : read LFM2 tiling params from GGUF metadata

Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from
processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp
with the previous constants as defaults. do_image_splitting=false maps to
min_tiles=max_tiles=1 (no tiling).

Rebased version of #25524, original implementation by Michael DM (@michael-dm).

Co-authored-by: Michael DM <usable_siren180@simplelogin.com>

* fix test

---------

Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
2026-10-20 16:05:30 +02:00
Kashif RasulandSigbjørn Skjæret 4c3c5810f8 hybrid dna tokenizer fixes (#30825)
* vocab : support Carbon-14B HybridDNA tokenizer

Carbon-14B auto-tags untagged DNA runs and uses N-padded tail tokens
(tail=pad_n). Also strip whitespace inside <dna> tags and handle a stray
</dna> like the HF tokenizer.

* convert : fix special tokens and merges for older Carbon models

Carbon-500M/3B/8B don't register the Qwen added tokens and ship no
tokenizer.json, so take the token types, merges and special ids from the
base tokenizer.

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* address review comments

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
2026-10-19 23:01:40 +02:00
fzowlandSigbjørn Skjæret 412a363c65 qwen3 : bidirectional attention and dense_2 output projection (#30607)
* qwen3 : bidirectional attention and dense_2 output projection

Assisted-by: Grok

* convert : read use_bidirectional_attention in base

Assisted-by: Grok

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-19 09:29:02 +03:00
Xue ZhenliangandSigbjørn Skjæret 62318cd57d model : add Kolibri-1 (Aleph-Alpha) support (#30673)
* model : add Kolibri-1 (Aleph-Alpha) support

Kolibri-1 is a bilingual German/English MoE reasoning model with 78B
parameters and 3.46B active per token: 50 layers, 384 routed experts plus
one shared expert.

Attention interleaves four sliding layers at a 513 window with one full
layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE.
GQA runs 48 query heads against 4 KV heads at head_dim 128 with a
QK-RMSNorm, and every block carries four RMSNorms, so the attention and the
MLP output are normalized before they rejoin the residual stream. Routing
picks the experts from logits + expert_bias and weights them with the
sigmoid of the unbiased logits, without renorm.

The converter reuses AfmoeModel for the per-expert checkpoint layout and
the expert selection bias, and writes the shared expert and the per layer
sliding window pattern. The released FP8 checkpoint needs no special code,
it goes through the generic weight_scale_inv path. Three tensor names are
added to the mapping (post_attn_norm, post_ffn_norm,
moe.router.expert_bias) and the 128k vocabulary registers the qwen2
pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA
and the sliding window pattern.

Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash)

* Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Add fusion records for kolibri1

* Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-17 13:00:17 +02:00
Daniel Han 01fc3aae84 model: add Beam (Reflection) (#30670) 2026-10-17 00:07:55 +02:00
Gabe Goodhart 23f7aabc16 model : Granite Switch Decision (#30676)
* feat(convert): Add decision.chat to support decision models that can also chat

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(convert): Add conversion for Granite Switch as decision model

The template follows from the usage sample:
https://huggingface.co/ibm-granite/granite-switch-4.1-8b#guardian-and-decision-side-by-side

Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(common): Add support for chat+decision models in common

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(common): Add granite switch decision model enum

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(server): Plumb through decision_chat bool to support decision+chat models

Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(server): Add Granite Switch decision support

Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Update server README for granite switch decisions / chat+decisions

Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* Revert "feat(convert): Add decision.chat to support decision models that can also chat"

This reverts commit 952ef567647e53ac9beda7a1f79d6927240d068d.

* Revert "feat(common): Add support for chat+decision models in common"

This reverts commit 93ed782e3a8b2fd5968732c6525c7ad29ad4bea2.

* Revert "feat(server): Plumb through decision_chat bool to support decision+chat models"

This reverts commit 97e9af60ddaccbd82f1eb0a488adff43b16ee095.

* fix: If Granite Switch supports decisions, it always supports text too

Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Remove reference to decision.chat in server README

Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-16 20:00:41 +02:00
Mikhail Podvitskii 9a16f1dbe4 model : MTP support for Mellum2.1 (#30612)
* model : MTP support for Mellum

* model : inline rope construction in Mellum
2026-10-16 10:20:03 +02:00
Aaron Teo b84ea5cf5e convert : speed up model endian conversion and add more datatype support (#30647)
* convert: speed up model endian conversion

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* convert: explicitly state noop

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* convert: add missing data type support

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-16 10:18:29 +02:00
Sh1m13 495c5805ce gguf-py : add numpy Q1_0 and Q2_0 de/quantization support (#30630)
* gguf-py : add numpy Q1_0 and Q2_0 de/quantization support

* cont: added q1_0 / q2_0 to the test list removing the small test

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:26:14 +02:00
Xuan-Son Nguyen 2851b3bcbf mtmd: add mtmd_audio_stream + support voxtral-realtime (#30629)
* mtmd: add mtmd_audio_stream + support voxtral-realtime

* address review
2026-10-16 01:46:42 +02:00
Matt Rutkowski e287bd8729 convert : fix regression in mapping of Granite speech tensors (#30628)
Signed-off-by: Matt Rutkowski <mrutkows@us.ibm.com>
2026-10-16 00:31:14 +02:00
ba3c05f71d model: add Muse Spark Open (text, vision and audio) (#30616)
* model: add Muse Spark Open (text, vision and audio)

Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.

Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.

The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Muse Spark: drop the redundant forward expand

* Muse Spark: drop the redundant forward expand and tighten the comments

* Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-graph.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 23:27:12 +02:00
Kartik Gulia cd0a5b6a18 models : add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support (#30520)
* add support

* remove unneeded bos fix

* add video flag

* fix flake error

* remove video-specific budgeting

* convert : fix lilicorr-dflash bid clash

* spec : enforce lilicorr n_tokens condition

* spec : fix lilicorr walk

[no release]
2026-10-15 16:34:56 +03:00