Commit Graph
186 Commits
Author SHA1 Message Date
PascalandGeorgi Gerganov d6be065572 llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (#31851)
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls

Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.

* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends

Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.

* tests: cover GGML_PREC_BF16 in the flash attention tests

The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.

* llama: request GGML_PREC_BF16 for the KQ multiplication

* cont : add TODO in clip.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-08 08:37:46 +01:00
Xuan-Son Nguyen a2654ec431 mtmd: move log-mel computation to cgraph (#31808)
* mtmd: move log-mel computation to cgraph

* rm stray file

* also include source in mtmd_audio_track

* rm stray file

* rm mtmd_audio_stft_op

* add test
2026-11-07 23:53:11 +01:00
Xuan-Son NguyenandPascal 9f0e5098a5 mtmd: support MageVL-Cowork model (#31819)
* mtmd: support MageVL-Cowork model

* mtmd: keep magevl rope_freqs at model level and check its shape

The rope factors are shared by every layer, so they live in clip_model
instead of a pointer copied into each clip_layer. The loader now checks
that v.rope_freqs is F32 with d_head elements and that d_head splits
into the 4:6:6 pairs the graph expects.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-07 19:36:44 +01:00
Xuan-Son Nguyen e14a779719 mtmd: enable cmake unity build (#31760) 2026-11-06 00:16:13 +01:00
Xuan-Son NguyenandPascal 5851a8e505 model, mtmd: drama-3 tts support (#31613)
* model, mtmd: drama-3 tts support

* clean up

* mtmd: drama3 helper stops at the codebook tower limit

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-04 15:15:09 +01:00
b97627967c model, mtmd: add chatterbox TTS support (#30993)
* add chatterbox TTS support

Co-authored-by: Pascal <admin@serveurperso.com>

* add built-in voice support

* rm redundant code

* clean up

* clean up 2

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* improve

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-03 23:47:11 +01:00
Prabhsimran Singh 28b812d6cb mtmd : add Nemotron 3.5 Transcribe support (#31471)
* mtmd : add Nemotron 3.5 Transcribe support

Assisted-by: Claude

* convert : fix vocab of draft models for Nemotron-H targets

Assisted-by: Claude

* convert : reject non-BF16 Transcribe checkpoints

Assisted-by: Claude

* mtmd : make speaker FIFO predictions local

Assisted-by: Claude

* mtmd : normalize the mel in encode_windows

Assisted-by: Claude

* convert : map the generated speaker kernel

Assisted-by: Claude

* mtmd : validate Nemotron Transcribe mmproj shapes

Assisted-by: Claude
2026-11-03 14:27:05 +01:00
d02b4c0d66 mtmd: Pocket tts - support for voices without voice cloning (#31429)
* feat: support PocketTTS voices with no voice cloning

The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.

Assisted-by: Claude Fable 5.1

* convert : pocket-tts voices never fail the conversion, log the drift of the state

The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.

Assisted-by: Claude Fable 5.1

* mtmd : list the predefined voices, expose them in GET /props

mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.

Assisted-by: Claude Fable 5.1

* server : an unknown voice is a bad request

A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.

Assisted-by: Claude Fable 5.1

* refactor api

* fix conversion

* small renaming

* fix lint

* tts : a model with predefined voices speaks with the first one by default

Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.

Assisted-by: Claude Fable 5.1

* mtmd : one comment per voice accessor, converter states the rule only

The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.

Assisted-by: Claude Fable 5.1

* clean up redundant code and comments

* mtmd: match the LFM3-Audio voice name case-insensitively

The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-02 18:39:51 +01:00
boshandXuan-Son Nguyen 7bd1808308 mtmd : resize Gemma 4 images without letterbox padding (#30922)
* mtmd : resize Gemma 4 images without letterbox padding

* Apply suggestion from @ngxson

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
2026-10-31 18:55:33 +01:00
Pascal cf24317bea common, mtmd: bound control vector layer indices and M-RoPE position loops (#31336)
* common: bound control vector layer indices by the model layer count

A direction tensor layer index is parsed from the tensor name with no
upper bound, and the buffer size and write offset are computed in int.
A large index overflows both, so the accumulation writes before the
start of the buffer.

Pass the model layer count to the loader, reject indices above it like
the existing zero index check, and compute the sizes in size_t.

* mtmd: fill M-RoPE positions by whole merge tiles

The position loops of the Qwen-VL, GLM, PaddleOCR and Qwen2.5-VL
encoders hardcode a 2x2 merge inside a stride of n_merge, so a patch
grid that is not a multiple of the merge writes past the positions
buffer. Iterate over whole merge tiles with n_merge inner steps, which
bounds the writes by the patch count and gives the same positions on
aligned grids.

PaddleOCR also requires a merge aligned image at graph build, as the
Qwen-VL family already does.
2026-10-28 23:15:01 +01:00
Tarek DakhranandXuan Son Nguyen d4f1b18b3a mtmd: enable video input support for LFM3 model family (#31283)
* mtmd : add LFM3-VL and LFM3-Omni video input

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* mtmd : reserve the soundtrack command to avoid a GCC 11 warning

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-28 12:23:06 +01:00
Tarek DakhranandXuan Son Nguyen 19ff817fb2 model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B (#31257)
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* convert : read the LFM3-Audio config.json as released

A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).

Assisted-by: Claude Opus 5.5

* mtmd : disable flash attention for the LFM3-Audio depthformer

Assisted-by: Claude Opus 5.5

* tts : add a voice option for LFM3-Audio and LFM3-Omni

The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.

Assisted-by: Claude Opus 5.5

* clean up template handling

* convert : detect LFM3-Audio and LFM3-Omni by model_type

Assisted-by: Claude Opus 5.5

* move banned tokens --> logits bias --> mtmd_helper

* leave mtmd-cli unchanged

* no more template_asr_system

* left over

* llama : keep the LFM3-Audio encoder convs and codec unquantized

Assisted-by: Claude Opus 5.5

* common, mtmd : shorten comments

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 18:47:14 +01:00
19cce69199 model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B (#31252)
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5

* tests : set the SWA rope base of the LFM3 test model

Assisted-by: Claude Opus 5.5

* llama : enable recurrent state rollback for LFM3

Assisted-by: Claude Opus 5.5

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>

* gguf-py : align the lfm3vl mapping comments

Assisted-by: Claude Opus 5.5

* llama : fix CRLF line endings from the review suggestions

Assisted-by: Claude Opus 5.5

* tests : add the LFM3 Metal fusion baseline

Assisted-by: Claude Opus 5.5

* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
2026-10-27 17:06:40 +01:00
Aman GuptaandDaniel Han 74bca39500 model: DSV4.1 flash (#31086)
* model : add DeepSeek-V4.1-Flash (deepseek41)

* deepseek41 : remove MXFP8, convert the Engram table to BF16

* deepseek41 : compute the compressed-KV read layers once in the cache

* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le

* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather

* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer

* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle

* deepseek41 : keep il last in build_hc_mixes

* deepseek41 : sort the top-k rows before the sparse gather

* tests : add deepseek41 to the Metal fusion baseline

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-10-25 09:43:51 +08:00
Xuan-Son Nguyen 7894e8b924 mtmd: move build_inp_temporal_merge to base class (#31065) 2026-10-24 03:38:25 +02:00
Yuri KhrustalevandXuan Son Nguyen edc353842c mtmd : lfm2 resize matches torchvision (#30943)
* mtmd : lfm2 bilinear resize matches torchvision

Lfm2VlImageProcessor and d1-omni vision.py resize uint8 images with
torchvision (antialias=True). That is Pillow's algorithm, except that the
weights are int16 with as many fractional bits as the largest weight allows,
where Pillow uses 22 bits. The Pillow port was 1-2 levels off on about 1% of
the values of the d1-omni and LFM2.5-VL crops.

RESIZE_ALGO_BILINEAR_TORCH rounds the weights of the Pillow port to that
precision and scales them back to 22 bits, which gives the output of
torchvision with the same passes, and "bilinear" maps to it for LFM2 and
d1-omni. Other models keep the Pillow algorithms.

* mtmd : lfm2 bicubic resize matches torchvision

Lfm2VlImageProcessor resizes with torchvision for the bicubic checkpoints too
(d1-3B, LFM2-VL), so "bicubic" maps to RESIZE_ALGO_BICUBIC_TORCH for LFM2: the
Pillow bicubic filter (a = -0.5, as torchvision with antialias) with the int16
weights of torchvision. The Pillow port was 1-2 levels off on about 0.25% of
the values of the d1-3B crops.

* use bitfield, add hparams setter

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-24 02:21:06 +02:00
Hans 9ae73e68f5 mtmd : add Soprano TTS support (#28607)
Add Soprano-80M and Soprano-1.1-80M (https://huggingface.co/ekwek/Soprano-1.1-80M)
as an audio generation model: a Qwen3 backbone plus a Vocos decoder that runs
as a gen-only mmproj.

- convert: detect Soprano checkpoints and export the backbone as qwen3 and the
  decoder.pth Vocos weights as the mmproj
- vocab: add the soprano pre-tokenizer (lowercase, whitespace collapse,
  per-digit split, [UNK] fallback)
- mtmd: add the Vocos graph and ISTFT, allow loading a gen-only mmproj for
  Soprano and include the gen-audio context in mtmd_get_memory_usage
- mtmd-helper: add the Soprano pipeline; the decoder runs over the whole
  utterance, so stream mode sends the audio in one chunk after flush()
- tts: document Soprano usage

Assisted-by: Codex
Assisted-by: Claude Opus 5.5
2026-10-24 01:09:50 +02:00
tc-mb a08ab75ae7 mtmd : add MiniCPM-OCR 1.0 support (#30979)
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
2026-10-22 19:22:15 +02:00
Maphist0andSigbjørn Skjæret 7775ffb7e4 model: Add support for SenseNova U1 model (#28919)
* model : add SenseNova U1 understanding support

Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.

- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
  tensors and concatenating temporal and spatial Q/K normalization
  weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
  metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.

Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.

* conversion : treat one-expert SenseNova U1 configs as dense

* mtmd : add SenseNova U1 vision support

Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.

The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.

* mtmd : fix SenseNova U1 vision token layout

* model : clean up SenseNova U1 RoPE metadata

Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.

Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].

* mtmd : simplify SenseNova U1 projector integration

Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.

Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.

* mtmd : fix SenseNova U1 image positions and attention

* fix spacing

* mtmd : restore generic test input chunks

* mtmd : restore SenseNova U1 integration after upstream merge

* Update conversion/sensenova_u1.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/sensenova-u1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* model : remove SenseNova U1 BF16 activation casts

Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.

Assisted-by: Codex

* Update src/models/sensenova-u1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-21 21:54:16 +02:00
Xuan-Son Nguyen 0a1ef309cd mtmd: add state object for mtmd_gen (#30902) 2026-10-21 12:59:54 +02:00
Xuan-Son NguyenandPascal 5c0b49a23b server : add POST /tts endpoint for TTS models (#26603)
* mtmd: add audio out stream api

* add /tts endpoint

* add docs

* wire up mtmd_helper_model_can_chat

* upload speaker_ref via form-data

* expose seed to api

* fix header for pcm

* wip

* fix penalty_last_n

* fix server_output_limits

* fix pocket-tts stream

* wip

* add server_schema for tts

* add server routing

* server: coerce seed and stream from multipart form fields

Form fields arrive as strings, so -F seed=42 and -F stream=true were
rejected by the schema. seed joins the integer fields, parsed as a long
long so a full uint32 seed fits, and stream is read as a boolean when
it comes as a string.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-21 01:19:42 +02:00
Takamichi Ohara 52ffa02745 model : support Nemotron-3(.5)-Embed (#30877) 2026-10-20 22:48:45 +02:00
Yuri KhrustalevandXuan Son Nguyen 4a45de4f2c mtmd : lfm2 tiles and layout match transformers (#30261)
* mtmd : lfm2 tiles stretch the image to the grid

Lfm2VlImageProcessor.crop_image_to_patches resizes the image straight to
(512 * rows, 512 * cols) before it cuts the tiles. llama.cpp kept the aspect
ratio and added black bars for LFM2 (only d1-omni stretched), so the tiles of
d1-3B and every LFM2-VL / LFM2.5-VL model differed from transformers unless
the image had the aspect ratio of the grid. With PAD_NONE the tiles are the
same as in transformers.

* mtmd : lfm2 layout with Python rounding in double precision

Lfm2VlImageProcessor.smart_resize rounds with Python's round (half to even)
and computes beta and the tile grid aspect ratio in double. llama.cpp used
std::round and float, so 31926 of the 16.7M sizes in [1,4096]^2 got another
thumbnail size or grid than transformers (and d1-omni vision.py), e.g.
400x600 -> 416x608 instead of 384x608 and 720x1600 -> 2x5 tiles instead of
2x4. The lfm2 smart resize and grid choice now follow the Python code and
match it on every size of that range.

* fix

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-20 18:54:46 +02:00
Xuan-Son Nguyen 5ba3b9fd3d mtmd: fix gemma 4 ffn act (#30882) 2026-10-20 18:33:32 +02:00
Tarek DakhranandMichael DM 36140dbee2 mtmd : read LFM2 tiling params from GGUF metadata (#30840)
* mtmd : read LFM2 tiling params from GGUF metadata

Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from
processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp
with the previous constants as defaults. do_image_splitting=false maps to
min_tiles=max_tiles=1 (no tiling).

Rebased version of #25524, original implementation by Michael DM (@michael-dm).

Co-authored-by: Michael DM <usable_siren180@simplelogin.com>

* fix test

---------

Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
2026-10-20 16:05:30 +02:00
bosh f711c8f303 mtmd : match the transformers Qwen-VL resize (#30736) 2026-10-20 01:14:40 +02:00
Xuan-Son Nguyen 2851b3bcbf mtmd: add mtmd_audio_stream + support voxtral-realtime (#30629)
* mtmd: add mtmd_audio_stream + support voxtral-realtime

* address review
2026-10-16 01:46:42 +02:00
ba3c05f71d model: add Muse Spark Open (text, vision and audio) (#30616)
* model: add Muse Spark Open (text, vision and audio)

Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.

Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.

The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Muse Spark: drop the redundant forward expand

* Muse Spark: drop the redundant forward expand and tighten the comments

* Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-graph.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 23:27:12 +02:00
Kartik Gulia cd0a5b6a18 models : add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support (#30520)
* add support

* remove unneeded bos fix

* add video flag

* fix flake error

* remove video-specific budgeting

* convert : fix lilicorr-dflash bid clash

* spec : enforce lilicorr n_tokens condition

* spec : fix lilicorr walk

[no release]
2026-10-15 16:34:56 +03:00
69f201a205 model : support MiniCPM-V 4.7 (#29416)
* mtmd : add MiniCPM-V 4.7 support

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* model : allow mrope time from an extra position slot

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* Update conversion/minicpm.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Slim down comments

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix for "do not hand-wrap comments"

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix ci

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* rm 3d repo for pr one

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* gguf: add rope.section_order metadata

* fix comments

* handle grid layout

* allow compat

---------

Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-10 20:41:59 +02:00
Xuan-Son Nguyen 0fd868cbca mtmd: add build_inp_attn_mask (#30259) 2026-10-10 11:22:12 +02:00
Tarek Dakhran a657f7e981 model : add LiquidAI/d1-omni-600M decision model (#30114)
* model : add LiquidAI/d1-omni-600M decision model

Assisted-by: Claude Opus 5.5

* mtmd : keep conformer GLU sigmoid on CUDA

Assisted-by: Claude Opus 5.5

* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal

Assisted-by: Claude Opus 5.5

* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files

Assisted-by: Claude Opus 5.5
2026-10-08 02:07:49 +02:00
Tarek Dakhran 88dcc460d6 model : add LiquidAI/d1-3B decision model (#30110)
* model : add LiquidAI/d1-3b decision model

mtmd : read LFM2 image resize algo from GGUF

Assisted-by: Claude Opus 5.5

* common : rename decision type d1 to lfm2-d1

Assisted-by: Claude Opus 5.5
2026-10-07 22:04:39 +02:00
Terrence Zhao 50a6c5cf7c mtmd: add cohere2 vision support (#30062)
* cohere2 vision model

* address comments

* remove redundant mapping

* follow existing patterns

* fused linear_1
2026-10-07 20:07:50 +02:00
Xuan-Son Nguyen da263e7275 models: support pplx-decider (#30044) 2026-10-06 16:07:46 +02:00
Xuan-Son Nguyen 552f18f912 mtmd: cap max_image to ubatch for non_causal models (#29773) 2026-10-01 11:55:11 +02:00
649dcb1036 add GLM-5.3-Flash (GLM5-Next) support (#27773)
* Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx

* Add initial MTP support

* Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements.

* Review driven changes, remove env vars, protect tensors

* Strip MTP for initial PR

* Clean up after mtp strip

* Clean up after mtp strip

* Update speculative.cpp

* Update llama-context.h

* Clean up after mtp strip

* Fix tokenizer ignore merges

* Improve quantization protection selection

* Refactor mhc helpers, graph base

* Lint Fixes

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Skip glm5-next in model saver, fix CRLF

* Skip glm5-next in sweep

* Remove T4 fallback

* Review cleanup

* Review suggestions

* Defer separate MTP gguf handling to MTP PR, drop filter

* Repad n_head_kv

* kpool init apply

* Order by descending score

* Drop guard

* read kpool from hparams, clarify kpool cache flags, remove kpool_build_state(nullptr)

* Add glm5-next support to model saver and add arch test fixture

* Review cleanup

* Kpool pooled caching clarify

* Add multi stream support

* Finish Rebase

* Sparse FA fir DSA prefill

* Const

* Update llama-model.cpp to fix rebase error

* gguf-py : merge tensor map entries for HC tensors

* model : use build_gdn_l2_norm in GLM5_NEXT implementation

* chore : remove trailing whitespace

* model : use new OP precision setting API in GLM5_NEXT implementation

* mtmd : use ggml_swiglu_clamp in GLM5V and apply the image token limit

The two clamps around swiglu_split are what ggml_swiglu_clamp already does,
so the clamp bounds collapse back to one value. GLM5V also never called
set_limit_image_tokens(), so --image-max-tokens had no effect.

Assisted-by: Claude Opus 5
(cherry picked from commit 46d18e12d422be4cc04a70e4a9a9e0168bb3d5b7)

* llama : keep the GLM5-Next k-pool layout across ubatches

The layout was rebuilt from a full cell scan on every ubatch. Pools are fixed
by the positions relative to the sequence's first one, so the layout now lives
on the memory and a ubatch only appends to it.

A sequence edit no longer stales every pooled key either, only the ones at or
after the edited position, which makes a tail seq_rm free. The pooling subgraph
is built unconditionally so the graph shape no longer changes every kpool
tokens, and the pool axis is folded into rows before soft_max, which otherwise
exceeds the CUDA gridDim.y limit past n_kv 262144.

Assisted-by: Claude Opus 5
(cherry picked from commit 5d1c40b93e17fddbf73b785efe43e0d02ccb3977)

* model : write the GLM5-Next recurrent rollback checkpoints

The conv state and the delta net state were only written to the live row, so a
rollback restored whatever the checkpoint rows happened to hold. Take the same
route as kimi-k3: build_recurrent_attn for the state, and write all K_rs conv
groups. That also drops a state view that assumed contiguous rows.

Enroll the arch in test-recurrent-state-rollback, which catches this under its
garbage-filled cache pass.

Assisted-by: Claude Opus 5
(cherry picked from commit 5ace37e86d5d448e83ef5dde5632c748185b18cd)

* llama: fix PR #27773 test-save-load-state restore failure

Clear the attention and indexer cache data after a failed hybrid state restore so restored NaNs cannot affect a later sequence.

Assisted-by: Codex

* llama: fix PR #27773 gpu-rocm graph reallocation

Reserve the full GLM5-Next pool capacity and dirty pool count. The gpu-rocm Test step aborts when n_new grows while the graph node count stays fixed; CUDA, Vulkan, Metal, and WebGPU checks report the same error.

Assisted-by: Codex

* llama : fix GLM5-Next k-pool layout staleness after edits and shared teardown

Two defects in the cross-ubatch k-pool layout added by the k-pool commit:

1. Wrong results. An edited sequence only rebuilt its pool layout when its cell
   count changed, so if the first ubatch after an edit added back exactly as many
   cells as were removed, the stale position-to-cell list survived. With a unified
   cache and more than one sequence, where another sequence takes the freed cells,
   the reused layout points at the wrong cells (CPU: large logit drift, CUDA: NaN).
   Rebuild whenever the sequence is stale, not only on a size mismatch.

2. Slowdown. "shared" mode was assumed to end only with an edit that forces a
   rebuild, but sharing also ends when the other sequence is removed. The survivor
   kept shared = true, pinning cache_safe off and re-pooling every pool on every
   ubatch (server trigger: n>1 completions with -kvu, via the seq_cp in
   copy_state_to). In seq_rm, if the layout has shared cells, stale every sequence
   so one rebuild re-derives sharing and cache_safe returns to 1.

Assisted-by: Claude Opus 5

* llama : fix build_attn_mha stream stride for non-contiguous q

build_attn_mha split the batch into streams with a stream stride of
q->nb[3]/n_stream. That only equals one stream's span, (ne[2]/n_stream)*nb[2],
when q is contiguous. GLM5-Next is nope-only, so it does not concat a rope part
and passes the permuted q_absorbed straight in, where nb[3] != ne[2]*nb[2]; the
stride was then n_head times too large and every stream s >= 1 read another
head's queries. Split-KV (-np N without --kv-unified) multi-stream prefill was
wrong for every stream past the first. Unified KV and decode were unaffected
(n_stream == 1, and decode takes the gather path). Other MLA models concat rope
so q is contiguous and the computed value is unchanged for them.

Compute the stride from the token dimension, which is identical for a
contiguous q.

Assisted-by: Claude Opus 5

* llama : re-derive GLM5-Next k-pool sharing on state_read/state_drop

The shared-cell teardown added to seq_rm (stale every sequence when the layout
has shared cells, so a survivor does not keep shared = true and pin cache_safe
off) was missing from the other paths that can free shared cells: state_read
and state_drop staled only the one sequence. Apply the same re-derivation there
and correct the comment that claimed sharing ends only via an edit or seq_rm.

Assisted-by: Claude Opus 5

* quant : drop duplicate GLM5-Next hc_ filter

The hc_ name filter was listed twice in the GLM5_NEXT protection block.

Assisted-by: Claude Opus 5

* glm5-next: scope K-pool cache access to indexed operations

* glm5-next: keep K-pool access in hybrid index memory

* glm5-next: keep mHC graph builders model-local

* glm5-next: mark only touched pools per ubatch

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
2026-09-30 14:20:32 +08:00
Tobyandaetherbird f830688e91 model : add Ling 3.0 VL support (#29151)
* model : fold Ling 3.0 VL into the BailingMoeV3 architecture

Assisted-by: Scout

* model : keep shared NORM rope list intact when gating bailingmoe3 on mrope sections

---------

Co-authored-by: aetherbird <aetherbird@users.noreply.github.com>
2026-09-24 08:57:31 +02:00
Xuan-Son Nguyen 73c941b111 mtmd: add various sanity checks (#29276) 2026-09-22 17:53:05 +02:00
Alexey Kopytko 5b335f413e ggml : check for allocation failures to prevent crashes (#28149)
* ggml : check for allocation failures to prevent crashes

* wording
2026-09-18 16:58:25 +03:00
Georgi Gerganov 5a6caa05fc ggml : update ggml_prec specification (#26675)
* ggml : update ggml_prec specification

[no ci]

* cont : add GGML_PREC_BF16

* cont : rework API

* cont : use new API

* cont : swap arg order

* cont : support for MUL_MAT_ID

* cont : fix accidental remove of "break;"

* cont : return bools, add doc TAG_GGML_PREC, clean-up

* cont : add search tag

* cont : ws
2026-09-08 09:06:24 +03:00
Xuan-Son Nguyen 163a40796f model, mtmd: fix gemma4 vision handling (#28335)
* model, mtmd: fix gemma4 vision handling

* nits
2026-09-04 12:23:27 +02:00
Xuan-Son Nguyen 7798007a29 mtmd: support DeepSeek-V4-Flash-Vision-Exp (#28133)
* mtmd: support DeepSeek-V4-Flash-Vision-Exp

* handle min/max token counts from CLI

* rm debugging

* use GGML_ROPE_TYPE_VISION

* nits

* apply review comments

* correct token count
2026-09-02 16:43:43 +02:00
Pascal 0f3a71be15 mtmd: Fix Qwen3-tts-0.6b (#28231)
* mtmd: load the qwen3-tts code predictor proj_in as optional

The talker and the code predictor share the hidden size on the 0.6B
checkpoints, so the reference builds no small_to_mtp_projection and
the conversion emits no tensor for it. The graph already falls back
to identity when the weight is missing, the loader now agrees.

* mtmd: keep the qwen3-tts code predictor ffn_down in F32

The code predictor carries a massive activation: its layer 2 FFN
intermediate peaks around 1.5e5, well past the 65504 ceiling of F16.
mul_mat casts its input to the weight type, so an F16 ffn_down turns
that peak into inf, the residual follows, and the next rms_norm yields
NaN. Reference forward in float32 gives 145109 against 145396 measured
in the graph.
2026-09-02 12:46:16 +02:00
Xuan-Son Nguyen 56db501e73 mtmd: use pillow-accurate algo, correct resize_algo for all models (#27594)
* mtmd: use pillow-accurate resize algo, correct resize_algo for all models

* speed optimization
2026-08-23 18:35:41 +02:00
Xuan-Son Nguyen b21e4de745 mtmd: use ggml_rope_set_offset (#27521)
* mtmd: use ggml_rope_set_offset

* add comment
2026-08-22 16:33:47 +02:00
Xuan-Son NguyenandSigbjørn Skjæret 54ee5ee643 mtmd: support dots3-note vision+audio (#27524)
* text: conversion

* init impl

* mtmd: conversion

* impl mtmd cpp

* Update gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-08-22 10:35:50 +02:00
John-Henry LimandXuan Son Nguyen 681c29d36a mtmd: add --mmproj-device argument (#23255)
* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var

* feat: load mmproj device backend immediately, add -mmdev shortflag

* fix: its a pointer now get the name

* clean up

* gen docs

* nits

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-20 18:45:37 +02:00
Xuan-Son Nguyen 27df9199d1 fix: check gguf array type before reading (#27075)
* fix: check gguf array type before reading

* update skill
2026-08-15 11:45:30 +02:00
9d57ce456c mtmd: fix Granite4 Vision image sequence assembly (#26653)
* mtmd: fix granite 4v grid assembly

(cherry picked from commit 91f82eb1b489bff0dc3f649edf2cb9e0b7f656e9)

* mtmd: fix truncation for scaled image height and width before unpad

Signed-off-by: Hemanth Battu <hbattu@ibm.com>

* mtmd: remove MTMD_DUMP_EMBD debug scaffolding

Signed-off-by: Hemanth Battu <hbattu@ibm.com>

* clean up comments, clarify about anyres_info excluded from serialization

* add_newline is now dead code

---------

Signed-off-by: Hemanth Battu <hbattu@ibm.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Hemanth Battu <hbattu@ibm.com>
2026-08-15 00:25:26 +02:00