Commit Graph
278 Commits
Author SHA1 Message Date
Xuan-Son Nguyen 2851b3bcbf mtmd: add mtmd_audio_stream + support voxtral-realtime (#30629)
* mtmd: add mtmd_audio_stream + support voxtral-realtime

* address review
2026-10-16 01:46:42 +02:00
Said Taghadouini 2f729cfa4c mtmd: lightonocr does not require longest edge to always be 1540, native resolution is the default (#30600)
Assisted-by: Codex
2026-10-15 23:53:02 +02:00
ba3c05f71d model: add Muse Spark Open (text, vision and audio) (#30616)
* model: add Muse Spark Open (text, vision and audio)

Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.

Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.

The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Muse Spark: drop the redundant forward expand

* Muse Spark: drop the redundant forward expand and tighten the comments

* Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-graph.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 23:27:12 +02:00
Kartik Gulia cd0a5b6a18 models : add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support (#30520)
* add support

* remove unneeded bos fix

* add video flag

* fix flake error

* remove video-specific budgeting

* convert : fix lilicorr-dflash bid clash

* spec : enforce lilicorr n_tokens condition

* spec : fix lilicorr walk

[no release]
2026-10-15 16:34:56 +03:00
69f201a205 model : support MiniCPM-V 4.7 (#29416)
* mtmd : add MiniCPM-V 4.7 support

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* model : allow mrope time from an extra position slot

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* Update conversion/minicpm.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Slim down comments

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix for "do not hand-wrap comments"

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* fix ci

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* rm 3d repo for pr one

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

* gguf: add rope.section_order metadata

* fix comments

* handle grid layout

* allow compat

---------

Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-10 20:41:59 +02:00
Xuan-Son Nguyen 0fd868cbca mtmd: add build_inp_attn_mask (#30259) 2026-10-10 11:22:12 +02:00
Tarek Dakhran a657f7e981 model : add LiquidAI/d1-omni-600M decision model (#30114)
* model : add LiquidAI/d1-omni-600M decision model

Assisted-by: Claude Opus 5.5

* mtmd : keep conformer GLU sigmoid on CUDA

Assisted-by: Claude Opus 5.5

* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal

Assisted-by: Claude Opus 5.5

* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files

Assisted-by: Claude Opus 5.5
2026-10-08 02:07:49 +02:00
Tarek Dakhran 88dcc460d6 model : add LiquidAI/d1-3B decision model (#30110)
* model : add LiquidAI/d1-3b decision model

mtmd : read LFM2 image resize algo from GGUF

Assisted-by: Claude Opus 5.5

* common : rename decision type d1 to lfm2-d1

Assisted-by: Claude Opus 5.5
2026-10-07 22:04:39 +02:00
Terrence Zhao 50a6c5cf7c mtmd: add cohere2 vision support (#30062)
* cohere2 vision model

* address comments

* remove redundant mapping

* follow existing patterns

* fused linear_1
2026-10-07 20:07:50 +02:00
Xuan-Son Nguyen da263e7275 models: support pplx-decider (#30044) 2026-10-06 16:07:46 +02:00
Adrien Gallouët 836d57176d mtmd : fix deprecated strdup warning on Windows (#29863)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-03 19:47:54 +02:00
Xuan-Son Nguyen 68e79bd8cd skill: note about model-specific CLI arguments + testings (#29808)
* skill: note about adding model-specific CLI arguments

* add testing instructions
2026-10-01 19:53:23 +02:00
Xuan-Son Nguyen 552f18f912 mtmd: cap max_image to ubatch for non_causal models (#29773) 2026-10-01 11:55:11 +02:00
Pascal b04642061d cli: exit on stdin EOF and drop the console wide Ctrl+C broadcast (#29722)
* cli: exit on stdin EOF and drop the console wide Ctrl+C broadcast

On Windows the simple input reader sends CTRL_C_EVENT to every process
attached to the console when stdin reaches EOF, killing unrelated
processes such as a supervising agent. The CLI only stopped on EOF
because of that self inflicted SIGINT; on POSIX, and with the advanced
reader, it spins forever printing prompts.

Drop the broadcast so both platforms just return an empty read, and
treat an empty read as EOF in the chat loop and the model selection,
since a submitted line always ends with a newline.

* cli: keep the newline of a trailing "/" and stop mtmd-cli on EOF

A lone "/" came back as an empty read and was taken for EOF, and
mtmd-cli only stopped on EOF through the removed broadcast.
2026-09-30 16:24:46 +02:00
649dcb1036 add GLM-5.3-Flash (GLM5-Next) support (#27773)
* Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx

* Add initial MTP support

* Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements.

* Review driven changes, remove env vars, protect tensors

* Strip MTP for initial PR

* Clean up after mtp strip

* Clean up after mtp strip

* Update speculative.cpp

* Update llama-context.h

* Clean up after mtp strip

* Fix tokenizer ignore merges

* Improve quantization protection selection

* Refactor mhc helpers, graph base

* Lint Fixes

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Skip glm5-next in model saver, fix CRLF

* Skip glm5-next in sweep

* Remove T4 fallback

* Review cleanup

* Review suggestions

* Defer separate MTP gguf handling to MTP PR, drop filter

* Repad n_head_kv

* kpool init apply

* Order by descending score

* Drop guard

* read kpool from hparams, clarify kpool cache flags, remove kpool_build_state(nullptr)

* Add glm5-next support to model saver and add arch test fixture

* Review cleanup

* Kpool pooled caching clarify

* Add multi stream support

* Finish Rebase

* Sparse FA fir DSA prefill

* Const

* Update llama-model.cpp to fix rebase error

* gguf-py : merge tensor map entries for HC tensors

* model : use build_gdn_l2_norm in GLM5_NEXT implementation

* chore : remove trailing whitespace

* model : use new OP precision setting API in GLM5_NEXT implementation

* mtmd : use ggml_swiglu_clamp in GLM5V and apply the image token limit

The two clamps around swiglu_split are what ggml_swiglu_clamp already does,
so the clamp bounds collapse back to one value. GLM5V also never called
set_limit_image_tokens(), so --image-max-tokens had no effect.

Assisted-by: Claude Opus 5
(cherry picked from commit 46d18e12d422be4cc04a70e4a9a9e0168bb3d5b7)

* llama : keep the GLM5-Next k-pool layout across ubatches

The layout was rebuilt from a full cell scan on every ubatch. Pools are fixed
by the positions relative to the sequence's first one, so the layout now lives
on the memory and a ubatch only appends to it.

A sequence edit no longer stales every pooled key either, only the ones at or
after the edited position, which makes a tail seq_rm free. The pooling subgraph
is built unconditionally so the graph shape no longer changes every kpool
tokens, and the pool axis is folded into rows before soft_max, which otherwise
exceeds the CUDA gridDim.y limit past n_kv 262144.

Assisted-by: Claude Opus 5
(cherry picked from commit 5d1c40b93e17fddbf73b785efe43e0d02ccb3977)

* model : write the GLM5-Next recurrent rollback checkpoints

The conv state and the delta net state were only written to the live row, so a
rollback restored whatever the checkpoint rows happened to hold. Take the same
route as kimi-k3: build_recurrent_attn for the state, and write all K_rs conv
groups. That also drops a state view that assumed contiguous rows.

Enroll the arch in test-recurrent-state-rollback, which catches this under its
garbage-filled cache pass.

Assisted-by: Claude Opus 5
(cherry picked from commit 5ace37e86d5d448e83ef5dde5632c748185b18cd)

* llama: fix PR #27773 test-save-load-state restore failure

Clear the attention and indexer cache data after a failed hybrid state restore so restored NaNs cannot affect a later sequence.

Assisted-by: Codex

* llama: fix PR #27773 gpu-rocm graph reallocation

Reserve the full GLM5-Next pool capacity and dirty pool count. The gpu-rocm Test step aborts when n_new grows while the graph node count stays fixed; CUDA, Vulkan, Metal, and WebGPU checks report the same error.

Assisted-by: Codex

* llama : fix GLM5-Next k-pool layout staleness after edits and shared teardown

Two defects in the cross-ubatch k-pool layout added by the k-pool commit:

1. Wrong results. An edited sequence only rebuilt its pool layout when its cell
   count changed, so if the first ubatch after an edit added back exactly as many
   cells as were removed, the stale position-to-cell list survived. With a unified
   cache and more than one sequence, where another sequence takes the freed cells,
   the reused layout points at the wrong cells (CPU: large logit drift, CUDA: NaN).
   Rebuild whenever the sequence is stale, not only on a size mismatch.

2. Slowdown. "shared" mode was assumed to end only with an edit that forces a
   rebuild, but sharing also ends when the other sequence is removed. The survivor
   kept shared = true, pinning cache_safe off and re-pooling every pool on every
   ubatch (server trigger: n>1 completions with -kvu, via the seq_cp in
   copy_state_to). In seq_rm, if the layout has shared cells, stale every sequence
   so one rebuild re-derives sharing and cache_safe returns to 1.

Assisted-by: Claude Opus 5

* llama : fix build_attn_mha stream stride for non-contiguous q

build_attn_mha split the batch into streams with a stream stride of
q->nb[3]/n_stream. That only equals one stream's span, (ne[2]/n_stream)*nb[2],
when q is contiguous. GLM5-Next is nope-only, so it does not concat a rope part
and passes the permuted q_absorbed straight in, where nb[3] != ne[2]*nb[2]; the
stride was then n_head times too large and every stream s >= 1 read another
head's queries. Split-KV (-np N without --kv-unified) multi-stream prefill was
wrong for every stream past the first. Unified KV and decode were unaffected
(n_stream == 1, and decode takes the gather path). Other MLA models concat rope
so q is contiguous and the computed value is unchanged for them.

Compute the stride from the token dimension, which is identical for a
contiguous q.

Assisted-by: Claude Opus 5

* llama : re-derive GLM5-Next k-pool sharing on state_read/state_drop

The shared-cell teardown added to seq_rm (stale every sequence when the layout
has shared cells, so a survivor does not keep shared = true and pin cache_safe
off) was missing from the other paths that can free shared cells: state_read
and state_drop staled only the one sequence. Apply the same re-derivation there
and correct the comment that claimed sharing ends only via an edit or seq_rm.

Assisted-by: Claude Opus 5

* quant : drop duplicate GLM5-Next hc_ filter

The hc_ name filter was listed twice in the GLM5_NEXT protection block.

Assisted-by: Claude Opus 5

* glm5-next: scope K-pool cache access to indexed operations

* glm5-next: keep K-pool access in hybrid index memory

* glm5-next: keep mHC graph builders model-local

* glm5-next: mark only touched pools per ubatch

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
2026-09-30 14:20:32 +08:00
Pascal 526c43b8f7 mtmd: fix GCC 15 stringop-overflow in decode_embd_batch (#29607) 2026-09-29 01:21:37 +02:00
Pascal 57b557cb95 models: pad on the left with ggml_pad_ext (#29567)
* models: pad on the left with ggml_pad_ext

The Parakeet, LFM2-Audio, Granite Speech and Gemma 4 audio encoders
build a left padding as a right pad followed by a roll, and DFlash2
concatenates a zero filled block in front of the previous tokens.
ggml_pad_ext does both in one node now that every backend supports a
left padding. The Gemma 4 audio embeddings are bit identical.

* models: skip the DFlash2 taps that only read padding

A tap at or past block_size shifts every row out of the block, so its
term is zero. The loop runs min(kernel_size, block_size) taps.
2026-09-28 20:56:15 +02:00
Xuan-Son Nguyen f1ea206218 batch: migrate speculative, mtmd and server to batch_ext (#29385)
* adapt common

* add common_batch

* wip

* wip: spec

* cont

* common_speculative_process

* server_batch to use common_batch

* rm some stale calls

Assisted-by: Claude Fable 5.1

* migrate mtmd

* handle imrope, handle return val of add()/add_embd()

* add spec zeros vector

* add warning on zero fill path
2026-09-28 19:52:45 +02:00
Sarah Wu 0c6a6a7ce5 Enables Windows ARM64 build with MSVC cl.exe (#28362)
* can reproduce the issue vlad sees

* fix fma issue

* drop volatile

* fix volatile runtime task

* add arm flag if needed

* fix hsum compile error

* fix syntax in quants

* strengthen sve probing

* make the syntax fixes one liners

* remove debug code

* formatting

* remove macro for float

* drive down gcc instruction count

* support armec

* fix CI comments

address CI comments

fix cross compile issue

remove warning

fix style and fix fma probing

fix style

* add documentation

* update documentation
2026-09-28 10:07:27 +02:00
Yuri Khrustalev fcc891545b mtmd: fix mel preprocessor in LFM2 audio (#29403)
which resulted in different greedy transcripts for 4.5% of English and 6.5% of Japanese
test utterances. In Japanese, some differences changed entire words.

This change:

* uses `log(x + 2^-24)` instead of clamping to the log floor
* uses a symmetric Hann window, equivalent to `torch.hann_window(periodic=False)`
* adds the normalization epsilon to the standard deviation instead of inside the square root

Only the `lfm2a` preprocessor opts into these behaviors. Other audio preprocessors are unchanged.

Tested on top of 84e76d8 using `llama-server` with CUDA and `temperature=0`, compared against
http://github.com/Liquid4All/liquid-audio fp32.

Test set:

* 200 LibriSpeech `test-clean` utterances (EN)
* 200 Common Voice `ja` test utterances (JP)
* identical 16 kHz audio passed to both implementations

| Greedy transcript identical to `liquid-audio` | Without fix |    With fix |
| --------------------------------------------- | ----------: | ----------: |
| EN F16                                        |     191/200 | 200/200 |
| JP F32                                        |     187/200 | 200/200 |
| JP F16                                        |     187/200 | 199/200 |

The remaining JP F16 difference is a comma and matches the reference implementation's own bf16
output.

Mel relative L2 error versus `liquid-audio`:

* EN: 3.2% -> ~2e-6 median
* JP: 3.9% -> ~2e-6 median
2026-09-25 18:52:44 +02:00
Tobyandaetherbird f830688e91 model : add Ling 3.0 VL support (#29151)
* model : fold Ling 3.0 VL into the BailingMoeV3 architecture

Assisted-by: Scout

* model : keep shared NORM rope list intact when gating bailingmoe3 on mrope sections

---------

Co-authored-by: aetherbird <aetherbird@users.noreply.github.com>
2026-09-24 08:57:31 +02:00
Xuan-Son Nguyen 73c941b111 mtmd: add various sanity checks (#29276) 2026-09-22 17:53:05 +02:00
Alexey Kopytko 5b335f413e ggml : check for allocation failures to prevent crashes (#28149)
* ggml : check for allocation failures to prevent crashes

* wording
2026-09-18 16:58:25 +03:00
Daniel Bevenius f3a184b153 cmake : remove precompiled headers (#28892)
This commit removes the precompiled headers that I added in Commit
3bcfeb700  ("cmake : add PCH and unity build to improve build times
(#28091)").

The motivation for this is that this looked good when developing this
but has caused multiple issues that I had taken into consideration and
we have decided to remove it and only keep the unity builds from the
above commit.

Refs: https://github.com/ggml-org/llama.cpp/pull/28882#issuecomment-5662272126
2026-09-14 16:07:28 +02:00
Daniel Bevenius 3bcfeb700f cmake : add PCH and unity build to improve build times (#28091)
* scripts : add initial profiling script (wip)

* src : add precompile headers (PCH) for models.h

* common : add common.h as PCH

* ggml : add PCH for ggml-impl.h

* mtmd : use PCH for models.h

* scripts : add script to build with Server/Tools/Tests

* server : add PCH for common.h

* docs: add profiling progress notes (wip)

* ggml : add exclude for GCC + SVE on ARM

Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33393906061/job/99493756214?pr=28091

* ggml : attempt to fix use of std::hardware_destructive_inference_size

Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33396221677/job/99501265689?pr=28091

* squash! ggml : attempt to fix use of std::hardware_destructive_inference_size

Add a version check for GCC 12 to conditionally apply the `-Winterference-size`
pragma.

* editorconfig : exclude profiling reports dir

This directory will not be included in the merge later and this commit
can be ignore at that point. Just fixing to keep CI happy.

* ggml : skip PCH for gcc on non-x86 architectures

* tests : add PCH for peg-parser/tests.h

There are 7 peg-parser tests that can share one PCH instead of then each
parsing the full tests.h.

* common : add PCH for chat.h

* docs : update linux build profiling full results

Just updating after a number of PCH additions. These are not exact
figures and will vary a bit from run to run, but they give a general idea
of the performance impact of PCH.

* cmake : introduce unity build for models

This commit introduces a unity build for the models to improve
compilation time.

The improvements were roughly the following:
```console
+------------------------+-----+------------+------------+------------+
| Build                  | TUs | Frontend   | Backend    | Total      |
+------------------------+-----+------------+------------+------------+
| Full,    master        | 396 |   811.0 s  |   692.2 s  | 1,503.2 s  |
| Full,    with PCH      | 405 |   380.0 s  |   664.7 s  | 1,044.7 s  |
| Full,    with PCH + UB | 264 |   357.7 s  |   635.7 s  |   993.4 s  |
+------------------------+-----+------------+------------+------------+

TU   = Translation Unit.
Full = includes Server, Tools, and Tests.
PCH  = precompiled headers.
UB   = unity build for models.
```

* docs : update linux profiling table with unitiy build results

* docs : update mac profiling results to include unity build [no ci]

* docs: remove profiling reports

* scripts : merge build profile scripts into one script

I was lazy before and just copied the first script to enable Tests,
Server, and Tools. This now merges them into a single script.

* Revert "editorconfig : exclude profiling reports dir" [no ci]

This reverts commit 2922a12118a0730d2f7632bcba265b44a0856c59.

* src : rename ggml_view_2d_slice to gemma3n_view_2d_slice

This is to be consistent with the rename in gemma4.cpp which was
required to avoid a name clash.

* cmake : add build profile script for windows [no ci]

This commit adds a port of the scripts/build-profile.sh script to
windows powershell.

This was developed on Windows on ARM but should work on X64 as well but
needs to be tested there as well.
2026-09-11 13:01:29 +02:00
Xuan-Son Nguyen 6de9cdb26b mtmd: propagate video ID to bitmap (#28601) 2026-09-09 10:37:36 +02:00
Pepper Gray ca86fb222e llama : add missing headers (#28566)
* fix compile-error: add missing header

Bug: #28557
Signed-off-by: Pepper Gray <hello@peppergray.xyz>

* fix compile-error: add missing header

Bug: #28559
Signed-off-by: Pepper Gray <hello@peppergray.xyz>

* fix compile-error: add missing header

Bug: #28560
Signed-off-by: Pepper Gray <hello@peppergray.xyz>

* fix compile-error: add missing header

Bug: #28561
Signed-off-by: Pepper Gray <hello@peppergray.xyz>

* fix compile-error: add missing header

Bug: #28562
Signed-off-by: Pepper Gray <hello@peppergray.xyz>

* fix compile-error: add missing header

Bug: #28564
Signed-off-by: Pepper Gray <hello@peppergray.xyz>

---------

Signed-off-by: Pepper Gray <hello@peppergray.xyz>
2026-09-08 12:59:53 +02:00
Georgi Gerganov 5a6caa05fc ggml : update ggml_prec specification (#26675)
* ggml : update ggml_prec specification

[no ci]

* cont : add GGML_PREC_BF16

* cont : rework API

* cont : use new API

* cont : swap arg order

* cont : support for MUL_MAT_ID

* cont : fix accidental remove of "break;"

* cont : return bools, add doc TAG_GGML_PREC, clean-up

* cont : add search tag

* cont : ws
2026-09-08 09:06:24 +03:00
Xuan-Son Nguyen 163a40796f model, mtmd: fix gemma4 vision handling (#28335)
* model, mtmd: fix gemma4 vision handling

* nits
2026-09-04 12:23:27 +02:00
Xuan-Son Nguyen de8656bd94 mtmd: propagate const to preproc class (#28310) 2026-09-03 12:57:10 +02:00
Mads Marquart f45576aa86 mtmd : add const in various places (#28307)
* mtmd : mark context as const in more methods

Mark `mtmd_context` as `const` in:
- mtmd_bitmap_init_lazy
- mtmd_tokenize
- mtmd_tokenize_from_parts
- mtmd_helper_support_video
- mtmd_helper_bitmap_init_from_file
- mtmd_helper_bitmap_init_from_buf
- mtmd_helper_video_init
- mtmd_helper_video_init_from_buf
- mtmd_helper_model_can_chat

The tokenization functions in particular are useful to have marked
`const`, as that allows more easily telling the compiler that we can
safely tokenize from multiple threads (`mtmd_tokenize` is already
documented as thread-safe, this just reifies that in the signature).

* mtmd : mark tokenization input pointer as const

Mark the `bitmaps` and `parts` pointers in `mtmd_tokenize` and
`mtmd_tokenize_from_parts` as `const`. This allows more easily calling
these with immutable arrays / vectors.

* mtmd : mark llama_context as const in mtmd_helper_model_can_chat
2026-09-03 12:12:49 +02:00
Xuan-Son Nguyen 67a17c17ca mtmd: fix idefics3 preproc (#28273) 2026-09-03 01:00:57 +02:00
Xuan-Son Nguyen 7339054744 mtmd: add mtmd_tokenize_from_parts() (#28250)
* add mtmd_tokenize_from_parts

* use it in mtmd-cli

* move add_special to call level
2026-09-02 21:20:10 +02:00
Xuan-Son Nguyen 9400c8946e model: correctly support input vision for deepseek4 (#28154)
* model: correctly support input vision for deepseek4

* nits
2026-09-02 19:14:46 +02:00
Xuan-Son Nguyen 7798007a29 mtmd: support DeepSeek-V4-Flash-Vision-Exp (#28133)
* mtmd: support DeepSeek-V4-Flash-Vision-Exp

* handle min/max token counts from CLI

* rm debugging

* use GGML_ROPE_TYPE_VISION

* nits

* apply review comments

* correct token count
2026-09-02 16:43:43 +02:00
Pascal 0f3a71be15 mtmd: Fix Qwen3-tts-0.6b (#28231)
* mtmd: load the qwen3-tts code predictor proj_in as optional

The talker and the code predictor share the hidden size on the 0.6B
checkpoints, so the reference builds no small_to_mtp_projection and
the conversion emits no tensor for it. The graph already falls back
to identity when the weight is missing, the loader now agrees.

* mtmd: keep the qwen3-tts code predictor ffn_down in F32

The code predictor carries a massive activation: its layer 2 FFN
intermediate peaks around 1.5e5, well past the 65504 ceiling of F16.
mul_mat casts its input to the weight type, so an F16 ffn_down turns
that peak into inf, the residual follows, and the next rms_norm yields
NaN. Reference forward in float32 gives 145109 against 145396 measured
in the graph.
2026-09-02 12:46:16 +02:00
Xuan-Son Nguyen f29551215b args: add --video-* CLI arguments (#24318)
* args: add --video-* CLI arguments

* gen docs

* nits

* add mtmd_helper_init_opt
2026-08-27 12:11:12 +02:00
Xuan-Son Nguyenandrkfg 160c6b0bdd mtmd: video: fix moov atom at the end of file (#27596)
* mtmd: video: fix moov at the end of file

Co-authored-by: rkfg <rkfg@rkfg.me>

* fix SIGPIPE

* windows: handle broken pipe case

---------

Co-authored-by: rkfg <rkfg@rkfg.me>
2026-08-24 09:59:04 +02:00
Xuan-Son Nguyen 56db501e73 mtmd: use pillow-accurate algo, correct resize_algo for all models (#27594)
* mtmd: use pillow-accurate resize algo, correct resize_algo for all models

* speed optimization
2026-08-23 18:35:41 +02:00
Xuan-Son Nguyen b21e4de745 mtmd: use ggml_rope_set_offset (#27521)
* mtmd: use ggml_rope_set_offset

* add comment
2026-08-22 16:33:47 +02:00
Xuan-Son NguyenandSigbjørn Skjæret 54ee5ee643 mtmd: support dots3-note vision+audio (#27524)
* text: conversion

* init impl

* mtmd: conversion

* impl mtmd cpp

* Update gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-08-22 10:35:50 +02:00
Xuan-Son Nguyen d775b8967a mtmd: support webp via ffmpeg (#27520) 2026-08-22 01:38:05 +02:00
John-Henry LimandXuan Son Nguyen 681c29d36a mtmd: add --mmproj-device argument (#23255)
* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var

* feat: load mmproj device backend immediately, add -mmdev shortflag

* fix: its a pointer now get the name

* clean up

* gen docs

* nits

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-20 18:45:37 +02:00
Xuan-Son Nguyen 95c409c136 mtmd: add mtmd_bitmap_set_mergeable (#27348) 2026-08-19 13:48:22 +02:00
Saba Fallah 9731ad3f29 mtmd: deepseek-ocr SAM ggml_conv_2d with the im2col kept in F32 (#26727) 2026-08-18 19:05:19 +02:00
BlackFoilandXuan Son Nguyen 7acdbb1f19 mtmd: fix LFM2 image tiling threshold (#27057)
* mtmd: fix LFM2 image tiling threshold

* refactor testing

* fix

* fix on windows

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-18 11:11:19 +02:00
Georgi Gerganov 27e345b574 build : fix xcframework + cmake clean-up (#27304)
* xcframework : fix build

* mtmd : remove unused include path

* vendor : use vendor::hash alias target in cmake

CMake reserves "::" in target names for imported/alias targets, so the real
target keeps the name vendor-hash and a vendor::hash ALIAS target is added.
Consumers (mtmd, llama-gguf-hash) now link against the namespaced alias.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* vendor : add cmake targets for all vendored libs with vendor:: aliases

Add INTERFACE targets for the header-only vendor libs (miniaudio, nlohmann,
sheredom, stb) and ALIAS targets named vendor::<lib> for all of them,
including cpp-httplib and hash. Each exposes the vendor/ root so includes
are namespaced, e.g. <nlohmann/json.hpp>.

Consolidate the per-lib add_subdirectory calls into a single
add_subdirectory(vendor), keeping the cpp-httplib gate on LLAMA_BUILD_COMMON.
Consumers (llama-common, mtmd) now link the aliases instead of relying on
raw vendor/ include paths.

hash: consumers now include via "hash/hash.h"; the vendor/hash dir is kept
as a PRIVATE include so the synced upstream sources compile unmodified.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* readme : use foo/bar names in acknowledgements

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* ocd : fix valign
2026-08-18 11:16:51 +03:00
Xuan-Son Nguyen 533b18257b server: save processed mtmd chunks as placeholder (#27278) 2026-08-17 21:29:50 +02:00
Xuan-Son Nguyen ed1c3a20f5 mtmd: use sha256 for input hashing (#27274)
* mtmd: use sha256 for input hashing

* void conflict with boringssl
2026-08-17 21:20:12 +02:00
Tarek DakhranandXuan-Son Nguyen b75ecd1971 mtmd : skip thumbnail for non-tiled LFM2 images (#27246)
* mtmd : skip thumbnail for non-tiled LFM2 images

Matches HF behavior.

* Update tools/mtmd/mtmd-image.cpp

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
2026-08-17 19:41:32 +02:00