Commit Graph
7 Commits
Author SHA1 Message Date
Pascal cfb9d4b754 mtmd: require an F32 position embedding for HunyuanVL (#32131)
HunyuanVL resamples its position embedding on the host into a float
buffer sized for F32, then reads ggml_nbytes() of the tensor into it.
The graph never uses the tensor, so any type loads, and an F64 tensor
overflows the buffer at the first image request.

Require the position embedding to be present and F32 when loading the
HunyuanVL tensors.
2026-11-12 12:15:28 +01:00
Pascal 72f4603c7c mtmd: read the MiniCPM-V projection width from ne[1] (#32132)
The MiniCPM-V resampler ends with build_mm(mm_model_proj), whose output
width is ne[1], while n_mmproj_embd() returns ne[0]. Both are equal in
real models, the projection is square. A non square projection passes
the text model n_embd check and only aborts at the encode, on the
output buffer size check.

Return ne[1], so the width matches the graph output.
2026-11-12 12:14:50 +01:00
Pascal 8c388364e1 mtmd: require the MLP projector and patch embedding tensors (#32129)
* mtmd: require the MLP projector and patch embedding tensors

The loader marks the patch embedding and the LLaVA projector tensors
as optional, while the graphs and n_mmproj_embd() dereference them
unconditionally. A GGUF without them crashes on a null tensor during
the warmup.

Load the patch embedding as required for the vision encoders that
start with it, which is all of them except GEMMA3NV, DEEPSEEKOCR and
DEEPSEEKOCR2. Load mm.0 and mm.2 as required for MLP, and mm.0, mm.1,
mm.3 and mm.4 for the Yi-type MLP_NORM variant, selected by mm.3.

The graph no longer tests mm.2 and the yasa2 assert on the patch
embedding is covered by the loader.

* mtmd: keep the Yi-type llava TODO

The Yi-type variant is still detected by the presence of mm.3, the
TODO on that detection stays.
2026-11-12 12:14:36 +01:00
Pascal 08f49598c1 mtmd: require a 2x2 spatial merge for the Qwen-VL projectors (#32130)
The Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Ling3-VL graphs build their
input with a temporal merge that assumes a 2x2 patch grid, and the
Qwen2-VL merger reshapes by 4. Any other spatial_merge_size from the
GGUF reaches a GGML_ASSERT at the first image request and aborts the
server.

Reject a spatial_merge_size other than 2 when loading the hparams.
2026-11-12 12:14:10 +01:00
Xuan-Son Nguyen 7c2653acc7 mtmd: fix string format, rename to mtmd_format (#32040) 2026-11-10 20:23:25 +01:00
Xuan-Son Nguyen 351025ec53 mtmd: add kimi K3.1 vision (#31980)
* mtmd: support kimi k3.1 vision

* fix preproc (both k2.5 and k3.1) and projector mlp

* small renaming
2026-11-10 11:28:47 +01:00
Xuan-Son Nguyen 7d7f028753 mtmd: refactoring: remove clip.h, consolidate code into mtmd (#31865) 2026-11-08 15:49:06 +01:00