* ggml-cpu: use VNNI for Q2_0 dot products
Keep the upstream AVX2 unpacking, accumulation, and fallback. Use VNNI for the integer dot product on supported x86 CPUs.
Assisted-by: Codex
* ggml-cpu: use VBMI for Q2_0 unpacking
Use multishift to unpack Q2_0 weights on CPUs with AVX-512 VBMI and VL. Retain the existing shift-and-mask unpacking on other CPUs.
Assisted-by: Codex
* Update ggml/src/ggml-cpu/arch/x86/quants.c
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* fixed support for DFlash2 for Ripple 65B
* spec: Add NoPE handling to DFlash
(cherry picked from commit 89ea8e3e77e3359d17902a2db136022d5d419d88)
* spec: Fix NoPE not being picked up by DFlash
(cherry picked from commit 43393b95fda2ce7e7d243e2650bf7b1cadf1c732)
---------
Co-authored-by: James Pickering <james_pic@hotmail.com>
* Fail tests on missing or unsupported type
* Fix output
* Remove whitespace
* Fail if no tests ran regardless of backend support
* Revert unnecessary cosmetic change
* lfm3 : put the normed tensor first in residual adds to enable norm fusion
Assisted-by: Claude Opus 5.5
* tests : update Metal fusion baseline for lfm3
Assisted-by: Claude Opus 5.5
* convert : add Inkling2 text GGUF conversion
Assisted-by: Codex
* llama : implement Inkling2 text inference
Assisted-by: Codex
* tests : cover Inkling2 text support
Assisted-by: Codex
* conversion : dequantize packed Inkling2 experts in row chunks
Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.
Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.
Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.
Assisted-by: Codex
* convert : add Inkling2 vision mmproj support
Assisted-by: Codex
* mtmd : implement Inkling2 still-image encoder
Assisted-by: Codex
* inkling2 : scope image loading
Assisted-by: Codex
* inkling2 : enable flash attention with folded bias and sinks
Assisted-by: Codex
* cuda : route per-head flash attention masks through vec
Assisted-by: Codex
* mtmd : support Inkling2 temporal frame means
Assisted-by: Codex
* style : restore shared image comment
Assisted-by: Codex
* tests : enlarge Inkling2 context fixture
Assisted-by: Codex
* inkling2 : load per-layer expert feed-forward widths
Assisted-by: Codex
* tests : add Inkling2 Metal fusion baselines
Assisted-by: Codex
* inkling2 : group and align registry entries
Assisted-by: Fable
* convert : regenerate the Inkling2 tokenizer hash with the update script
Assisted-by: Fable
* conversion : revert the print_registered_models change
Assisted-by: Fable
* conversion : sort the Inkling2 model map entries
Assisted-by: Fable
* convert : map Inkling2 vision checkpoint names in the tensor table
Assisted-by: Fable
* convert : map Inkling2 text checkpoint names in the tensor table
Assisted-by: Fable
* tests : document the Inkling2 packed expert conversion test
Assisted-by: Fable
* tests : cover Inkling2 checkpoint and transformers tensor names
Assisted-by: Fable
* llama : place Inkling2 fields and list entries by file convention
Assisted-by: Fable
* gguf-py : align Inkling2 tensor mapping comments
Assisted-by: Fable
* mtmd : place Inkling2 entries by file convention
Assisted-by: Fable
* tests : place Inkling2 cases after the existing ones
Assisted-by: Fable
* cuda : index per-head flash attention masks in the tile and mma kernels
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : project the relative bias before the gather, drop arch branches
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* convert : require Inkling2 hparams, add inkling2 pre-tokenizer
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* mtmd : cap Inkling2 image size, drop video frame averaging
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : fix CI build, Meta device assert and Windows conversion test
Assisted-by: Claude
* tests : use the default sliding window for the Inkling2 test model
Assisted-by: Claude
* simplify mtmd part
* give _v arch for vision
* remove alpha channel handling, it should be a follow up
* explicit rule about audio + vision models
* add audio support
* rm audio
* rm gguf mtmd rope base
* address review comment
* nits
* do not normalize embd from mtmd
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* common : reject out-of-range grammar_triggers type instead of aborting (#30835)
* tests : remove the invalid grammar trigger type regression test (#30835)
* vulkan: Explicitly include SPIRV-Headers
Distros that don't put the headers in a system include path need
to specify it to the compiler. So we use the appropriate CMake
target to include it.
Assisted-by: Codex
* vulkan: Add explicit headers location to workaround OSX CI runner conf
* vulkan: optimize Flash Attention cm1 for RDNA3 and RDNA4
* reduce shmem use through buffer reuse
* clean up and fixes
* limit q_in_regs to prefill
* vulkan: add flipped FA path, each subgroup owns 16 Q rows
* cleanup
* move flip shader into own file
* replace kvsep with direct load mode
* fix nvidia hang, tune AMD
* only store pv in sfsh if necessary
* fix skip condition
* nvidia tuning
* reduce occupancy on RDNA4 for large attention head sizes
* fix sparse FA fp16 underflow bug
Template soft_max_f32_parallelize_cols_single_row to keep column
elements in registers when they fit, reducing memory traffic from
5 global memory passes to 2 passes.
Assisted-by: Gemini 3.8 Flash
* vulkan: optimize matrix multiplication for narrow N
Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.
* Attempt to handle wave64
* add n=16 tile for cm1
* apply RDNA split_k heuristic to narrow_n generally
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* chat: allow tool calls inside reasoning blocks for Ministral-3 and Qwen3
Fixes#29248Fixes#20837
* chat: address review feedback for Ministral-3 reasoning end
- Remove <function= from Qwen3 (intentionally not used outside outdated coder models)
- Ministral-3: use choice between [/THINK] and peek([TOOL_CALLS]) instead of optional
* chat: add [TOOL_CALLS] to Ministral-3 thinking_end_tags
* chat: simplify Ministral-3 reasoning end choice
* chat: restore peek lookahead for Ministral-3 tool call start
* chat: keep [TOOL_CALLS] only in parser, not in thinking_end_tags
* cont : revert tool call tag removal and update tests
---------
Co-authored-by: Alen Požun <kukirin-ui@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
The thought channel is normally closed with <channel|>, but the model
may open it and go straight to a tool call without closing it. Stop the
reasoning at <channel|> OR <|tool_call>, so an unclosed thought does
not swallow the following tool call.
Fixes: #32231
Assisted-by: Qwen 3.8 27b
The qwen2vl graph passed NULL for the ffn_up/gate/down scales, so the
per-tensor NVFP4 scales loaded by the generic pass were never applied
and the model produced garbage (PPL ~1e6 on CUDA, NaN on CPU).
Pass the scales to build_ffn like qwen2 does.
Fixes#32101
* mtmd : allow custom video decoder
* also adapt to wllama requirements
* clean up
* add test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>