* vulkan: make the radix top-k tie selection deterministic
The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.
Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.
* optimize shader for AMD, Intel and Nvidia
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.
Fixes#30576
Assisted-by: GitHub Copilot
Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.
Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.
Also trim the output_config.effort test to a single level.
* Fail tests on missing or unsupported type
* Fix output
* Remove whitespace
* Fail if no tests ran regardless of backend support
* Revert unnecessary cosmetic change
* lfm3 : put the normed tensor first in residual adds to enable norm fusion
Assisted-by: Claude Opus 5.5
* tests : update Metal fusion baseline for lfm3
Assisted-by: Claude Opus 5.5
* convert : add Inkling2 text GGUF conversion
Assisted-by: Codex
* llama : implement Inkling2 text inference
Assisted-by: Codex
* tests : cover Inkling2 text support
Assisted-by: Codex
* conversion : dequantize packed Inkling2 experts in row chunks
Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.
Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.
Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.
Assisted-by: Codex
* convert : add Inkling2 vision mmproj support
Assisted-by: Codex
* mtmd : implement Inkling2 still-image encoder
Assisted-by: Codex
* inkling2 : scope image loading
Assisted-by: Codex
* inkling2 : enable flash attention with folded bias and sinks
Assisted-by: Codex
* cuda : route per-head flash attention masks through vec
Assisted-by: Codex
* mtmd : support Inkling2 temporal frame means
Assisted-by: Codex
* style : restore shared image comment
Assisted-by: Codex
* tests : enlarge Inkling2 context fixture
Assisted-by: Codex
* inkling2 : load per-layer expert feed-forward widths
Assisted-by: Codex
* tests : add Inkling2 Metal fusion baselines
Assisted-by: Codex
* inkling2 : group and align registry entries
Assisted-by: Fable
* convert : regenerate the Inkling2 tokenizer hash with the update script
Assisted-by: Fable
* conversion : revert the print_registered_models change
Assisted-by: Fable
* conversion : sort the Inkling2 model map entries
Assisted-by: Fable
* convert : map Inkling2 vision checkpoint names in the tensor table
Assisted-by: Fable
* convert : map Inkling2 text checkpoint names in the tensor table
Assisted-by: Fable
* tests : document the Inkling2 packed expert conversion test
Assisted-by: Fable
* tests : cover Inkling2 checkpoint and transformers tensor names
Assisted-by: Fable
* llama : place Inkling2 fields and list entries by file convention
Assisted-by: Fable
* gguf-py : align Inkling2 tensor mapping comments
Assisted-by: Fable
* mtmd : place Inkling2 entries by file convention
Assisted-by: Fable
* tests : place Inkling2 cases after the existing ones
Assisted-by: Fable
* cuda : index per-head flash attention masks in the tile and mma kernels
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : project the relative bias before the gather, drop arch branches
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* convert : require Inkling2 hparams, add inkling2 pre-tokenizer
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* mtmd : cap Inkling2 image size, drop video frame averaging
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : fix CI build, Meta device assert and Windows conversion test
Assisted-by: Claude
* tests : use the default sliding window for the Inkling2 test model
Assisted-by: Claude
* simplify mtmd part
* give _v arch for vision
* remove alpha channel handling, it should be a follow up
* explicit rule about audio + vision models
* add audio support
* rm audio
* rm gguf mtmd rope base
* address review comment
* nits
* do not normalize embd from mtmd
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* vulkan: optimize matrix multiplication for narrow N
Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.
* Attempt to handle wave64
* add n=16 tile for cm1
* apply RDNA split_k heuristic to narrow_n generally
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* chat: allow tool calls inside reasoning blocks for Ministral-3 and Qwen3
Fixes#29248Fixes#20837
* chat: address review feedback for Ministral-3 reasoning end
- Remove <function= from Qwen3 (intentionally not used outside outdated coder models)
- Ministral-3: use choice between [/THINK] and peek([TOOL_CALLS]) instead of optional
* chat: add [TOOL_CALLS] to Ministral-3 thinking_end_tags
* chat: simplify Ministral-3 reasoning end choice
* chat: restore peek lookahead for Ministral-3 tool call start
* chat: keep [TOOL_CALLS] only in parser, not in thinking_end_tags
* cont : revert tool call tag removal and update tests
---------
Co-authored-by: Alen Požun <kukirin-ui@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
The thought channel is normally closed with <channel|>, but the model
may open it and go straight to a tool call without closing it. Stop the
reasoning at <channel|> OR <|tool_call>, so an unclosed thought does
not swallow the following tool call.
Fixes: #32231
Assisted-by: Qwen 3.8 27b
* mtmd : allow custom video decoder
* also adapt to wllama requirements
* clean up
* add test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* grammar : limit the number of alternatives
* grammar : introduce an AST for grammar parsing
* grammar : revert AST refactor and alternatives limit
This reverts commits:
- 84f3b57c4d6c482e7187a30735d16b02de3784fa
- 393a707ebb7e6f798abdb8af34cb1bc102f4aaa6
* grammar: optimize big grammar matching by using left-factorization
* grammar : preserve character class boundaries during left factoring
* grammar : prevent infinite expansion during left factoring
* grammar : use ASCII characters in comments
* grammar : test empty suffixes after left factoring
* grammar : deduplicate left factoring logic
* grammar : add comments
* grammar : use number parameters and clarify RSS output
* grammar : clarify character class test names
* grammar : remove performance benchmark
* grammar : share immutable rules
---------
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
* llama : remove deprecated `llama_set_warmup`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : remove remaining deprecated API
Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* fit-params : remove stale `llama-ext.h` include
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : document `llama_model_dflash_selector_top_k`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : promote the NextN / speculative API from `llama-ext.h`
Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : polish the NextN API promotion
Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : prune deprecated API
* cont : bump RPC version
* cont : minor
* ggml : remove outdated documentation from ggml.h
The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:
- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`
The block also ended with 5 empty `TODO` sections.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)
* rpc : bump RPC protocol major version
The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.
The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: losewayy <1134285612@qq.com>
* CUDA: few-row MMQ for 2-24 src1 columns
New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.
* nit fixes
* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat
* few-row quantize: DS4 sum from the quantized values, as in #31247
* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build
* few-row MMQ: dispatch order MMVQ, few-row, MMQ
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back
* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ
* ggml-cuda: Tests for the fused scale
* ggml-cuda: reuse existing graph to test fusion in mmq epiloque
* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency
* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions
* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
* additional FA test cases
Assisted-by: Codex
* vulkan: allow dequantized KV optimization with CM2
Enable the KV dequantization optimization with coopmat2, guarded by a heuristic
based on query tokens*GQA ratio.
While doing this we saw very large speedups for certain head sizes, and it was
because those head sizes were incorrectly treated as unaligned. So also fix
those to be treated as aligned, which should help for when the dequant
optimization doesn't apply.
* gqa uses nbq2 for the stride, so we need to check that value for alignment
* tests : use VARS_TO_STR18 for flash attention
Assisted-by: Codex
MMF tiles 32 rows per block, so a BF16 matrix with few rows (e.g. the
qwen4exp hyper-connection projections, 320 rows with K = 10240) runs only
a handful of blocks and leaves most SMs idle at batch sizes 2..8. MMVF
runs a block per row and wins there when K is long enough.
MMVF's cost grows with rows x batch while MMF's stays roughly flat in that
range, so use MMVF only while rows * ne11 * 256 <= nsm * min(K, max_k) and
K >= 1024. The budget scales with the SM count, which keeps GPUs with few
SMs (e.g. the A16) on MMF. Blackwell allows up to 8 columns and
max_k = 6144; older GPUs stop at 5 columns and max_k = 4096.
Add eval cases at 128x2560, 512x2560 and 320x10240 for n = 2, 3, 4, 5 and 8.
* cuda : add MMA FlashAttention support for head sizes 40 and 72
Head sizes 40 and 72 (used e.g. in vision towers of Gemma 3/4 and Qwen3-VL) were excluded from every tensor-core dispatch branch and fell back to the tile kernel.
The fp16 MMA fragment covers 16 halfs in the head direction, so the in-kernel geometry is padded to the next multiple of 16 (40 -> 48, 72 -> 80). Register arrays, shared memory tile strides and MMA loop bounds use the padded size, while global memory loads and stores the real one.
Enabled on NVIDIA Turing and newer while Volta and AMD keep the tile kernel. The RDNA/CDNA config rows and the mma_tile_sizes specialisations are needed because the explicit instantiations are compiled for every target of a translation unit.
* tests : add the head size = 40 encoder perf shape
* cuda : address review comments on the head size 40 and 72 kernels
* Initial CUDA WIP version
* Use fp8 intrinsics
* Do W8A16 for FP8 in GEMV case
(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB
| model | size | params | backend | ngl | n_ubatch | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 | 2781.53 ± 8.48 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 | 37.56 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 @ d32000 | 1616.31 ± 2.61 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 @ d32000 | 34.11 ± 0.03 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 | 2542.00 ± 0.51 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 | 37.36 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 @ d32000 | 1762.86 ± 1.33 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 @ d32000 | 34.09 ± 0.02 |
* Optimize GEMV kernel for FP8
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2
Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099
* Remove trailing whitespace
* Add supported for strided copies to/from FP8 in CUDA
* Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh
* Fix odd-rows for convert/getrows, possible with FP8
* Fix FP8 fallback in cuBlas
* Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8
* Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte
* Add todo about accumulator/precision honoring
This is beyond the scope of the stacked PR
* Avoid races in fp8_fallback
* Allow Hopper and other BW SKUs for FP8
* Use intrinsic over explicit manual conversions
* Resolve TODO
As per spec, we don't have to preserve signs
* Add commend on cublaslt workspaces
* Use convert_unary_cont_cuda over block dequant in convert.cu
* Use get_rows_cuda_float for fp8 kv cache
* Make FP8 copies work on all IHV's
* More cleanup
* Simplify by falling back to shared cuBLAS helper
* Fuse weight_scale multiplication for FP8 also
* More code-cleanup
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4
Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix
* Use F32 activations in fallback mmvf path
* Explicitly link against cublaslt
* Move FP8 conversion intrinsics into ggml_cuda_cast
Cleaner this way
* WIP src1/prec for fp8 in MMVF
* Remove comment
* Keep only f8 x f8 KV cache for now
* Just do FP8 x FP32 x FP32 in MMVF as its fast enough
* Update ggml/src/ggml-cuda/ggml-cuda.cu
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
* Add batched FP8 GEMMS in cublas as suggested by code review
https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644
* Extract fp8x2 -> float2 load into helper
* Searapte copy checks to/from fp8
* Compiler massaging to avoid repeated address offsets
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>