Commit Graph
1369 Commits
Author SHA1 Message Date
Aldehir Rojas 4ddf2b9470 chat : implement inkling parser (#32381) 2026-11-16 19:48:24 +01:00
Rudransh Shekhar 7804be0634 arg : allow up to llama_max_devices() configs for --tensor-split and --fit-target (#13293) (#31435) 2026-11-16 17:07:45 +01:00
Xuan-Son Nguyen ef6bc6f367 jinja: impl scope-chain key lookup (#32396)
* jinja: impl scope-chain key lookup

* add test for call_statement lambda
2026-11-16 15:48:46 +01:00
Niklas Wenzel 62a6d3a302 llama : migrate to llama_batch_ext (#32356)
* llama : migrate to llama_batch_ext

* mtmd : migrate zonos3-mini code

Added in https://github.com/ggml-org/llama.cpp/pull/32020
2026-11-16 15:16:12 +01:00
Jeff Bolz ef3848f9d7 vulkan: Support split_k for conv3d. (#29929)
This tries to use the 256x128 shape (coopmat2-only) with split_k when
it appears profitable. It reuses the mul_mat split_k reduce shader.
2026-11-16 15:25:12 +02:00
Xuan-Son Nguyen e1925b3695 jinja: support json sort_keys=true (#32366) 2026-11-16 12:28:19 +01:00
PascalandRuben Ortlam adf554d239 vulkan: make the radix top-k tie selection deterministic (#31931)
* vulkan: make the radix top-k tie selection deterministic

The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.

Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.

* optimize shader for AMD, Intel and Nvidia

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-16 11:41:44 +01:00
liminfei-amd f477c1b90a cuda : fix MUL_MAT_ID abort with odd src0 columns on AMD (#32385)
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.

Fixes #30576

Assisted-by: GitHub Copilot

Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
2026-11-16 17:53:06 +08:00
Titaniumtown 21fa9c043f kv-cells: avoid an O(n_cells) scan per sequence removal (#32380) 2026-11-16 10:52:09 +01:00
Pascal a946eb541a server: answer reasoning_effort levels the template rejects with 400 (#32332)
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.

Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.

Also trim the output_config.effort test to a single level.
2026-11-16 06:55:56 +01:00
Xuan-Son NguyenandAldehir Rojas b4c0e00a4f jinja: add 4 missing features (#32352)
* jinja: add 4 missing features

* apply suggestion

Co-authored-by: Aldehir Rojas <hello@alde.dev>

* fix error msg

---------

Co-authored-by: Aldehir Rojas <hello@alde.dev>
2026-11-16 00:40:44 +01:00
Xuan-Son Nguyen 7ae3c6b231 mtmd: add zonos3-mini TTS model (#32020)
* zonos3 init

* add audio gen pipeline

* fix string_format

* add default voice as gguf metadata

* better ref

* support voice clone

* add delivery params

* optimize speed

* nits

* nits 2

* add text normalization

* nits
2026-11-15 19:44:44 +01:00
Xuan Son Nguyen f4d0673917 Revert "jinja: add 4 missing features"
This reverts commit a681a53b60.
2026-11-15 19:35:31 +01:00
Xuan Son Nguyen a681a53b60 jinja: add 4 missing features 2026-11-15 19:33:06 +01:00
Roshan Kumar 56fd8a375f server : ignore empty stop strings (#32093) 2026-11-15 17:23:05 +01:00
Xuan-Son Nguyen dd099a6bf3 chat: fix inkling 2 begin/end of image tokens placement (#32294)
* chat: fix inkling 2 begin/end of image tokens placement

* add test
2026-11-15 09:41:44 +01:00
Xuan-Son Nguyen 429460dc0e server: refactor, add server_slot_sched, prompt running hash (#32154)
* server: refactor, add server_slot_sched, prompt running hash

* use sha256 hash
2026-11-15 09:37:17 +01:00
Ed Addario 5a4afd8077 tests: Fail test-backend-ops and exit with error status 1 when missing and/or unsupported types and/or unsupported backends are provided in the command line (#30279)
* Fail tests on missing or unsupported type

* Fix output

* Remove whitespace

* Fail if no tests ran regardless of backend support

* Revert unnecessary cosmetic change
2026-11-15 08:26:23 +01:00
Tarek Dakhran e574a17b06 model : metal : workaround fusion for LFM3 (#32296)
* lfm3 : put the normed tensor first in residual adds to enable norm fusion

Assisted-by: Claude Opus 5.5

* tests : update Metal fusion baseline for lfm3

Assisted-by: Claude Opus 5.5
2026-11-15 01:05:20 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5c008e0008 feat: add Inkling 2 support (text + vision) (#32217)
* convert : add Inkling2 text GGUF conversion

Assisted-by: Codex

* llama : implement Inkling2 text inference

Assisted-by: Codex

* tests : cover Inkling2 text support

Assisted-by: Codex

* conversion : dequantize packed Inkling2 experts in row chunks

Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.

Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.

Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.

Assisted-by: Codex

* convert : add Inkling2 vision mmproj support

Assisted-by: Codex

* mtmd : implement Inkling2 still-image encoder

Assisted-by: Codex

* inkling2 : scope image loading

Assisted-by: Codex

* inkling2 : enable flash attention with folded bias and sinks

Assisted-by: Codex

* cuda : route per-head flash attention masks through vec

Assisted-by: Codex

* mtmd : support Inkling2 temporal frame means

Assisted-by: Codex

* style : restore shared image comment

Assisted-by: Codex

* tests : enlarge Inkling2 context fixture

Assisted-by: Codex

* inkling2 : load per-layer expert feed-forward widths

Assisted-by: Codex

* tests : add Inkling2 Metal fusion baselines

Assisted-by: Codex

* inkling2 : group and align registry entries

Assisted-by: Fable

* convert : regenerate the Inkling2 tokenizer hash with the update script

Assisted-by: Fable

* conversion : revert the print_registered_models change

Assisted-by: Fable

* conversion : sort the Inkling2 model map entries

Assisted-by: Fable

* convert : map Inkling2 vision checkpoint names in the tensor table

Assisted-by: Fable

* convert : map Inkling2 text checkpoint names in the tensor table

Assisted-by: Fable

* tests : document the Inkling2 packed expert conversion test

Assisted-by: Fable

* tests : cover Inkling2 checkpoint and transformers tensor names

Assisted-by: Fable

* llama : place Inkling2 fields and list entries by file convention

Assisted-by: Fable

* gguf-py : align Inkling2 tensor mapping comments

Assisted-by: Fable

* mtmd : place Inkling2 entries by file convention

Assisted-by: Fable

* tests : place Inkling2 cases after the existing ones

Assisted-by: Fable

* cuda : index per-head flash attention masks in the tile and mma kernels

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : project the relative bias before the gather, drop arch branches

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* convert : require Inkling2 hparams, add inkling2 pre-tokenizer

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* mtmd : cap Inkling2 image size, drop video frame averaging

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : fix CI build, Meta device assert and Windows conversion test

Assisted-by: Claude

* tests : use the default sliding window for the Inkling2 test model

Assisted-by: Claude

* simplify mtmd part

* give _v arch for vision

* remove alpha channel handling, it should be a follow up

* explicit rule about audio + vision models

* add audio support

* rm audio

* rm gguf mtmd rope base

* address review comment

* nits

* do not normalize embd from mtmd

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-15 00:51:31 +01:00
Pranav DhinakarandMax Krasnyansky dca216ad63 hexagon: add TIMESTEP_EMBEDDING HVX kernel (#32228)
* hexagon: add TIMESTEP_EMBEDDING HVX kernel

* hex-timestep: add dma, improve kparams, and work-splits, and tracing

* timestep: add testcases that cover WAN2.2

* hex-timestep: relax the gates to cont_rows

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-14 15:03:31 -08:00
Jeff BolzandRuben Ortlam d20426bbbe vulkan: optimize matrix multiplication for narrow N (#31990)
* vulkan: optimize matrix multiplication for narrow N

Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.

* Attempt to handle wave64

* add n=16 tile for cm1

* apply RDNA split_k heuristic to narrow_n generally

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-14 23:36:05 +01:00
Tarek Dakhran 0cc522e45d chat : allow undeclared tool calls in LFM2 tool-call grammar (#32164)
* chat : honor OpenAI tool strict flag in LFM2 tool-call grammar


* chat : drop strict, always allow undeclared LFM2 tool calls
2026-11-14 23:34:18 +01:00
3c311796f2 chat : allow tool calls inside reasoning blocks for Ministral-3 (#32271)
* chat: allow tool calls inside reasoning blocks for Ministral-3 and Qwen3

Fixes #29248
Fixes #20837

* chat: address review feedback for Ministral-3 reasoning end

- Remove <function= from Qwen3 (intentionally not used outside outdated coder models)
- Ministral-3: use choice between [/THINK] and peek([TOOL_CALLS]) instead of optional

* chat: add [TOOL_CALLS] to Ministral-3 thinking_end_tags

* chat: simplify Ministral-3 reasoning end choice

* chat: restore peek lookahead for Ministral-3 tool call start

* chat: keep [TOOL_CALLS] only in parser, not in thinking_end_tags

* cont : revert tool call tag removal and update tests

---------

Co-authored-by: Alen Požun <kukirin-ui@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
2026-11-14 23:33:35 +01:00
Cristian Adam 1c2adc75f6 chat : allow Gemma4 tool call during thought (#32232)
The thought channel is normally closed with <channel|>, but the model
may open it and go straight to a tool call without closing it. Stop the
reasoning at <channel|> OR <|tool_call>, so an unclosed thought does
not swallow the following tool call.

Fixes: #32231

Assisted-by: Qwen 3.8 27b
2026-11-14 23:32:19 +01:00
miyan a956a852c9 Fix ctz64 on 32bit (#32273) 2026-11-14 19:06:52 +01:00
Jhen-Jie HongandXuan Son Nguyen e3483e40b0 mtmd : allow custom video decoder (#32188)
* mtmd : allow custom video decoder

* also adapt to wllama requirements

* clean up

* add test

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-14 14:12:07 +01:00
HaoJun ZHANGandXuan Son Nguyen 4f2187e73d jinja : reduce template rendering overhead (#32023)
* jinja : reduce value allocation and lookup overhead

Assisted-by: GPT-6 Astra

* jinja : address rendering review feedback

Assisted-by: GPT-6 Astra

* jinja : test multipart strings and simplify lookup

Assisted-by: GPT-6 Astra

* fix "undefined hint"

* fix divisibleby 0

* clean up

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-14 13:34:20 +01:00
Hera 0cce7580b8 mtmd : avoid padding-only Whisper audio chunks (#30403)
Assisted-by: Claude Opus 5
2026-11-14 13:33:59 +01:00
Ehsan BateniandMax Krasnyansky 61dbf49228 Hexagon: add support for IQ2_XXS and NVFP4 (#31954)
* ggml-hexagon: IQ2_XXS and NVFP4 tiled repack layouts

* ggml-hexagon: IQ2_XXS HVX matmul kernel

* ggml-hexagon: NVFP4 HVX matmul kernel

* ggml-hexagon: NVFP4 per-tensor scale

* ggml-hexagon: HMX paths for IQ2_XXS and NVFP4

* ggml-hexagon: supports_op coverage for NVFP4 and IQ2_XXS

* fix HMX IQ2_XXS dequant scale underflow in fp16

* hexagon: IQ2_XXS 2-gather tile decode with compact grid, small-weight tests

* IQ2_XXS vpopcount sign parity and gather pipelining

* pipeline IQ2_XXS HMX dequant gathers and add large_w tests

* hex-iq2-xxs: rebase and cleanup

Remove unused helper functions.
Add lut_size to the overall vtcm required size.
Trim down (way way down) the new tests.

* hex-iq2-xxs: actually trim down the tests

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 20:57:05 -08:00
Todor BoinovskiandMax Krasnyansky ded522f830 hexagon: add support for SSM_SCAN op (#31921)
* hexagon: add SSM_SCAN Op support

* hex-ssm-scan: fix multi-device spliting

* hex-ssm-scan: remove redundant checks and make kernel use host config

* hex-ssm-scan: remove direct ddr access

* hex-ssm-scan: remove DMA ordering

* hex-ops: update supported ops

* hex-ssm-scan: rewrite the dma pipeline to match other ops (dbl.buf, etc)

* hex-ssm-scan: remove all fallbacks, only dma/vtcm paths

* hex-ssm-scan: add kernel_type to kparams and remove redundant recomputes

* hex-ssm-scan: avoid stalls in the overall pipeline (dma flush between chunks)

* hex-ssm-scan: vectorize softplus

* hex-ssm-scan: optimize hvx loops

* hex-ssm-scan: add HMX support

* hex-ssm-scan: add support for GGML_HEXAGON_SSM_SELECT

* hex-ssm-scan: cleanup kernel names

* hex-ssm-scan: simplify mdev checks

* hex-mdev: improve mdev row-split solver

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 12:27:51 -08:00
anujj 31953bcf74 ggml : add BF16 support to RMS_NORM (CPU, CUDA) (#32135)
* ggml : add BF16 support to RMS_NORM (CPU, CUDA)

* cuda : use one helper for the F32/BF16 RMS_NORM launches
2026-11-13 15:41:17 +02:00
o7siandPiotr Wilkin 108f73adfe grammar : optimize matching with left factoring (#29557)
* grammar : limit the number of alternatives

* grammar : introduce an AST for grammar parsing

* grammar : revert AST refactor and alternatives limit

This reverts commits:
- 84f3b57c4d6c482e7187a30735d16b02de3784fa
- 393a707ebb7e6f798abdb8af34cb1bc102f4aaa6

* grammar: optimize big grammar matching by using left-factorization

* grammar : preserve character class boundaries during left factoring

* grammar : prevent infinite expansion during left factoring

* grammar : use ASCII characters in comments

* grammar : test empty suffixes after left factoring

* grammar : deduplicate left factoring logic

* grammar : add comments

* grammar : use number parameters and clarify RSS output

* grammar : clarify character class test names

* grammar : remove performance benchmark

* grammar : share immutable rules

---------

Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
2026-11-13 13:52:21 +02:00
Awss 5676dfbe01 chat : accept </ifm|think> as K2 Horizon closer at medium/low effort (#30335) 2026-11-13 00:11:58 +01:00
Georgi Gerganov 8ca5a8008b llama : prune (#30565)
* llama : remove deprecated `llama_set_warmup`

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : remove remaining deprecated API

Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* fit-params : remove stale `llama-ext.h` include

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : document `llama_model_dflash_selector_top_k`

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : promote the NextN / speculative API from `llama-ext.h`

Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : polish the NextN API promotion

Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-12 20:59:09 +02:00
Georgi Gerganovandlosewayy 29dbbad2b1 ggml : prune (#29801)
* ggml : prune deprecated API

* cont : bump RPC version

* cont : minor

* ggml : remove outdated documentation from ggml.h

The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:

- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`

The block also ended with 5 empty `TODO` sections.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)

* rpc : bump RPC protocol major version

The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.

The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: losewayy <1134285612@qq.com>
2026-11-12 20:58:24 +02:00
anujj 0fb8f15770 Title: CUDA: few-row MMQ for BS 2-24 (#31533)
* CUDA: few-row MMQ for 2-24 src1 columns

New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.

* nit fixes

* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat

* few-row quantize: DS4 sum from the quantized values, as in #31247

* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build

* few-row MMQ: dispatch order MMVQ, few-row, MMQ
2026-11-12 21:12:45 +08:00
Konrad Moren 0d69255c52 CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints (#26048)
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back

* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ

* ggml-cuda: Tests for the fused scale

* ggml-cuda: reuse existing graph to test fusion in mmq epiloque

* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency

* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions

* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
2026-11-12 10:28:36 +01:00
Aldehir Rojas 93cf736a8d chat : refactor gemma4 to build loose schema rules and clean up (#31915)
* chat : build a schema-shaped PEG parser for Gemma 4 tool arguments

* chat : enforce only the key structure of Gemma 4 tool arguments

* chat : remove old gemma4 workarounds

* cont : add node.as<>() to simplify static casts

* cont : add depth_guard and use numerical rule names

* cont : clean up

* cont : extract gemma 4 parsers

* cont : clean up

* cont : clean up

* cont : remove dynamic cast
2026-11-12 09:44:42 +02:00
Xuan-Son Nguyen 664f031f49 ci: move test models to ggml-org-dev (#32096)
* ci: move test models to ggml-org-dev

* fix test spec
2026-11-11 21:17:16 +01:00
CC 4180722eeb metal : prevent out-of-bounds reads in SOLVE_TRI kernel (#32103) 2026-11-11 22:02:06 +02:00
Aldehir Rojas cd2db800db jinja : support chained comparisons and argument unpacking (#32072) 2026-11-11 11:53:04 -06:00
Xuan-Son Nguyen f7b886fe6f chat: allow TTS model to provide custom text normalizer via jinja (#32053)
* chat: allow TTS model to provide custom text normalizer via jinja

* rm tts_norm_is_reserved_name
2026-11-11 17:34:55 +01:00
Oliver Simons 1648d64117 test-fp8.cpp: use only the public ggml API (#32083) 2026-11-11 13:33:12 +02:00
Leon af6b5243d6 webgpu: add PAD_REFLECT_1D support (#31877)
* webgpu : add PAD_REFLECT_1D support

* webgpu: remove redundant PAD_REFLECT_1D contiguity check

* tests: simplify PAD_REFLECT_1D setup
2026-11-11 12:11:06 +02:00
Jeff Bolz d2b3eb9b60 vulkan: allow dequantized KV optimization with CM2 (#31526)
* additional FA test cases

Assisted-by: Codex

* vulkan: allow dequantized KV optimization with CM2

Enable the KV dequantization optimization with coopmat2, guarded by a heuristic
based on query tokens*GQA ratio.

While doing this we saw very large speedups for certain head sizes, and it was
because those head sizes were incorrectly treated as unaligned. So also fix
those to be treated as aligned, which should help for when the dequant
optimization doesn't apply.

* gqa uses nbq2 for the stride, so we need to check that value for alignment

* tests : use VARS_TO_STR18 for flash attention

Assisted-by: Codex
2026-11-11 12:08:06 +02:00
Oliver Simons 03d6c824e1 tests : use only the public ggml API (#32077) 2026-11-11 12:01:24 +02:00
Aman Gupta 459ab1cdd3 cuda : use MMVF for BF16 matrices of few rows at small batch sizes (#31959)
MMF tiles 32 rows per block, so a BF16 matrix with few rows (e.g. the
qwen4exp hyper-connection projections, 320 rows with K = 10240) runs only
a handful of blocks and leaves most SMs idle at batch sizes 2..8. MMVF
runs a block per row and wins there when K is long enough.

MMVF's cost grows with rows x batch while MMF's stays roughly flat in that
range, so use MMVF only while rows * ne11 * 256 <= nsm * min(K, max_k) and
K >= 1024. The budget scales with the SM count, which keeps GPUs with few
SMs (e.g. the A16) on MMF. Blackwell allows up to 8 columns and
max_k = 6144; older GPUs stop at 5 columns and max_k = 4096.

Add eval cases at 128x2560, 512x2560 and 320x10240 for n = 2, 3, 4, 5 and 8.
2026-11-11 10:46:13 +02:00
Sihan Yu 9d7b2d7397 cuda : fattn mma head sizes 40 72 (#30285)
* cuda : add MMA FlashAttention support for head sizes 40 and 72

Head sizes 40 and 72 (used e.g. in vision towers of Gemma 3/4 and Qwen3-VL) were excluded from every tensor-core dispatch branch and fell back to the tile kernel.

The fp16 MMA fragment covers 16 halfs in the head direction, so the in-kernel geometry is padded to the next multiple of 16 (40 -> 48, 72 -> 80). Register arrays, shared memory tile strides and MMA loop bounds use the padded size, while global memory loads and stores the real one.

Enabled on NVIDIA Turing and newer while Volta and AMD keep the tile kernel. The RDNA/CDNA config rows and the mma_tile_sizes specialisations are needed because the explicit instantiations are compiled for every target of a translation unit.

* tests : add the head size = 40 encoder perf shape

* cuda : address review comments on the head size 40 and 72 kernels
2026-11-11 10:45:01 +02:00
Oliver SimonsandAman Gupta 9e9619d030 CUDA: Add OCP FP8 support (#28413)
* Initial CUDA WIP version

* Use fp8 intrinsics

* Do W8A16 for FP8 in GEMV case

(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
  Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB
| model                          |       size |     params | backend    | ngl | n_ubatch |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |          pp8196 |       2781.53 ± 8.48 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |           tg256 |         37.56 ± 0.02 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio | pp8196 @ d32000 |       1616.31 ± 2.61 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |  tg256 @ d32000 |         34.11 ± 0.03 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |          pp8196 |       2542.00 ± 0.51 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |           tg256 |         37.36 ± 0.02 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio | pp8196 @ d32000 |       1762.86 ± 1.33 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |  tg256 @ d32000 |         34.09 ± 0.02 |

* Optimize GEMV kernel for FP8

* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2

Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099

* Remove trailing whitespace

* Add supported for strided copies to/from FP8 in CUDA

* Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh

* Fix odd-rows for convert/getrows, possible with FP8

* Fix FP8 fallback in cuBlas

* Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8

* Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte

* Add todo about accumulator/precision honoring

This is beyond the scope of the stacked PR

* Avoid races in fp8_fallback

* Allow Hopper and other BW SKUs for FP8

* Use intrinsic over explicit manual conversions

* Resolve TODO

As per spec, we don't have to preserve signs

* Add commend on cublaslt workspaces

* Use convert_unary_cont_cuda over block dequant in convert.cu

* Use get_rows_cuda_float for fp8 kv cache

* Make FP8 copies work on all IHV's

* More cleanup

* Simplify by falling back to shared cuBLAS helper

* Fuse weight_scale multiplication for FP8 also

* More code-cleanup

* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4

Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix

* Use F32 activations in fallback mmvf path

* Explicitly link against cublaslt

* Move FP8 conversion intrinsics into ggml_cuda_cast

Cleaner this way

* WIP src1/prec for fp8 in MMVF

* Remove comment

* Keep only f8 x f8 KV cache for now

* Just do FP8 x FP32 x FP32 in MMVF as its fast enough

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

* Add batched FP8 GEMMS in cublas as suggested by code review

https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644

* Extract fp8x2 -> float2 load into helper

* Searapte copy checks to/from fp8

* Compiler massaging to avoid repeated address offsets

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
2026-11-11 10:44:30 +02:00