Commit Graph
586 Commits
Author SHA1 Message Date
Georgi Gerganov ea40a3a1e1 metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (#31615)
* metal : support scale-bias routing norm in the TOPK_MOE fusion

The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sqrt(softplus) MoE routing with a selection bias

Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sigmoid MoE routing with a selection bias

The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.

- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
  GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
  tail; the selection-bias ADD is required, same as the sqrt(softplus)
  chains
- generalize the gating function constant from a sqrt(softplus) bool
  to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
  scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
  CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
  weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
  deepseek2/32, kimi, minimax, ...) now fuse their routing, and
  muse-spark prefill drops one MOE_REDUCE fire because the reordered
  graph allocates the routing weights inside the reduce output range,
  which the dst/input overlap check correctly declines

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* cont : remove low-signal comments

* ocd : valign ggml_metal_fusions
2026-11-05 13:52:56 +02:00
Takamichi Ohara 1fa7f1c488 ggml : support truncate=false for YaRN (#30206)
* ggml : support `truncate=false` for YaRN

YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.

Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.

* support CUDA backend

* add tests

* move RoPE helpers to internal header

* format
2026-11-05 12:02:05 +02:00
Hoang Vien DuyandGeorgi Gerganov f7b41cca4f metal : use mul_mm for 32 <= K < 64 (#30982)
* metal : use mul_mm for 32 <= K < 64

* metal : add bf16 x bf16 mul mat kernel

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-05 10:58:41 +02:00
Leon 553d2c2e7e webgpu : add MEAN support (#31591)
Assisted-by: Codex
2026-11-05 10:29:14 +02:00
Pascal a01dff908b tests: fix CI flakes of TOPK_MOE caused by near ties (#31426)
* tests: fix CI flakes of TOPK_MOE caused by near ties

With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.

Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.

* tests: build the TOPK_MOE logits by inverting the gating function

Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.

The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
2026-11-05 07:31:48 +01:00
anujj 5ce06fc71f Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ (#31296)
* cuda: fuse NVFP4 gate/up and GLU in MMQ

* cuda: honor the requested src1 precision in fused gate/up MMQ

* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels

* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming

* cuda: check gate_scale before loading the fused NVFP4 gate scale
2026-11-05 10:47:33 +08:00
Todor BoinovskiandMax Krasnyansky bee23322f1 hexagon: add LIGHTNING_INDEXER support (#31289)
* hexagon: initial support for LIGHTNING_INDEXER op

* hex-lght-idxr: hmx for prefill

* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute

* hex-lght-idxr: pipelined hmx path for decode and prefill

* hex-lid: add support for K (f32)

* hex-lid: add support for K (f32)

* hex-lid: add support for q4_0

* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills

* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel

* hex-lit: update dma pipeline to use canonical dbl buffering in all paths

* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths

* hex-lit: add support for dims collapse

* hex-lit: make kparams constistent with other ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 21:17:30 -08:00
Todor Boinovski 7071ba378a hexagon: add TOP_K_MASK op (#31552)
* hexagon: add TOP_K_MASK op

* Simplify Hexagon TOP_K_MASK kernel and do not duplicate host configuration
2026-11-03 12:04:03 -08:00
Todor BoinovskiandMax Krasnyansky 20a51f24a8 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
* hexagon: add support for DSV4_HC_PRE

* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization

* hex-dsv4-hc-pre: double-buffered DMA for prefill

* hex-dsv4-hc-pre: refactor DMA block setup

* hex-dsv4-hc-pre: precompute kernel params

* hex-dsv4-hc-pre: support for n_hc != 4

* hexagon: add support for DSV4_HC_POST

* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block

* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks

* hex-run: add support for test-llama-archs tool

* hex-dsv4: refactoring and optimizations

Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.

Tested with test-llama-archs -a qwen4ext and test-backend-ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 08:26:18 -08:00
Leon 713319e284 webgpu: add ARANGE support (#31455) 2026-11-03 07:54:00 +02:00
51fe42c5d1 hexagon: tiled hvx permute (#31446)
* hexagon: add tiled HVX transpose path for permuted CONT/CPY

Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.

Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.

Host:
  - drop extent-1 dims and merge dims that stay adjacent in src into a
    canonical view, stored in the kernel params. There are no minimum
    size limits.
  - the copy is treated as a 2D transpose between dst dim 0 and the
    contiguous src dim, batched over the remaining dims.
  - tiles are sized to 64 KB per buffer and halved until they fit in
    session VTCM. blkk is capped so the halfword vgather offsets fit in
    16 bits.

Kernel, per tile:
  - 2D DMA of blk0 strided src rows (blkk contiguous elements each)
    into VTCM
  - vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
  - 2D DMA of blkk contiguous rows out to dst

Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.

Small dims:
  - packed mode: when dim k follows dim 0 and dim 0 divides the HVX
    lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
    whole dst rows, and each tile goes out as one contiguous DMA.
  - when the contiguous src dim fits in one vector, it is staged exactly
    and the tile is made long along dim 0.

All other copies select the same kernels as before.

Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.

* hexagon: cpy transpose cleanups

- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)

Assisted-by: Claude Opus 5.5

* hexagon: use cpy transpose path for ne[0] == 1 permutes

- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests

Assisted-by: Claude Opus 5.5

* hex-cpy: further improve cpy

Bring in improvements from aizip branch/PR.

Cleanup kparams and replace divs with fastdiv.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-cpy: avoid vtcm bank conflicts

* hex-cpy: trim down the comments

* cpy-test: trim down new tests that cover permute scenarios

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
2026-11-02 21:13:38 -08:00
Aman GuptaandGeorgi Gerganov 1b6935b227 llama : flash_attn_ext_rows - more efficient unified kv for multi-seq (#29510)
* llama : use llm_attn_mask and kv_rows for unified attention

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* cont : optimize `get_seq_cells`

* llama : build the kv_rows lists from per-sequence cell bitmaps

Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.

Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).

* cont : make get_kv_rows_p_min private

* cont : minor metal

* llama : use kv_rows for every ubatch of a unified cache

A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.

Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.

* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-02 18:13:33 +02:00
Mark TodorovichandMax Krasnyansky a91eab67d8 hexagon : fix DMA transfers with strides > 24 bits (#31442)
* hexagon : fix DMA transfers with strides > 24 bits

On v75+ the 2D DMA descriptor stores row_size and strides in 24 bits.
dma_queue_push sent oversized strides to fallback_2d, which only splits
nrows, so strides of 16 MB or more were truncated - wrong addresses.
Fix will fall back to one transfer per row in that case.

Also added two CONT tests with source strides over 16 MB.

* hex-dma: refactor the fallback paths to remove duplication

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-31 16:49:23 -07:00
Oliver SimonsandSigbjørn Skjæret 68d20b884c Quant: OCP FP8 E4M3 support (#27512)
* WIP OCP FP8 E4M3 support

* Fix python type-checker

* Materialize input_scales also for FP8 when converting

* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4

1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
   dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
   weights are reshaped

* Allow remaining FP8 tensors to be dequantized during conversion

* Allow tensor-strategy for FP8-only compressed-tensor ckpts

* Fix and test gguf-py behavior for FP8_E4M3

* Disallow QKV-fusion for FP8 temporarily

* Fix conversion of fused qkvz & FP8 for Qwen3-Next

* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws

* Emit warnings when not fusing FP8 during conversion

* More FP8 conversion fixes

* Add supported for strided copies to/from FP8 in CPU

* remove unneeded scratch buffer from FP8 cpu cpy/dup path

* add explanatory comments and TODOs

* Add dedicated encoder and decoder tests for FP8

* Whitelist ggml_op_get_rows in metal instead of blacklisting

required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list

https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512

* Fix more conversion bugs

* Defer weight materialization during conversion

* Blacklist fp8 in metal backend instead of white-list

* Adjust indents

* Fix lint + execute tests

* Add fast CPU kernel for F8_E4M3

* Conversion fixes

* Fix remaining gaps

* Enable tied weights with shared scales

* Make mandatory-F32 rules apply to FP8 during conversion

* Fix python type checks

https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512

* Fix kaleidai unit-test failures

https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150

* Relax numeric precision constraints for pre-check

* Leave fast-FP8 CPU on RISC-V for follow-up

We already exclude arm from the fast-path

* Allow output scale tensors to be located in any gguf shard

* Move test_conversion_fp8.py test execution to behavior-check.yaml

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it

* Install all requirements also on Windows for behavior-check

Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512

* Unmap lazy tensors before clean-up in Windows

Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512

* LF line endings

https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512

* Extract FP8 tests from llama-arch to its own test-fixture

See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221

* Update gguf-py/gguf/scripts/gguf_convert_endian.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-29 18:14:37 +01:00
Jesse Symons fade717ed8 tests : remove duplicate FA test cases added in #31125 (#31357)
Requested in the #31125 review. The cases duplicate coverage that is
already in the file and add CI time. Test-only, nothing changes in the
kernel, the docs or the CMake option.

Assisted-by: Claude
2026-10-29 10:10:25 +02:00
Max Krasnyansky 8a199ad31a hexagon: support for UPSCALE op (#31331)
* hex-upscale: basic upscale ops (not fully vectorized)

* hex-upscale: optimize nearest 2x, 4x and bilinear 4x

* hex-upscale: add support for non-contig rows and dims collapsing

* hex-upscale: use vgather instead of scalar staging

* hex-upscale: move all kernel select logic to the host, review comments and cleanups

* tests: add permuted UPSCALE test to cover qwen3-vl image->text scenarios
2026-10-29 00:22:34 -07:00
Jesse Symons f7c01c5226 [SYCL] add DPAS flash attention kernel for quantized KV cache (#31125)
* sycl : add DPAS flash attention kernel for quantized KV cache

Decode and small batches (1 to 31 query tokens) now run flash attention
straight on q4_0 and q8_0 KV blocks, instead of scalar dequant or
converting the whole cache to f16 per call. Q.K uses s8 x s4 and s8 x s8
DPAS on XMX, V.P stays f16. dpas.s4.s8 is inline vISA since joint_matrix
has no s4.

Covers head sizes 64 to 512, GQA up to 32 and attention sinks. Prompts
stay on the existing kernels. GGML_SYCL_FA_DPAS=0 turns it off.

B580, Llama 3.1 8B Q4_K_M, q4_0 KV, tg128: 31.3 to 54.2 t/s at 16K
context. FLASH_ATTN_EXT 4582/4584, same 2 failures as master.

Assisted-by: Claude

* sycl : 192 head dropped, Wildcat Lake added to docs

Assisted-by: Claude

* sycl : stricter checks from review

Assisted-by: Claude

* [SYCL] : rename XMX macro, add GGML_SYCL_SUPPORT_XMX option, document DPAS

Assisted-by: Claude
2026-10-29 14:01:21 +08:00
Marco ColomboandMax Krasnyansky 2e0d86bebc Hexagon: IM2COL improvement - block based HVX parallelization (#31050)
* hexagon: IM2COL over multiple threads with output-col blocks

Assisted-by: Claude Opus 5.5

* hex-im2col: fix padding handling

* hex-im2col: loop unroll common cases

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-28 19:04:18 -07:00
anujj f5222f25c1 cuda: fuse MoE (MUL_MAT_ID) gate/up and GLU in MMQ (#31188) 2026-10-28 18:34:28 +05:30
Yonghye Kwon a7af149a2a cuda : add top-k MoE fusion for 384 experts (#31288)
Assisted-by: Codex
2026-10-28 10:13:01 +02:00
Pascal 922f53ed33 vulkan: fix CI llvmpipe by allowing dim 3 broadcast in MUL_MAT (#31239)
* vulkan: allow dim 3 broadcast in MUL_MAT

supports_op rejected every MUL_MAT whose src0 and src1 differ in ne3,
although mul_mm, mul_mmq and mul_mat_vec already broadcast src0 over
dim 3 through broadcast3. Such a matmul then ran on the CPU with a
copy of the weight, and the split layout changed with the number of
streams in the ubatch.

Keep the strict check for MUL_MAT_ID only, and route the broadcast
cases away from mul_mat_vec_nc, the one path that requires equal ne3.

* tests : harden MUL_MAT with permuted dim 3 broadcast cases

Harden the MUL_MAT coverage with a weight broadcast over dim 3 against
a permuted per-stream activation, the shape of an MLA output
projection with a non unified KV cache, for a single token, several
heads and several tokens per stream.
2026-10-27 14:39:03 +02:00
Georgi GerganovandPascal de5122d879 ggml : relax scale to non-contiguous views (#31146)
* ggml : relax scale to non-contiguous views

SCALE requires src0 to be 1D-contiguous. Relax this to dim-0
contiguity with arbitrary row strides (ggml_is_contiguous_rows),
same as the unary ops in the Metal backend.

- ggml: relax the assert in ggml_scale_impl
- cpu: address src0 rows through their byte strides (f32, bf16)
- cuda: stride-aware kernel, fastdiv row decomposition
- metal: support bf16 scale (guarded unary instantiations)
- sycl: keep the gate at contiguous, the kernel is flat f32

Add view variants to test_scale. Verified on CPU, Metal and CUDA.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)

* cuda : skip the softcap fusion on non-contiguous views

The fused softcap kernel reads src and writes dst as dense arrays,
so a view reaching SCALE -> TANH -> SCALE now that SCALE accepts
non-contiguous rows would be read wrongly. Keep such graphs on the
strided kernels until the fusion becomes stride aware.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-27 07:55:31 +02:00
Pascal 4d71b42fd8 ggml-cpu : rewrite concat as one row kernel for all types (#31179)
Every dst row comes from src0 or src1, or for dim 0 from the row of
src0 followed by the row of src1, so a single kernel covers every
type: a dense source row is one memcpy, quantized types included,
and a strided source is copied element by element with the element
size known at compile time.

Rows are split across threads instead of i2 alone, which is often 1,
and are copied in tiles of 32: for a transposed source the elements
are walked in the outer loop, so the rows of a tile read neighboring
addresses instead of jumping a full row stride per element.

Add transposed b cases to test_concat, the layout of the delta-net
conv input, and a perf case for it.
2026-10-26 16:18:45 +02:00
virajwad 70c5f26324 vulkan: rms norm optimization using subgroup reductions (#29882)
* RMS Norm Subgroup reductions optimizaton

* Fix build error

* Improved optimization

* Make usage of BLOCK_SIZE consistent

* Add fast path to skip 2nd subgroupAdd

* Add AMD Vendor to subgroup reduction opt

* Add comment clarifying rms_dict and merge_maps behavior

* Add unroll

* Rebase

* Fixed syntax issue

* Add 128/256/512 block sizes to non-subgroup path

* Remove trailing whitespace
2026-10-26 16:14:31 +02:00
feni6andfeni6 9b8446b914 metal : stop overflow from happening in large matrix output offsets (#30072)
* Metal : stop overflow from happening in large matrix output offsets

Assisted-by: Codex

* metal : widen batch offsets and MUL_MAT_ID token indices

Assisted-by: Codex

---------

Co-authored-by: feni6 <270037094+feni6@users.noreply.github.com>
2026-10-26 10:13:39 +02:00
Aman GuptaandGeorgi Gerganov a074215352 deepseekv4.1: follow-up review fixes (#31137)
* kv-cache : use ggml_backend_tensor_copy to refill the DSV4 K mirrors

* llama : move the Engram pad id to llama_hparams and the token map to llama_model

The saver no longer needs the deepseek41 model class.

* deepseek41 : drop the F32 src1 request on the HC mixes matmul

* ggml : add softplus TODOs in the backends that still use log(1 + exp(x))

* convert : make the DSV4/DSV4.1 modify_tensors generators, drop the redundant vision filter

* deepseek41 : clean up comments

* deepseek41 : drop the F32 src1 requests on the router and indexer matmuls

* convert : use format_tensor_name directly in DeepseekV4Model

* s/comp_kv_src/comp_il_src/g

* cont : fix TODOs for log1p

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-26 15:52:02 +08:00
anujj c49edf0963 CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows (#29948)
* cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows

For prompt processing, run the gate and up matmuls of a dense FFN as one
mul_mat_q launch and apply the GLU in the write-back.

- The loaders take the gate weights as a second pointer and fill the weight
  tile with up and gate rows interleaved in chunks of 8, so each thread holds
  an up value and the matching gate value (rows r and r + 8 of an MMA tile).
- The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[]
  per thread. The fused write-back computes the GLU from registers and writes
  half the rows.
- The fused path uses the regular tiling (no stream-k) with the column tiles
  innermost so that neighbouring blocks share weight tiles.
- Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.

* cuda: assert the fused MMQ gate argument and trim redundant comments

* cuda: move the fused MMQ hint check to should_fuse and add TODOs

* cuda: pass prec_src1 to the fused MMQ write-back

* cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back

* cuda: require has_fusion explicitly, drop the fusion-only src1 precision check

* cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile

* cuda: inline the MMQ tiling-efficiency check

* tests: one fused gate/up MMQ case per type and GLU op

* cuda: always launch the fused MMQ kernel with one block per tile

* tests: shorten the fused gate/up MMQ test comment

* cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs

* cuda: run the fused MMQ kernel on the plain tiling path

* cuda: put the column tiles innermost in the MMQ plain tiling path
2026-10-25 11:24:38 +02:00
Aman Gupta a55c16e4ee ggml-cuda: fuse qwen MRoPE + set-rows (#31025) 2026-10-25 11:15:45 +02:00
Jeff Bolz e321bdad7e vulkan: Add a larger tile size for conv3d (coopmat2 only) (#29920)
* vulkan: Add a larger tile size for conv3d (coopmat2 only)

This adds a larger tile size, in practice it is only for coopmat2 where
we don't need shared memory to store the output (would use more than 64KB).

Add some additional representative backend tests. At least, this is what codex
came up with from looking at sd.cpp Wan VAE.

* only apply large tile size for newer driver versions
2026-10-25 11:13:48 +02:00
Aman GuptaandDaniel Han 74bca39500 model: DSV4.1 flash (#31086)
* model : add DeepSeek-V4.1-Flash (deepseek41)

* deepseek41 : remove MXFP8, convert the Engram table to BF16

* deepseek41 : compute the compressed-KV read layers once in the cache

* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le

* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather

* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer

* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle

* deepseek41 : keep il last in build_hc_mixes

* deepseek41 : sort the top-k rows before the sparse gather

* tests : add deepseek41 to the Metal fusion baseline

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-10-25 09:43:51 +08:00
Ehsan Bateni 78a8ae14be hexagon: KV-head team split for multicore HMX flash attention (#31055)
* hexagon: KV-head team split for HMX flash attention in multicore

* hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag

0 = off (token-split only); 1 = head-split, plus team-split when
n_kv_heads < N_cores so stranded cores still participate (e.g. gemma-4
with 2 KV heads on 4 cores). Where heads divide evenly across cores,
plain head-split is used.
2026-10-24 12:19:32 -07:00
Georgi Gerganov 3f03b17724 metal : remove mul_mm_id amax rescaling (#30924)
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the
per expert-token amax scaling done in the compute graph
(ref: https://github.com/ggml-org/llama.cpp/pull/30861), which keeps the
MoE down-projection activations within f16 range before the op runs.

Drop the two-stage reduction kernels, their scratch buffer and the
GGML_PREC_F32 dispatch added in https://github.com/ggml-org/llama.cpp/pull/26223,
as well as the test cases that drive MUL_MAT_ID src1 out of f16 range.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-24 18:40:19 +03:00
Xuan-Son Nguyenandyomaytk 701fd25b3a ggml : allow MUL_MAT_ID -1 index to skip the computation (#26631)
* ggml : allow MUL_MAT_ID -1 index to skip the computation

* nits

* update vulkan code

* ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec

(cherry picked from commit d8a4b9efb6e2b9abf3a866e847756b7e6bbc6b69)

* fix cuda

* openvino

* exclude CUDA from this PR

* add TODO tag

---------

Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
2026-10-24 12:15:36 +02:00
Alexey Kopytko 5c31142682 qwen4exp : build the QSA selection without the kq_mask (#30965)
* qwen4exp : build the QSA selection without the kq_mask

* clean

* lang

* qwen4exp : reshape the fused QSA selection to the kq_mask streams

ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while
build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream].
With several streams the shape assert failed at graph reserve.

* test the QSA optimization

* reword

* update build_fused_top_k_mask instead of handwiring

* simplify

* keep il at the end for consistency
2026-10-23 19:49:08 +02:00
Max Krasnyansky 643b0cdacb hexagon: Op Fusion refactoring, optimizations and new fusions like BINARY_NX (back-to-back ADDs, etc) (#30843)
* hex-fusion: support for GELU+MUL, REPEAT+GET_ROWS and BINARY_NX (back-to-back ADD/MUL/...)

* hex-fusion: factor out fusion logic into separate .h and .cpp and update all related code

* hex-fusion: allow for RMS_NORM+MUL fusions for non-contig tensors

* hex-fusion: streamline fusion eligibility checks and cover missing corner cases

* hex-fusion: streamline fusion flow

* hex-opbatch: make n_tensors limit more scalable

* hex-bufs: adjust the trace buff sizing

* hex-glu: cleanup glu-ops interfaces and add kernel_params

* hex-kparams: clean up kparams handling in unary, ssm-conv, and other ops

* hex-kparams: update the rest of the ops PAD,DIAG,... to use kparams

* hex-unary: add support for collapsing dims

* hex-ops: cleanup redundant runtime checks

* hex-ops: consistent use of kernel_type in all kparams

* hex-fusion: remove redundant check in gdn+cpy

* hex-fusion: further streamlining of fusion rules and code

* hex-fusion: cleanup naming

* hex-opfmt: cleanup op formatting

* hex-fusion: bump fusion lookahead to 64

* hex-fusion: fuse SSM_CONV+SILU

* hex-fusion: fix hvx register spills in fused ssm_conv+silu

* hex-fusion: add more GLU fusions

* hex-fusion: remove duplicate checks and use ggml-common helpers

* hex-kparams: refactor kparams into htp-kparams.h/cpp

* hex-fusion: fix binary-nx fusion regression

* hex-kparams: another pass at cleaning up kparams (removing unused, etc)

* hex-kparams: streamling nameing and move _check_ wrappers into .h

* hex-kparams: templetize htp_kparams_check_<op> to reduce code bloat

* hex-ops: de-duplicate op dispatch switches

* hex-kparams: trim down exposed kparams calls

* hex-kparams: simplify type casting of kparams structs

* hex-fence: de-dup fence tensor construction

* hex-opbatch: simplify tensor shape init/checks

* hex-mm: cleanup leftovers of the original QK defines

* hex-fa: fix stack overflow on older npu firmware

* hex-ctx: move dma_caches into ops_ctx
2026-10-22 21:05:44 -07:00
budiony d81d0d8fb9 Multi-column on Intel XMX matrix engines refactored (#29864)
* Multi-column on Intel XMX matrix engines refactored

Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).

* Fixed regression bug

real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4

* Removed GGML_TYPE_Q8_0 activation

* rewritten xmx_mul_mat

64-byte alignment and 16-byte pitch check on the activations

* Implemented fused gate/up kernel

One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.

* Removed the temporary XMX_NOGLU environment switch

* Q2_K min-sum optimization

Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group

* Added dispatch check and GRF sweep

tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.

* Renamed variables

* Added 2 environment variables

GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16):  the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.

* Update esimd.hpp

1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.

2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.

The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
2026-10-22 21:39:01 +08:00
Aman GuptaandPascal d454a204ef llama: create new fused op for QSA mask (#30640)
* llama: create new fused op for QSA scoring

* enroll glm5.3-next too

* tests : initialize the remaining tensors in test_kpool_mask

The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check.

* ggml: generalize the k-pool mask op into ggml_top_k_mask

The k-pool mask op only covered the pooled QSA selection of qwen4exp
and glm5-next, while the DSA models and the DeepSeek V4 CSA build the
same top-k mask by hand with fill, set_rows and add.

ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask
cells of the live picks and the extra cells, and sets the rest to
-inf. Every row tensor is 4D in the layout of top_k, and the mask is
broadcast to those rows, so a selection can also be made per head.
score, blocks and extra are optional: without score every pick is
live, without blocks a pick is a cell, and extra holds the cells a
row keeps without a pick.

llm_graph_context::build_fused_top_k_mask views the row tensors in
the layout of kq_mask and registers the fused node. The k-pool
models, the DSA attention of build_attn and the DeepSeek V4 CSA mask
use it when the probe enables the op, and keep their current path on
backends without it.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-22 01:02:22 +08:00
bosh 479a52779f ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table (#30737)
* ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table

* ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path

* ggml-cpu : add the VXE gelu_quick path

* tests : add test-gelu-quick for the f32 CPU kernels

* tests : drop inf/nan inputs from test-gelu-quick
2026-10-21 09:34:37 +02:00
Anant Shrivastava 493253fe5a Sycl iq3 crash fix (#30604) 2026-10-21 01:04:13 -04:00
Tarek Dakhran 39da130fcd ggml-cuda : support ssm_conv kernel size 7 (#30873) 2026-10-20 17:47:44 +02:00
Georgi Gerganov df11702136 metal : optimize argmax and pool_1d (#30862)
* metal : optimize argmax and pool_1d

Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* tests : add pool_1d perf cases

Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : minor fix
2026-10-20 16:46:00 +03:00
Sam MurdochandSam 3bb63cbcbf sycl : fix get_rows on reordered quantized weights (#30430)
Co-authored-by: Sam <sam@garagellm.tail7dd858.ts.net>
2026-10-20 07:47:21 +08:00
Andrea Rossetti 0878f7c304 ggml: support non-contiguous SUM (CPU, CUDA, Vulkan, WebGPU) (#28851)
* ggml: support non-contiguous SUM (CPU, CUDA)

* tests: add non-contiguous SUM cases

* tests: SUM over non-contiguous views

* vulkan: support non-contiguous SUM

* webgpu: support non-contiguous SUM
2026-10-18 18:57:36 +02:00
Todd Malsbary 547208dca2 Restore ESIMD kernels on FFN path (#30239)
* Reenable ESIMD on the FFN path

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Extend ESIMD fusion on FFN path to include Q8_0

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Remove comment explicitly referencing ESIMD in test-backend-ops

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
2026-10-18 00:02:38 -04:00
Sigbjørn Skjæret f2f667b02d cuda : add grouped experts top-k fusion (#29181)
* fuse grouped experts

* add tests

* hip fix

* add to graph_optimize

* apply suggestion from am17an

* hip fix

* fix hip fix

* fix fix fix

* gate on hip version instead
2026-10-17 13:55:33 +02:00
Max Krasnyansky b797b9ca81 hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round

Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors

* hex-get-rows: vectorize raw k-quant ukernels

* hex-binary: refactor dma-loop thread functions (no need for macros, etc)

* hex-quant: move/cleanup flat and tiled dequant ukernels

* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights

* hex-get-rows: remove redundant cast

* hex-repack: proper filtering for BF16 that were leaking through during the tests

* hex-repack: streamling tiling rules (broken MM for now)

* hex-repack: fix mmid after tiling updates

* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars

* hex-mm: fix valid_rows calc in various kernels

* hex-mm: remove unused hvx_mm_nx_2d_flat

* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16

* hex-mm: another pass at hvx-tiled f16 matmul

* hex-get-rows: optimize small src1

* hex-get-rows: one more pass at recovering get-rows perf regression

* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py

* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul

* hex-mm: fold tiled_f16 checks into the general flow

* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op

* hex-quant: proper tail handling in new dequant ukernels

* hex-get-rows: remove unused code and cleanup support checks

* hex-mm: cleanup ne1 padding/tiling checks

* hex-mm: cleanup fusion checks for tiled tensors

* hex-repack: add a little can_repack helper to reduce duplication

* hex-repack: never repack host buffers

* hex-repack: factor out repack functions into separate .h and .cpp

* hex-tensor: cleanup tensor ifaces for tiled/padded sizes

* hex-repack: update k-quant tiling comments and cleanup #defines

* hex-dcache: factor out l2/dcache dirty tracking and flushing

* hex-tensor: update docs and remove #defs duplication

* hex-tensor: remove unused/incorrect get_data interface

* hex-get-rows: cleanup vtcm layout build

* tests: disable huge MM tests

* hex-dcache: add missing header
2026-10-15 20:30:11 -07:00
Nathan WilsonandRuben Ortlam 733f57a999 vulkan : copy strided f16 KV once on AMD so reads spread across memory channels (#27703)
* vulkan : copy strided f16 KV once so reads spread across memory channels

Assisted-by: Claude (Opus 4.8 / Opus 5)

* tests : cover the f16 KV copy to scratch

Assisted-by: Claude (Opus 4.8 / Opus 5)

* vulkan : gate the f16 KV scratch copy to AMD

Assisted-by: Claude (Opus 4.8 / Opus 5)

* vulkan : reuse the generic copy shader for the f16 KV scratch fill

The f16 clause needs only a strided copy, not a dequant, so drop the
dedicated shader and dispatch the existing copy pipeline instead. The
copy is described in source memory order so the strided access lands on
the writes, which stream, instead of the reads, which would concentrate
on a few memory channels.

Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5)

* vulkan : extend the strided KV scratch copy to bf16

Assisted-by: Claude Opus 5

* vulkan : clean up comments

Assisted-by: Claude Opus 5

* consolidate copy shaders

* disable on RDNA4

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-10-15 10:09:02 +03:00
cqderekandravi9 5a638aedb7 cpu: add F16 SWIGLU_OAI reference implementation and validation (#29200)
* cpu: add F16 SWIGLU_OAI reference implementation and F16 activation op test coverage

Adds ggml_compute_forward_swiglu_oai_f16() to the CPU backend, mirroring
the existing F32 path, and extends test-backend-ops with F16 test cases
for SWIGLU_OAI so downstream backends(e.g. Hexagon) can be CPU-diffed
for correctness.

* ggml-openvino: fix f16 SWIGLU_OAI type mismatch in alpha/one constants

translate_glu_swiglu_oai creates alpha_const and one as f32 constants
unconditionally, so f16 inputs hit a type mismatch in Swish/Add on GPU
("Swish inputs must have the same type but they are: f16 and f32").
translate_glu_swiglu_clamp already upcasts to f32 and converts back at
the end; apply the same pattern here.

---------

Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
2026-10-15 10:07:45 +03:00
Ethan Guo 6e66cb5d4e metal : add BF16 variant of the tensor API flash attention kernel (#30491)
* metal : add BF16 variant of the tensor API flash attention kernel

* cont : move tensor FA kernels to kernels/fa_tensor.metal
2026-10-15 09:39:47 +03:00
62df1815bc cuda, hip: GDN chunked kernel (#29353)
* cuda, hip: GDN chunked kernel

Ported-from: https://github.com/tetherto/qvac-fabric-llm.cpp/pull/257
Source-commit: 53f0027d8070715fe9ad2131f3201bb0001d4eed
Co-authored-by: Aman Gupta <aman.gupta@tether.io>

* fix older builds

* review fixes

* cuda: enable GDN MMA on tested Ampere and Ada GPUs

Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback.

Assisted-by: Codex

* cuda : use refactored swizzle API in GDN MMA

* ggml : GDN only guarantees the final state when n_tokens > K

* cuda : add short GDN MMA phase comments

---------

Co-authored-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
2026-10-14 22:00:58 +08:00