Commit Graph
3322 Commits
Author SHA1 Message Date
hey-gmandOliver Simons 1167d3f42c CUDA: fix CCCL version guard breaking on major version rollover (#29453)
* CUDA: fix CCCL version guard breaking on major version rollover

The guard compared the major and minor components independently:

    CCCL_MAJOR_VERSION >= 3 && CCCL_MINOR_VERSION >= 1

Minor resets to 0 whenever a new major series is cut, so on CCCL 4.x
this evaluates as 4 >= 3 && 0 >= 1, i.e. false. STRIDED_ITERATOR_AVAILABLE
stops being defined and argsort silently falls back to the
init_offsets path. Nothing warns and the build still succeeds, so the
regression is a quiet performance loss rather than a compile error.

CCCL already exposes the version as a single packed integer in
MMMmmmpp form, which is what its own version header uses:

    CCCL_VERSION = MAJOR * 1000000 + MINOR * 1000 + PATCH

so 3.4.3 is 3004003 and ">= 3.1" is a plain ">= 3001000". One
comparison, with no component arithmetic left to get wrong.

Checked against a hand-written "version >= 3.1" reference over 2.9.9,
3.0.0, 3.1.0, 3.1.99, 3.2.0, 3.4.3, 3.9.9, 3.99.99, 4.0.0, 4.2.7 and
5.0.0: no divergences. The old guard disagreed at 4.0.0 and 5.0.0.

Verified on RTX 4070 (sm_89), CUDA 13.4, CCCL 3.4.3:

  - cmake --build build --config Release: exit 0
  - test-backend-ops test -o ARGSORT -b CUDA0: 98/98 passed, CUDA0 OK

Note that a passing regression test does not on its own prove the guard
is still taken, since the fallback path passes too. Preprocessing the
real translation unit confirms the strided-iterator branch is the one
compiled in: counting_iterator is present, init_offsets is not.

Signed-off-by: Heitor <heitorgm@outlook.com>

* Update ggml/src/ggml-cuda/argsort.cu

* Apply suggestion from @ORippler

---------

Signed-off-by: Heitor <heitorgm@outlook.com>
Co-authored-by: Oliver Simons <osimons@nvidia.com>
2026-10-08 15:29:34 +02:00
gianni-cor ff5888f999 vulkan : fix TOP_K for +inf/NaN inputs and k = 1 on negative values (#30107)
The bucket search in topk_nary_search.comp started from the range
[0, 0xFF800000), which ends just below the ordered-uint mapping of +inf,
so +inf and NaN were never counted. A workgroup block with fewer than k
countable values left the ballot empty and the shader read uninitialized
shared state (hang/device lost on NVIDIA, wrong indices on AMD), and a few
+inf in a block were selected without being counted, dropping real top
values.

Map NaN to -inf on input, start from [0, 0xFFFFFFFF) so every value is
counted, and clamp the top bucket's end (2^32) instead of wrapping to 0.

The k = 1 path compared float bits as signed integers, which orders
negative values backwards; compare floats instead.

Add test_top_k_inf to test-backend-ops: negative values, fewer than k
+inf and many -inf, for k = 1, 10, 40.

Assisted-by: Claude Opus 5.5
2026-10-08 12:14:35 +02:00
Jeff Bolz dac3087394 vulkan: extend sparse FA support to coopmat2 (#30003) 2026-10-08 11:36:20 +02:00
Muhammad SaadandJohannes Gäßler 08246a28f6 cuda : support arbitrary striding for unary ops on f16, f32, and bf16 (#29781)
* cuda : support arbitrary striding for unary ops on f16, f32, and bf16

* Remove added newline

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-10-08 09:21:42 +02:00
bri-prism 46baf1f1fe sycl: FWHT optimizations (#29605) 2026-10-08 02:50:22 -04:00
Max Krasnyansky 097f5b5332 hex-mmadd: do not assume aligned read/write when bias-add is fused (#30133) 2026-10-08 09:48:06 +03:00
Bertay Eren d888016041 cuda : Use byte strides for roll to allow non-contiguous ROLL operations (#29547) 2026-10-08 09:46:38 +03:00
Eden fda1866613 CUDA: fix norm family kernels when ne[2]/ne[3] exceed grid dim limits (#28175) 2026-10-08 09:46:21 +03:00
Titaniumtown ff30363a0e sycl: fuse the delta-net alpha gate (add + unary + mul) (#29687)
* sycl: fuse the delta-net alpha gate (add + unary + mul)

* tests: cover the fused add + unary + mul chain

* sycl: give the fused alpha gate a flat path and pin the node skip
2026-10-08 09:45:21 +03:00
cwriterandcwriter 000bee54a5 sycl: stage bulk uploads (model loading) through a pinned ring buffer (#29608)
Co-authored-by: cwriter <cwriter@localhost>
2026-10-08 02:30:00 -04:00
SXX 24e41838e0 ggml-cuda: assign four GDN state columns per warp (#30087)
* ggml-cuda: assign two GDN state columns per warp

* ggml-cuda: use 4 GDN state columns per warp at S_v=128

* ggml-cuda: default cols_per_warp=4

* ggml-cuda: address GDN review nits
2026-10-08 11:37:34 +05:30
cwriterandcwriter 847f447c31 sycl: add grouped MoE XMX GEMM (#29245)
Co-authored-by: cwriter <cwriter@localhost>
2026-10-08 01:41:13 -04:00
Titaniumtown 9b4ed0ca57 sycl: remove duplicate block-size defines from op headers (#29507)
All were duplicates of the values in ggml/src/ggml-sycl/presets.hpp
2026-10-08 08:39:25 +03:00
Max KrasnyanskyandJhen-Jie Hong 9c2e0e491a hexagon: enable alloc_buffer_n (#30126)
* hex-bufs: add support for alloc_buffer_n

* hex-bufs: add support for splitting large tensors into separate buffers

* hex-bufs: update GGML_HEXAGON_MBUF to accept three values dyn,static,total

* hex-bufs: bump dyn. default to 512MB since 128MB causes perf regressions with big MOEs

* hex-run: add --no-embd-offload option to simplify command lines on devices that need it

* Update scripts/snapdragon/run.py

Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>

---------

Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
2026-10-07 18:30:06 -07:00
kurquhar 06cad0b9e7 hexagon: Q6_K weight dequant speedup (#30121)
* hexagon: Q6_K weight dequant speedup

Assisted-by: OpenCode

* unroll by another factor of 2

Assisted-by: OpenCode
2026-10-07 17:13:21 -07:00
kurquhar aa5e0092fd hexagon: support tiled Q4_K and Q6_K GET_ROWS (#30115)
* hexagon: support tiled Q4_K GET_ROWS

Assisted-by: OpenCode

* properly reject Q4_K views

Assisted-by: OpenCode

* hexagon: support tiled Q6_K GET_ROWS

Assisted-by: OpenCode
2026-10-07 17:04:45 -07:00
kurquhar 70815103c8 hexagon: improved GELU accuracy (#30104)
Assisted-by: OpenCode
2026-10-07 16:57:09 -07:00
bri-prism b86d2f0754 cuda: FWHT kernels for block widths above 512 (#29100)
The CUDA FWHT covers widths 64 to 512. It runs one row per warp and keeps N/32
values per lane, so wider blocks need more registers per lane than that layout
allows.

fwht_cuda_block runs one row per thread block with 256 threads, so each thread
keeps N/256 values. Stages below the warp width still shuffle, those up to the
block width go through shared memory, and the rest stay in registers. Same
butterfly and sign convention as the warp kernel.

Widths 64 to 512 keep the warp kernel. 1024 through 8192 use the new one, for
both F32 and F16 sources. ggml_cuda_op_mul_mat_use_fwht (the shared
supports_op/dispatch predicate added in #29096) does not check width, so it
needed no change here: any width it admits that ggml_cuda_op_fwht can't serve
already falls through correctly to the cuBLAS path.

Rebased onto current master with #29096's F16 commit underneath it, since this
depends on the same F16 template infrastructure; that commit applied cleanly,
the only conflict was in test-backend-ops.cpp where an unrelated intervening
commit's own test additions landed near this block.

test-backend-ops on an A10 (lambdalabs): MUL_MAT 1297/1297, including all
FWHT/Hadamard cases (18 existing, 4 new F32 wide, 4 new F16 wide, 2 new
many-rows, 1 too-big boundary moved to 16384).
2026-10-07 20:56:18 +02:00
ShobhitandGeorgi Gerganov 42c787e8c1 cuda: update uncoalesced memory reads in pool2d (#29425)
* cuda: update uncoalesced memory reads in pool2d

* cuda: Added perf and __restrict__ in pool2d

* Update POOL2D_WARP_KERNEL_MIN_WINDOW macro

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* cuda: Fix compiler bugs

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-07 19:53:43 +02:00
pratiknarola-t 988190680d metal : few-row MMA mat-mul for the remaining src0 types (#30065)
The generic few-row MMA kernel works for any type with a 16-weight
dequantizer, so it now also takes BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K,
TQ2_0 and the IQ types. Each type starts at the row count where it beats
the current kernels on an M3 Ultra: 5 rows for TQ2_0, 4 for BF16, 3
for MXFP4, Q2_0, Q2_K and IQ4_NL, and 2 for the others.

test-backend-ops perf -o MUL_MAT, m=4096, k=14336, M3 Ultra, time of this
change over master (mean of two interleaved runs each): 0.23 to 0.98 from
the threshold to 8 rows, 0.24 to 0.33 at 9 to 16 rows, and 0.99 to 1.01 at
1 and 512 rows.
2026-10-07 16:49:37 +03:00
Niko MaroulisandGeorgi Gerganov 7e8324f5fe metal : fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT (#30100)
* metal : fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT

ggml_metal_op_mul_mat_mma picks the residual of a fused MUL_MAT+ADD as
"the ADD operand whose op is not MUL_MAT". When both operands of the ADD
are mat-mul outputs (x = W1 @ u + W2 @ v), that test is true for both, so
the residual resolves to the fused mat-mul's own, never-written output and
the kernel adds whatever that buffer holds.

The fusion check (ggml_metal_mul_mat_add_operand) already selects the
operand by identity; make the encoder do the same.

Clef decision models hit this in their head (proj_option_context @ ctx +
proj_option_lexical @ lex, 9 option rows): on Metal, /v1/systemone
probabilities collapse toward uniform (billing 0.28 where the CPU backend
gives 0.977, Cloudflare_clef-flash Q8_0), deterministic per memory layout,
correct with GGML_METAL_FUSION_DISABLE=1. Not a quantization issue: the
same file is right on CPU.

Add a MUL_MAT_ADD mode to test-backend-ops where the residual is a second
mat-mul; on Metal it fails 27 of 28 cases before this change (the one pass
is f16 n=2, under the MMA row threshold, so nothing fuses).

* Update tests/test-backend-ops.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-07 16:41:50 +03:00
R0CKSTAR ad21565331 musa: use the tile lightning indexer kernel (#30080)
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
2026-10-07 11:22:56 +02:00
Masashi Yoshimura 36a73916ee ggml-webgpu: fix flash_attn supports_op check for overlapping KV (#28205) 2026-10-07 09:56:53 +02:00
Neo ZhangandGeorgi Gerganov 005a1e127a [SYCL] fix the issue in mixed different model GPUs in FA (#29071)
* fix mixed different model GPUs issue

* Update ggml/src/ggml-sycl/ggml-sycl.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-07 10:24:43 +03:00
Anant Shrivastava d2a79e6046 sycl: accelerate GLM MLA prefill with MKL flash attention (#29171)
* sycl: accelerate GLM MLA prefill with MKL flash attention

GLM-4.7 Flash uses an MLA shape with 576-wide Q/K heads, a 512-wide
V head, GQA 20, and F16 KV. The SYCL dispatcher rejects this shape
because the normal MKL flash-attention gate requires matching K/V
widths and caps the head dimension at 512, so prompt processing falls
back to the substantially slower TILE kernel.

Admit only the validated 576/576/512, GQA-20 F16 shape to the existing
MKL pipeline. Keep all other mismatched K/V shapes on their current
fallback paths.

Handle GLM's V cache as a narrower strided view of K rows. Select the
strided F16 descriptor when row stride is padded, and alias K/V
dequantization buffers only when their logical widths match. Restrict
the stride exception to a real V view sharing K's row stride.

Add the exact 576/512, GQA-20 prompt-path backend test.

On an Intel Arc Pro B70 at master e613ef2, pp8192 improves from
432.80 to 1292.29 tok/s (2.99x, +198.6%). tg256 remains unchanged
within noise at 45.67 versus 45.65 tok/s. The exact MLA test passes
and debug output confirms MKL dispatch.

* sycl: store MKL flash attention scores in F16

Keep the QK GEMM output in F16 instead of F32. The online softmax still
converts each score to F32 for its max, exponent, and sum, so the
per-element math is unchanged apart from score rounding, and the F32
matrix was being written only to be consumed as F16 probabilities.

The F32 score matrix is the largest flash-attention intermediate on this
path; storing it as F16 halves its size and traffic. This builds on the
coalesced softmax loads from 1aa2954bd, which read each score row
cooperatively, so the smaller dtype pays off.

Measured on an Intel Arc Pro B70 with the dispatch from the previous
commit, -ngl 999 -b 4096 -ub 1024 -ctk f16 -ctv f16 -fa on:

pp8192   1583.9 -> 1657.1 tok/s (+4.6%)
pp64000   610.0 ->  684.5 tok/s (+12.2%)
pp131072  ~354  ->  402.1 tok/s (+13.5%)

tg256 at 8k context is unchanged (32.64), and the FLASH_ATTN_EXT suite
shows no new failures. The exact GLM MLA backend cases pass against CPU.
Adjust the ~354 baseline figure if you prefer citing only measured pairs (the 131k dispatch-only point came from the equivalent maintained build). Optionally add Assisted-by: <tool name> per the contribution guidelines since AI contributed to the change.

* Revert "sycl: store MKL flash attention scores in F16"

This reverts commit 265f9748164dc88e02678425b75f520ccd528b32.
2026-10-07 10:24:07 +03:00
Pratyush Kumar 5e5b628eb5 sycl : fattn_kv_buffers cleanup (#27689) 2026-10-07 10:23:28 +03:00
Ruben Ortlam 4d756bc72b vulkan: fix amd iGPU slow checkpoint read (#30049) 2026-10-07 08:20:04 +02:00
Clemens Wasser 78651c410d sycl: add IQ3_S multi-column MMVQ (#29500) 2026-10-07 13:28:23 +08:00
Mendy Berger f498f864fb ggml-webgpu: no dawn native features on wasi (#27069) 2026-10-07 13:29:30 +09:00
Max Krasnyansky c479922ac5 hexagon: CPY/CONCAT/CONT/DUP overhaul to use DMA/HVX for all cases (#30067)
* hex-cpy: replace more paths with dma and simplify l2flush

* hex-cpy: use DMA in all sametype paths

* hex-cpy: rewrite the rest of the copy paths (diff type) to use dma

* hex-concat: use dma for multi-dev path which also removes the need for l2-line alignment

* hex-concat: proper support for mdev splitting

* hex-concat: cleanup ctx and kern params usage

* hex-cpy: cleanup contex and remove left-over non-dma checks

* hex-cpy: clean dma_cpy naming

* hex-cpy: proper kernel params and kernel selection

* hex-build: resolve left-over rebase conflicts

* hex-cpy: update dev guide to clarify 128 byte alignment requirement

* hex-concat: make sure we go through mdev barrier

* hex-concat: make sure to flush dma-queue

* hex-cpy/concat: cleanup kparams and vtcm layout handling

* hex-cpy: remove dead check for contig (routed to diff kernel) and update comments

* hex-dev: update developer guide based on latest changes

* hex-cpy: safe skip of noop copies

* hex-dup: route DUP to CPY
2026-10-06 15:24:17 -07:00
qiao_px 5ad1c5da0a cuda : add BF16 support for XIELU (#29955)
The XIELU CUDA kernel template is already generic over the element
type; only the F32/F16 type assertion and the else-if dispatch were
missing. Add the nv_bfloat16 branch to the launcher, and drop the
temporary supports_op gate in ggml-cuda.cu that rejected BF16+XIELU.

test-backend-ops gains two BF16 cases ([10,5,4,3] and [512,16,1,1]).
docs/ops/CUDA.csv and docs/ops.md are regenerated; the F32 xIELU row
flips from no to yes as well, i.e. the previous record was stale.

Tested:
- Mac CPU: xIELU F32/F16/BF16, 6/6
- Mac Metal: existing F32/F16, 4/4; BF16 still unsupported
- RTX 4090 CUDA: xIELU F32/F16/BF16, 6/6
- RTX 4090 CUDA BF16-only: 2/2
- git diff --check passes
2026-10-06 22:36:21 +02:00
Harkirat Gill 51ce9c11a6 ggml-cuda: use per-thread stream for buffer-init padding memset (#28782)
* ggml-cuda: use per-thread stream for buffer-init padding memset

* ci : re-enable test-backend-ops -j for ROCm
2026-10-06 21:04:05 +02:00
lhez 2207c8e57c opencl: fix OOB read in adreno xmem GEMM (#30041) 2026-10-06 09:15:25 -07:00
Aman GuptaandGeorgi Gerganov a46709b683 RPC: add -sm tensor (#26610)
* rpc: allow -sm tensor

* fix flush for apple rdma

* move graph_uids to rpc_dispatcher

* cont : fix conflict

* cont: stop spinning dispatcher thread

* remove meta backend change

* add TODO to simplify logic

* rpc: bump major version

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-06 20:55:39 +05:30
Pascal 65840ed53c ggml: fix CLAMP on non-contiguous views (CPU, CUDA) (#29517)
* ggml: fix CLAMP on non-contiguous views (CPU, CUDA)

CUDA clamped ggml_nelements values flat and ignored the view strides.
CPU addressed row j as j*nb01 and ignored nb02/nb03. Both now follow the
strides of dims 1..3; CUDA supports_op requires contiguous rows, like
Metal. test_clamp gains a non-contiguous view case.

* cuda: clamp kernel uses fastdiv for the view strides
2026-10-06 16:18:34 +02:00
thelittlefiremanandJohannes Gäßler ab09ea4c14 cuda: BF16/FP16 conversion to f32 chunking (#29442)
* ggml-cuda: chunk large BF16/FP16 to F32 conversions

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* ggml-cuda: respect dst stride in chunked cuBLAS matmul

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-10-06 17:16:49 +03:00
Foad Abo Dahood a043d38a62 metal : fix excess threadgroup memory in quantized flash attention (#29340) 2026-10-06 16:40:16 +03:00
Ian McKellar 58cb9138e4 vulkan : check for null vkEnumerateInstanceVersion (#29872)
A 1.0 loader (e.g. Android 8.1) has no vkEnumerateInstanceVersion, so
backend init called a null pointer. Treat it like any loader under 1.2.

Fixes #29871.

Assisted-by: Claude Opus 5.5
2026-10-06 15:28:54 +02:00
Johannes Gäßler 4f54067615 HIP: use -O0 for host code in debug builds (#29795) 2026-10-06 13:53:00 +02:00
Aman GuptaandGeorgi Gerganov 6753a033f0 ggml: refactor selective expert copying to user code (#29943)
* ggml: refactor selective expert copying to user code

* tests: enroll two models into selective expert copy test

* tests: use deepseek2 as test model

* improve comment in ggml-backend.h

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* cont: fix whitespace

* cont : better comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-06 10:59:45 +03:00
Ravi PanchumarthyandMustafa Cavus b9a5a00b86 ggml-openvino: fix CI tests; fix GPU regressions. (#30037)
* ggml-openvino: skip unselected graph branches and support DUP

Upstream #29622 adds a mixed token/embd branch to every input
embedding graph through ggml_build_forward_select(). Its nodes are
not flagged for compute, but the backend translated them anyway,
and the DUP in that branch was unsupported, so the scheduler split
the graph and passed the embeddings across the split with a fixed
token count. The first single-token decode then failed
(test-thread-safety on CPU and GPU).

Build the OV model from the compute nodes only, and translate a
same-type contiguous DUP like CONT so the graph stays on one backend.

* ggml-openvino: make inp_scale_rows token dim dynamic

#29622 also moves the per-token embedding scale (gemma3, gemma3n,
gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim
and pad it per chunk on the static (NPU) path.

* ggml-openvino: skip GPU MUL_MAT op tests with unbound Q4_1/Q4_K weights

Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU
plugin fails to compile that form for some row counts with "clFinish,
error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on
the MUL_MAT cases added in #29869 (e.g. m=1000, n=2, k=1024). Model
weights use a u4 zero point and are not affected.

Report these cases as unsupported on GPU until the plugin is fixed.
Op tests check support before allocating, so the check matches unbound
weights only; model loading probes with a dummy buffer and keeps its
weights on the GPU.

* ggml-openvino: create FILL in the output type

translate_fill always built an f32 constant, so an f16 FILL produced
f32 data and the copy back overran the f16 output buffer. Use the
output type for the constant.

* ggml-openvino: reject CONCAT with a quantized type

Quantized inputs are dequantized when translated, so the backend cannot
write a quantized CONCAT output. Report it as unsupported, as for CPY
to a quantized type.

* ggml-openvino: handle the single recurrent state gather of build_rs

#29856 changed build_rs to gather all recurrent states with one GET_ROWS
on the s_copy leaf and take the ubatch and extra states as views of it.
The stateful path matched only the previous form, a GET_ROWS per view of
s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU
("is_axis_valid(axis, r)" in a Concat).

For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the
active-state gather, keep the rank-4 layout of reshapes that read a view
of it, and map the copy of the empty extra-state view to the single-slot
remainder writeback. Do not warn about the dynamic dim of empty views.

* openvino: align eltwise operand ranks to work around a GPU-plugin defect

* openvino: match the MoE fusion on the rank-3 stateful graph

* ggml-openvino: do not unsqueeze an RMS norm output in AlignEltwiseOperandRanks

The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract
whose operand ranks differ. In gemma-3 the lower-rank operand of the
post-attention residual add is the norm output, and unsqueezing it makes
the GPU plugin compute the layer wrongly: gemma-3 returns empty answers
on GPU with stateful execution.

Skip the rewrite when the lower-rank operand is an RMS norm output.

* docs : update OpenVINO validated models

---------

Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
2026-10-06 09:57:04 +03:00
Todor BoinovskiandMax Krasnyansky 5e03bdd870 hexagon: ssm-conv updates (#29971)
* hexagon: ssm-conv double-buffered DMA for prefill and decode restructuring

* hex-ssm-conv: remove divs from loops and fix trace events

* hex-dma: improved SSM_CONV dma pipeline and streamlined dma_queue

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-05 17:42:40 -07:00
Aparna M PandMax Krasnyansky 50569eb87d hexagon: add pool op support (#29995)
* hexagon: add pool_2d support

* hexagon: add pool_1d support

* hex-pool: dma changes

* hex-pool: Optimize HTP pooling boundaries and DMA pipelining

* hex-pool: code cleanup and correctness fixes

* hex-pool: re-write the DMA pipeline

* hex-pool: pool chunking support

* hex-pool: remove/vectorize all scalar paths

* hex-pool: simplify chunk solver (no need for a loop)

* hex-pool: remove redundant checks

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-05 15:34:32 -07:00
8345f33395 hexagon: matmul and flash-atten scalability updates (#29974)
* hexagon: head-parallel flash_attn partitioning for row-split multicore

In row-split mode each core computes its output row shard of every
MUL_MAT, but flash_attn was previously partitioning by Q tokens
(flat qrow split) instead of by heads. This forced every core to
read the full KV cache (all n_kv_heads), negating the memory
bandwidth benefit of multicore on flash_attn.

Change both HMX and HVX flash_attn kernels to partition by KV heads
when n_kv_heads is divisible by n_cores: core i processes heads
[i*n_kv_heads/N, (i+1)*n_kv_heads/N) exclusively, reading only its
head shard of the KV cache. Falls back to the original token-block
split when n_kv_heads % n_cores != 0 (e.g. Gemma-4 with 2 KV heads
on 4 cores).

Controlled by GGML_HEXAGON_FA_HEAD_SPLIT (default 1 = on).
The flag is packed into bit 1 of the existing is_dst_fp32 kparams
byte to stay within the 128-byte kernel_params blob limit.

Measured gains at 4c row-split (PP t/s, ubatch=1024):
  Qwen3-0.6B:    6977 -> 11026  (+58%)
  llama-3.2-3B:  3717 ->  5522  (+49%)
  Qwen3.5-4B:    2739 ->  2855   (+4%)
  Gemma-4 MoE:   no change (MoE FFN dominates, fallback path)

TG is unchanged (flash_attn is a small fraction of decode time
relative to the matmul+barrier cost per layer).

* hex-fa: cleanup kern_params and head-split selection

* hex-fa: add -fa-head-split option to run.py

* hex-mdev: update matmul solver to account for reduced work in row-split scenarios

* hex-mmid: better work splitting by expers in multi-dev scenarios

* hex-fa: update HMX gating based on the model/n-hvx/ctx-len sweep

* hex-fa: precompute softcap/scale on the host

* hexagon: flatten matmul into 2d to use HMX in multi-sequence

* hex-mm: cleanup kparams and use collapse to 3/4D -> 2D mapping

* hex-mm: fix typo in collapse fallback

* hex-mm: another pass at consistent naming for act tensors

* hex-mm: add support for colapsing dims in fused matmuls

* hex-build: fix WoS build errors

* hex-mm: make sure to enforce dst stride in can_collapse

* hex-fa: add a onliner commit for head-split check

* hex-fa: remove unused local head_split var

* hex-fa: tighten up can_split checks

* hex-mm: update unfused paths to use act instead src1

* hex-mm: make sure to check all dsts for splitting

* hexagon: fix the second weight chunk address in the batched HMX matmul prologue

* hexagon: F16 activation and ragged N in the HMX matmul

* hex-mm: tighten the ragged/split checks in mdev cases

* hex-mm: enable MM fusion for F16 activations

* hex-mm: pass tiled sizes to the solver in fused paths

* hex-mmid: remove scalar divs from expert mapping loops

* hex-mmid: proper cacheline safety enforcement for mdev splits

* hex-mm: improve solver for mdev split scanarios and tail handling

* hex-mm: remove redundant checks

* hex-mm: fix fused HMX MUL_MAT_NX drops the final partial tile for quantized weights

* hex-mm: better handling of ragged shapes (removes scalar memset of vtcm)

---------

Co-authored-by: ebateni <ebateni@qti.qualcomm.com>
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
2026-10-05 08:55:21 -07:00
Aman Gupta e117148a41 CUDA: make the alloc_deps check batch independent (#29986)
Fixes #29980
2026-10-05 19:14:23 +05:30
Ruben Ortlam 6c59c40076 vulkan: fix Flash Attention shmem write out of bounds (#29988) 2026-10-05 15:15:00 +02:00
virajwad 3c9e747f7e vulkan: revert mul_mat_id tile selection PR #29182 (#29936)
* Fix Intel prefill regression on MoE models

* Revert the n_per_expert change back to nei1
2026-10-05 16:11:21 +03:00
Pascal b809b886d9 cuda: use the vector lightning indexer kernel on MUSA (#29990)
* cuda: stage the lightning indexer queries in head passes for MUSA

MUSA archs 21 and 22 cap static shared memory at 28 KB, and the tile
kernel staged the queries of all four heads next to the key tile for
33 KB. The queries are now staged in passes of
LIGHTNING_INDEXER_TILE_HEADS_PER_PASS heads: two on MUSA for 25 KB,
four elsewhere where the single pass folds to the previous kernel.

* cuda: use the vector lightning indexer kernel on MUSA

Address review from am17an: the tile kernel stays off MUSA, whose archs
21 and 22 cap static shared memory at 28 KB, below the 33 KB the tile
needs, so MUSA keeps the vector kernel it ran before. This replaces the
head passes, CUDA and ROCm run the merged kernel unchanged.
2026-10-05 15:09:53 +02:00
Konrad Moren 8b2fbaf32c CUDA: Optimize accumulation in mmq for NVFP4 type (#29857)
* ggml_cuda: optimize accumulation in mmq_vec_dot_fp4_fp4_mma for better performance

* remove whitespace

* fix: correct indentation in mma_block_scaled_fp4 loop
2026-10-05 13:43:07 +02:00
François-Xavier Gsell 806eee9841 vulkan: fix stale prealloc_y reuse across flash attention and soft_max (#29591)
Assisted-by: Claude
2026-10-05 10:38:26 +02:00