Commit Graph
3800 Commits
Author SHA1 Message Date
Saejin Kim-HeinertandSigbjørn Skjæret b54752df40 ggml-cpu: add x86 VNNI Q2_0 dot product (#26348)
* ggml-cpu: use VNNI for Q2_0 dot products

Keep the upstream AVX2 unpacking, accumulation, and fallback. Use VNNI for the integer dot product on supported x86 CPUs.

Assisted-by: Codex

* ggml-cpu: use VBMI for Q2_0 unpacking

Use multishift to unpack Q2_0 weights on CPUs with AVX-512 VBMI and VL. Retain the existing shift-and-mask unpacking on other CPUs.

Assisted-by: Codex

* Update ggml/src/ggml-cpu/arch/x86/quants.c

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-15 16:19:33 +01:00
uvos 278a89cb78 CUDA/HIP: Refactor the use of _sync warp primitives (#31617)
Uses the rocm-native primitives, increases the rocm version requirement to 6.3 and makes top-k selectable warp.
2026-11-15 10:58:36 +01:00
shaofeiqi 6d3568e31d opencl: add A7 Q6_K non-MoE binary kernel (#30949) 2026-11-14 21:49:26 -08:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5c008e0008 feat: add Inkling 2 support (text + vision) (#32217)
* convert : add Inkling2 text GGUF conversion

Assisted-by: Codex

* llama : implement Inkling2 text inference

Assisted-by: Codex

* tests : cover Inkling2 text support

Assisted-by: Codex

* conversion : dequantize packed Inkling2 experts in row chunks

Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.

Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.

Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.

Assisted-by: Codex

* convert : add Inkling2 vision mmproj support

Assisted-by: Codex

* mtmd : implement Inkling2 still-image encoder

Assisted-by: Codex

* inkling2 : scope image loading

Assisted-by: Codex

* inkling2 : enable flash attention with folded bias and sinks

Assisted-by: Codex

* cuda : route per-head flash attention masks through vec

Assisted-by: Codex

* mtmd : support Inkling2 temporal frame means

Assisted-by: Codex

* style : restore shared image comment

Assisted-by: Codex

* tests : enlarge Inkling2 context fixture

Assisted-by: Codex

* inkling2 : load per-layer expert feed-forward widths

Assisted-by: Codex

* tests : add Inkling2 Metal fusion baselines

Assisted-by: Codex

* inkling2 : group and align registry entries

Assisted-by: Fable

* convert : regenerate the Inkling2 tokenizer hash with the update script

Assisted-by: Fable

* conversion : revert the print_registered_models change

Assisted-by: Fable

* conversion : sort the Inkling2 model map entries

Assisted-by: Fable

* convert : map Inkling2 vision checkpoint names in the tensor table

Assisted-by: Fable

* convert : map Inkling2 text checkpoint names in the tensor table

Assisted-by: Fable

* tests : document the Inkling2 packed expert conversion test

Assisted-by: Fable

* tests : cover Inkling2 checkpoint and transformers tensor names

Assisted-by: Fable

* llama : place Inkling2 fields and list entries by file convention

Assisted-by: Fable

* gguf-py : align Inkling2 tensor mapping comments

Assisted-by: Fable

* mtmd : place Inkling2 entries by file convention

Assisted-by: Fable

* tests : place Inkling2 cases after the existing ones

Assisted-by: Fable

* cuda : index per-head flash attention masks in the tile and mma kernels

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : project the relative bias before the gather, drop arch branches

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* convert : require Inkling2 hparams, add inkling2 pre-tokenizer

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* mtmd : cap Inkling2 image size, drop video frame averaging

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : fix CI build, Meta device assert and Windows conversion test

Assisted-by: Claude

* tests : use the default sliding window for the Inkling2 test model

Assisted-by: Claude

* simplify mtmd part

* give _v arch for vision

* remove alpha channel handling, it should be a follow up

* explicit rule about audio + vision models

* add audio support

* rm audio

* rm gguf mtmd rope base

* address review comment

* nits

* do not normalize embd from mtmd

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-15 00:51:31 +01:00
Pranav DhinakarandMax Krasnyansky dca216ad63 hexagon: add TIMESTEP_EMBEDDING HVX kernel (#32228)
* hexagon: add TIMESTEP_EMBEDDING HVX kernel

* hex-timestep: add dma, improve kparams, and work-splits, and tracing

* timestep: add testcases that cover WAN2.2

* hex-timestep: relax the gates to cont_rows

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-14 15:03:31 -08:00
Aaron Teo acf5d54562 llamafile: introduce more Q0 quant support for s390x (#32203)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-14 23:38:06 +01:00
Richard Palethorpe 3bc612fe23 vulkan: Explicitly include SPIRV-Headers (#31416)
* vulkan: Explicitly include SPIRV-Headers

Distros that don't put the headers in a system include path need
to specify it to the compiler. So we use the appropriate CMake
target to include it.

Assisted-by: Codex

* vulkan: Add explicit headers location to workaround OSX CI runner conf
2026-11-14 23:37:14 +01:00
Ruben Ortlam d8f30a5b6d vulkan: optimize prompt processing Flash Attention for RDNA3 and RDNA4 (#31323)
* vulkan: optimize Flash Attention cm1 for RDNA3 and RDNA4

* reduce shmem use through buffer reuse

* clean up and fixes

* limit q_in_regs to prefill

* vulkan: add flipped FA path, each subgroup owns 16 Q rows

* cleanup

* move flip shader into own file

* replace kvsep with direct load mode

* fix nvidia hang, tune AMD

* only store pv in sfsh if necessary

* fix skip condition

* nvidia tuning

* reduce occupancy on RDNA4 for large attention head sizes

* fix sparse FA fp16 underflow bug
2026-11-14 23:36:58 +01:00
Kartik Sirohi e964175653 cuda : keep columns in registers for cooperative softmax (#28982)
Template soft_max_f32_parallelize_cols_single_row to keep column
elements in registers when they fit, reducing memory traffic from
5 global memory passes to 2 passes.

Assisted-by: Gemini 3.8 Flash
2026-11-14 23:36:34 +01:00
Jeff BolzandRuben Ortlam d20426bbbe vulkan: optimize matrix multiplication for narrow N (#31990)
* vulkan: optimize matrix multiplication for narrow N

Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.

* Attempt to handle wave64

* add n=16 tile for cm1

* apply RDNA split_k heuristic to narrow_n generally

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-14 23:36:05 +01:00
Ruben Ortlam 19f31e61b0 vulkan: use GPU copies on UMA with uncached memory on Intel (#32024) 2026-11-14 23:35:54 +01:00
Nicolas Mowen 07037ba133 sycl : optimize mtmd image processing (#32109) 2026-11-14 23:35:16 +01:00
lhez 1ae047c926 opencl: fixes for q4_k mv oob access, f16 swiglu_oai, cpy with strided dst, batched group_norm and bf16 sigmoid (#32251)
* opencl: fix q4_K mv oob read

Assisted-by: Codex

* opencl: fix kernel dispatch and oob writes for  f16 swiglu_oai

Assisted-by: Codex

* opencl: fix the row broadcast for add/mul

Assisted-by: Codex

* opencl: respect destination tensor strides for cpy

Assisted-by: Codex

* opencl: handle batches in ne3 for group_norm

Assisted-by: Codex

* opencl: avoid double rounding for bf16 sigmoid

Assisted-by: Codex
2026-11-14 23:32:48 +01:00
Jhen-Jie Hong c6c0ec6d5c hexagon: fix fa no-mask tail (#32243) 2026-11-14 10:06:01 -08:00
Aman Gupta c56be864f4 ggml-backend : start a new split at an incompatible weight in a split without inputs (#31775) 2026-11-14 22:14:48 +08:00
Ehsan BateniandMax Krasnyansky 61dbf49228 Hexagon: add support for IQ2_XXS and NVFP4 (#31954)
* ggml-hexagon: IQ2_XXS and NVFP4 tiled repack layouts

* ggml-hexagon: IQ2_XXS HVX matmul kernel

* ggml-hexagon: NVFP4 HVX matmul kernel

* ggml-hexagon: NVFP4 per-tensor scale

* ggml-hexagon: HMX paths for IQ2_XXS and NVFP4

* ggml-hexagon: supports_op coverage for NVFP4 and IQ2_XXS

* fix HMX IQ2_XXS dequant scale underflow in fp16

* hexagon: IQ2_XXS 2-gather tile decode with compact grid, small-weight tests

* IQ2_XXS vpopcount sign parity and gather pipelining

* pipeline IQ2_XXS HMX dequant gathers and add large_w tests

* hex-iq2-xxs: rebase and cleanup

Remove unused helper functions.
Add lut_size to the overall vtcm required size.
Trim down (way way down) the new tests.

* hex-iq2-xxs: actually trim down the tests

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 20:57:05 -08:00
Aman Chadha(IVIXMMI) 9183a552e5 metal : fix mul_mv destination offset overflow (#32221) 2026-11-14 07:49:32 +08:00
Todor BoinovskiandMax Krasnyansky ded522f830 hexagon: add support for SSM_SCAN op (#31921)
* hexagon: add SSM_SCAN Op support

* hex-ssm-scan: fix multi-device spliting

* hex-ssm-scan: remove redundant checks and make kernel use host config

* hex-ssm-scan: remove direct ddr access

* hex-ssm-scan: remove DMA ordering

* hex-ops: update supported ops

* hex-ssm-scan: rewrite the dma pipeline to match other ops (dbl.buf, etc)

* hex-ssm-scan: remove all fallbacks, only dma/vtcm paths

* hex-ssm-scan: add kernel_type to kparams and remove redundant recomputes

* hex-ssm-scan: avoid stalls in the overall pipeline (dma flush between chunks)

* hex-ssm-scan: vectorize softplus

* hex-ssm-scan: optimize hvx loops

* hex-ssm-scan: add HMX support

* hex-ssm-scan: add support for GGML_HEXAGON_SSM_SELECT

* hex-ssm-scan: cleanup kernel names

* hex-ssm-scan: simplify mdev checks

* hex-mdev: improve mdev row-split solver

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 12:27:51 -08:00
thelittlefireman b79696f1f5 ggml-cuda: hip: tune GCN MMQ configs (#30021) 2026-11-13 18:12:51 +01:00
Max Krasnyansky 9daa0aa22b hexagon: improve buffer allocations and mappings (mmap stacking) (#32144)
* hex-bufs: combine queue and fence buffers

* hex-bufs: use n_threads for queue sizing

* hex-mmap: optimize buffer stacking at the host to avoid iova fragmentation

* hex-mmap: add simply retry for spurios mmap failures

* hex-mmap: use eager mode for extended mmap retries
2026-11-13 07:56:20 -08:00
ynankani fe14cad51b CUDA: add SM103 SASS to non-native builds (#30099)
Signed-off-by: ynankani <ynankani@nvidia.com>
2026-11-13 20:32:24 +05:30
anujj 31953bcf74 ggml : add BF16 support to RMS_NORM (CPU, CUDA) (#32135)
* ggml : add BF16 support to RMS_NORM (CPU, CUDA)

* cuda : use one helper for the F32/BF16 RMS_NORM launches
2026-11-13 15:41:17 +02:00
Yonghye Kwon ae93877c8f vulkan: add TOP_K_MASK (#31544)
* vulkan: add TOP_K_MASK

Assisted-by: Claude Opus 5.5

* vulkan: fix the TOP_K_MASK barrier and use the common offset helper

Assisted-by: Claude Opus 5.5

* vulkan: declare ggml_vk_top_k_mask in the common header

Assisted-by: Claude Opus 5.5
2026-11-13 14:02:36 +01:00
8c2c4c39c5 ggml-vulkan: add optional device keepalive (#29267)
* ggml-vulkan: add optional device keepalive

* simplify implementation, move ownership to backend, change env var to seconds

---------

Co-authored-by: Acmmi <acmmi@Acmmis-MacBook-Air.local>
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-13 13:53:55 +02:00
Pascal 2d8a325029 CUDA: split the PAD_REFLECT_1D grid to stay within the HIP grid limit (#32179)
The kernel launches one block per row and 256-wide tile, so the
{2, 16777217} test case asks for 16777217 blocks of 256 threads. HIP
refuses a grid dimension above 2^32 threads and the ROCm jobs fail
with "invalid configuration argument".

The host now launches the grid in chunks of at most UINT32_MAX / 256
blocks and passes the chunk offset to the kernel. Real sizes fit in a
single chunk and keep their timings.
2026-11-13 11:15:25 +02:00
Georgi Gerganov 3bfde28f91 ggml : bump version to 1.0.0 (ggml/1690)
* ggml : bump version to 1.0.0

* examples : bump test-cmake (ggml/0)
2026-11-13 10:37:44 +02:00
SaidSultaniandMax Krasnyansky 92d6c8a5ca fix(hexagon): make host build work with clang-cl on Windows ARM64 (#32011)
* fix(hexagon): make host build work with clang-cl on Windows ARM64

Two CMake fixes for building ggml-hexagon natively on Windows-on-ARM64
with x64-hosted clang-cl (validated on Snapdragon X Elite, discussion in
qualcomm/GenieX#1390):

- Some clang drivers (e.g. x64-hosted toolchains used on ARM64) don't
  predefine the arch macro, which makes winnt.h fail with "No Target
  Architecture". Define _M_ARM64/_ARM64_ explicitly for the host TUs.
- The Hexagon SDK's incs/stddef include (AEEStdDef.h and friends) poisons
  the MSVC header chain when it leaks onto dependent TUs' include paths
  via htp_iface's PUBLIC includes. Keep it PRIVATE to the interface stub
  and system-level for the backend's own TUs (which include <AEEStdErr.h>
  directly).

Originally validated 2026-08-26 as reported in the issue; applies
cleanly to 15759e0.

* hex-cmake: trim down excessive comments

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-12 13:52:56 -08:00
Georgi Gerganovandlosewayy 29dbbad2b1 ggml : prune (#29801)
* ggml : prune deprecated API

* cont : bump RPC version

* cont : minor

* ggml : remove outdated documentation from ggml.h

The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:

- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`

The block also ended with 5 empty `TODO` sections.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)

* rpc : bump RPC protocol major version

The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.

The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: losewayy <1134285612@qq.com>
2026-11-12 20:58:24 +02:00
Aparna M P 2a4daf0e08 hexagon: add Q5_0 and Q5_1 support (#32094)
* hexagon: add q5_0 and q5_1 support

* hex-mm: dedicated q5_0 kernels
2026-11-12 08:33:16 -08:00
anujj 0fb8f15770 Title: CUDA: few-row MMQ for BS 2-24 (#31533)
* CUDA: few-row MMQ for 2-24 src1 columns

New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.

* nit fixes

* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat

* few-row quantize: DS4 sum from the quantized values, as in #31247

* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build

* few-row MMQ: dispatch order MMVQ, few-row, MMQ
2026-11-12 21:12:45 +08:00
Kong Jiale a8575a3ef4 ggml : route GATED_DELTA_NET to the SpacemiT RVV kernel (#31897)
The SpacemiT compute case was commented out, so the op stayed on the scalar CPU path. Type dispatch stays with the other ops, and use_ref keeps the reference path on the generic CPU kernel.

Assisted-by: Codex

Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
2026-11-12 12:20:25 +02:00
sliu39 a963f65957 vulkan : Intel FA kernel for prefill. (#29357)
* vulkan : Intel customized FA shader for prefill usage

* vulkan : adding check of max_bias/logit_softcap for Intel FA kernel path

* vulkan : add m16n16k16/SIMD32 support for Xe FA prefill shader

* vulkan : Intel FA shader and host refinement, with accuracy issues fixes

* vulkan : Limit the SLM usage to 48 KiB for Intel prefill FA shader

* vulkan : Update Intel FA shaders to handle cases with large head dimension K, unaligned Q head stride and multi head mask cases

* vulkan : Intel FA shader and host refinement to align with existing design, extreme cases handling, shape check relaxing.

* vulkan : Intel FA shader and host update to support q as tensor view, unaligned batch stride q. Add env variable for enforcing XE FA path for other vendor GPUs with 16x16x16 CM1 and SIMD32 support

* vulkan : Inte FA shader and host update to fix the potential SLM over 48 KiB issue, enhance the buffer access robust and add asymmetric qk/pv ration support.

* vulkan : Inte FA shader and host update to add SLM 48 KiB size check, remove unnecessary extensions and finetune the unroll/dont_unroll loops

* vulkan : Intel FA shader and host update to add SLM size check function, use real device max SLM size in shader specialization constants.

* vulkan : Intel FA shader and host update to consolidate host helper function, remove redundant buffer sync, bypass unused pipeline and buffer, fix subgroup shmem conflict and add negative case handling.

* vulkan : Intel FA host update to correct get_xe_fa_tuning_params() function log message, and add workgroupMemoryExplicitLayout/workgroupMemoryExplicitLayout16BitAccess prerequisite‌ feature check.
2026-11-12 11:19:17 +01:00
Marco Colombo ab4a77b9ad ggml-cpu: use the 4x4 simd_gemm tile on ARM NEON (#32128)
Assisted-by: Claude Fable 5.2
2026-11-12 12:13:59 +02:00
Konrad Moren 0d69255c52 CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints (#26048)
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back

* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ

* ggml-cuda: Tests for the fused scale

* ggml-cuda: reuse existing graph to test fusion in mmq epiloque

* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency

* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions

* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
2026-11-12 10:28:36 +01:00
Aman Gupta 3bce8bd50c ggml-backend : prefetch only inputs computed by earlier splits (#32100)
* ggml-backend : prefetch only inputs computed by earlier splits

The copy callback of a split runs after the prefetch and can write tensors that a later split reads, e.g. the MoE cache uploads missed experts into its bank. A prefetch of such a tensor reads it before the write.

* llama : skip the MoE cache when its device cannot run the cached mul_mat_id

Otherwise the scheduler runs the matmul on another backend and copies the whole bank every ubatch.
2026-11-12 17:24:40 +08:00
Kong Jiale 81f21f4b27 ggml : dispatch SpacemiT Q4_1 32x256 through the HP kernel (#31929)
The 32x256 pack already keeps a separate uint8 zero point. Complete the M1 path and use the same HP kernel as Q4_0.

Assisted-by: Codex

Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
2026-11-12 10:59:09 +02:00
Titaniumtown 09338b85e8 sycl: make oneDNN FA cache thread-safe and memoize runtime failures (#31138) 2026-11-12 08:58:26 +02:00
Neo Zhang dcbff21661 sycl : fix AOT for more than 4GB (#31873)
* fix aot for >4g garbage output issue; add --aot in build script

* correct the aot flag

* support no-xmx build, update the code for no-xmx building, rename nossl to no-ssl
2026-11-12 08:54:58 +02:00
zhaoyshineandzhaoyshine f3d7725a77 sycl : use cached device-id in set_main_device (#32013)
ggml_sycl_set_main_device() runs at the start of every graph compute. Its
early-return check called dpct::get_current_device_id(), which locks the
device manager mutex and looks up the thread-to-device map. Use the
thread-local cached id that the rest of the backend already uses.

Assisted-by: Claude Code

Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
2026-11-12 08:50:55 +02:00
Joseph Patricio 7021f7e099 CUDA: add cuda debug buffer pool (#31919)
* CUDA: add cuda debug buffer pool

* CUDA: add code comment for debug pool with compute-sanitizer
2026-11-12 08:50:32 +02:00
shaofeiqiandLi He e0949d09b1 opencl: add bin kernel kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin (#30947)
* opencl: add A7 Q4_0 non-MoE binary kernel

* opencl: refactor

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-11 21:53:51 -08:00
Feiteng 5d4c8eeb24 CUDA: skip empty MoE tiles in MMQ stream-k fixup (#32067)
* CUDA: skip empty MoE tiles in MMQ stream-k fixup

mul_mat_q_stream_k_fixup loads ids_dst[col_low + j] for all j < J before
checking j_sup. mul_mat_q returns early on col_low == col_high, the fixup
kernel did not. mm_ids_helper emits one extra tile per expert that may be
empty; for trailing experts col_low == ncols_dst, so the load reads one
element past the J_best-1 padding added in #29953. The kernel returns at
j >= j_sup before storing anything, so results are unaffected.

Return early when j_sup <= 0, like the main kernel. The return is
block-uniform (tsj is loaded once per block) and precedes __syncthreads().

Add test_mul_mat_id_unused_tail to test-backend-ops: only the first
n_mats/2 experts are selected, so the trailing experts have zero rows and
their tiles are empty; m=33, k=512 keeps tile efficiency low enough that
stream-k splits along K and launches the fixup kernel.

* CUDA: drop the extra MUL_MAT_ID test case and the explanatory comment (review)
2026-11-12 11:45:56 +08:00
e1f7593c32 opencl: add decomposed FA for Adreno (#32046)
* opencl: add decomposed FA

Assisted-by: Claude Opus 5.5
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: build FA MQ kernel on first FA call

* so allows warmup to early build neccessary kernels

Assisted-by: Claude Opus 5.5

* opencl: handle inf for fully masked row

* opencl: refine condition for using fa decomp

* opencl: support dp4a for PV mm

Assisted-by: Codex

* opencl: add direct PV for FA decomp to avoid V transpose for small batch

Assisted-by: Codex

* opencl: enable direct pv by default

* opencl: expand X2E decomposed FA to smaller heads and batches

- Lower the default head dimension threshold from 256 to 128 on X2E.
- Allow DK64 with P8, no attention sinks, GQA >= 2 and n_kv >= max(1024, 2*n_q).
- Allow P8 with V transpose for 2-31 queries when direct PV is unavailable, DK >= 128 and n_kv >= 1024.

Assisted-by: Codex

* opencl: adjust format

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-11-11 16:18:24 -08:00
CC 4180722eeb metal : prevent out-of-bounds reads in SOLVE_TRI kernel (#32103) 2026-11-11 22:02:06 +02:00
Ruben Ortlam 055c3c26ef vulkan: support batched tensor copy API (#31418)
* vulkan: support batched tensor copy API

* allocate staging buffer only for copies that actually need it
2026-11-11 21:56:39 +02:00
Tarek DakhranandClaude Opus 5.5 2e325dcd57 backend : re-instantiate CUDA graphs when the cluster dims of a kernel change (#31284)
* CUDA: re-instantiate graphs when the cluster dims of a kernel change

* CUDA: keep the cluster dims with the graph instance, check them only on Hopper+

Assisted-by: Claude Opus 5.5

* CUDA: use CUDA_CHECK and a helper struct for the graph cluster dims

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* CUDA: add a TODO for the graph cluster dims test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:00:45 +02:00
fox5611 432433637b ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call (#31838)
* ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call

Assisted-by: Claude Opus 5

* ROCm/HIP: corrected by only setting the workspace on ROCm 7.0 or newer
2026-11-11 12:32:39 +01:00
Pascal 7f24aacde3 cpu: accumulate flash attention V in F32 (#29810)
* cpu: accumulate flash attention V in F32 in the one-chunk kernel

The one-chunk kernel accumulated an F16 V cache in an F16 running sum
that is only normalized at the end, which overflows past 65504 on long
contexts: inf, then NaN on every following token.

Every V type now accumulates in F32, like the tiled kernel. F16 rows go
through ggml_vec_mad_f16_f32, which converts and multiplies in the same
register (F16C/AVX512, NEON, RVV) with no round trip through a temporary
F32 row, so decode keeps master speed on x86 and loses about 4% at 4k
context on ARM, where master used native F16 FMA.

* cpu: widen the F16 V rows like ggml_vec_dot_f16_f32

ggml_vec_mad_f16_f32 required F16C and FMA, so AVX builds without FMA
fell back to a scalar loop and ran 3x slower than master. It now
follows the guards of ggml_vec_dot_f16_f32: AVX512, then AVX with F16C
through GGML_F32x8_FMA, then NEON and RVV, then the scalar loop.
2026-11-11 11:55:19 +01:00
Leon af6b5243d6 webgpu: add PAD_REFLECT_1D support (#31877)
* webgpu : add PAD_REFLECT_1D support

* webgpu: remove redundant PAD_REFLECT_1D contiguity check

* tests: simplify PAD_REFLECT_1D setup
2026-11-11 12:11:06 +02:00
Jeff Bolz d2b3eb9b60 vulkan: allow dequantized KV optimization with CM2 (#31526)
* additional FA test cases

Assisted-by: Codex

* vulkan: allow dequantized KV optimization with CM2

Enable the KV dequantization optimization with coopmat2, guarded by a heuristic
based on query tokens*GQA ratio.

While doing this we saw very large speedups for certain head sizes, and it was
because those head sizes were incorrectly treated as unaligned. So also fix
those to be treated as aligned, which should help for when the dequant
optimization doesn't apply.

* gqa uses nbq2 for the stride, so we need to check that value for alignment

* tests : use VARS_TO_STR18 for flash attention

Assisted-by: Codex
2026-11-11 12:08:06 +02:00