Commit Graph
622 Commits
Author SHA1 Message Date
Giorgio Sarno ebc9fb933e CUDA: use block reductions for whole-row pool_1d (#31855)
* CUDA: use block reductions for whole-row pool_1d

Assisted-by: Claude Code

* CUDA: template pool1d row kernel by ggml_op_pool
2026-11-17 15:51:15 +01:00
Neo Zhang dde9de82cb sycl : Fix FA mkl issue: a mask broadcast over heads or batches (#31536)
* fix the SYCL fa with mkl error of a mask broadcast over heads or batches; add argument cache-ram to start-svr.sh

* add argument cache-ram to win-start-svr.bat

* fix format issue, remove --cache-ram in script: move to another PR
2026-11-17 10:50:32 +02:00
Nikhil Jain 140a279874 webgpu : Implement timestep embedding (#31695)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* resolve merge conflicts

* save state

* save kernel

* add kernel back in

* fix merge issues with kernels

* updates

* update tests

* change invocation count and workgroup to be u32

* clang format

* clang format

* update docs

* format

* remove extra file
2026-11-17 10:41:37 +02:00
Jeff Bolz ef3848f9d7 vulkan: Support split_k for conv3d. (#29929)
This tries to use the 256x128 shape (coopmat2-only) with split_k when
it appears profitable. It reuses the mul_mat split_k reduce shader.
2026-11-16 15:25:12 +02:00
PascalandRuben Ortlam adf554d239 vulkan: make the radix top-k tie selection deterministic (#31931)
* vulkan: make the radix top-k tie selection deterministic

The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.

Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.

* optimize shader for AMD, Intel and Nvidia

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-16 11:41:44 +01:00
liminfei-amd f477c1b90a cuda : fix MUL_MAT_ID abort with odd src0 columns on AMD (#32385)
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.

Fixes #30576

Assisted-by: GitHub Copilot

Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
2026-11-16 17:53:06 +08:00
Ed Addario 5a4afd8077 tests: Fail test-backend-ops and exit with error status 1 when missing and/or unsupported types and/or unsupported backends are provided in the command line (#30279)
* Fail tests on missing or unsupported type

* Fix output

* Remove whitespace

* Fail if no tests ran regardless of backend support

* Revert unnecessary cosmetic change
2026-11-15 08:26:23 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5c008e0008 feat: add Inkling 2 support (text + vision) (#32217)
* convert : add Inkling2 text GGUF conversion

Assisted-by: Codex

* llama : implement Inkling2 text inference

Assisted-by: Codex

* tests : cover Inkling2 text support

Assisted-by: Codex

* conversion : dequantize packed Inkling2 experts in row chunks

Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.

Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.

Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.

Assisted-by: Codex

* convert : add Inkling2 vision mmproj support

Assisted-by: Codex

* mtmd : implement Inkling2 still-image encoder

Assisted-by: Codex

* inkling2 : scope image loading

Assisted-by: Codex

* inkling2 : enable flash attention with folded bias and sinks

Assisted-by: Codex

* cuda : route per-head flash attention masks through vec

Assisted-by: Codex

* mtmd : support Inkling2 temporal frame means

Assisted-by: Codex

* style : restore shared image comment

Assisted-by: Codex

* tests : enlarge Inkling2 context fixture

Assisted-by: Codex

* inkling2 : load per-layer expert feed-forward widths

Assisted-by: Codex

* tests : add Inkling2 Metal fusion baselines

Assisted-by: Codex

* inkling2 : group and align registry entries

Assisted-by: Fable

* convert : regenerate the Inkling2 tokenizer hash with the update script

Assisted-by: Fable

* conversion : revert the print_registered_models change

Assisted-by: Fable

* conversion : sort the Inkling2 model map entries

Assisted-by: Fable

* convert : map Inkling2 vision checkpoint names in the tensor table

Assisted-by: Fable

* convert : map Inkling2 text checkpoint names in the tensor table

Assisted-by: Fable

* tests : document the Inkling2 packed expert conversion test

Assisted-by: Fable

* tests : cover Inkling2 checkpoint and transformers tensor names

Assisted-by: Fable

* llama : place Inkling2 fields and list entries by file convention

Assisted-by: Fable

* gguf-py : align Inkling2 tensor mapping comments

Assisted-by: Fable

* mtmd : place Inkling2 entries by file convention

Assisted-by: Fable

* tests : place Inkling2 cases after the existing ones

Assisted-by: Fable

* cuda : index per-head flash attention masks in the tile and mma kernels

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : project the relative bias before the gather, drop arch branches

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* convert : require Inkling2 hparams, add inkling2 pre-tokenizer

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* mtmd : cap Inkling2 image size, drop video frame averaging

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : fix CI build, Meta device assert and Windows conversion test

Assisted-by: Claude

* tests : use the default sliding window for the Inkling2 test model

Assisted-by: Claude

* simplify mtmd part

* give _v arch for vision

* remove alpha channel handling, it should be a follow up

* explicit rule about audio + vision models

* add audio support

* rm audio

* rm gguf mtmd rope base

* address review comment

* nits

* do not normalize embd from mtmd

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-15 00:51:31 +01:00
Pranav DhinakarandMax Krasnyansky dca216ad63 hexagon: add TIMESTEP_EMBEDDING HVX kernel (#32228)
* hexagon: add TIMESTEP_EMBEDDING HVX kernel

* hex-timestep: add dma, improve kparams, and work-splits, and tracing

* timestep: add testcases that cover WAN2.2

* hex-timestep: relax the gates to cont_rows

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-14 15:03:31 -08:00
Jeff BolzandRuben Ortlam d20426bbbe vulkan: optimize matrix multiplication for narrow N (#31990)
* vulkan: optimize matrix multiplication for narrow N

Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.

* Attempt to handle wave64

* add n=16 tile for cm1

* apply RDNA split_k heuristic to narrow_n generally

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-14 23:36:05 +01:00
Ehsan BateniandMax Krasnyansky 61dbf49228 Hexagon: add support for IQ2_XXS and NVFP4 (#31954)
* ggml-hexagon: IQ2_XXS and NVFP4 tiled repack layouts

* ggml-hexagon: IQ2_XXS HVX matmul kernel

* ggml-hexagon: NVFP4 HVX matmul kernel

* ggml-hexagon: NVFP4 per-tensor scale

* ggml-hexagon: HMX paths for IQ2_XXS and NVFP4

* ggml-hexagon: supports_op coverage for NVFP4 and IQ2_XXS

* fix HMX IQ2_XXS dequant scale underflow in fp16

* hexagon: IQ2_XXS 2-gather tile decode with compact grid, small-weight tests

* IQ2_XXS vpopcount sign parity and gather pipelining

* pipeline IQ2_XXS HMX dequant gathers and add large_w tests

* hex-iq2-xxs: rebase and cleanup

Remove unused helper functions.
Add lut_size to the overall vtcm required size.
Trim down (way way down) the new tests.

* hex-iq2-xxs: actually trim down the tests

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 20:57:05 -08:00
Todor BoinovskiandMax Krasnyansky ded522f830 hexagon: add support for SSM_SCAN op (#31921)
* hexagon: add SSM_SCAN Op support

* hex-ssm-scan: fix multi-device spliting

* hex-ssm-scan: remove redundant checks and make kernel use host config

* hex-ssm-scan: remove direct ddr access

* hex-ssm-scan: remove DMA ordering

* hex-ops: update supported ops

* hex-ssm-scan: rewrite the dma pipeline to match other ops (dbl.buf, etc)

* hex-ssm-scan: remove all fallbacks, only dma/vtcm paths

* hex-ssm-scan: add kernel_type to kparams and remove redundant recomputes

* hex-ssm-scan: avoid stalls in the overall pipeline (dma flush between chunks)

* hex-ssm-scan: vectorize softplus

* hex-ssm-scan: optimize hvx loops

* hex-ssm-scan: add HMX support

* hex-ssm-scan: add support for GGML_HEXAGON_SSM_SELECT

* hex-ssm-scan: cleanup kernel names

* hex-ssm-scan: simplify mdev checks

* hex-mdev: improve mdev row-split solver

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 12:27:51 -08:00
anujj 31953bcf74 ggml : add BF16 support to RMS_NORM (CPU, CUDA) (#32135)
* ggml : add BF16 support to RMS_NORM (CPU, CUDA)

* cuda : use one helper for the F32/BF16 RMS_NORM launches
2026-11-13 15:41:17 +02:00
Georgi Gerganovandlosewayy 29dbbad2b1 ggml : prune (#29801)
* ggml : prune deprecated API

* cont : bump RPC version

* cont : minor

* ggml : remove outdated documentation from ggml.h

The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:

- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`

The block also ended with 5 empty `TODO` sections.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)

* rpc : bump RPC protocol major version

The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.

The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: losewayy <1134285612@qq.com>
2026-11-12 20:58:24 +02:00
anujj 0fb8f15770 Title: CUDA: few-row MMQ for BS 2-24 (#31533)
* CUDA: few-row MMQ for 2-24 src1 columns

New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.

* nit fixes

* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat

* few-row quantize: DS4 sum from the quantized values, as in #31247

* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build

* few-row MMQ: dispatch order MMVQ, few-row, MMQ
2026-11-12 21:12:45 +08:00
Konrad Moren 0d69255c52 CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints (#26048)
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back

* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ

* ggml-cuda: Tests for the fused scale

* ggml-cuda: reuse existing graph to test fusion in mmq epiloque

* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency

* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions

* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
2026-11-12 10:28:36 +01:00
CC 4180722eeb metal : prevent out-of-bounds reads in SOLVE_TRI kernel (#32103) 2026-11-11 22:02:06 +02:00
Leon af6b5243d6 webgpu: add PAD_REFLECT_1D support (#31877)
* webgpu : add PAD_REFLECT_1D support

* webgpu: remove redundant PAD_REFLECT_1D contiguity check

* tests: simplify PAD_REFLECT_1D setup
2026-11-11 12:11:06 +02:00
Jeff Bolz d2b3eb9b60 vulkan: allow dequantized KV optimization with CM2 (#31526)
* additional FA test cases

Assisted-by: Codex

* vulkan: allow dequantized KV optimization with CM2

Enable the KV dequantization optimization with coopmat2, guarded by a heuristic
based on query tokens*GQA ratio.

While doing this we saw very large speedups for certain head sizes, and it was
because those head sizes were incorrectly treated as unaligned. So also fix
those to be treated as aligned, which should help for when the dequant
optimization doesn't apply.

* gqa uses nbq2 for the stride, so we need to check that value for alignment

* tests : use VARS_TO_STR18 for flash attention

Assisted-by: Codex
2026-11-11 12:08:06 +02:00
Aman Gupta 459ab1cdd3 cuda : use MMVF for BF16 matrices of few rows at small batch sizes (#31959)
MMF tiles 32 rows per block, so a BF16 matrix with few rows (e.g. the
qwen4exp hyper-connection projections, 320 rows with K = 10240) runs only
a handful of blocks and leaves most SMs idle at batch sizes 2..8. MMVF
runs a block per row and wins there when K is long enough.

MMVF's cost grows with rows x batch while MMF's stays roughly flat in that
range, so use MMVF only while rows * ne11 * 256 <= nsm * min(K, max_k) and
K >= 1024. The budget scales with the SM count, which keeps GPUs with few
SMs (e.g. the A16) on MMF. Blackwell allows up to 8 columns and
max_k = 6144; older GPUs stop at 5 columns and max_k = 4096.

Add eval cases at 128x2560, 512x2560 and 320x10240 for n = 2, 3, 4, 5 and 8.
2026-11-11 10:46:13 +02:00
Sihan Yu 9d7b2d7397 cuda : fattn mma head sizes 40 72 (#30285)
* cuda : add MMA FlashAttention support for head sizes 40 and 72

Head sizes 40 and 72 (used e.g. in vision towers of Gemma 3/4 and Qwen3-VL) were excluded from every tensor-core dispatch branch and fell back to the tile kernel.

The fp16 MMA fragment covers 16 halfs in the head direction, so the in-kernel geometry is padded to the next multiple of 16 (40 -> 48, 72 -> 80). Register arrays, shared memory tile strides and MMA loop bounds use the padded size, while global memory loads and stores the real one.

Enabled on NVIDIA Turing and newer while Volta and AMD keep the tile kernel. The RDNA/CDNA config rows and the mma_tile_sizes specialisations are needed because the explicit instantiations are compiled for every target of a translation unit.

* tests : add the head size = 40 encoder perf shape

* cuda : address review comments on the head size 40 and 72 kernels
2026-11-11 10:45:01 +02:00
Oliver SimonsandAman Gupta 9e9619d030 CUDA: Add OCP FP8 support (#28413)
* Initial CUDA WIP version

* Use fp8 intrinsics

* Do W8A16 for FP8 in GEMV case

(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
  Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB
| model                          |       size |     params | backend    | ngl | n_ubatch |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |          pp8196 |       2781.53 ± 8.48 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |           tg256 |         37.56 ± 0.02 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio | pp8196 @ d32000 |       1616.31 ± 2.61 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |      512 |        dio |  tg256 @ d32000 |         34.11 ± 0.03 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |          pp8196 |       2542.00 ± 0.51 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |           tg256 |         37.36 ± 0.02 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio | pp8196 @ d32000 |       1762.86 ± 1.33 |
| qwen35 27B NVFP4               |  19.56 GiB |    27.32 B | CUDA       |  -1 |     2048 |        dio |  tg256 @ d32000 |         34.09 ± 0.02 |

* Optimize GEMV kernel for FP8

* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2

Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099

* Remove trailing whitespace

* Add supported for strided copies to/from FP8 in CUDA

* Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh

* Fix odd-rows for convert/getrows, possible with FP8

* Fix FP8 fallback in cuBlas

* Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8

* Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte

* Add todo about accumulator/precision honoring

This is beyond the scope of the stacked PR

* Avoid races in fp8_fallback

* Allow Hopper and other BW SKUs for FP8

* Use intrinsic over explicit manual conversions

* Resolve TODO

As per spec, we don't have to preserve signs

* Add commend on cublaslt workspaces

* Use convert_unary_cont_cuda over block dequant in convert.cu

* Use get_rows_cuda_float for fp8 kv cache

* Make FP8 copies work on all IHV's

* More cleanup

* Simplify by falling back to shared cuBLAS helper

* Fuse weight_scale multiplication for FP8 also

* More code-cleanup

* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4

Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix

* Use F32 activations in fallback mmvf path

* Explicitly link against cublaslt

* Move FP8 conversion intrinsics into ggml_cuda_cast

Cleaner this way

* WIP src1/prec for fp8 in MMVF

* Remove comment

* Keep only f8 x f8 KV cache for now

* Just do FP8 x FP32 x FP32 in MMVF as its fast enough

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

* Add batched FP8 GEMMS in cublas as suggested by code review

https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644

* Extract fp8x2 -> float2 load into helper

* Searapte copy checks to/from fp8

* Compiler massaging to avoid repeated address offsets

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
2026-11-11 10:44:30 +02:00
Oliver Simons 202c1a229f llama : Add FP8 KV-cache support (#28392)
* Add FP8 KV-cache support

* Add tests for scaled, FP8 KV-cache to test-backend-ops

* Fix get_*_scale mapping and dequnatize FP8 also before rope shift

* Normalizes cache-scale names independently of projection layout.

While fused KQV is currently impractival for non-shared scales, this is
an improvement for future use regardless

* Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs

* Update comment wording

* Move E2E FP8 KV cache tests to test-llama-archs

They belong there as they are not just a test for ggml's backend ops

* Increase support surface, remove seed

* make scaled kv generic

* Fix CI failings for test-kv-cache

https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392

* CI: execute kv-cache-scale tests

* exclude fp8-kv tests from OpenVINO

* Fix linting errors of test_kv_cache_scales

* Disable kv-ffload=false + multi-GPU tests for FP8

See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413

* Adjust test label for test-kv-cache.cpp

* Fix conflict-resolution mistakes in test-kv-cache.cpp

https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392

* Move KV tests to test-fp8.cpp

https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4410518158

* Remove tmp file for test-kv-cache-scales.py on Windows as well

* Fix fp8 kv cache test by population kv_source_impl

* Exclude models that consume KV cache directly

* Remove obsolete comment

* Allow BF16 scales as well for KV cache

* Rename *_cache_scale to *_cache_s

https://github.com/ggml-org/llama.cpp/pull/28392#discussion_r4517194999

* Move `is_type_scaled()` to `llama-kv-cache.cpp`
2026-11-10 18:08:32 +02:00
Georgi Gerganov 494d0f47bd ggml : add MUL_MAT src1 precision tests and gating (#31896) 2026-11-10 17:03:27 +02:00
18c5a95e2b sycl: Add device-side MoE scheduler (#30156)
* sycl: Add device-side MoE scheduler

Assisted-by: Codex

* Add width-heuristic override bit and Q2_K and Q3_K types

Assisted-by: Codex

* tests: cover skipped MUL_MAT_ID slots on the device MoE schedule

Assisted-by: Claude Opus 5.5

* sycl: keep graphs on for lazily reordered MUL_MAT and reject BN=64 combos on the device MoE schedule

The graph check disabled SYCL graphs until every MUL_MAT weight with a
reorder extra was reordered, which never happens for some (dense MXFP4,
the fused GLU plain path), so graphs stayed off and the check reran on
every compute. Upstream already reorders those during recording with
the async memory extension, so only MUL_MAT_ID weights are reordered
ahead of recording now.

ggml_sycl_grouped_dequant_gemm_device_supported() accepted any XMX
combo, but the device schedule only runs with BN == GGML_SYCL_FG_BN, so
a 32x64 combo (GGML_SYCL_XMX_GATHER_SHAPES=4 or 8) made the graph check
pass and the compute fall back to the host sync path while recording.

Assisted-by: Claude Opus 5.5

* sycl: keep the ggml_sycl_mul_mat_id_needs_sync name for the graph check

Assisted-by: Claude Opus 5.5

* fix: stop erroneously reordering host-offloaded tensors

With -ncmoe, prompt processing offloads the CPU-resident expert matmuls
to the GPU: the scheduler uploads the experts into a copy in the compute
buffer before every run. init_tensor gave that copy a reorder record, so
the MUL_MAT_ID gather path reordered every offloaded expert tensor on
every ubatch, and the next upload overwrote the result.

Qwen3-30B-A3B Q4_K_M, 1x B60, llama-perplexity -c 4096 -ub 2048
-ncmoe 48, 8 chunks, on master ac8e1694:
  master                        116 s, PPL 7.739
  master XMX_GATHER_TYPES=0      67 s, PPL 7.738
  with this change               67 s, PPL 7.729

Refs ggml-org/llama.cpp#31262

Assisted-by: Claude Opus 5.5

* Remove SYCL-specific test comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* sycl : name the MUL_MAT_ID path predicates after what they decide

ggml_sycl_mul_mat_id_mmvq_ok -> ggml_sycl_mul_mat_id_should_use_mmvq
ggml_sycl_mul_mat_id_mmvq_shape_ok -> ggml_sycl_mul_mat_id_mmvq_eligible
ggml_sycl_mul_mat_id_grouped_shape_ok -> ggml_sycl_mul_mat_id_device_sched_eligible
node_is_device_local -> graph_node_is_in_local_device
ggml_sycl_gg_rows -> ggml_sycl_grouped_gemm_rows
ggml_sycl_gg_tile -> ggml_sycl_grouped_gemm_tile (from #29245, renamed to match)

"eligible" is a necessary condition; the type and width checks follow
elsewhere. "should_use" means the path runs. No functional change.

Assisted-by: Claude Opus 5.5

* sycl : reorder Q3_K MoE experts and load them as vectors in the grouped GEMM

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: cwriter <cwriter@localhost>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-10 09:42:25 +02:00
ae4893a092 vulkan: add tiled-transpose concat shader for Qwen models (#30591)
* vulkan: tiled transpose for dim-0 concat with a transposed source

The delta-net conv-state path does ggml_transpose() straight into a dim-0
ggml_concat(), so the generic concat kernel reads src1 with the transposed
stride, one element per thread, and neighbouring lanes land on different cache
lines. Stage a 32x32 tile through shared memory instead, which keeps both the
load and the store coalesced. 33 columns pads away the bank conflicts.

Gated narrowly: dim 0, 2D, unquantized 4-byte elements, contiguous src0/dst
rows, and src1 actually transposed. Everything else falls through to the
existing kernel.

Qwen3.6-35B-A3B on gfx1151/RADV, 30 gated-delta-net layers: the CONCAT op goes
11877 us -> 957 us, and pp2048 at ub2048 goes 1224 -> 1628 t/s. Generated text
is unchanged.

The approach follows the tiled transposed concat Gaetan Puleo wrote for the HIP
backend in his Strix Halo fork.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* enable by default, add smaller shader sizes

* clean up comments

* improve shader

* Add non-contiguous concat tests

Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>

* handle x or y axis overflow

---------

Co-authored-by: Nathan Wilson <nath.flagman151@passmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
2026-11-10 09:41:45 +02:00
losewayy 4b46d96538 metal : make ggml_tri select instead of scaling by the comparison (#30711)
* metal: make ggml_tri select instead of scaling by the comparison

The kernel multiplied by the 0/1 comparison, which turns a +-inf outside the triangle into NaN, while the CPU kernel writes a plain zero there. Select, like the CPU does.

Adds test_tri_special, which puts +-inf into the cells that have to be discarded, for all four triangle types. It fails against the previous kernel and passes with this change.

* Just trim down the repetitive comments.
2026-11-09 13:31:16 +02:00
Pascal 6590d25a32 tests: bound the test_rope tolerance by the f32 angle resolution (#31874)
The rotation angle of a position p is rounded in f32 to about
p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach
it through different rounding sequences (cumulative product versus
pow). With positions up to 8192 this puts the NMSE floor between
backends near 8e-8, so the default 1e-7 threshold fails at random on
Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for
n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
2026-11-09 09:31:13 +01:00
Xuan-Son Nguyen 7d7f028753 mtmd: refactoring: remove clip.h, consolidate code into mtmd (#31865) 2026-11-08 15:49:06 +01:00
57d04820df ggml : update CUDA/Metal radix Top-K to make tied selections deterministic (#31084)
* ggml : fix for the radix TOP_K thread timing bug

* make the key an ordered 64-bit pair

* a rewrite without shift-skip logic

* fix casts

* warp-contiguous layout

* reduce the scope removing vulkan+metal changes

* turn it off for Metal/ROCm, note about cub::DeviceTopK

* fix the hip build

* skip Vulkan as well

* popc lambda from fattn

* do block_cols math in int64_t

* make it allowlist bc SYCL looks non deterministic too

* ggml : make Metal radix Top-K tied selections deterministic

The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.

Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.

This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* tests : enable deterministic top-k check on Metal, add radix-path tie cases

Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.

Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* ggml : use coalesced accesses in the Metal radix Top-K compaction

The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.

Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* tests : remove the extra top-k tie and perf cases

The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal: scan the radix bins and warp totals with a simdgroup

The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.

* tests: enable the deterministic top-k check on ROCm

HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.

* cuda: remove the redundant radix top-k counter reset

Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-08 16:42:57 +02:00
sargentmajordandSargentmajord 9982004913 ggml-cuda : add d_state=64 fast path to SSM_SCAN kernel (#31818)
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.

Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).

Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.

Assisted-by: Claude Sonnet 5

Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
2026-11-08 15:48:20 +02:00
LeonandMasashi Yoshimura a691b23458 webgpu : add COUNT_EQUAL support (#31578)
* webgpu : add COUNT_EQUAL support

Assisted-by: Codex

* webgpu : reduce small COUNT_EQUAL inputs in one dispatch

Assisted-by: Codex

* webgpu : simplify COUNT_EQUAL and reuse existing tests

* webgpu : move COUNT_EQUAL pipeline cache declaration

* Update ggml/src/ggml-webgpu/ggml-webgpu.cpp

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>

* tests : simplify COUNT_EQUAL coverage

---------

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>
2026-11-08 11:50:09 +02:00
PascalandGeorgi Gerganov d6be065572 llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (#31851)
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls

Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.

* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends

Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.

* tests: cover GGML_PREC_BF16 in the flash attention tests

The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.

* llama: request GGML_PREC_BF16 for the KQ multiplication

* cont : add TODO in clip.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-08 08:37:46 +01:00
Nikhil Jain cf659f240b webgpu : Implement group norm kernel (#31490)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* Add WebGPU group_norm kernel

* clean up

* resolve merge conflicts

* parameterize data type; clean up cmake changes

* add nullptr check

* clean up webkit experiments

* more cleanup

* more cleanup

* clang format

* Remove unused feature check for WebGPU

* clean up supports_op condition and update documentation

* add perf tests for group_norm

* clean up doc with script and undo change in test-backend-ops

* Do 2d dispatch of workgroups

* Do 2d dispatch of workgroups

* Unstage unrelated changes

* undo docs churn

* add new test case

* undo small change

* undo small change
2026-11-07 10:01:27 +02:00
669204ee2b hexagon: conv_2d optimizations and support for conv_3d (#31814)
* hexagon: Generalization of CONV_2D implementation for CONV_3D

- New op HTP_OP_CONV_3D is wired through htp-ops.h, main.c (shares op_conv_2d) and the op remap in htp-kparams.cpp.
- conv_3d runs on the existing HMX tiled (implicit GEMM) conv2d path. KD and KH are folded into one virtual kernel row (KD*KH), and output rows are counted as N*OD*OH.
- htp_kparams_solve_conv3d checks the conv_3d layout and calls the tiled solver htp_kparams_conv2d_tiled, which takes an is3d flag.
- The tiled kernel takes the sizes and strides that differ between conv_2d and conv_3d from a struct htp_conv_geom. Only the input stage unfolds depth.
- 256-entry DMA queue can no longer overflow

Assisted-by: Claude Code

* hexagon: modify checking to eliminate divide-by-zero hazards

* hexagon: conv2d tiled: compute the conv_3d unfold divisors on the host

The KH and OH fastdiv values used to unfold conv_3d kh into (kd, kh) and
oh into (od, oh) were built per op on the device, each with a 64-bit
divide. Pass them in the kernel params instead; they fit in the space the
window variant already reserves, so the struct size does not change.

Assisted-by: Claude Code

* hexagon : preserve fold scheduling in conv2d for small size

hexagon : support arbitrary positive conv2d strides
Support positive horizontal strides for WHCN and CWHN inputs. Preserve
load/shuffle paths for strides 1/2 and use gather for larger strides.

hexagon : improve conv2d tile size selection

Prefer the largest feasible output-channel block and derive spatial
chunk capacity from VTCM usage. Compare capacity and thread-aligned
candidates using per-thread work and chunk overhead.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-conv: fix compiler warning

---------

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-06 11:51:57 -08:00
Georgi Gerganov 9a1d03c7ca tests : override hardware_concurrency in CI (#31797)
The self-hosted CPU runners are containers with 8 CPUs, but
std::thread::hardware_concurrency reports the cores of the host. The
table mode of test-llama-context sizes its pool from that count, so it
runs 4 models x 8 threads on an 8-CPU job and hangs.

Add TEST_ARG_HARDWARE_CONCURRENCY and TEST_ARG_PARALLEL_WORKERS as the
defaults for the auto-selected counts of test-llama-context and of
test-backend-ops, and set them to 8 and 4 for cpu-x64-high-perf.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-11-06 14:52:26 +02:00
Georgi Gerganov ea40a3a1e1 metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (#31615)
* metal : support scale-bias routing norm in the TOPK_MOE fusion

The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sqrt(softplus) MoE routing with a selection bias

Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sigmoid MoE routing with a selection bias

The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.

- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
  GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
  tail; the selection-bias ADD is required, same as the sqrt(softplus)
  chains
- generalize the gating function constant from a sqrt(softplus) bool
  to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
  scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
  CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
  weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
  deepseek2/32, kimi, minimax, ...) now fuse their routing, and
  muse-spark prefill drops one MOE_REDUCE fire because the reordered
  graph allocates the routing weights inside the reduce output range,
  which the dst/input overlap check correctly declines

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* cont : remove low-signal comments

* ocd : valign ggml_metal_fusions
2026-11-05 13:52:56 +02:00
Takamichi Ohara 1fa7f1c488 ggml : support truncate=false for YaRN (#30206)
* ggml : support `truncate=false` for YaRN

YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.

Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.

* support CUDA backend

* add tests

* move RoPE helpers to internal header

* format
2026-11-05 12:02:05 +02:00
Hoang Vien DuyandGeorgi Gerganov f7b41cca4f metal : use mul_mm for 32 <= K < 64 (#30982)
* metal : use mul_mm for 32 <= K < 64

* metal : add bf16 x bf16 mul mat kernel

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-05 10:58:41 +02:00
Leon 553d2c2e7e webgpu : add MEAN support (#31591)
Assisted-by: Codex
2026-11-05 10:29:14 +02:00
Pascal a01dff908b tests: fix CI flakes of TOPK_MOE caused by near ties (#31426)
* tests: fix CI flakes of TOPK_MOE caused by near ties

With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.

Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.

* tests: build the TOPK_MOE logits by inverting the gating function

Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.

The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
2026-11-05 07:31:48 +01:00
anujj 5ce06fc71f Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ (#31296)
* cuda: fuse NVFP4 gate/up and GLU in MMQ

* cuda: honor the requested src1 precision in fused gate/up MMQ

* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels

* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming

* cuda: check gate_scale before loading the fused NVFP4 gate scale
2026-11-05 10:47:33 +08:00
Todor BoinovskiandMax Krasnyansky bee23322f1 hexagon: add LIGHTNING_INDEXER support (#31289)
* hexagon: initial support for LIGHTNING_INDEXER op

* hex-lght-idxr: hmx for prefill

* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute

* hex-lght-idxr: pipelined hmx path for decode and prefill

* hex-lid: add support for K (f32)

* hex-lid: add support for K (f32)

* hex-lid: add support for q4_0

* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills

* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel

* hex-lit: update dma pipeline to use canonical dbl buffering in all paths

* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths

* hex-lit: add support for dims collapse

* hex-lit: make kparams constistent with other ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 21:17:30 -08:00
Todor Boinovski 7071ba378a hexagon: add TOP_K_MASK op (#31552)
* hexagon: add TOP_K_MASK op

* Simplify Hexagon TOP_K_MASK kernel and do not duplicate host configuration
2026-11-03 12:04:03 -08:00
Todor BoinovskiandMax Krasnyansky 20a51f24a8 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
* hexagon: add support for DSV4_HC_PRE

* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization

* hex-dsv4-hc-pre: double-buffered DMA for prefill

* hex-dsv4-hc-pre: refactor DMA block setup

* hex-dsv4-hc-pre: precompute kernel params

* hex-dsv4-hc-pre: support for n_hc != 4

* hexagon: add support for DSV4_HC_POST

* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block

* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks

* hex-run: add support for test-llama-archs tool

* hex-dsv4: refactoring and optimizations

Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.

Tested with test-llama-archs -a qwen4ext and test-backend-ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 08:26:18 -08:00
Leon 713319e284 webgpu: add ARANGE support (#31455) 2026-11-03 07:54:00 +02:00
51fe42c5d1 hexagon: tiled hvx permute (#31446)
* hexagon: add tiled HVX transpose path for permuted CONT/CPY

Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.

Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.

Host:
  - drop extent-1 dims and merge dims that stay adjacent in src into a
    canonical view, stored in the kernel params. There are no minimum
    size limits.
  - the copy is treated as a 2D transpose between dst dim 0 and the
    contiguous src dim, batched over the remaining dims.
  - tiles are sized to 64 KB per buffer and halved until they fit in
    session VTCM. blkk is capped so the halfword vgather offsets fit in
    16 bits.

Kernel, per tile:
  - 2D DMA of blk0 strided src rows (blkk contiguous elements each)
    into VTCM
  - vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
  - 2D DMA of blkk contiguous rows out to dst

Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.

Small dims:
  - packed mode: when dim k follows dim 0 and dim 0 divides the HVX
    lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
    whole dst rows, and each tile goes out as one contiguous DMA.
  - when the contiguous src dim fits in one vector, it is staged exactly
    and the tile is made long along dim 0.

All other copies select the same kernels as before.

Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.

* hexagon: cpy transpose cleanups

- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)

Assisted-by: Claude Opus 5.5

* hexagon: use cpy transpose path for ne[0] == 1 permutes

- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests

Assisted-by: Claude Opus 5.5

* hex-cpy: further improve cpy

Bring in improvements from aizip branch/PR.

Cleanup kparams and replace divs with fastdiv.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-cpy: avoid vtcm bank conflicts

* hex-cpy: trim down the comments

* cpy-test: trim down new tests that cover permute scenarios

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
2026-11-02 21:13:38 -08:00
Aman GuptaandGeorgi Gerganov 1b6935b227 llama : flash_attn_ext_rows - more efficient unified kv for multi-seq (#29510)
* llama : use llm_attn_mask and kv_rows for unified attention

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* cont : optimize `get_seq_cells`

* llama : build the kv_rows lists from per-sequence cell bitmaps

Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.

Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).

* cont : make get_kv_rows_p_min private

* cont : minor metal

* llama : use kv_rows for every ubatch of a unified cache

A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.

Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.

* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-02 18:13:33 +02:00
Mark TodorovichandMax Krasnyansky a91eab67d8 hexagon : fix DMA transfers with strides > 24 bits (#31442)
* hexagon : fix DMA transfers with strides > 24 bits

On v75+ the 2D DMA descriptor stores row_size and strides in 24 bits.
dma_queue_push sent oversized strides to fallback_2d, which only splits
nrows, so strides of 16 MB or more were truncated - wrong addresses.
Fix will fall back to one transfer per row in that case.

Also added two CONT tests with source strides over 16 MB.

* hex-dma: refactor the fallback paths to remove duplication

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-31 16:49:23 -07:00
Oliver SimonsandSigbjørn Skjæret 68d20b884c Quant: OCP FP8 E4M3 support (#27512)
* WIP OCP FP8 E4M3 support

* Fix python type-checker

* Materialize input_scales also for FP8 when converting

* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4

1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
   dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
   weights are reshaped

* Allow remaining FP8 tensors to be dequantized during conversion

* Allow tensor-strategy for FP8-only compressed-tensor ckpts

* Fix and test gguf-py behavior for FP8_E4M3

* Disallow QKV-fusion for FP8 temporarily

* Fix conversion of fused qkvz & FP8 for Qwen3-Next

* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws

* Emit warnings when not fusing FP8 during conversion

* More FP8 conversion fixes

* Add supported for strided copies to/from FP8 in CPU

* remove unneeded scratch buffer from FP8 cpu cpy/dup path

* add explanatory comments and TODOs

* Add dedicated encoder and decoder tests for FP8

* Whitelist ggml_op_get_rows in metal instead of blacklisting

required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list

https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512

* Fix more conversion bugs

* Defer weight materialization during conversion

* Blacklist fp8 in metal backend instead of white-list

* Adjust indents

* Fix lint + execute tests

* Add fast CPU kernel for F8_E4M3

* Conversion fixes

* Fix remaining gaps

* Enable tied weights with shared scales

* Make mandatory-F32 rules apply to FP8 during conversion

* Fix python type checks

https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512

* Fix kaleidai unit-test failures

https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150

* Relax numeric precision constraints for pre-check

* Leave fast-FP8 CPU on RISC-V for follow-up

We already exclude arm from the fast-path

* Allow output scale tensors to be located in any gguf shard

* Move test_conversion_fp8.py test execution to behavior-check.yaml

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it

* Install all requirements also on Windows for behavior-check

Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512

* Unmap lazy tensors before clean-up in Windows

Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512

* LF line endings

https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512

* Extract FP8 tests from llama-arch to its own test-fixture

See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221

* Update gguf-py/gguf/scripts/gguf_convert_endian.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-29 18:14:37 +01:00