Commit Graph
600 Commits
Author SHA1 Message Date
Oliver Simons 202c1a229f llama : Add FP8 KV-cache support (#28392)
* Add FP8 KV-cache support

* Add tests for scaled, FP8 KV-cache to test-backend-ops

* Fix get_*_scale mapping and dequnatize FP8 also before rope shift

* Normalizes cache-scale names independently of projection layout.

While fused KQV is currently impractival for non-shared scales, this is
an improvement for future use regardless

* Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs

* Update comment wording

* Move E2E FP8 KV cache tests to test-llama-archs

They belong there as they are not just a test for ggml's backend ops

* Increase support surface, remove seed

* make scaled kv generic

* Fix CI failings for test-kv-cache

https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392

* CI: execute kv-cache-scale tests

* exclude fp8-kv tests from OpenVINO

* Fix linting errors of test_kv_cache_scales

* Disable kv-ffload=false + multi-GPU tests for FP8

See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413

* Adjust test label for test-kv-cache.cpp

* Fix conflict-resolution mistakes in test-kv-cache.cpp

https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392

* Move KV tests to test-fp8.cpp

https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4410518158

* Remove tmp file for test-kv-cache-scales.py on Windows as well

* Fix fp8 kv cache test by population kv_source_impl

* Exclude models that consume KV cache directly

* Remove obsolete comment

* Allow BF16 scales as well for KV cache

* Rename *_cache_scale to *_cache_s

https://github.com/ggml-org/llama.cpp/pull/28392#discussion_r4517194999

* Move `is_type_scaled()` to `llama-kv-cache.cpp`
2026-11-10 18:08:32 +02:00
Georgi Gerganov 494d0f47bd ggml : add MUL_MAT src1 precision tests and gating (#31896) 2026-11-10 17:03:27 +02:00
18c5a95e2b sycl: Add device-side MoE scheduler (#30156)
* sycl: Add device-side MoE scheduler

Assisted-by: Codex

* Add width-heuristic override bit and Q2_K and Q3_K types

Assisted-by: Codex

* tests: cover skipped MUL_MAT_ID slots on the device MoE schedule

Assisted-by: Claude Opus 5.5

* sycl: keep graphs on for lazily reordered MUL_MAT and reject BN=64 combos on the device MoE schedule

The graph check disabled SYCL graphs until every MUL_MAT weight with a
reorder extra was reordered, which never happens for some (dense MXFP4,
the fused GLU plain path), so graphs stayed off and the check reran on
every compute. Upstream already reorders those during recording with
the async memory extension, so only MUL_MAT_ID weights are reordered
ahead of recording now.

ggml_sycl_grouped_dequant_gemm_device_supported() accepted any XMX
combo, but the device schedule only runs with BN == GGML_SYCL_FG_BN, so
a 32x64 combo (GGML_SYCL_XMX_GATHER_SHAPES=4 or 8) made the graph check
pass and the compute fall back to the host sync path while recording.

Assisted-by: Claude Opus 5.5

* sycl: keep the ggml_sycl_mul_mat_id_needs_sync name for the graph check

Assisted-by: Claude Opus 5.5

* fix: stop erroneously reordering host-offloaded tensors

With -ncmoe, prompt processing offloads the CPU-resident expert matmuls
to the GPU: the scheduler uploads the experts into a copy in the compute
buffer before every run. init_tensor gave that copy a reorder record, so
the MUL_MAT_ID gather path reordered every offloaded expert tensor on
every ubatch, and the next upload overwrote the result.

Qwen3-30B-A3B Q4_K_M, 1x B60, llama-perplexity -c 4096 -ub 2048
-ncmoe 48, 8 chunks, on master ac8e1694:
  master                        116 s, PPL 7.739
  master XMX_GATHER_TYPES=0      67 s, PPL 7.738
  with this change               67 s, PPL 7.729

Refs ggml-org/llama.cpp#31262

Assisted-by: Claude Opus 5.5

* Remove SYCL-specific test comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* sycl : name the MUL_MAT_ID path predicates after what they decide

ggml_sycl_mul_mat_id_mmvq_ok -> ggml_sycl_mul_mat_id_should_use_mmvq
ggml_sycl_mul_mat_id_mmvq_shape_ok -> ggml_sycl_mul_mat_id_mmvq_eligible
ggml_sycl_mul_mat_id_grouped_shape_ok -> ggml_sycl_mul_mat_id_device_sched_eligible
node_is_device_local -> graph_node_is_in_local_device
ggml_sycl_gg_rows -> ggml_sycl_grouped_gemm_rows
ggml_sycl_gg_tile -> ggml_sycl_grouped_gemm_tile (from #29245, renamed to match)

"eligible" is a necessary condition; the type and width checks follow
elsewhere. "should_use" means the path runs. No functional change.

Assisted-by: Claude Opus 5.5

* sycl : reorder Q3_K MoE experts and load them as vectors in the grouped GEMM

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: cwriter <cwriter@localhost>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-10 09:42:25 +02:00
ae4893a092 vulkan: add tiled-transpose concat shader for Qwen models (#30591)
* vulkan: tiled transpose for dim-0 concat with a transposed source

The delta-net conv-state path does ggml_transpose() straight into a dim-0
ggml_concat(), so the generic concat kernel reads src1 with the transposed
stride, one element per thread, and neighbouring lanes land on different cache
lines. Stage a 32x32 tile through shared memory instead, which keeps both the
load and the store coalesced. 33 columns pads away the bank conflicts.

Gated narrowly: dim 0, 2D, unquantized 4-byte elements, contiguous src0/dst
rows, and src1 actually transposed. Everything else falls through to the
existing kernel.

Qwen3.6-35B-A3B on gfx1151/RADV, 30 gated-delta-net layers: the CONCAT op goes
11877 us -> 957 us, and pp2048 at ub2048 goes 1224 -> 1628 t/s. Generated text
is unchanged.

The approach follows the tiled transposed concat Gaetan Puleo wrote for the HIP
backend in his Strix Halo fork.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* enable by default, add smaller shader sizes

* clean up comments

* improve shader

* Add non-contiguous concat tests

Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>

* handle x or y axis overflow

---------

Co-authored-by: Nathan Wilson <nath.flagman151@passmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
2026-11-10 09:41:45 +02:00
losewayy 4b46d96538 metal : make ggml_tri select instead of scaling by the comparison (#30711)
* metal: make ggml_tri select instead of scaling by the comparison

The kernel multiplied by the 0/1 comparison, which turns a +-inf outside the triangle into NaN, while the CPU kernel writes a plain zero there. Select, like the CPU does.

Adds test_tri_special, which puts +-inf into the cells that have to be discarded, for all four triangle types. It fails against the previous kernel and passes with this change.

* Just trim down the repetitive comments.
2026-11-09 13:31:16 +02:00
Pascal 6590d25a32 tests: bound the test_rope tolerance by the f32 angle resolution (#31874)
The rotation angle of a position p is rounded in f32 to about
p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach
it through different rounding sequences (cumulative product versus
pow). With positions up to 8192 this puts the NMSE floor between
backends near 8e-8, so the default 1e-7 threshold fails at random on
Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for
n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
2026-11-09 09:31:13 +01:00
Xuan-Son Nguyen 7d7f028753 mtmd: refactoring: remove clip.h, consolidate code into mtmd (#31865) 2026-11-08 15:49:06 +01:00
57d04820df ggml : update CUDA/Metal radix Top-K to make tied selections deterministic (#31084)
* ggml : fix for the radix TOP_K thread timing bug

* make the key an ordered 64-bit pair

* a rewrite without shift-skip logic

* fix casts

* warp-contiguous layout

* reduce the scope removing vulkan+metal changes

* turn it off for Metal/ROCm, note about cub::DeviceTopK

* fix the hip build

* skip Vulkan as well

* popc lambda from fattn

* do block_cols math in int64_t

* make it allowlist bc SYCL looks non deterministic too

* ggml : make Metal radix Top-K tied selections deterministic

The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.

Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.

This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* tests : enable deterministic top-k check on Metal, add radix-path tie cases

Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.

Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* ggml : use coalesced accesses in the Metal radix Top-K compaction

The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.

Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* tests : remove the extra top-k tie and perf cases

The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal: scan the radix bins and warp totals with a simdgroup

The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.

* tests: enable the deterministic top-k check on ROCm

HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.

* cuda: remove the redundant radix top-k counter reset

Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-08 16:42:57 +02:00
sargentmajordandSargentmajord 9982004913 ggml-cuda : add d_state=64 fast path to SSM_SCAN kernel (#31818)
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.

Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).

Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.

Assisted-by: Claude Sonnet 5

Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
2026-11-08 15:48:20 +02:00
LeonandMasashi Yoshimura a691b23458 webgpu : add COUNT_EQUAL support (#31578)
* webgpu : add COUNT_EQUAL support

Assisted-by: Codex

* webgpu : reduce small COUNT_EQUAL inputs in one dispatch

Assisted-by: Codex

* webgpu : simplify COUNT_EQUAL and reuse existing tests

* webgpu : move COUNT_EQUAL pipeline cache declaration

* Update ggml/src/ggml-webgpu/ggml-webgpu.cpp

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>

* tests : simplify COUNT_EQUAL coverage

---------

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>
2026-11-08 11:50:09 +02:00
PascalandGeorgi Gerganov d6be065572 llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (#31851)
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls

Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.

* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends

Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.

* tests: cover GGML_PREC_BF16 in the flash attention tests

The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.

* llama: request GGML_PREC_BF16 for the KQ multiplication

* cont : add TODO in clip.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-08 08:37:46 +01:00
Nikhil Jain cf659f240b webgpu : Implement group norm kernel (#31490)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* Add WebGPU group_norm kernel

* clean up

* resolve merge conflicts

* parameterize data type; clean up cmake changes

* add nullptr check

* clean up webkit experiments

* more cleanup

* more cleanup

* clang format

* Remove unused feature check for WebGPU

* clean up supports_op condition and update documentation

* add perf tests for group_norm

* clean up doc with script and undo change in test-backend-ops

* Do 2d dispatch of workgroups

* Do 2d dispatch of workgroups

* Unstage unrelated changes

* undo docs churn

* add new test case

* undo small change

* undo small change
2026-11-07 10:01:27 +02:00
669204ee2b hexagon: conv_2d optimizations and support for conv_3d (#31814)
* hexagon: Generalization of CONV_2D implementation for CONV_3D

- New op HTP_OP_CONV_3D is wired through htp-ops.h, main.c (shares op_conv_2d) and the op remap in htp-kparams.cpp.
- conv_3d runs on the existing HMX tiled (implicit GEMM) conv2d path. KD and KH are folded into one virtual kernel row (KD*KH), and output rows are counted as N*OD*OH.
- htp_kparams_solve_conv3d checks the conv_3d layout and calls the tiled solver htp_kparams_conv2d_tiled, which takes an is3d flag.
- The tiled kernel takes the sizes and strides that differ between conv_2d and conv_3d from a struct htp_conv_geom. Only the input stage unfolds depth.
- 256-entry DMA queue can no longer overflow

Assisted-by: Claude Code

* hexagon: modify checking to eliminate divide-by-zero hazards

* hexagon: conv2d tiled: compute the conv_3d unfold divisors on the host

The KH and OH fastdiv values used to unfold conv_3d kh into (kd, kh) and
oh into (od, oh) were built per op on the device, each with a 64-bit
divide. Pass them in the kernel params instead; they fit in the space the
window variant already reserves, so the struct size does not change.

Assisted-by: Claude Code

* hexagon : preserve fold scheduling in conv2d for small size

hexagon : support arbitrary positive conv2d strides
Support positive horizontal strides for WHCN and CWHN inputs. Preserve
load/shuffle paths for strides 1/2 and use gather for larger strides.

hexagon : improve conv2d tile size selection

Prefer the largest feasible output-channel block and derive spatial
chunk capacity from VTCM usage. Compare capacity and thread-aligned
candidates using per-thread work and chunk overhead.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-conv: fix compiler warning

---------

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-06 11:51:57 -08:00
Georgi Gerganov 9a1d03c7ca tests : override hardware_concurrency in CI (#31797)
The self-hosted CPU runners are containers with 8 CPUs, but
std::thread::hardware_concurrency reports the cores of the host. The
table mode of test-llama-context sizes its pool from that count, so it
runs 4 models x 8 threads on an 8-CPU job and hangs.

Add TEST_ARG_HARDWARE_CONCURRENCY and TEST_ARG_PARALLEL_WORKERS as the
defaults for the auto-selected counts of test-llama-context and of
test-backend-ops, and set them to 8 and 4 for cpu-x64-high-perf.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-11-06 14:52:26 +02:00
Georgi Gerganov ea40a3a1e1 metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (#31615)
* metal : support scale-bias routing norm in the TOPK_MOE fusion

The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sqrt(softplus) MoE routing with a selection bias

Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sigmoid MoE routing with a selection bias

The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.

- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
  GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
  tail; the selection-bias ADD is required, same as the sqrt(softplus)
  chains
- generalize the gating function constant from a sqrt(softplus) bool
  to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
  scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
  CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
  weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
  deepseek2/32, kimi, minimax, ...) now fuse their routing, and
  muse-spark prefill drops one MOE_REDUCE fire because the reordered
  graph allocates the routing weights inside the reduce output range,
  which the dst/input overlap check correctly declines

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* cont : remove low-signal comments

* ocd : valign ggml_metal_fusions
2026-11-05 13:52:56 +02:00
Takamichi Ohara 1fa7f1c488 ggml : support truncate=false for YaRN (#30206)
* ggml : support `truncate=false` for YaRN

YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.

Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.

* support CUDA backend

* add tests

* move RoPE helpers to internal header

* format
2026-11-05 12:02:05 +02:00
Hoang Vien DuyandGeorgi Gerganov f7b41cca4f metal : use mul_mm for 32 <= K < 64 (#30982)
* metal : use mul_mm for 32 <= K < 64

* metal : add bf16 x bf16 mul mat kernel

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-05 10:58:41 +02:00
Leon 553d2c2e7e webgpu : add MEAN support (#31591)
Assisted-by: Codex
2026-11-05 10:29:14 +02:00
Pascal a01dff908b tests: fix CI flakes of TOPK_MOE caused by near ties (#31426)
* tests: fix CI flakes of TOPK_MOE caused by near ties

With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.

Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.

* tests: build the TOPK_MOE logits by inverting the gating function

Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.

The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
2026-11-05 07:31:48 +01:00
anujj 5ce06fc71f Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ (#31296)
* cuda: fuse NVFP4 gate/up and GLU in MMQ

* cuda: honor the requested src1 precision in fused gate/up MMQ

* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels

* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming

* cuda: check gate_scale before loading the fused NVFP4 gate scale
2026-11-05 10:47:33 +08:00
Todor BoinovskiandMax Krasnyansky bee23322f1 hexagon: add LIGHTNING_INDEXER support (#31289)
* hexagon: initial support for LIGHTNING_INDEXER op

* hex-lght-idxr: hmx for prefill

* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute

* hex-lght-idxr: pipelined hmx path for decode and prefill

* hex-lid: add support for K (f32)

* hex-lid: add support for K (f32)

* hex-lid: add support for q4_0

* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills

* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel

* hex-lit: update dma pipeline to use canonical dbl buffering in all paths

* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths

* hex-lit: add support for dims collapse

* hex-lit: make kparams constistent with other ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 21:17:30 -08:00
Todor Boinovski 7071ba378a hexagon: add TOP_K_MASK op (#31552)
* hexagon: add TOP_K_MASK op

* Simplify Hexagon TOP_K_MASK kernel and do not duplicate host configuration
2026-11-03 12:04:03 -08:00
Todor BoinovskiandMax Krasnyansky 20a51f24a8 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
* hexagon: add support for DSV4_HC_PRE

* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization

* hex-dsv4-hc-pre: double-buffered DMA for prefill

* hex-dsv4-hc-pre: refactor DMA block setup

* hex-dsv4-hc-pre: precompute kernel params

* hex-dsv4-hc-pre: support for n_hc != 4

* hexagon: add support for DSV4_HC_POST

* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block

* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks

* hex-run: add support for test-llama-archs tool

* hex-dsv4: refactoring and optimizations

Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.

Tested with test-llama-archs -a qwen4ext and test-backend-ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 08:26:18 -08:00
Leon 713319e284 webgpu: add ARANGE support (#31455) 2026-11-03 07:54:00 +02:00
51fe42c5d1 hexagon: tiled hvx permute (#31446)
* hexagon: add tiled HVX transpose path for permuted CONT/CPY

Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.

Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.

Host:
  - drop extent-1 dims and merge dims that stay adjacent in src into a
    canonical view, stored in the kernel params. There are no minimum
    size limits.
  - the copy is treated as a 2D transpose between dst dim 0 and the
    contiguous src dim, batched over the remaining dims.
  - tiles are sized to 64 KB per buffer and halved until they fit in
    session VTCM. blkk is capped so the halfword vgather offsets fit in
    16 bits.

Kernel, per tile:
  - 2D DMA of blk0 strided src rows (blkk contiguous elements each)
    into VTCM
  - vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
  - 2D DMA of blkk contiguous rows out to dst

Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.

Small dims:
  - packed mode: when dim k follows dim 0 and dim 0 divides the HVX
    lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
    whole dst rows, and each tile goes out as one contiguous DMA.
  - when the contiguous src dim fits in one vector, it is staged exactly
    and the tile is made long along dim 0.

All other copies select the same kernels as before.

Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.

* hexagon: cpy transpose cleanups

- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)

Assisted-by: Claude Opus 5.5

* hexagon: use cpy transpose path for ne[0] == 1 permutes

- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests

Assisted-by: Claude Opus 5.5

* hex-cpy: further improve cpy

Bring in improvements from aizip branch/PR.

Cleanup kparams and replace divs with fastdiv.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-cpy: avoid vtcm bank conflicts

* hex-cpy: trim down the comments

* cpy-test: trim down new tests that cover permute scenarios

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
2026-11-02 21:13:38 -08:00
Aman GuptaandGeorgi Gerganov 1b6935b227 llama : flash_attn_ext_rows - more efficient unified kv for multi-seq (#29510)
* llama : use llm_attn_mask and kv_rows for unified attention

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* cont : optimize `get_seq_cells`

* llama : build the kv_rows lists from per-sequence cell bitmaps

Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.

Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).

* cont : make get_kv_rows_p_min private

* cont : minor metal

* llama : use kv_rows for every ubatch of a unified cache

A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.

Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.

* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-02 18:13:33 +02:00
Mark TodorovichandMax Krasnyansky a91eab67d8 hexagon : fix DMA transfers with strides > 24 bits (#31442)
* hexagon : fix DMA transfers with strides > 24 bits

On v75+ the 2D DMA descriptor stores row_size and strides in 24 bits.
dma_queue_push sent oversized strides to fallback_2d, which only splits
nrows, so strides of 16 MB or more were truncated - wrong addresses.
Fix will fall back to one transfer per row in that case.

Also added two CONT tests with source strides over 16 MB.

* hex-dma: refactor the fallback paths to remove duplication

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-31 16:49:23 -07:00
Oliver SimonsandSigbjørn Skjæret 68d20b884c Quant: OCP FP8 E4M3 support (#27512)
* WIP OCP FP8 E4M3 support

* Fix python type-checker

* Materialize input_scales also for FP8 when converting

* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4

1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
   dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
   weights are reshaped

* Allow remaining FP8 tensors to be dequantized during conversion

* Allow tensor-strategy for FP8-only compressed-tensor ckpts

* Fix and test gguf-py behavior for FP8_E4M3

* Disallow QKV-fusion for FP8 temporarily

* Fix conversion of fused qkvz & FP8 for Qwen3-Next

* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws

* Emit warnings when not fusing FP8 during conversion

* More FP8 conversion fixes

* Add supported for strided copies to/from FP8 in CPU

* remove unneeded scratch buffer from FP8 cpu cpy/dup path

* add explanatory comments and TODOs

* Add dedicated encoder and decoder tests for FP8

* Whitelist ggml_op_get_rows in metal instead of blacklisting

required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list

https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512

* Fix more conversion bugs

* Defer weight materialization during conversion

* Blacklist fp8 in metal backend instead of white-list

* Adjust indents

* Fix lint + execute tests

* Add fast CPU kernel for F8_E4M3

* Conversion fixes

* Fix remaining gaps

* Enable tied weights with shared scales

* Make mandatory-F32 rules apply to FP8 during conversion

* Fix python type checks

https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512

* Fix kaleidai unit-test failures

https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150

* Relax numeric precision constraints for pre-check

* Leave fast-FP8 CPU on RISC-V for follow-up

We already exclude arm from the fast-path

* Allow output scale tensors to be located in any gguf shard

* Move test_conversion_fp8.py test execution to behavior-check.yaml

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it

* Install all requirements also on Windows for behavior-check

Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512

* Unmap lazy tensors before clean-up in Windows

Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512

* LF line endings

https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512

* Extract FP8 tests from llama-arch to its own test-fixture

See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221

* Update gguf-py/gguf/scripts/gguf_convert_endian.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-29 18:14:37 +01:00
Jesse Symons fade717ed8 tests : remove duplicate FA test cases added in #31125 (#31357)
Requested in the #31125 review. The cases duplicate coverage that is
already in the file and add CI time. Test-only, nothing changes in the
kernel, the docs or the CMake option.

Assisted-by: Claude
2026-10-29 10:10:25 +02:00
Max Krasnyansky 8a199ad31a hexagon: support for UPSCALE op (#31331)
* hex-upscale: basic upscale ops (not fully vectorized)

* hex-upscale: optimize nearest 2x, 4x and bilinear 4x

* hex-upscale: add support for non-contig rows and dims collapsing

* hex-upscale: use vgather instead of scalar staging

* hex-upscale: move all kernel select logic to the host, review comments and cleanups

* tests: add permuted UPSCALE test to cover qwen3-vl image->text scenarios
2026-10-29 00:22:34 -07:00
Jesse Symons f7c01c5226 [SYCL] add DPAS flash attention kernel for quantized KV cache (#31125)
* sycl : add DPAS flash attention kernel for quantized KV cache

Decode and small batches (1 to 31 query tokens) now run flash attention
straight on q4_0 and q8_0 KV blocks, instead of scalar dequant or
converting the whole cache to f16 per call. Q.K uses s8 x s4 and s8 x s8
DPAS on XMX, V.P stays f16. dpas.s4.s8 is inline vISA since joint_matrix
has no s4.

Covers head sizes 64 to 512, GQA up to 32 and attention sinks. Prompts
stay on the existing kernels. GGML_SYCL_FA_DPAS=0 turns it off.

B580, Llama 3.1 8B Q4_K_M, q4_0 KV, tg128: 31.3 to 54.2 t/s at 16K
context. FLASH_ATTN_EXT 4582/4584, same 2 failures as master.

Assisted-by: Claude

* sycl : 192 head dropped, Wildcat Lake added to docs

Assisted-by: Claude

* sycl : stricter checks from review

Assisted-by: Claude

* [SYCL] : rename XMX macro, add GGML_SYCL_SUPPORT_XMX option, document DPAS

Assisted-by: Claude
2026-10-29 14:01:21 +08:00
Marco ColomboandMax Krasnyansky 2e0d86bebc Hexagon: IM2COL improvement - block based HVX parallelization (#31050)
* hexagon: IM2COL over multiple threads with output-col blocks

Assisted-by: Claude Opus 5.5

* hex-im2col: fix padding handling

* hex-im2col: loop unroll common cases

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-28 19:04:18 -07:00
anujj f5222f25c1 cuda: fuse MoE (MUL_MAT_ID) gate/up and GLU in MMQ (#31188) 2026-10-28 18:34:28 +05:30
Yonghye Kwon a7af149a2a cuda : add top-k MoE fusion for 384 experts (#31288)
Assisted-by: Codex
2026-10-28 10:13:01 +02:00
Pascal 922f53ed33 vulkan: fix CI llvmpipe by allowing dim 3 broadcast in MUL_MAT (#31239)
* vulkan: allow dim 3 broadcast in MUL_MAT

supports_op rejected every MUL_MAT whose src0 and src1 differ in ne3,
although mul_mm, mul_mmq and mul_mat_vec already broadcast src0 over
dim 3 through broadcast3. Such a matmul then ran on the CPU with a
copy of the weight, and the split layout changed with the number of
streams in the ubatch.

Keep the strict check for MUL_MAT_ID only, and route the broadcast
cases away from mul_mat_vec_nc, the one path that requires equal ne3.

* tests : harden MUL_MAT with permuted dim 3 broadcast cases

Harden the MUL_MAT coverage with a weight broadcast over dim 3 against
a permuted per-stream activation, the shape of an MLA output
projection with a non unified KV cache, for a single token, several
heads and several tokens per stream.
2026-10-27 14:39:03 +02:00
Georgi GerganovandPascal de5122d879 ggml : relax scale to non-contiguous views (#31146)
* ggml : relax scale to non-contiguous views

SCALE requires src0 to be 1D-contiguous. Relax this to dim-0
contiguity with arbitrary row strides (ggml_is_contiguous_rows),
same as the unary ops in the Metal backend.

- ggml: relax the assert in ggml_scale_impl
- cpu: address src0 rows through their byte strides (f32, bf16)
- cuda: stride-aware kernel, fastdiv row decomposition
- metal: support bf16 scale (guarded unary instantiations)
- sycl: keep the gate at contiguous, the kernel is flat f32

Add view variants to test_scale. Verified on CPU, Metal and CUDA.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)

* cuda : skip the softcap fusion on non-contiguous views

The fused softcap kernel reads src and writes dst as dense arrays,
so a view reaching SCALE -> TANH -> SCALE now that SCALE accepts
non-contiguous rows would be read wrongly. Keep such graphs on the
strided kernels until the fusion becomes stride aware.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-27 07:55:31 +02:00
Pascal 4d71b42fd8 ggml-cpu : rewrite concat as one row kernel for all types (#31179)
Every dst row comes from src0 or src1, or for dim 0 from the row of
src0 followed by the row of src1, so a single kernel covers every
type: a dense source row is one memcpy, quantized types included,
and a strided source is copied element by element with the element
size known at compile time.

Rows are split across threads instead of i2 alone, which is often 1,
and are copied in tiles of 32: for a transposed source the elements
are walked in the outer loop, so the rows of a tile read neighboring
addresses instead of jumping a full row stride per element.

Add transposed b cases to test_concat, the layout of the delta-net
conv input, and a perf case for it.
2026-10-26 16:18:45 +02:00
virajwad 70c5f26324 vulkan: rms norm optimization using subgroup reductions (#29882)
* RMS Norm Subgroup reductions optimizaton

* Fix build error

* Improved optimization

* Make usage of BLOCK_SIZE consistent

* Add fast path to skip 2nd subgroupAdd

* Add AMD Vendor to subgroup reduction opt

* Add comment clarifying rms_dict and merge_maps behavior

* Add unroll

* Rebase

* Fixed syntax issue

* Add 128/256/512 block sizes to non-subgroup path

* Remove trailing whitespace
2026-10-26 16:14:31 +02:00
feni6andfeni6 9b8446b914 metal : stop overflow from happening in large matrix output offsets (#30072)
* Metal : stop overflow from happening in large matrix output offsets

Assisted-by: Codex

* metal : widen batch offsets and MUL_MAT_ID token indices

Assisted-by: Codex

---------

Co-authored-by: feni6 <270037094+feni6@users.noreply.github.com>
2026-10-26 10:13:39 +02:00
Aman GuptaandGeorgi Gerganov a074215352 deepseekv4.1: follow-up review fixes (#31137)
* kv-cache : use ggml_backend_tensor_copy to refill the DSV4 K mirrors

* llama : move the Engram pad id to llama_hparams and the token map to llama_model

The saver no longer needs the deepseek41 model class.

* deepseek41 : drop the F32 src1 request on the HC mixes matmul

* ggml : add softplus TODOs in the backends that still use log(1 + exp(x))

* convert : make the DSV4/DSV4.1 modify_tensors generators, drop the redundant vision filter

* deepseek41 : clean up comments

* deepseek41 : drop the F32 src1 requests on the router and indexer matmuls

* convert : use format_tensor_name directly in DeepseekV4Model

* s/comp_kv_src/comp_il_src/g

* cont : fix TODOs for log1p

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-10-26 15:52:02 +08:00
anujj c49edf0963 CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows (#29948)
* cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows

For prompt processing, run the gate and up matmuls of a dense FFN as one
mul_mat_q launch and apply the GLU in the write-back.

- The loaders take the gate weights as a second pointer and fill the weight
  tile with up and gate rows interleaved in chunks of 8, so each thread holds
  an up value and the matching gate value (rows r and r + 8 of an MMA tile).
- The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[]
  per thread. The fused write-back computes the GLU from registers and writes
  half the rows.
- The fused path uses the regular tiling (no stream-k) with the column tiles
  innermost so that neighbouring blocks share weight tiles.
- Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.

* cuda: assert the fused MMQ gate argument and trim redundant comments

* cuda: move the fused MMQ hint check to should_fuse and add TODOs

* cuda: pass prec_src1 to the fused MMQ write-back

* cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back

* cuda: require has_fusion explicitly, drop the fusion-only src1 precision check

* cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile

* cuda: inline the MMQ tiling-efficiency check

* tests: one fused gate/up MMQ case per type and GLU op

* cuda: always launch the fused MMQ kernel with one block per tile

* tests: shorten the fused gate/up MMQ test comment

* cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs

* cuda: run the fused MMQ kernel on the plain tiling path

* cuda: put the column tiles innermost in the MMQ plain tiling path
2026-10-25 11:24:38 +02:00
Aman Gupta a55c16e4ee ggml-cuda: fuse qwen MRoPE + set-rows (#31025) 2026-10-25 11:15:45 +02:00
Jeff Bolz e321bdad7e vulkan: Add a larger tile size for conv3d (coopmat2 only) (#29920)
* vulkan: Add a larger tile size for conv3d (coopmat2 only)

This adds a larger tile size, in practice it is only for coopmat2 where
we don't need shared memory to store the output (would use more than 64KB).

Add some additional representative backend tests. At least, this is what codex
came up with from looking at sd.cpp Wan VAE.

* only apply large tile size for newer driver versions
2026-10-25 11:13:48 +02:00
Aman GuptaandDaniel Han 74bca39500 model: DSV4.1 flash (#31086)
* model : add DeepSeek-V4.1-Flash (deepseek41)

* deepseek41 : remove MXFP8, convert the Engram table to BF16

* deepseek41 : compute the compressed-KV read layers once in the cache

* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le

* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather

* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer

* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle

* deepseek41 : keep il last in build_hc_mixes

* deepseek41 : sort the top-k rows before the sparse gather

* tests : add deepseek41 to the Metal fusion baseline

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-10-25 09:43:51 +08:00
Ehsan Bateni 78a8ae14be hexagon: KV-head team split for multicore HMX flash attention (#31055)
* hexagon: KV-head team split for HMX flash attention in multicore

* hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag

0 = off (token-split only); 1 = head-split, plus team-split when
n_kv_heads < N_cores so stranded cores still participate (e.g. gemma-4
with 2 KV heads on 4 cores). Where heads divide evenly across cores,
plain head-split is used.
2026-10-24 12:19:32 -07:00
Georgi Gerganov 3f03b17724 metal : remove mul_mm_id amax rescaling (#30924)
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the
per expert-token amax scaling done in the compute graph
(ref: https://github.com/ggml-org/llama.cpp/pull/30861), which keeps the
MoE down-projection activations within f16 range before the op runs.

Drop the two-stage reduction kernels, their scratch buffer and the
GGML_PREC_F32 dispatch added in https://github.com/ggml-org/llama.cpp/pull/26223,
as well as the test cases that drive MUL_MAT_ID src1 out of f16 range.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-24 18:40:19 +03:00
Xuan-Son Nguyenandyomaytk 701fd25b3a ggml : allow MUL_MAT_ID -1 index to skip the computation (#26631)
* ggml : allow MUL_MAT_ID -1 index to skip the computation

* nits

* update vulkan code

* ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec

(cherry picked from commit d8a4b9efb6e2b9abf3a866e847756b7e6bbc6b69)

* fix cuda

* openvino

* exclude CUDA from this PR

* add TODO tag

---------

Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
2026-10-24 12:15:36 +02:00
Alexey Kopytko 5c31142682 qwen4exp : build the QSA selection without the kq_mask (#30965)
* qwen4exp : build the QSA selection without the kq_mask

* clean

* lang

* qwen4exp : reshape the fused QSA selection to the kq_mask streams

ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while
build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream].
With several streams the shape assert failed at graph reserve.

* test the QSA optimization

* reword

* update build_fused_top_k_mask instead of handwiring

* simplify

* keep il at the end for consistency
2026-10-23 19:49:08 +02:00
Max Krasnyansky 643b0cdacb hexagon: Op Fusion refactoring, optimizations and new fusions like BINARY_NX (back-to-back ADDs, etc) (#30843)
* hex-fusion: support for GELU+MUL, REPEAT+GET_ROWS and BINARY_NX (back-to-back ADD/MUL/...)

* hex-fusion: factor out fusion logic into separate .h and .cpp and update all related code

* hex-fusion: allow for RMS_NORM+MUL fusions for non-contig tensors

* hex-fusion: streamline fusion eligibility checks and cover missing corner cases

* hex-fusion: streamline fusion flow

* hex-opbatch: make n_tensors limit more scalable

* hex-bufs: adjust the trace buff sizing

* hex-glu: cleanup glu-ops interfaces and add kernel_params

* hex-kparams: clean up kparams handling in unary, ssm-conv, and other ops

* hex-kparams: update the rest of the ops PAD,DIAG,... to use kparams

* hex-unary: add support for collapsing dims

* hex-ops: cleanup redundant runtime checks

* hex-ops: consistent use of kernel_type in all kparams

* hex-fusion: remove redundant check in gdn+cpy

* hex-fusion: further streamlining of fusion rules and code

* hex-fusion: cleanup naming

* hex-opfmt: cleanup op formatting

* hex-fusion: bump fusion lookahead to 64

* hex-fusion: fuse SSM_CONV+SILU

* hex-fusion: fix hvx register spills in fused ssm_conv+silu

* hex-fusion: add more GLU fusions

* hex-fusion: remove duplicate checks and use ggml-common helpers

* hex-kparams: refactor kparams into htp-kparams.h/cpp

* hex-fusion: fix binary-nx fusion regression

* hex-kparams: another pass at cleaning up kparams (removing unused, etc)

* hex-kparams: streamling nameing and move _check_ wrappers into .h

* hex-kparams: templetize htp_kparams_check_<op> to reduce code bloat

* hex-ops: de-duplicate op dispatch switches

* hex-kparams: trim down exposed kparams calls

* hex-kparams: simplify type casting of kparams structs

* hex-fence: de-dup fence tensor construction

* hex-opbatch: simplify tensor shape init/checks

* hex-mm: cleanup leftovers of the original QK defines

* hex-fa: fix stack overflow on older npu firmware

* hex-ctx: move dma_caches into ops_ctx
2026-10-22 21:05:44 -07:00
budiony d81d0d8fb9 Multi-column on Intel XMX matrix engines refactored (#29864)
* Multi-column on Intel XMX matrix engines refactored

Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).

* Fixed regression bug

real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4

* Removed GGML_TYPE_Q8_0 activation

* rewritten xmx_mul_mat

64-byte alignment and 16-byte pitch check on the activations

* Implemented fused gate/up kernel

One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.

* Removed the temporary XMX_NOGLU environment switch

* Q2_K min-sum optimization

Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group

* Added dispatch check and GRF sweep

tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.

* Renamed variables

* Added 2 environment variables

GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16):  the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.

* Update esimd.hpp

1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.

2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.

The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
2026-10-22 21:39:01 +08:00