Commit Graph
3890 Commits
Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 8742446dba ggml-vulkan: env knob for the concat transpose fast path
GGML_VK_CONCAT_FASTPATH=0 forces the generic path and =log reports each
fast-path dispatch, so validation can prove the path fires at prefill and
not at decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 7194d47311 ggml-vulkan: skip the concat transpose fast path below one tile
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a
loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of
40 layers) build_conv_state emits a transposed concat per recurrent layer, so
the fast path runs 30 times per decoded token.

At one token the transposed source is a single column, so there is no
uncoalesced stride left to fix, but the path still costs two dispatches and an
unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes
the command stream. Measured -17% tg256 on RX 580; the barrier, not the
half-empty transpose dispatch, is nearly all of it.

Require the transposed source to be at least one TILE_DIM wide. That source is
qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the
fast path, decode and small speculative drafts take the generic one.

pp8192 +11% retained, tg256 back to parity, generation byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
Lumpiasty fa24747925 ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.

Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty cfda297cbb ggml-vulkan: tiled transpose fast-path for concat with transposed source
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).

Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).

Assisted-by: opencode
2026-11-21 19:32:13 +01:00
LumpiastyandClaude Opus 5 e1bc1f7398 vulkan: record why flash-attn shmem staging stays off for GCN
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.

Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.

Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-11-21 19:32:13 +01:00
Lumpiasty b5085151f2 ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.

Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.

Assisted-by: Claude
2026-11-21 19:32:13 +01:00
Xuan-Son Nguyen 19bc41dbb4 model: apply .scale tensors automatically + fix correctness (#32669)
* model: apply .scale tensors automatically

* rm build_moe_ffn "_s" overload

* use llama_prec_policy

* fix ci

* build_lora_mm_id: rm w_s
2026-11-21 16:30:01 +01:00
Frosty40andTitaniumtown e50c254a22 sycl: bound CONV_TRANSPOSE_1D input iteration (#31109)
Used angelX to optimize SYCL/drama 3 coordination, results are as follows

Each output reads only inputs with i*s0 <= idx < i*s0 + K, in the same ascending order as before.

Arc Pro B70, oneAPI 2026.1, test-backend-ops perf, base 9b56ccf00 vs patch, median of 3 same-boot A/B/A rounds, A/A 1.000-1.001:

| input [ne0, ne1] | kernel [K, OC, IC] | s0 | base us | patch us | speedup |
|---|---|---|---|---|---|
| [16, 1536] | [16, 768, 1536] | 8 | 3910.57 | 1348.59 | 2.90x |
| [128, 768] | [16, 384, 768] | 8 | 37735.00 | 1665.26 | 22.66x |
| [1024, 384] | [8, 192, 384] | 4 | 270371.75 | 1471.74 | 183.71x |
| [4096, 192] | [4, 96, 192] | 2 | 534614.63 | 752.74 | 710.23x |

Shapes are the drama 3 codec decoder upsampling stack, added as perf cases. Chained 4-layer stack at 1/4/24 input frames: base 66.3/847.4/28638.5 ms, patch 2.3/6.7/30.7 ms. Kernel numbers, not end-to-end TTS latency.

Correctness: test-backend-ops -o CONV_TRANSPOSE_1D 116/116 on base and patch, and 116/116 rebased on 3f03b1772. Seeded output dump (7.46 MB) byte-identical to base.

Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857). With only those cases excluded: 20145/20173 on base and patch, and every patch-run failure also fails on base.

Co-authored-by: Titaniumtown <titaniumtown@proton.me>
2026-11-21 16:20:29 +02:00
Kazutoshi, Akao 660be5506b ggml : improvements to the memory access pattern in ggml_compute_forward_mul_mat_one_chunk (#30846) 2026-11-21 16:16:40 +02:00
Titaniumtown c661033349 sycl: add IQ4_NL and IQ4_XS weight reorder (#31998) 2026-11-21 15:11:14 +02:00
Titaniumtown 44c20b1c0a sycl: column-blocked gated_delta_net kernel (#32456) 2026-11-21 14:38:00 +02:00
Clemens Wasser a818a46f3d sycl: add fused ADD, RMS_NORM, MUL and vectorize rms norm (#31753)
* sycl: vectorize contiguous rms norm

* sycl: add fused ADD, RMS_NORM, MUL

* tests: test fused ADD, RMS_NORM, MUL at decode widths
2026-11-21 14:20:22 +02:00
Johannes Gäßler b019be0c86 CUDA: abort on UB in fastdiv (#32698) 2026-11-21 13:46:59 +02:00
Ruben Ortlam 362ac92974 vulkan: unroll FA coopmat1 array due to a driver bug on AMD Windows (#32466)
* vulkan: unroll FA coopmat1 array due to a driver bug on AMD Windows

* simplify code to make it clearer
2026-11-21 13:46:20 +02:00
KaiFelixBennettandRuben Ortlam fb7090df56 vulkan: improve RDNA4 MMVQ performance for Q3_K, Q5_K and IQ4_XS (#31804)
* vulkan: keep default MMVQ rows for Q3_K, Q5_K, IQ4_XS on RDNA4 AMD proprietary driver

Assisted-by: Claude Opus 5.5

* clean up mmvq shader load logic

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-21 13:41:55 +02:00
Aaron Teo 3da11de1f1 ggml-zdnn: fix ztensor load race condition (#32279)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-21 13:05:23 +02:00
shalinib-ibm 93cb37ea7b ggml-cpu: fix nc aligment in tinyBLAS_PPC_Q0 (#32659)
Fixes https://github.com/ggml-org/llama.cpp/issues/31680

The matmul_tiled() implementation in tinyBLAS_Q0_PPC requires nc to be a multiple of 8 to perform matrix multiplication correctly.

The previous alignment logic contained a special case for n == 4, which allowed n_aligned to be 4. Consequently, the tiled path could be selected with nc = 4, violating the kernel's tile-width requirement.

This caused incorrect matrix multiplication results, observed as garbled inference output when running  Flint Q4_K_M and Gemma 3 27B Instruct.
2026-11-21 12:58:49 +02:00
cwriter 418fe97b5f sycl: Improve Grouped XMX GEMM by avoiding SLM staging (#31428) 2026-11-20 22:37:40 -05:00
Pelayo Garcia BárcenaandJohannes Gäßler e3c3f666a4 hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1) (#32389)
* hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1)

* hip: write the RDNA1 MMVQ values explicitly instead of recursing into the generic table

* hip: list the RDNA1 MMVQ table before RDNA2 in the enum

* Update ggml/src/ggml-cuda/mmvq.cu

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-11-20 22:52:19 +01:00
Tarek Dakhran a83d68b562 metal : allow the norm chain in either operand of the fused MUL/ADD (#32300)
Assisted-by: Claude Opus 5.5
2026-11-20 19:55:44 +01:00
Georgi Gerganov 7716ec8cbc ggml : major version bump reminders + deprecate ggml_ftype (#32647) 2026-11-20 19:36:42 +02:00
Ruben Ortlam 6f5c9ad217 vulkan: add support for COL2IM_1D misalign bytes and split workgroups (#32673) 2026-11-20 19:11:59 +02:00
Georgi Gerganov 4b69c79cca cuda : update GDN supports_op check (#32660) 2026-11-20 14:35:20 +02:00
Aaron Teo bed8ec2f8d llamafile(s390x): add QK data type support (#32600)
* llamafile: add QK support for s390x

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* llamafile: clean up for s390x

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-20 09:47:41 +02:00
sys2008 52e8db6550 kleidiai : allow OHOS (HarmonyOS) AArch64 target (#32265)
* kleidiai : allow OHOS (HarmonyOS) AArch64 target

* kleidiai : allow OHOS AArch64 target

Assisted-by: GLM
2026-11-20 09:46:59 +02:00
Nikhil Jain d7d1b632da webgpu : fix 2D indexing for copy kernel (#32576)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* resolve merge conflicts

* Add 2d indexing to copy kernel

* move test
2026-11-20 09:45:44 +02:00
Teng Mei ea84d70903 CUDA: optimize Orin single-column MMVQ (#31268)
* CUDA: optimize Orin single-column MMVQ

* Create a separate Orin branch in `calc_rows_per_block`
2026-11-20 09:44:04 +02:00
Chinmay ShrivastavaandJonSnow1807 9fbe30c2a0 CUDA: keep the MUL_MAT_ID precision in the per-expert fallback (#31345)
* tests: add CUDA-only MUL_MAT/MUL_MAT_ID cases with results above the fp16 range

IQ1_M weights have no MMQ kernel, so once the batch is bigger than the MMVQ
limit the path is dequantize + cuBLAS. These cases produce results around
1.3e5, past the fp16 max of 65504, so an fp16 result buffer turns them into inf
while the CPU reference stays finite. They are only added for the CUDA backend.
Vulkan deliberately accumulates quantized matmuls in fp16 at default precision
and would fail them by design.

* CUDA: use an fp32 result for quantized src0 in the cuBLAS fallback

On the F16 compute path, quantized weights went through cuBLAS with fp16
accumulation and an fp16 result buffer that was converted to fp32 afterwards.
Anything above 65504 became inf and the next layer turned into NaN (see #30951:
gpt-oss-120b with IQ1_M experts at -ub 512). Set prefer_f32_output for
quantized src0 so the GEMM still uses fp16 operands but accumulates and stores
in fp32, the same mode already used on Volta, RDNA4 and CDNA. Every other
quantized path already accumulates in fp32.

Diagnosis and fix by denizyuret in #30951.

* CUDA: keep the MUL_MAT_ID precision in the per-expert fallback

The per-expert dst_slice kept op = GGML_OP_NONE, so ggml_prec_get_acc returned UNDEFINED and a requested precision was ignored. Set the op to GGML_OP_MUL_MAT. Also revert the fp32 result change and remove the tests.

---------

Co-authored-by: JonSnow1807 <JonSnow1807@users.noreply.github.com>
2026-11-20 09:42:16 +02:00
Amadeus dyw c0b97be0b2 vulkan: optimize RMS_NORM_MUL_ROPE loop control (#31149) 2026-11-20 09:38:43 +02:00
Ruben OrtlamandJeff Bolz 16f80bd0c0 vulkan: chunked GATED_DELTA_NET implementation (#30207)
* vulkan: add cm2 chunked GATED_DELTA_NET shader

* fix shmem bank conflict bottleneck

* optimize further

* add coopmat maintenance1 variant

* optimize for RDNA4

* add gmem version RDNA4

* optimize for Ampere

* fix nvidia shmem overuse

* fix cm2 validation errors

* improve cm2 perf

* improve cm1 perf

* clean up

* enable by default

* show cm maintenance1 marker in device info

* properly check for cm_m1 features

* add tests

* add v32 mode, improve chunked condition

* allow other subgroup sizes

* vulkan: specialize the GDN CM2 workgroup size

Assisted-by: Codex

* vulkan: coopmat2 GDN implementation

* work around Nvidia shader compiler crash

* remove shared shader cm2 code

---------

Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
2026-11-20 09:38:14 +02:00
Leon 2d8d003937 webgpu : add COL2IM_1D support (#31708) 2026-11-20 09:36:58 +02:00
7e960f40fb opencl: fold GQA head for f16 FA for better efficiency (#32631)
Assisted-by: Codex

Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-11-19 22:26:28 -08:00
Hongqiang Wang 8bf88ba936 vulkan: disable subgroup arithmetic on Qualcomm's driver (#30826)
On Adreno, subgroupAdd() after divergent code returns wrong sums. The
fused ADD+RMS_NORM partial-sum path hits this, so the decode-step norm
is garbage and models emit stray tokens after long prompts.

Clear subgroup_arithmetic for the Qualcomm proprietary driver, which
also turns off add_rms_fusion and the other subgroupAdd paths.

test-backend-ops on Adreno 850 and X2-90: ADD_RMS_NORM 20/25 -> 25/25,
RMS_NORM_MUL_ADD 33/45 -> 45/45. llama 3B Q4_0 on Adreno 850: pp512
and tg128 unchanged.
2026-11-20 05:58:46 +01:00
Johannes Gäßler b94959a64d CUDA: implement support for duplicate expert IDs (#32505)
implement support for duplicate expert IDs
2026-11-19 23:21:24 +01:00
c406f4367e opencl: support dp4a for KQ mm in decomposed FA (#32577)
* opencl: support int8 KQ for DK = 512

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex

* opencl: allow query chunk up to 96 for int8 KQ

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex

* opencl: extend `build_program_from_source` to allow non-fatal failure

Assisted-by: Codex

* opencl: add int8 KQ for decomposed FA with DK <= 256

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-11-19 11:30:19 -08:00
anujj 7fce5439fe ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA) (#32387)
* ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA)

* cuda : SET_ROWS from BF16 into quantized types and F8_E4M3

* cuda : fuse BF16 rope into a BF16 cache

* tests : trim the BF16 rope cases to one per code path
2026-11-19 23:43:55 +05:30
9a080f44b5 ggml-openvino: fix OpenVINO CI, minor fixes and optimizations (#32590)
* ggml-openvino: round the GPU KV length up to a power of two

Review notes (follow-ups, not blocking):
- convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
  copies it into the ov::Tensor. Writing the padded rows straight into
  padded.data() avoids the second allocation and copy (up to 16 MB per
  call at a 16K context).
- The padding is skipped when the mask is not contiguous, and the KV
  inputs are then sliced to more rows than the mask has columns. Masks
  are always contiguous today; a GGML_ASSERT would make that explicit.
- compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
  is_stateful_enabled() is local to ggml-openvino.cpp.

* ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

* ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

* ggml-openvino: Remove solution C and keep A as default

* ggml-openvino: match the output count before reusing a compiled NPU prefill

The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.

* ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding

* ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix

- compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
  ran before the new one and still exported result_norm, so the
  result_norm exclusion had no effect. Keep only the new block.
- compute_node_dynamic_dims checked is_activation_input twice in a row.
  Remove the second copy.
- process_view_input_new created the dynamic Split on split_dim. With
  stateful execution the input drops the leading size-1 axis, so use
  split_axis, which already accounts for the rank difference.

* ggml-openvino: hash the output flags into the model-cache fingerprint

The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.

* ggml-openvino: support f16 SCALE and f16/bf16 TRI

test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.

SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.

* ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
 layer

The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.

* enable mamba based models on OV CPU-GPU

* revert changes in tests

* docs : add Falcon-H1 to the OpenVINO validated models

Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.

* ggml-openvino: simplify the SSM_SCAN build check and clean up

- Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
  always provides the op. Drop the no-op LANGUAGE CXX source property.
- Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
  with -Wextra-semi, which fails builds with fatal warnings.
- Add ggml_openvino_is_static() for the repeated NPU or
  GGML_OPENVINO_FORCE_STATIC check.
- Share the f16 and f32 KQ mask padding code.
- Drop the redundant inline on in-class static functions.

---------

Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
2026-11-19 07:22:26 -08:00
Bhargav Krish c7468107ef cuda: Scatter kernel for set_n (#32489)
* cuda: add scatter for batched tensor writes

* update test case to mirror get_n test case

* save restore timing logs

* refactor code into batch-copy.cu file
2026-11-19 20:14:47 +05:30
ac830a9cbb ggml : add backend scheduler tests (#27258)
* Adds initial scheduler stress tests

* Adds new sleep op to simplify scheduler testing

* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321

* Adds experimental vulkan GGML_OP_SLEEP

* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.

* Cleanup: Removes CUDA specific comments and nvtx-markers

* Fix: forgot to add sleep specific test case

* Fix oversights identified by CI

* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior

* Fix: Early exit in CPU-only settings (cause for graviton CI fails)

* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds

* Exclude ACCEL devices from new tests due to incompatibility

* Fixes vulkan test failure in llvmpipe

* Adds asserts, improves comments and some checks

* Fixes nitpicks in white-space and variable naming

* Adds comments that certain vk drivers could time OP_SLEEP incorrectly

* Drops limitiation to CPU/GPU in test-sleep

* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)

* Implements feedback for OP_SLEEP to work in place (no copy)

* Apply non-code suggestions  from JohannesGaesslers code review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail

* Make clang-tidy happy

* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)

* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name

* Drops unnecessary shader_clock.comp test and guards

* Replaces sched_ prefixes with helper_ prefix

* Moves test suite to new RAII types for better memory management

* Adds sleep option to test_chain_all_backends

* Adds sleep option to first-run arm of the y-shaped test

* Fix switchcase order for legibility in stress_test_dag

* Simplfies stress_test_linked_list via double for-loop

* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)

* Fix RPC Proto patch due to GGML_OPS bump

* Adds ggml-cpp.h treatment to test-sleep.cpp

* Adds TODO pointers for subsequent additional tests.

* Apply suggestion to ggml.h

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)

* Apply batched suggestions from gg code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Completes callsite change of gg-suggestion

* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-19 16:40:13 +02:00
Alexey Kopytko f8caa5ea30 metal : fix the second argument of NSMakeRange (#32602) 2026-11-19 13:49:22 +02:00
SS1618 4db8b7638f webgpu: add POOL_2D support (#32250)
* Shader, pipeline, and launcher

* Update docs and add newline to wgsl

* Formatting fixes
2026-11-19 13:35:10 +02:00
PascalandGeorgi Gerganov 41d8297fe4 metal : CONV_2D tensor API for M5, split-K for all chips (#32542)
* metal: add a tensor API conv_2d kernel and split long reductions

The conv_2d matmul kernel runs on simdgroup matrices only, and gives
each threadgroup the whole reduction of its output tile. On M5 this
leaves the tensor cores unused, and convolutions with few output
positions and a long reduction (IC * KW * KH in the thousands) run a
handful of threadgroups on an otherwise idle GPU.

Add kernel_conv_2d_mm_tensor for devices with the tensor API: like
kernel_mul_mm, it gathers the patches of 64 output positions into
threadgroup memory as the left operand and reads the kernel rows
from device memory as the right operand. Devices without the tensor
API keep the simdgroup kernel.

Both kernels can now split the reduction until there are enough
threadgroups: each split writes its partial sums to a buffer placed
after the output, as flash attention does, and a small kernel adds
them in a fixed order, so the result stays deterministic. CONV_2D is
marked as an op whose allocation may exceed its output size.

test-backend-ops -o CONV_2D passes, with and without
GGML_METAL_TENSOR_DISABLE.

CONV_2D, F16 kernel, us/run on an M5 Max:
  [19,19,256,16] k [4,4,256,4096]     19008 -> 5235
  [16,16,128,8] k [3,3,128,512]         333 -> 124
  [1536,1536,32,1] k [3,3,32,64]      12086 -> 6512
  [33,1,1280,1] k [3,1,1280,1280] s2    659 -> 33
  [2054,1,768,1] k [7,1,768,768]       3161 -> 699

* metal: walk the conv_2d patches incrementally, size splits by output

The patch gather of both conv_2d matmul kernels spent most of its
time on address math: 64-bit multiplies, divisions and four bound
checks per value. Walk the reduction index with 32-bit byte offsets
inside the image, updated by additions, with one unsigned compare per
bound; the host checks that an image fits. The tensor kernel no
longer gathers the reduction indices past the end of its split.

The number of splits now targets about 1M partial sums, with at least
3 reduction steps per split, instead of counting the tiles of one
kernel.

CONV_2D, F16 kernel, us/run on an M5 Max:
  [19,19,256,16] k [4,4,256,4096]     4998 -> 4512
  [1536,1536,32,1] k [3,3,32,64]      6721 -> 5354
  [4096,1,1024,1] k [7,1,1024,1024]   2701 -> 2236
  [2054,1,768,1] k [7,1,768,768]      1050 -> 871
  with GGML_METAL_TENSOR_DISABLE:
  [4096,1,1024,1] k [7,1,1024,1024]   9047 -> 8283

* nit: share the CONV_2D simdgroup count between host and kernel

kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per
threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel
constant and res.nsg, so the dispatch and the kernel cannot drift apart.

* metal: fall back to the generic conv_2d kernel for images past 4 GiB

The matmul conv_2d kernels walk the patches with 32-bit byte offsets
inside an image. Instead of asserting on nb13, compute the largest
offset actually read from ne/nb, which also covers strided views of
the input, and route images that do not fit to the generic kernel.

* cont : comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-19 11:48:44 +02:00
Bartowski 76f186df71 ggml : IQ2 creation speedup through caching (#31910)
* IQ2 quant creation speedup from caching

* Rename variables for clarity

* Explicit casts

* Initialize all values, add comment to tolerance
2026-11-19 10:04:58 +02:00
Birk Skyum 7a4117eef9 vulkan : enable MMQ with mixed signedness integer dots (#31045)
* vulkan : enable MMQ with mixed signedness integer dots

* vulkan: specialize MMQ shaders for mixed-signedness dot products

* vulkan: track integer dot product support with signed, mixed and unsigned flags

* vulkan: move signed dot check to the MMVQ call sites
2026-11-19 08:15:09 +02:00
Jeff Bolz 573aa89de6 vulkan: Implement stream-k MUL_MAT (#28528)
* vulkan: Implement stream-k MUL_MAT

Stream-k tries to split chunks of 256 K elements evenly across all SMs.
Workgroups that finish a tile write to dst, others write to scratch, and
there's a resolve shader similar to split-k.

The shader code is implemented for scalar+cm1+cm2, but only currently enabled
for cm2. The heuristic for enabling on coopmat2 is pretty simple due to the
large tile size (one workgroup per SM), but would be more complex for the tile
sizes being used for cm1, and it doesn't make sense for me to tune that on
NVIDIA devices.

Assisted-by: Codex

* address review feedback
2026-11-19 08:14:14 +02:00
Daniil Sterkhov ecafc67145 vulkan: use floating-point zero for storage-only FP16 in DIAG/TRI (#32379)
* vulkan: Change integer zero to floating-point zero for storage-only FP16 in DIAG/TRI.

Assisted-by: OpenAI Codex

* delete narrow test
2026-11-19 08:13:06 +02:00
Aman Gupta d4f5c26049 CUDA: use 64-bit dst offsets in mul_mat_q (#32374)
The MMQ kernels computed the dst tile offset (jt*J*stride_col_dst) in
32-bit int. For a dst with more than 2^31 elements, column tiles that
start past INT_MAX/nrows wrap around and write out of bounds.

This happens with an all-logits output head, e.g. llama-perplexity
-c 32768 -b 32768 -ub 16384 on a 248320-vocab model: the q8_0 output
matmul has a 248320 x 16384 dst (4.07e9 elements) and aborts with an
illegal memory access.

Widen the offsets in mul_mat_q and the stream-k fixup to int64_t, and
add a (disabled by default, ~9 GB) test-backend-ops case.

Assisted-by: Claude Opus 5.5
2026-11-19 08:11:44 +02:00
lhez 08c252e221 opencl: allow adjusting KV padding in decomposed FA (#32522)
Assisted-by: Codex
2026-11-18 20:11:46 -08:00
En YaoandXuan Son Nguyen b001d7bb22 ggml : WASM Relaxed SIMD Enhancement (#19590)
* add relaxed simd

* fix indentation

* remove comments and whitespace

* wasm: fix relaxed SIMD kernels and add q3_K/q6_K support

* rm trailing space

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-19 01:36:17 +01:00
Adrien Gallouët edec462cc0 ggml, ci : use llvm 22.1.8 (#32548)
* ggml, ci : use llvm 22.1.8

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Update build-cpu.yml too

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Fix clang 22

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Fix release.yml (arm64 with vulkan)

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Fix server.yml

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-18 21:10:23 +01:00