Commit Graph
142 Commits
Author SHA1 Message Date
Georgi Gerganov 494d0f47bd ggml : add MUL_MAT src1 precision tests and gating (#31896) 2026-11-10 17:03:27 +02:00
Takamichi Ohara 9f0d80cf4a ggml : support YaRN truncation config on openvino, sycl, vulkan, webgpu, rpc (#30802)
* openvino : support YaRN truncation config

* sycl : support YaRN truncation config

* vulkan : support YaRN truncation config

* webgpu : support YaRN truncation config

* rpc : support YaRN truncation config
2026-11-09 11:59:07 +02:00
LeonandMasashi Yoshimura a691b23458 webgpu : add COUNT_EQUAL support (#31578)
* webgpu : add COUNT_EQUAL support

Assisted-by: Codex

* webgpu : reduce small COUNT_EQUAL inputs in one dispatch

Assisted-by: Codex

* webgpu : simplify COUNT_EQUAL and reuse existing tests

* webgpu : move COUNT_EQUAL pipeline cache declaration

* Update ggml/src/ggml-webgpu/ggml-webgpu.cpp

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>

* tests : simplify COUNT_EQUAL coverage

---------

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>
2026-11-08 11:50:09 +02:00
PascalandGeorgi Gerganov d6be065572 llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (#31851)
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls

Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.

* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends

Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.

* tests: cover GGML_PREC_BF16 in the flash attention tests

The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.

* llama: request GGML_PREC_BF16 for the KQ multiplication

* cont : add TODO in clip.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-08 08:37:46 +01:00
Nikhil Jain cf659f240b webgpu : Implement group norm kernel (#31490)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* Add WebGPU group_norm kernel

* clean up

* resolve merge conflicts

* parameterize data type; clean up cmake changes

* add nullptr check

* clean up webkit experiments

* more cleanup

* more cleanup

* clang format

* Remove unused feature check for WebGPU

* clean up supports_op condition and update documentation

* add perf tests for group_norm

* clean up doc with script and undo change in test-backend-ops

* Do 2d dispatch of workgroups

* Do 2d dispatch of workgroups

* Unstage unrelated changes

* undo docs churn

* add new test case

* undo small change

* undo small change
2026-11-07 10:01:27 +02:00
Masashi Yoshimura 5d3ed2ae7e webgpu: add batched tensor API (get_tensor_n and set_tensor_n) (#31473)
* add set_tensor_n and get_tensor_n

* fix for editor-config checker

* remove the outdated comment
2026-11-06 11:35:47 +02:00
Takamichi Ohara 1fa7f1c488 ggml : support truncate=false for YaRN (#30206)
* ggml : support `truncate=false` for YaRN

YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.

Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.

* support CUDA backend

* add tests

* move RoPE helpers to internal header

* format
2026-11-05 12:02:05 +02:00
Leon 553d2c2e7e webgpu : add MEAN support (#31591)
Assisted-by: Codex
2026-11-05 10:29:14 +02:00
Masashi Yoshimura b393d73bb0 webgpu: return maxBufferSize from get_max_size and avoid whole-buffer bindings in memset/clear (#31348)
* Fix get_max_size and improve the memset shader

* Fix buffer_memset to  handle the buffer size more than max binding size.
2026-11-04 15:46:06 +02:00
Leon 713319e284 webgpu: add ARANGE support (#31455) 2026-11-03 07:54:00 +02:00
Aman GuptaandGeorgi Gerganov 1b6935b227 llama : flash_attn_ext_rows - more efficient unified kv for multi-seq (#29510)
* llama : use llm_attn_mask and kv_rows for unified attention

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* cont : optimize `get_seq_cells`

* llama : build the kv_rows lists from per-sequence cell bitmaps

Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.

Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).

* cont : make get_kv_rows_p_min private

* cont : minor metal

* llama : use kv_rows for every ubatch of a unified cache

A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.

Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.

* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-02 18:13:33 +02:00
Pascal 1c5ed702c9 webgpu : fix CI by keeping scratch alloc sizes monotonic in batch length (#31422)
* webgpu: keep scratch alloc sizes monotonic in the batch length

The flash attention vec path and the MMVQ path append a scratch area
after dst, but get_alloc_size only accounted for it when the current
batch takes that path. The size was therefore not monotonic in the
batch length: a graph reserved at n_ubatch (tiled path, no scratch)
or at one token did not cover batches of 2 to 19 query rows, and
GGML_SCHED_NO_REALLOC builds aborted on an unexpected reallocation.

Pass the row count to both path predicates and size the scratch with
the row count clamped to the longest batch the short path serves.

* nit: align the mmvq pipeline key with clang-format 22

* webgpu: address review @yomaytk, name the mat-vec row threshold
2026-11-02 09:36:04 +02:00
Georgi Gerganov 418f905a4e ggml : add batched tensor copy API (#30881)
* ggml : add generic batched tensor copy API

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : use batched tensor copy API in context io

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : add batched tensor copy API

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : batch device io tensor copies

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : add CUDA support for batched tensor APIs

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : remove async tensor_n APIs and add null buffer hooks

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : fix batched tensor 2D detection

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : group batched tensor operations by buffer type

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : minor indentantion

* cont : const correctness

* tests : add mock buffer type tests for batched tensor API

Restructure test-backend-tensor to run a unified test set on each backend buffer type and on an ad-hoc mock buffer type. The mock set exercises the dispatch logic in ggml_backend_tensor_{set,get,copy}_n: hook invocation, per-tensor fallback, cpy_tensor_n all-or-nothing, and buft grouping.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : add TODO
2026-10-29 20:52:17 +02:00
Evan Huus 9125a2ebe8 ci : fix race in webgpu memset (#31162) 2026-10-26 18:02:12 +09:00
Xuan-Son Nguyenandyomaytk 701fd25b3a ggml : allow MUL_MAT_ID -1 index to skip the computation (#26631)
* ggml : allow MUL_MAT_ID -1 index to skip the computation

* nits

* update vulkan code

* ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec

(cherry picked from commit d8a4b9efb6e2b9abf3a866e847756b7e6bbc6b69)

* fix cuda

* openvino

* exclude CUDA from this PR

* add TODO tag

---------

Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
2026-10-24 12:15:36 +02:00
Chanho Lee 0b1281aeef webgpu : add Q2_0 support (#30914)
* webgpu : add Q2_0 row dequantization

* webgpu : add Q2_0 matrix multiplication
2026-10-22 08:30:04 +03:00
Andrea Rossetti 0878f7c304 ggml: support non-contiguous SUM (CPU, CUDA, Vulkan, WebGPU) (#28851)
* ggml: support non-contiguous SUM (CPU, CUDA)

* tests: add non-contiguous SUM cases

* tests: SUM over non-contiguous views

* vulkan: support non-contiguous SUM

* webgpu: support non-contiguous SUM
2026-10-18 18:57:36 +02:00
Masashi Yoshimura fcac19079f webgpu: (Fix CI) fix mul_mat_id with duplicate expert ids (#30451)
* Fix mm-id of webgpu in the case of ids duplication

* formatting

* fix comments to avoid breaking up sentences in comments
2026-10-14 08:24:23 +09:00
Masashi Yoshimura ba6439a6b5 webgpu: use 2D workgroup dispatch for all the ops which use 1D dispatch (e.g., rms_norm) (#30219)
* remove 1D workgroups dispatching

* formatting
2026-10-09 16:00:37 +03:00
Pascal e94acad853 ci: fix Models Backend webgpu by supporting GGML_OP_DUP (#30216)
The mixed batch path of PR-29622 writes the token rows with set_rows
into a dup of the embeddings. WebGPU did not support DUP, so the dup
ran on the CPU while the set_rows writing into it was scheduled on
WebGPU, which then bound a CPU buffer and crashed. DUP is the same copy
as CPY and CONT and now goes through the same path.
2026-10-09 12:27:32 +02:00
Masashi Yoshimura 36a73916ee ggml-webgpu: fix flash_attn supports_op check for overlapping KV (#28205) 2026-10-07 09:56:53 +02:00
Mendy Berger f498f864fb ggml-webgpu: no dawn native features on wasi (#27069) 2026-10-07 13:29:30 +09:00
Masashi Yoshimura 11fe02151f webgpu: add f16 support to fill/set_rows (#29897) 2026-10-04 09:07:47 +09:00
Georgi Gerganov 631109b34d ggml : add alloc_buffer_n to buffer type interface (#23671)
* ggml : add `alloc_buffer_n` to buffer type interface

Add alloc_buffer_n method to ggml_backend_buffer_type_i
interface, with a public API ggml_backend_buft_alloc_buffer_n.

- Default implementation in ggml-backend.cpp handles multi-buffer
  splitting and tensor allocation via ggml_tallocr
- Meta buffer type provides custom implementation that creates
  per-device sub-contexts and delegates to simple buffer types
- ggml_backend_alloc_ctx_tensors_from_buft now collects tensors
  into a list and delegates to the new API
- Remove temporary ggml_backend_meta_alloc_ctx_tensors_from_buft
- Add NULL alloc_buffer_n to all existing buffer type
  interfaces (cpu, metal, openvino, hexagon, webgpu, zdnn, virtgpu, repack)

Assisted-by: llama.cpp:local pi

* cont : fix `cur_buf_size` init after flushing a buffer

* ggml : add TODO tag for shared buffer split logic

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* tests : add alloc_buffer_n coverage

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* cont : fix compile warnings

* tests : add descriptions for alloc_buffer_n tests

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* ggml : address review comments on alloc_buffer_n

- restore GGML_LOG_ERROR on buffer alloc / tensor init failure in the
  default impl (name the failing tensor)
- check the malloc result and drop the _impl indirection in
  ggml_backend_alloc_ctx_tensors_from_buft
- remove comments that restate the code
- fix the TAG_ALLOC_SHARED_BUFFER_SPLIT typo

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* ggml : add get_alloc_size_n to buffer type interface

- Add ggml_backend_buft_get_alloc_size_n public API
- Add optional get_alloc_size_n callback to ggml_backend_buffer_type_i
- Share tensor->buffer planning between alloc_buffer_n default and get_alloc_size_n default
- Replace unchecked realloc with std::vector in alloc_buffer_n default
- Make ggml_backend_alloc_ctx_tensors_from_buft_size use the new API
- Add test-alloc coverage for get_alloc_size_n

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* cont : report malloc failure
2026-10-02 11:08:08 +03:00
Masashi Yoshimura 4b1622afb7 webgpu: add bfloat16 support for MUL_MAT/MUL_MAT_ID/GET_ROWS- #29358 (#29358) 2026-10-01 22:09:07 +09:00
Masashi Yoshimura 0c1e57098b webgpu: fix SSM_SCAN binding aliasing (#29750) 2026-10-01 11:11:48 +09:00
jbooth f00a64c147 webgpu: Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor (#29471)
* Fix:  Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor

* Clang formatting
2026-09-28 16:38:10 +03:00
Michael de Gansandyomaytk 0f8a414b75 metal : gate mul_mm_id src1 rescale behind ggml_prec (#29029)
* metal : gate mul_mm_id src1 rescale behind ggml_prec

Assisted-by: Claude Fable 5.1

* ggml-webgpu: reject MUL_MAT_ID when src1 precision is F32

* cuda/vulkan: reject MUL_MAT_ID in supports_op when src1 prec is F32

fix `supports_op` to return false for failing backends when the specified src1 precision is f32

Assisted-by: Claude Fable 5.1

---------

Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
2026-09-22 18:32:28 +03:00
Masashi Yoshimura 932a68e068 webgpu : add fused gdn + cpy (#28976) 2026-09-21 10:39:30 +03:00
Masashi Yoshimura 44be98f057 ggml-webgpu: fix supports_op condition for GET_ROWS (#28978)
* fix get_rows vec4 handling

* Add src strides checking to vec4_aligned of get_rows and the new test case.
2026-09-18 20:47:07 +09:00
Pascal 3f5e94d7c2 webgpu: align tensor bindings to the type block size (#28382)
Walk the binding offset back until the distance to the tensor is a
whole number of blocks, so block quantized views get a valid element
offset in the shader.
2026-09-12 06:40:21 +02:00
Mendy BergerandMasashi Yoshimura d3146f2b56 ggml-webgpu: Update to a recent version of Dawn (#28683)
* ggml-webgpu: Update to a recent version of Dawn

* No module scanning

* Accept review suggestion to update comment

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>

---------

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>
2026-09-12 10:47:29 +09:00
Pascal c0b1871bc7 webgpu: format the GET_ROWS case block (#28542)
Brace on its own line and body indented one level, matching the
surrounding cases, so the webgpu clang-format check passes.
2026-09-07 15:55:14 +02:00
Jeff BolzandAiChiTuDouPian 0cae43063c vulkan: support type-aligned GET_ROWS (#28253)
* vulkan: fall back to CPU for GET_ROWS with misaligned offsets

The Vulkan GET_ROWS shader asserts when a tensor's backing-buffer offset
plus view_offs is misaligned w.r.t. minStorageBufferOffsetAlignment
(see init_pushconst_tensor_offsets). Previously this caused a hard crash
on models using ggml_view + ggml_get_rows (e.g. Qwen3-TTS, Qwen3-VL).

Return false from supports_op() in the misaligned case so the scheduler
falls back to CPU, matching the existing pattern for PAD_REFLECT_1D and
other unsupported op/shape combinations.

Repro: llama-tts -m Qwen3-TTS-*.gguf -mm mmproj-*.gguf -ngl 99
Crash: GGML_ASSERT(dst->op != GGML_OP_GET_ROWS || (a_offset == 0 && ...)) failed

* vulkan: trim comment for GET_ROWS misalign fallback

* vulkan: fix file corruption in gated_linear_attn struct

* vulkan: properly handle misaligned offsets in GET_ROWS quantized path

- get_rows_quant.comp was missing get_aoffset()/get_boffset()/get_doffset()
  calls that are already present in get_rows.comp, causing GGML_ASSERT crashes
  when GET_ROWS operates on views with non-zero view_offs, as produced by
  KV cache slices in Qwen3-TTS and Qwen3-VL.
- Remove the defensive misalignment GGML_ASSERT in init_pushconst_tensor_offsets
  for the binary push-constants specialization, since both get_rows.comp and
  get_rows_quant.comp now correctly apply per-tensor base offsets.
- Remove the workaround CPU fallback in supports_op() for GET_ROWS, since the
  Vulkan backend now handles misaligned offsets natively (no more bailout).
- Add backend test coverage with view_src0=true (ggml_view_4d into a padded
  tensor) for F32, F16, Q4_0, Q4_K, Q8_0, and I32 types, exercising both the
  non-quantized (get_rows.comp) and quantized (get_rows_quant.comp) paths
  with non-zero view_offs that reproduce the original Qwen3-TTS crash.

* tests: trim redundant comments in test_get_rows vs0 region

* tests: trim redundant comments in test_get_rows vs0 region (follow-up)

* vulkan: bind tensor base for binary ops, pass full view_offs via push constants

For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, MUL, etc.),
bind the view_src base and pass the full view_offs divided by type_size via
push constant misalign_offsets. This avoids truncation when misalign_bytes is
not a multiple of quantized block size.

ggml_vk_tensor_subbuffer gains a use_view_offs parameter. When false, the
binding points to vk_tensor_offset (base) and size includes view_offs.
init_pushconst_tensor_offsets<binary> computes a/b/d_offset directly from
tensor->view_offs, which is always row-aligned and therefore exact.

Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).

All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).

* vulkan: bind aligned offset for binary ops, pass adjusted misalign via push constants

For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.

ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types).

ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (target - adjusted_misalign) instead of the view_src base, keeping the
offset small enough for 16-bit/8-bit push constant fields.

Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).

All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).

* vulkan: bind aligned offset for binary ops, fix UMA offset mismatch

For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.

Added ggml_vk_tensor_physical_offset to unify physical offset lookup across
UMA and non-UMA devices. On UMA, resolves via ggml_vk_host_get(tensor->data);
otherwise uses vk_tensor_offset(t) + t->view_offs. Both get_misalign_bytes and
the new ggml_vk_get_adjusted_misalign helper build on top of this function,
so buffer bindings and push constant offsets are always consistent regardless
of device memory model.

ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types) while
remaining small enough for 16-bit/8-bit push constant fields
(adjusted_misalign < lcm(align, type_size)).

ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (physical_offset - adjusted_misalign) on both UMA and discrete GPUs,
fixing a bug where the UMA host_get path previously skipped the adjusted
misalign binding and returned the target offset directly.

Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).

All 223 GET_ROWS tests pass on Vulkan (NVIDIA GeForce RTX 5060 Ti).

* finish misalignment fix

* supports_op changes for openvino/webgpu

---------

Co-authored-by: AiChiTuDouPian <15327701848@qq.com>
2026-09-07 12:22:10 +02:00
8e53fcefd2 webgpu : avoid crash when offset is not multiple of 4 in WebGPU ggml_backend_tensor_get() implementation (#28045)
* webgpu : avoid crash when offset is not multiple of 4 in WebGPU ggml_backend_tensor_get() implementation

* chore : improve code readability

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-08-31 16:04:38 +02:00
Aman Gupta 0190529ec4 ggml: add SWIGLU_CLAMP (#27930)
* ggml: add SWIGLU_CLAMP

* add vulkan shader
2026-08-30 23:00:02 +08:00
Xuan-Son Nguyen 749f688fca ggml: support ggml_rope_set_offset on opencl, sycl, wgpu, hexagon (#27345)
* ggml: support ggml_rope_set_offset on opencl, sycl, wgpu, hexagon

* rm inplace optimization
2026-08-21 00:36:57 +02:00
Xuan-Son Nguyen fe8156f789 ggml: add ggml_rope_set_offset (+ metal support) (#27120)
* add params

* cpu kernel

* metal kernel

* add test backend ops

* gate other backends

* ggml: (cuda) support ggml_rope_set_offset (#27121)

* rm cuda supports_op guard, fix webgpu clang-format

* ggml: support ggml_rope_set_offset on vulkan (#27344)

* ggml: support ggml_rope_set_offset on vulkan

* remove inplace optimization
2026-08-19 14:04:57 +02:00
Masashi Yoshimura 5112b9738b ggml-webgpu: add mulmat with overlapping src0/src1 (e.g., for minimax-01) (#27321) 2026-08-19 16:29:33 +09:00
1692f9e50b ggml : recurrent state rollback for ggml_ssm_scan (#26623)
* Initial changes for Recurrent state rollback for nemotron for cpu and cuda

* Removing CPU RS rollback. Will enable it in subsequent PRs

* addition of test case

* Removing assert and calling runtime API to check if op is supported

* removing extra API and updating the call sites for K

* replace static cuda detection to runtime fused_op api

* address review comments and fallback when SSM rollback not supprted

* Adding changes for supporting RS-rollback in CPU. Also added test-backend-ops for cpu and cuda

* removing memory manipulation as rs rollback is now supported in CPU

* removing the static probe which is not needed now

* correcting the format

* address review comments

* enabling test for all the backends, unsupported backends will fallback to CPU

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* choose different graph based on the result of fused_ssm_op is supported or not and also handled memory->n_rs_seq >1 case incase of op is not supported

* Support K > 1 in ssm_scan for all backends

* Fix CI Issues

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
2026-08-14 17:20:40 +03:00
153d324bcf llama: add default load-mode auto, which avoids mmap on iGPUs (#26081)
* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used

* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>

* set mmap_support to false on OpenCL backend

* fix order of load modes

* use -1 for auto

* resolve load mode auto earlier to correctly pick gpu host or cpu memory

* add load mode auto to llama-bench

* bump virtgpu api version, regenerate docs

---------

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-08-11 09:20:46 +03:00
Masashi Yoshimura 84f7129467 ggml-webgpu: fix CI errors from #25025 and #25262 (#26566)
* test new flash_attn test

* rebase and fix to disable subgrou matrices when max_kv_tile == 0

* delete log output

* Add i32 support to cpy and enables the all ops test

* restore the non target ci tests

* comment out of TODO of build-cpu.yml

* fix format
2026-08-11 07:10:00 +03:00
Masashi Yoshimura f401bb1390 ggml-webgpu : refactor several wgsl files and simplify flash_attn wgsl. (#26134) 2026-08-10 09:29:41 +03:00
Masashi Yoshimura 7a2db1a0cf ggml-webgpu: add support for f16 repeat (#26307) 2026-08-02 08:28:31 +02:00
Masashi Yoshimura 8f4646a63e ggml-webgpu: improve flash_attn_vec for quantized KV at long contexts (#25956)
* improve fa of quantized kv cache

* Fix some bugs and some comments.

* fix v type check and some comments

* Fix build error caused by rebasing

* editorconfig checking pass
2026-07-31 09:08:40 +03:00
Reese Levine bc71c24c9d ggml-webgpu: Fix some binding alias issues to support all archs, fix recurrent-state-rollback test (#25931)
* Add overlap glu variant to support all archs, fix recurrent-state-rollback test

* format

* Fix all arch overlapped ranges

* format

* diagnose bus error on apple ci

* More testing

* more testing

* more targeted testing

* Fix bug in alignment for > 4gb buffer offsets

* Fix bug in view offsets

* Try avoiding multi_buffers

* not fixed yet, more logging :(

* Handle edge case in set_rows

* Try looking at view source

* Skip deepseek32 for now and clean up trace infrastructure

* simplify skipping

* last cleanup

* actually final cleanup

* update handling of overlap

* format

* try skipping other failing model
2026-07-28 21:13:06 +03:00
m1el 7347430f44 webgpu : add CONV_2D_DW (depthwise conv2d) kernel (#25847)
* webgpu : add CONV_2D_DW (depthwise conv2d) kernel

Implement GGML_OP_CONV_2D_DW for the WebGPU backend,
ported from the Vulkan backend's conv2d_dw.comp.

Assisted-by: Claude Opus-4.8

* Remove unnecessary comments in webgpu support

* update supported ops tables, triggered by adding webgpu CONV_2D_DW
2026-07-22 17:24:44 +09:00
Masashi Yoshimura 6c5de1cc83 ggml-webgpu: add support for NVFP4 (#25143) 2026-06-30 17:20:04 +09:00
Jeff Bolz 72a9269172 vulkan: support all backend tests for SQR/SQRT/SIN/COS/CLAMP/LEAKY_RELU/NORM (#24582)
* vulkan: make SQR/SQRT/SIN/COS/CLAMP/LEAKY_RELU use unary.comp

* vulkan: make NORM support noncontig

* add noncontiguous row test cases for norm/l2_norm, handle this in the CPU backend and l2_norm.comp

* fix supports_op for cuda and webgpu
2026-06-23 09:48:24 -05:00
Masashi Yoshimura 7c908502ea ggml-webgpu: improve MTP inference by using mat-vec path for small batches (#24811)
* ggml-webgpu: improve small batches decoding

* Add barrier to the NUM_COLS loop in mul-mat-vec
2026-06-23 17:13:55 +09:00