Commit Graph
3667 Commits
Author SHA1 Message Date
EugeoSynthesisThirtyTwoandYour Name 162e2bb2e4 ggml : fix bf16 (#31609)
Co-authored-by: Your Name <you@example.com>
2026-11-04 10:10:22 +02:00
lhez 6900fb5ac8 opencl: improve q2_0 GEMV for Adreno (#31577)
* opencl: improve q2_0 decoding and small f32 matmuls

Assisted-by: Codex

* opencl: optimize small f32 matmuls on Adreno 8xx

Assisted-by: Codex
2026-11-04 10:07:14 +02:00
Pascal bbb02c93e9 ggml: bound views by the bytes their strides reach (#31584)
* ggml: bound views by the bytes their strides reach

A view was checked against its source with the size of a contiguous
tensor of the same shape, computed before its strides were set. Views
whose rows overlap failed that check although they stay inside their
source, and views with wider strides passed it while reaching past
the end. The view functions now pass their strides to the tensor
constructor, which checks the bytes they actually reach, like
ggml_nbytes. The conv windows of delta_net are created with all their
rows at once instead of being widened after the check.

* ggml: assert that a strided view starts from contiguous elements

The view functions set nb[0] to the type size, so the source must have
contiguous elements along dim 0 for the view to read them.
2026-11-04 09:47:11 +02:00
Georgi Gerganov 960b1d51b7 ggml : write host-buffer tensor data directly in gguf writer (#31144)
The gguf writer staged each tensor into a temporary vector before
writing it. For host-buffer tensors the data pointer is directly
writable, so skip the staging copy and write it straight to the file.
This avoids the per-tensor allocation, zero-init and memcpy.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
2026-11-04 09:29:26 +02:00
Todor BoinovskiandMax Krasnyansky bee23322f1 hexagon: add LIGHTNING_INDEXER support (#31289)
* hexagon: initial support for LIGHTNING_INDEXER op

* hex-lght-idxr: hmx for prefill

* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute

* hex-lght-idxr: pipelined hmx path for decode and prefill

* hex-lid: add support for K (f32)

* hex-lid: add support for K (f32)

* hex-lid: add support for q4_0

* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills

* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel

* hex-lit: update dma pipeline to use canonical dbl buffering in all paths

* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths

* hex-lit: add support for dims collapse

* hex-lit: make kparams constistent with other ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 21:17:30 -08:00
Todor Boinovski 7071ba378a hexagon: add TOP_K_MASK op (#31552)
* hexagon: add TOP_K_MASK op

* Simplify Hexagon TOP_K_MASK kernel and do not duplicate host configuration
2026-11-03 12:04:03 -08:00
Zack LiandMax Krasnyansky 6b9c0cfcba hexagon: faster TOP_K for the backend sampler (#31279)
* hexagon: faster TOP_K for the backend sampler (skip tiles, then block scan)

Skip the bitonic sort for tiles whose max cannot reach the running k-th
best value, then replace the tile sort with top_k_scan(): reject 256-float
blocks with one HVX max-reduce and insert only surviving elements into a
sorted k-entry list. Used for k in 2..64 and rows wider than 1024, in both
the single-row and multi-row paths.

Ties keep the first-seen index. NaN is treated as -INF.

* hex-top-k: eliminate scalar reads from vtcm

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 10:06:28 -08:00
Todor BoinovskiandMax Krasnyansky 20a51f24a8 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
* hexagon: add support for DSV4_HC_PRE

* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization

* hex-dsv4-hc-pre: double-buffered DMA for prefill

* hex-dsv4-hc-pre: refactor DMA block setup

* hex-dsv4-hc-pre: precompute kernel params

* hex-dsv4-hc-pre: support for n_hc != 4

* hexagon: add support for DSV4_HC_POST

* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block

* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks

* hex-run: add support for test-llama-archs tool

* hex-dsv4: refactoring and optimizations

Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.

Tested with test-llama-archs -a qwen4ext and test-backend-ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 08:26:18 -08:00
Aaron Teo a464960ed3 ggml-zdnn: fix fattn test failure (#31583) 2026-11-03 23:51:07 +08:00
Jasmine-tim 0978a6e902 ggml: avoid signed overflow in ggml_roll bounds check (#30375)
ggml_roll validated each shift with abs(shift), but abs(INT_MIN) is signed
integer overflow UB evaluated before the bounds check runs. Compare with a
signed-safe test instead.
2026-11-03 13:26:29 +02:00
mmalpe-nvandGaurav Garg 8fc0376721 cuda: potential fix for Q8_1 correction sum (#31247)
* cuda: potential fix for Q8_1 correction sum

* Update ggml/src/ggml-cuda/quantize.cu

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>

* addressed review comment to change the quantize_mmq_q8_1 flow as well

---------

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
2026-11-03 15:54:42 +05:30
Bhargav Krish 740251a920 cuda : gather batched tensor reads into mapped host memory (PR to master) (#31391)
* cuda: gather batched tensor reads into mapped host memory

Assisted-by: Codex

* add comments

Assisted-by: Codex

* simplify the implementation based on checks that exist earlier in the flow

* simplify batched tensor gather indexing

* add conditional dispatch

* address comments - make host registration raii, better test mindful of ci time, assert instead of return false

* cuda : gather small reads and use pinned DMA for large reads

* cuda : use std::vector for host gather descriptors

* cuda : add 8-byte gather copies, generalize tensor test and log registration failures

* common buffer for descriptors

* simplify stream sync + increase to 128k descriptor pinned buf
2026-11-03 15:31:50 +05:30
zyang-dev 71efc559a0 Add nccl support for multi-node tensor parallelism (#28967)
* Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available.

* Drop ggml_backend_comm_allreduce_rank_t and reuse ggml_backend_comm_allreduce_tensor for per-rank communicators

* reuse ggml_backend_cuda_comm_allreduce_nccl for per-rank communicators. Remove the helper function backend_comm_proc_address

* Change ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t to more generic multi-process backend communicator interfaces

* set maxCTAs in ncclConfig_t to 8 for dgx spark in ggml_backend_cuda_comm_init_rank
2026-11-03 14:56:19 +05:30
Aman GuptaandPratik Narola 259d91709d meta: support GLM-5.3 next sm tensor (#30499)
* meta: support GLM-5.3 next sm tensor

* glm5-next : cont q_absorbed for -sm tensor with non-unified KV

* meta : keep GGML_OP_ARANGE in ggml.h order

* glm5-next : split MLA heads in pairs for -sm tensor

The MLA cache has a single KV head, so the heads on a device are its GQA
ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3
devices the 16 heads of the test model were split 5/5/6 and it aborted.

* meta : keep the split for views inside one slab of the split axis

With a non-unified KV cache, build_attn_mha splits q into streams with a
view. For a permuted q, e.g. the absorbed query of glm5-next, that view
aborted with "view of permuted tensor not implemented". The split is now
kept if every other dim of the source and the view stays inside one slab
of the split axis. The per-device strides are only rescaled above it.

From tetherto/qvac-fabric-llm.cpp#353.

Co-authored-by: Pratik Narola <pratik.narola@tether.io>

* glm5-next : drop the q_absorbed cont

The meta backend can now split the per-stream view of the permuted q, so
the cont is not needed for -sm tensor with a non-unified KV cache. With
the cont, the q of flash attention was not contiguous, and when FA ran on
the CPU (WebGPU) the meta get_tensor aborted.

---------

Co-authored-by: Pratik Narola <pratik.narola@tether.io>
2026-11-03 10:51:10 +02:00
Georgi Gerganov 36ee3caf7a metal : fix cpy kernel for dst non-contiguous in dim 0 (#31581)
kernel_cpy_t_t indexed the dst innermost dim with an implicit element-size
stride instead of args.nb0, so cpy into a permuted view (nb0 != type size)
wrote the wrong addresses. caught by the new test-backend-ops cpy case with
permute_dst = {1, 0, 2, 3} from #31446. addressing is unchanged when dst is
contiguous in dim 0.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-03 10:41:24 +02:00
FamousMonsterrandFamousMonsterr fac3a894fc vulkan: skip maxStorageBufferRange pre-check for non-compute tensors (#31114)
* vulkan: skip maxStorageBufferRange pre-check for non-compute tensors

Pre-allocated leaf tensors and views (GGML_OP_NONE / GGML_OP_VIEW) are never
executed by the backend, but were still rejected by the descriptor-size
pre-check in ggml_backend_vk_device_supports_op when larger than
maxStorageBufferRange on devices without shader_64b_indexing. The scheduler
then aborts with 'pre-allocated tensor (...) in a buffer (Vulkan0) that cannot
run the operation (NONE)' - e.g. the MoE expert cache banks (llama-moe-cache)
of a 256-expert model with --moe-cache-mib on GCN GPUs, where a whole bank
(2.4 GB) exceeds the limit while the consuming ops only access small
per-step views.

Skip the size pre-check for non-compute ops (NONE and VIEW); the buffer type
allocation limit still bounds the actual allocation, and compute ops keep
validating their sources.

Tested on AMD Radeon Pro VII (gfx906, no shader_64b_indexing):
- before: GGML_ABORT 'pre-allocated tensor (moe_cache.0.blk.0.ffn_down_exps.weight) ... cannot run the operation (NONE)' with llama-cli -ngl 99 --cpu-moe --moe-cache-mib 6144 (Cyber-Tiel-Coder-35B-A3B, qwen35moe, 256 experts per layer)
- after: loads and generates at 8.1 t/s (vs 0.6 t/s without the cache)
- regression: -ncmoe 16 path unchanged (19.2 t/s), qwen3moe/qwen35moe without cache unchanged

Fixes #30697

* vulkan: use ggml_op_is_empty for the non-compute size-check skip

---------

Co-authored-by: FamousMonsterr <FamousMonsterr@users.noreply.github.com>
2026-11-03 08:53:04 +01:00
Aman GuptaandPascal b74298c109 delta_net_base: refactor conv state (#30878)
* delta_net: access conv state via views

* fix sm tensor

* delta_net: fill the conv slot index on the host

The rollback slot index of the conv windows is a constant per graph
shape. It is now an I32 input of the recurrent memory filled on the
host, instead of an arange and an argsort computed on the device, and
the meta backend no longer needs a mirrored split for ARANGE.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-03 08:28:35 +01:00
Dongsheng LuoandDongsheng Luo 37b0f9c36a vulkan: fix out-of-bounds NVFP4 UE4M3 scale lookup (#31079)
* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup

On the non-OCP path, ue4m3_fp32_lut has 128 entries and is indexed with the raw scale byte. ggml_validate_row_data accepts every byte and ggml_ue4m3_to_fp32 ignores bit 7, so a byte >= 0x80 reads past the shared array. Vulkan robustness does not cover shared memory.

Size the table, both init loops, and the two host shared-memory budgets to 256 entries. ue4m3_to_fp32_build() already masks the exponent and treats only 0 and 0x7F as special, so entries 128-255 match the CPU. A 256-entry table keeps 0xFF distinct from 0x7F: the CPU decodes 0xFF as a non-zero value and 0x7F as zero.

Each NVFP4 pipeline uses 512 more bytes of shared memory. ggml_fp32_to_ue4m3 never sets bit 7, so models quantized by llama.cpp are unaffected.

Assisted-by: Claude
Assisted-by: Codex

* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup

Assisted-by: Codex

---------

Co-authored-by: Dongsheng Luo <7137760+flyingdoog@users.noreply.github.com>
2026-11-03 07:58:56 +01:00
hesiru 00f113eceb cuda : prevent premature eviction of graphs about to be used (#30662)
* cuda : prevent premature eviction of graphs about to be used

* add last_graph_use_time for entire cuda graph map and testcase for cuda graph

Assisted-by: Codex

* delete the test
2026-11-03 08:06:03 +02:00
shaofeiqiandLi He e6192de0a6 opencl: add initial iq4_xs support (#31339)
* opencl: add iq4_xs AOS mv

Assisted-by: Codex

* opencl: add iq4_xs SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq4_xs noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq4_xs dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: optimize iq4_xs adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

Assisted-by: Codex

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-02 21:58:19 -08:00
Leon 713319e284 webgpu: add ARANGE support (#31455) 2026-11-03 07:54:00 +02:00
51fe42c5d1 hexagon: tiled hvx permute (#31446)
* hexagon: add tiled HVX transpose path for permuted CONT/CPY

Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.

Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.

Host:
  - drop extent-1 dims and merge dims that stay adjacent in src into a
    canonical view, stored in the kernel params. There are no minimum
    size limits.
  - the copy is treated as a 2D transpose between dst dim 0 and the
    contiguous src dim, batched over the remaining dims.
  - tiles are sized to 64 KB per buffer and halved until they fit in
    session VTCM. blkk is capped so the halfword vgather offsets fit in
    16 bits.

Kernel, per tile:
  - 2D DMA of blk0 strided src rows (blkk contiguous elements each)
    into VTCM
  - vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
  - 2D DMA of blkk contiguous rows out to dst

Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.

Small dims:
  - packed mode: when dim k follows dim 0 and dim 0 divides the HVX
    lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
    whole dst rows, and each tile goes out as one contiguous DMA.
  - when the contiguous src dim fits in one vector, it is staged exactly
    and the tile is made long along dim 0.

All other copies select the same kernels as before.

Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.

* hexagon: cpy transpose cleanups

- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)

Assisted-by: Claude Opus 5.5

* hexagon: use cpy transpose path for ne[0] == 1 permutes

- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests

Assisted-by: Claude Opus 5.5

* hex-cpy: further improve cpy

Bring in improvements from aizip branch/PR.

Cleanup kparams and replace divs with fastdiv.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-cpy: avoid vtcm bank conflicts

* hex-cpy: trim down the comments

* cpy-test: trim down new tests that cover permute scenarios

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
2026-11-02 21:13:38 -08:00
Aman GuptaandGeorgi Gerganov 1b6935b227 llama : flash_attn_ext_rows - more efficient unified kv for multi-seq (#29510)
* llama : use llm_attn_mask and kv_rows for unified attention

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* cont : optimize `get_seq_cells`

* llama : build the kv_rows lists from per-sequence cell bitmaps

Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.

Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).

* cont : make get_kv_rows_p_min private

* cont : minor metal

* llama : use kv_rows for every ubatch of a unified cache

A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.

Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.

* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-02 18:13:33 +02:00
Aaron Teo 6ed812057c ggml-zdnn: share buffers with host (#31481)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-02 14:11:07 +02:00
Yuri Khrustalev ea24b952cd ggml : use vm_allocate instead of posix_memalign on all Apple platforms (#31437)
* ggml : use `vm_allocate` instead of `posix_memalign` on iOS

* ggml : use vm_allocate on all Apple platforms, run test-host-buffer-release by default
2026-11-02 14:08:21 +02:00
lhez 84d9ce0fad opencl: add topk_mask (#31441) 2026-11-02 13:59:05 +02:00
dcbb091677 vulkan: add tensor-parallel allreduce function (#25051)
* vulkan: add tensor-parallel all-reduce communication

* vulkan: require shared semaphores for tensor communication

Assisted-by: Codex

* vulkan: reduce directly from shared host buffers

Assisted-by: Codex

* vulkan: fix ring all-reduce races with 3+ devices

Two hazards in the tensor-parallel ring all-reduce made -sm tensor output
non-deterministic with three GPUs:

- With n = 3, step t's host-slot region starts at t * cels * 2 bytes, which
  is not a multiple of 32, so consecutive steps' transfer copies share a
  32-byte sector of the imported host memory. On NVIDIA the later copy's
  leading bytes (up to 10 F16 values at the start of every chunk) end up
  stale in host memory, even when the copies are ordered by a semaphore.
  Give every step its own 256-byte-aligned region. n = 2 and n = 4 already
  had aligned regions for the usual sizes, so their layout is unchanged.

- A single-shot all-reduce leaves every device's host slot being read by all
  peers, but the next ring round only ordered its first transfer after nextd,
  and only from round 2 on. With n >= 3 a device could overwrite its slot
  while prevd was still summing it. Wait for all peers on the first ring
  round after a single-shot all-reduce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vulkan: fold the tensor-parallel all-reduce into the compute submissions

With -sm tensor the meta backend drives every device from one thread and calls
graph_compute once per subgraph (two per layer), so decode was bound by the CPU
cost of submitting, not by the GPUs: ~1500 vkQueueSubmit per token for a 64-layer
model on two GPUs, with the GPUs idle ~40% of the time.

- Don't split subgraphs by flops while a communicator is active. The heuristic
  was tuned for whole graphs and turned each small subgraph into ~5 submits.
- Fold the small (single-shot) all-reduce into the compute submissions: the
  last command buffer of a subgraph stays open, the all-reduce appends the
  gather copy to it and submits it, then opens the next command buffer with the
  peer waits and the reduce, which the next subgraph is recorded into. Each
  subgraph now costs one submit per device. The reserved progress value is
  attached by whichever path submits the open context.
- Alternate between two host slots, placed outside the ring's regions, so the
  gather no longer needs a WAR wait on the peers' previous reduce.
- On synchronize, submit the deferred work of every device in the group first,
  so the tail of the graph runs on all devices in parallel.
- Wait for the transfer queue as well as the compute queue before freeing or
  reallocating communication buffers, and don't leave a stale cap behind when a
  reallocation fails.

qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 32.2 -> 49.7 t/s, 3 GPUs 20.9 ->
40.0, 4 GPUs 15.0 -> 27.6; pp512 on 2 GPUs 2138 -> 2320. Greedy output is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* - Two-phase all-reduce over shared host memory: write own slot + signal, wait peers + add from their slots
- One exported timeline per device, no backend hooks; comm work recorded in the backend compute ctx
- F16 wire above 1 MiB, F32 below; small reductions ride with the next graph submit
- Double-buffered, grow-only host slots; zero-size splits zeroed
- Remove ring, transfer-queue leg, device staging, shrink, Windows shims
- comm_init returns nullptr when unsupported; add GGML_VK_COMM_DISABLE
- No flops-based submit splitting for graphs below 1 GFLOP

[rebased onto the ring all-reduce fix and the folded all-reduce, which this design replaces; their backend hooks are removed here]

* add minimal submit split size

* replace opaque_fd with sync_fd for spec compliance

* add precision check

* add semaphore pool

* rename env var, add logging if requirements not met

* reduce comments

* code improvements

* vulkan: use posix_memalign for the tensor-parallel host buffers

std::aligned_alloc is unavailable on Android bionic below API 28, which
upstream switched away from in #30828; do the same for the communicator's
host slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vulkan: synchronize the tensor-parallel all-reduce through the host

Exchanging SYNC_FD semaphores costs a kernel wake-up on every all-reduce:
on NVIDIA the waiting GPU starts ~35 us after its peer has finished, which
is most of the idle time in a decode step. OPAQUE_FD semaphores avoid that
but must not be imported into another physical device.

Use only device-local timeline semaphores instead. Each device signals
done[i] once its partial sum is in host memory (after a barrier that makes
it available to the host), and a bridge thread signals go[i] from the host
once all peers of i are done. The waiting side makes the host memory
visible with a host-write barrier, since its submit predates the data.
The thread spins for up to 200 us on each wait (GGML_VK_ALLREDUCE_SPIN_US)
and then blocks, so it is idle when there is no work. This also takes the
per-round fd export, dup and import calls off the main thread and drops the
VK_KHR_external_semaphore_fd requirement and the binary semaphore pool.

Graphs below 1 GFLOP, such as tensor-parallel decode slices, are no longer
split into several submits.

qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 37.2 -> 45.4 t/s, 3 GPUs
24.1 -> 35.5, 4 GPUs 16.5 -> 27.1; pp512 unchanged. Greedy output is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* use sync_fd on mesa drivers

* refactor for readability

* move sync_fd semaphore handling to thread, trigger mesa submit threads

* move allreduce functions to new cpp file

* clean comments

* smaller fixes, code improvements

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-02 13:57:56 +02:00
ravel7524 07ec7c8af0 vulkan: improve small-n split-K selection on RDNA3 and RDNA4 (#30146)
* vulkan: improve small-n split-K selection on RDNA3 and RDNA4

* Clean up test cases by removing commented code

Removed tests as requested
2026-11-02 13:54:09 +02:00
Amadeus dyw cb02250b5b vulkan: avoid redundant scans when zeroing skipped rows (#31266)
* vulkan: avoid redundant scans when zeroing skipped rows

* vulkan: remove unreachable zero-expert fallback
2026-11-02 13:53:27 +02:00
92es 2c93f0b6e1 sycl : fast ggml_sycl_comm_exchange() on Intel Arc Pro B70s (#30956)
* Update ggml-sycl.cpp

Increases -sm tensor prompt eval t/s by +19% with zero PPL change.

* Update ggml-sycl.cpp

Replace first pair of wait() calls with ext_oneapi_submit_barrier().  The wait() calls was discovered to cause MTP degradation.

* Update ggml-sycl.cpp

Adding comm_exchange function pointer to switch between 2 and 4 buffer ggml_sycl_comm_exchange versions @ initialization time (versus hot path) controlled by SYCL_ENABLE_DEFAULT_CONTEXTS=1 setting.

* Update ggml-sycl.cpp

adding back 2nd half of 4-buffer free.

* Update ggml-sycl.cpp

Removing extra whitespace.

* Update ggml-sycl.cpp

Excluding Windows until it is validated on Windows.
2026-11-02 13:52:53 +02:00
Titaniumtown 5e18eb021b sycl: dmmv code cleanup (#31325)
* sycl: fold q8_0 reorder dmmv into the generic template

* sycl: template the dmmv non-K launchers

* sycl: template the dmmv K-quant launchers
2026-11-02 13:51:20 +02:00
Uber a7a490a490 ggml : fix clang-cl build with AVX512 BF16 (#31276)
* Correctly gate immintrin.h inclusion when building with clang-cl and avx512bf16

* ggml : explicit vector casts for clang-cl
2026-11-02 18:39:53 +08:00
Pascal 1c5ed702c9 webgpu : fix CI by keeping scratch alloc sizes monotonic in batch length (#31422)
* webgpu: keep scratch alloc sizes monotonic in the batch length

The flash attention vec path and the MMVQ path append a scratch area
after dst, but get_alloc_size only accounted for it when the current
batch takes that path. The size was therefore not monotonic in the
batch length: a graph reserved at n_ubatch (tiled path, no scratch)
or at one token did not cover batches of 2 to 19 query rows, and
GGML_SCHED_NO_REALLOC builds aborted on an unexpected reallocation.

Pass the row count to both path predicates and size the scratch with
the row count clamped to the longest batch the short path serves.

* nit: align the mmvq pipeline key with clang-format 22

* webgpu: address review @yomaytk, name the mat-vec row threshold
2026-11-02 09:36:04 +02:00
Michael Taylor dde879bc4f sycl: Try PkgConfig for LevelZero includes / libs (#31076) 2026-11-01 01:46:43 -05:00
shaofeiqiandLi He 060b2fea4e opencl: add initial iq3 support (#31338)
* opencl: add iq3_s AOS mv

Assisted-by: Codex

* opencl: add iq3_s SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq3_s noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq3_s dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: optimize iq3_s adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq3_xxs AOS mv

Assisted-by: Codex

* opencl: add iq3_xxs SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq3_xxs noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq3_xxs dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: optimize iq3_xxs adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-31 22:26:47 -07:00
Mark TodorovichandMax Krasnyansky a91eab67d8 hexagon : fix DMA transfers with strides > 24 bits (#31442)
* hexagon : fix DMA transfers with strides > 24 bits

On v75+ the 2D DMA descriptor stores row_size and strides in 24 bits.
dma_queue_push sent oversized strides to fallback_2d, which only splits
nrows, so strides of 16 MB or more were truncated - wrong addresses.
Fix will fall back to one transfer per row in that case.

Also added two CONT tests with source strides over 16 MB.

* hex-dma: refactor the fallback paths to remove duplication

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-31 16:49:23 -07:00
shaofeiqiandLi He 0ec4027a2a opencl: add fwht (#31434)
Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-30 22:13:48 -07:00
Jesse Symons 8956c769e8 sycl: fix f32 to q2_0/q4_0/q5_0 rounding in CPY (#31120) 2026-10-30 21:54:41 -04:00
shaofeiqiandLi He 792fcf4e75 opencl: add initial iq1 support (#31341)
* opencl: add iq1_s AOS mv

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq1_s SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq1_s noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq1_s dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: optimize iq1_s adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq1_m AOS mv

Assisted-by: Codex

* opencl: add iq1_m SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq1_m noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq1_m dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: optimize iq1_m adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-30 10:43:30 -07:00
shaofeiqiandLi He e0f142d5dd opencl: add initial q2_0 support (#31340)
* opencl: add q2_0 AOS mv

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add q2_0 SOA mv and tiled mm

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add q2_0 noshuffle gemm/gemv for adreno

Assisted-by: Codex

* opencl: add q2_0 dp4a for adreno

* opencl: optimize q2_0 adreno decode and narrow batches

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-10-29 21:30:05 -07:00
Michael Taylor 36a0e2ffc0 sycl: FLASH_ATTN_EXT failures (#30761) 2026-10-29 17:37:28 -04:00
Georgi Gerganov 418f905a4e ggml : add batched tensor copy API (#30881)
* ggml : add generic batched tensor copy API

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : use batched tensor copy API in context io

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : add batched tensor copy API

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : batch device io tensor copies

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : add CUDA support for batched tensor APIs

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : remove async tensor_n APIs and add null buffer hooks

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : fix batched tensor 2D detection

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* ggml : group batched tensor operations by buffer type

Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : minor indentantion

* cont : const correctness

* tests : add mock buffer type tests for batched tensor API

Restructure test-backend-tensor to run a unified test set on each backend buffer type and on an ad-hoc mock buffer type. The mock set exercises the dispatch logic in ggml_backend_tensor_{set,get,copy}_n: hook invocation, per-tensor fallback, cpy_tensor_n all-or-nothing, and buft grouping.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* cont : add TODO
2026-10-29 20:52:17 +02:00
Slobodan JosicandMario Limonciello eb4e244be7 ROCm/HIP: add RDNA4_M family for gfx1170/gfx1171 (#31373)
gfx117X is more like RDNA4, with 128b WMMA. But they're not identical. Adjust the defines accordingly to have a new RDNA4_M and use it where it makes sense.


Assisted-by: Codex
Assisted-by: GitHub Copilot

Co-authored-by: Mario Limonciello <mario.limonciello@amd.com>
2026-10-29 18:23:29 +01:00
Oliver SimonsandSigbjørn Skjæret 68d20b884c Quant: OCP FP8 E4M3 support (#27512)
* WIP OCP FP8 E4M3 support

* Fix python type-checker

* Materialize input_scales also for FP8 when converting

* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4

1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
   dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
   weights are reshaped

* Allow remaining FP8 tensors to be dequantized during conversion

* Allow tensor-strategy for FP8-only compressed-tensor ckpts

* Fix and test gguf-py behavior for FP8_E4M3

* Disallow QKV-fusion for FP8 temporarily

* Fix conversion of fused qkvz & FP8 for Qwen3-Next

* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws

* Emit warnings when not fusing FP8 during conversion

* More FP8 conversion fixes

* Add supported for strided copies to/from FP8 in CPU

* remove unneeded scratch buffer from FP8 cpu cpy/dup path

* add explanatory comments and TODOs

* Add dedicated encoder and decoder tests for FP8

* Whitelist ggml_op_get_rows in metal instead of blacklisting

required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list

https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512

* Fix more conversion bugs

* Defer weight materialization during conversion

* Blacklist fp8 in metal backend instead of white-list

* Adjust indents

* Fix lint + execute tests

* Add fast CPU kernel for F8_E4M3

* Conversion fixes

* Fix remaining gaps

* Enable tied weights with shared scales

* Make mandatory-F32 rules apply to FP8 during conversion

* Fix python type checks

https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512

* Fix kaleidai unit-test failures

https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150

* Relax numeric precision constraints for pre-check

* Leave fast-FP8 CPU on RISC-V for follow-up

We already exclude arm from the fast-path

* Allow output scale tensors to be located in any gguf shard

* Move test_conversion_fp8.py test execution to behavior-check.yaml

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it

* Install all requirements also on Windows for behavior-check

Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512

* Unmap lazy tensors before clean-up in Windows

Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512

* LF line endings

https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512

* Extract FP8 tests from llama-arch to its own test-fixture

See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221

* Update gguf-py/gguf/scripts/gguf_convert_endian.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-29 18:14:37 +01:00
uvos 78febbad9d HIP/CUDA: enable sparse attn on CDNA (#31259)
* HIP/CUDA: enable sparse attn on CDNA

* Use __ballot for __ballot_sync on old ROCm versions

* Ignore new large head fattn mma variant
2026-10-29 10:14:11 +02:00
lhezandHongqiang Wang 22183e51f9 opencl: initial support for iq2 (#31267)
* opencl: add iq2_s AOS mv

* opencl: add iq2_s SOA mv and tiled mm

* opencl: add iq2_s noshuffle gemm/gemv for adreno

* opencl: add iq2_s dp4a for adreno

* opencl: optimize iq2_s adreno decode and narrow batches

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add iq2_xs AOS mv

* opencl: add iq2_xs SOA mv and tiled mm

* opencl: add iq2_xs noshuffle gemm/gemv for adreno

* opencl: add iq2_xs dp4a for adreno

* opencl: optimize iq2_xs adreno decode and narrow batches

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: add iq2_xxs AOS mv

* opencl: add iq2_xxs SOA mv and tiled mm

* opencl: add iq2_xxs noshuffle gemm/gemv for adreno

* opencl: add iq2_xxs dp4a for adreno

* opencl: optimize iq2_xxs adreno decode and narrow batches

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-29 10:12:11 +02:00
Aman Gupta 6d194531e1 metal : faster MXFP4 mat-vec, full threadgroups for unary/binary/GLU (#31075)
* metal : faster MXFP4 mat-vec kernel

Decode the fp4 values with the fp16 bit trick instead of a threadgroup lookup table, and load the packed nibbles as aligned 32-bit words. The kernel no longer uses threadgroup memory.

* metal : launch contiguous unary and binary ops with full threadgroups

The contiguous unary path and the row-broadcast binary path used one thread per threadgroup. Use up to 256 threads per threadgroup and bound-check the tail.

* metal : split GLU rows across threadgroups

A GLU row ran on one threadgroup, so single-row decode used a single threadgroup for the whole op. Split each row over threadgroups of up to 256 threads.
2026-10-29 10:11:29 +02:00
Georgi Gerganov c9daee149d ggml-metal : fix unary kernels compile on older Metal toolchains (#31360)
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-29 10:09:51 +02:00
Max Krasnyansky 8a199ad31a hexagon: support for UPSCALE op (#31331)
* hex-upscale: basic upscale ops (not fully vectorized)

* hex-upscale: optimize nearest 2x, 4x and bilinear 4x

* hex-upscale: add support for non-contig rows and dims collapsing

* hex-upscale: use vgather instead of scalar staging

* hex-upscale: move all kernel select logic to the host, review comments and cleanups

* tests: add permuted UPSCALE test to cover qwen3-vl image->text scenarios
2026-10-29 00:22:34 -07:00
Trivikram ReddyandMax Krasnyansky da6c30ade8 hexagon: improve DDR bw utilization in transposed CONCAT (#31277)
* hex-concat: double row block size for transposed CONCAT

* hex-concat: double buffer scratchpads

* hex-concat: chunked src1 rows in transposed concat

* hex-concat: pad src1 VTCM stride to spread gather lanes

* hex-vgather: apply similar vgather opts to SSM_CONV and GDN

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-28 23:57:18 -07:00