Pinned async host uploads can record into compute_ctx before graph
compute, so the perf logger's opening timestamp must append after them
instead of asserting the context is expired. Without this, profiling
(GGML_VK_PERF_LOGGER) crashes whenever host-register pinning is on.
Assisted-by: Claude
VK_EXT_external_memory_host import fails on RADV for file-backed mmap pages,
so register_host_buffer no-op'd and every MoE-expert upload paid a slow
single-threaded pageable memcpy into the staging buffer each eval (~3 GB/s
effective on the 35B, ~25% of PCIe bandwidth).
When the import fails, fall back to a one-time copy of the region into a
host-visible Vulkan buffer registered in device->pinned_memory under the
mmap's address range. The existing pinned fast path then DMAs straight from
that buffer each eval at full PCIe bandwidth - no per-eval CPU memcpy and no
blocking sync. Costs one-time host-visible memory equal to the CPU-resident
weight region.
Assisted-by: opencode
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).
Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).
Assisted-by: opencode
Export register_host_buffer/unregister via the backend reg so the existing
GGML_CUDA_REGISTER_HOST path in llama-model-loader pins mmap'd expert
weights on Vulkan. Imports host pages through VK_EXT_external_memory_host
into device->pinned_memory, letting the existing pinned fast path in
ggml_vk_buffer_write_2d_async DMA straight from system RAM instead of
bouncing through the staging buffer + blocking host memcpy.
A single Vulkan buffer cannot cover a multi-GB mmap (capped at
device->max_buffer_size), so the region is imported in page-aligned
chunks; a bound check makes tensors straddling a chunk boundary fall back
to staging instead of reading out of bounds. Opt-in via
GGML_CUDA_REGISTER_HOST=1 or GGML_VK_REGISTER_HOST=1, no-op otherwise.
Assisted-by: opencode
Profiling showed the 2-slot rotation stalls ~5.5ms per layer waiting for
the down_exps upload: 3 tensors per MoE layer need 3 slots for a full
layer of lookahead. Default is now 3 (one layer), configurable via the
env var value, degrading to however many slots fit on OOM.
Also fixes a use-after-free: on a failed slot regrow the repointed
staging tensors kept dangling pointers into freed device memory, which
graph reuse carried into later evals. Staging repoints are now restored
right after kernel launch, slots are sized once from the graph max, and
allocation happens before freeing.
Qwen3.6-35B-A3B pp2048 on RTX 3060: 1643 -> 1880 t/s (mainline: 1143).
At large batch sizes virtually every expert is used, so the per-layer
routing-ids readback that mainline waits on buys nothing while forcing a
full device sync per expert tensor (3x per MoE layer). Above a batch
threshold, upload the full expert tensors through a second backend
instance on the same device (own stream) with two event-ordered staging
slots, so uploads for tensor N+1 overlap compute of tensor N.
Qwen3.6-35B-A3B pp2048 on RTX 3060 (-ncmoe 26, ub 2048): 1383 -> 1663 t/s.
Generation output verified token-identical; decode path unaffected.
Opt-in via GGML_SCHED_PREFETCH_EXPERTS=1.
k_get_rows_float did a scalar one-element-per-thread copy and recomputed the
row-invariant work (index load, fast_div_modulo, src/dst row pointers) for
every element. Hoist that out of the per-element loop, and add a vectorized
path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the
contiguous same-type (no-cast) case.
The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at
runtime on 16-byte alignment of the base pointers and all row strides and on
ne00 % VEC == 0. Vectorizing divides the block count by VEC, so a small
single-row gather can drop below the device CU count and regress; an
occupancy gate keeps those on the block-rich scalar path.
On Strix Halo (gfx1151) the DeltaNet recurrent-state gather (ne00=524288)
drops 18.6us -> 13.0us (rocprofv3 HW timestamps), faster than the Vulkan
backend, with no regression on the small conv-state gather; total get_rows
-27%. test-backend-ops GET_ROWS passes (47/47).
Assisted-by: Claude Opus 4.8
* vulkan: Support Q2_0
The backend perf tests for mat-vec-mul weren't very good at first (worse than
q2_k), doubling the rows per workgroup made a big difference.
* reorder
* resolve merge conflict, adjust err threshold for f16->q2_0 set_rows
* opencl: load quant as uint in mv_q4_k_f32_flat
helps older compilers (e.g., E031.41, boosts 2x),
no impact on newer compilers (e.g., E031.45 or newer)
* opencl: load quant as uint in mv_q5_K_f32_flat
helps older compilers (e.g., E031.41)
* opencl: format
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hex-mm: fix artificial limit in the solver that restricted number of act-prep threads
* hex-mm: fix warning
* hex-prof: do not apply --top to the timeline report
* hmx-mm: add suport for tiled act-processing to better distribute hvx work
* hex-l2: add tracing for l2flush events
* workqueue: redo the legacy workpool api to match hmx-queue and dma-queue
* hmx-mm: fix f32 activation buffer alignmnet for nhvx=5,6,7
* hex-work: minor cleanup for work-queue apis
* hex-work: further cleanup of the work-queue api
* hex-l2: optimize l2flushes at the opbatch level
* hex-work: remove unused mask
* hex-work: no need to drop hvx ctx in the work-queue
* hex-work: add explicit wakeup/suspend and make threads spin
* hex-bufs: mark any non-weight tensor as compute
* hex-dma: dma-queue support for alias queues and cached dma
* hex-l2: track tensor aliases and delay or skip flushes as much as possible
* hex-l2: simplify tensor alias handling
* hex-l2: handle overlapping views as a circular list of aliases
* hex-tens: add flags helper
* hex-l2: add helper for marking tensors clearn/dirty
* hex-l2: mark binary and rope outputs as l2-clean and keep the rest as is for now
* hex-l2: proper support for handling all tensor overlap scenarios
* hex-trace: instrument matmul init code and cleanup trace checks
* hex-thread: introduce dedicated main thread with explicit stack and priority
* hex-l2: track dirty state as bitmap and introduce threaded flush
* hex-trace: remove redundant checks for ctx != null
* hex-l2: allocate entire context as one buffer and l2fetch it after big flushes
* hex-l2: disable tensor clearing in binary and rope for now seems to cause issues with fusion
* hmx-mm: update act proc to use fastdivs and fix DMA overflow
* hmx-mm: make MUL_MAT_ID kernels robust to multi-chunk cases (start_row>0)
* hex-queue: remove obsolete queue interfaces and flush hmx-queue at the end of the op-batch
* hex-queue: dont use early wakeup for small op-batches
* hex-tensors: properly cap max_tensors in op-batches and dirty_map
* hex-l2: make sure threaded l2flush does proper rounding
* hex-l2: factor out htp_tensor_flush for reuse (if needed)
* hex-l2: optimize tensor flushes by coalescing flush-all
* hex-l2: optimize multi-threaded flush
* hex-drv: futureproof version checks
* hexagon: fix errors and warnings on windows
* hex-main: update main thread to only use dspqueue_read, dspqueue_peek is not available on some platforms
* hex-main: add fallback mode for dspqueue with callbacks
* hex-main: introduce fallback mode for using dspqueue callbacks for full op processing
* hex-main: remove early wakeup, not helping and seems to cause some errors with certain batch sizes
* hex-l2: make sure to use invalidate version of flushall
* hex-l2: dont try to trace early l2flush at the start of op-batch
* hex-main: remove offset_ctx that must be zero anyway
* hex-hmx: fix hmx_queue_depth to use idx_write - idx_read
* hex-hmx: use atomic_load for idx_read/write
* hex-main: add static assert to make sure n_threads are aligned
The current integration treats SME as a single capability (CPU_FEATURE_SME)
with no distinction between SME(v1) and SME2. The kernels dispatched under
CPU_FEATURE_SME use SME2-specific instructions, making dispatch incorrect
on SME(v1)-only hardware.
We introduce build-time and runtime distinction between SME and SME2, and
wire SME(v1) and SME2 kernels based on actual hardware support.
* support cuda virtual devices
* disable NCCL path when virtual devices are used
* label virtual devices in description; add GPUx2 server CI jobs
* code refactor
PR #16308 set info.devices[id].integrated = false unconditionally for all
CUDA/HIP devices as a workaround for corrupted output on Jetson Orin
(#15034). On HIP/ROCm the device's real hipDeviceProp_t.integrated flag is
needed: with the cached field forced to false, supports_buft() refuses
CUDA host buffers on AMD APU/UMA parts, while get_type() already reads
prop.integrated (#23007) — an inconsistency that breaks integrated-GPU
host-buffer use on ROCm.
Guard the workaround so it only applies to non-HIP (CUDA) builds and
restore prop.integrated for HIP, keeping the Jetson workaround intact for
CUDA.
Fixes#23977
Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
* CUDA: dedup MoE gate/up activation quantization (fp4)
For MoE gate/up projections the src1 activation is broadcast across the
routed experts (ne11 == 1), so ids_src1 maps every one of a token's
n_expert_used slots to the same physical row. The MMQ path therefore
re-quantized each token's activation n_expert_used times.
For fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of
once per expert. For NVFP4 a single quantize+scatter kernel
(quantize_scatter_mmq_nvfp4) quantizes each token once and writes the
resulting block_fp4_mmq straight to all n_expert_used slots, using an
inverse token->compact-row map (build_tok2c). MXFP4, and
GGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique
rows then gather into the expert-sorted layout (gather_mmq_fp4_blocks).
Both are bit-identical to the previous gather-then-quantize path (identical
source data, deterministic per-block quantization), verified by
test-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the
default, gather, and per-expert paths) and by coherent end-to-end
generation. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original
per-expert path.
Same-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192
(nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks):
activation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused
quantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The
fused path avoids materializing and re-reading the 8x compact buffer,
writing the expert copies directly from registers.
* CUDA: bounds-check token ids in build_tok2c_kernel
Guard against malformed ids_src1: skip out-of-range token ids (t < 0 or
t >= n_tokens) and drop entries beyond n_expert_used per token instead of
writing past the token's tok2c region. No behavior change for valid MoE
routing data; test-backend-ops MUL_MAT_ID 790/790.
* Refactor the code based on review comments
- Removed previously added kernels that were not necessary anymore\
- Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows.
* Adding q8_1 support for dedup and addressing review comments
* Add pragma unrolls
* Remove redundant cudaMemsetAsync call
* Removing follow up redundancies
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
* opencl: workaround for A850 compiler compat
* opencl: fix DX compiler version parsing and cleanup
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* opencl: exclude Adreno A7x from using Adreno MoE kernels
Some compilers for A7x devices miscompile the repack kernels, corrupting
the weights and causing MoE models to generate garbage output
* opencl: exclude A6x and unknown Adreno from MoE weights repack
* cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel)
* chore : remove indentation of #pragma unroll
* cuda : remove unnecessary kernel template declarations
* cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants.
* cuda : relax MMA architecture requirements to Turing in lightning indexer implementation
* chore : renamed variables
* chore : rename ggml_cuda_op_lightning_indexer() to ggml_cuda_lightning_indexer()
* chore : TODO for AMD rocWMMA
* chore : whitespace formatting
* chore : another variable rename to fix problems caused by shadowing
* chore : yet another rename, this time uppercased all constants
* cuda : added alignment checks for Q and K tensors in lightning indexer implementation
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* opencl: route `sub_group_shuffle_xor` to qcom ext when KHR ext is unavailable
KHR `sub_group_shuffle_xor` is not defined by compiler when
`cl_qcom_subgroup_shuffle` is present, causing certain FA
kernels fail to build. Define the KHR shuffle_xor using
the qcom extension.
* opencl: skip FA kernels with mixed and quant types for A7x to avoid compiler crash
* metal: fuse snake activation (mul, sin, sqr, mul, add)
Mirror the CUDA, Vulkan and CPU snake fusion: same matcher on the naive
5-op chain, same F32 contract on a and inv_b, same F32/F16/BF16 kernel
with F32 compute. Follows the Metal backend idioms: bf16 instantiation
gated behind GGML_METAL_HAS_BF16 and concurrency ranges checked on the
remaining chain nodes before encoding, as done by the bin fusion.
Covered by the existing backend-agnostic SNAKE_FUSE tests.
* metal: absorb snake fusion into ggml_metal_op_bin
Extract the matcher to ggml_metal_op_can_fuse_snake, mirroring the
Vulkan naming, and dispatch the fused path from ggml_metal_op_bin.
The encode loop switch is back to a single call per case.
Address review from ggerganov
* metal: fix indentation in ggml_metal_op_can_fuse_snake
Raise the threshold for minimum buffer size from 1 GiB to 4 GiB, based
on real-world experiments of overcommitting device memory with model
weights larger than available VRAM, for example Qwen3.5-35B-A3B-Q8
running on a B70.
Also add a debug message to better track USM system allocations.
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
* [SYCL] F16 (default) Flash Attention with XMX engine via oneDNN graph API; Qwen3.6-27b-Q8_0 prefill speed up x1.21 at p=512 and x4.26 at p=80k
* [SYCL] Address review on FA oneDNN path. Result: llama-bench---pp512; 32% increase with fa1; llama-perplexity---0.11% difference; tested model: mradermacher/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf
* PR-25222 revision v2: addressed audits
* [SYCL] flash-attn oneDNN SDPA KV F16 rev 3.0: add BMG gate + multi-device sync. Narrow the scrope of this PR to Battlemage only (bmg; Xe2). Other archs (e.g., alchemist) fall back to existing FA kernel. When device_count >1, apply stream -> wait_and_throw(), validated working path for multi-gpu sync fix by @maxious.
Co-authored-by: maxious <81432+maxious@users.noreply.github.com>
* updated comment on bmg gate, noted the issue
---------
Co-authored-by: scientist3 <scientist.3@users.noreply.github.com>
Co-authored-by: hmscider <hmscider@users.noreply.github.com>
Co-authored-by: maxious <81432+maxious@users.noreply.github.com>
The f16 GEMV kernels take a vectorized path for ne00 >= 128 that casts the row
pointers to half4 or float4. When the row stride is not aligned, the wide load
becomes misaligned. On devices that require natural alignment for vector loads,
the kernel reads garbage. This is the case Intel GPUs and the kernels produce
incorrect results there. Adreno happpens to be byte addressable and the kernels
happen to work.
* opencl: do not fail backend init on devices without cl_khr_integer_dot_product
* opencl: do not call dp4 kernels when dp is unavailable
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* vulkan/cpu: Support f16 as SET_ROWS src.
This adds full support for f16 SET_ROWS (equivalent to f32) to vulkan and CPU
backends, and adds more backend tests.
* Set DenormPreserve 16 when supported, to try to fix failures on Intel
* tune error threshold
* update metal supports_op
* ggml: uniformize im2col dst_type for all conv ops
* Update ggml/src/ggml.c
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml : uniformize im2col casting logic across all conv ops
* fix : allow im2col_f16 to accept any kernel type
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Lumpiasty
thecodacus