* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used
* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* set mmap_support to false on OpenCL backend
* fix order of load modes
* use -1 for auto
* resolve load mode auto earlier to correctly pick gpu host or cpu memory
* add load mode auto to llama-bench
* bump virtgpu api version, regenerate docs
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggml_roll only asserts nb[0] == ggml_type_size, so a permuted src is a
valid input, but the CUDA and Metal roll kernels index by ne alone and
never read the nb strides. A non-contiguous src therefore produced
silently wrong results. Neither backend declared a contiguity
requirement in supports_op, so the scheduler did not fall back to the
CPU implementation, which does handle strides correctly.
Add the requirement to both backends, matching the existing
GGML_OP_ROPE guard, and add a permuted test_roll case.
* ci: prepare for amd rocm ci
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: fix editorconfig-checker
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: fix device not recognised
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: rename gpu-amd to gpu-hip
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: gpu-hip to gpu-rocm
haha
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* CUDA: allow integrated-GPU host output buffer in debug assert
On integrated GPUs (APUs), the scheduler can legitimately place a graph
node's output on the host-visible buffer, which ggml_cuda_compute_forward
already handles. The debug assert in ggml_cuda_graph_evaluate_and_capture
required every node output to be on the device buffer, so a debug build
aborts on such a node (e.g. attn_residual ADD -> ROCm_Host on RDNA3.5).
The source-tensor assert directly below already permits this via the
integrated + cuda_host exception; apply the same exception to the node's
own output buffer. Debug-only; no effect on release/compute.
Fixes test-recurrent-state-rollback on gfx1151 (Strix Halo).
* ci: enable unified memory for ROCm gfx1151 job
Work around a coherence issue on integrated RDNA3.5 (gfx1151) where GPU
kernels reading mmap-loaded weights can return incorrect output, which
makes test-llama-archs (and real inference) intermittently wrong.
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 uses managed memory, which restores
coherence. Remove once the underlying ROCm/HIP issue is fixed.
* test-llama-archs: skip jamba on HIP backend
jamba produces incorrect output (~0.55 NMSE vs CPU) on the HIP backend on
RDNA3.5 (gfx1151); the SSM kernels need separate investigation. Skip it
for now, matching the existing per-backend carve-outs (WebGPU), so the
ROCm CI can run the test for the remaining architectures.
* ci: use HIP_LAUNCH_BLOCKING for ROCm gfx1151 job
The gfx1151 ROCm CI job produced incorrect inference output (qwen3 perplexity ~88 vs ~9.4) due to an async-execution correctness issue in the HIP path. Serializing kernel launches with HIP_LAUNCH_BLOCKING=1 restores correctness. This replaces the earlier GGML_CUDA_ENABLE_UNIFIED_MEMORY workaround, which did not fix batched inference.
* test-backend-sampler: skip top-k subtests on HIP backend
The ROCm backend does not support the TOP_K/ARGSORT op at vocab scale (no CUB; bitonic argsort is capped at ncols <= 1024), so top-k/top-p backend samplers cannot be offloaded. The penalties, set_sampler, mixed, and top_p subtests assert that offload happened, so they fail on HIP. Skip them until TOP_K is supported on the ROCm backend.
* Update tests/test-backend-sampler.cpp
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
* Update tests/test-backend-sampler.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Jim Wu <ywu@xilinx.com>
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* CUDA: Fix data-races when reusing block_reduce
block_reduce currently doesn't resync after reading from SMEM, causing
potential data-races when reusing SMEM for multiple reductions.
One may consider simply always adding this in block_reduce, but this
comes at a potential perf cost
* double-buffering for single-row softmax
* double-buffering for norm as well
* Add comment
* Add explanatory comment to block_reduce
* Specify need for + do memory barrier only in multi-warp scenario
* Implement review-suggestion from @gaugarg-nv
* add bool cwhn = true to conv_2d test cases
* add layout check at graph building time
* extend layout checks for conv2d.cu kernel
* in CPU back-end kernel needs to be stored contiguously to prevent test failures with cwhn=1
* trim white space
* do op support check in vulkan backend
* fix CI failure and vulkan run-time assert failure by introducing new graph build-time check in ggml_backend_vk_device_supports_op
* add additional check in support_op function for Vulkan to fix run-time assert failure
ggml_cuda_should_use_mmq() selects MMQ purely from the quantization
type. The current MMQ configurations are designed and maintained against
a minimum of 48 KiB per-block shared memory, the limit provided by
NVIDIA Pascal GPUs and later. On devices that report less, no supported
MMQ tile fits and mul_mat_q_switch_J() aborts when every tile size
exceeds the device's per-block shared memory budget.
Disable MMQ when smpbo < 48 KiB so the caller falls back to the BLAS
path instead of hitting GGML_ABORT. Some current MUSA QY1 devices
report only 28 KiB and are covered by this guard.
Reproduced on a Moore Threads MTT S70 (arch mp_21, 28 KiB shared memory
per block) with an RWKV-7 0.1B Q8_0 model:
$ llama-bench -m rwkv7-g1d-0.1b-Q8_0.gguf -p 128 -n 0
J_best=0
ggml/src/ggml-cuda/template-instances/../mmq.cuh:1521: fatal error
(core dumped)
Only prefill (batch > 1) is affected; token generation is fine. After
the fix the same device falls back to the BLAS path:
Q8_0 pp128 1470.7 t/s, tg8 55.3 t/s (was: abort)
FP16 unchanged
Q4_K_M unchanged
This matches a -DGGML_CUDA_FORCE_CUBLAS=ON build (pp128 1464.2 t/s),
which confirms the fallback path is the one being taken.
This is not MUSA-specific: any device with less than 48 KiB per-block
shared memory is affected.
Co-authored-by: KakaruHayate <KakaruHayate@users.noreply.github.com>
* ggml-cuda: add chunked SSD matmul for Mamba-2 prefill acceleration
* cuda: added SSD CICD fixes for CUDA / HIP / MUSA / MSVC.
* ggml-cuda: review comments fixed.
* ggml-cuda: Fuse M matrix materialization into pre_matmul kernel and enabled test.
* ggml-cuda: test updates and fixes
* ggml-cuda: test updates to remove hardcoding of tensor initialise data limits.
* ggml-cuda: ssd minor review comment fixed.
* ggml-cuda: ssd minor CICD fixed.
* CUDA SSD: Fixes correctness by promoting s0_stride_seq to int64_t, improves memory coalescing in ssm_ssd_prepare_dt_kernel, and boosts efficiency by merging B_weighted and C_scaled; also addresses prior review comments.
* cuda: fix sdata read-write race in prepare_dt fallback scan loop
* Squash history before conflict-resolution during rebase on master
WIP commit
Add 32-byte loads, restore per-block amax
Use nvfp4x4 intrinsic when available
Fuse per-channel amax and quantization kernels
Do pointer arithmetic only once on x
Remove unnecessary ternary in the load
We assert on host side that ne00 is 64-aligned
Add back scale-search, but optimize it with intrinsics
Code cleanup
Make scale in MMQ-epilogue NVFP4-specific/restrictive for now
Remove unneeded include, add comment
Fix trailing whitespace
Guard __builtin_align__(32) struct to NVIDIA
Seems like HIP doesn't have this available, see https://github.com/ggml-org/llama.cpp/actions/runs/29438651734/job/87431623001
* compiler massaging to avoid unnecessary LDCs
* kvalues_mxfp4 -> kvalues_nvfp4 in quantize_mmq_nvfp4
* Always pass in src1_scale.ptr
* Extract ggml_cuda_is_aligned helper
* cuda: add k-quant support to GET_ROWS
Device-side embedding lookups require GET_ROWS to handle the k-quants
used by common GGUF recipes (Q4_K_M stores token_embd as q6_K). Without
it the backend rejects the op and the scheduler falls back to the host,
copying the full embedding matrix back on every token in single-device
graphs.
Factor the super-block dequantizers out of the dequantize_block kernels
in convert.cu into shared device functions in dequantize.cuh and reuse
them from a new k_get_rows_kq kernel : one thread block dequantizes one
(dst row, super-block) pair with the existing thread layouts, 32 threads
for q4_K and 64 for the other k-quants.
Covers q2_K to q6_K in get_rows_cuda and supports_op. i-quants are left
as a TODO.
* cuda: add i-quant support to GET_ROWS
Extends the shared super-block dequantizers to the nine i-quants and
reuses them from k_get_rows_kq with the 32-thread layout of the matching
convert.cu kernels. supports_op gates the k-quant and i-quant path on
ne0 being a multiple of QK_K, which iq4_nl does not guarantee on its
own (QK4_NL sub-blocks). mxfp4 is left as a TODO.
* cuda: add mxfp4 support to GET_ROWS
Moves the mxfp4 dequantizer into the shared super-block helpers and
reuses it from k_get_rows_kq with the 32-thread layout of the matching
convert.cu kernel. mxfp4 joins the ne0 % QK_K gate in supports_op since
its 32-value sub-blocks do not guarantee QK_K-aligned rows on their own.
This closes GET_ROWS type coverage on CUDA: every quantized GGML type
now takes the direct device path.
* cuda: gate the GET_ROWS row size only for 32-value sub-block types
Address review from @pwilkin: the i-quant commit replaced the return
shared by the whole supported type cascade, so f16/f32/bf16/i32 and the
legacy quants also inherited the ne0 % QK_K == 0 gate and any row size
that is not a multiple of 256 fell back to the scheduler. Split the
cascade: unconditional support is restored everywhere, the gate stays
only on iq4_nl and mxfp4 whose 32-value sub-blocks do not guarantee the
QK_K super-blocks the kernel iterates on.
k_get_rows_float did a scalar one-element-per-thread copy and recomputed the
row-invariant work (index load, fast_div_modulo, src/dst row pointers) for
every element. Hoist that out of the per-element loop, and add a vectorized
path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the
contiguous same-type (no-cast) case.
The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at
runtime on 16-byte alignment of the base pointers and all row strides and on
ne00 % VEC == 0. Vectorizing divides the block count by VEC, so a small
single-row gather can drop below the device CU count and regress; an
occupancy gate keeps those on the block-rich scalar path.
On Strix Halo (gfx1151) the DeltaNet recurrent-state gather (ne00=524288)
drops 18.6us -> 13.0us (rocprofv3 HW timestamps), faster than the Vulkan
backend, with no regression on the small conv-state gather; total get_rows
-27%. test-backend-ops GET_ROWS passes (47/47).
Assisted-by: Claude Opus 4.8
* support cuda virtual devices
* disable NCCL path when virtual devices are used
* label virtual devices in description; add GPUx2 server CI jobs
* code refactor
PR #16308 set info.devices[id].integrated = false unconditionally for all
CUDA/HIP devices as a workaround for corrupted output on Jetson Orin
(#15034). On HIP/ROCm the device's real hipDeviceProp_t.integrated flag is
needed: with the cached field forced to false, supports_buft() refuses
CUDA host buffers on AMD APU/UMA parts, while get_type() already reads
prop.integrated (#23007) — an inconsistency that breaks integrated-GPU
host-buffer use on ROCm.
Guard the workaround so it only applies to non-HIP (CUDA) builds and
restore prop.integrated for HIP, keeping the Jetson workaround intact for
CUDA.
Fixes#23977
Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
* CUDA: dedup MoE gate/up activation quantization (fp4)
For MoE gate/up projections the src1 activation is broadcast across the
routed experts (ne11 == 1), so ids_src1 maps every one of a token's
n_expert_used slots to the same physical row. The MMQ path therefore
re-quantized each token's activation n_expert_used times.
For fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of
once per expert. For NVFP4 a single quantize+scatter kernel
(quantize_scatter_mmq_nvfp4) quantizes each token once and writes the
resulting block_fp4_mmq straight to all n_expert_used slots, using an
inverse token->compact-row map (build_tok2c). MXFP4, and
GGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique
rows then gather into the expert-sorted layout (gather_mmq_fp4_blocks).
Both are bit-identical to the previous gather-then-quantize path (identical
source data, deterministic per-block quantization), verified by
test-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the
default, gather, and per-expert paths) and by coherent end-to-end
generation. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original
per-expert path.
Same-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192
(nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks):
activation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused
quantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The
fused path avoids materializing and re-reading the 8x compact buffer,
writing the expert copies directly from registers.
* CUDA: bounds-check token ids in build_tok2c_kernel
Guard against malformed ids_src1: skip out-of-range token ids (t < 0 or
t >= n_tokens) and drop entries beyond n_expert_used per token instead of
writing past the token's tok2c region. No behavior change for valid MoE
routing data; test-backend-ops MUL_MAT_ID 790/790.
* Refactor the code based on review comments
- Removed previously added kernels that were not necessary anymore\
- Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows.
* Adding q8_1 support for dedup and addressing review comments
* Add pragma unrolls
* Remove redundant cudaMemsetAsync call
* Removing follow up redundancies
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
* cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel)
* chore : remove indentation of #pragma unroll
* cuda : remove unnecessary kernel template declarations
* cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants.
* cuda : relax MMA architecture requirements to Turing in lightning indexer implementation
* chore : renamed variables
* chore : rename ggml_cuda_op_lightning_indexer() to ggml_cuda_lightning_indexer()
* chore : TODO for AMD rocWMMA
* chore : whitespace formatting
* chore : another variable rename to fix problems caused by shadowing
* chore : yet another rename, this time uppercased all constants
* cuda : added alignment checks for Q and K tensors in lightning indexer implementation
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
If a Cuda device has no or limited available memory, the actual call
to cudaMemGetInfo() itself can cause a fatal crash due to a cuda out
of memory error (there is not enough memory to actually query memory)
This causes an issue because we query memory for all devices at
startup even if the user isn't trying to use the device for inference.
Fix this by making the error non-fatal and assigning zero total/free
memory to the device. This will have the downstream effect of the fit
algorithm not trying to put any layers on it, which is desired outcome
vs hard crashing.
this also prevents crashes in cuda enabled builds when user explicitly
passes '-dev none'
* ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage
* ggml : allocate tmp_dst only only once before the loop
* chore : whitespaces
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml : use chunked processing in both CUDA CUB top-k and argsort implementations
* chore : separate argsort_f32_i32_cuda_bitonic() call from return statement
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* chore : replace ternary operators with min/max
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* cuda: fix snake fusion type predicate, a and inv_b are F32
The matcher required a->type == x->type while launch_snake reads both
as const float *, matching the CPU and Metal contract where a and inv_b
stay F32. F16/BF16 chains never fused and fell back to the naive path,
and a hypothetical all F16 chain would have read F16 bits as float.
Aligns the predicate and the comment with ggml-cpu.c
* cuda: reject snake fusion on non-contiguous operands
The kernel reads x[idx] and a[c] / inv_b[c] linearly, so a
non-contiguous view passing the matcher would silently read wrong data.
Mirror the contiguity guard already present in the CPU, Vulkan and
Metal matchers.
* Update ggml-cuda.cu - Turing P2P access fix.
* Add original code as fallback behaviour when NCCL or P2P is not set/true.
* Update ggml/src/ggml-cuda/ggml-cuda.cu to add comment as per suggestion
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* cuda : concat implementation for quantized types
* chore : apply am17an clever suggestion to shorten the code
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* cuda: enable topk-moe fusion for 288 experts
The topk-moe fusion only accepted power-of-2 expert counts (or the
special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash)
fell back to the unfused per-layer routing chain: softmax/sigmoid,
argsort, get_rows, sum_rows, div, clamp, scale. At batch size 1 that
is ~330 extra tiny graph nodes per token.
288 is a multiple of the warp size, so the existing kernel already
handles it; this adds the missing template instantiation and accepts
288 in the eligibility check.
Measured on gfx1151 with Step-3.7-Flash IQ4_XS (llama-bench,
-b 4096 -ub 4096 -fa 1 -dio 1 -ctk q8_0 -ctv q8_0; machine idle,
before/after paired so pp4096 stays matched as a load control):
test | before | after
----------------+----------------+----------------
pp4096 | 460.99 ± 0.45 | 462.47 ± 0.34 (unchanged)
tg128 | 19.10 ± 0.04 | 19.56 ± 0.03 (+2.4%)
tg128 @ d30000 | 12.68 ± 0.04 | 12.69 ± 0.03 (unchanged)
Prompt processing is unaffected (the fusion only touches decode
routing). The decode gain is ~+2.4% at shallow context and fades with
depth: by 30k tokens each step is attention-bound over the KV cache,
so removing the fixed routing overhead is no longer visible.
Assisted-By: Claude Fable 5 <noreply@anthropic.com>
* Update tests/test-backend-ops.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Add comment for case 288 in topk-moe.cu
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Remove redundant CUDA copies after gated_delta_net.
Currently, GDN writes recurrent state snapshots into its output tail, then the graph immediately copies those snapshots into ssm_states_all. With MTP draft length 3, target decode uses K=4, so that becomes 4 extra ggml_cuda_cpy calls.
The change detects that gated_delta_net -> view -> cpy pattern and makes the CUDA GDN kernel write the state snapshot(s) directly into the recurrent cache, skipping the intermediate tail writes and copy kernels when safe.
* Address review comments
Rafail Giavrimis