Incrementing `ref_count` at the beginning is important later
in the `free()` method of the `ggml_backend_opencl_context` at program end.
If we do not increment the `ref_count`, the result would be -1 here,
and consequently, the profiling data would not be flushed and written.
( #ifdef GGML_OPENCL_PROFILING )
The Adreno KQ/KQV image1d kernels (ggml_cl_mul_mat_kq_kqv_adreno) ignore
dim 3 entirely: the sub-buffer covers only nb02*ne02 bytes and the kernel
receives no ne03/ne13/nb03/nb13 arguments. With the unified KV cache,
multi-sequence batches (e.g. llama-perplexity with its default -b 2048,
n_seq=4, or a multi-slot llama-server) present KQ/KQV as 4D tensors with
ne3 = n_stream, so every stream past the first reads the first stream's
K/V and produces garbage. Flash attention masks the bug where it is
enabled; devices where FA is declined (e.g. Adreno 740) hit it with
default settings.
Route ne03/ne13 > 1 to the general path, which handles dim 3, and honor
view_offs when creating the sub-buffers (currently always 0 for tensors
reaching this function, but the function would silently misread any
future view).
Llama-3.2-1B-Instruct Q4_0, wiki.test.raw, 8 chunks, -ngl 99:
- Adreno 740, default: PPL 1817.64 -> 15.61
- Adreno 740, -fa 0: PPL 1941.64 -> 15.61
- Adreno 840, -fa 0: PPL 1943.90 -> 15.50
- single-stream (-b 512) results unchanged (15.6090)
- test-backend-ops -o MUL_MAT on 740: identical before/after (909 OK,
12 pre-existing q6_K failures)
* opencl: workaround for A850 compiler compat
* opencl: fix DX compiler version parsing and cleanup
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* opencl: exclude Adreno A7x from using Adreno MoE kernels
Some compilers for A7x devices miscompile the repack kernels, corrupting
the weights and causing MoE models to generate garbage output
* opencl: exclude A6x and unknown Adreno from MoE weights repack
* opencl: route `sub_group_shuffle_xor` to qcom ext when KHR ext is unavailable
KHR `sub_group_shuffle_xor` is not defined by compiler when
`cl_qcom_subgroup_shuffle` is present, causing certain FA
kernels fail to build. Define the KHR shuffle_xor using
the qcom extension.
* opencl: skip FA kernels with mixed and quant types for A7x to avoid compiler crash
* opencl: do not fail backend init on devices without cl_khr_integer_dot_product
* opencl: do not call dp4 kernels when dp is unavailable
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* opencl: fix garbled output for Q6_K weights with ne01 % 128 != 0 on Adreno
Observed with granite-3.1-3b-a800m-instruct, whose vocab is an odd number.
Route Q6_K dense mul_mat with ne01 % 128 != 0 off the noshuffle path:
decode (ne1==1) uses the correct flat GEMV and the matching GEMM (ne1>1)
falls back to CPU (the flat convert has no verified small-batch GEMM kernel
for these shapes). All standard hidden/FFN/vocab dims are multiples of 128
and keep the noshuffle path.
* opencl: reserve alignment slack for the SOA subbuffer carve in alloc size
set_tensor carves quantized weights into per-component subbuffers (d/q,
ql/qh/s/d, ...) whose origins are each rounded up to the device base
address alignment. When a component's size is not a multiple of the
alignment, the carve extends past ggml_nbytes(tensor) and the last
subbuffer overlaps the next tensor in the pool -- e.g. q6_K [1536, 49155]:
size_s = 49155*96 ends 32 bytes past a 128-byte boundary, so the d
subbuffer ends 96 bytes past the tensor's allocation, and whichever of the
two neighboring tensors is uploaded last silently corrupts the other (here:
the last vocab rows' block scales). This affects any quant type whose
component sizes can be misaligned, on any shape with ne01 not a multiple of
the alignment granularity; standard power-of-two dims are unaffected.
Implement get_alloc_size for the OpenCL buffer type and reserve the
worst-case carve slack (4 aligned gaps; 5 components max, q5_K) for
quantized tensors. Costs at most 512 bytes per quantized tensor at the
observed 128-byte alignment.
* opencl: use lm based q6_k mm when ne1 is not multiple of 128
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* opencl: ragged-tile MoE prefill GEMM (skip padded expert tiles)
The MoE prefill GEMM groups tokens into TILESIZE_N=32 per-expert tiles; at low
tokens-per-expert most tiles are mostly padding. When a tile's upper 16 slots
are all padding (router index 0xFFFFFFFF), skip the second dotx16_reduce8 half.
Numerically identical (skipped lanes are padding). Applied to all eight *_f32_ns
MoE GEMMs; default on, opt out with GGML_OPENCL_MOE_RAGGED_FP16=0.
* opencl: quarter-granularity ragged MoE tile-skip (8-col skip-groups)
Replace the two half-tile dotx16_reduce8 calls in the 8 *_f32_ns MoE GEMMs with
four dotx8_reduce4 (8-column) calls, skipping each empty trailing skip-group
independently. Padding is always trailing, so the kernel rounds the valid count
up to the skip granularity and skips fully-padding groups. Byte-identical to the
non-skipped path. New env GGML_OPENCL_MOE_RAGGED_GRAN={8,16,32} (quarter/half/
off); default quarter.
* opencl: move ragged moe env var in cl_init
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* opencl: vec flash-attention decode kernels for f16/q8_0/q4_0 KV
* opencl: improve non FA KQ mv kernels
* opencl: tweaks for multiquery FA
* opencl: some tweaks for FA q1 kernels
* opencl: FA with DK=DV=512 for gemma-4
* opencl: various fixes
* opencl: cleanup
* opencl: fix FA decode crash for DK=512 (gemma-4)
The DK=512 decode-only program does not create the f32_f16 prefill
kernel, so the compiled check in ensure_fa_variant never hit and
supports_op gave inconsistent answers for the same op. block_n is also
unset for DK=512 decode; guard it to avoid an out-of-range read at
dispatch.
* opencl: run DK=512 FA decode on CPU
DK=512 decode is bandwidth-bound and faster on the CPU than the GPU,
increasingly so with depth. Decline it in supports_op; prefill stays on the GPU.
* opencl: compile MQ_GQA=8 FA kernels in a minimal program
The full program compiled with -D MQ_GQA=8 runs the Adreno compiler out
of memory at DK>=256. Only the vec_mq kernels are used from this
program, so compile it with FA_MQ_ONLY, which excludes everything else.
Also include the program name in the compile error log.
* opencl: remove stray token in flash_attn_f32_f16.cl
A stray "." broke the f32_f16 program build.
* opencl: split f16-KV FA decode finer (FD_KV_PER_SPLIT_F16)
The 2048 default under-fills the GPU on single-query f16-KV decode;
use 512 for f16 KV to get more splits. Quantized KV keeps 2048.
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* opencl: rework FA kernel for f16 and f32
* opencl: flash-attention prefill prepass kernels
- flash_attn_kv_pad_f16 pads the tail KV tile to a BLOCK_N multiple
- flash_attn_mask_pad_f16 pads the matching mask tile
- flash_attn_blk_f16 classifies each KV tile per query block as
fully masked / mixed / fully unmasked, so
the main kernel can skip fully-masked tiles
and the mask lookup for fully-unmasked ones
* opencl: FA kernels for q4_0 and q8_0
* opencl: `set_rows` for f32 to q8_0/q4_0
* opencl: dequant kernels for q4_0 and q8_0
* opencl: add FA tile tuning table with override
* opencl: wire host side for FA
* opencl: q4_0 MoE tensors are also SOA'ed
* opencl: cosmetic fix
* opencl: refactor, also clarify some code paths in comments
* opencl: fix inifity for `-cl-finite-math-only`
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* Make ggml_gated_delta_net take only the initial recurrent state (D, 1, n_seqs) and passes the snapshot count K as an op parameter instead of inferring it from state->ne[1].
Remove the padding hack and copy all emitted snapshots into the recurrent cache with a single strided ggml_cpy
* Make GDN changes in all backends. Address review comments.
* Fix CI build errors
* opencl: add general q5_0 support
* opencl: add general q5_1 support
* opencl: support non-uniform workgrp size
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* OP_GATED_DELTA_NET impl
* add back lanes_per_column declaration
* removed has_subgroup_arithmetic and has_subgroup_clustered_reduce
* removed trailing spaces and fixes indentation. Hard coded subgroup size for Adreno and Intel. Return not supported when K>1 state snapshot
* support for K>1 state snapshot
* removed picky indent multiple of 4 fixes
* removed return that won\'t be executed
* opencl: refactor initialization
* opencl: refactor GPU identification
* opencl: rename for consistency
* opencl: cache global mem size in dev_ctx
* opencl: adjust log level
* opencl: load argsort and flash_attn kernels in supports_op
* argsort kernel must be built for supports_op for querying the max
workgroups
* flash_attn kernel has many variants, only load them when needed
* opencl: add q4_k moe support
* opencl: add q5_k moe support
* opencl: add q6_k moe support
* opencl: adjust format
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* opencl: add q5_0 moe support
* opencl: add q5_1 moe support
* opencl: avoid potential leak
* opencl: suppress unused var warning when building for non-Adreno
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* Q4_1 MoE CLC pass sanity check
* remove unnecessary code
* opencl: remove unnecessary asserts and reformat
* opencl: fix supports_op for q4_1 moe
* q4_1 moe is supported by Adreno with certain shapes
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* Q4_0 MoE CLC pass sanity check
* release program
* opencl: fix whitespace
* opencl: remove unused cl_program
* opencl: break #if block to make it more clear
* opencl: adjust format
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
akleine