Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

opencl: fix q6_K flat mul_mat for Adreno A6x/A7x GPUs with older E031 compilers (#26476)

Browse Source 
* opencl: decline KV-convert flash_attn variants on Adreno A7X (compiler SIGSEGV)

The Adreno 740 (A7X) compiler E031.41 crashes inside clBuildProgram when
building the flash_attn programs whose KV path is mixed-type or dequantized:
flash_attn_f32_f16, flash_attn_f32_q8_0, flash_attn_f32_q4_0. It is a driver
crash rather than a compile-error return, so build_program_from_source_ex()
cannot catch it. The uniform f32 and f16 programs build correctly.

Decline the three KV-convert variants on the A7X in supports_op so they never
lazy-compile; those attention layers run on the CPU backend instead. Same
idiom as the existing Intel DK=512 and X1E carve-outs.

test-backend-ops FLASH_ATTN_EXT on the 740: 226 OK / 0 FAIL, previously exit
139. Other parts are unaffected - the gate is dead code there.

* opencl: fix q6_K flat mul_mat on older Adreno E031 compilers, gated

kernel_mul_mv_q6_K_f32_flat produces ~10x-wrong output on the older Adreno
E031 compilers while q4_K and q5_K are correct. Four codegen defects, each
confirmed on-device against the CPU reference:

  1. 64-bit ulong arithmetic is miscompiled, so every weight and scale read
     hit the wrong address - the primary cause, and why q5_K (int offsets)
     was unaffected. The block index is computed in int and widened only
     inside the pointer expression.
  2. The vectorized dequant (int4/float4 bit-ops, convert_*4, dot()) is
     miscompiled; the 6-bit weights are reconstructed and the dot done
     scalar.
  3. vload4 of the f32 activations is miscompiled; replaced by a
     scalar-indexed load.
  4. The accumulation is miscompiled unless a side effect forces the partial
     sums to materialize. A printf under a guard the compiler cannot prove
     false acts as a zero-cost optimizer barrier; its placement is
     load-bearing.

The defect tracks the compiler, not the GPU generation: it reproduces on
E031.38 (Adreno 642L) and E031.41 (Adreno 740) and is fixed by E031.45
(Adreno 619), so the workarounds are gated on the compiler version. Where
they are not needed they cost real throughput - 42.4 -> 35.1 GFLOPS on an
Adreno 840 q6_K GEMV. The explicit compiler-type check is required, not
redundant: newer_than_or_same() is false for every non-E031 compiler, so
negating it alone would enable the workarounds on E17 and DX.

test-backend-ops MUL_MAT is 919/919 on the Adreno 740, 642L, 619, 840 and
850; the 740 and 642L were 909/919 before. The 642L additionally needs the
A6X per-kernel-program support to reach these tests at all.
This commit is contained in:
Hongqiang Wang
2026-08-20 10:58:35 -07:00
committed by GitHub
parent 6b4fa88a6c
commit 6503355df0
2 changed files with 136 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-opencl/ggml-opencl.cpp
+41 -1
View File
@@ -582,6 +582,8 @@ struct ggml_backend_opencl_context {
bool adreno_use_bin_kernels;
get_adreno_bin_kernel_func_t get_adreno_bin_kernel_func = nullptr;
ggml_cl_compiler_version adreno_cl_compiler_version;
// The q6_K flat mul_mat codegen workarounds are needed by old E031 compilers only.
bool q6_k_flat_old_compiler;
std::string kernel_compile_opts; // cached for lazy-compiled kernels.
@@ -1931,8 +1933,14 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
#else
const std::string kernel_src = read_file("mul_mv_q6_k_f32_flat.cl");
#endif
// The codegen workarounds in this kernel are a measured 13-20% loss on
// compilers that do not need them, so only the affected ones build them;
// everyone else gets the original source.
const std::string q6k_opts = backend_ctx->q6_k_flat_old_compiler
? compile_opts + " -DADRENO_OLD_COMPILER=1"
: compile_opts;
cl_program prog =
build_program_from_source(backend_ctx, kernel_src.c_str(), compile_opts);
build_program_from_source(backend_ctx, kernel_src.c_str(), q6k_opts);
CL_CHECK((backend_ctx->kernel_mul_mv_q6_K_f32_flat = clCreateKernel(prog, "kernel_mul_mv_q6_K_f32_flat", &err), err));
CL_CHECK(clReleaseProgram(prog));
@@ -5917,6 +5925,16 @@ static ggml_backend_opencl_context * ggml_cl_init(ggml_backend_dev_t dev) {
(backend_ctx->adreno_cl_compiler_version.type == E031 && backend_ctx->adreno_cl_compiler_version.major >= 47) ||
(backend_ctx->adreno_cl_compiler_version.type == DX && backend_ctx->adreno_cl_compiler_version.major >= 17);
// The q6_K flat mul_mat miscompile is a defect of the older E031 compilers, not a
// property of any GPU generation: it reproduces on E031.38 (Adreno 642L) and E031.41
// (Adreno 740) and is fixed by E031.45 (Adreno 619). Gate on the compiler so parts
// that do not need the workarounds do not pay for them. The explicit type check is
// required: newer_than_or_same() is false for every non-E031 compiler, so negating it
// alone would enable the workarounds on E17/DX.
backend_ctx->q6_k_flat_old_compiler =
backend_ctx->adreno_cl_compiler_version.type == E031 &&
!backend_ctx->adreno_cl_compiler_version.newer_than_or_same(E031, 45, 0, 0);
size_t ext_str_size;
clGetDeviceInfo(device, CL_DEVICE_EXTENSIONS, 0, NULL, &ext_str_size);
char *ext_buffer = (char *)alloca(ext_str_size + 1);
@@ -7496,6 +7514,7 @@ static bool ggml_opencl_supports_op(ggml_backend_dev_t dev, const struct ggml_te
v->type == GGML_TYPE_F16 && op->type == GGML_TYPE_F16;
const bool is_f32_f16 = q->type == GGML_TYPE_F32 && k->type == GGML_TYPE_F16 &&
v->type == GGML_TYPE_F16 && op->type == GGML_TYPE_F32;
const bool is_f32_q8_0 = q->type == GGML_TYPE_F32 && k->type == GGML_TYPE_Q8_0 &&
v->type == GGML_TYPE_Q8_0 && op->type == GGML_TYPE_F32 &&
dk % 32 == 0 && dv % 32 == 0;
@@ -7503,6 +7522,21 @@ static bool ggml_opencl_supports_op(ggml_backend_dev_t dev, const struct ggml_te
v->type == GGML_TYPE_Q4_0 && op->type == GGML_TYPE_F32 &&
dk % 32 == 0 && dv % 32 == 0;
// A7X (Adreno 740, compiler E031.41) SIGSEGVs inside clBuildProgram
// building the flash_attn programs whose KV path is mixed-type or
// dequantized — f32_f16, q8_0, q4_0 (reproduced at DK=40 and DK=64; it
// is DK-independent). It is a driver crash, not codegen-wrong-output, so
// it cannot be caught in-process (fatal=false only handles clean compile
// errors). The uniform f16_f16 / f32_f32 programs compile fine on this
// compiler, so decline only the KV-convert variants; ggml then runs
// those (f16-KV / quant-KV) attention layers on the CPU backend.
// Negative compiler carve-out, same idiom as the Intel DK=512 decline
// below and the X1E driver-quirk guards.
if (backend_ctx && backend_ctx->adreno_gen == ADRENO_GPU_GEN::A7X &&
(is_f32_f16 || is_f32_q8_0 || is_f32_q4_0)) {
return false;
}
// Asymmetric KV: host-dequants both sides to F32, uses f32 kernel.
auto is_kv_type_ok = [](ggml_type t) {
return t == GGML_TYPE_F16 || t == GGML_TYPE_F32 ||
@@ -20583,6 +20617,12 @@ static void ggml_cl_mul_mat(ggml_backend_t backend, const ggml_tensor * src0, co
CL_CHECK(clSetKernelArg(kernel, 14, sizeof(int), &ne1));
CL_CHECK(clSetKernelArg(kernel, 15, sizeof(int), &r2));
CL_CHECK(clSetKernelArg(kernel, 16, sizeof(int), &r3));
// The optimizer-barrier arg exists only in the ADRENO_OLD_COMPILER build of
// this kernel; conformant compilers get the original 17-arg signature.
if (backend_ctx->q6_k_flat_old_compiler) {
cl_uchar q6k_mask = 0xFF; // never 0xFE in prod; see the kernel note
CL_CHECK(clSetKernelArg(kernel, 17, sizeof(cl_uchar), &q6k_mask));
}
#else
kernel = backend_ctx->kernel_mul_mv_q6_K_f32;