From 78ffbaa18533c2ed51eab2ef5ae0702330efc3e9 Mon Sep 17 00:00:00 2001 From: Hongqiang Wang Date: Mon, 9 Nov 2026 23:00:55 -0800 Subject: [PATCH] vulkan: cap scalar flash attention at 8 rows for large heads on Qualcomm (#30829) With head sizes above 128 and KV >= 1024, scalar flash attention uses 16 rows per workgroup. On the Qualcomm Adreno driver this variant spills registers to scratch memory (264 bytes per invocation on the X1-85, none at 8 rows) and runs about 40x slower. The largest test-backend-ops case then becomes a single dispatch of several seconds, past the Windows GPU timeout, and the test never completes on the X1 Elite. Use at most 8 rows per workgroup on Qualcomm when hsk or hsv > 128. Smaller heads keep 16 rows, which is faster for them on the X2 Elite. Adreno X1-85, FLASH_ATTN_EXT hsk=256 kv=4096: nb=64 711 ms -> 17 ms, nb=512 no result -> 125 ms; test-backend-ops completes. Adreno X2-90: Gemma 4 E4B pp512 at depth 4096 with flash attention 81-90 -> 117 t/s. --- ggml/src/ggml-vulkan/ggml-vulkan.cpp | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp index 946c4f565..0eb759428 100644 --- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp +++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp @@ -1260,6 +1260,12 @@ static vk_fa_tuning_params get_fa_tuning_params_scalar(const vk_device& device, result.block_rows /= 2; } + // With head sizes above 128, 16 rows per workgroup spill registers on Qualcomm and run about + // 40x slower; at large KV a single dispatch can then exceed the Windows GPU timeout. + if (device->vendor_id == VK_VENDOR_ID_QUALCOMM && (hsk > 128 || hsv > 128)) { + result.block_rows = std::min(result.block_rows, 8u); + } + // On AMD RDNA, for small head sizes and big batch size the shader uses few registers, so too many subgroups get scheduled // at once and end up thrashing the cache. Fix this by setting a large (unused) shmem buffer that reduces occupancy. // This targets an occupancy of 4 subgroups per SIMD.