Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Extended SYCL oneDNN SDPA to non-FP16 KV caches (Q4_0–Q8_0 and FP32) (#25874)

Browse Source 
* sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches

Extends the oneDNN SDPA path (PR #25222) to handle non-F16 KV caches by
dequantizing or converting K/V to dense FP16 on-device before feeding
them into the SDPA graph. The fused systolic kernel then runs identically
to the native FP16 path.

Supported KV types:
  - Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl
  - F32: cont_to_f16_sycl<float>
  - BF16 and IQ types are excluded (no conversion kernel available)

Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only).
F16 KV runs at any length (existing behavior).

Also includes the stream sync fix (stream->wait_and_throw() unconditional,
PR #25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are
always dequantized to separate buffers).

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR)
Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Ozymandias_EBON
2026-08-04 10:07:23 +08:00
committed by GitHub
co-authored by Claude
parent 0ef6e55edb
commit 66fa168a56
2 changed files with 142 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-sycl/fattn.cpp
+20 -5
View File
@@ -97,7 +97,7 @@ static void ggml_sycl_flash_attn_ext_vec(ggml_backend_sycl_context & ctx, ggml_t
enum best_fattn_kernel {
BEST_FATTN_KERNEL_NONE = 0,
BEST_FATTN_KERNEL_VEC = 100,
BEST_FATTN_KERNEL_ONEDNN = 150, // added enum for onednn==150
BEST_FATTN_KERNEL_ONEDNN = 150, // oneDNN SDPA: native F16 (PR #25222)
BEST_FATTN_KERNEL_TILE = 200,
BEST_FATTN_KERNEL_MKL = 300,
};
@@ -130,6 +130,14 @@ static best_fattn_kernel ggml_sycl_get_best_fattn_kernel(const int device, const
bool gqa_opt_applies = gqa_ratio >= 2 && mask && max_bias == 0.0f && K->ne[1] % FATTN_KQ_STRIDE == 0;
// XMX-accelerated path: oneDNN SDPA (native F16 and dequant+non-F16).
// ONEDNN requires min 32 query tokens — short-circuit decode to avoid
// calling _supported() on every decode FA call.
if (Q->ne[1] >= 32
&& ggml_sycl_flash_attn_ext_onednn_supported(dst)) {
return BEST_FATTN_KERNEL_ONEDNN;
}
// MKL path: XMX-accelerated GEMM for prompt processing (all KV cache types).
// The MKL kernel converts non-F16 K/V to F16 via to_fp16_sycl before GEMM,
// so quantized, F16, BF16, and F32 caches all benefit from XMX acceleration.
@@ -167,7 +175,6 @@ static best_fattn_kernel ggml_sycl_get_best_fattn_kernel(const int device, const
return BEST_FATTN_KERNEL_MKL;
}
}
for (const ggml_tensor * t : {Q, K, V, mask}) {
if (t == nullptr || ggml_is_quantized(t->type)) {
continue;
@@ -215,6 +222,7 @@ static best_fattn_kernel ggml_sycl_get_best_fattn_kernel(const int device, const
switch (K->type) {
case GGML_TYPE_F32:
case GGML_TYPE_F16:
case GGML_TYPE_BF16:
break;
case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
@@ -233,8 +241,11 @@ static best_fattn_kernel ggml_sycl_get_best_fattn_kernel(const int device, const
return BEST_FATTN_KERNEL_NONE;
}
// For small batch sizes the vector kernel may be preferable over the kernels optimized for large batch sizes:
const bool can_use_vector_kernel = Q->ne[0] <= 512 && Q->ne[0] % 64 == 0 && K->ne[1] % FATTN_KQ_STRIDE == 0;
// For small batch sizes the vector kernel may be preferable over the kernels optimized for large batch sizes.
// BF16 is excluded: the VEC kernel has no BF16 template (it needs GGML_SYCL_FA_ALL_QUANTS for non-F16/Q4_0/Q8_0).
const bool has_bf16 = (K->type == GGML_TYPE_BF16 || V->type == GGML_TYPE_BF16);
const bool can_use_vector_kernel = Q->ne[0] <= 512 && Q->ne[0] % 64 == 0 && K->ne[1] % FATTN_KQ_STRIDE == 0
&& !has_bf16;
// Fused-XMX path: oneDNN Graph SDPA (flash attention). Strictly
// additive -- taken only when statically supported, otherwise falls through to VEC/TILE below.
@@ -276,6 +287,7 @@ void ggml_sycl_flash_attn_ext(ggml_backend_sycl_context & ctx, ggml_tensor * dst
const char * kname = "TILE";
best_fattn_kernel k = ggml_sycl_get_best_fattn_kernel(ctx.device, dst);
if (k == BEST_FATTN_KERNEL_MKL) kname = "MKL";
if (k == BEST_FATTN_KERNEL_ONEDNN) kname = "ONEDNN";
if (k == BEST_FATTN_KERNEL_VEC) kname = "VEC";
int64_t delta = 0;
if (Dk == 256) {
@@ -292,7 +304,8 @@ void ggml_sycl_flash_attn_ext(ggml_backend_sycl_context & ctx, ggml_tensor * dst
(long long)V_dbg->ne[1]);
}
switch (ggml_sycl_get_best_fattn_kernel(ggml_sycl_get_device(), dst)) {
const best_fattn_kernel fk = ggml_sycl_get_best_fattn_kernel(ggml_sycl_get_device(), dst);
switch (fk) {
case BEST_FATTN_KERNEL_NONE:
GGML_ABORT("Not support Flash-Attention");
case BEST_FATTN_KERNEL_ONEDNN:
@@ -331,6 +344,7 @@ void ggml_sycl_flash_attn_ext(ggml_backend_sycl_context & ctx, ggml_tensor * dst
q->wait();
const char * kname = "???";
best_fattn_kernel kb = ggml_sycl_get_best_fattn_kernel(ctx.device, dst);
if (kb == BEST_FATTN_KERNEL_ONEDNN) kname = "ONEDNN";
if (kb == BEST_FATTN_KERNEL_MKL) kname = "MKL";
if (kb == BEST_FATTN_KERNEL_TILE) kname = "TILE";
if (kb == BEST_FATTN_KERNEL_VEC) kname = "VEC";
@@ -354,6 +368,7 @@ void ggml_sycl_flash_attn_ext(ggml_backend_sycl_context & ctx, ggml_tensor * dst
}
}
}
}
bool ggml_sycl_flash_attn_ext_supported(int device, const ggml_tensor * dst) {