Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
be876204aa23bc0d8f890981fde2853555b132f9
llama.cpp/ggml/src/ggml-sycl
T
History
Titaniumtown be876204aa sycl: bind the f16 KV cache in place for the oneDNN SDPA path (#27468) 
Measured at a live KV length of 34816 (32768 depth plus one 2048 ubatch),
on Qwen3.8 27B Q4_K_S:

  per tensor         4 * 34816 * 256 * 2 B  =  71.3 MB
  staged per call    K and V, so 2x         = 142.6 MB
  traffic per call   read once, write once  = 285.2 MB
  traffic per ubatch 285.2 MB * 16 calls    =   4.56 GB

One ubatch is one ggml_cgraph submission (llama_context::process_ubatch ->
graph_compute), so that 4.56 GB is the cost of a single 2048-token prefill
chunk, and it scales with the live KV length: the first ubatch of the same run,
at seq = 2048, moves 0.27 GB.

Reproduce the two measured inputs with:

  GGML_SCHED_DEBUG=2 llama-bench -m MODEL -p 8 -n 0 -r 1 -ngl 0 \
      -fa on -ctk f16 -ctv f16 -v > nd.txt 2>&1
  grep -E 'n_layer|n_head_kv|n_embd_head_k' nd.txt
  awk '/node #  0 /{g++} g==1 && /\(FLASH_ATTN\)/{n++} END{print n+0}' nd.txt
2026-08-28 11:53:31 +03:00
..
dpct
sycl: fix multiple warnings in compiling sycl backend (#26713)
2026-08-21 08:15:40 +03:00
template-instances
sycl : add flash-attn support for head size 512 (#21654)
2026-04-09 09:36:48 +03:00
add-id.cpp
sycl : fix wrong variable check by assert (#20903)
2026-03-25 11:48:37 +02:00
add-id.hpp
[SYCL] Support gpt-oss by OPs add-id, mul_mat for mxfp4, swiglu_oai (#17826)
2025-12-15 10:35:15 +08:00
backend.hpp
sycl: fuse RMS_NORM + MUL (#26015)
2026-07-31 09:17:53 +03:00
binbcast.cpp
[SYCL] support bf16 on bin_bcast OP and unary OPs (#24838)
2026-06-22 14:09:02 +03:00
binbcast.hpp
[SYCL] fix UT fault cases: count-equal, argsort, pad OPs (#16521)
2025-10-12 21:53:35 +08:00
CMakeLists.txt
sycl(build): parallelize ocloc invocations (#25903)
2026-07-27 15:33:11 +03:00
col2im-1d.cpp
[SYCL] support op col2im_1d (#25264)
2026-07-07 11:07:46 +03:00
col2im-1d.hpp
[SYCL] support op col2im_1d (#25264)
2026-07-07 11:07:46 +03:00
common.cpp
[SYCL] rename GGML_SYCL_SUPPORT_LEVEL_ZERO (#24719)
2026-06-18 11:18:26 +03:00
common.hpp
sycl : Add DMMV ESIMD Q3_K kernel (#26251)
2026-08-13 11:32:36 +03:00
concat.cpp
sycl : enhance concat to support Q4_0, Q4_1, Q5_0, Q5_1, Q8_0 (#26800)
2026-08-13 11:24:28 +03:00
concat.hpp
SYCL: Refactor ggml_sycl_compute_forward (#11121)
2025-01-10 08:13:03 +08:00
conv2d-dw.cpp
sycl : support kernel type fp16 for conv2d_dw (#25653)
2026-07-15 10:31:10 +03:00
conv2d-dw.hpp
[SYCL] support OPs: conv_2d, conv_2d_dw, conv2d_transpose (#24600)
2026-06-18 09:40:03 +03:00
conv2d-transpose.cpp
[SYCL] support OPs: conv_2d, conv_2d_dw, conv2d_transpose (#24600)
2026-06-18 09:40:03 +03:00
conv2d-transpose.hpp
[SYCL] support OPs: conv_2d, conv_2d_dw, conv2d_transpose (#24600)
2026-06-18 09:40:03 +03:00
conv2d.cpp
[SYCL] support OPs: conv_2d, conv_2d_dw, conv2d_transpose (#24600)
2026-06-18 09:40:03 +03:00
conv2d.hpp
[SYCL] support OPs: conv_2d, conv_2d_dw, conv2d_transpose (#24600)
2026-06-18 09:40:03 +03:00
conv3d.cpp
sycl : fix the failed UT cases of conv_3d (#24900)
2026-06-25 08:27:58 +03:00
conv3d.hpp
[SYCL] Add conv_3d (#24691)
2026-06-17 17:20:01 +03:00
conv.cpp
Revert "sycl: add usage of enqueue_functions extension (#14244)" (#15910)
2025-09-12 09:15:12 +08:00
conv.hpp
SYCL: Refactor ggml_sycl_compute_forward (#11121)
2025-01-10 08:13:03 +08:00
convert.cpp
sycl : add Q2_K reordered MMVQ and ESIMD kernels (again) (#27490)
2026-08-22 10:09:26 +03:00
convert.hpp
sycl: Improve mul_mat_id memory efficiency and add BF16 fast path (#22119)
2026-04-22 20:32:56 +08:00
count-equal.cpp
[SYCL] supprt Flash Attention for fp32/fp16/Q4/Q5/Q8 (#20190)
2026-03-08 12:00:07 +08:00
count-equal.hpp
[SYCL] fix UT fault cases: count-equal, argsort, pad OPs (#16521)
2025-10-12 21:53:35 +08:00
cpy.cpp
sycl: fix thread/block count in quantized cpy kernel launches (#27160)
2026-08-17 07:32:16 +03:00
cpy.hpp
[SYCL] support the missed types in cpy (#26005)
2026-07-31 10:25:16 +03:00
cross_entropy_loss.cpp
[SYCL] support OP cross_entropy_loss, cross_entropy_loss_back (#25236)
2026-07-07 10:48:50 +03:00
cross_entropy_loss.hpp
[SYCL] support OP cross_entropy_loss, cross_entropy_loss_back (#25236)
2026-07-07 10:48:50 +03:00
cumsum.cpp
sycl: add FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, GATED_DELTA_NET (#22149)
2026-05-07 18:51:33 +03:00
cumsum.hpp
sycl: add FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, GATED_DELTA_NET (#22149)
2026-05-07 18:51:33 +03:00
dequantize.hpp
sycl : add Q2_K reordered MMVQ and ESIMD kernels (again) (#27490)
2026-08-22 10:09:26 +03:00
diag.cpp
sycl: add FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, GATED_DELTA_NET (#22149)
2026-05-07 18:51:33 +03:00
diag.hpp
sycl: add FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, GATED_DELTA_NET (#22149)
2026-05-07 18:51:33 +03:00
dmmv.cpp
sycl : add Q2_K reordered MMVQ and ESIMD kernels (again) (#27490)
2026-08-22 10:09:26 +03:00
dmmv.hpp
llama : reorganize source code + improve CMake (#8006)
2024-06-26 18:33:02 +03:00
dsv4-hc.cpp
sycl : Support DSv4 OPs: LIGHTNING_INDEXER,DSV4_HC_COMB,DSV4_HC_POST,DSV4_HC_PRE (#26568)
2026-08-07 08:22:23 +03:00
dsv4-hc.hpp
sycl : Support DSv4 OPs: LIGHTNING_INDEXER,DSV4_HC_COMB,DSV4_HC_POST,DSV4_HC_PRE (#26568)
2026-08-07 08:22:23 +03:00
element_wise.cpp
sycl: fix multiple warnings in compiling sycl backend (#26713)
2026-08-21 08:15:40 +03:00
element_wise.hpp
sycl: fuse mul_mat(gate) + mul_mat(up) + GLU for q4_K dense FFN (#26779)
2026-08-14 02:26:23 -04:00
esimd.hpp
sycl : add Q2_K reordered MMVQ and ESIMD kernels (again) (#27490)
2026-08-22 10:09:26 +03:00
fattn-buffers.cpp
SYCL: reduce allocation overhead during flash attention (#22732)
2026-05-09 09:30:39 +03:00
fattn-buffers.hpp
SYCL: reduce allocation overhead during flash attention (#22732)
2026-05-09 09:30:39 +03:00
fattn-common.hpp
[SYCL] Support Q4_1, Q5_0, Q5_1 in Flash-attention (#23812)
2026-06-01 09:53:53 +03:00
fattn-mkl.cpp
sycl: fix multiple warnings in compiling sycl backend (#26713)
2026-08-21 08:15:40 +03:00
fattn-onednn.cpp
sycl: bind the f16 KV cache in place for the oneDNN SDPA path (#27468)
2026-08-28 11:53:31 +03:00
fattn-onednn.hpp
[SYCL] Flash Attention with XMX engine via oneDNN (#25222)
2026-07-15 10:26:53 +03:00
fattn-tile.cpp
sycl : add flash-attn support for head size 512 (#21654)
2026-04-09 09:36:48 +03:00
fattn-tile.hpp
sycl : add flash-attn support for head size 512 (#21654)
2026-04-09 09:36:48 +03:00
fattn-vec.hpp
sycl : fix error Error OP FLASH_ATTN_EXT on arc770 (#26441)
2026-08-07 08:17:56 +03:00
fattn.cpp
Extended SYCL oneDNN SDPA to non-FP16 KV caches (Q4_0–Q8_0 and FP32) (#25874)
2026-08-04 10:07:23 +08:00
fattn.hpp
SYCL: add oneMKL GEMM flash attention for XMX-accelerated prompt proc… (#25025)
2026-07-31 10:43:16 +03:00
fill.cpp
sycl: add FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, GATED_DELTA_NET (#22149)
2026-05-07 18:51:33 +03:00
fill.hpp
sycl: add FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, GATED_DELTA_NET (#22149)
2026-05-07 18:51:33 +03:00
fusion.cpp
sycl: fuse mul_mat(gate) + mul_mat(up) + GLU for q4_K dense FFN (#26779)
2026-08-14 02:26:23 -04:00
fusion.hpp
sycl: fuse UNARY(silu|sigmoid|softplus) + MUL (#26411)
2026-08-13 11:41:38 +03:00
fwht.cpp
sycl: honor GGML_HINT_SRC0_IS_HADAMARD (#27298)
2026-08-18 21:21:25 +08:00
fwht.hpp
sycl: honor GGML_HINT_SRC0_IS_HADAMARD (#27298)
2026-08-18 21:21:25 +08:00
gated_delta_net.cpp
sycl: fuse the gated-delta-net state writeback cpy (#26643)
2026-08-14 09:00:36 +03:00
gated_delta_net.hpp
sycl: fuse the gated-delta-net state writeback cpy (#26643)
2026-08-14 09:00:36 +03:00
gemm.hpp
sycl: Improve mul_mat_id memory efficiency and add BF16 fast path (#22119)
2026-04-22 20:32:56 +08:00
getrows.cpp
sycl : fix get_rows Q2_K, Q4_K, Q5_K (#25656)
2026-07-15 10:32:28 +03:00
getrows.hpp
SYCL: Remove misleading ggml_sycl_op_flatten function (#12387)
2025-03-31 11:25:24 +02:00
ggml-sycl.cpp
sycl : mark tq2_0 as not supported (#27660)
2026-08-25 09:04:58 +03:00
gla.cpp
Revert "sycl: add usage of enqueue_functions extension (#14244)" (#15910)
2025-09-12 09:15:12 +08:00
gla.hpp
SYCL: Add gated linear attention kernel (#11175)
2025-01-15 11:20:17 +08:00
im2col.cpp
sycl: fix multiple warnings in compiling sycl backend (#26713)
2026-08-21 08:15:40 +03:00
im2col.hpp
[SYCL] Add OP im2col_3d (#22903)
2026-05-11 08:01:47 +03:00
lightning-indexer.cpp
sycl : Support DSv4 OPs: LIGHTNING_INDEXER,DSV4_HC_COMB,DSV4_HC_POST,DSV4_HC_PRE (#26568)
2026-08-07 08:22:23 +03:00
lightning-indexer.hpp
sycl : Support DSv4 OPs: LIGHTNING_INDEXER,DSV4_HC_COMB,DSV4_HC_POST,DSV4_HC_PRE (#26568)
2026-08-07 08:22:23 +03:00
mmq.cpp
Revert "sycl: add usage of enqueue_functions extension (#14244)" (#15910)
2025-09-12 09:15:12 +08:00
mmq.hpp
llama : reorganize source code + improve CMake (#8006)
2024-06-26 18:33:02 +03:00
mmvq.cpp
sycl : mark tq2_0 as not supported (#27660)
2026-08-25 09:04:58 +03:00
mmvq.hpp
sycl: fuse mul_mat(gate) + mul_mat(up) + GLU for q4_K dense FFN (#26779)
2026-08-14 02:26:23 -04:00
norm.cpp
sycl: fix multiple warnings in compiling sycl backend (#26713)
2026-08-21 08:15:40 +03:00
norm.hpp
sycl: fuse RMS_NORM + MUL (#26015)
2026-07-31 09:17:53 +03:00
opt-step.cpp
[SYCL] support OP OPT_STEP_ADAMW, OPT_STEP_SGD (#25268)
2026-08-17 07:31:29 +03:00
opt-step.hpp
[SYCL] support OP OPT_STEP_ADAMW, OPT_STEP_SGD (#25268)
2026-08-17 07:31:29 +03:00
outprod.cpp
sycl : support MUL_MAT and OUT_PROD with Q1_0 (#24721)
2026-06-18 11:17:37 +03:00
outprod.hpp
SYCL: Refactor ggml_sycl_compute_forward (#11121)
2025-01-10 08:13:03 +08:00
pad_reflect_1d.cpp
refactor pad_reflect_1d to make the UT case pass (#17204)
2025-11-28 08:50:56 +08:00
pad_reflect_1d.hpp
refactor pad_reflect_1d to make the UT case pass (#17204)
2025-11-28 08:50:56 +08:00
pad.cpp
sycl: support non-contiguous input in PAD op (#22148)
2026-05-09 08:05:22 +08:00
pad.hpp
[SYCL] fix UT fault cases: count-equal, argsort, pad OPs (#16521)
2025-10-12 21:53:35 +08:00
pool.cpp
[SYCL] add to support pool_1d, move pool_1d/2d code to pool.cpp/hpp (#24584)
2026-06-15 10:01:07 +03:00
pool.hpp
[SYCL] add to support pool_1d, move pool_1d/2d code to pool.cpp/hpp (#24584)
2026-06-15 10:01:07 +03:00
presets.hpp
sycl: *glu flat path (#26354)
2026-08-07 08:24:40 +03:00
quantize.hpp
sycl: refactor quantization to q8_1 (#14815)
2025-07-28 11:05:53 +01:00
quants.hpp
sycl : add Q2_K reordered MMVQ and ESIMD kernels (again) (#27490)
2026-08-22 10:09:26 +03:00
repeat_back.cpp
SYCL: optimized repeat_back kernel (3× fewer asm instructions, 2× faster)Feature/sycl repeat back opt (#16869)
2025-11-03 09:35:33 +08:00
repeat_back.hpp
sycl: add REPEAT_BACK operation support (#16734)
2025-10-27 09:19:50 +08:00
roll.cpp
sycl: add ROLL operation support (#16665)
2025-10-27 09:20:24 +08:00
roll.hpp
sycl: add ROLL operation support (#16665)
2025-10-27 09:20:24 +08:00
rope.cpp
ggml: support ggml_rope_set_offset on opencl, sycl, wgpu, hexagon (#27345)
2026-08-21 00:36:57 +02:00
rope.hpp
fix op rope, add rope_back (#20293)
2026-03-11 09:53:34 +08:00
set_rows.cpp
sycl : mark tq2_0 as not supported (#27660)
2026-08-25 09:04:58 +03:00
set_rows.hpp
SYCL: Initial set_rows kernel implementation (#14562)
2025-07-10 09:29:38 +01:00
set.cpp
SYCL SET operator optimized for F32 tensors (#16350)
2025-10-17 10:36:40 +08:00
set.hpp
SYCL SET operator optimized for F32 tensors (#16350)
2025-10-17 10:36:40 +08:00
softmax.cpp
sycl : clamp softmax input to avoid underflow (#24941)
2026-06-26 15:02:42 +08:00
softmax.hpp
[SYCL] refactor soft_max, add soft_max_back (#16472)
2025-10-09 10:25:11 +03:00
solve_tri.cpp
sycl: add FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, GATED_DELTA_NET (#22149)
2026-05-07 18:51:33 +03:00
solve_tri.hpp
sycl: add FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, GATED_DELTA_NET (#22149)
2026-05-07 18:51:33 +03:00
ssm_conv.cpp
sycl: coalesce the ssm_conv window loads (#26612)
2026-08-07 21:09:32 +03:00
ssm_conv.hpp
sycl: add SSM_CONV operation support (#16800)
2025-10-28 09:50:33 +08:00
ssm_scan.cpp
ggml : recurrent state rollback for ggml_ssm_scan (#26623)
2026-08-14 17:20:40 +03:00
ssm_scan.hpp
sycl: add FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, GATED_DELTA_NET (#22149)
2026-05-07 18:51:33 +03:00
sycl_hw.cpp
[SYCL] Optimize Q4_0 mul_mat for Arc770, add scripts (#22291)
2026-04-25 09:20:14 +03:00
sycl_hw.hpp
[SYCL] Optimize Q4_0 mul_mat for Arc770, add scripts (#22291)
2026-04-25 09:20:14 +03:00
topk-moe.cpp
sycl: add fused top-k MoE (#25217)
2026-07-13 09:56:41 +03:00
topk-moe.hpp
sycl: add fused top-k MoE (#25217)
2026-07-13 09:56:41 +03:00
tsembd.cpp
ggml : fix padding in timestep embedding kernels (#15932)
2025-09-16 15:25:57 +02:00
tsembd.hpp
SYCL: Refactor ggml_sycl_compute_forward (#11121)
2025-01-10 08:13:03 +08:00
type.hpp
sycl : support nvfp4 type in mul_mat (#21227)
2026-04-01 13:54:15 +03:00
upscale.cpp
sycl: disable Q1_0 in backend and cleanup unused variables (#21807)
2026-04-13 09:44:58 +08:00
upscale.hpp
[SYCL] ehance UPSCALE to support all UT cases (#20637)
2026-03-17 10:01:52 +08:00
vecdotq.hpp
sycl : add Q2_K reordered MMVQ and ESIMD kernels (again) (#27490)
2026-08-22 10:09:26 +03:00
wkv.cpp
Remove support for Nvidia & AMD GPU, because the oneAPI plugin for Nvidia & AMD GPU is unavailable: download/installation channels are out of work. (#19246)
2026-02-02 21:06:21 +08:00
wkv.hpp
llama: Add support for RWKV v7 architecture (#12412)
2025-03-18 07:27:50 +08:00