From c233ce9b512c1d9694747cd0d83d76ccdfd31ab5 Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Thu, 10 Sep 2026 17:06:01 +0200 Subject: [PATCH] ggml-vulkan: skip the concat transpose fast path below one tile The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of 40 layers) build_conv_state emits a transposed concat per recurrent layer, so the fast path runs 30 times per decoded token. At one token the transposed source is a single column, so there is no uncoalesced stride left to fix, but the path still costs two dispatches and an unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes the command stream. Measured -17% tg256 on RX 580; the barrier, not the half-empty transpose dispatch, is nearly all of it. Require the transposed source to be at least one TILE_DIM wide. That source is qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the fast path, decode and small speculative drafts take the generic one. pp8192 +11% retained, tg256 back to parity, generation byte-identical. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ --- ggml/src/ggml-vulkan/ggml-vulkan.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp index 74c339930..2b22efdfd 100644 --- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp +++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp @@ -13579,6 +13579,8 @@ static void ggml_vk_concat(ggml_backend_vk_context * ctx, vk_context& subctx, co const uint32_t ts = ggml_type_size(s->type); return s->nb[1] == ts // dim1 innermost && s->nb[0] == (size_t) s->ne[1] * ts // consistent 2D transpose + && s->ne[0] >= 32 // at least one full transpose tile (TILE_DIM); at 1 token the + // tiled path is pure overhead, the generic copy is faster && s->ne[2] == 1 && s->ne[3] == 1; }; const uint32_t dst_ts = ggml_type_size(dst->type);