ggml-vulkan: skip the concat transpose fast path below one tile
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of 40 layers) build_conv_state emits a transposed concat per recurrent layer, so the fast path runs 30 times per decoded token. At one token the transposed source is a single column, so there is no uncoalesced stride left to fix, but the path still costs two dispatches and an unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes the command stream. Measured -17% tg256 on RX 580; the barrier, not the half-empty transpose dispatch, is nearly all of it. Require the transposed source to be at least one TILE_DIM wide. That source is qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the fast path, decode and small speculative drafts take the generic one. pp8192 +11% retained, tg256 back to parity, generation byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
@@ -13579,6 +13579,8 @@ static void ggml_vk_concat(ggml_backend_vk_context * ctx, vk_context& subctx, co
|
||||
const uint32_t ts = ggml_type_size(s->type);
|
||||
return s->nb[1] == ts // dim1 innermost
|
||||
&& s->nb[0] == (size_t) s->ne[1] * ts // consistent 2D transpose
|
||||
&& s->ne[0] >= 32 // at least one full transpose tile (TILE_DIM); at 1 token the
|
||||
// tiled path is pure overhead, the generic copy is faster
|
||||
&& s->ne[2] == 1 && s->ne[3] == 1;
|
||||
};
|
||||
const uint32_t dst_ts = ggml_type_size(dst->type);
|
||||
|
||||
Reference in New Issue
Block a user