From fac3a894fc04901c3e5abaff457e024c6b52e04f Mon Sep 17 00:00:00 2001 From: FamousMonsterr Date: Tue, 3 Nov 2026 10:53:04 +0300 Subject: [PATCH] vulkan: skip maxStorageBufferRange pre-check for non-compute tensors (#31114) * vulkan: skip maxStorageBufferRange pre-check for non-compute tensors Pre-allocated leaf tensors and views (GGML_OP_NONE / GGML_OP_VIEW) are never executed by the backend, but were still rejected by the descriptor-size pre-check in ggml_backend_vk_device_supports_op when larger than maxStorageBufferRange on devices without shader_64b_indexing. The scheduler then aborts with 'pre-allocated tensor (...) in a buffer (Vulkan0) that cannot run the operation (NONE)' - e.g. the MoE expert cache banks (llama-moe-cache) of a 256-expert model with --moe-cache-mib on GCN GPUs, where a whole bank (2.4 GB) exceeds the limit while the consuming ops only access small per-step views. Skip the size pre-check for non-compute ops (NONE and VIEW); the buffer type allocation limit still bounds the actual allocation, and compute ops keep validating their sources. Tested on AMD Radeon Pro VII (gfx906, no shader_64b_indexing): - before: GGML_ABORT 'pre-allocated tensor (moe_cache.0.blk.0.ffn_down_exps.weight) ... cannot run the operation (NONE)' with llama-cli -ngl 99 --cpu-moe --moe-cache-mib 6144 (Cyber-Tiel-Coder-35B-A3B, qwen35moe, 256 experts per layer) - after: loads and generates at 8.1 t/s (vs 0.6 t/s without the cache) - regression: -ncmoe 16 path unchanged (19.2 t/s), qwen3moe/qwen35moe without cache unchanged Fixes #30697 * vulkan: use ggml_op_is_empty for the non-compute size-check skip --------- Co-authored-by: FamousMonsterr --- ggml/src/ggml-vulkan/ggml-vulkan.cpp | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp index 2e886c9be..89f0470cb 100644 --- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp +++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp @@ -15550,13 +15550,14 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm } return true; }; - // reject any tensors larger than the max buffer size + // non-compute ops are never executed here; their size is bounded by the buffer allocation limit + const bool non_compute = ggml_op_is_empty(op->op); for (int i = 0; i < GGML_MAX_SRC; i++) { - if (op->src[i] && !tensor_size_supported(ggml_nbytes(op->src[i]))) { + if (op->src[i] && !non_compute && !tensor_size_supported(ggml_nbytes(op->src[i]))) { return false; } } - if (!tensor_size_supported(ggml_nbytes(op))) { + if (!non_compute && !tensor_size_supported(ggml_nbytes(op))) { return false; }