vulkan: skip maxStorageBufferRange pre-check for non-compute tensors (#31114)
* vulkan: skip maxStorageBufferRange pre-check for non-compute tensors Pre-allocated leaf tensors and views (GGML_OP_NONE / GGML_OP_VIEW) are never executed by the backend, but were still rejected by the descriptor-size pre-check in ggml_backend_vk_device_supports_op when larger than maxStorageBufferRange on devices without shader_64b_indexing. The scheduler then aborts with 'pre-allocated tensor (...) in a buffer (Vulkan0) that cannot run the operation (NONE)' - e.g. the MoE expert cache banks (llama-moe-cache) of a 256-expert model with --moe-cache-mib on GCN GPUs, where a whole bank (2.4 GB) exceeds the limit while the consuming ops only access small per-step views. Skip the size pre-check for non-compute ops (NONE and VIEW); the buffer type allocation limit still bounds the actual allocation, and compute ops keep validating their sources. Tested on AMD Radeon Pro VII (gfx906, no shader_64b_indexing): - before: GGML_ABORT 'pre-allocated tensor (moe_cache.0.blk.0.ffn_down_exps.weight) ... cannot run the operation (NONE)' with llama-cli -ngl 99 --cpu-moe --moe-cache-mib 6144 (Cyber-Tiel-Coder-35B-A3B, qwen35moe, 256 experts per layer) - after: loads and generates at 8.1 t/s (vs 0.6 t/s without the cache) - regression: -ncmoe 16 path unchanged (19.2 t/s), qwen3moe/qwen35moe without cache unchanged Fixes #30697 * vulkan: use ggml_op_is_empty for the non-compute size-check skip --------- Co-authored-by: FamousMonsterr <FamousMonsterr@users.noreply.github.com>
This commit is contained in:
1 parent
b74298c109
commit
fac3a894fc
1 file changed
+4
-3
@@ -15550,13 +15550,14 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
|
||||
}
|
||||
return true;
|
||||
};
|
||||
// reject any tensors larger than the max buffer size
|
||||
// non-compute ops are never executed here; their size is bounded by the buffer allocation limit
|
||||
const bool non_compute = ggml_op_is_empty(op->op);
|
||||
for (int i = 0; i < GGML_MAX_SRC; i++) {
|
||||
if (op->src[i] && !tensor_size_supported(ggml_nbytes(op->src[i]))) {
|
||||
if (op->src[i] && !non_compute && !tensor_size_supported(ggml_nbytes(op->src[i]))) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
if (!tensor_size_supported(ggml_nbytes(op))) {
|
||||
if (!non_compute && !tensor_size_supported(ggml_nbytes(op))) {
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user