sycl: make --fit respect --fit-target better (#27629)

improve the --fit algorithm to take into account the actual peak
required VRAM for a given context size on a SYCL backend.

This includes both properly accounting for how much VRAM is required
when the allocated context is fully used (which makes the reported
context drop below what it did before, but stop it OOMing) as well
as preventing some overly-conservative calculations which meant too much
VRAM was being reserved.

Tested on a Arc b70 with unsloth's qwen3.8 (Q4_K_XL), able to get 262144 context,
fully usable, with q8_0 KV and MTP and 4k ubatch size using --fit-target 1
This commit is contained in:
Nick Farrell
2026-08-29 05:00:09 -04:00
committed by GitHub
parent c9ca51c1f6
commit cc83d7b482
6 changed files with 167 additions and 41 deletions
+4 -1
View File
@@ -955,7 +955,10 @@ static size_t ggml_backend_sycl_buffer_type_get_max_size(ggml_backend_buffer_typ
}
static size_t ggml_backend_sycl_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {
size_t size = ggml_nbytes(tensor);
// Reserve the additional scratch so it's visible to the graph allocator
size_t size = tensor->op == GGML_OP_FLASH_ATTN_EXT
? ggml_sycl_flash_attn_ext_get_alloc_size(tensor)
: ggml_nbytes(tensor);
int64_t ne0 = tensor->ne[0];
if (ggml_is_quantized(tensor->type)) {