ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc

Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.

Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).

Assisted-by: Claude
This commit is contained in:
2026-07-26 14:55:50 +02:00
parent c2c8d377cb
commit 994fd757fc
4 changed files with 49 additions and 0 deletions
+5
View File
@@ -104,6 +104,11 @@ extern "C" {
GGML_API enum ggml_status ggml_backend_graph_compute (ggml_backend_t backend, struct ggml_cgraph * cgraph);
GGML_API enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph);
// Free transient/scratch device memory the backend holds outside of any allocated buffer
// (compute preallocations, staging buffers). No-op if the backend does not implement it.
// The backend remains usable; scratch is reallocated lazily on the next compute.
GGML_API void ggml_backend_free_scratch(ggml_backend_t backend);
// NOTE: will be removed, use device version instead
GGML_API bool ggml_backend_supports_op(ggml_backend_t backend, const struct ggml_tensor * op);
GGML_API bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft);