ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc
Add an optional backend interface method free_scratch (with a public ggml_backend_free_scratch wrapper) that frees transient/scratch device memory a backend holds outside of any allocated buffer, keeping the backend usable - the scratch is reallocated lazily on the next compute. Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers and reset their sizes, so an idle/cold model does not hold the vision or matmul compute preallocations in VRAM. All other backends leave the hook null (no-op). Assisted-by: Claude
This commit is contained in:
@@ -104,6 +104,11 @@ extern "C" {
|
||||
GGML_API enum ggml_status ggml_backend_graph_compute (ggml_backend_t backend, struct ggml_cgraph * cgraph);
|
||||
GGML_API enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph);
|
||||
|
||||
// Free transient/scratch device memory the backend holds outside of any allocated buffer
|
||||
// (compute preallocations, staging buffers). No-op if the backend does not implement it.
|
||||
// The backend remains usable; scratch is reallocated lazily on the next compute.
|
||||
GGML_API void ggml_backend_free_scratch(ggml_backend_t backend);
|
||||
|
||||
// NOTE: will be removed, use device version instead
|
||||
GGML_API bool ggml_backend_supports_op(ggml_backend_t backend, const struct ggml_tensor * op);
|
||||
GGML_API bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft);
|
||||
|
||||
Reference in New Issue
Block a user