mtmd: release/restore the encoder weights from VRAM on demand

Add clip_release_device/clip_restore_device (and mtmd_release_device/
mtmd_restore_device wrappers over the vision + audio contexts) that free
the multimodal encoder's device weight buffer to a read-only host shadow
and rebuild it on demand, using the same shadow/free/reallocate pattern
as llama_model weights. No-op for a CPU-backed encoder. This lets the
server drop the ~hundreds-of-MiB vision encoder from VRAM when it is not
encoding an image.

Assisted-by: Claude
This commit is contained in:
2026-09-09 00:39:21 +02:00
parent c49d19cc64
commit 6564834854
4 changed files with 105 additions and 0 deletions
+18
View File
@@ -1116,6 +1116,24 @@ std::vector<std::vector<const mtmd_bitmap *>> mtmd_group_mergeable_bitmaps(std::
return output;
}
void mtmd_release_device(mtmd_context * ctx) {
if (ctx == nullptr) {
return;
}
if (ctx->ctx_v) { clip_release_device(ctx->ctx_v); }
if (ctx->ctx_a) { clip_release_device(ctx->ctx_a); }
}
bool mtmd_restore_device(mtmd_context * ctx) {
if (ctx == nullptr) {
return true;
}
bool ok = true;
if (ctx->ctx_v) { ok = clip_restore_device(ctx->ctx_v) && ok; }
if (ctx->ctx_a) { ok = clip_restore_device(ctx->ctx_a) && ok; }
return ok;
}
struct mtmd_tokenizer {
const mtmd_context * ctx;