ggml-vulkan: pre-stage host weights when import is unavailable

VK_EXT_external_memory_host import fails on RADV for file-backed mmap pages,
so register_host_buffer no-op'd and every MoE-expert upload paid a slow
single-threaded pageable memcpy into the staging buffer each eval (~3 GB/s
effective on the 35B, ~25% of PCIe bandwidth).

When the import fails, fall back to a one-time copy of the region into a
host-visible Vulkan buffer registered in device->pinned_memory under the
mmap's address range. The existing pinned fast path then DMAs straight from
that buffer each eval at full PCIe bandwidth - no per-eval CPU memcpy and no
blocking sync. Costs one-time host-visible memory equal to the CPU-resident
weight region.

Assisted-by: opencode
This commit is contained in:
2026-07-22 21:04:20 +02:00
parent 42e52045dc
commit 3972da9cc9
+17 -1
View File
@@ -18385,12 +18385,28 @@ static bool ggml_backend_vk_register_host_buffer(void * buffer, size_t size) {
uint8_t * p = static_cast<uint8_t *>(buffer);
size_t remaining = size;
bool dev_success = false;
bool import_ok = true; // flips to false once VK_EXT_external_memory_host import fails
while (remaining > 0) {
size_t cur = std::min(remaining, chunk);
vk_buffer buf = ggml_vk_buffer_from_host_ptr(device, p, cur);
vk_buffer buf;
if (import_ok) {
buf = ggml_vk_buffer_from_host_ptr(device, p, cur);
}
if (!buf || !buf->buffer) {
// Fallback for drivers that can't import file-backed mmap pages (e.g. RADV):
// make a one-time copy of the region into a host-visible Vulkan buffer. The GPU
// then DMAs straight from it every eval at full PCIe bandwidth, instead of paying
// a slow single-threaded pageable memcpy into the staging buffer each time.
import_ok = false;
buf = ggml_vk_create_buffer_check(device, cur,
vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent | vk::MemoryPropertyFlagBits::eHostCached,
vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent);
if (buf && buf->buffer && buf->ptr) {
memcpy(buf->ptr, p, cur);
} else {
break;
}
}
{
std::lock_guard<std::shared_mutex> guard(device->pinned_memory_mutex);
device->pinned_memory.emplace_back(p, cur, buf);