ggml-vulkan: pre-stage host weights when import is unavailable
VK_EXT_external_memory_host import fails on RADV for file-backed mmap pages, so register_host_buffer no-op'd and every MoE-expert upload paid a slow single-threaded pageable memcpy into the staging buffer each eval (~3 GB/s effective on the 35B, ~25% of PCIe bandwidth). When the import fails, fall back to a one-time copy of the region into a host-visible Vulkan buffer registered in device->pinned_memory under the mmap's address range. The existing pinned fast path then DMAs straight from that buffer each eval at full PCIe bandwidth - no per-eval CPU memcpy and no blocking sync. Costs one-time host-visible memory equal to the CPU-resident weight region. Assisted-by: opencode
This commit is contained in:
@@ -18385,12 +18385,28 @@ static bool ggml_backend_vk_register_host_buffer(void * buffer, size_t size) {
|
||||
uint8_t * p = static_cast<uint8_t *>(buffer);
|
||||
size_t remaining = size;
|
||||
bool dev_success = false;
|
||||
bool import_ok = true; // flips to false once VK_EXT_external_memory_host import fails
|
||||
while (remaining > 0) {
|
||||
size_t cur = std::min(remaining, chunk);
|
||||
vk_buffer buf = ggml_vk_buffer_from_host_ptr(device, p, cur);
|
||||
vk_buffer buf;
|
||||
if (import_ok) {
|
||||
buf = ggml_vk_buffer_from_host_ptr(device, p, cur);
|
||||
}
|
||||
if (!buf || !buf->buffer) {
|
||||
// Fallback for drivers that can't import file-backed mmap pages (e.g. RADV):
|
||||
// make a one-time copy of the region into a host-visible Vulkan buffer. The GPU
|
||||
// then DMAs straight from it every eval at full PCIe bandwidth, instead of paying
|
||||
// a slow single-threaded pageable memcpy into the staging buffer each time.
|
||||
import_ok = false;
|
||||
buf = ggml_vk_create_buffer_check(device, cur,
|
||||
vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent | vk::MemoryPropertyFlagBits::eHostCached,
|
||||
vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent);
|
||||
if (buf && buf->buffer && buf->ptr) {
|
||||
memcpy(buf->ptr, p, cur);
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
{
|
||||
std::lock_guard<std::shared_mutex> guard(device->pinned_memory_mutex);
|
||||
device->pinned_memory.emplace_back(p, cur, buf);
|
||||
|
||||
Reference in New Issue
Block a user