From d7e32623cdc3ba3527248936a2ba015fb4b19e3a Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Tue, 21 Jul 2026 17:17:23 +0200 Subject: [PATCH] docs(readme): document Vulkan behavior of the MoE-offload flags Pinning helps on Vulkan too, but GGML_SCHED_PREFETCH_EXPERTS regresses there (second backend shares one device queue, no overlap). Add an RX 580 Polaris benchmark and the context-dependent --n-cpu-moe guidance. Assisted-by: Claude --- README.md | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 217eb2eb8..9b8b6aea5 100644 --- a/README.md +++ b/README.md @@ -20,8 +20,8 @@ environment variables, and produce **token-identical** output to mainline. | Env var | What it does | | --- | --- | -| `GGML_CUDA_REGISTER_HOST=1` | Page-locks (pins) the mmap'd CPU expert weights so host→device copies go straight over DMA instead of through the driver's hidden bounce buffer (~6–7 → ~20 GB/s). | -| `GGML_SCHED_PREFETCH_EXPERTS=1` | Prefetches each layer's experts on a second CUDA stream, so the weight uploads overlap compute instead of stalling the GPU. | +| `GGML_CUDA_REGISTER_HOST=1` | Page-locks (pins) the mmap'd CPU expert weights so host->device copies go straight over DMA instead of through the driver's hidden bounce buffer (~6-7 -> ~20 GB/s). Works on CUDA and Vulkan (also honored as `GGML_VK_REGISTER_HOST`). Note: it is a presence check, so `=0` still enables it. | +| `GGML_SCHED_PREFETCH_EXPERTS=1` | Prefetches each layer's experts on a second stream, so the weight uploads overlap compute instead of stalling the GPU. **CUDA only** - on the Vulkan backend the second backend instance shares one device queue, giving no overlap, so this regresses (see Vulkan note below). Leave it off on Vulkan. | ### Benchmark @@ -40,6 +40,38 @@ Result: **~1143 → ~1880 t/s** prefill (**+64%**) — same GPU, same settings, Branches: [`fable5/host-register`](https://github.com/thecodacus/llama.cpp/tree/fable5/host-register) (pinning only) · [`fable5/prefetch-experts`](https://github.com/thecodacus/llama.cpp/tree/fable5/prefetch-experts) (both — this branch). +### Vulkan (older AMD, e.g. RX 580 / Polaris) + +On the Vulkan backend the two optimizations behave differently from CUDA: + +- **Pinning (`GGML_CUDA_REGISTER_HOST=1`) is the win.** It brings the H2D expert uploads up to full PCIe DMA rate. +- **`GGML_SCHED_PREFETCH_EXPERTS=1` regresses here - do not use it.** The prefetch runs through a second backend instance that shares the same Vulkan device queue, so uploads do not actually overlap compute; it only adds overhead. +- **Tune `--n-cpu-moe` to context length.** Keeping some expert layers resident in spare VRAM helps short prompts, but at long context the growing KV cache needs that VRAM back, so residency backfires. Use a low value (more resident) for short prompts and a high value (all experts on host) for long context. +- Keep flash attention on (`-fa 1`, the default) and do not quantize the KV cache on this hardware (`-ctk/-ctv q8_0` is slower - the dequant overhead in the scalar FA kernel outweighs the bandwidth saving). + +Measured on an **RX 580 8GB** (Polaris, PCIe 3.0 x16, no fp16 / no matrix cores) with **Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`: + +```bash +# baseline (pinning off): +./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 99 -p 2048 -n 0 -r 2 -b 2048 -ub 2048 + +# pinned, all experts on host: +GGML_CUDA_REGISTER_HOST=1 \ +./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 99 -p 2048 -n 0 -r 2 -b 2048 -ub 2048 + +# pinned + a few expert layers kept resident (best for short prompts): +GGML_CUDA_REGISTER_HOST=1 \ +./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 28 -p 2048 -n 0 -r 2 -b 2048 -ub 2048 +``` + +| Config | pp2048 (t/s) | +| --- | ---: | +| baseline (unpinned) | ~252 | +| pinned, `-ncmoe 99` | ~294 | +| pinned, `-ncmoe 28` | ~308 | + +Result: **~252 -> ~308 t/s** prefill (**+22%**), token-identical. At long context the bottleneck shifts to attention compute (GPU-bound), where these MoE-transfer optimizations no longer apply; prefer a high `--n-cpu-moe` there for VRAM headroom. + ## Recent API changes - [Changelog for `libllama` API](https://github.com/ggml-org/llama.cpp/issues/9289)