diff --git a/README.md b/README.md index 9b8b6aea5..53b60fd86 100644 --- a/README.md +++ b/README.md @@ -42,35 +42,46 @@ Branches: [`fable5/host-register`](https://github.com/thecodacus/llama.cpp/tree/ ### Vulkan (older AMD, e.g. RX 580 / Polaris) -On the Vulkan backend the two optimizations behave differently from CUDA: +On the Vulkan backend the CUDA-oriented flags above behave differently, and this fork adds a +Polaris-specific flash-attention fix. Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16, +no fp16, no matrix cores) with **Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`: -- **Pinning (`GGML_CUDA_REGISTER_HOST=1`) is the win.** It brings the H2D expert uploads up to full PCIe DMA rate. -- **`GGML_SCHED_PREFETCH_EXPERTS=1` regresses here - do not use it.** The prefetch runs through a second backend instance that shares the same Vulkan device queue, so uploads do not actually overlap compute; it only adds overhead. -- **Tune `--n-cpu-moe` to context length.** Keeping some expert layers resident in spare VRAM helps short prompts, but at long context the growing KV cache needs that VRAM back, so residency backfires. Use a low value (more resident) for short prompts and a high value (all experts on host) for long context. -- Keep flash attention on (`-fa 1`, the default) and do not quantize the KV cache on this hardware (`-ctk/-ctv q8_0` is slower - the dequant overhead in the scalar FA kernel outweighs the bandwidth saving). +- **Flash-attention `mask_opt` is now enabled for GCN large head sizes (this fork's own change).** + Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips + fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on + a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B + (head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical. +- **For a long-running server, load with `--no-mmap`, not pinning.** `GGML_CUDA_REGISTER_HOST=1` + (pinning) gives ~+17% in an isolated `llama-bench` run, but in a server the RADV host-pointer + import fails and the fallback pre-stage buffer allocation fails for large / co-resident models, so + it silently reverts to slow staging (and can trip warnings/OOM). `--no-mmap` (weights in RAM) is + both faster and clean there. Pinning is still fine for one-off `llama-bench` numbers. +- **`GGML_SCHED_PREFETCH_EXPERTS=1` regresses - do not use it** on Vulkan (its second backend shares + one device queue, so uploads never overlap compute). +- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp). +- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short + prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that + VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`). -Measured on an **RX 580 8GB** (Polaris, PCIe 3.0 x16, no fp16 / no matrix cores) with **Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`: - -```bash -# baseline (pinning off): -./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 99 -p 2048 -n 0 -r 2 -b 2048 -ub 2048 - -# pinned, all experts on host: -GGML_CUDA_REGISTER_HOST=1 \ -./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 99 -p 2048 -n 0 -r 2 -b 2048 -ub 2048 - -# pinned + a few expert layers kept resident (best for short prompts): -GGML_CUDA_REGISTER_HOST=1 \ -./build/bin/llama-bench -m MODEL -ngl 99 -ncmoe 28 -p 2048 -n 0 -r 2 -b 2048 -ub 2048 -``` +Prefill throughput (isolated `llama-bench`, pinned unless noted): | Config | pp2048 (t/s) | | --- | ---: | -| baseline (unpinned) | ~252 | -| pinned, `-ncmoe 99` | ~294 | -| pinned, `-ncmoe 28` | ~308 | +| baseline, unpinned, `ncmoe 40` | ~252 | +| pinned, `ncmoe 40` | ~294 | +| pinned, `ncmoe 28` | ~308 | +| server default (`--no-mmap`, `ncmoe 40`) | ~285 | +| + `mask_opt`, @ 32k context | **+12%** | -Result: **~252 -> ~308 t/s** prefill (**+22%**), token-identical. At long context the bottleneck shifts to attention compute (GPU-bound), where these MoE-transfer optimizations no longer apply; prefer a high `--n-cpu-moe` there for VRAM headroom. +**Recommended RX 580 / Polaris serving command** (per model): + +```bash +llama-server -hf : --no-mmap -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 +``` + +Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high +for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so +`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path. ## Recent API changes