From f9d41c711f0ba971e7d29a5c035b0e9d5fced4e0 Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Wed, 9 Sep 2026 00:41:59 +0200 Subject: [PATCH] docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends --- README.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/README.md b/README.md index aae3bcd35..c2b3e7b95 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,82 @@ +## This fork - Polaris / GCN tuning for large MoE models + +Changes and measurements for running large MoE models with their experts offloaded to system RAM +(`--n-cpu-moe`) on an old GCN card. The two code changes below are auto-on, need no flag, and are +**token-identical** to mainline. Everything else here is tuning guidance. + +Findings on an **RX 580 8GB** (Polaris / GCN, PCIe 3.0 x16, no fp16, no matrix cores) with +**Qwen3.5-35B-A3B Q4_K_M**, `-b 2048 -ub 2048`: + +- **Flash-attention `mask_opt` is enabled for GCN large head sizes (this fork's own change).** + Upstream disables it on GCN; it is a **lossless** win in high-context prefill - it skips + fully-masked causal blocks and the per-block mask add on fully-visible ones, which is real work on + a card whose attention is compute-bound (no matrix cores). Auto-on, no flag. On Qwen3.5-35B + (head_dim 256): pp2048 **+8% @ 16k, +12% @ 32k**, growing with depth; perplexity bit-identical. +- **`-b 2048 -ub 2048` is the biggest prefill lever** (the default `-ub 512` roughly halves pp). +- **Tune `--n-cpu-moe` to context length.** Keep some expert layers resident in spare VRAM for short + prompts (e.g. `ncmoe 28` on the 35B, ~+5% over all-host); at long context the KV cache needs that + VRAM, so raise it (`ncmoe 40`, all experts on host). Keep flash attention on (`-fa 1`). + +**Recommended RX 580 / Polaris serving command** (per model): + +```bash +llama-server -hf : -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 +``` + +Lower `--n-cpu-moe` (e.g. 28) if the model plus your context budget leave spare VRAM; keep it high +for long-context / agentic use. At long context the bottleneck is attention compute (GPU-bound), so +`mask_opt` (above) is where the remaining prefill wins come from, not the MoE-transfer path. + +### Very large MoE (experts bigger than the GTT limit) + +Measured on the same RX 580 with **Laguna-S-2.1 118B IQ2_M** (48 layers, 256 experts, 10 used, +experts 30.7 GiB of a 34.7 GiB file, all host-resident at `--n-cpu-moe 48`). The advice above changes +in this regime: + +- **`--no-mmap` stops being an option.** Its pinned host buffer is charged against the amdgpu GTT + limit (~31.4 GiB here, about half of system RAM). At 30.7 GiB of experts the model no longer + loads, and the allocation spike can OOM the box. Use mmap and accept the staging copy. +- **The routing-ids readback is pure overhead at prefill batch sizes** and this fork now skips it. + To decide which experts to upload, the scheduler read the ids back from the device that had just + produced them, forcing a full pipeline flush once per MoE layer per eval. With 2048 tokens x 10 + experts over 256 experts every expert is used anyway. Skipping it is exact - `mul_mat_id` only + reads the rows the ids point at. Auto-on above `4 * n_expert` ids; decode keeps the old path. + **pp2048 +4.7% @ 32k depth, +0.9% @ 16k, tg +3%.** +- **Where the time actually goes** (`GGML_VK_PERF_LOGGER=1`, depth 0, 22.65 s per 2048-token eval, + 17.18 s of it GPU-busy so ~24% is H2D stall): expert `MUL_MAT_ID` **52%**, attention projections + **24%**, `FLASH_ATTN_EXT` **16%**, everything else 8%. The expert matmuls run at 1686-2139 GFLOP/s + while dense `MUL_MAT` q5_K/q6_K in the same graph reaches 3096-3691 - **the single largest + remaining opportunity on this hardware is closing that gap**, not the transfer path. +- **Interleaved SWA keeps its own small KV cache**, so a sliding-window layer costs the same at any + depth (`n_kv` pinned at `n_swa * n_seq_max + n_ubatch`). On this model 36 of 48 layers are O(1) in + depth and the entire high-context slowdown comes from the 12 full-attention layers. +- **`--parallel 1`** is worth setting for a solo large model: the server otherwise auto-selects 4 + slots, and the SWA cache is sized `n_swa * n_seq_max + n_ubatch`, so 4 slots cost 4096 cells + instead of 2560. Measured **223 MiB of VRAM freed** at 64k context. + +Dead ends measured on this hardware, recorded so they are not retried: + +| Change | Result | +| --- | ---: | +| flash-attn `shmem_staging` enabled for GCN | **-6.7% @ 16k, -7.4% @ 32k** | +| `-b 4096 -ub 4096` (to amortize the fixed per-eval expert upload) | flat (-1%) | +| `--n-cpu-moe` 44 instead of 48 | +1.5%, but does not fit at 64k ctx | +| `mask_opt` gate relaxed below head_dim 256 | -18.5% @ 16k | + +`shmem_staging` looks like a certain win (without it each rowgroup re-reads the whole K/V block +through a 16 KiB L1) but the `kvsh` stride of `D/4+1` dwords is 4 mod 32, which costs an 8-way LDS +bank conflict on wave64 - that `+1` padding is tuned for warp32. `-ub 4096` fails because halving the +number of expert uploads is exactly cancelled by intra-ubatch attention growing quadratically. + +**Serving note that outweighs all of the above.** With a model this large, anything that restarts the +process is far more expensive than any kernel win: the server's prompt cache is RAM-only with no disk +backing, so a restart forces a full re-prefill of the conversation. If a model swapper can evict this +model to run a small helper model (chat-title generation and the like), fix that first - keeping the +process alive across a swap took a repeat turn from a 21,960 ms prefill down to 225 ms. + ## Quick start A few options to get `llama.cpp` installed on your machine: