diff --git a/scripts/rx580-bench/README.md b/scripts/rx580-bench/README.md index ecdb33a01..3833caaf8 100644 --- a/scripts/rx580-bench/README.md +++ b/scripts/rx580-bench/README.md @@ -17,8 +17,12 @@ Two things, deliberately: Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k and 32k tokens. 2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random - tokens, `pp2048`, `pp8192`, `tg32`. This is what prior work on this box - recorded, so it is the continuity metric. + tokens: prefill `pp2048`, `pp8192` at depth 0, and decode `tg256@d4096` + (256 generated tokens on top of a 4096-token context, `-p 0 -n 256 + -d 4096`). The prefill cells are the continuity metric with prior work on + this box. Decode used to be `tg32` at the default depth; that is not a valid + metric here (see "Reading the numbers"), and old `tg32` rows in + `results.txt` are history only. They do not measure the same thing and they do not agree. `llama-bench` pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a @@ -100,9 +104,14 @@ Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16, The `llama-bench` equivalent of that config is: ``` --t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 32 -r 3 +-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 0 -r 3 +-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 0 -n 256 -d 4096 -r 3 ``` +`lbsweep.sh` runs these as two invocations (prefill, then decode). Override +with `LB_P` (prefill sizes), `LB_N` / `LB_D` (decode tokens / depth; `LB_N=0` +skips decode) and `LB_R` (repetitions). + ## The corpus slices are constants, not recalibrated `pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this @@ -128,15 +137,22 @@ or the corpus changes. * `prompt_n` is the token count actually prefilled. It must equal the target. * `prompt_ms` is wall-clock prefill time. * `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better. -* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0. -* `tg32` is token generation at DEFAULT depth (n_ctx 32). **It is NOT a valid - decode metric on this box.** At that size decode is launch/barrier bound, not - compute bound, and measures a regime serving never reaches: it read -36% - between two upstream commits that are at parity in real use (the cause was - the view-alias fix in ggml_vk_graph_optimize). For decode use **tg256 with an - explicit depth**, e.g. `-p 0 -n 256 -d 2048`. tg is also NOT monotonic in - n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 - - so only ever compare same-depth cells. +* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0. A depth + suffix (`pp2048@d16384`) means the prompt is processed on top of a context + of that many tokens. +* `tg256@d4096` is the decode metric: 256 generated tokens at depth 4096 + (`-p 0 -n 256 -d 4096`). Decode is CPU-bound with `--n-cpu-moe 40` and + swings with node CPU contention (about 60 co-tenant pods), so compare + medians across interleaved rounds and expect occasional catastrophic + samples. +* `tg32` (old rows, no depth suffix) is token generation at the DEFAULT depth + (n_ctx 32). **It is NOT a valid decode metric on this box.** At that size + decode is launch/barrier bound, not compute bound, and measures a regime + serving never reaches: it read -36% between two upstream commits that are at + parity at tg256 (the cause was the view-alias fix in + ggml_vk_graph_optimize). tg is also NOT monotonic in n_ctx here - on one + binary n_ctx 256 measured slower than both 32 and 544 - so only ever compare + same-depth cells. `summarize.py` flags any depth-less tg row. ## Noise floor: deltas under about 5 percent are unproven @@ -196,21 +212,36 @@ If a fresh run disagrees with these by much more than the noise floor, something is wrong (something else on the GPU, a thermal problem, a bad build) before you believe you found a speedup. -Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config, current upstream -(`f3f1a8f27`) and the deployed fork, measured 2026-09-08/09: +Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config. + +llama-bench, current production fork (20261024 rebase, `6c480f7e7`) and the +previous production fork (`5751c977b`), measured 2026-10-24 by +`fork-validate.sh --perf` (2 interleaved arms each, `-r 2` prefill, `-r 3` +decode): + +| metric | about | +|---|---| +| pp2048 (depth 0) | 274-286 t/s (274 current, 286 previous; the -4% is inside the noise floor) | +| pp2048@d16384 | 201-206 t/s | +| pp2048@d32768 | 158-160 t/s | +| tg256@d4096 | 17.8 t/s | + +Corpus harness, measured 2026-09-08/09 on the then-current upstream +(`f3f1a8f27`) and the fork deployed then (not re-measured since): | metric | clean upstream | fork | |---|---|---| | corpus pp @ 4k | about 240 t/s | about 255 t/s | | corpus pp @ 16k | about 206 t/s | about 225 t/s | | corpus pp @ 32k | about 171 t/s | about 195 t/s | -| llama-bench pp2048 | see results.txt | about 285 t/s | -| llama-bench tg32 | | not a valid metric, see above | +| llama-bench pp8192 | about 222 t/s | about 246 t/s | -Older reference points for the same model/config: `pp2048` depth-0 about -285 t/s. The historical "tg32 about 19 t/s" figure is retained only as trivia - -do not treat it as a target, and see the tg32 warning above. +Invalid-metric history: the 2026-09 runs recorded `tg32` (7-15 t/s, swinging +by run) and older notes mention "tg32 about 19 t/s". Neither is a decode +reference; see the tg32 warning above. Model load with `-lm none` takes about 26 s warm, longer cold. A full 3-round A/B run of both harnesses takes roughly 100 minutes; -add about 16 minutes per extra arm per round. +add about 16 minutes per extra arm per round. The tg256@d4096 decode cell +costs about 1.2 minutes per arm per round (model load, the 4096-token depth +prefill and 3 x 256 tokens), measured 2026-10-24. diff --git a/scripts/rx580-bench/lbsweep.sh b/scripts/rx580-bench/lbsweep.sh index 9b33ec1ad..af7f96f13 100755 --- a/scripts/rx580-bench/lbsweep.sh +++ b/scripts/rx580-bench/lbsweep.sh @@ -4,11 +4,24 @@ # Runs ON THE POD. Normally invoked by bench.sh. # # Usage: lbsweep.sh