The llama-bench sweep measured decode as tg32 at the default depth and the README called it the continuity metric, while tg32 at n_ctx 32 is launch / barrier bound on this card and is not a valid decode metric (it once read -36% between builds at parity at tg256). lbsweep.sh now runs prefill (-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as two invocations and labels depth cells (tg256@d4096). summarize.py marks old depth-less tg rows as invalid-metric history. The README examples and the reference values are updated to the 2026-10-24 production numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
248 lines
10 KiB
Markdown
248 lines
10 KiB
Markdown
# rx580-bench
|
|
|
|
Prompt-processing benchmark harness for the RX 580 (Polaris) serving pod.
|
|
|
|
It exists so that no future session has to rediscover how to measure this
|
|
machine. It encodes the production config, fixed corpus slices, the repeat
|
|
and median discipline, and the noise floor.
|
|
|
|
## What it measures
|
|
|
|
Two things, deliberately:
|
|
|
|
1. **Corpus prompt processing** (the number we actually care about). Real
|
|
Polish prose from `pan-tadeusz.txt` pushed through `llama-server`'s
|
|
`/completion` endpoint with `n_predict: 1`, `cache_prompt: false`. This
|
|
mimics agentic tool-result parsing: a big blob of real text arriving cold.
|
|
Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k
|
|
and 32k tokens.
|
|
2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random
|
|
tokens: prefill `pp2048`, `pp8192` at depth 0, and decode `tg256@d4096`
|
|
(256 generated tokens on top of a 4096-token context, `-p 0 -n 256
|
|
-d 4096`). The prefill cells are the continuity metric with prior work on
|
|
this box. Decode used to be `tg32` at the default depth; that is not a valid
|
|
metric here (see "Reading the numbers"), and old `tg32` rows in
|
|
`results.txt` are history only.
|
|
|
|
They do not measure the same thing and they do not agree. `llama-bench`
|
|
pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
|
|
growing KV cache, so they fall off with prompt length. Both are useful.
|
|
|
|
## How to run it
|
|
|
|
One command from the workstation:
|
|
|
|
```
|
|
./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean
|
|
```
|
|
|
|
Interleaved A/B, which is the only valid way to compare two builds:
|
|
|
|
```
|
|
./scripts/rx580-bench/run.sh \
|
|
--build /root/arms/new-clean/build --label new-clean \
|
|
--build-b /root/arms/new-fork/build --label-b new-fork \
|
|
--rounds 3
|
|
```
|
|
|
|
Summarize whatever has accumulated:
|
|
|
|
```
|
|
./scripts/rx580-bench/run.sh --summarize
|
|
```
|
|
|
|
`run.sh` copies the harness onto the pod and execs it there. The wrapper is
|
|
thin on purpose: all the methodology lives in the pod-side scripts, so it can
|
|
also be driven directly on the pod if kubectl is inconvenient.
|
|
|
|
The run is detached on the pod and then tailed, so Ctrl-C on the wrapper does
|
|
not orphan a half-finished run. `bench.sh` restarts `llama-swap` from an EXIT
|
|
trap on success, failure and interrupt alike.
|
|
|
|
### Pod access
|
|
|
|
```
|
|
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
|
|
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
|
|
```
|
|
|
|
The pod has `curl` and `python3` but **no `jq`**; the harness uses python3 for
|
|
all JSON.
|
|
|
|
## Files
|
|
|
|
| file | side | what |
|
|
|---|---|---|
|
|
| `run.sh` | workstation | one-command wrapper; installs and execs the rest |
|
|
| `bench.sh` | pod | orchestrator: stops llama-swap, waits for idle GPU, runs both harnesses, restarts llama-swap via trap |
|
|
| `ppbench.py` | pod | the corpus prompt-processing harness |
|
|
| `lbsweep.sh` | pod | the llama-bench sweep |
|
|
| `summarize.py` | pod | turns `results.txt` into a median/min-max table with delta significance |
|
|
|
|
Results append to `/root/bench/results.txt` as parseable `RESULT` /
|
|
`SUMMARY` / `LBRESULT` lines. Server logs land in `/root/bench/server-*.log`.
|
|
|
|
## The config it assumes
|
|
|
|
Production serving config, matching `/root/config.yaml`:
|
|
|
|
```
|
|
-t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 -lm none
|
|
--ctx-size 40960 --no-warmup
|
|
```
|
|
|
|
Model:
|
|
|
|
```
|
|
/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
|
|
```
|
|
|
|
Qwen3.6-35B-A3B, 20.8 GiB, 40 layers, 256 experts of which 8 are used.
|
|
Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16,
|
|
12 CPU cores, 62 GB RAM.
|
|
|
|
The `llama-bench` equivalent of that config is:
|
|
|
|
```
|
|
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 0 -r 3
|
|
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 0 -n 256 -d 4096 -r 3
|
|
```
|
|
|
|
`lbsweep.sh` runs these as two invocations (prefill, then decode). Override
|
|
with `LB_P` (prefill sizes), `LB_N` / `LB_D` (decode tokens / depth; `LB_N=0`
|
|
skips decode) and `LB_R` (repetitions).
|
|
|
|
## The corpus slices are constants, not recalibrated
|
|
|
|
`pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this
|
|
model's tokenizer it runs **2.6702 chars/token**, and `/completion` reports
|
|
`prompt_n` identical to `/tokenize` (BOS offset 0).
|
|
|
|
The slices are `corpus[0:NCHARS]` and are baked into `ppbench.py`:
|
|
|
|
| target tokens | chars | utf-8 bytes |
|
|
|---|---|---|
|
|
| 4096 | 10776 | 11608 |
|
|
| 16384 | 43858 | 47175 |
|
|
| 32768 | 87165 | 93843 |
|
|
|
|
These hit the target `prompt_n` exactly. Every build is therefore measured on
|
|
byte-identical input. `ppbench.py` warns loudly if the reported `prompt_n`
|
|
ever stops matching the target, which is the signal that the table is stale.
|
|
Only recalibrate (`ppbench.py <build> <label> --calibrate`) if the model file
|
|
or the corpus changes.
|
|
|
|
## Reading the numbers
|
|
|
|
* `prompt_n` is the token count actually prefilled. It must equal the target.
|
|
* `prompt_ms` is wall-clock prefill time.
|
|
* `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better.
|
|
* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0. A depth
|
|
suffix (`pp2048@d16384`) means the prompt is processed on top of a context
|
|
of that many tokens.
|
|
* `tg256@d4096` is the decode metric: 256 generated tokens at depth 4096
|
|
(`-p 0 -n 256 -d 4096`). Decode is CPU-bound with `--n-cpu-moe 40` and
|
|
swings with node CPU contention (about 60 co-tenant pods), so compare
|
|
medians across interleaved rounds and expect occasional catastrophic
|
|
samples.
|
|
* `tg32` (old rows, no depth suffix) is token generation at the DEFAULT depth
|
|
(n_ctx 32). **It is NOT a valid decode metric on this box.** At that size
|
|
decode is launch/barrier bound, not compute bound, and measures a regime
|
|
serving never reaches: it read -36% between two upstream commits that are at
|
|
parity at tg256 (the cause was the view-alias fix in
|
|
ggml_vk_graph_optimize). tg is also NOT monotonic in n_ctx here - on one
|
|
binary n_ctx 256 measured slower than both 32 and 544 - so only ever compare
|
|
same-depth cells. `summarize.py` flags any depth-less tg row.
|
|
|
|
## Noise floor: deltas under about 5 percent are unproven
|
|
|
|
This is the single most important thing in this directory.
|
|
|
|
`llama-bench`'s within-run error bars understate **cross-invocation** variance
|
|
by roughly **14x**. The same config measured `276.7 +/- 0.9` in one invocation
|
|
and `296.4 +/- 0.6` in another, 7 percent apart, with error bars that claimed
|
|
0.3 percent precision.
|
|
|
|
Therefore:
|
|
|
|
* Any comparison you want to draw a conclusion from must live **inside a
|
|
single `llama-bench` invocation**, using comma-separated sweeps.
|
|
* Two different *builds* cannot share an invocation, so run them
|
|
**interleaved A/B/A/B/A/B across at least 3 rounds** and compare medians.
|
|
That is what `--build-b` does.
|
|
* State explicitly that any cross-build delta under about 5 percent is
|
|
unproven. `summarize.py` labels them `unproven` for you.
|
|
|
|
## Traps
|
|
|
|
* **`-tb` is not a `llama-bench` flag.** It is a `llama-server` flag. Passing
|
|
it makes `llama-bench` print usage and exit silently, mid-sweep, which looks
|
|
exactly like a run that produced nothing. Always check the row count.
|
|
* **A ubatch larger than the prompt never fills.** With `-ub 2048`, prompt
|
|
lengths must be multiples of 2048 or the sweep measures nothing meaningful.
|
|
* **Nothing else may touch the GPU.** `bench.sh` stops `llama-swap` and
|
|
refuses to start if any `llama-*` process is still alive.
|
|
* **Never leave the pod not serving.** `bench.sh` restarts `llama-swap` from
|
|
an EXIT trap. If you bypass the script, restart it by hand:
|
|
`supervisorctl -c /root/supervisord.conf start llama-swap`.
|
|
* **Do not run `llama-cli` detached with closed stdin** on this box; it loops
|
|
forever. Use the server harness or `llama-bench -d` for depth timing.
|
|
* **`--mmap` / `--no-mmap` / `--mlock` and llama-bench's `-mmp` are gone**
|
|
upstream (#28334). Use `-lm none` (`--load-mode none`) on both
|
|
`llama-server` and `llama-bench`. The old flags now fail: the server refuses
|
|
to start, `llama-bench` silently produces no rows. `/root/config.yaml` must
|
|
be updated to `-lm none` as well before deploying a build past that commit.
|
|
|
|
## Optional: the MoE expert cache
|
|
|
|
Upstream #29887 added `--moe-cache-mib N` (default 0, off): a device cache of
|
|
the most recently used experts of the CPU-resident (`--n-cpu-moe`) layers. It
|
|
mostly helps decode and small batches; large prefill batches only read from it.
|
|
It is a server option only - `llama-bench` has no flag for it - so measure it
|
|
with the corpus harness, e.g.
|
|
`LLAMA_ARG_MOE_CACHE_MIB=1024 bench.sh --corpus-only --build DIR --label NAME`
|
|
(the environment reaches the spawned server). The variable applies to every
|
|
arm of the run, so an on/off comparison spans two runs and the noise floor
|
|
above applies in full. The cache takes VRAM from the same 8 GB as the KV
|
|
cache and compute buffers.
|
|
|
|
## Known-good reference values
|
|
|
|
If a fresh run disagrees with these by much more than the noise floor,
|
|
something is wrong (something else on the GPU, a thermal problem, a bad
|
|
build) before you believe you found a speedup.
|
|
|
|
Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config.
|
|
|
|
llama-bench, current production fork (20261024 rebase, `6c480f7e7`) and the
|
|
previous production fork (`5751c977b`), measured 2026-10-24 by
|
|
`fork-validate.sh --perf` (2 interleaved arms each, `-r 2` prefill, `-r 3`
|
|
decode):
|
|
|
|
| metric | about |
|
|
|---|---|
|
|
| pp2048 (depth 0) | 274-286 t/s (274 current, 286 previous; the -4% is inside the noise floor) |
|
|
| pp2048@d16384 | 201-206 t/s |
|
|
| pp2048@d32768 | 158-160 t/s |
|
|
| tg256@d4096 | 17.8 t/s |
|
|
|
|
Corpus harness, measured 2026-09-08/09 on the then-current upstream
|
|
(`f3f1a8f27`) and the fork deployed then (not re-measured since):
|
|
|
|
| metric | clean upstream | fork |
|
|
|---|---|---|
|
|
| corpus pp @ 4k | about 240 t/s | about 255 t/s |
|
|
| corpus pp @ 16k | about 206 t/s | about 225 t/s |
|
|
| corpus pp @ 32k | about 171 t/s | about 195 t/s |
|
|
| llama-bench pp8192 | about 222 t/s | about 246 t/s |
|
|
|
|
Invalid-metric history: the 2026-09 runs recorded `tg32` (7-15 t/s, swinging
|
|
by run) and older notes mention "tg32 about 19 t/s". Neither is a decode
|
|
reference; see the tg32 warning above.
|
|
|
|
Model load with `-lm none` takes about 26 s warm, longer cold.
|
|
A full 3-round A/B run of both harnesses takes roughly 100 minutes;
|
|
add about 16 minutes per extra arm per round. The tg256@d4096 decode cell
|
|
costs about 1.2 minutes per arm per round (model load, the 4096-token depth
|
|
prefill and 3 x 256 tokens), measured 2026-10-24.
|