llama.cpp/scripts/rx580-bench/README.md

# rx580-bench

Prompt-processing benchmark harness for the RX 580 (Polaris) serving pod.

It exists so that no future session has to rediscover how to measure this
machine. It encodes the production config, fixed corpus slices, the repeat
and median discipline, and the noise floor.

## What it measures

Two things, deliberately:

1. **Corpus prompt processing** (the number we actually care about). Real
   Polish prose from `pan-tadeusz.txt` pushed through `llama-server`'s
   `/completion` endpoint with `n_predict: 1`, `cache_prompt: false`. This
   mimics agentic tool-result parsing: a big blob of real text arriving cold.
   Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k
   and 32k tokens.
2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random
   tokens, `pp2048`, `pp8192`, `tg32`. This is what prior work on this box
   recorded, so it is the continuity metric.

They do not measure the same thing and they do not agree. `llama-bench`
pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
growing KV cache, so they fall off with prompt length. Both are useful.

## How to run it

One command from the workstation:

```
./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean
```

Interleaved A/B, which is the only valid way to compare two builds:

```
./scripts/rx580-bench/run.sh \
  --build   /root/arms/new-clean/build --label   new-clean \
  --build-b /root/arms/new-fork/build  --label-b new-fork \
  --rounds 3
```

Summarize whatever has accumulated:

```
./scripts/rx580-bench/run.sh --summarize
```

`run.sh` copies the harness onto the pod and execs it there. The wrapper is
thin on purpose: all the methodology lives in the pod-side scripts, so it can
also be driven directly on the pod if kubectl is inconvenient.

The run is detached on the pod and then tailed, so Ctrl-C on the wrapper does
not orphan a half-finished run. `bench.sh` restarts `llama-swap` from an EXIT
trap on success, failure and interrupt alike.

### Pod access

```
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
```

The pod has `curl` and `python3` but **no `jq`**; the harness uses python3 for
all JSON.

## Files

| file | side | what |
|---|---|---|
| `run.sh` | workstation | one-command wrapper; installs and execs the rest |
| `bench.sh` | pod | orchestrator: stops llama-swap, waits for idle GPU, runs both harnesses, restarts llama-swap via trap |
| `ppbench.py` | pod | the corpus prompt-processing harness |
| `lbsweep.sh` | pod | the llama-bench sweep |
| `summarize.py` | pod | turns `results.txt` into a median/min-max table with delta significance |

Results append to `/root/bench/results.txt` as parseable `RESULT` /
`SUMMARY` / `LBRESULT` lines. Server logs land in `/root/bench/server-*.log`.

## The config it assumes

Production serving config, matching `/root/config.yaml`:

```
-t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --no-mmap
--ctx-size 40960 --no-warmup
```

Model:

```
/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
```

Qwen3.6-35B-A3B, 20.8 GiB, 40 layers, 256 experts of which 8 are used.
Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16,
12 CPU cores, 62 GB RAM.

The `llama-bench` equivalent of that config is:

```
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -mmp 0 -p 2048,8192 -n 32 -r 3
```

## The corpus slices are constants, not recalibrated

`pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this
model's tokenizer it runs **2.6702 chars/token**, and `/completion` reports
`prompt_n` identical to `/tokenize` (BOS offset 0).

The slices are `corpus[0:NCHARS]` and are baked into `ppbench.py`:

| target tokens | chars | utf-8 bytes |
|---|---|---|
| 4096 | 10776 | 11608 |
| 16384 | 43858 | 47175 |
| 32768 | 87165 | 93843 |

These hit the target `prompt_n` exactly. Every build is therefore measured on
byte-identical input. `ppbench.py` warns loudly if the reported `prompt_n`
ever stops matching the target, which is the signal that the table is stale.
Only recalibrate (`ppbench.py <build> <label> --calibrate`) if the model file
or the corpus changes.

## Reading the numbers

* `prompt_n` is the token count actually prefilled. It must equal the target.
* `prompt_ms` is wall-clock prefill time.
* `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better.
* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0.
* `tg32` is token generation at DEFAULT depth (n_ctx 32). **It is NOT a valid
  decode metric on this box.** At that size decode is launch/barrier bound, not
  compute bound, and measures a regime serving never reaches: it read -36%
  between two upstream commits that are at parity in real use (the cause was
  the view-alias fix in ggml_vk_graph_optimize). For decode use **tg256 with an
  explicit depth**, e.g. `-p 0 -n 256 -d 2048`. tg is also NOT monotonic in
  n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 -
  so only ever compare same-depth cells.

## Noise floor: deltas under about 5 percent are unproven

This is the single most important thing in this directory.

`llama-bench`'s within-run error bars understate **cross-invocation** variance
by roughly **14x**. The same config measured `276.7 +/- 0.9` in one invocation
and `296.4 +/- 0.6` in another, 7 percent apart, with error bars that claimed
0.3 percent precision.

Therefore:

* Any comparison you want to draw a conclusion from must live **inside a
  single `llama-bench` invocation**, using comma-separated sweeps.
* Two different *builds* cannot share an invocation, so run them
  **interleaved A/B/A/B/A/B across at least 3 rounds** and compare medians.
  That is what `--build-b` does.
* State explicitly that any cross-build delta under about 5 percent is
  unproven. `summarize.py` labels them `unproven` for you.

## Traps

* **`-tb` is not a `llama-bench` flag.** It is a `llama-server` flag. Passing
  it makes `llama-bench` print usage and exit silently, mid-sweep, which looks
  exactly like a run that produced nothing. Always check the row count.
* **A ubatch larger than the prompt never fills.** With `-ub 2048`, prompt
  lengths must be multiples of 2048 or the sweep measures nothing meaningful.
* **Nothing else may touch the GPU.** `bench.sh` stops `llama-swap` and
  refuses to start if any `llama-*` process is still alive.
* **Never leave the pod not serving.** `bench.sh` restarts `llama-swap` from
  an EXIT trap. If you bypass the script, restart it by hand:
  `supervisorctl -c /root/supervisord.conf start llama-swap`.
* **Do not run `llama-cli` detached with closed stdin** on this box; it loops
  forever. Use the server harness or `llama-bench -d` for depth timing.
* `--no-mmap` is deprecated upstream in favour of `--load-mode` / `-lm none`,
  but still accepted and still what production passes.

## Known-good reference values

If a fresh run disagrees with these by much more than the noise floor,
something is wrong (something else on the GPU, a thermal problem, a bad
build) before you believe you found a speedup.

Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config, current upstream
(`f3f1a8f27`) and the deployed fork, measured 2026-09-08/09:

| metric | clean upstream | fork |
|---|---|---|
| corpus pp @ 4k | about 240 t/s | about 255 t/s |
| corpus pp @ 16k | about 206 t/s | about 225 t/s |
| corpus pp @ 32k | about 171 t/s | about 195 t/s |
| llama-bench pp2048 | see results.txt | about 285 t/s |
| llama-bench tg32 | | not a valid metric, see above |

Older reference points for the same model/config: `pp2048` depth-0 about
285 t/s. The historical "tg32 about 19 t/s" figure is retained only as trivia -
do not treat it as a target, and see the tg32 warning above.

Model load with `--no-mmap` takes about 26 s warm, longer cold.
A full 3-round A/B run of both harnesses takes roughly 100 minutes;
add about 16 minutes per extra arm per round.