Files
llama.cpp/scripts/rx580-bench
LumpiastyandClaude Opus 5.5 62d442e7bf scripts: rx580-bench - decode at tg256 depth 4096, not tg32
The llama-bench sweep measured decode as tg32 at the default depth and the
README called it the continuity metric, while tg32 at n_ctx 32 is launch /
barrier bound on this card and is not a valid decode metric (it once read
-36% between builds at parity at tg256). lbsweep.sh now runs prefill
(-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as
two invocations and labels depth cells (tg256@d4096). summarize.py marks
old depth-less tg rows as invalid-metric history. The README examples and
the reference values are updated to the 2026-10-24 production numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 00:33:22 +02:00
..

rx580-bench

Prompt-processing benchmark harness for the RX 580 (Polaris) serving pod.

It exists so that no future session has to rediscover how to measure this machine. It encodes the production config, fixed corpus slices, the repeat and median discipline, and the noise floor.

What it measures

Two things, deliberately:

  1. Corpus prompt processing (the number we actually care about). Real Polish prose from pan-tadeusz.txt pushed through llama-server's /completion endpoint with n_predict: 1, cache_prompt: false. This mimics agentic tool-result parsing: a big blob of real text arriving cold. Reported as timings.prompt_per_second at prompt lengths of about 4k, 16k and 32k tokens.
  2. llama-bench sweep (the controlled cross-check). Synthetic random tokens: prefill pp2048, pp8192 at depth 0, and decode tg256@d4096 (256 generated tokens on top of a 4096-token context, -p 0 -n 256 -d 4096). The prefill cells are the continuity metric with prior work on this box. Decode used to be tg32 at the default depth; that is not a valid metric here (see "Reading the numbers"), and old tg32 rows in results.txt are history only.

They do not measure the same thing and they do not agree. llama-bench pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a growing KV cache, so they fall off with prompt length. Both are useful.

How to run it

One command from the workstation:

./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean

Interleaved A/B, which is the only valid way to compare two builds:

./scripts/rx580-bench/run.sh \
  --build   /root/arms/new-clean/build --label   new-clean \
  --build-b /root/arms/new-fork/build  --label-b new-fork \
  --rounds 3

Summarize whatever has accumulated:

./scripts/rx580-bench/run.sh --summarize

run.sh copies the harness onto the pod and execs it there. The wrapper is thin on purpose: all the methodology lives in the pod-side scripts, so it can also be driven directly on the pod if kubectl is inconvenient.

The run is detached on the pod and then tailed, so Ctrl-C on the wrapper does not orphan a half-finished run. bench.sh restarts llama-swap from an EXIT trap on success, failure and interrupt alike.

Pod access

export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'

The pod has curl and python3 but no jq; the harness uses python3 for all JSON.

Files

file side what
run.sh workstation one-command wrapper; installs and execs the rest
bench.sh pod orchestrator: stops llama-swap, waits for idle GPU, runs both harnesses, restarts llama-swap via trap
ppbench.py pod the corpus prompt-processing harness
lbsweep.sh pod the llama-bench sweep
summarize.py pod turns results.txt into a median/min-max table with delta significance

Results append to /root/bench/results.txt as parseable RESULT / SUMMARY / LBRESULT lines. Server logs land in /root/bench/server-*.log.

The config it assumes

Production serving config, matching /root/config.yaml:

-t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 -lm none
--ctx-size 40960 --no-warmup

Model:

/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf

Qwen3.6-35B-A3B, 20.8 GiB, 40 layers, 256 experts of which 8 are used. Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16, 12 CPU cores, 62 GB RAM.

The llama-bench equivalent of that config is:

-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 0 -r 3
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 0 -n 256 -d 4096 -r 3

lbsweep.sh runs these as two invocations (prefill, then decode). Override with LB_P (prefill sizes), LB_N / LB_D (decode tokens / depth; LB_N=0 skips decode) and LB_R (repetitions).

The corpus slices are constants, not recalibrated

pan-tadeusz.txt is 447334 chars / 482907 bytes of Polish text. Against this model's tokenizer it runs 2.6702 chars/token, and /completion reports prompt_n identical to /tokenize (BOS offset 0).

The slices are corpus[0:NCHARS] and are baked into ppbench.py:

target tokens chars utf-8 bytes
4096 10776 11608
16384 43858 47175
32768 87165 93843

These hit the target prompt_n exactly. Every build is therefore measured on byte-identical input. ppbench.py warns loudly if the reported prompt_n ever stops matching the target, which is the signal that the table is stale. Only recalibrate (ppbench.py <build> <label> --calibrate) if the model file or the corpus changes.

Reading the numbers

  • prompt_n is the token count actually prefilled. It must equal the target.
  • prompt_ms is wall-clock prefill time.
  • tps / prompt_per_second is prompt_n / prompt_ms * 1000. Higher better.
  • pp2048 / pp8192 are llama-bench prefill throughput at depth 0. A depth suffix (pp2048@d16384) means the prompt is processed on top of a context of that many tokens.
  • tg256@d4096 is the decode metric: 256 generated tokens at depth 4096 (-p 0 -n 256 -d 4096). Decode is CPU-bound with --n-cpu-moe 40 and swings with node CPU contention (about 60 co-tenant pods), so compare medians across interleaved rounds and expect occasional catastrophic samples.
  • tg32 (old rows, no depth suffix) is token generation at the DEFAULT depth (n_ctx 32). It is NOT a valid decode metric on this box. At that size decode is launch/barrier bound, not compute bound, and measures a regime serving never reaches: it read -36% between two upstream commits that are at parity at tg256 (the cause was the view-alias fix in ggml_vk_graph_optimize). tg is also NOT monotonic in n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 - so only ever compare same-depth cells. summarize.py flags any depth-less tg row.

Noise floor: deltas under about 5 percent are unproven

This is the single most important thing in this directory.

llama-bench's within-run error bars understate cross-invocation variance by roughly 14x. The same config measured 276.7 +/- 0.9 in one invocation and 296.4 +/- 0.6 in another, 7 percent apart, with error bars that claimed 0.3 percent precision.

Therefore:

  • Any comparison you want to draw a conclusion from must live inside a single llama-bench invocation, using comma-separated sweeps.
  • Two different builds cannot share an invocation, so run them interleaved A/B/A/B/A/B across at least 3 rounds and compare medians. That is what --build-b does.
  • State explicitly that any cross-build delta under about 5 percent is unproven. summarize.py labels them unproven for you.

Traps

  • -tb is not a llama-bench flag. It is a llama-server flag. Passing it makes llama-bench print usage and exit silently, mid-sweep, which looks exactly like a run that produced nothing. Always check the row count.
  • A ubatch larger than the prompt never fills. With -ub 2048, prompt lengths must be multiples of 2048 or the sweep measures nothing meaningful.
  • Nothing else may touch the GPU. bench.sh stops llama-swap and refuses to start if any llama-* process is still alive.
  • Never leave the pod not serving. bench.sh restarts llama-swap from an EXIT trap. If you bypass the script, restart it by hand: supervisorctl -c /root/supervisord.conf start llama-swap.
  • Do not run llama-cli detached with closed stdin on this box; it loops forever. Use the server harness or llama-bench -d for depth timing.
  • --mmap / --no-mmap / --mlock and llama-bench's -mmp are gone upstream (#28334). Use -lm none (--load-mode none) on both llama-server and llama-bench. The old flags now fail: the server refuses to start, llama-bench silently produces no rows. /root/config.yaml must be updated to -lm none as well before deploying a build past that commit.

Optional: the MoE expert cache

Upstream #29887 added --moe-cache-mib N (default 0, off): a device cache of the most recently used experts of the CPU-resident (--n-cpu-moe) layers. It mostly helps decode and small batches; large prefill batches only read from it. It is a server option only - llama-bench has no flag for it - so measure it with the corpus harness, e.g. LLAMA_ARG_MOE_CACHE_MIB=1024 bench.sh --corpus-only --build DIR --label NAME (the environment reaches the spawned server). The variable applies to every arm of the run, so an on/off comparison spans two runs and the noise floor above applies in full. The cache takes VRAM from the same 8 GB as the KV cache and compute buffers.

Known-good reference values

If a fresh run disagrees with these by much more than the noise floor, something is wrong (something else on the GPU, a thermal problem, a bad build) before you believe you found a speedup.

Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config.

llama-bench, current production fork (20261024 rebase, 6c480f7e7) and the previous production fork (5751c977b), measured 2026-10-24 by fork-validate.sh --perf (2 interleaved arms each, -r 2 prefill, -r 3 decode):

metric about
pp2048 (depth 0) 274-286 t/s (274 current, 286 previous; the -4% is inside the noise floor)
pp2048@d16384 201-206 t/s
pp2048@d32768 158-160 t/s
tg256@d4096 17.8 t/s

Corpus harness, measured 2026-09-08/09 on the then-current upstream (f3f1a8f27) and the fork deployed then (not re-measured since):

metric clean upstream fork
corpus pp @ 4k about 240 t/s about 255 t/s
corpus pp @ 16k about 206 t/s about 225 t/s
corpus pp @ 32k about 171 t/s about 195 t/s
llama-bench pp8192 about 222 t/s about 246 t/s

Invalid-metric history: the 2026-09 runs recorded tg32 (7-15 t/s, swinging by run) and older notes mention "tg32 about 19 t/s". Neither is a decode reference; see the tg32 warning above.

Model load with -lm none takes about 26 s warm, longer cold. A full 3-round A/B run of both harnesses takes roughly 100 minutes; add about 16 minutes per extra arm per round. The tg256@d4096 decode cell costs about 1.2 minutes per arm per round (model load, the 4096-token depth prefill and 3 x 256 tokens), measured 2026-10-24.