The llama-bench sweep measured decode as tg32 at the default depth and the README called it the continuity metric, while tg32 at n_ctx 32 is launch / barrier bound on this card and is not a valid decode metric (it once read -36% between builds at parity at tg256). lbsweep.sh now runs prefill (-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as two invocations and labels depth cells (tg256@d4096). summarize.py marks old depth-less tg rows as invalid-metric history. The README examples and the reference values are updated to the 2026-10-24 production numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rx580-bench
Prompt-processing benchmark harness for the RX 580 (Polaris) serving pod.
It exists so that no future session has to rediscover how to measure this machine. It encodes the production config, fixed corpus slices, the repeat and median discipline, and the noise floor.
What it measures
Two things, deliberately:
- Corpus prompt processing (the number we actually care about). Real
Polish prose from
pan-tadeusz.txtpushed throughllama-server's/completionendpoint withn_predict: 1,cache_prompt: false. This mimics agentic tool-result parsing: a big blob of real text arriving cold. Reported astimings.prompt_per_secondat prompt lengths of about 4k, 16k and 32k tokens. llama-benchsweep (the controlled cross-check). Synthetic random tokens: prefillpp2048,pp8192at depth 0, and decodetg256@d4096(256 generated tokens on top of a 4096-token context,-p 0 -n 256 -d 4096). The prefill cells are the continuity metric with prior work on this box. Decode used to betg32at the default depth; that is not a valid metric here (see "Reading the numbers"), and oldtg32rows inresults.txtare history only.
They do not measure the same thing and they do not agree. llama-bench
pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
growing KV cache, so they fall off with prompt length. Both are useful.
How to run it
One command from the workstation:
./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean
Interleaved A/B, which is the only valid way to compare two builds:
./scripts/rx580-bench/run.sh \
--build /root/arms/new-clean/build --label new-clean \
--build-b /root/arms/new-fork/build --label-b new-fork \
--rounds 3
Summarize whatever has accumulated:
./scripts/rx580-bench/run.sh --summarize
run.sh copies the harness onto the pod and execs it there. The wrapper is
thin on purpose: all the methodology lives in the pod-side scripts, so it can
also be driven directly on the pod if kubectl is inconvenient.
The run is detached on the pod and then tailed, so Ctrl-C on the wrapper does
not orphan a half-finished run. bench.sh restarts llama-swap from an EXIT
trap on success, failure and interrupt alike.
Pod access
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
The pod has curl and python3 but no jq; the harness uses python3 for
all JSON.
Files
| file | side | what |
|---|---|---|
run.sh |
workstation | one-command wrapper; installs and execs the rest |
bench.sh |
pod | orchestrator: stops llama-swap, waits for idle GPU, runs both harnesses, restarts llama-swap via trap |
ppbench.py |
pod | the corpus prompt-processing harness |
lbsweep.sh |
pod | the llama-bench sweep |
summarize.py |
pod | turns results.txt into a median/min-max table with delta significance |
Results append to /root/bench/results.txt as parseable RESULT /
SUMMARY / LBRESULT lines. Server logs land in /root/bench/server-*.log.
The config it assumes
Production serving config, matching /root/config.yaml:
-t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 -lm none
--ctx-size 40960 --no-warmup
Model:
/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
Qwen3.6-35B-A3B, 20.8 GiB, 40 layers, 256 experts of which 8 are used. Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16, 12 CPU cores, 62 GB RAM.
The llama-bench equivalent of that config is:
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 0 -r 3
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 0 -n 256 -d 4096 -r 3
lbsweep.sh runs these as two invocations (prefill, then decode). Override
with LB_P (prefill sizes), LB_N / LB_D (decode tokens / depth; LB_N=0
skips decode) and LB_R (repetitions).
The corpus slices are constants, not recalibrated
pan-tadeusz.txt is 447334 chars / 482907 bytes of Polish text. Against this
model's tokenizer it runs 2.6702 chars/token, and /completion reports
prompt_n identical to /tokenize (BOS offset 0).
The slices are corpus[0:NCHARS] and are baked into ppbench.py:
| target tokens | chars | utf-8 bytes |
|---|---|---|
| 4096 | 10776 | 11608 |
| 16384 | 43858 | 47175 |
| 32768 | 87165 | 93843 |
These hit the target prompt_n exactly. Every build is therefore measured on
byte-identical input. ppbench.py warns loudly if the reported prompt_n
ever stops matching the target, which is the signal that the table is stale.
Only recalibrate (ppbench.py <build> <label> --calibrate) if the model file
or the corpus changes.
Reading the numbers
prompt_nis the token count actually prefilled. It must equal the target.prompt_msis wall-clock prefill time.tps/prompt_per_secondisprompt_n / prompt_ms * 1000. Higher better.pp2048/pp8192are llama-bench prefill throughput at depth 0. A depth suffix (pp2048@d16384) means the prompt is processed on top of a context of that many tokens.tg256@d4096is the decode metric: 256 generated tokens at depth 4096 (-p 0 -n 256 -d 4096). Decode is CPU-bound with--n-cpu-moe 40and swings with node CPU contention (about 60 co-tenant pods), so compare medians across interleaved rounds and expect occasional catastrophic samples.tg32(old rows, no depth suffix) is token generation at the DEFAULT depth (n_ctx 32). It is NOT a valid decode metric on this box. At that size decode is launch/barrier bound, not compute bound, and measures a regime serving never reaches: it read -36% between two upstream commits that are at parity at tg256 (the cause was the view-alias fix in ggml_vk_graph_optimize). tg is also NOT monotonic in n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 - so only ever compare same-depth cells.summarize.pyflags any depth-less tg row.
Noise floor: deltas under about 5 percent are unproven
This is the single most important thing in this directory.
llama-bench's within-run error bars understate cross-invocation variance
by roughly 14x. The same config measured 276.7 +/- 0.9 in one invocation
and 296.4 +/- 0.6 in another, 7 percent apart, with error bars that claimed
0.3 percent precision.
Therefore:
- Any comparison you want to draw a conclusion from must live inside a
single
llama-benchinvocation, using comma-separated sweeps. - Two different builds cannot share an invocation, so run them
interleaved A/B/A/B/A/B across at least 3 rounds and compare medians.
That is what
--build-bdoes. - State explicitly that any cross-build delta under about 5 percent is
unproven.
summarize.pylabels themunprovenfor you.
Traps
-tbis not allama-benchflag. It is allama-serverflag. Passing it makesllama-benchprint usage and exit silently, mid-sweep, which looks exactly like a run that produced nothing. Always check the row count.- A ubatch larger than the prompt never fills. With
-ub 2048, prompt lengths must be multiples of 2048 or the sweep measures nothing meaningful. - Nothing else may touch the GPU.
bench.shstopsllama-swapand refuses to start if anyllama-*process is still alive. - Never leave the pod not serving.
bench.shrestartsllama-swapfrom an EXIT trap. If you bypass the script, restart it by hand:supervisorctl -c /root/supervisord.conf start llama-swap. - Do not run
llama-clidetached with closed stdin on this box; it loops forever. Use the server harness orllama-bench -dfor depth timing. --mmap/--no-mmap/--mlockand llama-bench's-mmpare gone upstream (#28334). Use-lm none(--load-mode none) on bothllama-serverandllama-bench. The old flags now fail: the server refuses to start,llama-benchsilently produces no rows./root/config.yamlmust be updated to-lm noneas well before deploying a build past that commit.
Optional: the MoE expert cache
Upstream #29887 added --moe-cache-mib N (default 0, off): a device cache of
the most recently used experts of the CPU-resident (--n-cpu-moe) layers. It
mostly helps decode and small batches; large prefill batches only read from it.
It is a server option only - llama-bench has no flag for it - so measure it
with the corpus harness, e.g.
LLAMA_ARG_MOE_CACHE_MIB=1024 bench.sh --corpus-only --build DIR --label NAME
(the environment reaches the spawned server). The variable applies to every
arm of the run, so an on/off comparison spans two runs and the noise floor
above applies in full. The cache takes VRAM from the same 8 GB as the KV
cache and compute buffers.
Known-good reference values
If a fresh run disagrees with these by much more than the noise floor, something is wrong (something else on the GPU, a thermal problem, a bad build) before you believe you found a speedup.
Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config.
llama-bench, current production fork (20261024 rebase, 6c480f7e7) and the
previous production fork (5751c977b), measured 2026-10-24 by
fork-validate.sh --perf (2 interleaved arms each, -r 2 prefill, -r 3
decode):
| metric | about |
|---|---|
| pp2048 (depth 0) | 274-286 t/s (274 current, 286 previous; the -4% is inside the noise floor) |
| pp2048@d16384 | 201-206 t/s |
| pp2048@d32768 | 158-160 t/s |
| tg256@d4096 | 17.8 t/s |
Corpus harness, measured 2026-09-08/09 on the then-current upstream
(f3f1a8f27) and the fork deployed then (not re-measured since):
| metric | clean upstream | fork |
|---|---|---|
| corpus pp @ 4k | about 240 t/s | about 255 t/s |
| corpus pp @ 16k | about 206 t/s | about 225 t/s |
| corpus pp @ 32k | about 171 t/s | about 195 t/s |
| llama-bench pp8192 | about 222 t/s | about 246 t/s |
Invalid-metric history: the 2026-09 runs recorded tg32 (7-15 t/s, swinging
by run) and older notes mention "tg32 about 19 t/s". Neither is a decode
reference; see the tg32 warning above.
Model load with -lm none takes about 26 s warm, longer cold.
A full 3-round A/B run of both harnesses takes roughly 100 minutes;
add about 16 minutes per extra arm per round. The tg256@d4096 decode cell
costs about 1.2 minutes per arm per round (model load, the 4096-token depth
prefill and 3 x 256 tokens), measured 2026-10-24.