scripts: rx580-bench - decode at tg256 depth 4096, not tg32

The llama-bench sweep measured decode as tg32 at the default depth and the
README called it the continuity metric, while tg32 at n_ctx 32 is launch /
barrier bound on this card and is not a valid decode metric (it once read
-36% between builds at parity at tg256). lbsweep.sh now runs prefill
(-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as
two invocations and labels depth cells (tg256@d4096). summarize.py marks
old depth-less tg rows as invalid-metric history. The README examples and
the reference values are updated to the 2026-10-24 production numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 00:33:22 +02:00
1 parent f664b0f20e
commit 62d442e7bf
3 files changed
+99 -37

No files matched your search

+51 -20
View File
@@ -17,8 +17,12 @@ Two things, deliberately:
Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k
and 32k tokens.
2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random
tokens, `pp2048`, `pp8192`, `tg32`. This is what prior work on this box
recorded, so it is the continuity metric.
tokens: prefill `pp2048`, `pp8192` at depth 0, and decode `tg256@d4096`
(256 generated tokens on top of a 4096-token context, `-p 0 -n 256
-d 4096`). The prefill cells are the continuity metric with prior work on
this box. Decode used to be `tg32` at the default depth; that is not a valid
metric here (see "Reading the numbers"), and old `tg32` rows in
`results.txt` are history only.
They do not measure the same thing and they do not agree. `llama-bench`
pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
@@ -100,9 +104,14 @@ Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16,
The `llama-bench` equivalent of that config is:
```
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 32 -r 3
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 0 -r 3
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 0 -n 256 -d 4096 -r 3
```
`lbsweep.sh` runs these as two invocations (prefill, then decode). Override
with `LB_P` (prefill sizes), `LB_N` / `LB_D` (decode tokens / depth; `LB_N=0`
skips decode) and `LB_R` (repetitions).
## The corpus slices are constants, not recalibrated
`pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this
@@ -128,15 +137,22 @@ or the corpus changes.
* `prompt_n` is the token count actually prefilled. It must equal the target.
* `prompt_ms` is wall-clock prefill time.
* `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better.
* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0.
* `tg32` is token generation at DEFAULT depth (n_ctx 32). **It is NOT a valid
decode metric on this box.** At that size decode is launch/barrier bound, not
compute bound, and measures a regime serving never reaches: it read -36%
between two upstream commits that are at parity in real use (the cause was
the view-alias fix in ggml_vk_graph_optimize). For decode use **tg256 with an
explicit depth**, e.g. `-p 0 -n 256 -d 2048`. tg is also NOT monotonic in
n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 -
so only ever compare same-depth cells.
* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0. A depth
suffix (`pp2048@d16384`) means the prompt is processed on top of a context
of that many tokens.
* `tg256@d4096` is the decode metric: 256 generated tokens at depth 4096
(`-p 0 -n 256 -d 4096`). Decode is CPU-bound with `--n-cpu-moe 40` and
swings with node CPU contention (about 60 co-tenant pods), so compare
medians across interleaved rounds and expect occasional catastrophic
samples.
* `tg32` (old rows, no depth suffix) is token generation at the DEFAULT depth
(n_ctx 32). **It is NOT a valid decode metric on this box.** At that size
decode is launch/barrier bound, not compute bound, and measures a regime
serving never reaches: it read -36% between two upstream commits that are at
parity at tg256 (the cause was the view-alias fix in
ggml_vk_graph_optimize). tg is also NOT monotonic in n_ctx here - on one
binary n_ctx 256 measured slower than both 32 and 544 - so only ever compare
same-depth cells. `summarize.py` flags any depth-less tg row.
## Noise floor: deltas under about 5 percent are unproven
@@ -196,21 +212,36 @@ If a fresh run disagrees with these by much more than the noise floor,
something is wrong (something else on the GPU, a thermal problem, a bad
build) before you believe you found a speedup.
Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config, current upstream
(`f3f1a8f27`) and the deployed fork, measured 2026-09-08/09:
Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config.
llama-bench, current production fork (20261024 rebase, `6c480f7e7`) and the
previous production fork (`5751c977b`), measured 2026-10-24 by
`fork-validate.sh --perf` (2 interleaved arms each, `-r 2` prefill, `-r 3`
decode):
| metric | about |
|---|---|
| pp2048 (depth 0) | 274-286 t/s (274 current, 286 previous; the -4% is inside the noise floor) |
| pp2048@d16384 | 201-206 t/s |
| pp2048@d32768 | 158-160 t/s |
| tg256@d4096 | 17.8 t/s |
Corpus harness, measured 2026-09-08/09 on the then-current upstream
(`f3f1a8f27`) and the fork deployed then (not re-measured since):
| metric | clean upstream | fork |
|---|---|---|
| corpus pp @ 4k | about 240 t/s | about 255 t/s |
| corpus pp @ 16k | about 206 t/s | about 225 t/s |
| corpus pp @ 32k | about 171 t/s | about 195 t/s |
| llama-bench pp2048 | see results.txt | about 285 t/s |
| llama-bench tg32 | | not a valid metric, see above |
| llama-bench pp8192 | about 222 t/s | about 246 t/s |
Older reference points for the same model/config: `pp2048` depth-0 about
285 t/s. The historical "tg32 about 19 t/s" figure is retained only as trivia -
do not treat it as a target, and see the tg32 warning above.
Invalid-metric history: the 2026-09 runs recorded `tg32` (7-15 t/s, swinging
by run) and older notes mention "tg32 about 19 t/s". Neither is a decode
reference; see the tg32 warning above.
Model load with `-lm none` takes about 26 s warm, longer cold.
A full 3-round A/B run of both harnesses takes roughly 100 minutes;
add about 16 minutes per extra arm per round.
add about 16 minutes per extra arm per round. The tg256@d4096 decode cell
costs about 1.2 minutes per arm per round (model load, the 4096-token depth
prefill and 3 x 256 tokens), measured 2026-10-24.
+39 -14
View File
@@ -4,11 +4,24 @@
# Runs ON THE POD. Normally invoked by bench.sh.
#
# Usage: lbsweep.sh <build_dir> <label>
# Env overrides: RESULTS, MODEL, LB_P, LB_N, LB_R
# Env overrides: RESULTS, MODEL, LB_P, LB_N, LB_D, LB_R
#
# Two llama-bench invocations per call:
# prefill -p $LB_P -n 0 at depth 0 (default pp2048,pp8192)
# decode -p 0 -n $LB_N -d $LB_D (default tg256 @ d4096)
# LB_N=0 skips the decode invocation.
#
# Everything that must be compared lives inside ONE llama-bench invocation
# (-p takes a comma-separated list), because cross-invocation variance on this
# box is about 7 percent while the within-run error bars are about 0.3 percent.
# Prefill and decode cells are never compared with each other, so they may
# live in separate invocations.
#
# Decode is measured at an explicit depth on purpose: tg32 at the default
# depth (n_ctx 32) is launch/barrier bound on this card and is NOT a valid
# decode metric (it once read -36% between two builds that were at parity at
# tg256). Decode rows are labelled with their depth (tg256@d4096); old tg32
# rows in results.txt are invalid-metric history.
#
# Flag traps, learned the hard way:
# * -tb is NOT a llama-bench flag. It exists on llama-server only.
@@ -30,36 +43,42 @@ fi
RESULTS="${RESULTS:-/root/bench/results.txt}"
MODEL="${MODEL:-/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf}"
LB_P="${LB_P:-2048,8192}"
LB_N="${LB_N:-32}"
LB_N="${LB_N:-256}"
LB_D="${LB_D:-4096}"
LB_R="${LB_R:-3}"
STAMP=$(date -u +%Y-%m-%dT%H:%M:%S)
export LD_LIBRARY_PATH="$BUILD_DIR/bin"
export LABEL BUILD_DIR STAMP RESULTS
OUT=$("$BUILD_DIR/bin/llama-bench" \
-m "$MODEL" \
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none \
-p "$LB_P" -n "$LB_N" -r "$LB_R" -o json 2>/dev/null)
lb() {
"$BUILD_DIR/bin/llama-bench" \
-m "$MODEL" \
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none \
-r "$LB_R" -o json "$@" 2>/dev/null
}
if [ -z "$OUT" ]; then
echo "ERROR: llama-bench produced no output for $LABEL (bad flag?)" >&2
exit 3
fi
printf '%s' "$OUT" | python3 -c '
record() { # $1 = what, stdin = llama-bench JSON
WHAT="$1" python3 -c '
import json, sys, os
label = os.environ["LABEL"]
build = os.environ["BUILD_DIR"]
stamp = os.environ["STAMP"]
results = os.environ["RESULTS"]
rows = json.load(sys.stdin)
what = os.environ["WHAT"]
text = sys.stdin.read()
if not text.strip():
sys.stderr.write("ERROR: llama-bench produced no output for %s %s (bad flag?)\n" % (label, what))
sys.exit(3)
rows = json.loads(text)
if not rows:
sys.stderr.write("ERROR: llama-bench returned zero rows\n")
sys.stderr.write("ERROR: llama-bench returned zero rows for %s %s\n" % (label, what))
sys.exit(3)
out = open(results, "a")
for r in rows:
test = "pp%d" % r["n_prompt"] if r["n_prompt"] else "tg%d" % r["n_gen"]
if r.get("n_depth", 0):
test += "@d%d" % r["n_depth"]
line = ("LBRESULT label=%s build=%s test=%s tps=%.2f stddev=%.2f "
"samples=%s time=%s"
% (label, build, test, r["avg_ts"], r["stddev_ts"],
@@ -68,3 +87,9 @@ for r in rows:
print(line)
out.close()
'
}
lb -p "$LB_P" -n 0 | record prefill
if [ "$LB_N" -gt 0 ]; then
lb -p 0 -n "$LB_N" -d "$LB_D" | record decode
fi
+9 -3
View File
@@ -19,6 +19,7 @@ RE_CORPUS = re.compile(
r"prompt_n=(\d+) rep=\d+ prompt_ms=([\d.]+) tps=([\d.]+)")
RE_LB = re.compile(
r"^LBRESULT label=(\S+) build=\S+ test=(\S+) tps=([\d.]+)")
RE_TG_NODEPTH = re.compile(r"^tg\d+-llama-bench$")
def main():
@@ -55,13 +56,13 @@ def main():
order.sort(key=sort_key)
w = max(len(l) for l in labels) + 2
head = "metric".ljust(20) + "".join(l.ljust(max(w, 24)) for l in labels)
head = "metric".ljust(26) + "".join(l.ljust(max(w, 24)) for l in labels)
print("")
print("t/s, median (min-max), n samples")
print(head)
print("-" * len(head))
for metric in order:
row = metric.ljust(20)
row = metric.ljust(26)
for label in labels:
vals = data[metric].get(label)
if not vals:
@@ -71,6 +72,9 @@ def main():
median(vals), min(vals), max(vals), len(vals))
row += cell.ljust(max(w, 24))
print(row)
if any(RE_TG_NODEPTH.match(m) for m in order):
print("note: tgN rows without @d<depth> ran at n_ctx ~N (old default);"
" INVALID decode metric on this box, history only")
if len(labels) >= 2:
base = labels[0]
@@ -87,7 +91,9 @@ def main():
ma, mb = median(a), median(b)
pct = (mb - ma) / ma * 100.0
verdict = "SIGNIFICANT" if abs(pct) >= NOISE_FLOOR_PCT else "unproven"
print(" %-20s %+6.1f%% %s" % (metric, pct, verdict))
if RE_TG_NODEPTH.match(metric):
verdict = "INVALID metric (tg at n_ctx ~N), ignore"
print(" %-26s %+6.1f%% %s" % (metric, pct, verdict))
return 0