scripts: rx580-bench - decode at tg256 depth 4096, not tg32
The llama-bench sweep measured decode as tg32 at the default depth and the README called it the continuity metric, while tg32 at n_ctx 32 is launch / barrier bound on this card and is not a valid decode metric (it once read -36% between builds at parity at tg256). lbsweep.sh now runs prefill (-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as two invocations and labels depth cells (tg256@d4096). summarize.py marks old depth-less tg rows as invalid-metric history. The README examples and the reference values are updated to the 2026-10-24 production numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f664b0f20e
commit
62d442e7bf
3 files changed
+99
-37
No files matched your search
@@ -17,8 +17,12 @@ Two things, deliberately:
|
||||
Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k
|
||||
and 32k tokens.
|
||||
2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random
|
||||
tokens, `pp2048`, `pp8192`, `tg32`. This is what prior work on this box
|
||||
recorded, so it is the continuity metric.
|
||||
tokens: prefill `pp2048`, `pp8192` at depth 0, and decode `tg256@d4096`
|
||||
(256 generated tokens on top of a 4096-token context, `-p 0 -n 256
|
||||
-d 4096`). The prefill cells are the continuity metric with prior work on
|
||||
this box. Decode used to be `tg32` at the default depth; that is not a valid
|
||||
metric here (see "Reading the numbers"), and old `tg32` rows in
|
||||
`results.txt` are history only.
|
||||
|
||||
They do not measure the same thing and they do not agree. `llama-bench`
|
||||
pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
|
||||
@@ -100,9 +104,14 @@ Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16,
|
||||
The `llama-bench` equivalent of that config is:
|
||||
|
||||
```
|
||||
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 32 -r 3
|
||||
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 0 -r 3
|
||||
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 0 -n 256 -d 4096 -r 3
|
||||
```
|
||||
|
||||
`lbsweep.sh` runs these as two invocations (prefill, then decode). Override
|
||||
with `LB_P` (prefill sizes), `LB_N` / `LB_D` (decode tokens / depth; `LB_N=0`
|
||||
skips decode) and `LB_R` (repetitions).
|
||||
|
||||
## The corpus slices are constants, not recalibrated
|
||||
|
||||
`pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this
|
||||
@@ -128,15 +137,22 @@ or the corpus changes.
|
||||
* `prompt_n` is the token count actually prefilled. It must equal the target.
|
||||
* `prompt_ms` is wall-clock prefill time.
|
||||
* `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better.
|
||||
* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0.
|
||||
* `tg32` is token generation at DEFAULT depth (n_ctx 32). **It is NOT a valid
|
||||
decode metric on this box.** At that size decode is launch/barrier bound, not
|
||||
compute bound, and measures a regime serving never reaches: it read -36%
|
||||
between two upstream commits that are at parity in real use (the cause was
|
||||
the view-alias fix in ggml_vk_graph_optimize). For decode use **tg256 with an
|
||||
explicit depth**, e.g. `-p 0 -n 256 -d 2048`. tg is also NOT monotonic in
|
||||
n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 -
|
||||
so only ever compare same-depth cells.
|
||||
* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0. A depth
|
||||
suffix (`pp2048@d16384`) means the prompt is processed on top of a context
|
||||
of that many tokens.
|
||||
* `tg256@d4096` is the decode metric: 256 generated tokens at depth 4096
|
||||
(`-p 0 -n 256 -d 4096`). Decode is CPU-bound with `--n-cpu-moe 40` and
|
||||
swings with node CPU contention (about 60 co-tenant pods), so compare
|
||||
medians across interleaved rounds and expect occasional catastrophic
|
||||
samples.
|
||||
* `tg32` (old rows, no depth suffix) is token generation at the DEFAULT depth
|
||||
(n_ctx 32). **It is NOT a valid decode metric on this box.** At that size
|
||||
decode is launch/barrier bound, not compute bound, and measures a regime
|
||||
serving never reaches: it read -36% between two upstream commits that are at
|
||||
parity at tg256 (the cause was the view-alias fix in
|
||||
ggml_vk_graph_optimize). tg is also NOT monotonic in n_ctx here - on one
|
||||
binary n_ctx 256 measured slower than both 32 and 544 - so only ever compare
|
||||
same-depth cells. `summarize.py` flags any depth-less tg row.
|
||||
|
||||
## Noise floor: deltas under about 5 percent are unproven
|
||||
|
||||
@@ -196,21 +212,36 @@ If a fresh run disagrees with these by much more than the noise floor,
|
||||
something is wrong (something else on the GPU, a thermal problem, a bad
|
||||
build) before you believe you found a speedup.
|
||||
|
||||
Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config, current upstream
|
||||
(`f3f1a8f27`) and the deployed fork, measured 2026-09-08/09:
|
||||
Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config.
|
||||
|
||||
llama-bench, current production fork (20261024 rebase, `6c480f7e7`) and the
|
||||
previous production fork (`5751c977b`), measured 2026-10-24 by
|
||||
`fork-validate.sh --perf` (2 interleaved arms each, `-r 2` prefill, `-r 3`
|
||||
decode):
|
||||
|
||||
| metric | about |
|
||||
|---|---|
|
||||
| pp2048 (depth 0) | 274-286 t/s (274 current, 286 previous; the -4% is inside the noise floor) |
|
||||
| pp2048@d16384 | 201-206 t/s |
|
||||
| pp2048@d32768 | 158-160 t/s |
|
||||
| tg256@d4096 | 17.8 t/s |
|
||||
|
||||
Corpus harness, measured 2026-09-08/09 on the then-current upstream
|
||||
(`f3f1a8f27`) and the fork deployed then (not re-measured since):
|
||||
|
||||
| metric | clean upstream | fork |
|
||||
|---|---|---|
|
||||
| corpus pp @ 4k | about 240 t/s | about 255 t/s |
|
||||
| corpus pp @ 16k | about 206 t/s | about 225 t/s |
|
||||
| corpus pp @ 32k | about 171 t/s | about 195 t/s |
|
||||
| llama-bench pp2048 | see results.txt | about 285 t/s |
|
||||
| llama-bench tg32 | | not a valid metric, see above |
|
||||
| llama-bench pp8192 | about 222 t/s | about 246 t/s |
|
||||
|
||||
Older reference points for the same model/config: `pp2048` depth-0 about
|
||||
285 t/s. The historical "tg32 about 19 t/s" figure is retained only as trivia -
|
||||
do not treat it as a target, and see the tg32 warning above.
|
||||
Invalid-metric history: the 2026-09 runs recorded `tg32` (7-15 t/s, swinging
|
||||
by run) and older notes mention "tg32 about 19 t/s". Neither is a decode
|
||||
reference; see the tg32 warning above.
|
||||
|
||||
Model load with `-lm none` takes about 26 s warm, longer cold.
|
||||
A full 3-round A/B run of both harnesses takes roughly 100 minutes;
|
||||
add about 16 minutes per extra arm per round.
|
||||
add about 16 minutes per extra arm per round. The tg256@d4096 decode cell
|
||||
costs about 1.2 minutes per arm per round (model load, the 4096-token depth
|
||||
prefill and 3 x 256 tokens), measured 2026-10-24.
|
||||
@@ -4,11 +4,24 @@
|
||||
# Runs ON THE POD. Normally invoked by bench.sh.
|
||||
#
|
||||
# Usage: lbsweep.sh <build_dir> <label>
|
||||
# Env overrides: RESULTS, MODEL, LB_P, LB_N, LB_R
|
||||
# Env overrides: RESULTS, MODEL, LB_P, LB_N, LB_D, LB_R
|
||||
#
|
||||
# Two llama-bench invocations per call:
|
||||
# prefill -p $LB_P -n 0 at depth 0 (default pp2048,pp8192)
|
||||
# decode -p 0 -n $LB_N -d $LB_D (default tg256 @ d4096)
|
||||
# LB_N=0 skips the decode invocation.
|
||||
#
|
||||
# Everything that must be compared lives inside ONE llama-bench invocation
|
||||
# (-p takes a comma-separated list), because cross-invocation variance on this
|
||||
# box is about 7 percent while the within-run error bars are about 0.3 percent.
|
||||
# Prefill and decode cells are never compared with each other, so they may
|
||||
# live in separate invocations.
|
||||
#
|
||||
# Decode is measured at an explicit depth on purpose: tg32 at the default
|
||||
# depth (n_ctx 32) is launch/barrier bound on this card and is NOT a valid
|
||||
# decode metric (it once read -36% between two builds that were at parity at
|
||||
# tg256). Decode rows are labelled with their depth (tg256@d4096); old tg32
|
||||
# rows in results.txt are invalid-metric history.
|
||||
#
|
||||
# Flag traps, learned the hard way:
|
||||
# * -tb is NOT a llama-bench flag. It exists on llama-server only.
|
||||
@@ -30,36 +43,42 @@ fi
|
||||
RESULTS="${RESULTS:-/root/bench/results.txt}"
|
||||
MODEL="${MODEL:-/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf}"
|
||||
LB_P="${LB_P:-2048,8192}"
|
||||
LB_N="${LB_N:-32}"
|
||||
LB_N="${LB_N:-256}"
|
||||
LB_D="${LB_D:-4096}"
|
||||
LB_R="${LB_R:-3}"
|
||||
|
||||
STAMP=$(date -u +%Y-%m-%dT%H:%M:%S)
|
||||
export LD_LIBRARY_PATH="$BUILD_DIR/bin"
|
||||
export LABEL BUILD_DIR STAMP RESULTS
|
||||
|
||||
OUT=$("$BUILD_DIR/bin/llama-bench" \
|
||||
-m "$MODEL" \
|
||||
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none \
|
||||
-p "$LB_P" -n "$LB_N" -r "$LB_R" -o json 2>/dev/null)
|
||||
lb() {
|
||||
"$BUILD_DIR/bin/llama-bench" \
|
||||
-m "$MODEL" \
|
||||
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none \
|
||||
-r "$LB_R" -o json "$@" 2>/dev/null
|
||||
}
|
||||
|
||||
if [ -z "$OUT" ]; then
|
||||
echo "ERROR: llama-bench produced no output for $LABEL (bad flag?)" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
printf '%s' "$OUT" | python3 -c '
|
||||
record() { # $1 = what, stdin = llama-bench JSON
|
||||
WHAT="$1" python3 -c '
|
||||
import json, sys, os
|
||||
label = os.environ["LABEL"]
|
||||
build = os.environ["BUILD_DIR"]
|
||||
stamp = os.environ["STAMP"]
|
||||
results = os.environ["RESULTS"]
|
||||
rows = json.load(sys.stdin)
|
||||
what = os.environ["WHAT"]
|
||||
text = sys.stdin.read()
|
||||
if not text.strip():
|
||||
sys.stderr.write("ERROR: llama-bench produced no output for %s %s (bad flag?)\n" % (label, what))
|
||||
sys.exit(3)
|
||||
rows = json.loads(text)
|
||||
if not rows:
|
||||
sys.stderr.write("ERROR: llama-bench returned zero rows\n")
|
||||
sys.stderr.write("ERROR: llama-bench returned zero rows for %s %s\n" % (label, what))
|
||||
sys.exit(3)
|
||||
out = open(results, "a")
|
||||
for r in rows:
|
||||
test = "pp%d" % r["n_prompt"] if r["n_prompt"] else "tg%d" % r["n_gen"]
|
||||
if r.get("n_depth", 0):
|
||||
test += "@d%d" % r["n_depth"]
|
||||
line = ("LBRESULT label=%s build=%s test=%s tps=%.2f stddev=%.2f "
|
||||
"samples=%s time=%s"
|
||||
% (label, build, test, r["avg_ts"], r["stddev_ts"],
|
||||
@@ -68,3 +87,9 @@ for r in rows:
|
||||
print(line)
|
||||
out.close()
|
||||
'
|
||||
}
|
||||
|
||||
lb -p "$LB_P" -n 0 | record prefill
|
||||
if [ "$LB_N" -gt 0 ]; then
|
||||
lb -p 0 -n "$LB_N" -d "$LB_D" | record decode
|
||||
fi
|
||||
@@ -19,6 +19,7 @@ RE_CORPUS = re.compile(
|
||||
r"prompt_n=(\d+) rep=\d+ prompt_ms=([\d.]+) tps=([\d.]+)")
|
||||
RE_LB = re.compile(
|
||||
r"^LBRESULT label=(\S+) build=\S+ test=(\S+) tps=([\d.]+)")
|
||||
RE_TG_NODEPTH = re.compile(r"^tg\d+-llama-bench$")
|
||||
|
||||
|
||||
def main():
|
||||
@@ -55,13 +56,13 @@ def main():
|
||||
order.sort(key=sort_key)
|
||||
|
||||
w = max(len(l) for l in labels) + 2
|
||||
head = "metric".ljust(20) + "".join(l.ljust(max(w, 24)) for l in labels)
|
||||
head = "metric".ljust(26) + "".join(l.ljust(max(w, 24)) for l in labels)
|
||||
print("")
|
||||
print("t/s, median (min-max), n samples")
|
||||
print(head)
|
||||
print("-" * len(head))
|
||||
for metric in order:
|
||||
row = metric.ljust(20)
|
||||
row = metric.ljust(26)
|
||||
for label in labels:
|
||||
vals = data[metric].get(label)
|
||||
if not vals:
|
||||
@@ -71,6 +72,9 @@ def main():
|
||||
median(vals), min(vals), max(vals), len(vals))
|
||||
row += cell.ljust(max(w, 24))
|
||||
print(row)
|
||||
if any(RE_TG_NODEPTH.match(m) for m in order):
|
||||
print("note: tgN rows without @d<depth> ran at n_ctx ~N (old default);"
|
||||
" INVALID decode metric on this box, history only")
|
||||
|
||||
if len(labels) >= 2:
|
||||
base = labels[0]
|
||||
@@ -87,7 +91,9 @@ def main():
|
||||
ma, mb = median(a), median(b)
|
||||
pct = (mb - ma) / ma * 100.0
|
||||
verdict = "SIGNIFICANT" if abs(pct) >= NOISE_FLOOR_PCT else "unproven"
|
||||
print(" %-20s %+6.1f%% %s" % (metric, pct, verdict))
|
||||
if RE_TG_NODEPTH.match(metric):
|
||||
verdict = "INVALID metric (tg at n_ctx ~N), ignore"
|
||||
print(" %-26s %+6.1f%% %s" % (metric, pct, verdict))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user