Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

scripts: add the RX 580 benchmark harness

Browse Source 
Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
Lumpiasty
2026-09-10 17:06:01 +02:00
co-authored by Claude Opus 5
parent c233ce9b51
commit f9a5c231ed
6 changed files with 861 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/rx580-bench/README.md
+200
View File
@@ -0,0 +1,200 @@
# rx580-bench
Prompt-processing benchmark harness for the RX 580 (Polaris) serving pod.
It exists so that no future session has to rediscover how to measure this
machine. It encodes the production config, fixed corpus slices, the repeat
and median discipline, and the noise floor.
## What it measures
Two things, deliberately:
1. **Corpus prompt processing** (the number we actually care about). Real
Polish prose from `pan-tadeusz.txt` pushed through `llama-server`'s
`/completion` endpoint with `n_predict: 1`, `cache_prompt: false`. This
mimics agentic tool-result parsing: a big blob of real text arriving cold.
Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k
and 32k tokens.
2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random
tokens, `pp2048`, `pp8192`, `tg32`. This is what prior work on this box
recorded, so it is the continuity metric.
They do not measure the same thing and they do not agree. `llama-bench`
pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
growing KV cache, so they fall off with prompt length. Both are useful.
## How to run it
One command from the workstation:
```
./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean
```
Interleaved A/B, which is the only valid way to compare two builds:
```
./scripts/rx580-bench/run.sh \
--build /root/arms/new-clean/build --label new-clean \
--build-b /root/arms/new-fork/build --label-b new-fork \
--rounds 3
```
Summarize whatever has accumulated:
```
./scripts/rx580-bench/run.sh --summarize
```
`run.sh` copies the harness onto the pod and execs it there. The wrapper is
thin on purpose: all the methodology lives in the pod-side scripts, so it can
also be driven directly on the pod if kubectl is inconvenient.
The run is detached on the pod and then tailed, so Ctrl-C on the wrapper does
not orphan a half-finished run. `bench.sh` restarts `llama-swap` from an EXIT
trap on success, failure and interrupt alike.
### Pod access
```
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
```
The pod has `curl` and `python3` but **no `jq`**; the harness uses python3 for
all JSON.
## Files
| file | side | what |
|---|---|---|
| `run.sh` | workstation | one-command wrapper; installs and execs the rest |
| `bench.sh` | pod | orchestrator: stops llama-swap, waits for idle GPU, runs both harnesses, restarts llama-swap via trap |
| `ppbench.py` | pod | the corpus prompt-processing harness |
| `lbsweep.sh` | pod | the llama-bench sweep |
| `summarize.py` | pod | turns `results.txt` into a median/min-max table with delta significance |
Results append to `/root/bench/results.txt` as parseable `RESULT` /
`SUMMARY` / `LBRESULT` lines. Server logs land in `/root/bench/server-*.log`.
## The config it assumes
Production serving config, matching `/root/config.yaml`:
```
-t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --no-mmap
--ctx-size 40960 --no-warmup
```
Model:
```
/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
```
Qwen3.6-35B-A3B, 20.8 GiB, 40 layers, 256 experts of which 8 are used.
Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16,
12 CPU cores, 62 GB RAM.
The `llama-bench` equivalent of that config is:
```
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -mmp 0 -p 2048,8192 -n 32 -r 3
```
## The corpus slices are constants, not recalibrated
`pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this
model's tokenizer it runs **2.6702 chars/token**, and `/completion` reports
`prompt_n` identical to `/tokenize` (BOS offset 0).
The slices are `corpus[0:NCHARS]` and are baked into `ppbench.py`:
| target tokens | chars | utf-8 bytes |
|---|---|---|
| 4096 | 10776 | 11608 |
| 16384 | 43858 | 47175 |
| 32768 | 87165 | 93843 |
These hit the target `prompt_n` exactly. Every build is therefore measured on
byte-identical input. `ppbench.py` warns loudly if the reported `prompt_n`
ever stops matching the target, which is the signal that the table is stale.
Only recalibrate (`ppbench.py <build> <label> --calibrate`) if the model file
or the corpus changes.
## Reading the numbers
* `prompt_n` is the token count actually prefilled. It must equal the target.
* `prompt_ms` is wall-clock prefill time.
* `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better.
* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0.
* `tg32` is token generation at DEFAULT depth (n_ctx 32). **It is NOT a valid
decode metric on this box.** At that size decode is launch/barrier bound, not
compute bound, and measures a regime serving never reaches: it read -36%
between two upstream commits that are at parity in real use (the cause was
the view-alias fix in ggml_vk_graph_optimize). For decode use **tg256 with an
explicit depth**, e.g. `-p 0 -n 256 -d 2048`. tg is also NOT monotonic in
n_ctx here - on one binary n_ctx 256 measured slower than both 32 and 544 -
so only ever compare same-depth cells.
## Noise floor: deltas under about 5 percent are unproven
This is the single most important thing in this directory.
`llama-bench`'s within-run error bars understate **cross-invocation** variance
by roughly **14x**. The same config measured `276.7 +/- 0.9` in one invocation
and `296.4 +/- 0.6` in another, 7 percent apart, with error bars that claimed
0.3 percent precision.
Therefore:
* Any comparison you want to draw a conclusion from must live **inside a
single `llama-bench` invocation**, using comma-separated sweeps.
* Two different *builds* cannot share an invocation, so run them
**interleaved A/B/A/B/A/B across at least 3 rounds** and compare medians.
That is what `--build-b` does.
* State explicitly that any cross-build delta under about 5 percent is
unproven. `summarize.py` labels them `unproven` for you.
## Traps
* **`-tb` is not a `llama-bench` flag.** It is a `llama-server` flag. Passing
it makes `llama-bench` print usage and exit silently, mid-sweep, which looks
exactly like a run that produced nothing. Always check the row count.
* **A ubatch larger than the prompt never fills.** With `-ub 2048`, prompt
lengths must be multiples of 2048 or the sweep measures nothing meaningful.
* **Nothing else may touch the GPU.** `bench.sh` stops `llama-swap` and
refuses to start if any `llama-*` process is still alive.
* **Never leave the pod not serving.** `bench.sh` restarts `llama-swap` from
an EXIT trap. If you bypass the script, restart it by hand:
`supervisorctl -c /root/supervisord.conf start llama-swap`.
* **Do not run `llama-cli` detached with closed stdin** on this box; it loops
forever. Use the server harness or `llama-bench -d` for depth timing.
* `--no-mmap` is deprecated upstream in favour of `--load-mode` / `-lm none`,
but still accepted and still what production passes.
## Known-good reference values
If a fresh run disagrees with these by much more than the noise floor,
something is wrong (something else on the GPU, a thermal problem, a bad
build) before you believe you found a speedup.
Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config, current upstream
(`f3f1a8f27`) and the deployed fork, measured 2026-09-08/09:
| metric | clean upstream | fork |
|---|---|---|
| corpus pp @ 4k | about 240 t/s | about 255 t/s |
| corpus pp @ 16k | about 206 t/s | about 225 t/s |
| corpus pp @ 32k | about 171 t/s | about 195 t/s |
| llama-bench pp2048 | see results.txt | about 285 t/s |
| llama-bench tg32 | | not a valid metric, see above |
Older reference points for the same model/config: `pp2048` depth-0 about
285 t/s. The historical "tg32 about 19 t/s" figure is retained only as trivia -
do not treat it as a target, and see the tg32 warning above.
Model load with `--no-mmap` takes about 26 s warm, longer cold.
A full 3-round A/B run of both harnesses takes roughly 100 minutes;
add about 16 minutes per extra arm per round.
scripts/rx580-bench/bench.sh
Executable
+167
View File
@@ -0,0 +1,167 @@
#!/bin/sh
#
# RX 580 prompt-processing benchmark orchestrator. Runs ON THE POD.
#
# Stops llama-swap, waits for the GPU to go idle, runs the real-corpus
# prompt-processing harness and the llama-bench sweep for every arm, appends
# parseable results, and ALWAYS restarts llama-swap on exit (success, error
# or Ctrl-C).
#
# Usage:
# bench.sh --build DIR --label NAME
# [--build-b DIR --label-b NAME] second arm, interleaved
# [--arm DIR:LABEL ...] extra arms, repeatable (N-way)
# [--rounds N] default 3 multi-arm, 1 otherwise
# [--reps N] corpus repeats per size, default 3
# [--sizes 4096,16384,32768]
# [--corpus-only | --lb-only]
# [--results PATH] default /root/bench/results.txt
# [--keep-swap-down] do not restart llama-swap at exit
#
set -e
BENCH_DIR=/root/bench
SUPCONF=/root/supervisord.conf
RESULTS="$BENCH_DIR/results.txt"
MODEL=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
ARMS="" # space separated DIR|LABEL entries
BUILD_A=""; LABEL_A=""
BUILD_B=""; LABEL_B=""
ROUNDS=""
REPS=3
SIZES=4096,16384,32768
DO_CORPUS=1
DO_LB=1
KEEP_SWAP_DOWN=0
while [ $# -gt 0 ]; do
case "$1" in
--build) BUILD_A="$2"; shift 2 ;;
--label) LABEL_A="$2"; shift 2 ;;
--build-b) BUILD_B="$2"; shift 2 ;;
--label-b) LABEL_B="$2"; shift 2 ;;
--arm) ARMS="$ARMS ${2%%:*}|${2#*:}"; shift 2 ;;
--rounds) ROUNDS="$2"; shift 2 ;;
--reps) REPS="$2"; shift 2 ;;
--sizes) SIZES="$2"; shift 2 ;;
--results) RESULTS="$2"; shift 2 ;;
--corpus-only) DO_LB=0; shift ;;
--lb-only) DO_CORPUS=0; shift ;;
--keep-swap-down) KEEP_SWAP_DOWN=1; shift ;;
-h|--help) sed -n '2,20p' "$0"; exit 0 ;;
*) echo "unknown arg: $1" >&2; exit 2 ;;
esac
done
if [ -n "$BUILD_A" ] && [ -z "$LABEL_A" ]; then
echo "error: --build needs --label" >&2; exit 2
fi
if [ -n "$BUILD_B" ] && [ -z "$LABEL_B" ]; then
echo "error: --build-b needs --label-b" >&2; exit 2
fi
# --build/--label and --build-b/--label-b are sugar for the first two arms.
if [ -n "$BUILD_A" ]; then ARMS="$BUILD_A|$LABEL_A $ARMS"; fi
if [ -n "$BUILD_B" ]; then ARMS="$ARMS $BUILD_B|$LABEL_B"; fi
ARMS=$(echo $ARMS)
if [ -z "$ARMS" ]; then
echo "error: need at least one arm (--build/--label or --arm DIR:LABEL)" >&2
exit 2
fi
NARMS=$(echo "$ARMS" | wc -w)
if [ -z "$ROUNDS" ]; then
if [ "$NARMS" -gt 1 ]; then ROUNDS=3; else ROUNDS=1; fi
fi
for e in $ARMS; do
d=${e%%|*}
if [ ! -x "$d/bin/llama-server" ] || [ ! -x "$d/bin/llama-bench" ]; then
echo "error: $d lacks bin/llama-server or bin/llama-bench" >&2
exit 2
fi
done
cat <<'WARN'
-------------------------------------------------------------------------
RX 580 benchmark. Read this before trusting any number.
* Cross-invocation variance on this box is about 7 percent. llama-bench
within-run error bars understate it by roughly 14x. Any cross-build
delta under about 5 percent is UNPROVEN noise.
* Comparisons you care about must live inside ONE llama-bench invocation
(comma-separated sweeps), or be interleaved across at least 3 rounds.
That is what multiple arms plus --rounds does.
* Nothing else may touch the GPU while this runs. llama-swap is stopped
for the duration and restarted on exit, including on error or Ctrl-C.
* Traps: -tb is NOT a llama-bench flag. A ubatch larger than -p never
fills, so prompt lengths must be multiples of the 2048 ubatch. A bad
flag makes llama-bench print usage and exit silently mid-sweep.
-------------------------------------------------------------------------
WARN
echo "[bench] $NARMS arm(s), $ROUNDS round(s): $ARMS"
restore_swap() {
rc=$?
if [ "$KEEP_SWAP_DOWN" -eq 0 ]; then
echo ""
echo "[bench] restarting llama-swap"
supervisorctl -c "$SUPCONF" start llama-swap || true
supervisorctl -c "$SUPCONF" status llama-swap || true
else
echo "[bench] --keep-swap-down: llama-swap left STOPPED"
fi
exit $rc
}
trap restore_swap EXIT INT TERM
echo "[bench] stopping llama-swap"
supervisorctl -c "$SUPCONF" stop llama-swap || true
wait_gpu_idle() {
i=0
while [ $i -lt 60 ]; do
if ! pgrep -f 'bin/llama-server|bin/llama-bench|bin/llama-cli' >/dev/null 2>&1; then
return 0
fi
sleep 2
i=$((i+1))
done
echo "error: something is still on the GPU:" >&2
pgrep -af 'bin/llama-server|bin/llama-bench|bin/llama-cli' >&2
exit 3
}
echo "[bench] waiting for the GPU to go idle"
wait_gpu_idle
echo "[bench] GPU idle"
run_corpus() {
[ "$DO_CORPUS" -eq 1 ] || return 0
echo "===== corpus $2 ====="
wait_gpu_idle
python3 "$BENCH_DIR/ppbench.py" "$1" "$2" \
--sizes "$SIZES" --reps "$REPS" --results "$RESULTS"
}
run_lb() {
[ "$DO_LB" -eq 1 ] || return 0
echo "===== llama-bench $2 ====="
wait_gpu_idle
RESULTS="$RESULTS" MODEL="$MODEL" sh "$BENCH_DIR/lbsweep.sh" "$1" "$2"
}
# Interleave: every arm is measured once per round, so slow drift in the
# machine hits all arms roughly equally instead of biasing whichever ran first.
r=1
while [ "$r" -le "$ROUNDS" ]; do
echo ""
echo "########## ROUND $r / $ROUNDS ##########"
for e in $ARMS; do run_corpus "${e%%|*}" "${e#*|}"; done
for e in $ARMS; do run_lb "${e%%|*}" "${e#*|}"; done
r=$((r+1))
done
echo ""
echo "ALL_RUNS_DONE"
echo "[bench] raw results appended to $RESULTS"
python3 "$BENCH_DIR/summarize.py" "$RESULTS" || true
scripts/rx580-bench/lbsweep.sh
Executable
+69
View File
@@ -0,0 +1,69 @@
#!/bin/sh
#
# llama-bench sweep, the controlled cross-check for the corpus harness.
# Runs ON THE POD. Normally invoked by bench.sh.
#
# Usage: lbsweep.sh <build_dir> <label>
# Env overrides: RESULTS, MODEL, LB_P, LB_N, LB_R
#
# Everything that must be compared lives inside ONE llama-bench invocation
# (-p takes a comma-separated list), because cross-invocation variance on this
# box is about 7 percent while the within-run error bars are about 0.3 percent.
#
# Flag traps, learned the hard way:
# * -tb is NOT a llama-bench flag. It exists on llama-server only.
# * -ncmoe is the llama-bench spelling of --n-cpu-moe.
# * -mmp 0 is the llama-bench equivalent of the server's --no-mmap.
# * A bad flag makes llama-bench print usage and exit silently, which looks
# exactly like a sweep that produced no rows. Always check row count.
# * A ubatch larger than -p never fills, so keep -p a multiple of -ub.
#
set -e
BUILD_DIR="$1"
LABEL="$2"
if [ -z "$BUILD_DIR" ] || [ -z "$LABEL" ]; then
echo "usage: $0 <build_dir> <label>" >&2
exit 2
fi
RESULTS="${RESULTS:-/root/bench/results.txt}"
MODEL="${MODEL:-/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf}"
LB_P="${LB_P:-2048,8192}"
LB_N="${LB_N:-32}"
LB_R="${LB_R:-3}"
STAMP=$(date -u +%Y-%m-%dT%H:%M:%S)
export LD_LIBRARY_PATH="$BUILD_DIR/bin"
export LABEL BUILD_DIR STAMP RESULTS
OUT=$("$BUILD_DIR/bin/llama-bench" \
-m "$MODEL" \
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -mmp 0 \
-p "$LB_P" -n "$LB_N" -r "$LB_R" -o json 2>/dev/null)
if [ -z "$OUT" ]; then
echo "ERROR: llama-bench produced no output for $LABEL (bad flag?)" >&2
exit 3
fi
printf '%s' "$OUT" | python3 -c '
import json, sys, os
label = os.environ["LABEL"]
build = os.environ["BUILD_DIR"]
stamp = os.environ["STAMP"]
results = os.environ["RESULTS"]
rows = json.load(sys.stdin)
if not rows:
sys.stderr.write("ERROR: llama-bench returned zero rows\n")
sys.exit(3)
out = open(results, "a")
for r in rows:
test = "pp%d" % r["n_prompt"] if r["n_prompt"] else "tg%d" % r["n_gen"]
line = ("LBRESULT label=%s build=%s test=%s tps=%.2f stddev=%.2f "
"samples=%s time=%s"
% (label, build, test, r["avg_ts"], r["stddev_ts"],
r["samples_ts"], stamp))
out.write(line + "\n")
print(line)
out.close()
'
scripts/rx580-bench/ppbench.py
Executable
+239
View File
@@ -0,0 +1,239 @@
#!/usr/bin/env python3
"""
Real-corpus prompt-processing benchmark for llama-server on the RX 580 pod.
Runs ON THE POD. Normally invoked by bench.sh, not directly.
Starts llama-server from a given build directory with the production config,
waits for /health, POSTs fixed slices of the Pan Tadeusz corpus to /completion,
records timings.prompt_n / prompt_ms / prompt_per_second, then shuts the server
down cleanly.
Prompt slices are FIXED CONSTANTS (see SLICES below), calibrated once against
the Qwen3.6-35B-A3B tokenizer, so that every build is measured on byte-identical
input. Do not re-calibrate for a normal run; use --calibrate only if the model
or the corpus changes.
Usage:
ppbench.py <build_dir> <label> [--sizes 4096,16384,32768] [--reps 3]
[--results PATH] [--calibrate]
"""
import json
import os
import signal
import subprocess
import sys
import time
import urllib.request
from statistics import median
BENCH_DIR = "/root/bench"
CORPUS = os.path.join(BENCH_DIR, "pan-tadeusz.txt")
DEFAULT_RESULTS = os.path.join(BENCH_DIR, "results.txt")
MODEL = ("/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/"
"snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/"
"Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf")
PORT = 8099
BASE = "http://127.0.0.1:%d" % PORT
# Production serving config. Keep in sync with /root/config.yaml (llama-swap).
# Traps encoded here on purpose:
# -b / -ub 2048 : a ubatch larger than the prompt never fills, so every
# prompt length below must stay a multiple of 2048.
# --no-mmap : deprecated upstream in favour of --load-mode / -lm none,
# but still accepted; it is what production passes today.
SERVER_ARGS = ["-t", "6", "-ngl", "99", "--n-cpu-moe", "40",
"-b", "2048", "-ub", "2048", "-fa", "1", "--no-mmap",
"--ctx-size", "40960", "--no-warmup",
"--port", str(PORT), "-m", MODEL]
# Calibrated once on 2026-09-08 against Qwen3.6-35B-A3B-UD-Q4_K_XL:
# Polish text of pan-tadeusz.txt runs 2.6702 chars/token, and /completion
# reports prompt_n identical to /tokenize (BOS offset 0).
# Each slice is corpus[0:NCHARS] read as UTF-8 text; NBYTES is the resulting
# UTF-8 byte length, recorded so the slice can be reproduced with byte tools.
# These produce EXACTLY the target prompt_n. Do not edit without recalibrating.
SLICES = {
4096: {"chars": 10776, "bytes": 11608},
16384: {"chars": 43858, "bytes": 47175},
32768: {"chars": 87165, "bytes": 93843},
}
DEFAULT_SIZES = [4096, 16384, 32768]
def post(path, payload, timeout=2400):
req = urllib.request.Request(
BASE + path,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST")
with urllib.request.urlopen(req, timeout=timeout) as r:
return json.loads(r.read().decode("utf-8"))
def health_ok():
try:
with urllib.request.urlopen(BASE + "/health", timeout=5) as r:
return json.loads(r.read().decode("utf-8")).get("status") == "ok"
except Exception:
return False
def start_server(build_dir, logpath):
env = dict(os.environ)
env["LD_LIBRARY_PATH"] = os.path.join(build_dir, "bin")
binary = os.path.join(build_dir, "bin", "llama-server")
if not os.path.exists(binary):
raise RuntimeError("no llama-server at %s" % binary)
log = open(logpath, "wb")
p = subprocess.Popen([binary] + SERVER_ARGS, stdout=log,
stderr=subprocess.STDOUT, env=env,
start_new_session=True)
deadline = time.time() + 1200
while time.time() < deadline:
if p.poll() is not None:
raise RuntimeError("llama-server exited early rc=%s, see %s"
% (p.returncode, logpath))
if health_ok():
return p
time.sleep(2)
stop_server(p)
raise RuntimeError("llama-server not healthy within 1200s, see %s" % logpath)
def stop_server(p):
if p is None or p.poll() is not None:
return
try:
os.killpg(os.getpgid(p.pid), signal.SIGTERM)
except Exception:
p.terminate()
for _ in range(120):
if p.poll() is not None:
return
time.sleep(1)
try:
os.killpg(os.getpgid(p.pid), signal.SIGKILL)
except Exception:
p.kill()
p.wait()
def n_tokens(text):
"""Exact token count via /tokenize. Cheap: no prefill."""
return len(post("/tokenize", {"content": text}, timeout=300)["tokens"])
def calibrate(corpus, targets):
"""Recompute the SLICES table. Only needed if model or corpus changes."""
probe = corpus[:2000]
tk = n_tokens(probe)
r = post("/completion", {"prompt": probe, "n_predict": 1,
"cache_prompt": False, "temperature": 0})
offset = r["timings"]["prompt_n"] - tk
cpt = len(probe) / float(tk)
sys.stderr.write("calibrate: %.4f chars/token, prompt_n offset %d\n"
% (cpt, offset))
for target in targets:
want = target - offset
lo, hi = 1, len(corpus)
guess = min(len(corpus), max(1, int(want * cpt)))
best = None
for _ in range(60):
got = n_tokens(corpus[:guess])
if got == want:
best = guess
break
if got < want:
lo = guess + 1
else:
hi = guess - 1
if lo > hi:
break
guess = (lo + hi) // 2
if best is None:
best = guess
nbytes = len(corpus[:best].encode("utf-8"))
sys.stderr.write("calibrate: %d tokens -> chars=%d bytes=%d\n"
% (target, best, nbytes))
def main():
if len(sys.argv) < 3:
sys.stderr.write(__doc__)
return 2
build_dir = os.path.abspath(sys.argv[1])
label = sys.argv[2]
sizes = list(DEFAULT_SIZES)
reps = 3
results_path = DEFAULT_RESULTS
do_calibrate = "--calibrate" in sys.argv
args = sys.argv[3:]
for i, a in enumerate(args):
if a == "--sizes":
sizes = [int(x) for x in args[i + 1].split(",")]
elif a == "--reps":
reps = int(args[i + 1])
elif a == "--results":
results_path = args[i + 1]
corpus = open(CORPUS, encoding="utf-8").read()
logpath = os.path.join(BENCH_DIR, "server-%s-%d.log" % (label, int(time.time())))
p = None
try:
sys.stderr.write("[%s] starting llama-server from %s\n" % (label, build_dir))
t0 = time.time()
p = start_server(build_dir, logpath)
sys.stderr.write("[%s] healthy after %.1fs\n" % (label, time.time() - t0))
if do_calibrate:
calibrate(corpus, sizes)
return 0
stamp = time.strftime("%Y-%m-%dT%H:%M:%S")
out = open(results_path, "a")
for target in sizes:
if target not in SLICES:
sys.stderr.write("no calibrated slice for %d tokens, skipping\n" % target)
continue
nchars = SLICES[target]["chars"]
prompt = corpus[:nchars]
rates, mss, ns = [], [], set()
for rep in range(1, reps + 1):
r = post("/completion", {"prompt": prompt, "n_predict": 1,
"cache_prompt": False, "temperature": 0})
t = r["timings"]
rates.append(t["prompt_per_second"])
mss.append(t["prompt_ms"])
ns.add(t["prompt_n"])
line = ("RESULT corpus label=%s build=%s target=%d chars=%d "
"prompt_n=%d rep=%d prompt_ms=%.2f tps=%.2f time=%s"
% (label, build_dir, target, nchars, t["prompt_n"], rep,
t["prompt_ms"], t["prompt_per_second"], stamp))
out.write(line + "\n")
out.flush()
sys.stderr.write(line + "\n")
if ns != {target}:
sys.stderr.write("WARNING: prompt_n %s != target %d; slice table "
"is stale, rerun with --calibrate\n" % (sorted(ns), target))
s = ("SUMMARY corpus label=%s target=%d prompt_n=%s median_tps=%.2f "
"min_tps=%.2f max_tps=%.2f median_ms=%.1f time=%s"
% (label, target, sorted(ns), median(rates), min(rates),
max(rates), median(mss), stamp))
out.write(s + "\n")
out.flush()
sys.stderr.write(s + "\n")
out.close()
finally:
stop_server(p)
sys.stderr.write("[%s] server stopped\n" % label)
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/rx580-bench/run.sh
Executable
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env bash
#
# One-command entry point for the RX 580 prompt-processing benchmark.
#
# Runs from your workstation. Copies the pod-side harness into the pod and
# execs it there over kubectl. All the real work happens on the pod; this is
# a thin wrapper so there is exactly one command to remember.
#
# Examples:
# ./scripts/rx580-bench/run.sh --build /root/llama.cpp/build --label fork-before
# ./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean \
# --build-b /root/arms/new-fork/build --label-b new-fork
# ./scripts/rx580-bench/run.sh --summarize
# ./scripts/rx580-bench/run.sh --install-only
#
# Every argument other than the wrapper-only flags below is passed straight
# through to bench.sh on the pod. See bench.sh --help.
#
set -euo pipefail
KUBECONFIG_PATH="${KUBECONFIG:-/home/user/Projects/klaster/talos/generated/kubeconfig}"
NS="${RX580_NS:-llama}"
DEPLOY="${RX580_DEPLOY:-deploy/supervisord}"
POD_BENCH_DIR=/root/bench
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export KUBECONFIG="$KUBECONFIG_PATH"
kx() { kubectl -n "$NS" exec -i "$DEPLOY" -- sh -c "$1"; }
install_harness() {
echo "[run] installing harness into $POD_BENCH_DIR"
kx "mkdir -p $POD_BENCH_DIR"
for f in bench.sh ppbench.py lbsweep.sh summarize.py; do
kx "cat > $POD_BENCH_DIR/$f" < "$HERE/$f"
done
kx "chmod +x $POD_BENCH_DIR/bench.sh $POD_BENCH_DIR/lbsweep.sh $POD_BENCH_DIR/ppbench.py $POD_BENCH_DIR/summarize.py"
# The corpus is large and rarely changes; only push it if missing.
if ! kx "test -s $POD_BENCH_DIR/pan-tadeusz.txt" 2>/dev/null; then
if [ -f "$HERE/pan-tadeusz.txt" ]; then
echo "[run] uploading corpus"
kx "cat > $POD_BENCH_DIR/pan-tadeusz.txt" < "$HERE/pan-tadeusz.txt"
else
echo "[run] ERROR: $POD_BENCH_DIR/pan-tadeusz.txt missing on the pod and" >&2
echo " no local copy at $HERE/pan-tadeusz.txt to upload." >&2
exit 4
fi
fi
}
ARGS=()
INSTALL_ONLY=0
SUMMARIZE=0
for a in "$@"; do
case "$a" in
--install-only) INSTALL_ONLY=1 ;;
--summarize) SUMMARIZE=1 ;;
*) ARGS+=("$a") ;;
esac
done
install_harness
if [ "$INSTALL_ONLY" -eq 1 ]; then
echo "[run] harness installed; not running anything"
exit 0
fi
if [ "$SUMMARIZE" -eq 1 ]; then
kx "python3 $POD_BENCH_DIR/summarize.py ${ARGS[*]:-}"
exit 0
fi
if [ ${#ARGS[@]} -eq 0 ]; then
kx "sh $POD_BENCH_DIR/bench.sh --help"
exit 2
fi
# A full 3-round A/B run takes hours; run detached on the pod and tail it, so a
# dropped kubectl connection cannot orphan a half-finished run with llama-swap
# still down. bench.sh restarts llama-swap from its own EXIT trap either way.
STAMP="$(date -u +%Y%m%d-%H%M%S)"
LOG="$POD_BENCH_DIR/run-$STAMP.log"
echo "[run] starting detached run on the pod, log: $LOG"
kx "cd $POD_BENCH_DIR && nohup sh $POD_BENCH_DIR/bench.sh ${ARGS[*]} > $LOG 2>&1 & echo started"
echo "[run] tailing until ALL_RUNS_DONE (safe to Ctrl-C: the pod keeps running)"
kx "i=0; while [ \$i -lt 100000 ]; do
if grep -qE 'ALL_RUNS_DONE|Traceback|error:' $LOG; then break; fi
sleep 15; i=\$((i+1));
done; cat $LOG"
scripts/rx580-bench/summarize.py
Executable
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""
Summarize a results.txt produced by bench.sh into a per-label table.
Usage: summarize.py [results.txt]
Prints median and min-max of every metric for every label, and flags
cross-label deltas against the ~5 percent noise floor of this machine.
"""
import re
import sys
from statistics import median
NOISE_FLOOR_PCT = 5.0
RE_CORPUS = re.compile(
r"^RESULT corpus label=(\S+) build=\S+ target=(\d+) chars=\d+ "
r"prompt_n=(\d+) rep=\d+ prompt_ms=([\d.]+) tps=([\d.]+)")
RE_LB = re.compile(
r"^LBRESULT label=(\S+) build=\S+ test=(\S+) tps=([\d.]+)")
def main():
path = sys.argv[1] if len(sys.argv) > 1 else "/root/bench/results.txt"
# data[metric][label] = list of tps
data = {}
order = []
labels = []
for line in open(path):
m = RE_CORPUS.match(line)
if m:
label, target, _pn, _ms, tps = m.groups()
metric = "pp%s-corpus" % target
else:
m = RE_LB.match(line)
if not m:
continue
label, test, tps = m.groups()
metric = "%s-llama-bench" % test
if metric not in data:
data[metric] = {}
order.append(metric)
data[metric].setdefault(label, []).append(float(tps))
if label not in labels:
labels.append(label)
if not data:
print("no parseable results in %s" % path)
return 1
def sort_key(m):
n = re.search(r"(\d+)", m)
return (0 if "corpus" in m else 1, int(n.group(1)) if n else 0)
order.sort(key=sort_key)
w = max(len(l) for l in labels) + 2
head = "metric".ljust(20) + "".join(l.ljust(max(w, 24)) for l in labels)
print("")
print("t/s, median (min-max), n samples")
print(head)
print("-" * len(head))
for metric in order:
row = metric.ljust(20)
for label in labels:
vals = data[metric].get(label)
if not vals:
row += "-".ljust(max(w, 24))
else:
cell = "%.1f (%.1f-%.1f) n=%d" % (
median(vals), min(vals), max(vals), len(vals))
row += cell.ljust(max(w, 24))
print(row)
if len(labels) >= 2:
base = labels[0]
print("")
print("deltas vs %s (noise floor %.0f%%, anything under it is UNPROVEN)"
% (base, NOISE_FLOOR_PCT))
for other in labels[1:]:
print(" %s vs %s:" % (other, base))
for metric in order:
a = data[metric].get(base)
b = data[metric].get(other)
if not a or not b:
continue
ma, mb = median(a), median(b)
pct = (mb - ma) / ma * 100.0
verdict = "SIGNIFICANT" if abs(pct) >= NOISE_FLOOR_PCT else "unproven"
print(" %-20s %+6.1f%% %s" % (metric, pct, verdict))
return 0
if __name__ == "__main__":
sys.exit(main())