--perf averaged the two per-arm llama-bench means of each build, so one
catastrophic sample (CPU contention from the ~60 other pods on the node) moved
the result, against the fork's own rule of medians, never means. llama-bench
now writes -o jsonl (every repetition sample) next to the markdown table, and
the gates use the median of all samples of a build pooled over its two arms
(4 per prefill cell, 6 for decode); the table shows min-max and n.
--perf-report TSV re-aggregates a saved <scratch>/perf.tsv without a GPU.
Checked offline on the pod: the 2026-09 A/B samples in results.txt give
pp2048 reference 250.24 (old mean) vs 247.76 (new median), candidate 280.15
vs 276.93; a synthetic 2026-10-24-like set with one 180 t/s candidate sample
read -9.3% (FAIL) the old way and -0.2% (PASS) with medians.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>