Files
llama.cpp/scripts/fork-validate.sh
T
LumpiastyandClaude Opus 5.5 fe57bc93a0 scripts: fork-validate.sh --diskcache - the mmproj cases
Production runs the 35B and gemma with an mmproj, and the first deploy of the
disk prompt cache aborted on that path. Every case had used --no-mmproj. Now
--no-mmproj is in DC_MM, separate from DC_ARGS, and two cases run with the
0.8B's mmproj (--no-mmproj-offload, arbiter env, LLAMA_MMPROJ_ONDEMAND=1),
against their own reference:
- restore-mmproj: turn 1 is saved on SIGTERM and restored after the restart,
  prompt_n equal to the reference, output identical;
- mmproj-image: an image request (no restore, answer "red"), then the text
  chat (restored, identical), then an image again, then SIGTERM with media in
  the slot (no state copied, exit 0, the file count unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 21:38:58 +01:00

1189 lines
73 KiB
Bash

#!/usr/bin/env bash
# fork-validate.sh - validate a candidate build of the RX 580 llama.cpp fork against a reference build.
#
# Runs ON THE POD (Vulkan / RADV, RX 580). Prints PASS/FAIL/SKIP per check plus a summary; exits
# nonzero if any check FAILs. Only touches its own scratch dir under /tmp, its own scratch ports and
# processes it started itself (killed by explicit PID). Build dirs are only read.
#
# fork-validate.sh --build /root/arms/X/build --ref /root/llama.cpp/build [checks] [options]
#
# Checks (no check flag = --ops --concat --arbiter):
# --ops test-backend-ops on Vulkan0 for CONCAT, FLASH_ATTN_EXT, MUL_MAT, MUL_MAT_ID, GET_ROWS
# -> the fork's kernels and upstream's agree with the CPU reference; asserts the case count
# is > 0 because "-b Vulkan" (no index) silently runs nothing and still prints OK.
# --concat Vulkan concat tiled-transpose fast path (ggml_vk_concat, gate src_transposed_2d)
# -> it fires at prefill and NOT at decode on a real hybrid (delta-net) model; it is the
# fork's biggest prefill win, and taking it at decode once cost -17% decode.
# Needs a build carrying the GGML_VK_CONCAT_FASTPATH knob (=0 off, =log report);
# either --build has it or pass --concat-build DIR (candidate + the knob patch).
# --arbiter on-demand VRAM sharing (LLAMA_SLEEP_VRAM_ONLY ...) with two real servers on a scratch
# arena -> a cold server really gives its VRAM back (v-vram-release: the card-wide
# delta over a baseline taken before the scratch servers start must drop by >= 70% of
# what it added warm, and come back on restore; production's own DRM clients are
# re-read each time and the check SKIPs if they moved by > 64 MiB), handoffs never
# strand, release/restore is bit-exact, multimodal survives it,
# and every shutdown path (mid-release, mid-wait, mid-generation, cold, warm, kill -9 of
# a waiter) exits cleanly and hands the GPU token on. These are the past production
# incidents (stranded requests, wedged token, exit 134 on cold teardown).
# Then a-handoff-latency: ordinary handoff latency of --build vs --ref in the same
# session (fresh server pairs, blocks ref,cand,cand,ref x 16 handoffs) -> FAIL if the
# candidate median exceeds the reference median by more than max(25%, 200 ms). Two
# production deploys were lost to arbiter changes that only showed up here.
# Needs >= 1536 MiB free VRAM (FAILs otherwise); beside a warm gemma-26B only about
# 1.2 GiB is free, so then run it in the --perf swapdown window (about 2 minutes).
# --perf llama-bench, candidate vs reference, alternating ref,cand,cand,ref, on the 35B MoE
# -> no prefill regression > 5% at depth 0/16k/32k (FAIL), decode tg256@d4096
# reported; peak VRAM per arm (FAIL if > 95% of the card). Compares MEDIANS of every
# repetition sample pooled over a build's two arms (4 per prefill cell, 6 for decode),
# never means: CPU contention on the node produces single catastrophic samples.
# Needs llama-swap STOPPED (run it under /root/bench/moe/swapdown.sh); refuses to
# run if any other llama process is up. Samples are kept in <scratch>/perf.tsv.
# --moecache MODEL
# one arbiter server with --moe-cache-mib N on a MoE model, forced cold/warm twice
# -> greedy output identical before/after (the expert-cache SLRU is re-initialised on
# release; a stale SLRU maps experts to empty banks and produces garbage).
# --moe-exact MODEL
# the MoE routing-ids readback skip (F1, sched_copy_experts) is exact: MODEL (a MoE,
# e.g. the --perf model) with every expert on the host, greedy, one prompt of >= 2x
# the skip threshold ceil(4 * n_expert / n_expert_used) tokens (read from the GGUF;
# 128 on the 35B) -> candidate output byte-identical to the reference, and, if the
# candidate has the LLAMA_MOE_IDS_SKIP knob, to the candidate with the skip forced off
# (=0); =log proves the skip engaged. Without the knob skip-engaged is SKIPped with the
# batch-size argument. Needs the GPU to itself like --perf (same guard); about 3 model
# loads, so run it in the same swapdown window as --perf.
# --diskcache the disk prompt cache (--cache-disk-path) on the small hybrid model, CPU only (no VRAM;
# runs beside production), plus the GPU path when >= 1.5 GiB VRAM is free
# -> a slot state saved on SIGTERM (plain, arbiter warm, arbiter cold) is restored after a
# restart: only the suffix is processed (prompt_n == a server that never restarted) and
# the greedy answer is identical; no restore for another conversation or another model
# path / n_ctx / KV type; truncated and corrupt files are removed, never used; the size
# limit evicts the oldest file; a write error (RLIMIT_FSIZE) is logged and exit stays 0;
# with an mmproj loaded (as production runs) a text chat is restored the same way, and an
# image prompt is never restored or saved and does not crash.
#
# Options:
# --concat-build DIR build with the GGML_VK_CONCAT_FASTPATH knob for --concat (default: --build)
# --small MODEL small hybrid model for --concat/--arbiter (default: Qwen3.5-0.8B UD-Q4_K_XL)
# --mmproj FILE its multimodal projector (default: mmproj-F16 next to it)
# --perf-model MODEL model for --perf (default: Qwen3.6-35B-A3B UD-Q4_K_XL)
# --moecache-args S extra server args for --moecache (default: "-ngl 99 --n-cpu-moe 30 -c 4096")
# --moecache-mib N cache size for --moecache (default 2048; 256 never hits on a 26B and proves nothing)
# --port-base N scratch ports N..N+2 (default 8091)
# --keep keep the scratch dir even when everything passed (it is kept on any FAIL)
# --moe-exact-args S server args for --moe-exact (default: "-ngl 99 --n-cpu-moe 99 -c 4096 -b 2048
# -ub 2048 -t 6"; keep the experts on the host and -ub >= the prompt)
# --perf-report TSV only re-aggregate a saved <scratch>/perf.tsv of an earlier --perf run and
# apply the same gates (no GPU, no --build/--ref needed); runs nothing else
#
# Requirements: bash, coreutils, awk, sed, grep, curl, flock(1); gdb optional (makes the
# SIGTERM-during-release case deterministic; it is not in the pod image and is lost on a pod restart,
# then the case falls back to polling the log). No python/jq.
#
# Test only: FV_TEST_SHARED_GPU=1 skips the exclusive-GPU guard of --perf / --moe-exact, for running
# the script against stub binaries. Never set it for a real measurement.
set -u
# ---------------------------------------------------------------- defaults
HF=/root/.cache/huggingface/hub
SMALL_DIR=$HF/models--unsloth--Qwen3.5-0.8B-GGUF/snapshots/6ab461498e2023f6e3c1baea90a8f0fe38ab64d0
SMALL=$SMALL_DIR/Qwen3.5-0.8B-UD-Q4_K_XL.gguf
MMPROJ=
PERF_MODEL=$HF/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
MOECACHE_MODEL=
MOECACHE_ARGS="-ngl 99 --n-cpu-moe 30 -c 4096"
MOECACHE_MIB=2048
MOEX_MODEL=
MOEX_ARGS="-ngl 99 --n-cpu-moe 99 -c 4096 -b 2048 -ub 2048 -t 6"
BUILD= ; REF= ; CONCAT_BUILD=
PORT_BASE=8091
KEEP=0
DO_OPS=0 DO_CONCAT=0 DO_ARB=0 DO_PERF=0 DO_MOEC=0 DO_MOEX=0 DO_DC=0
PERF_REPORT=
usage() { awk 'NR > 1 && /^#/ {sub(/^# ?/, ""); print; next} NR > 1 {exit}' "$0"; exit 2; }
while [ $# -gt 0 ]; do
case "$1" in
--build) BUILD=$2; shift 2 ;;
--ref) REF=$2; shift 2 ;;
--ops) DO_OPS=1; shift ;;
--concat) DO_CONCAT=1; shift ;;
--arbiter) DO_ARB=1; shift ;;
--diskcache) DO_DC=1; shift ;;
--perf) DO_PERF=1; shift ;;
--moecache) DO_MOEC=1; MOECACHE_MODEL=$2; shift 2 ;;
--moecache-args) MOECACHE_ARGS=$2; shift 2 ;;
--moecache-mib) MOECACHE_MIB=$2; shift 2 ;;
--moe-exact) DO_MOEX=1; MOEX_MODEL=$2; shift 2 ;;
--moe-exact-args) MOEX_ARGS=$2; shift 2 ;;
--concat-build) CONCAT_BUILD=$2; shift 2 ;;
--small) SMALL=$2; shift 2 ;;
--mmproj) MMPROJ=$2; shift 2 ;;
--perf-model) PERF_MODEL=$2; shift 2 ;;
--port-base) PORT_BASE=$2; shift 2 ;;
--keep) KEEP=1; shift ;;
--perf-report) PERF_REPORT=$(readlink -f "$2"); shift 2 ;;
-h|--help) usage ;;
*) echo "unknown argument: $1" >&2; usage ;;
esac
done
[ -n "$PERF_REPORT" ] || { [ -n "$BUILD" ] && [ -n "$REF" ]; } || usage
if [ $((DO_OPS + DO_CONCAT + DO_ARB + DO_PERF + DO_MOEC + DO_MOEX + DO_DC)) -eq 0 ]; then
DO_OPS=1 DO_CONCAT=1 DO_ARB=1
fi
[ -n "$MMPROJ" ] || MMPROJ=$(dirname "$SMALL")/mmproj-F16.gguf
PA=$PORT_BASE PB=$((PORT_BASE + 1)) PC=$((PORT_BASE + 2))
# ---------------------------------------------------------------- reporting
RESULTS=()
NFAIL=0
ts() { date +%H:%M:%S; }
log() { echo "[$(ts)] $*"; }
pass() { RESULTS+=("PASS $1: $2"); echo "[$(ts)] PASS $1: $2"; }
fail() { RESULTS+=("FAIL $1: $2"); echo "[$(ts)] FAIL $1: $2"; NFAIL=$((NFAIL + 1)); }
skip() { RESULTS+=("SKIP $1: $2"); echo "[$(ts)] SKIP $1: $2"; }
# check NAME CONDITION-RESULT(0/1) MESSAGE
check() { if [ "$2" -eq 0 ]; then pass "$1" "$3"; else fail "$1" "$3"; fi; }
now_ms() { date +%s%3N; }
# ---------------------------------------------------------------- scratch + cleanup
S=$(mktemp -d /tmp/fork-validate.XXXXXX)
ARENA=$S/arena
mkdir -p "$ARENA"
cd "$S" || exit 2 # anything written to a relative path lands in the scratch dir
PIDS=() # "pid:starttime" of every background process we start (starttime guards PID reuse)
pid_alive() { # alive and not a zombie
[ -r "/proc/$1/stat" ] || return 1
local st; st=$(sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | awk '{print $1}') || return 1
[ -n "$st" ] && [ "$st" != Z ]
}
starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | awk '{print $20}'; }
track() { PIDS+=("$1:$(starttime "$1")"); }
ours() { # "pid:starttime" -> pid if that exact process is still alive
local p=${1%%:*} t=${1#*:}
pid_alive "$p" && [ "$(starttime "$p")" = "$t" ] && echo "$p"
}
cleanup() {
local e p
for e in "${PIDS[@]}"; do p=$(ours "$e") && kill -TERM "$p" 2>/dev/null; done
local i=0
while [ $i -lt 50 ]; do
local any=0
for e in "${PIDS[@]}"; do ours "$e" > /dev/null && any=1; done
[ $any -eq 0 ] && break
sleep 0.1; i=$((i + 1))
done
for e in "${PIDS[@]}"; do p=$(ours "$e") && kill -KILL "$p" 2>/dev/null; done
wait 2>/dev/null
}
finish() {
local rc=$?
trap - EXIT INT TERM HUP
cleanup
echo
echo "==================== fork-validate summary ===================="
echo "candidate: $BUILD ($(git -C "$BUILD/.." rev-parse --short HEAD 2>/dev/null || echo '?'))"
echo "reference: $REF ($(git -C "$REF/.." rev-parse --short HEAD 2>/dev/null || echo '?'))"
local r; for r in "${RESULTS[@]}"; do echo "$r"; done
local np ns
np=$(printf '%s\n' "${RESULTS[@]}" | grep -c '^PASS')
ns=$(printf '%s\n' "${RESULTS[@]}" | grep -c '^SKIP')
echo "total: $np PASS, $NFAIL FAIL, $ns SKIP"
if [ $NFAIL -gt 0 ] || [ $KEEP -eq 1 ] || [ $rc -ne 0 ]; then
echo "scratch kept: $S"
else
rm -rf "$S"
fi
[ $NFAIL -gt 0 ] && exit 1
exit $rc
}
trap finish EXIT
trap 'echo "interrupted"; NFAIL=$((NFAIL + 1)); exit 130' INT TERM HUP
# ---------------------------------------------------------------- environment sanity
for v in $(env | sed -n 's/^\(GGML_[A-Za-z0-9_]*\|LLAMA_\(SLEEP\|VRAM\|MMPROJ\|ARG\|CACHE\|LOG\|MOE\)[A-Za-z0-9_]*\)=.*/\1/p'); do
log "unsetting inherited $v"; unset "$v"
done
[ -n "$PERF_REPORT" ] || for d in "$BUILD" "$REF"; do
[ -x "$d/bin/llama-server" ] || { echo "no llama-server in $d/bin" >&2; exit 2; }
done
case "$ARENA" in /dev/shm/llama-vram*) echo "refusing the production arena" >&2; exit 2 ;; esac
VRAM_USED=$(ls /sys/class/drm/card*/device/mem_info_vram_used 2>/dev/null | head -1)
VRAM_TOTAL=$(ls /sys/class/drm/card*/device/mem_info_vram_total 2>/dev/null | head -1)
vram_mb() { echo $(( $(cat "$VRAM_USED") / 1048576 )); }
port_busy() { # LISTEN socket on the port (v4 or v6)
local hex; hex=$(printf '%04X' "$1")
awk -v h=":$hex" '$4=="0A" && substr($2, length($2)-4)==h {f=1} END {exit !f}' /proc/net/tcp /proc/net/tcp6 2>/dev/null
}
[ -n "$PERF_REPORT" ] || for p in $PA $PB $PC; do
port_busy "$p" && { echo "scratch port $p is in use" >&2; exit 2; }
done
log "scratch dir $S, ports $PA/$PB/$PC, VRAM used $(vram_mb) MiB"
# ---------------------------------------------------------------- helpers: servers
# start_server NAME BUILD PORT ENV-ASSIGNMENTS... -- SERVER-ARGS... (sets SPID, log at $S/NAME.log)
# SRV_WRAP: optional command prefix that execs its arguments (the PID stays the server's)
SRV_WRAP=()
start_server() {
local name=$1 build=$2 port=$3; shift 3
local envs=()
while [ $# -gt 0 ] && [ "$1" != -- ]; do envs+=("$1"); shift; done
shift
"${SRV_WRAP[@]}" env LD_LIBRARY_PATH="$build/bin" "${envs[@]}" "$build/bin/llama-server" \
--host 127.0.0.1 --port "$port" "$@" >> "$S/$name.log" 2>&1 &
SPID=$!
track "$SPID"
echo "=== start pid $SPID $(date +%T.%N)" >> "$S/$name.log.events"
}
# wait_health PORT PID TIMEOUT_S
wait_health() {
local t0; t0=$(now_ms)
while [ $(( $(now_ms) - t0 )) -lt $(( $3 * 1000 )) ]; do
pid_alive "$2" || return 1
[ "$(curl -s -m 2 -o /dev/null -w '%{http_code}' "http://127.0.0.1:$1/health")" = 200 ] && return 0
sleep 0.2
done
return 1
}
# wait_exit PID TIMEOUT_MS -> prints elapsed ms, returns 1 on timeout. Then reap with `wait PID`.
wait_exit() {
local t0; t0=$(now_ms)
while pid_alive "$1"; do
[ $(( $(now_ms) - t0 )) -ge "$2" ] && { echo $(( $(now_ms) - t0 )); return 1; }
sleep 0.02
done
echo $(( $(now_ms) - t0 ))
}
# stop_server PID -> SIGTERM, wait up to 20 s, sets RC and EXIT_MS
stop_server() {
kill -TERM "$1" 2>/dev/null
EXIT_MS=$(wait_exit "$1" 20000) || { kill -KILL "$1" 2>/dev/null; }
wait "$1" 2>/dev/null; RC=$?
}
# ---------------------------------------------------------------- helpers: locks (/proc/locks)
lock_id() { # maj:min:inode as /proc/locks prints it (hex dev, decimal inode)
local d; d=$(( 16#$(stat -c %D "$1") ))
printf '%02x:%02x:%s' $(( (d >> 8) & 0xfff )) $(( (d & 0xff) | ((d >> 12) & 0xfff00) )) "$(stat -c %i "$1")"
}
lock_pids() { # FILE WRITE|READ -> PIDs holding a flock on it (granted, not blocked)
[ -e "$1" ] || return 0
awk -v id="$(lock_id "$1")" -v m="$2" '$2=="FLOCK" && $4==m && $6==id {print $5}' /proc/locks
}
token_holder() { lock_pids "$ARENA/token.lock" WRITE | head -1; }
holder_num() { local h; h=$(token_holder); echo "${h:-0}"; } # for arithmetic: 0 = nobody
# wait_holder PID TIMEOUT_MS : until PID holds token.lock ("" = until nobody does)
wait_holder() {
local t0; t0=$(now_ms)
while [ "$(token_holder)" != "$1" ]; do
[ $(( $(now_ms) - t0 )) -ge "$2" ] && return 1
sleep 0.01
done
echo $(( $(now_ms) - t0 ))
}
wait_want() { # PID TIMEOUT_MS : until PID holds want.lock shared (registered waiter)
local t0; t0=$(now_ms)
until lock_pids "$ARENA/want.lock" READ | grep -qx "$1"; do
[ $(( $(now_ms) - t0 )) -ge "$2" ] && return 1
sleep 0.01
done
}
# ---------------------------------------------------------------- helpers: VRAM attribution
# mem_info_vram_used is card-wide and includes production. Production's own share is read from the
# DRM fdinfo of every process this script did not start (drm-memory-vram, deduplicated by
# drm-client-id; summing fds double-counts shared buffers), so a production model that warms or
# cools during a measurement is detected instead of being attributed to the scratch servers.
drm_vram_kib() { # PID... -> KiB of VRAM held by the DRM clients of these processes
local p fs=()
for p in "$@"; do [ -d "/proc/$p/fdinfo" ] && fs+=("/proc/$p/fdinfo/"*); done
[ ${#fs[@]} -gt 0 ] || { echo 0; return; }
cat "${fs[@]}" 2>/dev/null | awk '/^pos:/ {c = ""} /^drm-client-id:/ {c = $2}
/^drm-memory-vram:/ && c != "" {v = $2; if ($3 == "MiB") v *= 1024; m[c] = v}
END {for (k in m) s += m[k]; print s + 0}'
}
prod_vram_mb() { # MiB of VRAM held by every process this script did not start
local p e mine=" " others=()
for e in "${PIDS[@]}"; do mine+="${e%%:*} "; done
for p in /proc/[0-9]*; do
p=${p#/proc/}
case "$mine" in *" $p "*) ;; *) others+=("$p") ;; esac
done
echo $(( $(drm_vram_kib "${others[@]}") / 1024 ))
}
# ---------------------------------------------------------------- helpers: requests
json_str() { sed -n 's/.*"'"$1"'":"\(\([^"\\]\|\\.\)*\)".*/\1/p' "$2" | head -1; }
json_num() { grep -o "\"$1\":[0-9]*" "$2" | head -1 | cut -d: -f2; }
# complete PORT OUTFILE JSON-BODY [TIMEOUT_S] -> prints HTTP code ("000" on timeout/refused)
complete() {
curl -sS -m "${4:-60}" -o "$2" -w '%{http_code}' -H 'Content-Type: application/json' \
-d "$3" "http://127.0.0.1:$1/completion" 2>>"$S/curl.err" || true
}
PROMPT='The history of the printing press began in'
GREEDY="\"temperature\":0,\"top_k\":1,\"seed\":42"
greedy_body() { # N_PREDICT CACHE_PROMPT [PROMPT]
echo "{\"prompt\":\"${3:-$PROMPT}\",\"n_predict\":$1,$GREEDY,\"cache_prompt\":$2}"
}
LONG_BODY="{\"prompt\":\"Write a very long story about a lighthouse keeper.\",\"n_predict\":4000,\"ignore_eos\":true,$GREEDY,\"stream\":true}"
# stream_bg PORT OUTFILE BODY -> starts a streaming request, sets CPID
stream_bg() {
curl -sS -N -m 300 -H 'Content-Type: application/json' -d "$3" \
"http://127.0.0.1:$1/completion" > "$2" 2>&1 &
CPID=$!; track "$CPID"
}
wait_tokens() { # FILE N TIMEOUT_MS : until N SSE data lines arrived
local t0; t0=$(now_ms)
while [ "$(grep -c '^data: ' "$1" 2>/dev/null)" -lt "$2" ]; do
[ $(( $(now_ms) - t0 )) -ge "$3" ] && return 1
sleep 0.05
done
}
scan_logs() { # NAME-PATTERN... -> prints offending lines
grep -h -n -E 'GGML_ASSERT|Assertion|terminate called|Segmentation fault|core dumped|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST|vk::.*Error|GGML_ABORT|fatal error' "$@" 2>/dev/null | head -20
}
# ================================================================ 1. ops
run_ops() {
local op out passed total
for op in CONCAT FLASH_ATTN_EXT MUL_MAT MUL_MAT_ID GET_ROWS; do
out=$S/ops-$op.log
log "ops: test-backend-ops -b Vulkan0 -o $op"
( cd "$S" && LD_LIBRARY_PATH="$BUILD/bin" timeout 1800 "$BUILD/bin/test-backend-ops" -b Vulkan0 -o "$op" ) \
> "$out" 2>&1
local rc=$?
# strip ANSI colours; "N/M tests passed" is printed once per tested backend
read -r passed total < <(sed 's/\x1b\[[0-9;]*m//g' "$out" | sed -n 's/^ *\([0-9]*\)\/\([0-9]*\) tests passed.*/\1 \2/p' | head -1)
passed=${passed:-0}; total=${total:-0}
if [ $rc -eq 0 ] && [ "$total" -gt 0 ] && [ "$passed" -eq "$total" ] \
&& sed 's/\x1b\[[0-9;]*m//g' "$out" | grep -q 'Backend Vulkan0: OK'; then
pass "ops/$op" "$passed/$total cases on Vulkan0"
else
fail "ops/$op" "rc=$rc, $passed/$total passed (0 cases = wrong backend name); see $out"
fi
done
}
# ================================================================ 2. concat fast path
# Method: llama-bench on the small hybrid model (qwen35: delta-net layers concat the conv state with
# the transposed qkv in build_conv_state). (1) with the knob =log, count fast-path dispatches in a
# prefill-only run (must be > 0, all with >= 32 tokens) and a decode-only run (must be 0, while the
# perf logger proves CONCAT nodes did execute); (2) GGML_VK_PERF_LOGGER CONCAT time at prefill: the
# unmodified candidate must match knob-on and be >= 2x faster than knob-off, which ties the
# candidate binary itself (no knob) to the fast path.
concat_us() { # LOG -> "count avg_us" over all perf-logger CONCAT lines
awk '/^CONCAT: / {n += $2; t += $(NF-1)} END {if (n) printf "%d %.2f\n", n, t/n; else print "0 0"}' "$1"
}
run_concat() {
local tb=$CONCAT_BUILD
[ -n "$tb" ] || tb=$BUILD
if ! grep -qa GGML_VK_CONCAT_FASTPATH "$tb/bin/libggml-vulkan.so" 2>/dev/null; then
fail concat "no GGML_VK_CONCAT_FASTPATH knob in $tb (build the candidate + knob patch, pass --concat-build)"
return
fi
[ -f "$SMALL" ] || { fail concat "small model missing: $SMALL"; return; }
local common=(-m "$SMALL" -ngl 99 -fa 1 -lm none -t 6)
local pp=(-p 512 -n 0 -b 512 -ub 512 -r 3)
local tg=(-p 0 -n 32 -r 1)
lb() { # BUILD OUT ENV... -- ARGS
local b=$1 out=$2; shift 2
local envs=(); while [ "$1" != -- ]; do envs+=("$1"); shift; done; shift
env LD_LIBRARY_PATH="$b/bin" GGML_VK_PERF_LOGGER=1 "${envs[@]}" "$b/bin/llama-bench" "${common[@]}" "$@" \
> "$out" 2>&1
}
log "concat: llama-bench prefill/decode arms (perf logger on)"
lb "$tb" "$S/cc-pp-log.log" GGML_VK_CONCAT_FASTPATH=log -- "${pp[@]}"
lb "$tb" "$S/cc-tg-log.log" GGML_VK_CONCAT_FASTPATH=log -- "${tg[@]}"
lb "$tb" "$S/cc-pp-off.log" GGML_VK_CONCAT_FASTPATH=0 -- "${pp[@]}"
lb "$BUILD" "$S/cc-pp-cand.log" -- "${pp[@]}"
lb "$tb" "$S/cc-pp-off2.log" GGML_VK_CONCAT_FASTPATH=0 -- "${pp[@]}"
lb "$BUILD" "$S/cc-pp-cand2.log" -- "${pp[@]}"
cat "$S/cc-pp-off2.log" >> "$S/cc-pp-off.log"; cat "$S/cc-pp-cand2.log" >> "$S/cc-pp-cand.log"
local n_pp n_pp_small n_tg n_tg_concat
n_pp=$(grep -c 'transpose fast path' "$S/cc-pp-log.log")
n_pp_small=$(sed -n 's/.*transpose fast path, transposed ne0=\([0-9]*\).*/\1/p' "$S/cc-pp-log.log" | awk '$1 < 32' | wc -l)
n_tg=$(grep -c 'transpose fast path' "$S/cc-tg-log.log")
read -r n_tg_concat _ < <(concat_us "$S/cc-tg-log.log")
local ne0s; ne0s=$(sed -n 's/.*transposed ne0=\([0-9]*\).*/\1/p' "$S/cc-pp-log.log" | sort -n | uniq -c | awk '{printf "%sx%s ", $1, $2}')
check concat/prefill-fires $(( n_pp > 0 && n_pp_small == 0 ? 0 : 1 )) \
"$n_pp fast-path dispatches in the prefill run (transposed ne0: ${ne0s:-none})"
check concat/decode-skips $(( n_tg == 0 && n_tg_concat > 0 ? 0 : 1 )) \
"$n_tg fast-path dispatches over $n_tg_concat executed CONCAT nodes in the decode run"
local c_on c_off c_cand on off cand
read -r c_on on < <(concat_us "$S/cc-pp-log.log")
read -r c_off off < <(concat_us "$S/cc-pp-off.log")
read -r c_cand cand < <(concat_us "$S/cc-pp-cand.log")
local ok
ok=$(awk -v on="$on" -v off="$off" -v c="$cand" 'BEGIN {print (c > 0 && on > 0 && off/c >= 2 && c/on < 1.5 && on/c < 1.5) ? 0 : 1}')
check concat/candidate-timing "$ok" \
"prefill CONCAT avg: candidate ${cand} us, knob-on ${on} us, knob-off ${off} us (off/cand $(awk -v a="$off" -v b="$cand" 'BEGIN {printf "%.1f", (b > 0) ? a / b : 0}')x)"
local t_on t_off t_cand
t_on=$(grep -E '\| +pp512 ' "$S/cc-pp-log.log" | awk -F'|' '{print $(NF-1)}' | awk '{print $1}' | head -1)
t_off=$(grep -E '\| +pp512 ' "$S/cc-pp-off.log" | awk -F'|' '{print $(NF-1)}' | awk '{print $1}' | tr '\n' ' ')
t_cand=$(grep -E '\| +pp512 ' "$S/cc-pp-cand.log" | awk -F'|' '{print $(NF-1)}' | awk '{print $1}' | tr '\n' ' ')
log "concat: pp512 t/s (perf logger on, so absolute speed is not representative): on $t_on | off $t_off | candidate $t_cand"
}
# ================================================================ 3. arbiter
ARB_ENV=(LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$ARENA" LLAMA_SLEEP_EVICT_KV=1 LLAMA_MMPROJ_ONDEMAND=1)
ARB_ARGS=(-m "$SMALL" -c 8192 -ngl 99 -fa 1 -lm none --parallel 1 --no-warmup --reasoning off)
A_PID= ; B_PID=
start_a() { start_server A "$BUILD" "$PA" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --mmproj "$MMPROJ"; A_PID=$SPID; }
start_b() { start_server B "$BUILD" "$PB" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --no-mmproj; B_PID=$SPID; }
sleeps() { grep -c 'server entering sleeping state' "$1"; }
make_red_ppm() { # 128x128 solid red, binary PPM (stb_image reads it; no image tools needed)
local f=$S/red.ppm i
printf '\377\000\000' > "$S/px"
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do cat "$S/px" "$S/px" > "$S/px2"; mv "$S/px2" "$S/px"; done
{ printf 'P6\n128 128\n255\n'; cat "$S/px"; } > "$f"
base64 -w0 "$f"
}
run_arbiter() {
[ -f "$SMALL" ] && [ -f "$MMPROJ" ] || { fail arbiter "model/mmproj missing: $SMALL $MMPROJ"; return; }
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
if [ $free_mb -lt 1536 ]; then
fail arbiter "only $free_mb MiB VRAM free (< 1536): run inside the swapdown window"; return
fi
# card-wide and production VRAM before any scratch server exists (see prod_vram_mb)
local v0 p0; v0=$(vram_mb); p0=$(prod_vram_mb)
log "arbiter: $free_mb MiB VRAM free (card $v0 MiB used, production DRM clients $p0 MiB); starting A (mmproj) on :$PA"
start_a
wait_health "$PA" "$A_PID" 180 || { fail arbiter/start "A did not become healthy, see $S/A.log"; return; }
# no-handoff reference: A has held the token since its load and never released
local code ref_a
code=$(complete "$PA" "$S/ref-a.json" "$(greedy_body 48 false)")
ref_a=$(json_str content "$S/ref-a.json")
[ "$code" = 200 ] && [ -n "$ref_a" ] || { fail arbiter/start "reference request on A: HTTP $code"; return; }
[ "$(token_holder)" = "$A_PID" ] || { fail arbiter/start "A serves but does not hold token.lock"; return; }
# ---- (v) VRAM really drops when A goes cold, and comes back when it warms again
# A is alone on the scratch arena; a fake waiter on want.lock makes it release. Its footprint is
# the card-wide delta against the baseline taken before A started; production's DRM clients are
# re-read at every point, and if they moved the check is SKIPped (it cannot attribute the delta).
local v_warm p_warm v_cold p_cold v_back p_back f_warm f_cold w k=0
sleep 0.3
v_warm=$(vram_mb); p_warm=$(prod_vram_mb); f_warm=$(( $(drm_vram_kib "$A_PID") / 1024 ))
local added=$(( v_warm - v0 )) need
need=$(( added * 7 / 10 ))
( exec 9<"$ARENA/want.lock"; flock -s 9; exec sleep 120 ) & w=$!; track "$w"
if ! wait_holder "" 10000 > /dev/null; then
fail arbiter/v-vram-release "A did not drop the token for a waiter within 10 s (holder $(token_holder))"
else
# the release frees before it unlocks; give the driver a moment to return the pages
while [ $(( v_warm - $(vram_mb) )) -lt $need ] && [ $k -lt 30 ]; do sleep 0.1; k=$((k + 1)); done
fi
v_cold=$(vram_mb); p_cold=$(prod_vram_mb); f_cold=$(( $(drm_vram_kib "$A_PID") / 1024 ))
kill "$w" 2>/dev/null; wait "$w" 2>/dev/null
code=$(complete "$PA" "$S/v-a.json" "$(greedy_body 48 false)")
sleep 0.3
v_back=$(vram_mb); p_back=$(prod_vram_mb)
local pmove; pmove=$(printf '%s\n' "$p_warm" "$p_cold" "$p_back" | awk -v b="$p0" '{d = $1 - b; if (d < 0) d = -d; if (d > m) m = d} END {print m + 0}')
local vmsg="card-wide over baseline: warm +${added} MiB, cold +$((v_cold - v0)) MiB (dropped $((v_warm - v_cold)), need >= ${need} = 70%), restored +$((v_back - v0)) MiB; A's own DRM clients warm ${f_warm} / cold ${f_cold} MiB; production ${p0} -> ${p_warm}/${p_cold}/${p_back} MiB"
if [ "$pmove" -gt 64 ]; then
skip arbiter/v-vram-release "production VRAM moved by ${pmove} MiB during the measurement, cannot attribute: $vmsg"
elif [ "$added" -lt 64 ]; then
fail arbiter/v-vram-release "A added only ${added} MiB when warm (measurement broken?): $vmsg"
else
check arbiter/v-vram-release $(( v_warm - v_cold >= need ? 0 : 1 )) "$vmsg"
check arbiter/v-vram-restore $(( code == 200 && v_back - v0 >= need && $(holder_num) == A_PID ? 0 : 1 )) \
"HTTP $code, A holds the token: $([ "$(token_holder)" = "$A_PID" ] && echo yes || echo NO), restored +$((v_back - v0)) MiB (need >= ${need}), output $([ "$(json_str content "$S/v-a.json")" = "$ref_a" ] && echo identical || echo DIFFERENT)"
[ "$(json_str content "$S/v-a.json")" = "$ref_a" ] || fail arbiter/v-vram-output "output after the release/restore differs from the reference"
fi
log "arbiter: starting B (text-only) on :$PB"
start_b
wait_health "$PB" "$B_PID" 180 || { fail arbiter/start "B did not become healthy, see $S/B.log"; return; }
pass arbiter/start "A=$A_PID B=$B_PID up, reference output ${#ref_a} chars"
# ---- (a)+(b) alternating handoffs
local i port pid name bad=0 stranded=0 diff_a=0 diff_b=0 ref_b= lat=() holder_bad=0
for i in $(seq 0 30); do
if [ $((i % 2)) -eq 0 ]; then port=$PB pid=$B_PID name=B; else port=$PA pid=$A_PID name=A; fi
local t0 t1 out=$S/h$i.json
t0=$(now_ms)
code=$(complete "$port" "$out" "$(greedy_body 48 false)" 60)
t1=$(now_ms)
[ $i -gt 0 ] && lat+=($((t1 - t0)))
if [ "$code" = 000 ]; then stranded=$((stranded + 1)); continue; fi
[ "$code" != 200 ] && { bad=$((bad + 1)); continue; }
local c; c=$(json_str content "$out")
if [ $name = A ]; then [ "$c" = "$ref_a" ] || diff_a=$((diff_a + 1))
else if [ -z "$ref_b" ]; then ref_b=$c; elif [ "$c" != "$ref_b" ]; then diff_b=$((diff_b + 1)); fi; fi
[ "$(token_holder)" = "$pid" ] || holder_bad=$((holder_bad + 1))
done
local med max
med=$(printf '%s\n' "${lat[@]}" | sort -n | awk '{a[NR]=$1} END {print a[int((NR+1)/2)]}')
max=$(printf '%s\n' "${lat[@]}" | sort -n | tail -1)
check arbiter/a-handoffs $(( stranded == 0 && bad == 0 && holder_bad == 0 ? 0 : 1 )) \
"30 handoffs: $stranded stranded, $bad non-200, $holder_bad served-without-token; request latency median ${med} ms, max ${max} ms"
check arbiter/b-greedy-identical $(( diff_a == 0 && diff_b == 0 ? 0 : 1 )) \
"A: 15/15 == no-handoff reference ($diff_a differ); B: $diff_b of 15 differ from its first; A==B: $([ "$ref_a" = "$ref_b" ] && echo yes || echo no)"
# ---- (b') KV / prompt cache survives an evict + restore (LLAMA_SLEEP_EVICT_KV=1)
local p1 p2 c1 c2
code=$(complete "$PA" "$S/kv1.json" "$(greedy_body 32 true 'Explain in detail how a refrigerator moves heat from inside to outside')")
complete "$PB" "$S/kv-b.json" "$(greedy_body 8 false)" > /dev/null
[ "$(token_holder)" = "$B_PID" ] || log "warning: B does not hold the token before the KV re-check"
complete "$PA" "$S/kv2.json" "$(greedy_body 32 true 'Explain in detail how a refrigerator moves heat from inside to outside')" > /dev/null
p1=$(json_num prompt_n "$S/kv1.json"); p2=$(json_num prompt_n "$S/kv2.json")
c1=$(json_str content "$S/kv1.json"); c2=$(json_str content "$S/kv2.json")
check arbiter/b-kv-survives $(( ${p2:-99} < ${p1:-0} && ${#c1} > 0 ? 0 : 1 )) \
"prompt_n $p1 -> $p2 after evict/restore (cache reused), output $([ "$c1" = "$c2" ] && echo identical || echo DIFFERENT)"
[ "$c1" = "$c2" ] || fail arbiter/b-kv-output "cached-prompt output changed across evict/restore"
# ---- (c) image after release/restore
complete "$PB" "$S/c-b.json" "$(greedy_body 8 false)" > /dev/null # A goes cold
local cold_ok=1; [ "$(token_holder)" = "$B_PID" ] && cold_ok=0
local img; img=$(make_red_ppm)
printf '{"messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"data:image/x-portable-pixmap;base64,%s"}},{"type":"text","text":"What is the main color of this image? Answer with one word."}]}],"max_tokens":16,%s}' \
"$img" "$GREEDY" > "$S/c-body.json"
code=$(curl -sS -m 120 -o "$S/c-a.json" -w '%{http_code}' -H 'Content-Type: application/json' \
-d @"$S/c-body.json" "http://127.0.0.1:$PA/v1/chat/completions" 2>>"$S/curl.err")
local ans; ans=$(json_str content "$S/c-a.json")
check arbiter/c-image-after-restore $(( cold_ok == 0 && ${#code} > 0 && code == 200 ? 0 : 1 )) \
"A cold before: $([ $cold_ok -eq 0 ] && echo yes || echo NO), HTTP $code, answer: '$ans'"
echo "$ans" | grep -qi red || fail arbiter/c-image-answer "expected 'red', got '$ans'"
# text right after the image path (encoder out, weights back)
code=$(complete "$PA" "$S/c-a2.json" "$(greedy_body 48 false)")
[ "$code" = 200 ] && [ "$(json_str content "$S/c-a2.json")" = "$ref_a" ] \
|| fail arbiter/c-text-after-image "HTTP $code, output differs from reference after the image request"
# ---- (g) kill -9 a waiter that holds want.lock -> the holder stops yielding
stream_bg "$PA" "$S/g-a.sse" "$LONG_BODY"; local gcl=$CPID
if ! wait_tokens "$S/g-a.sse" 3 30000; then fail arbiter/g-kill9-waiter "A did not start streaming"; else
local s_before; s_before=$(sleeps "$S/A.log")
complete "$PB" "$S/g-b.json" "$(greedy_body 8 false)" 60 > /dev/null &
local gbc=$!; track "$gbc"
if ! wait_want "$B_PID" 10000; then
fail arbiter/g-kill9-waiter "B never registered on want.lock (setup)"
else
kill -KILL "$B_PID"; wait "$B_PID" 2>/dev/null
kill "$gcl" 2>/dev/null; wait "$gcl" 2>/dev/null # end A's generation (client abort)
sleep 2
local wl; wl=$(lock_pids "$ARENA/want.lock" READ | wc -l)
check arbiter/g-kill9-waiter $(( $(holder_num) == A_PID && $(sleeps "$S/A.log") == s_before && wl == 0 ? 0 : 1 )) \
"after kill -9 of waiting B: want.lock readers $wl, token holder $(token_holder) (A=$A_PID), A sleeps +$(( $(sleeps "$S/A.log") - s_before ))"
fi
wait "$gbc" 2>/dev/null
fi
log "arbiter: restarting B"
start_b
wait_health "$PB" "$B_PID" 180 || { fail arbiter/restart "B did not come back"; return; }
# ---- (e) SIGTERM while A waits for the token -> exits fast, never restores
stream_bg "$PB" "$S/e-b.sse" "$LONG_BODY"; local ecl=$CPID
wait_tokens "$S/e-b.sse" 3 30000 || { fail arbiter/e-term-waiting "B did not start streaming"; }
complete "$PA" "$S/e-a.json" "$(greedy_body 8 false)" 30 > /dev/null &
local eac=$!; track "$eac"
if wait_want "$A_PID" 10000; then
sleep 0.3
local t0 ms took_token=0
t0=$(now_ms); kill -TERM "$A_PID"
while pid_alive "$A_PID"; do
[ "$(token_holder)" = "$A_PID" ] && took_token=1
[ $(( $(now_ms) - t0 )) -gt 10000 ] && break
sleep 0.01
done
ms=$(( $(now_ms) - t0 )); wait "$A_PID"; RC=$?
check arbiter/e-term-waiting $(( RC == 0 && ms <= 1500 && took_token == 0 && $(holder_num) == B_PID ? 0 : 1 )) \
"exit $RC in ${ms} ms, A took the token: $([ $took_token = 1 ] && echo YES || echo no), holder now $(token_holder) (B=$B_PID), log: $(grep -c 'no longer waiting for the GPU token' "$S/A.log") abort line(s)"
else
fail arbiter/e-term-waiting "A never registered on want.lock (setup)"
stop_server "$A_PID"
fi
kill "$ecl" 2>/dev/null; wait "$ecl" "$eac" 2>/dev/null
log "arbiter: restarting A"
start_a
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
# ---- (d) SIGTERM during A's VRAM release -> exit 0 within ~2 s, B gets the token
# A is warm (fresh load, first release also captures the host shadow: the longest release)
local how sig_ms
if command -v gdb > /dev/null; then
how=gdb
gdb -p "$A_PID" -batch -nx -ex 'set pagination off' -ex 'handle all nostop noprint pass' \
-ex 'break llama_context_release_device' -ex 'continue' \
-ex "shell date +%s%3N > $S/d.sig" -ex 'delete' -ex 'signal SIGTERM' > "$S/d-gdb.log" 2>&1 &
local gpid=$!; track "$gpid"
local k=0; until grep -q '^Continuing' "$S/d-gdb.log" || [ $k -gt 300 ]; do sleep 0.1; k=$((k + 1)); done
sleep 0.5
else
how=log-poll
fi
local s0; s0=$(sleeps "$S/A.log")
complete "$PB" "$S/d-b.json" "$(greedy_body 8 false)" 60 > "$S/d-b.code" &
local dbc=$!; track "$dbc"
if [ $how = log-poll ]; then
local k=0; until [ "$(sleeps "$S/A.log")" -gt "$s0" ] || [ $k -gt 2000 ]; do sleep 0.005; k=$((k + 1)); done
now_ms > "$S/d.sig"; kill -TERM "$A_PID"
else
local k=0; until [ -s "$S/d.sig" ] || [ $k -gt 600 ]; do sleep 0.05; k=$((k + 1)); done
fi
if [ ! -s "$S/d.sig" ]; then
fail arbiter/d-term-in-release "release breakpoint never hit ($how), see $S/d-gdb.log"
kill "$gpid" 2>/dev/null; stop_server "$A_PID"
else
while pid_alive "$A_PID" && [ $(( $(now_ms) - $(cat "$S/d.sig") )) -lt 20000 ]; do sleep 0.01; done
sig_ms=$(( $(now_ms) - $(cat "$S/d.sig") ))
wait "$A_PID"; RC=$?
wait "$dbc"
local dcode; dcode=$(cat "$S/d-b.code")
sleep 0.2
check arbiter/d-term-in-release $(( RC == 0 && sig_ms <= 2500 && dcode == 200 && $(holder_num) == B_PID ? 0 : 1 )) \
"SIGTERM inside llama_context_release_device ($how): A exit $RC in ${sig_ms} ms, B HTTP $dcode, holder $(token_holder) (B=$B_PID)"
fi
log "arbiter: restarting A"
start_a
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
# ---- (f) SIGTERM mid-generation (streaming)
stream_bg "$PA" "$S/f-a.sse" "$LONG_BODY"; local fcl=$CPID
if ! wait_tokens "$S/f-a.sse" 10 30000; then fail arbiter/f-term-streaming "A did not start streaming"; else
local t0 free_ms=-1 cl_ms=-1 exit_ms=-1
t0=$(now_ms); kill -TERM "$A_PID"
while [ $(( $(now_ms) - t0 )) -lt 10000 ]; do
[ $free_ms -lt 0 ] && [ -z "$(token_holder)" ] && free_ms=$(( $(now_ms) - t0 ))
[ $cl_ms -lt 0 ] && ! pid_alive "$fcl" && cl_ms=$(( $(now_ms) - t0 ))
[ $exit_ms -lt 0 ] && ! pid_alive "$A_PID" && exit_ms=$(( $(now_ms) - t0 ))
[ $free_ms -ge 0 ] && [ $cl_ms -ge 0 ] && [ $exit_ms -ge 0 ] && break
sleep 0.01
done
wait "$A_PID"; RC=$?; wait "$fcl" 2>/dev/null
local errev; errev=$(grep -c '"error"' "$S/f-a.sse")
check arbiter/f-term-streaming $(( RC == 0 && errev > 0 && cl_ms >= 0 && free_ms >= 0 && free_ms <= 2500 && exit_ms >= 0 ? 0 : 1 )) \
"A exit $RC in ${exit_ms} ms, token free in ${free_ms} ms, client closed in ${cl_ms} ms with $errev error event(s) after $(grep -c '^data: ' "$S/f-a.sse") data lines"
fi
log "arbiter: restarting A"
start_a
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
# ---- (h) clean shutdown warm and cold
[ "$(token_holder)" = "$A_PID" ] || log "warning: A is not warm before the warm-shutdown case"
stop_server "$A_PID"
check arbiter/h-shutdown-warm $(( RC == 0 ? 0 : 1 )) "warm A: exit $RC in ${EXIT_MS} ms"
start_a
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
complete "$PB" "$S/h-b.json" "$(greedy_body 8 false)" > /dev/null # A goes cold
local acold=no; [ "$(token_holder)" = "$B_PID" ] && acold=yes
stop_server "$A_PID"
check arbiter/h-shutdown-cold $([ $RC -eq 0 ] && [ $acold = yes ] && echo 0 || echo 1) \
"cold A (B holds token: $acold): exit $RC in ${EXIT_MS} ms"
stop_server "$B_PID"
check arbiter/h-shutdown-warm-b $(( RC == 0 ? 0 : 1 )) "warm B: exit $RC in ${EXIT_MS} ms"
local bad_lines; bad_lines=$(scan_logs "$S/A.log" "$S/B.log")
check arbiter/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) \
"$([ -z "$bad_lines" ] && echo 'no asserts / Vulkan errors in A/B logs' || echo "$bad_lines" | head -3 | tr '\n' ' ')"
run_handoff_ab
}
# handoff_block BUILD TAG N : a fresh pair of BUILD on the scratch arena (A with mmproj, B text-only,
# the same config as case a), one warm-up request on A, then N+1 alternating requests starting with
# B. Request 0 is not a handoff (B holds the token after its load); the latency of requests 1..N is
# appended to $S/lat-TAG.txt. HB_BAD counts setup failures, non-200s and answers without the token.
handoff_block() {
local build=$1 tag=$2 n=$3 a b i port pid code t0 t1
HB_BAD=0
: > "$S/lat-$tag.txt"
start_server "L$tag-A" "$build" "$PA" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --mmproj "$MMPROJ"; a=$SPID
if ! wait_health "$PA" "$a" 180; then HB_BAD=1; stop_server "$a"; return 1; fi
complete "$PA" "$S/L$tag-warm.json" "$(greedy_body 8 false)" > /dev/null
start_server "L$tag-B" "$build" "$PB" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --no-mmproj; b=$SPID
if ! wait_health "$PB" "$b" 180; then HB_BAD=1; stop_server "$b"; stop_server "$a"; return 1; fi
for i in $(seq 0 "$n"); do
if [ $((i % 2)) -eq 0 ]; then port=$PB pid=$b; else port=$PA pid=$a; fi
t0=$(now_ms)
code=$(complete "$port" "$S/L$tag-$i.json" "$(greedy_body 48 false)" 60)
t1=$(now_ms)
if [ "$code" != 200 ] || [ "$(token_holder)" != "$pid" ]; then HB_BAD=$((HB_BAD + 1)); continue; fi
[ "$i" -gt 0 ] && echo $((t1 - t0)) >> "$S/lat-$tag.txt"
done
stop_server "$b"; stop_server "$a"
}
# ---- (a') ordinary handoff latency, candidate vs reference, in one session
# Two production deploys were lost to arbiter changes that passed the pathological cases but made
# every ordinary handoff slower. Blocks run ref, cand, cand, ref (a fixed order once gave the later
# slot about 9%); each build's median is over all its handoffs. FAIL if the candidate median exceeds
# the reference median by more than max(25%, 200 ms): small absolute numbers are noisy.
run_handoff_ab() {
local n=16 blk k=0 bad_r=0 bad_c=0 tag
log "arbiter: handoff latency A/B, blocks ref,cand,cand,ref x $n handoffs"
: > "$S/lat-ref.txt"; : > "$S/lat-cand.txt"
for blk in ref cand cand ref; do
k=$((k + 1)); tag=$k-$blk
local b; [ $blk = ref ] && b=$REF || b=$BUILD
handoff_block "$b" "$tag" "$n"
if [ $blk = ref ]; then bad_r=$((bad_r + HB_BAD)); else bad_c=$((bad_c + HB_BAD)); fi
cat "$S/lat-$tag.txt" >> "$S/lat-$blk.txt"
log "arbiter: block $tag median $(median < "$S/lat-$tag.txt") ms over $(wc -l < "$S/lat-$tag.txt") handoffs, $HB_BAD bad"
done
local mr mc nr nc maxr maxc limit
mr=$(median < "$S/lat-ref.txt"); mc=$(median < "$S/lat-cand.txt")
nr=$(wc -l < "$S/lat-ref.txt"); nc=$(wc -l < "$S/lat-cand.txt")
maxr=$(sort -n "$S/lat-ref.txt" | tail -1); maxc=$(sort -n "$S/lat-cand.txt" | tail -1)
if [ -z "$mr" ] || [ -z "$mc" ] || [ $bad_r -gt 0 ] || [ $bad_c -gt 0 ]; then
fail arbiter/a-handoff-latency "blocks incomplete: reference $nr handoffs ($bad_r bad), candidate $nc ($bad_c bad); see $S/L*-*.log"
return
fi
limit=$(awk -v r="$mr" 'BEGIN {d = r * 0.25; if (d < 200) d = 200; printf "%.0f", r + d}')
check arbiter/a-handoff-latency "$(awk -v c="$mc" -v l="$limit" 'BEGIN {print (c > l) ? 1 : 0}')" \
"median handoff latency candidate ${mc} ms (n $nc, max $maxc) vs reference ${mr} ms (n $nr, max $maxr); limit ${limit} ms = ref + max(25%, 200 ms)"
local bad_lines; bad_lines=$(scan_logs "$S"/L*-A.log "$S"/L*-B.log)
[ -z "$bad_lines" ] || fail arbiter/a-handoff-latency-logs "$(echo "$bad_lines" | head -3 | tr '\n' ' ')"
}
# ================================================================ 4. perf
vram_sampler() { # OUTFILE : max of mem_info_vram_used every 0.25 s, written continuously
local m=0 v
while :; do v=$(cat "$VRAM_USED"); [ "$v" -gt "$m" ] && { m=$v; echo "$m" > "$1"; }; sleep 0.25; done
}
# median of the numbers on stdin (one per line); empty input -> empty output
median() { sort -g | awk '{a[NR] = $1} END {if (NR) printf "%.2f", (NR % 2) ? a[(NR + 1) / 2] : (a[NR / 2] + a[NR / 2 + 1]) / 2}'; }
bench_samples() { # JSONL (llama-bench -o jsonl) -> "test<TAB>t/s", one row per repetition sample
sed -n 's/.*"n_prompt": *\([0-9]*\),.*"n_gen": *\([0-9]*\),.*"n_depth": *\([0-9]*\),.*"samples_ts": *\[\([^]]*\)\].*/\1 \2 \3 \4/p' "$1" |
awk '{t = ($1 > 0) ? "pp" $1 : "tg" $2; if ($3 > 0) t = t " @ d" $3
for (i = 4; i <= NF; i++) {v = $i; gsub(/,/, "", v); if (v != "") print t "\t" v}}'
}
run_perf() {
gpu_exclusive perf || return
[ -f "$PERF_MODEL" ] || { fail perf "model missing: $PERF_MODEL"; return; }
[ -x "$REF/bin/llama-bench" ] && [ -x "$BUILD/bin/llama-bench" ] || { fail perf "llama-bench missing"; return; }
local common=(-m "$PERF_MODEL" -t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -o jsonl -oe md)
local arm k=0 base; base=$(vram_mb)
log "perf: idle VRAM $base MiB; arms ref,cand,cand,ref"
for arm in ref cand cand ref; do
k=$((k + 1))
local b; [ $arm = ref ] && b=$REF || b=$BUILD
local tag=$k-$arm
vram_sampler "$S/perf-$tag.vram" & local vs=$!; track "$vs"
# stdout: one JSON line per test with every repetition sample; stderr: the markdown table + logs
log "perf: arm $tag pp sweep"
LD_LIBRARY_PATH="$b/bin" "$b/bin/llama-bench" "${common[@]}" -p 2048 -n 0 -d 0,16384,32768 -r 2 \
> "$S/perf-$tag-pp.jsonl" 2> "$S/perf-$tag-pp.log"
log "perf: arm $tag tg"
LD_LIBRARY_PATH="$b/bin" "$b/bin/llama-bench" "${common[@]}" -p 0 -n 256 -d 4096 -r 3 \
> "$S/perf-$tag-tg.jsonl" 2> "$S/perf-$tag-tg.log"
kill "$vs" 2>/dev/null; wait "$vs" 2>/dev/null
{ bench_samples "$S/perf-$tag-pp.jsonl"; bench_samples "$S/perf-$tag-tg.jsonl"; } | sed "s/^/$arm\t/" >> "$S/perf.tsv"
echo "$arm $(cat "$S/perf-$tag.vram" 2>/dev/null || echo 0)" >> "$S/perf-vram.txt"
done
perf_report "$S/perf.tsv"
local total; total=$(cat "$VRAM_TOTAL")
local vr vc
vr=$(awk '$1=="ref" && $2>m {m=$2} END {print m+0}' "$S/perf-vram.txt")
vc=$(awk '$1=="cand" && $2>m {m=$2} END {print m+0}' "$S/perf-vram.txt")
check perf/peak-vram $(( vc * 100 <= total * 95 ? 0 : 1 )) \
"peak VRAM reference $((vr / 1048576)) MiB, candidate $((vc / 1048576)) MiB of $((total / 1048576)) MiB (limit 95%)"
}
# perf_report TSV : "arm<TAB>test<TAB>t/s" rows, one per llama-bench repetition sample (arm = ref|cand).
# Gates on the MEDIAN of all samples of a build (pooled over its arms): a single catastrophic
# sample (CPU contention from the other pods on the node) moves a mean, not a median.
perf_report() {
local tsv=$1
echo
echo "perf: median t/s over every repetition sample of both arms (order ref,cand,cand,ref)"
printf '%-18s %24s %24s %8s\n' test "reference med (min-max, n)" "candidate med (min-max, n)" delta
local tests t r c d rs cs
tests=$(awk -F'\t' '{print $2}' "$tsv" | awk '!s[$0]++')
local IFS_=$IFS; IFS=$'\n'
for t in $tests; do
r=$(awk -F'\t' -v t="$t" '$1=="ref" && $2==t {print $3}' "$tsv" | median)
c=$(awk -F'\t' -v t="$t" '$1=="cand" && $2==t {print $3}' "$tsv" | median)
rs=$(awk -F'\t' -v t="$t" '$1=="ref" && $2==t {n++; if (n == 1 || $3 < lo) lo = $3; if (n == 1 || $3 > hi) hi = $3} END {if (n) printf "%.1f-%.1f, %d", lo, hi, n}' "$tsv")
cs=$(awk -F'\t' -v t="$t" '$1=="cand" && $2==t {n++; if (n == 1 || $3 < lo) lo = $3; if (n == 1 || $3 > hi) hi = $3} END {if (n) printf "%.1f-%.1f, %d", lo, hi, n}' "$tsv")
d=$(awk -v r="$r" -v c="$c" 'BEGIN {if (r > 0) printf "%+.1f%%", (c/r - 1)*100}')
printf '%-18s %24s %24s %8s\n' "$t" "$r ($rs)" "$c ($cs)" "$d"
case "$t" in pp*)
if [ -z "$r" ] || [ -z "$c" ] || [ "$(awk -v r="$r" -v c="$c" 'BEGIN {print (c < 0.95*r)}')" = 1 ]; then
fail "perf/$t" "candidate median $c vs reference median $r t/s ($d, limit -5%)"
else
pass "perf/$t" "candidate median $c vs reference median $r t/s ($d)"
fi ;;
tg*) RESULTS+=("INFO perf/$t: candidate median $c vs reference median $r t/s ($d; decode is CPU-bound and noisy here, not gated)") ;;
esac
done
IFS=$IFS_
local n_tests; n_tests=$(echo "$tests" | grep -c '^pp')
[ "$n_tests" -ge 3 ] || fail perf/rows "expected 3 pp rows, got $n_tests (see $S/perf-*-pp.log)"
}
# ================================================================ 5. MoE cache release/restore smoke
run_moecache() {
[ -f "$MOECACHE_MODEL" ] || { fail moecache "model missing: $MOECACHE_MODEL"; return; }
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
log "moecache: $free_mb MiB VRAM free; server on :$PC with --moe-cache-mib $MOECACHE_MIB"
local arena=$S/arena-moec; mkdir -p "$arena"
# -lv 4: the cache size and hit/miss stats are only logged at that verbosity
# shellcheck disable=SC2086
start_server M "$BUILD" "$PC" LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$arena" LLAMA_SLEEP_EVICT_KV=1 -- \
-m "$MOECACHE_MODEL" $MOECACHE_ARGS -fa 1 -lm none --parallel 1 --no-warmup --reasoning off -lv 4 \
--moe-cache-mib "$MOECACHE_MIB"
local mp=$SPID
wait_health "$PC" "$mp" 600 || { fail moecache "server did not start, see $S/M.log"; return; }
# chat endpoint: an instruct model fed a raw prompt can loop, which says nothing about the cache
local body="{\"messages\":[{\"role\":\"user\",\"content\":\"What are the three primary colors of light? Answer in one short sentence.\"}],\"max_tokens\":64,$GREEDY}"
local outs=() i code w
for i in 0 1 2; do
code=$(curl -sS -m 300 -o "$S/m$i.json" -w '%{http_code}' -H 'Content-Type: application/json' \
-d "$body" "http://127.0.0.1:$PC/v1/chat/completions" 2>>"$S/curl.err")
outs+=("$(json_str content "$S/m$i.json")")
[ "$code" = 200 ] || { fail moecache "request $i: HTTP $code"; stop_server "$mp"; return; }
[ $i -eq 2 ] && break
# force a release: a fake waiter holds want.lock shared until the server drops the token
( exec 9<"$arena/want.lock"; flock -s 9; exec sleep 120 ) & w=$!; track "$w"
local k=0; until [ -z "$(awk -v id="$(lock_id "$arena/token.lock")" '$2=="FLOCK" && $4=="WRITE" && $6==id' /proc/locks)" ] || [ $k -gt 600 ]; do sleep 0.05; k=$((k + 1)); done
log "moecache: cycle $((i + 1)) cold, VRAM $(vram_mb) MiB"
kill "$w"; wait "$w" 2>/dev/null
done
stop_server "$mp"
local size hits rel res
size=$(sed -n 's/.*MoE cache size = *\([0-9.]*\) MiB.*/\1/p' "$S/M.log" | head -1)
hits=$(sed -n 's/.*llama_moe_cache: .* hits = \([0-9]*\),.*/\1/p' "$S/M.log" | awk '{s += $1} END {print s+0}')
rel=$(grep -c 'released .* MiB of MoE cache' "$S/M.log"); res=$(grep -c 'restored .* MiB of MoE cache' "$S/M.log")
check moecache/active $(( hits > 0 ? 0 : 1 )) \
"cache ${size:-?} MiB, $hits hits (0 hits = cache too small to ever reuse an expert, so a stale SLRU would go unnoticed), released ${rel}x restored ${res}x"
local same=1; [ "${outs[0]}" = "${outs[1]}" ] && [ "${outs[1]}" = "${outs[2]}" ] && [ -n "${outs[0]}" ] && same=0
check moecache/identical $same "3 greedy answers across 2 cold/warm cycles $([ $same = 0 ] && echo identical || echo DIFFER): '$(echo "${outs[0]}" | cut -c1-80)'"
local coh=1; echo "${outs[2]}" | grep -qi red && echo "${outs[2]}" | grep -qi green && echo "${outs[2]}" | grep -qi blue && coh=0
check moecache/coherent $coh "answer after the 2nd restore names red, green and blue"
check moecache/shutdown $(( RC == 0 ? 0 : 1 )) "exit $RC"
local bad_lines; bad_lines=$(scan_logs "$S/M.log")
check moecache/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) "${bad_lines:-clean}"
}
# ================================================================ 6. MoE routing-ids readback skip (F1) exactness
# sched_copy_experts skips the ids readback when a split's ids tensor has n_tokens * n_expert_used >=
# 4 * n_expert entries, i.e. for any ubatch of at least ceil(4 * n_expert / n_expert_used) tokens
# (128 on a 256-expert / 8-used model). Method: the model with every MoE layer's experts on the host
# (so the expert-copy callback runs), greedy, one prompt of >= 2x that many tokens in a single
# ubatch: the prefill takes the skip and every decode step takes the readback. The candidate's
# output must be byte-identical to the reference's and, when the build has the LLAMA_MOE_IDS_SKIP
# knob, to the candidate with the skip forced off (=0), which proves the skip itself is exact even
# when the reference build carries it too; =log then shows the skip engaged.
gguf_u32() { # FILE KEY-SUFFIX -> uint32 value of the first GGUF metadata key ending in KEY-SUFFIX
local off t
off=$(head -c 67108864 "$1" | grep -abo -m1 -F "$2" | head -1 | cut -d: -f1)
[ -n "$off" ] || return 1
off=$((off + ${#2})) # key string -> uint32 value type -> value
t=$(od -An -tu4 -j "$off" -N4 "$1" | tr -d ' ')
[ "$t" = 4 ] || [ "$t" = 5 ] || return 1 # GGUF_TYPE_UINT32 / INT32
od -An -tu4 -j $((off + 4)) -N4 "$1" | tr -d ' '
}
gpu_exclusive() { # CHECK-NAME -> FAILs it and returns 1 unless nothing else can use the GPU
[ "${FV_TEST_SHARED_GPU:-}" = 1 ] && { log "FV_TEST_SHARED_GPU=1: skipping the exclusive-GPU guard (script testing only)"; return 0; }
if supervisorctl -c /root/supervisord.conf status llama-swap 2>/dev/null | grep -q RUNNING; then
fail "$1" "llama-swap is RUNNING; run under /root/bench/moe/swapdown.sh"; return 1
fi
local others; others=$(ps -eo pid=,comm= | awk '$2 ~ /^llama-(server|bench|cli)$/' | wc -l)
[ "$others" -eq 0 ] || { fail "$1" "$others other llama processes are running"; return 1; }
}
moex_prompt() { # N_LINES -> a deterministic prose-like prompt (JSON-safe: no quotes or backslashes)
local i
for i in $(seq 1 "$1"); do
printf 'Entry %d: the river town of Station%d ships %d barrels of grain and %d crates of salt each week to the harbour, and its council meets on day %d of the month. ' \
"$i" "$i" $(( (i * 37) % 101 + 3 )) $(( (i * 53) % 89 + 2 )) $(( (i * 7) % 28 + 1 ))
done
printf 'Question: which town ships the most barrels of grain each week? Answer:'
}
run_moe_exact() {
gpu_exclusive moe-exact || return
[ -f "$MOEX_MODEL" ] || { fail moe-exact "model missing: $MOEX_MODEL"; return; }
local n_exp n_used
n_exp=$(gguf_u32 "$MOEX_MODEL" .expert_count); n_used=$(gguf_u32 "$MOEX_MODEL" .expert_used_count)
if [ -z "$n_exp" ] || [ -z "$n_used" ] || [ "$n_exp" -eq 0 ] || [ "$n_used" -eq 0 ]; then
fail moe-exact "no expert_count / expert_used_count in $MOEX_MODEL (not a MoE model?)"; return
fi
local thr=$(( (4 * n_exp + n_used - 1) / n_used )) # tokens per ubatch where the skip starts
local lines=$(( (2 * thr + 39) / 40 + 2 )) # ~45 tokens per line -> >= 2x the threshold
local body
body="{\"prompt\":\"$(moex_prompt "$lines")\",\"n_predict\":64,$GREEDY,\"cache_prompt\":false}"
local knob=0; grep -qa LLAMA_MOE_IDS_SKIP "$BUILD/bin/libllama.so" 2>/dev/null && knob=1
log "moe-exact: n_expert $n_exp, n_expert_used $n_used -> skip at >= $thr tokens per ubatch; prompt of $lines lines; knob in candidate: $([ $knob = 1 ] && echo yes || echo no)"
# arm NAME BUILD ENV... : one server, one request; sets MX_CONTENT / MX_PN, returns 1 on failure
mx_arm() {
local name=$1 b=$2; shift 2
# shellcheck disable=SC2086
start_server "$name" "$b" "$PC" "$@" -- -m "$MOEX_MODEL" $MOEX_ARGS -fa 1 -lm none --parallel 1 --no-warmup
local p=$SPID code
if ! wait_health "$PC" "$p" 900; then fail "moe-exact/$name" "server did not start, see $S/$name.log"; stop_server "$p"; return 1; fi
code=$(complete "$PC" "$S/$name.json" "$body" 600)
stop_server "$p"
MX_CONTENT=$(json_str content "$S/$name.json"); MX_PN=$(json_num prompt_n "$S/$name.json")
[ "$code" = 200 ] && [ -n "$MX_CONTENT" ] || { fail "moe-exact/$name" "HTTP $code, empty output; see $S/$name.log"; return 1; }
[ "$RC" -eq 0 ] || log "warning: $name exited $RC"
}
local c_cand c_ref c_off pn
if [ $knob = 1 ]; then mx_arm X-cand "$BUILD" LLAMA_MOE_IDS_SKIP=log || return
else mx_arm X-cand "$BUILD" || return; fi
c_cand=$MX_CONTENT; pn=${MX_PN:-0}
if [ "$pn" -lt "$thr" ]; then
fail moe-exact/prompt "prompt_n $pn < threshold $thr: the skip never ran, nothing was tested"; return
fi
mx_arm X-ref "$REF" || return
c_ref=$MX_CONTENT
check moe-exact/identical-ref $([ "$c_cand" = "$c_ref" ] && echo 0 || echo 1) \
"prompt_n $pn (>= $thr), 64 greedy tokens: candidate $([ "$c_cand" = "$c_ref" ] && echo == || echo '!=') reference ('$(echo "$c_cand" | cut -c1-60)')"
if [ $knob = 1 ]; then
mx_arm X-cand-noskip "$BUILD" LLAMA_MOE_IDS_SKIP=0 || return
c_off=$MX_CONTENT
check moe-exact/identical-noskip $([ "$c_cand" = "$c_off" ] && echo 0 || echo 1) \
"candidate with the skip (LLAMA_MOE_IDS_SKIP=log) $([ "$c_cand" = "$c_off" ] && echo == || echo '!=') candidate with the readback forced (=0)"
local n_skip n_small
n_skip=$(grep -c 'sched_copy_experts: ids readback skipped' "$S/X-cand.log")
n_small=$(sed -n 's/.*ids readback skipped, n_ids=\([0-9]*\) n_expert=\([0-9]*\).*/\1 \2/p' "$S/X-cand.log" | awk '$1 < 4 * $2' | wc -l)
check moe-exact/skip-engaged $(( n_skip > 0 && n_small == 0 ? 0 : 1 )) \
"$n_skip skips logged ($n_small below 4*n_expert), $(grep -c 'ids readback skipped' "$S/X-cand-noskip.log") with the skip forced off"
else
skip moe-exact/skip-engaged "candidate has no LLAMA_MOE_IDS_SKIP knob to observe it; by construction the prefill ubatch had $pn x $n_used = $((pn * n_used)) ids >= 4 x $n_exp, so the skip branch ran"
fi
local bad_lines; bad_lines=$(scan_logs "$S"/X-*.log)
check moe-exact/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) "${bad_lines:-clean}"
}
# ================================================================ 7. disk prompt cache
# --cache-disk-path: on shutdown each slot's state goes to a file; after a restart a prompt that starts
# with a saved prompt (or with one of its checkpoints, for a hybrid model) is restored from the file.
# Method: chat requests on the small hybrid model, greedy. A reference server that never restarts
# gives turn 2 (turn 1 + answer + a new question); the candidate saves turn 1 on SIGTERM, restarts
# and must give the same answer with the same prompt_n (only the suffix processed). CPU only
# (-dev none -ngl 0, no VRAM), plus the GPU path (-ngl 99, arbiter env, warm and cold) if VRAM allows.
DC_ARGS=(-m "$SMALL" -c 4096 -dev none -ngl 0 -fa 1 -t 4 --parallel 1 --no-warmup --reasoning off)
DC_MM=(--no-mmproj) # the mmproj case sets --mmproj
DC_GPU=(-dev Vulkan0 -ngl 99)
DC_ENV=(LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$ARENA" LLAMA_SLEEP_EVICT_KV=1)
DC_PID=
dc_chat() { # PORT OUT MESSAGES-JSON -> HTTP code
curl -sS -m 300 -o "$2" -w '%{http_code}' -H 'Content-Type: application/json' \
-d "{\"messages\":$3,\"max_tokens\":32,$GREEDY}" "http://127.0.0.1:$1/v1/chat/completions" 2>>"$S/curl.err" || true
}
dc_msgs() { # USER1 [ANSWER1 USER2] -> messages JSON
if [ $# -eq 1 ]; then echo "[{\"role\":\"user\",\"content\":\"$1\"}]"
else echo "[{\"role\":\"user\",\"content\":\"$1\"},{\"role\":\"assistant\",\"content\":\"$2\"},{\"role\":\"user\",\"content\":\"$3\"}]"; fi
}
# dc_start NAME DIR ENV... -- ARGS... : DIR "" = no disk cache; sets DC_PID, returns 1 if not healthy
dc_start() {
local name=$1 dir=$2; shift 2
local envs=(); while [ "$1" != -- ]; do envs+=("$1"); shift; done; shift
start_server "$name" "$BUILD" "$PC" "${envs[@]}" -- "${DC_ARGS[@]}" "${DC_MM[@]}" ${dir:+--cache-disk-path "$dir"} "$@"
DC_PID=$SPID
wait_health "$PC" "$DC_PID" 180
}
# dc_term : SIGTERM DC_PID; sets RC, DC_EXIT_MS and DC_FREE_MS (token.lock free, -1 if never held)
dc_term() {
local t0; t0=$(now_ms); DC_FREE_MS=-1; DC_EXIT_MS=-1
local held=0; [ -n "$(token_holder)" ] && held=1
kill -TERM "$DC_PID"
while [ $(( $(now_ms) - t0 )) -lt 30000 ]; do
[ $held = 1 ] && [ $DC_FREE_MS -lt 0 ] && [ -z "$(token_holder)" ] && DC_FREE_MS=$(( $(now_ms) - t0 ))
pid_alive "$DC_PID" || { DC_EXIT_MS=$(( $(now_ms) - t0 )); break; }
sleep 0.01
done
pid_alive "$DC_PID" && kill -KILL "$DC_PID"
wait "$DC_PID" 2>/dev/null; RC=$?
}
# dc_ref TAG ARGS... : turn 1 and turn 2 on one server that never restarts -> DC_C1, DC_REF2, DC_PN_REF, DC_N2, DC_MS_FULL
dc_ref() {
local tag=$1; shift
dc_start "$tag" "" -- "$@" || { fail "diskcache/$tag" "reference server did not start, see $S/$tag.log"; return 1; }
local code; code=$(dc_chat "$PC" "$S/$tag-1.json" "$(dc_msgs "$DC_U1")")
DC_C1=$(json_str content "$S/$tag-1.json")
[ "$code" = 200 ] && [ -n "$DC_C1" ] || { fail "diskcache/$tag" "turn 1: HTTP $code"; stop_server "$DC_PID"; return 1; }
DC_M2=$(dc_msgs "$DC_U1" "$DC_C1" "$DC_U2")
code=$(dc_chat "$PC" "$S/$tag-2.json" "$DC_M2")
DC_REF2=$(json_str content "$S/$tag-2.json"); DC_PN_REF=$(json_num prompt_n "$S/$tag-2.json"); DC_N2=$(json_num prompt_tokens "$S/$tag-2.json")
stop_server "$DC_PID"
# the same turn 2 on a fresh server: the full prefill the cache saves
dc_start "$tag-full" "" -- "$@" && dc_chat "$PC" "$S/$tag-full.json" "$DC_M2" > /dev/null
DC_MS_FULL=$(sed -n 's/.*"prompt_ms":\([0-9.]*\).*/\1/p' "$S/$tag-full.json" | head -1)
stop_server "$DC_PID"
[ "$code" = 200 ] && [ -n "$DC_REF2" ] && [ "${DC_PN_REF:-0}" -gt 0 ] || { fail "diskcache/$tag" "turn 2: HTTP $code"; return 1; }
log "diskcache: reference $tag: turn 2 prompt_n $DC_PN_REF of $DC_N2 tokens, a full prefill of turn 2 took ${DC_MS_FULL} ms"
}
# dc_restore TAG DIR COLD ENV... -- ARGS... : turn 1, SIGTERM (COLD=1: forced cold by a fake waiter first),
# restart, turn 2 -> restored from disk, prompt_n == reference, output == reference
dc_restore() {
local tag=$1 dir=$2 cold=$3; shift 3
local envs=(); while [ "$1" != -- ]; do envs+=("$1"); shift; done; shift
rm -rf "$dir"
dc_start "$tag-a" "$dir" "${envs[@]}" -- "$@" || { fail "diskcache/$tag" "server did not start, see $S/$tag-a.log"; return 1; }
local code c1 w=; code=$(dc_chat "$PC" "$S/$tag-a.json" "$(dc_msgs "$DC_U1")"); c1=$(json_str content "$S/$tag-a.json")
if [ "$cold" = 1 ]; then
( exec 9<"$ARENA/want.lock"; flock -s 9; exec sleep 120 ) & w=$!; track "$w"
wait_holder "" 10000 > /dev/null || log "warning: $tag-a did not go cold"
fi
local was_cold=no; [ "$cold" = 1 ] && [ -z "$(token_holder)" ] && grep -q 'server entering sleeping state' "$S/$tag-a.log" && was_cold=yes
dc_term
[ -n "$w" ] && { kill "$w" 2>/dev/null; wait "$w" 2>/dev/null; }
local rc1=$RC exit1=$DC_EXIT_MS free1=$DC_FREE_MS nf
nf=$(find "$dir" -name '*.lpcd' 2>/dev/null | wc -l)
local f1; f1=$(find "$dir" -name '*.lpcd' 2>/dev/null | head -1); [ -n "$f1" ] && cp "$f1" "$S/$tag-turn1.lpcd"
dc_start "$tag-b" "$dir" "${envs[@]}" -- "$@" || { fail "diskcache/$tag" "restart failed, see $S/$tag-b.log"; return 1; }
code=$(dc_chat "$PC" "$S/$tag-b.json" "$DC_M2")
local c2 pn ms nres; c2=$(json_str content "$S/$tag-b.json"); pn=$(json_num prompt_n "$S/$tag-b.json")
ms=$(sed -n 's/.*"prompt_ms":\([0-9.]*\).*/\1/p' "$S/$tag-b.json" | head -1)
nres=$(grep -c 'disk cache: restored' "$S/$tag-b.log")
local ok=1
[ "$code" = 200 ] && [ "$rc1" -eq 0 ] && [ "$nf" -eq 1 ] && [ "$nres" -eq 1 ] && [ "$c1" = "$DC_C1" ] && \
[ "${pn:-0}" = "$DC_PN_REF" ] && [ "${pn:-0}" -lt "${DC_N2:-0}" ] && [ "$c2" = "$DC_REF2" ] && ok=0
[ "$cold" = 1 ] && [ $was_cold != yes ] && ok=1
check "diskcache/$tag" $ok \
"$([ "$cold" = 1 ] && echo "cold before SIGTERM: $was_cold, ")exit $rc1 in ${exit1} ms$([ "$free1" -ge 0 ] && echo ", token free in ${free1} ms"), $nf file(s); restart: $nres restore(s), turn 2 prompt_n $pn (reference $DC_PN_REF) of $DC_N2 tokens in ${ms} ms (full prefill ${DC_MS_FULL} ms), output $([ "$c2" = "$DC_REF2" ] && echo == || echo '!=') reference"
log "diskcache: $tag: $(grep -h 'disk cache: \(copied\|saved\)' "$S/$tag-a.log" | sed 's/.*disk cache: //' | tr '\n' ';') $(grep -h 'disk cache: restored' "$S/$tag-b.log" | sed 's/.*disk cache: //')"
stop_server "$DC_PID"
}
run_diskcache() {
[ -f "$SMALL" ] || { fail diskcache "small model missing: $SMALL"; return; }
DC_U1="$(moex_prompt 12)"
DC_U2="Which town ships the fewest crates of salt each week?"
dc_ref dc-ref || return
# (1) save on SIGTERM, restore after a restart: plain, arbiter env warm, arbiter env cold
dc_restore restore "$S/dc" 0 -- || return
[ -f "$S/restore-turn1.lpcd" ] || { fail diskcache "no file saved on SIGTERM, see $S/restore-a.log"; return; }
dc_restore restore-arbiter "$S/dc-arb" 0 "${DC_ENV[@]}" --
dc_restore restore-cold "$S/dc-cold" 1 "${DC_ENV[@]}" --
# shutdown cost: the same warm arbiter shutdown without the cache
rm -rf "$S/dc-none"
dc_start dc-none "" "${DC_ENV[@]}" -- && dc_chat "$PC" "$S/dc-none.json" "$(dc_msgs "$DC_U1")" > /dev/null
dc_term
log "diskcache: warm arbiter shutdown without the cache: exit $RC in ${DC_EXIT_MS} ms, token free in ${DC_FREE_MS} ms"
# (2) no restore for a prompt that does not start with a saved one, or under another key
local code pn n nres
dc_start dc-np "$S/dc" -- || { fail diskcache/start "see $S/dc-np.log"; return; }
code=$(dc_chat "$PC" "$S/dc-np.json" "$(dc_msgs "Variant B. $DC_U1")")
pn=$(json_num prompt_n "$S/dc-np.json"); n=$(json_num prompt_tokens "$S/dc-np.json"); nres=$(grep -c 'disk cache: restored' "$S/dc-np.log")
check diskcache/no-prefix $(( code == 200 && nres == 0 && pn == n ? 0 : 1 )) "other conversation: HTTP $code, $nres restore(s), prompt_n $pn of $n"
stop_server "$DC_PID"
local tag args
ln -sf "$SMALL" "$S/other-path.gguf"
for tag in key-ctx key-ctk key-model; do
case $tag in
key-ctx) args=(-c 8192) ;;
key-ctk) args=(-ctk q8_0) ;;
key-model) args=(-m "$S/other-path.gguf") ;;
esac
rm -rf "$S/dc-$tag"; mkdir -p "$S/dc-$tag"; cp "$S/restore-turn1.lpcd" "$S/dc-$tag/"
dc_start "dc-$tag" "$S/dc-$tag" -- "${args[@]}" || { fail "diskcache/$tag" "see $S/dc-$tag.log"; continue; }
code=$(dc_chat "$PC" "$S/dc-$tag.json" "$DC_M2")
pn=$(json_num prompt_n "$S/dc-$tag.json"); n=$(json_num prompt_tokens "$S/dc-$tag.json"); nres=$(grep -c 'disk cache: restored' "$S/dc-$tag.log")
check "diskcache/$tag" $(( code == 200 && nres == 0 && pn == n ? 0 : 1 )) \
"${args[*]}: HTTP $code, $nres restore(s), prompt_n $pn of $n; index: $(grep -h 'disk cache: .*prompts of this model' "$S/dc-$tag.log" | sed 's/.*, \([0-9]* prompts\)/\1/')"
stop_server "$DC_PID"
done
# (3) corrupt and truncated files are removed, never used
local size; size=$(stat -c %s "$S/restore-turn1.lpcd")
rm -rf "$S/dc-bad"; mkdir -p "$S/dc-bad"
head -c $((size / 2)) "$S/restore-turn1.lpcd" > "$S/dc-bad/trunc.lpcd"
cp "$S/restore-turn1.lpcd" "$S/dc-bad/flip.lpcd"
printf '\377' | dd of="$S/dc-bad/flip.lpcd" bs=1 seek=$((size / 2)) conv=notrunc 2>/dev/null
dc_start dc-bad "$S/dc-bad" -- || { fail diskcache/corrupt "see $S/dc-bad.log"; return; }
local trunc_gone=no; [ -e "$S/dc-bad/trunc.lpcd" ] || trunc_gone=yes
code=$(dc_chat "$PC" "$S/dc-bad.json" "$DC_M2")
pn=$(json_num prompt_n "$S/dc-bad.json"); nres=$(grep -c 'disk cache: restored' "$S/dc-bad.log")
local flip_gone=no; [ -e "$S/dc-bad/flip.lpcd" ] || flip_gone=yes
check diskcache/corrupt $([ $trunc_gone = yes ] && [ $flip_gone = yes ] && [ "$code" = 200 ] && [ "$nres" -eq 0 ] && [ "$pn" = "$DC_N2" ] && [ "$(json_str content "$S/dc-bad.json")" = "$DC_REF2" ] && echo 0 || echo 1) \
"truncated file removed at startup: $trunc_gone, byte-flipped file removed at load: $flip_gone, $nres restore(s), prompt_n $pn of $DC_N2, output $([ "$(json_str content "$S/dc-bad.json")" = "$DC_REF2" ] && echo == || echo '!=') reference"
stop_server "$DC_PID"
# (3') LRU size limit: room for one file only, the second save evicts the first
local cap=$(( size * 3 / 2 / 1048576 + 1 )) files total
rm -rf "$S/dc-lru"
dc_start dc-lru1 "$S/dc-lru" -- --cache-disk-size "$cap" && dc_chat "$PC" "$S/dc-lru1.json" "$(dc_msgs "$DC_U1")" > /dev/null
stop_server "$DC_PID"
dc_start dc-lru2 "$S/dc-lru" -- --cache-disk-size "$cap" && dc_chat "$PC" "$S/dc-lru2.json" "$(dc_msgs "Variant B. $DC_U1")" > /dev/null
stop_server "$DC_PID"
files=$(find "$S/dc-lru" -name '*.lpcd' | wc -l); total=$(du -cb "$S"/dc-lru/*.lpcd 2>/dev/null | tail -1 | cut -f1)
check diskcache/lru $(( files == 1 && ${total:-0} <= cap * 1048576 && $(grep -c 'size limit, removing' "$S/dc-lru2.log") == 1 ? 0 : 1 )) \
"limit $cap MiB (file ~$((size / 1048576)) MiB): $files file(s), $((${total:-0} / 1048576)) MiB after two saves, $(grep -c 'size limit, removing' "$S/dc-lru2.log") eviction(s)"
# (3'') write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored -> EFBIG): logged, exit 0, nothing left, next start serves
rm -rf "$S/dc-full"
SRV_WRAP=(bash -c 'trap "" XFSZ; ulimit -f 1024; exec "$@"' _)
dc_start dc-full "$S/dc-full" -- && dc_chat "$PC" "$S/dc-full.json" "$(dc_msgs "$DC_U1")" > /dev/null
SRV_WRAP=()
dc_term
local left; left=$(find "$S/dc-full" -type f 2>/dev/null | wc -l)
dc_start dc-full2 "$S/dc-full" -- || { fail diskcache/write-error "restart failed, see $S/dc-full2.log"; return; }
code=$(dc_chat "$PC" "$S/dc-full2.json" "$DC_M2")
check diskcache/write-error $([ "$RC" -eq 0 ] && [ "$left" -eq 0 ] && grep -q 'disk cache: failed to write' "$S/dc-full.log" && [ "$code" = 200 ] && echo 0 || echo 1) \
"exit $RC in ${DC_EXIT_MS} ms, '$(grep -h 'disk cache: failed to write' "$S/dc-full.log" | sed 's/.*: \([^:]*\), not saved/\1/' | head -1)', $left file(s) left, next start HTTP $code"
stop_server "$DC_PID"
# (4) with an mmproj loaded, as the production 35B and gemma-26B run: a text-only chat is saved and
# restored (server_tokens::get_tokens() asserts in that mode); a prompt with an image is never
# restored or saved and does not crash. Own reference: the CPU references above stay as they are.
if [ ! -f "$MMPROJ" ]; then
skip diskcache/mmproj "mmproj missing: $MMPROJ"
else
DC_MM=(--mmproj "$MMPROJ" --no-mmproj-offload)
local DC_ENV_MM=("${DC_ENV[@]}" LLAMA_MMPROJ_ONDEMAND=1)
if dc_ref dc-ref-mm; then
dc_restore restore-mmproj "$S/dc-mm" 0 "${DC_ENV_MM[@]}" --
local img ans1 ans2 c2 nres1 ncopy
img=$(make_red_ppm)
printf '{"messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"data:image/x-portable-pixmap;base64,%s"}},{"type":"text","text":"What is the main color of this image? Answer with one word."}]}],"max_tokens":16,%s}' \
"$img" "$GREEDY" > "$S/dc-img-body.json"
rm -rf "$S/dc-img"; mkdir -p "$S/dc-img"; cp "$S/restore-mmproj-turn1.lpcd" "$S/dc-img/" 2>/dev/null
if dc_start dc-img "$S/dc-img" "${DC_ENV_MM[@]}" --; then
# image first (no restore), then the text chat (restored), then an image again: the slot holds media at the SIGTERM
code=$(curl -sS -m 300 -o "$S/dc-img1.json" -w '%{http_code}' -H 'Content-Type: application/json' -d @"$S/dc-img-body.json" "http://127.0.0.1:$PC/v1/chat/completions" 2>>"$S/curl.err")
ans1=$(json_str content "$S/dc-img1.json"); nres1=$(grep -c 'disk cache: restored' "$S/dc-img.log")
dc_chat "$PC" "$S/dc-img2.json" "$DC_M2" > /dev/null; c2=$(json_str content "$S/dc-img2.json")
local code3; code3=$(curl -sS -m 300 -o "$S/dc-img3.json" -w '%{http_code}' -H 'Content-Type: application/json' -d @"$S/dc-img-body.json" "http://127.0.0.1:$PC/v1/chat/completions" 2>>"$S/curl.err")
ans2=$(json_str content "$S/dc-img3.json")
dc_term
ncopy=$(grep -c 'disk cache: copied' "$S/dc-img.log")
local nf; nf=$(find "$S/dc-img" -name '*.lpcd' | wc -l)
check diskcache/mmproj-image $([ "$code" = 200 ] && [ "$code3" = 200 ] && [ "$nres1" -eq 0 ] && [ "$(grep -c 'disk cache: restored' "$S/dc-img.log")" -eq 1 ] && \
[ "$c2" = "$DC_REF2" ] && [ "$RC" -eq 0 ] && [ "$ncopy" -eq 0 ] && [ "$nf" -eq 1 ] && echo "$ans1$ans2" | grep -qi red && echo 0 || echo 1) \
"image: HTTP $code '$ans1', $nres1 restore(s); text chat after it: $(grep -c 'disk cache: restored' "$S/dc-img.log") restore(s), output $([ "$c2" = "$DC_REF2" ] && echo == || echo '!=') reference; image again: HTTP $code3 '$ans2'; SIGTERM with media in the slot: exit $RC, $ncopy state(s) copied, $nf file(s)"
else
fail diskcache/mmproj-image "server did not start, see $S/dc-img.log"
fi
fi
DC_MM=(--no-mmproj)
fi
# (1') GPU: the state is saved from VRAM (warm) and from a host shadow (cold), restored into VRAM
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
if [ $free_mb -lt 1536 ]; then
skip diskcache/gpu "only $free_mb MiB VRAM free (< 1536)"
elif dc_ref dc-ref-gpu "${DC_GPU[@]}"; then
dc_restore gpu-warm "$S/dc-gpu" 0 "${DC_ENV[@]}" -- "${DC_GPU[@]}"
dc_restore gpu-cold "$S/dc-gpu" 1 "${DC_ENV[@]}" -- "${DC_GPU[@]}"
dc_start dc-none-gpu "" "${DC_ENV[@]}" -- "${DC_GPU[@]}" && dc_chat "$PC" "$S/dc-none-gpu.json" "$(dc_msgs "$DC_U1")" > /dev/null
dc_term
log "diskcache: warm GPU arbiter shutdown without the cache: exit $RC in ${DC_EXIT_MS} ms, token free in ${DC_FREE_MS} ms"
fi
local bad_lines; bad_lines=$(scan_logs "$S"/dc-*.log "$S"/restore*.log "$S"/gpu-*.log)
check diskcache/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) "${bad_lines:-clean}"
}
# ================================================================ main
if [ -n "$PERF_REPORT" ]; then
[ -r "$PERF_REPORT" ] || { echo "cannot read $PERF_REPORT" >&2; exit 2; }
perf_report "$PERF_REPORT"; exit 0
fi
[ $DO_OPS -eq 1 ] && run_ops
[ $DO_CONCAT -eq 1 ] && run_concat
[ $DO_ARB -eq 1 ] && run_arbiter
[ $DO_PERF -eq 1 ] && run_perf
[ $DO_MOEC -eq 1 ] && run_moecache
[ $DO_MOEX -eq 1 ] && run_moe_exact
[ $DO_DC -eq 1 ] && run_diskcache
exit 0