Runs beside production. The model is the small hybrid (Qwen3.5-0.8B), CPU only (-dev none -ngl 0, no VRAM), with chat requests and greedy decoding. A reference server that never restarts runs turn 1 and turn 2. The candidate saves turn 1 on SIGTERM and restarts. Turn 2 must then be restored from disk with the same prompt_n as the reference (only the suffix processed) and the same answer. Cases: - restore plain, with the arbiter env warm, and forced cold by a fake waiter on want.lock (the save from a released state). Each case records the exit time and SIGTERM-to-token-free, also once without the cache; - no restore for another conversation, or with another n_ctx, KV type or model path (other key); - a truncated file is removed at startup and a byte-flipped one at load, never used; - the size limit evicts the oldest file; - a write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored: EFBIG) is logged, the exit stays 0 with no file left, and the next start serves; - the same restore on the GPU (-ngl 99, arbiter env, warm and cold) when >= 1.5 GiB of VRAM is free, else SKIP. start_server gains SRV_WRAP, a command prefix that execs the server. First run (2026-10-25, beside production): 11 PASS, 0 FAIL; the GPU leg SKIPped (1219 MiB free). The first run found the CPU_REPACK release crash (exit 139 in the cold case). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1151 lines
70 KiB
Bash
1151 lines
70 KiB
Bash
#!/usr/bin/env bash
|
|
# fork-validate.sh - validate a candidate build of the RX 580 llama.cpp fork against a reference build.
|
|
#
|
|
# Runs ON THE POD (Vulkan / RADV, RX 580). Prints PASS/FAIL/SKIP per check plus a summary; exits
|
|
# nonzero if any check FAILs. Only touches its own scratch dir under /tmp, its own scratch ports and
|
|
# processes it started itself (killed by explicit PID). Build dirs are only read.
|
|
#
|
|
# fork-validate.sh --build /root/arms/X/build --ref /root/llama.cpp/build [checks] [options]
|
|
#
|
|
# Checks (no check flag = --ops --concat --arbiter):
|
|
# --ops test-backend-ops on Vulkan0 for CONCAT, FLASH_ATTN_EXT, MUL_MAT, MUL_MAT_ID, GET_ROWS
|
|
# -> the fork's kernels and upstream's agree with the CPU reference; asserts the case count
|
|
# is > 0 because "-b Vulkan" (no index) silently runs nothing and still prints OK.
|
|
# --concat Vulkan concat tiled-transpose fast path (ggml_vk_concat, gate src_transposed_2d)
|
|
# -> it fires at prefill and NOT at decode on a real hybrid (delta-net) model; it is the
|
|
# fork's biggest prefill win, and taking it at decode once cost -17% decode.
|
|
# Needs a build carrying the GGML_VK_CONCAT_FASTPATH knob (=0 off, =log report);
|
|
# either --build has it or pass --concat-build DIR (candidate + the knob patch).
|
|
# --arbiter on-demand VRAM sharing (LLAMA_SLEEP_VRAM_ONLY ...) with two real servers on a scratch
|
|
# arena -> a cold server really gives its VRAM back (v-vram-release: the card-wide
|
|
# delta over a baseline taken before the scratch servers start must drop by >= 70% of
|
|
# what it added warm, and come back on restore; production's own DRM clients are
|
|
# re-read each time and the check SKIPs if they moved by > 64 MiB), handoffs never
|
|
# strand, release/restore is bit-exact, multimodal survives it,
|
|
# and every shutdown path (mid-release, mid-wait, mid-generation, cold, warm, kill -9 of
|
|
# a waiter) exits cleanly and hands the GPU token on. These are the past production
|
|
# incidents (stranded requests, wedged token, exit 134 on cold teardown).
|
|
# Then a-handoff-latency: ordinary handoff latency of --build vs --ref in the same
|
|
# session (fresh server pairs, blocks ref,cand,cand,ref x 16 handoffs) -> FAIL if the
|
|
# candidate median exceeds the reference median by more than max(25%, 200 ms). Two
|
|
# production deploys were lost to arbiter changes that only showed up here.
|
|
# Needs >= 1536 MiB free VRAM (FAILs otherwise); beside a warm gemma-26B only about
|
|
# 1.2 GiB is free, so then run it in the --perf swapdown window (about 2 minutes).
|
|
# --perf llama-bench, candidate vs reference, alternating ref,cand,cand,ref, on the 35B MoE
|
|
# -> no prefill regression > 5% at depth 0/16k/32k (FAIL), decode tg256@d4096
|
|
# reported; peak VRAM per arm (FAIL if > 95% of the card). Compares MEDIANS of every
|
|
# repetition sample pooled over a build's two arms (4 per prefill cell, 6 for decode),
|
|
# never means: CPU contention on the node produces single catastrophic samples.
|
|
# Needs llama-swap STOPPED (run it under /root/bench/moe/swapdown.sh); refuses to
|
|
# run if any other llama process is up. Samples are kept in <scratch>/perf.tsv.
|
|
# --moecache MODEL
|
|
# one arbiter server with --moe-cache-mib N on a MoE model, forced cold/warm twice
|
|
# -> greedy output identical before/after (the expert-cache SLRU is re-initialised on
|
|
# release; a stale SLRU maps experts to empty banks and produces garbage).
|
|
# --moe-exact MODEL
|
|
# the MoE routing-ids readback skip (F1, sched_copy_experts) is exact: MODEL (a MoE,
|
|
# e.g. the --perf model) with every expert on the host, greedy, one prompt of >= 2x
|
|
# the skip threshold ceil(4 * n_expert / n_expert_used) tokens (read from the GGUF;
|
|
# 128 on the 35B) -> candidate output byte-identical to the reference, and, if the
|
|
# candidate has the LLAMA_MOE_IDS_SKIP knob, to the candidate with the skip forced off
|
|
# (=0); =log proves the skip engaged. Without the knob skip-engaged is SKIPped with the
|
|
# batch-size argument. Needs the GPU to itself like --perf (same guard); about 3 model
|
|
# loads, so run it in the same swapdown window as --perf.
|
|
# --diskcache the disk prompt cache (--cache-disk-path) on the small hybrid model, CPU only (no VRAM;
|
|
# runs beside production), plus the GPU path when >= 1.5 GiB VRAM is free
|
|
# -> a slot state saved on SIGTERM (plain, arbiter warm, arbiter cold) is restored after a
|
|
# restart: only the suffix is processed (prompt_n == a server that never restarted) and
|
|
# the greedy answer is identical; no restore for another conversation or another model
|
|
# path / n_ctx / KV type; truncated and corrupt files are removed, never used; the size
|
|
# limit evicts the oldest file; a write error (RLIMIT_FSIZE) is logged and exit stays 0.
|
|
#
|
|
# Options:
|
|
# --concat-build DIR build with the GGML_VK_CONCAT_FASTPATH knob for --concat (default: --build)
|
|
# --small MODEL small hybrid model for --concat/--arbiter (default: Qwen3.5-0.8B UD-Q4_K_XL)
|
|
# --mmproj FILE its multimodal projector (default: mmproj-F16 next to it)
|
|
# --perf-model MODEL model for --perf (default: Qwen3.6-35B-A3B UD-Q4_K_XL)
|
|
# --moecache-args S extra server args for --moecache (default: "-ngl 99 --n-cpu-moe 30 -c 4096")
|
|
# --moecache-mib N cache size for --moecache (default 2048; 256 never hits on a 26B and proves nothing)
|
|
# --port-base N scratch ports N..N+2 (default 8091)
|
|
# --keep keep the scratch dir even when everything passed (it is kept on any FAIL)
|
|
# --moe-exact-args S server args for --moe-exact (default: "-ngl 99 --n-cpu-moe 99 -c 4096 -b 2048
|
|
# -ub 2048 -t 6"; keep the experts on the host and -ub >= the prompt)
|
|
# --perf-report TSV only re-aggregate a saved <scratch>/perf.tsv of an earlier --perf run and
|
|
# apply the same gates (no GPU, no --build/--ref needed); runs nothing else
|
|
#
|
|
# Requirements: bash, coreutils, awk, sed, grep, curl, flock(1); gdb optional (makes the
|
|
# SIGTERM-during-release case deterministic; it is not in the pod image and is lost on a pod restart,
|
|
# then the case falls back to polling the log). No python/jq.
|
|
#
|
|
# Test only: FV_TEST_SHARED_GPU=1 skips the exclusive-GPU guard of --perf / --moe-exact, for running
|
|
# the script against stub binaries. Never set it for a real measurement.
|
|
|
|
set -u
|
|
|
|
# ---------------------------------------------------------------- defaults
|
|
HF=/root/.cache/huggingface/hub
|
|
SMALL_DIR=$HF/models--unsloth--Qwen3.5-0.8B-GGUF/snapshots/6ab461498e2023f6e3c1baea90a8f0fe38ab64d0
|
|
SMALL=$SMALL_DIR/Qwen3.5-0.8B-UD-Q4_K_XL.gguf
|
|
MMPROJ=
|
|
PERF_MODEL=$HF/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
|
|
MOECACHE_MODEL=
|
|
MOECACHE_ARGS="-ngl 99 --n-cpu-moe 30 -c 4096"
|
|
MOECACHE_MIB=2048
|
|
MOEX_MODEL=
|
|
MOEX_ARGS="-ngl 99 --n-cpu-moe 99 -c 4096 -b 2048 -ub 2048 -t 6"
|
|
BUILD= ; REF= ; CONCAT_BUILD=
|
|
PORT_BASE=8091
|
|
KEEP=0
|
|
DO_OPS=0 DO_CONCAT=0 DO_ARB=0 DO_PERF=0 DO_MOEC=0 DO_MOEX=0 DO_DC=0
|
|
PERF_REPORT=
|
|
|
|
usage() { awk 'NR > 1 && /^#/ {sub(/^# ?/, ""); print; next} NR > 1 {exit}' "$0"; exit 2; }
|
|
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--build) BUILD=$2; shift 2 ;;
|
|
--ref) REF=$2; shift 2 ;;
|
|
--ops) DO_OPS=1; shift ;;
|
|
--concat) DO_CONCAT=1; shift ;;
|
|
--arbiter) DO_ARB=1; shift ;;
|
|
--diskcache) DO_DC=1; shift ;;
|
|
--perf) DO_PERF=1; shift ;;
|
|
--moecache) DO_MOEC=1; MOECACHE_MODEL=$2; shift 2 ;;
|
|
--moecache-args) MOECACHE_ARGS=$2; shift 2 ;;
|
|
--moecache-mib) MOECACHE_MIB=$2; shift 2 ;;
|
|
--moe-exact) DO_MOEX=1; MOEX_MODEL=$2; shift 2 ;;
|
|
--moe-exact-args) MOEX_ARGS=$2; shift 2 ;;
|
|
--concat-build) CONCAT_BUILD=$2; shift 2 ;;
|
|
--small) SMALL=$2; shift 2 ;;
|
|
--mmproj) MMPROJ=$2; shift 2 ;;
|
|
--perf-model) PERF_MODEL=$2; shift 2 ;;
|
|
--port-base) PORT_BASE=$2; shift 2 ;;
|
|
--keep) KEEP=1; shift ;;
|
|
--perf-report) PERF_REPORT=$(readlink -f "$2"); shift 2 ;;
|
|
-h|--help) usage ;;
|
|
*) echo "unknown argument: $1" >&2; usage ;;
|
|
esac
|
|
done
|
|
[ -n "$PERF_REPORT" ] || { [ -n "$BUILD" ] && [ -n "$REF" ]; } || usage
|
|
if [ $((DO_OPS + DO_CONCAT + DO_ARB + DO_PERF + DO_MOEC + DO_MOEX + DO_DC)) -eq 0 ]; then
|
|
DO_OPS=1 DO_CONCAT=1 DO_ARB=1
|
|
fi
|
|
[ -n "$MMPROJ" ] || MMPROJ=$(dirname "$SMALL")/mmproj-F16.gguf
|
|
PA=$PORT_BASE PB=$((PORT_BASE + 1)) PC=$((PORT_BASE + 2))
|
|
|
|
# ---------------------------------------------------------------- reporting
|
|
RESULTS=()
|
|
NFAIL=0
|
|
ts() { date +%H:%M:%S; }
|
|
log() { echo "[$(ts)] $*"; }
|
|
pass() { RESULTS+=("PASS $1: $2"); echo "[$(ts)] PASS $1: $2"; }
|
|
fail() { RESULTS+=("FAIL $1: $2"); echo "[$(ts)] FAIL $1: $2"; NFAIL=$((NFAIL + 1)); }
|
|
skip() { RESULTS+=("SKIP $1: $2"); echo "[$(ts)] SKIP $1: $2"; }
|
|
# check NAME CONDITION-RESULT(0/1) MESSAGE
|
|
check() { if [ "$2" -eq 0 ]; then pass "$1" "$3"; else fail "$1" "$3"; fi; }
|
|
now_ms() { date +%s%3N; }
|
|
|
|
# ---------------------------------------------------------------- scratch + cleanup
|
|
S=$(mktemp -d /tmp/fork-validate.XXXXXX)
|
|
ARENA=$S/arena
|
|
mkdir -p "$ARENA"
|
|
cd "$S" || exit 2 # anything written to a relative path lands in the scratch dir
|
|
PIDS=() # "pid:starttime" of every background process we start (starttime guards PID reuse)
|
|
|
|
pid_alive() { # alive and not a zombie
|
|
[ -r "/proc/$1/stat" ] || return 1
|
|
local st; st=$(sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | awk '{print $1}') || return 1
|
|
[ -n "$st" ] && [ "$st" != Z ]
|
|
}
|
|
starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | awk '{print $20}'; }
|
|
track() { PIDS+=("$1:$(starttime "$1")"); }
|
|
ours() { # "pid:starttime" -> pid if that exact process is still alive
|
|
local p=${1%%:*} t=${1#*:}
|
|
pid_alive "$p" && [ "$(starttime "$p")" = "$t" ] && echo "$p"
|
|
}
|
|
|
|
cleanup() {
|
|
local e p
|
|
for e in "${PIDS[@]}"; do p=$(ours "$e") && kill -TERM "$p" 2>/dev/null; done
|
|
local i=0
|
|
while [ $i -lt 50 ]; do
|
|
local any=0
|
|
for e in "${PIDS[@]}"; do ours "$e" > /dev/null && any=1; done
|
|
[ $any -eq 0 ] && break
|
|
sleep 0.1; i=$((i + 1))
|
|
done
|
|
for e in "${PIDS[@]}"; do p=$(ours "$e") && kill -KILL "$p" 2>/dev/null; done
|
|
wait 2>/dev/null
|
|
}
|
|
finish() {
|
|
local rc=$?
|
|
trap - EXIT INT TERM HUP
|
|
cleanup
|
|
echo
|
|
echo "==================== fork-validate summary ===================="
|
|
echo "candidate: $BUILD ($(git -C "$BUILD/.." rev-parse --short HEAD 2>/dev/null || echo '?'))"
|
|
echo "reference: $REF ($(git -C "$REF/.." rev-parse --short HEAD 2>/dev/null || echo '?'))"
|
|
local r; for r in "${RESULTS[@]}"; do echo "$r"; done
|
|
local np ns
|
|
np=$(printf '%s\n' "${RESULTS[@]}" | grep -c '^PASS')
|
|
ns=$(printf '%s\n' "${RESULTS[@]}" | grep -c '^SKIP')
|
|
echo "total: $np PASS, $NFAIL FAIL, $ns SKIP"
|
|
if [ $NFAIL -gt 0 ] || [ $KEEP -eq 1 ] || [ $rc -ne 0 ]; then
|
|
echo "scratch kept: $S"
|
|
else
|
|
rm -rf "$S"
|
|
fi
|
|
[ $NFAIL -gt 0 ] && exit 1
|
|
exit $rc
|
|
}
|
|
trap finish EXIT
|
|
trap 'echo "interrupted"; NFAIL=$((NFAIL + 1)); exit 130' INT TERM HUP
|
|
|
|
# ---------------------------------------------------------------- environment sanity
|
|
for v in $(env | sed -n 's/^\(GGML_[A-Za-z0-9_]*\|LLAMA_\(SLEEP\|VRAM\|MMPROJ\|ARG\|CACHE\|LOG\|MOE\)[A-Za-z0-9_]*\)=.*/\1/p'); do
|
|
log "unsetting inherited $v"; unset "$v"
|
|
done
|
|
[ -n "$PERF_REPORT" ] || for d in "$BUILD" "$REF"; do
|
|
[ -x "$d/bin/llama-server" ] || { echo "no llama-server in $d/bin" >&2; exit 2; }
|
|
done
|
|
case "$ARENA" in /dev/shm/llama-vram*) echo "refusing the production arena" >&2; exit 2 ;; esac
|
|
VRAM_USED=$(ls /sys/class/drm/card*/device/mem_info_vram_used 2>/dev/null | head -1)
|
|
VRAM_TOTAL=$(ls /sys/class/drm/card*/device/mem_info_vram_total 2>/dev/null | head -1)
|
|
vram_mb() { echo $(( $(cat "$VRAM_USED") / 1048576 )); }
|
|
port_busy() { # LISTEN socket on the port (v4 or v6)
|
|
local hex; hex=$(printf '%04X' "$1")
|
|
awk -v h=":$hex" '$4=="0A" && substr($2, length($2)-4)==h {f=1} END {exit !f}' /proc/net/tcp /proc/net/tcp6 2>/dev/null
|
|
}
|
|
[ -n "$PERF_REPORT" ] || for p in $PA $PB $PC; do
|
|
port_busy "$p" && { echo "scratch port $p is in use" >&2; exit 2; }
|
|
done
|
|
log "scratch dir $S, ports $PA/$PB/$PC, VRAM used $(vram_mb) MiB"
|
|
|
|
# ---------------------------------------------------------------- helpers: servers
|
|
# start_server NAME BUILD PORT ENV-ASSIGNMENTS... -- SERVER-ARGS... (sets SPID, log at $S/NAME.log)
|
|
# SRV_WRAP: optional command prefix that execs its arguments (the PID stays the server's)
|
|
SRV_WRAP=()
|
|
start_server() {
|
|
local name=$1 build=$2 port=$3; shift 3
|
|
local envs=()
|
|
while [ $# -gt 0 ] && [ "$1" != -- ]; do envs+=("$1"); shift; done
|
|
shift
|
|
"${SRV_WRAP[@]}" env LD_LIBRARY_PATH="$build/bin" "${envs[@]}" "$build/bin/llama-server" \
|
|
--host 127.0.0.1 --port "$port" "$@" >> "$S/$name.log" 2>&1 &
|
|
SPID=$!
|
|
track "$SPID"
|
|
echo "=== start pid $SPID $(date +%T.%N)" >> "$S/$name.log.events"
|
|
}
|
|
# wait_health PORT PID TIMEOUT_S
|
|
wait_health() {
|
|
local t0; t0=$(now_ms)
|
|
while [ $(( $(now_ms) - t0 )) -lt $(( $3 * 1000 )) ]; do
|
|
pid_alive "$2" || return 1
|
|
[ "$(curl -s -m 2 -o /dev/null -w '%{http_code}' "http://127.0.0.1:$1/health")" = 200 ] && return 0
|
|
sleep 0.2
|
|
done
|
|
return 1
|
|
}
|
|
# wait_exit PID TIMEOUT_MS -> prints elapsed ms, returns 1 on timeout. Then reap with `wait PID`.
|
|
wait_exit() {
|
|
local t0; t0=$(now_ms)
|
|
while pid_alive "$1"; do
|
|
[ $(( $(now_ms) - t0 )) -ge "$2" ] && { echo $(( $(now_ms) - t0 )); return 1; }
|
|
sleep 0.02
|
|
done
|
|
echo $(( $(now_ms) - t0 ))
|
|
}
|
|
# stop_server PID -> SIGTERM, wait up to 20 s, sets RC and EXIT_MS
|
|
stop_server() {
|
|
kill -TERM "$1" 2>/dev/null
|
|
EXIT_MS=$(wait_exit "$1" 20000) || { kill -KILL "$1" 2>/dev/null; }
|
|
wait "$1" 2>/dev/null; RC=$?
|
|
}
|
|
|
|
# ---------------------------------------------------------------- helpers: locks (/proc/locks)
|
|
lock_id() { # maj:min:inode as /proc/locks prints it (hex dev, decimal inode)
|
|
local d; d=$(( 16#$(stat -c %D "$1") ))
|
|
printf '%02x:%02x:%s' $(( (d >> 8) & 0xfff )) $(( (d & 0xff) | ((d >> 12) & 0xfff00) )) "$(stat -c %i "$1")"
|
|
}
|
|
lock_pids() { # FILE WRITE|READ -> PIDs holding a flock on it (granted, not blocked)
|
|
[ -e "$1" ] || return 0
|
|
awk -v id="$(lock_id "$1")" -v m="$2" '$2=="FLOCK" && $4==m && $6==id {print $5}' /proc/locks
|
|
}
|
|
token_holder() { lock_pids "$ARENA/token.lock" WRITE | head -1; }
|
|
holder_num() { local h; h=$(token_holder); echo "${h:-0}"; } # for arithmetic: 0 = nobody
|
|
# wait_holder PID TIMEOUT_MS : until PID holds token.lock ("" = until nobody does)
|
|
wait_holder() {
|
|
local t0; t0=$(now_ms)
|
|
while [ "$(token_holder)" != "$1" ]; do
|
|
[ $(( $(now_ms) - t0 )) -ge "$2" ] && return 1
|
|
sleep 0.01
|
|
done
|
|
echo $(( $(now_ms) - t0 ))
|
|
}
|
|
wait_want() { # PID TIMEOUT_MS : until PID holds want.lock shared (registered waiter)
|
|
local t0; t0=$(now_ms)
|
|
until lock_pids "$ARENA/want.lock" READ | grep -qx "$1"; do
|
|
[ $(( $(now_ms) - t0 )) -ge "$2" ] && return 1
|
|
sleep 0.01
|
|
done
|
|
}
|
|
|
|
# ---------------------------------------------------------------- helpers: VRAM attribution
|
|
# mem_info_vram_used is card-wide and includes production. Production's own share is read from the
|
|
# DRM fdinfo of every process this script did not start (drm-memory-vram, deduplicated by
|
|
# drm-client-id; summing fds double-counts shared buffers), so a production model that warms or
|
|
# cools during a measurement is detected instead of being attributed to the scratch servers.
|
|
drm_vram_kib() { # PID... -> KiB of VRAM held by the DRM clients of these processes
|
|
local p fs=()
|
|
for p in "$@"; do [ -d "/proc/$p/fdinfo" ] && fs+=("/proc/$p/fdinfo/"*); done
|
|
[ ${#fs[@]} -gt 0 ] || { echo 0; return; }
|
|
cat "${fs[@]}" 2>/dev/null | awk '/^pos:/ {c = ""} /^drm-client-id:/ {c = $2}
|
|
/^drm-memory-vram:/ && c != "" {v = $2; if ($3 == "MiB") v *= 1024; m[c] = v}
|
|
END {for (k in m) s += m[k]; print s + 0}'
|
|
}
|
|
prod_vram_mb() { # MiB of VRAM held by every process this script did not start
|
|
local p e mine=" " others=()
|
|
for e in "${PIDS[@]}"; do mine+="${e%%:*} "; done
|
|
for p in /proc/[0-9]*; do
|
|
p=${p#/proc/}
|
|
case "$mine" in *" $p "*) ;; *) others+=("$p") ;; esac
|
|
done
|
|
echo $(( $(drm_vram_kib "${others[@]}") / 1024 ))
|
|
}
|
|
|
|
# ---------------------------------------------------------------- helpers: requests
|
|
json_str() { sed -n 's/.*"'"$1"'":"\(\([^"\\]\|\\.\)*\)".*/\1/p' "$2" | head -1; }
|
|
json_num() { grep -o "\"$1\":[0-9]*" "$2" | head -1 | cut -d: -f2; }
|
|
# complete PORT OUTFILE JSON-BODY [TIMEOUT_S] -> prints HTTP code ("000" on timeout/refused)
|
|
complete() {
|
|
curl -sS -m "${4:-60}" -o "$2" -w '%{http_code}' -H 'Content-Type: application/json' \
|
|
-d "$3" "http://127.0.0.1:$1/completion" 2>>"$S/curl.err" || true
|
|
}
|
|
PROMPT='The history of the printing press began in'
|
|
GREEDY="\"temperature\":0,\"top_k\":1,\"seed\":42"
|
|
greedy_body() { # N_PREDICT CACHE_PROMPT [PROMPT]
|
|
echo "{\"prompt\":\"${3:-$PROMPT}\",\"n_predict\":$1,$GREEDY,\"cache_prompt\":$2}"
|
|
}
|
|
LONG_BODY="{\"prompt\":\"Write a very long story about a lighthouse keeper.\",\"n_predict\":4000,\"ignore_eos\":true,$GREEDY,\"stream\":true}"
|
|
# stream_bg PORT OUTFILE BODY -> starts a streaming request, sets CPID
|
|
stream_bg() {
|
|
curl -sS -N -m 300 -H 'Content-Type: application/json' -d "$3" \
|
|
"http://127.0.0.1:$1/completion" > "$2" 2>&1 &
|
|
CPID=$!; track "$CPID"
|
|
}
|
|
wait_tokens() { # FILE N TIMEOUT_MS : until N SSE data lines arrived
|
|
local t0; t0=$(now_ms)
|
|
while [ "$(grep -c '^data: ' "$1" 2>/dev/null)" -lt "$2" ]; do
|
|
[ $(( $(now_ms) - t0 )) -ge "$3" ] && return 1
|
|
sleep 0.05
|
|
done
|
|
}
|
|
scan_logs() { # NAME-PATTERN... -> prints offending lines
|
|
grep -h -n -E 'GGML_ASSERT|Assertion|terminate called|Segmentation fault|core dumped|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST|vk::.*Error|GGML_ABORT|fatal error' "$@" 2>/dev/null | head -20
|
|
}
|
|
|
|
# ================================================================ 1. ops
|
|
run_ops() {
|
|
local op out passed total
|
|
for op in CONCAT FLASH_ATTN_EXT MUL_MAT MUL_MAT_ID GET_ROWS; do
|
|
out=$S/ops-$op.log
|
|
log "ops: test-backend-ops -b Vulkan0 -o $op"
|
|
( cd "$S" && LD_LIBRARY_PATH="$BUILD/bin" timeout 1800 "$BUILD/bin/test-backend-ops" -b Vulkan0 -o "$op" ) \
|
|
> "$out" 2>&1
|
|
local rc=$?
|
|
# strip ANSI colours; "N/M tests passed" is printed once per tested backend
|
|
read -r passed total < <(sed 's/\x1b\[[0-9;]*m//g' "$out" | sed -n 's/^ *\([0-9]*\)\/\([0-9]*\) tests passed.*/\1 \2/p' | head -1)
|
|
passed=${passed:-0}; total=${total:-0}
|
|
if [ $rc -eq 0 ] && [ "$total" -gt 0 ] && [ "$passed" -eq "$total" ] \
|
|
&& sed 's/\x1b\[[0-9;]*m//g' "$out" | grep -q 'Backend Vulkan0: OK'; then
|
|
pass "ops/$op" "$passed/$total cases on Vulkan0"
|
|
else
|
|
fail "ops/$op" "rc=$rc, $passed/$total passed (0 cases = wrong backend name); see $out"
|
|
fi
|
|
done
|
|
}
|
|
|
|
# ================================================================ 2. concat fast path
|
|
# Method: llama-bench on the small hybrid model (qwen35: delta-net layers concat the conv state with
|
|
# the transposed qkv in build_conv_state). (1) with the knob =log, count fast-path dispatches in a
|
|
# prefill-only run (must be > 0, all with >= 32 tokens) and a decode-only run (must be 0, while the
|
|
# perf logger proves CONCAT nodes did execute); (2) GGML_VK_PERF_LOGGER CONCAT time at prefill: the
|
|
# unmodified candidate must match knob-on and be >= 2x faster than knob-off, which ties the
|
|
# candidate binary itself (no knob) to the fast path.
|
|
concat_us() { # LOG -> "count avg_us" over all perf-logger CONCAT lines
|
|
awk '/^CONCAT: / {n += $2; t += $(NF-1)} END {if (n) printf "%d %.2f\n", n, t/n; else print "0 0"}' "$1"
|
|
}
|
|
run_concat() {
|
|
local tb=$CONCAT_BUILD
|
|
[ -n "$tb" ] || tb=$BUILD
|
|
if ! grep -qa GGML_VK_CONCAT_FASTPATH "$tb/bin/libggml-vulkan.so" 2>/dev/null; then
|
|
fail concat "no GGML_VK_CONCAT_FASTPATH knob in $tb (build the candidate + knob patch, pass --concat-build)"
|
|
return
|
|
fi
|
|
[ -f "$SMALL" ] || { fail concat "small model missing: $SMALL"; return; }
|
|
local common=(-m "$SMALL" -ngl 99 -fa 1 -lm none -t 6)
|
|
local pp=(-p 512 -n 0 -b 512 -ub 512 -r 3)
|
|
local tg=(-p 0 -n 32 -r 1)
|
|
lb() { # BUILD OUT ENV... -- ARGS
|
|
local b=$1 out=$2; shift 2
|
|
local envs=(); while [ "$1" != -- ]; do envs+=("$1"); shift; done; shift
|
|
env LD_LIBRARY_PATH="$b/bin" GGML_VK_PERF_LOGGER=1 "${envs[@]}" "$b/bin/llama-bench" "${common[@]}" "$@" \
|
|
> "$out" 2>&1
|
|
}
|
|
log "concat: llama-bench prefill/decode arms (perf logger on)"
|
|
lb "$tb" "$S/cc-pp-log.log" GGML_VK_CONCAT_FASTPATH=log -- "${pp[@]}"
|
|
lb "$tb" "$S/cc-tg-log.log" GGML_VK_CONCAT_FASTPATH=log -- "${tg[@]}"
|
|
lb "$tb" "$S/cc-pp-off.log" GGML_VK_CONCAT_FASTPATH=0 -- "${pp[@]}"
|
|
lb "$BUILD" "$S/cc-pp-cand.log" -- "${pp[@]}"
|
|
lb "$tb" "$S/cc-pp-off2.log" GGML_VK_CONCAT_FASTPATH=0 -- "${pp[@]}"
|
|
lb "$BUILD" "$S/cc-pp-cand2.log" -- "${pp[@]}"
|
|
cat "$S/cc-pp-off2.log" >> "$S/cc-pp-off.log"; cat "$S/cc-pp-cand2.log" >> "$S/cc-pp-cand.log"
|
|
|
|
local n_pp n_pp_small n_tg n_tg_concat
|
|
n_pp=$(grep -c 'transpose fast path' "$S/cc-pp-log.log")
|
|
n_pp_small=$(sed -n 's/.*transpose fast path, transposed ne0=\([0-9]*\).*/\1/p' "$S/cc-pp-log.log" | awk '$1 < 32' | wc -l)
|
|
n_tg=$(grep -c 'transpose fast path' "$S/cc-tg-log.log")
|
|
read -r n_tg_concat _ < <(concat_us "$S/cc-tg-log.log")
|
|
local ne0s; ne0s=$(sed -n 's/.*transposed ne0=\([0-9]*\).*/\1/p' "$S/cc-pp-log.log" | sort -n | uniq -c | awk '{printf "%sx%s ", $1, $2}')
|
|
check concat/prefill-fires $(( n_pp > 0 && n_pp_small == 0 ? 0 : 1 )) \
|
|
"$n_pp fast-path dispatches in the prefill run (transposed ne0: ${ne0s:-none})"
|
|
check concat/decode-skips $(( n_tg == 0 && n_tg_concat > 0 ? 0 : 1 )) \
|
|
"$n_tg fast-path dispatches over $n_tg_concat executed CONCAT nodes in the decode run"
|
|
|
|
local c_on c_off c_cand on off cand
|
|
read -r c_on on < <(concat_us "$S/cc-pp-log.log")
|
|
read -r c_off off < <(concat_us "$S/cc-pp-off.log")
|
|
read -r c_cand cand < <(concat_us "$S/cc-pp-cand.log")
|
|
local ok
|
|
ok=$(awk -v on="$on" -v off="$off" -v c="$cand" 'BEGIN {print (c > 0 && on > 0 && off/c >= 2 && c/on < 1.5 && on/c < 1.5) ? 0 : 1}')
|
|
check concat/candidate-timing "$ok" \
|
|
"prefill CONCAT avg: candidate ${cand} us, knob-on ${on} us, knob-off ${off} us (off/cand $(awk -v a="$off" -v b="$cand" 'BEGIN {printf "%.1f", (b > 0) ? a / b : 0}')x)"
|
|
local t_on t_off t_cand
|
|
t_on=$(grep -E '\| +pp512 ' "$S/cc-pp-log.log" | awk -F'|' '{print $(NF-1)}' | awk '{print $1}' | head -1)
|
|
t_off=$(grep -E '\| +pp512 ' "$S/cc-pp-off.log" | awk -F'|' '{print $(NF-1)}' | awk '{print $1}' | tr '\n' ' ')
|
|
t_cand=$(grep -E '\| +pp512 ' "$S/cc-pp-cand.log" | awk -F'|' '{print $(NF-1)}' | awk '{print $1}' | tr '\n' ' ')
|
|
log "concat: pp512 t/s (perf logger on, so absolute speed is not representative): on $t_on | off $t_off | candidate $t_cand"
|
|
}
|
|
|
|
# ================================================================ 3. arbiter
|
|
ARB_ENV=(LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$ARENA" LLAMA_SLEEP_EVICT_KV=1 LLAMA_MMPROJ_ONDEMAND=1)
|
|
ARB_ARGS=(-m "$SMALL" -c 8192 -ngl 99 -fa 1 -lm none --parallel 1 --no-warmup --reasoning off)
|
|
A_PID= ; B_PID=
|
|
start_a() { start_server A "$BUILD" "$PA" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --mmproj "$MMPROJ"; A_PID=$SPID; }
|
|
start_b() { start_server B "$BUILD" "$PB" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --no-mmproj; B_PID=$SPID; }
|
|
sleeps() { grep -c 'server entering sleeping state' "$1"; }
|
|
|
|
make_red_ppm() { # 128x128 solid red, binary PPM (stb_image reads it; no image tools needed)
|
|
local f=$S/red.ppm i
|
|
printf '\377\000\000' > "$S/px"
|
|
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do cat "$S/px" "$S/px" > "$S/px2"; mv "$S/px2" "$S/px"; done
|
|
{ printf 'P6\n128 128\n255\n'; cat "$S/px"; } > "$f"
|
|
base64 -w0 "$f"
|
|
}
|
|
|
|
run_arbiter() {
|
|
[ -f "$SMALL" ] && [ -f "$MMPROJ" ] || { fail arbiter "model/mmproj missing: $SMALL $MMPROJ"; return; }
|
|
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
|
|
if [ $free_mb -lt 1536 ]; then
|
|
fail arbiter "only $free_mb MiB VRAM free (< 1536): run inside the swapdown window"; return
|
|
fi
|
|
# card-wide and production VRAM before any scratch server exists (see prod_vram_mb)
|
|
local v0 p0; v0=$(vram_mb); p0=$(prod_vram_mb)
|
|
log "arbiter: $free_mb MiB VRAM free (card $v0 MiB used, production DRM clients $p0 MiB); starting A (mmproj) on :$PA"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/start "A did not become healthy, see $S/A.log"; return; }
|
|
|
|
# no-handoff reference: A has held the token since its load and never released
|
|
local code ref_a
|
|
code=$(complete "$PA" "$S/ref-a.json" "$(greedy_body 48 false)")
|
|
ref_a=$(json_str content "$S/ref-a.json")
|
|
[ "$code" = 200 ] && [ -n "$ref_a" ] || { fail arbiter/start "reference request on A: HTTP $code"; return; }
|
|
[ "$(token_holder)" = "$A_PID" ] || { fail arbiter/start "A serves but does not hold token.lock"; return; }
|
|
|
|
# ---- (v) VRAM really drops when A goes cold, and comes back when it warms again
|
|
# A is alone on the scratch arena; a fake waiter on want.lock makes it release. Its footprint is
|
|
# the card-wide delta against the baseline taken before A started; production's DRM clients are
|
|
# re-read at every point, and if they moved the check is SKIPped (it cannot attribute the delta).
|
|
local v_warm p_warm v_cold p_cold v_back p_back f_warm f_cold w k=0
|
|
sleep 0.3
|
|
v_warm=$(vram_mb); p_warm=$(prod_vram_mb); f_warm=$(( $(drm_vram_kib "$A_PID") / 1024 ))
|
|
local added=$(( v_warm - v0 )) need
|
|
need=$(( added * 7 / 10 ))
|
|
( exec 9<"$ARENA/want.lock"; flock -s 9; exec sleep 120 ) & w=$!; track "$w"
|
|
if ! wait_holder "" 10000 > /dev/null; then
|
|
fail arbiter/v-vram-release "A did not drop the token for a waiter within 10 s (holder $(token_holder))"
|
|
else
|
|
# the release frees before it unlocks; give the driver a moment to return the pages
|
|
while [ $(( v_warm - $(vram_mb) )) -lt $need ] && [ $k -lt 30 ]; do sleep 0.1; k=$((k + 1)); done
|
|
fi
|
|
v_cold=$(vram_mb); p_cold=$(prod_vram_mb); f_cold=$(( $(drm_vram_kib "$A_PID") / 1024 ))
|
|
kill "$w" 2>/dev/null; wait "$w" 2>/dev/null
|
|
code=$(complete "$PA" "$S/v-a.json" "$(greedy_body 48 false)")
|
|
sleep 0.3
|
|
v_back=$(vram_mb); p_back=$(prod_vram_mb)
|
|
local pmove; pmove=$(printf '%s\n' "$p_warm" "$p_cold" "$p_back" | awk -v b="$p0" '{d = $1 - b; if (d < 0) d = -d; if (d > m) m = d} END {print m + 0}')
|
|
local vmsg="card-wide over baseline: warm +${added} MiB, cold +$((v_cold - v0)) MiB (dropped $((v_warm - v_cold)), need >= ${need} = 70%), restored +$((v_back - v0)) MiB; A's own DRM clients warm ${f_warm} / cold ${f_cold} MiB; production ${p0} -> ${p_warm}/${p_cold}/${p_back} MiB"
|
|
if [ "$pmove" -gt 64 ]; then
|
|
skip arbiter/v-vram-release "production VRAM moved by ${pmove} MiB during the measurement, cannot attribute: $vmsg"
|
|
elif [ "$added" -lt 64 ]; then
|
|
fail arbiter/v-vram-release "A added only ${added} MiB when warm (measurement broken?): $vmsg"
|
|
else
|
|
check arbiter/v-vram-release $(( v_warm - v_cold >= need ? 0 : 1 )) "$vmsg"
|
|
check arbiter/v-vram-restore $(( code == 200 && v_back - v0 >= need && $(holder_num) == A_PID ? 0 : 1 )) \
|
|
"HTTP $code, A holds the token: $([ "$(token_holder)" = "$A_PID" ] && echo yes || echo NO), restored +$((v_back - v0)) MiB (need >= ${need}), output $([ "$(json_str content "$S/v-a.json")" = "$ref_a" ] && echo identical || echo DIFFERENT)"
|
|
[ "$(json_str content "$S/v-a.json")" = "$ref_a" ] || fail arbiter/v-vram-output "output after the release/restore differs from the reference"
|
|
fi
|
|
|
|
log "arbiter: starting B (text-only) on :$PB"
|
|
start_b
|
|
wait_health "$PB" "$B_PID" 180 || { fail arbiter/start "B did not become healthy, see $S/B.log"; return; }
|
|
pass arbiter/start "A=$A_PID B=$B_PID up, reference output ${#ref_a} chars"
|
|
|
|
# ---- (a)+(b) alternating handoffs
|
|
local i port pid name bad=0 stranded=0 diff_a=0 diff_b=0 ref_b= lat=() holder_bad=0
|
|
for i in $(seq 0 30); do
|
|
if [ $((i % 2)) -eq 0 ]; then port=$PB pid=$B_PID name=B; else port=$PA pid=$A_PID name=A; fi
|
|
local t0 t1 out=$S/h$i.json
|
|
t0=$(now_ms)
|
|
code=$(complete "$port" "$out" "$(greedy_body 48 false)" 60)
|
|
t1=$(now_ms)
|
|
[ $i -gt 0 ] && lat+=($((t1 - t0)))
|
|
if [ "$code" = 000 ]; then stranded=$((stranded + 1)); continue; fi
|
|
[ "$code" != 200 ] && { bad=$((bad + 1)); continue; }
|
|
local c; c=$(json_str content "$out")
|
|
if [ $name = A ]; then [ "$c" = "$ref_a" ] || diff_a=$((diff_a + 1))
|
|
else if [ -z "$ref_b" ]; then ref_b=$c; elif [ "$c" != "$ref_b" ]; then diff_b=$((diff_b + 1)); fi; fi
|
|
[ "$(token_holder)" = "$pid" ] || holder_bad=$((holder_bad + 1))
|
|
done
|
|
local med max
|
|
med=$(printf '%s\n' "${lat[@]}" | sort -n | awk '{a[NR]=$1} END {print a[int((NR+1)/2)]}')
|
|
max=$(printf '%s\n' "${lat[@]}" | sort -n | tail -1)
|
|
check arbiter/a-handoffs $(( stranded == 0 && bad == 0 && holder_bad == 0 ? 0 : 1 )) \
|
|
"30 handoffs: $stranded stranded, $bad non-200, $holder_bad served-without-token; request latency median ${med} ms, max ${max} ms"
|
|
check arbiter/b-greedy-identical $(( diff_a == 0 && diff_b == 0 ? 0 : 1 )) \
|
|
"A: 15/15 == no-handoff reference ($diff_a differ); B: $diff_b of 15 differ from its first; A==B: $([ "$ref_a" = "$ref_b" ] && echo yes || echo no)"
|
|
|
|
# ---- (b') KV / prompt cache survives an evict + restore (LLAMA_SLEEP_EVICT_KV=1)
|
|
local p1 p2 c1 c2
|
|
code=$(complete "$PA" "$S/kv1.json" "$(greedy_body 32 true 'Explain in detail how a refrigerator moves heat from inside to outside')")
|
|
complete "$PB" "$S/kv-b.json" "$(greedy_body 8 false)" > /dev/null
|
|
[ "$(token_holder)" = "$B_PID" ] || log "warning: B does not hold the token before the KV re-check"
|
|
complete "$PA" "$S/kv2.json" "$(greedy_body 32 true 'Explain in detail how a refrigerator moves heat from inside to outside')" > /dev/null
|
|
p1=$(json_num prompt_n "$S/kv1.json"); p2=$(json_num prompt_n "$S/kv2.json")
|
|
c1=$(json_str content "$S/kv1.json"); c2=$(json_str content "$S/kv2.json")
|
|
check arbiter/b-kv-survives $(( ${p2:-99} < ${p1:-0} && ${#c1} > 0 ? 0 : 1 )) \
|
|
"prompt_n $p1 -> $p2 after evict/restore (cache reused), output $([ "$c1" = "$c2" ] && echo identical || echo DIFFERENT)"
|
|
[ "$c1" = "$c2" ] || fail arbiter/b-kv-output "cached-prompt output changed across evict/restore"
|
|
|
|
# ---- (c) image after release/restore
|
|
complete "$PB" "$S/c-b.json" "$(greedy_body 8 false)" > /dev/null # A goes cold
|
|
local cold_ok=1; [ "$(token_holder)" = "$B_PID" ] && cold_ok=0
|
|
local img; img=$(make_red_ppm)
|
|
printf '{"messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"data:image/x-portable-pixmap;base64,%s"}},{"type":"text","text":"What is the main color of this image? Answer with one word."}]}],"max_tokens":16,%s}' \
|
|
"$img" "$GREEDY" > "$S/c-body.json"
|
|
code=$(curl -sS -m 120 -o "$S/c-a.json" -w '%{http_code}' -H 'Content-Type: application/json' \
|
|
-d @"$S/c-body.json" "http://127.0.0.1:$PA/v1/chat/completions" 2>>"$S/curl.err")
|
|
local ans; ans=$(json_str content "$S/c-a.json")
|
|
check arbiter/c-image-after-restore $(( cold_ok == 0 && ${#code} > 0 && code == 200 ? 0 : 1 )) \
|
|
"A cold before: $([ $cold_ok -eq 0 ] && echo yes || echo NO), HTTP $code, answer: '$ans'"
|
|
echo "$ans" | grep -qi red || fail arbiter/c-image-answer "expected 'red', got '$ans'"
|
|
# text right after the image path (encoder out, weights back)
|
|
code=$(complete "$PA" "$S/c-a2.json" "$(greedy_body 48 false)")
|
|
[ "$code" = 200 ] && [ "$(json_str content "$S/c-a2.json")" = "$ref_a" ] \
|
|
|| fail arbiter/c-text-after-image "HTTP $code, output differs from reference after the image request"
|
|
|
|
# ---- (g) kill -9 a waiter that holds want.lock -> the holder stops yielding
|
|
stream_bg "$PA" "$S/g-a.sse" "$LONG_BODY"; local gcl=$CPID
|
|
if ! wait_tokens "$S/g-a.sse" 3 30000; then fail arbiter/g-kill9-waiter "A did not start streaming"; else
|
|
local s_before; s_before=$(sleeps "$S/A.log")
|
|
complete "$PB" "$S/g-b.json" "$(greedy_body 8 false)" 60 > /dev/null &
|
|
local gbc=$!; track "$gbc"
|
|
if ! wait_want "$B_PID" 10000; then
|
|
fail arbiter/g-kill9-waiter "B never registered on want.lock (setup)"
|
|
else
|
|
kill -KILL "$B_PID"; wait "$B_PID" 2>/dev/null
|
|
kill "$gcl" 2>/dev/null; wait "$gcl" 2>/dev/null # end A's generation (client abort)
|
|
sleep 2
|
|
local wl; wl=$(lock_pids "$ARENA/want.lock" READ | wc -l)
|
|
check arbiter/g-kill9-waiter $(( $(holder_num) == A_PID && $(sleeps "$S/A.log") == s_before && wl == 0 ? 0 : 1 )) \
|
|
"after kill -9 of waiting B: want.lock readers $wl, token holder $(token_holder) (A=$A_PID), A sleeps +$(( $(sleeps "$S/A.log") - s_before ))"
|
|
fi
|
|
wait "$gbc" 2>/dev/null
|
|
fi
|
|
log "arbiter: restarting B"
|
|
start_b
|
|
wait_health "$PB" "$B_PID" 180 || { fail arbiter/restart "B did not come back"; return; }
|
|
|
|
# ---- (e) SIGTERM while A waits for the token -> exits fast, never restores
|
|
stream_bg "$PB" "$S/e-b.sse" "$LONG_BODY"; local ecl=$CPID
|
|
wait_tokens "$S/e-b.sse" 3 30000 || { fail arbiter/e-term-waiting "B did not start streaming"; }
|
|
complete "$PA" "$S/e-a.json" "$(greedy_body 8 false)" 30 > /dev/null &
|
|
local eac=$!; track "$eac"
|
|
if wait_want "$A_PID" 10000; then
|
|
sleep 0.3
|
|
local t0 ms took_token=0
|
|
t0=$(now_ms); kill -TERM "$A_PID"
|
|
while pid_alive "$A_PID"; do
|
|
[ "$(token_holder)" = "$A_PID" ] && took_token=1
|
|
[ $(( $(now_ms) - t0 )) -gt 10000 ] && break
|
|
sleep 0.01
|
|
done
|
|
ms=$(( $(now_ms) - t0 )); wait "$A_PID"; RC=$?
|
|
check arbiter/e-term-waiting $(( RC == 0 && ms <= 1500 && took_token == 0 && $(holder_num) == B_PID ? 0 : 1 )) \
|
|
"exit $RC in ${ms} ms, A took the token: $([ $took_token = 1 ] && echo YES || echo no), holder now $(token_holder) (B=$B_PID), log: $(grep -c 'no longer waiting for the GPU token' "$S/A.log") abort line(s)"
|
|
else
|
|
fail arbiter/e-term-waiting "A never registered on want.lock (setup)"
|
|
stop_server "$A_PID"
|
|
fi
|
|
kill "$ecl" 2>/dev/null; wait "$ecl" "$eac" 2>/dev/null
|
|
log "arbiter: restarting A"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
|
|
|
|
# ---- (d) SIGTERM during A's VRAM release -> exit 0 within ~2 s, B gets the token
|
|
# A is warm (fresh load, first release also captures the host shadow: the longest release)
|
|
local how sig_ms
|
|
if command -v gdb > /dev/null; then
|
|
how=gdb
|
|
gdb -p "$A_PID" -batch -nx -ex 'set pagination off' -ex 'handle all nostop noprint pass' \
|
|
-ex 'break llama_context_release_device' -ex 'continue' \
|
|
-ex "shell date +%s%3N > $S/d.sig" -ex 'delete' -ex 'signal SIGTERM' > "$S/d-gdb.log" 2>&1 &
|
|
local gpid=$!; track "$gpid"
|
|
local k=0; until grep -q '^Continuing' "$S/d-gdb.log" || [ $k -gt 300 ]; do sleep 0.1; k=$((k + 1)); done
|
|
sleep 0.5
|
|
else
|
|
how=log-poll
|
|
fi
|
|
local s0; s0=$(sleeps "$S/A.log")
|
|
complete "$PB" "$S/d-b.json" "$(greedy_body 8 false)" 60 > "$S/d-b.code" &
|
|
local dbc=$!; track "$dbc"
|
|
if [ $how = log-poll ]; then
|
|
local k=0; until [ "$(sleeps "$S/A.log")" -gt "$s0" ] || [ $k -gt 2000 ]; do sleep 0.005; k=$((k + 1)); done
|
|
now_ms > "$S/d.sig"; kill -TERM "$A_PID"
|
|
else
|
|
local k=0; until [ -s "$S/d.sig" ] || [ $k -gt 600 ]; do sleep 0.05; k=$((k + 1)); done
|
|
fi
|
|
if [ ! -s "$S/d.sig" ]; then
|
|
fail arbiter/d-term-in-release "release breakpoint never hit ($how), see $S/d-gdb.log"
|
|
kill "$gpid" 2>/dev/null; stop_server "$A_PID"
|
|
else
|
|
while pid_alive "$A_PID" && [ $(( $(now_ms) - $(cat "$S/d.sig") )) -lt 20000 ]; do sleep 0.01; done
|
|
sig_ms=$(( $(now_ms) - $(cat "$S/d.sig") ))
|
|
wait "$A_PID"; RC=$?
|
|
wait "$dbc"
|
|
local dcode; dcode=$(cat "$S/d-b.code")
|
|
sleep 0.2
|
|
check arbiter/d-term-in-release $(( RC == 0 && sig_ms <= 2500 && dcode == 200 && $(holder_num) == B_PID ? 0 : 1 )) \
|
|
"SIGTERM inside llama_context_release_device ($how): A exit $RC in ${sig_ms} ms, B HTTP $dcode, holder $(token_holder) (B=$B_PID)"
|
|
fi
|
|
log "arbiter: restarting A"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
|
|
|
|
# ---- (f) SIGTERM mid-generation (streaming)
|
|
stream_bg "$PA" "$S/f-a.sse" "$LONG_BODY"; local fcl=$CPID
|
|
if ! wait_tokens "$S/f-a.sse" 10 30000; then fail arbiter/f-term-streaming "A did not start streaming"; else
|
|
local t0 free_ms=-1 cl_ms=-1 exit_ms=-1
|
|
t0=$(now_ms); kill -TERM "$A_PID"
|
|
while [ $(( $(now_ms) - t0 )) -lt 10000 ]; do
|
|
[ $free_ms -lt 0 ] && [ -z "$(token_holder)" ] && free_ms=$(( $(now_ms) - t0 ))
|
|
[ $cl_ms -lt 0 ] && ! pid_alive "$fcl" && cl_ms=$(( $(now_ms) - t0 ))
|
|
[ $exit_ms -lt 0 ] && ! pid_alive "$A_PID" && exit_ms=$(( $(now_ms) - t0 ))
|
|
[ $free_ms -ge 0 ] && [ $cl_ms -ge 0 ] && [ $exit_ms -ge 0 ] && break
|
|
sleep 0.01
|
|
done
|
|
wait "$A_PID"; RC=$?; wait "$fcl" 2>/dev/null
|
|
local errev; errev=$(grep -c '"error"' "$S/f-a.sse")
|
|
check arbiter/f-term-streaming $(( RC == 0 && errev > 0 && cl_ms >= 0 && free_ms >= 0 && free_ms <= 2500 && exit_ms >= 0 ? 0 : 1 )) \
|
|
"A exit $RC in ${exit_ms} ms, token free in ${free_ms} ms, client closed in ${cl_ms} ms with $errev error event(s) after $(grep -c '^data: ' "$S/f-a.sse") data lines"
|
|
fi
|
|
log "arbiter: restarting A"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
|
|
|
|
# ---- (h) clean shutdown warm and cold
|
|
[ "$(token_holder)" = "$A_PID" ] || log "warning: A is not warm before the warm-shutdown case"
|
|
stop_server "$A_PID"
|
|
check arbiter/h-shutdown-warm $(( RC == 0 ? 0 : 1 )) "warm A: exit $RC in ${EXIT_MS} ms"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
|
|
complete "$PB" "$S/h-b.json" "$(greedy_body 8 false)" > /dev/null # A goes cold
|
|
local acold=no; [ "$(token_holder)" = "$B_PID" ] && acold=yes
|
|
stop_server "$A_PID"
|
|
check arbiter/h-shutdown-cold $([ $RC -eq 0 ] && [ $acold = yes ] && echo 0 || echo 1) \
|
|
"cold A (B holds token: $acold): exit $RC in ${EXIT_MS} ms"
|
|
stop_server "$B_PID"
|
|
check arbiter/h-shutdown-warm-b $(( RC == 0 ? 0 : 1 )) "warm B: exit $RC in ${EXIT_MS} ms"
|
|
|
|
local bad_lines; bad_lines=$(scan_logs "$S/A.log" "$S/B.log")
|
|
check arbiter/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) \
|
|
"$([ -z "$bad_lines" ] && echo 'no asserts / Vulkan errors in A/B logs' || echo "$bad_lines" | head -3 | tr '\n' ' ')"
|
|
|
|
run_handoff_ab
|
|
}
|
|
|
|
# handoff_block BUILD TAG N : a fresh pair of BUILD on the scratch arena (A with mmproj, B text-only,
|
|
# the same config as case a), one warm-up request on A, then N+1 alternating requests starting with
|
|
# B. Request 0 is not a handoff (B holds the token after its load); the latency of requests 1..N is
|
|
# appended to $S/lat-TAG.txt. HB_BAD counts setup failures, non-200s and answers without the token.
|
|
handoff_block() {
|
|
local build=$1 tag=$2 n=$3 a b i port pid code t0 t1
|
|
HB_BAD=0
|
|
: > "$S/lat-$tag.txt"
|
|
start_server "L$tag-A" "$build" "$PA" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --mmproj "$MMPROJ"; a=$SPID
|
|
if ! wait_health "$PA" "$a" 180; then HB_BAD=1; stop_server "$a"; return 1; fi
|
|
complete "$PA" "$S/L$tag-warm.json" "$(greedy_body 8 false)" > /dev/null
|
|
start_server "L$tag-B" "$build" "$PB" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --no-mmproj; b=$SPID
|
|
if ! wait_health "$PB" "$b" 180; then HB_BAD=1; stop_server "$b"; stop_server "$a"; return 1; fi
|
|
for i in $(seq 0 "$n"); do
|
|
if [ $((i % 2)) -eq 0 ]; then port=$PB pid=$b; else port=$PA pid=$a; fi
|
|
t0=$(now_ms)
|
|
code=$(complete "$port" "$S/L$tag-$i.json" "$(greedy_body 48 false)" 60)
|
|
t1=$(now_ms)
|
|
if [ "$code" != 200 ] || [ "$(token_holder)" != "$pid" ]; then HB_BAD=$((HB_BAD + 1)); continue; fi
|
|
[ "$i" -gt 0 ] && echo $((t1 - t0)) >> "$S/lat-$tag.txt"
|
|
done
|
|
stop_server "$b"; stop_server "$a"
|
|
}
|
|
|
|
# ---- (a') ordinary handoff latency, candidate vs reference, in one session
|
|
# Two production deploys were lost to arbiter changes that passed the pathological cases but made
|
|
# every ordinary handoff slower. Blocks run ref, cand, cand, ref (a fixed order once gave the later
|
|
# slot about 9%); each build's median is over all its handoffs. FAIL if the candidate median exceeds
|
|
# the reference median by more than max(25%, 200 ms): small absolute numbers are noisy.
|
|
run_handoff_ab() {
|
|
local n=16 blk k=0 bad_r=0 bad_c=0 tag
|
|
log "arbiter: handoff latency A/B, blocks ref,cand,cand,ref x $n handoffs"
|
|
: > "$S/lat-ref.txt"; : > "$S/lat-cand.txt"
|
|
for blk in ref cand cand ref; do
|
|
k=$((k + 1)); tag=$k-$blk
|
|
local b; [ $blk = ref ] && b=$REF || b=$BUILD
|
|
handoff_block "$b" "$tag" "$n"
|
|
if [ $blk = ref ]; then bad_r=$((bad_r + HB_BAD)); else bad_c=$((bad_c + HB_BAD)); fi
|
|
cat "$S/lat-$tag.txt" >> "$S/lat-$blk.txt"
|
|
log "arbiter: block $tag median $(median < "$S/lat-$tag.txt") ms over $(wc -l < "$S/lat-$tag.txt") handoffs, $HB_BAD bad"
|
|
done
|
|
local mr mc nr nc maxr maxc limit
|
|
mr=$(median < "$S/lat-ref.txt"); mc=$(median < "$S/lat-cand.txt")
|
|
nr=$(wc -l < "$S/lat-ref.txt"); nc=$(wc -l < "$S/lat-cand.txt")
|
|
maxr=$(sort -n "$S/lat-ref.txt" | tail -1); maxc=$(sort -n "$S/lat-cand.txt" | tail -1)
|
|
if [ -z "$mr" ] || [ -z "$mc" ] || [ $bad_r -gt 0 ] || [ $bad_c -gt 0 ]; then
|
|
fail arbiter/a-handoff-latency "blocks incomplete: reference $nr handoffs ($bad_r bad), candidate $nc ($bad_c bad); see $S/L*-*.log"
|
|
return
|
|
fi
|
|
limit=$(awk -v r="$mr" 'BEGIN {d = r * 0.25; if (d < 200) d = 200; printf "%.0f", r + d}')
|
|
check arbiter/a-handoff-latency "$(awk -v c="$mc" -v l="$limit" 'BEGIN {print (c > l) ? 1 : 0}')" \
|
|
"median handoff latency candidate ${mc} ms (n $nc, max $maxc) vs reference ${mr} ms (n $nr, max $maxr); limit ${limit} ms = ref + max(25%, 200 ms)"
|
|
local bad_lines; bad_lines=$(scan_logs "$S"/L*-A.log "$S"/L*-B.log)
|
|
[ -z "$bad_lines" ] || fail arbiter/a-handoff-latency-logs "$(echo "$bad_lines" | head -3 | tr '\n' ' ')"
|
|
}
|
|
|
|
# ================================================================ 4. perf
|
|
vram_sampler() { # OUTFILE : max of mem_info_vram_used every 0.25 s, written continuously
|
|
local m=0 v
|
|
while :; do v=$(cat "$VRAM_USED"); [ "$v" -gt "$m" ] && { m=$v; echo "$m" > "$1"; }; sleep 0.25; done
|
|
}
|
|
# median of the numbers on stdin (one per line); empty input -> empty output
|
|
median() { sort -g | awk '{a[NR] = $1} END {if (NR) printf "%.2f", (NR % 2) ? a[(NR + 1) / 2] : (a[NR / 2] + a[NR / 2 + 1]) / 2}'; }
|
|
bench_samples() { # JSONL (llama-bench -o jsonl) -> "test<TAB>t/s", one row per repetition sample
|
|
sed -n 's/.*"n_prompt": *\([0-9]*\),.*"n_gen": *\([0-9]*\),.*"n_depth": *\([0-9]*\),.*"samples_ts": *\[\([^]]*\)\].*/\1 \2 \3 \4/p' "$1" |
|
|
awk '{t = ($1 > 0) ? "pp" $1 : "tg" $2; if ($3 > 0) t = t " @ d" $3
|
|
for (i = 4; i <= NF; i++) {v = $i; gsub(/,/, "", v); if (v != "") print t "\t" v}}'
|
|
}
|
|
run_perf() {
|
|
gpu_exclusive perf || return
|
|
[ -f "$PERF_MODEL" ] || { fail perf "model missing: $PERF_MODEL"; return; }
|
|
[ -x "$REF/bin/llama-bench" ] && [ -x "$BUILD/bin/llama-bench" ] || { fail perf "llama-bench missing"; return; }
|
|
local common=(-m "$PERF_MODEL" -t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -o jsonl -oe md)
|
|
local arm k=0 base; base=$(vram_mb)
|
|
log "perf: idle VRAM $base MiB; arms ref,cand,cand,ref"
|
|
for arm in ref cand cand ref; do
|
|
k=$((k + 1))
|
|
local b; [ $arm = ref ] && b=$REF || b=$BUILD
|
|
local tag=$k-$arm
|
|
vram_sampler "$S/perf-$tag.vram" & local vs=$!; track "$vs"
|
|
# stdout: one JSON line per test with every repetition sample; stderr: the markdown table + logs
|
|
log "perf: arm $tag pp sweep"
|
|
LD_LIBRARY_PATH="$b/bin" "$b/bin/llama-bench" "${common[@]}" -p 2048 -n 0 -d 0,16384,32768 -r 2 \
|
|
> "$S/perf-$tag-pp.jsonl" 2> "$S/perf-$tag-pp.log"
|
|
log "perf: arm $tag tg"
|
|
LD_LIBRARY_PATH="$b/bin" "$b/bin/llama-bench" "${common[@]}" -p 0 -n 256 -d 4096 -r 3 \
|
|
> "$S/perf-$tag-tg.jsonl" 2> "$S/perf-$tag-tg.log"
|
|
kill "$vs" 2>/dev/null; wait "$vs" 2>/dev/null
|
|
{ bench_samples "$S/perf-$tag-pp.jsonl"; bench_samples "$S/perf-$tag-tg.jsonl"; } | sed "s/^/$arm\t/" >> "$S/perf.tsv"
|
|
echo "$arm $(cat "$S/perf-$tag.vram" 2>/dev/null || echo 0)" >> "$S/perf-vram.txt"
|
|
done
|
|
perf_report "$S/perf.tsv"
|
|
local total; total=$(cat "$VRAM_TOTAL")
|
|
local vr vc
|
|
vr=$(awk '$1=="ref" && $2>m {m=$2} END {print m+0}' "$S/perf-vram.txt")
|
|
vc=$(awk '$1=="cand" && $2>m {m=$2} END {print m+0}' "$S/perf-vram.txt")
|
|
check perf/peak-vram $(( vc * 100 <= total * 95 ? 0 : 1 )) \
|
|
"peak VRAM reference $((vr / 1048576)) MiB, candidate $((vc / 1048576)) MiB of $((total / 1048576)) MiB (limit 95%)"
|
|
}
|
|
# perf_report TSV : "arm<TAB>test<TAB>t/s" rows, one per llama-bench repetition sample (arm = ref|cand).
|
|
# Gates on the MEDIAN of all samples of a build (pooled over its arms): a single catastrophic
|
|
# sample (CPU contention from the other pods on the node) moves a mean, not a median.
|
|
perf_report() {
|
|
local tsv=$1
|
|
echo
|
|
echo "perf: median t/s over every repetition sample of both arms (order ref,cand,cand,ref)"
|
|
printf '%-18s %24s %24s %8s\n' test "reference med (min-max, n)" "candidate med (min-max, n)" delta
|
|
local tests t r c d rs cs
|
|
tests=$(awk -F'\t' '{print $2}' "$tsv" | awk '!s[$0]++')
|
|
local IFS_=$IFS; IFS=$'\n'
|
|
for t in $tests; do
|
|
r=$(awk -F'\t' -v t="$t" '$1=="ref" && $2==t {print $3}' "$tsv" | median)
|
|
c=$(awk -F'\t' -v t="$t" '$1=="cand" && $2==t {print $3}' "$tsv" | median)
|
|
rs=$(awk -F'\t' -v t="$t" '$1=="ref" && $2==t {n++; if (n == 1 || $3 < lo) lo = $3; if (n == 1 || $3 > hi) hi = $3} END {if (n) printf "%.1f-%.1f, %d", lo, hi, n}' "$tsv")
|
|
cs=$(awk -F'\t' -v t="$t" '$1=="cand" && $2==t {n++; if (n == 1 || $3 < lo) lo = $3; if (n == 1 || $3 > hi) hi = $3} END {if (n) printf "%.1f-%.1f, %d", lo, hi, n}' "$tsv")
|
|
d=$(awk -v r="$r" -v c="$c" 'BEGIN {if (r > 0) printf "%+.1f%%", (c/r - 1)*100}')
|
|
printf '%-18s %24s %24s %8s\n' "$t" "$r ($rs)" "$c ($cs)" "$d"
|
|
case "$t" in pp*)
|
|
if [ -z "$r" ] || [ -z "$c" ] || [ "$(awk -v r="$r" -v c="$c" 'BEGIN {print (c < 0.95*r)}')" = 1 ]; then
|
|
fail "perf/$t" "candidate median $c vs reference median $r t/s ($d, limit -5%)"
|
|
else
|
|
pass "perf/$t" "candidate median $c vs reference median $r t/s ($d)"
|
|
fi ;;
|
|
tg*) RESULTS+=("INFO perf/$t: candidate median $c vs reference median $r t/s ($d; decode is CPU-bound and noisy here, not gated)") ;;
|
|
esac
|
|
done
|
|
IFS=$IFS_
|
|
local n_tests; n_tests=$(echo "$tests" | grep -c '^pp')
|
|
[ "$n_tests" -ge 3 ] || fail perf/rows "expected 3 pp rows, got $n_tests (see $S/perf-*-pp.log)"
|
|
}
|
|
|
|
# ================================================================ 5. MoE cache release/restore smoke
|
|
run_moecache() {
|
|
[ -f "$MOECACHE_MODEL" ] || { fail moecache "model missing: $MOECACHE_MODEL"; return; }
|
|
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
|
|
log "moecache: $free_mb MiB VRAM free; server on :$PC with --moe-cache-mib $MOECACHE_MIB"
|
|
local arena=$S/arena-moec; mkdir -p "$arena"
|
|
# -lv 4: the cache size and hit/miss stats are only logged at that verbosity
|
|
# shellcheck disable=SC2086
|
|
start_server M "$BUILD" "$PC" LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$arena" LLAMA_SLEEP_EVICT_KV=1 -- \
|
|
-m "$MOECACHE_MODEL" $MOECACHE_ARGS -fa 1 -lm none --parallel 1 --no-warmup --reasoning off -lv 4 \
|
|
--moe-cache-mib "$MOECACHE_MIB"
|
|
local mp=$SPID
|
|
wait_health "$PC" "$mp" 600 || { fail moecache "server did not start, see $S/M.log"; return; }
|
|
# chat endpoint: an instruct model fed a raw prompt can loop, which says nothing about the cache
|
|
local body="{\"messages\":[{\"role\":\"user\",\"content\":\"What are the three primary colors of light? Answer in one short sentence.\"}],\"max_tokens\":64,$GREEDY}"
|
|
local outs=() i code w
|
|
for i in 0 1 2; do
|
|
code=$(curl -sS -m 300 -o "$S/m$i.json" -w '%{http_code}' -H 'Content-Type: application/json' \
|
|
-d "$body" "http://127.0.0.1:$PC/v1/chat/completions" 2>>"$S/curl.err")
|
|
outs+=("$(json_str content "$S/m$i.json")")
|
|
[ "$code" = 200 ] || { fail moecache "request $i: HTTP $code"; stop_server "$mp"; return; }
|
|
[ $i -eq 2 ] && break
|
|
# force a release: a fake waiter holds want.lock shared until the server drops the token
|
|
( exec 9<"$arena/want.lock"; flock -s 9; exec sleep 120 ) & w=$!; track "$w"
|
|
local k=0; until [ -z "$(awk -v id="$(lock_id "$arena/token.lock")" '$2=="FLOCK" && $4=="WRITE" && $6==id' /proc/locks)" ] || [ $k -gt 600 ]; do sleep 0.05; k=$((k + 1)); done
|
|
log "moecache: cycle $((i + 1)) cold, VRAM $(vram_mb) MiB"
|
|
kill "$w"; wait "$w" 2>/dev/null
|
|
done
|
|
stop_server "$mp"
|
|
local size hits rel res
|
|
size=$(sed -n 's/.*MoE cache size = *\([0-9.]*\) MiB.*/\1/p' "$S/M.log" | head -1)
|
|
hits=$(sed -n 's/.*llama_moe_cache: .* hits = \([0-9]*\),.*/\1/p' "$S/M.log" | awk '{s += $1} END {print s+0}')
|
|
rel=$(grep -c 'released .* MiB of MoE cache' "$S/M.log"); res=$(grep -c 'restored .* MiB of MoE cache' "$S/M.log")
|
|
check moecache/active $(( hits > 0 ? 0 : 1 )) \
|
|
"cache ${size:-?} MiB, $hits hits (0 hits = cache too small to ever reuse an expert, so a stale SLRU would go unnoticed), released ${rel}x restored ${res}x"
|
|
local same=1; [ "${outs[0]}" = "${outs[1]}" ] && [ "${outs[1]}" = "${outs[2]}" ] && [ -n "${outs[0]}" ] && same=0
|
|
check moecache/identical $same "3 greedy answers across 2 cold/warm cycles $([ $same = 0 ] && echo identical || echo DIFFER): '$(echo "${outs[0]}" | cut -c1-80)'"
|
|
local coh=1; echo "${outs[2]}" | grep -qi red && echo "${outs[2]}" | grep -qi green && echo "${outs[2]}" | grep -qi blue && coh=0
|
|
check moecache/coherent $coh "answer after the 2nd restore names red, green and blue"
|
|
check moecache/shutdown $(( RC == 0 ? 0 : 1 )) "exit $RC"
|
|
local bad_lines; bad_lines=$(scan_logs "$S/M.log")
|
|
check moecache/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) "${bad_lines:-clean}"
|
|
}
|
|
|
|
# ================================================================ 6. MoE routing-ids readback skip (F1) exactness
|
|
# sched_copy_experts skips the ids readback when a split's ids tensor has n_tokens * n_expert_used >=
|
|
# 4 * n_expert entries, i.e. for any ubatch of at least ceil(4 * n_expert / n_expert_used) tokens
|
|
# (128 on a 256-expert / 8-used model). Method: the model with every MoE layer's experts on the host
|
|
# (so the expert-copy callback runs), greedy, one prompt of >= 2x that many tokens in a single
|
|
# ubatch: the prefill takes the skip and every decode step takes the readback. The candidate's
|
|
# output must be byte-identical to the reference's and, when the build has the LLAMA_MOE_IDS_SKIP
|
|
# knob, to the candidate with the skip forced off (=0), which proves the skip itself is exact even
|
|
# when the reference build carries it too; =log then shows the skip engaged.
|
|
gguf_u32() { # FILE KEY-SUFFIX -> uint32 value of the first GGUF metadata key ending in KEY-SUFFIX
|
|
local off t
|
|
off=$(head -c 67108864 "$1" | grep -abo -m1 -F "$2" | head -1 | cut -d: -f1)
|
|
[ -n "$off" ] || return 1
|
|
off=$((off + ${#2})) # key string -> uint32 value type -> value
|
|
t=$(od -An -tu4 -j "$off" -N4 "$1" | tr -d ' ')
|
|
[ "$t" = 4 ] || [ "$t" = 5 ] || return 1 # GGUF_TYPE_UINT32 / INT32
|
|
od -An -tu4 -j $((off + 4)) -N4 "$1" | tr -d ' '
|
|
}
|
|
gpu_exclusive() { # CHECK-NAME -> FAILs it and returns 1 unless nothing else can use the GPU
|
|
[ "${FV_TEST_SHARED_GPU:-}" = 1 ] && { log "FV_TEST_SHARED_GPU=1: skipping the exclusive-GPU guard (script testing only)"; return 0; }
|
|
if supervisorctl -c /root/supervisord.conf status llama-swap 2>/dev/null | grep -q RUNNING; then
|
|
fail "$1" "llama-swap is RUNNING; run under /root/bench/moe/swapdown.sh"; return 1
|
|
fi
|
|
local others; others=$(ps -eo pid=,comm= | awk '$2 ~ /^llama-(server|bench|cli)$/' | wc -l)
|
|
[ "$others" -eq 0 ] || { fail "$1" "$others other llama processes are running"; return 1; }
|
|
}
|
|
moex_prompt() { # N_LINES -> a deterministic prose-like prompt (JSON-safe: no quotes or backslashes)
|
|
local i
|
|
for i in $(seq 1 "$1"); do
|
|
printf 'Entry %d: the river town of Station%d ships %d barrels of grain and %d crates of salt each week to the harbour, and its council meets on day %d of the month. ' \
|
|
"$i" "$i" $(( (i * 37) % 101 + 3 )) $(( (i * 53) % 89 + 2 )) $(( (i * 7) % 28 + 1 ))
|
|
done
|
|
printf 'Question: which town ships the most barrels of grain each week? Answer:'
|
|
}
|
|
run_moe_exact() {
|
|
gpu_exclusive moe-exact || return
|
|
[ -f "$MOEX_MODEL" ] || { fail moe-exact "model missing: $MOEX_MODEL"; return; }
|
|
local n_exp n_used
|
|
n_exp=$(gguf_u32 "$MOEX_MODEL" .expert_count); n_used=$(gguf_u32 "$MOEX_MODEL" .expert_used_count)
|
|
if [ -z "$n_exp" ] || [ -z "$n_used" ] || [ "$n_exp" -eq 0 ] || [ "$n_used" -eq 0 ]; then
|
|
fail moe-exact "no expert_count / expert_used_count in $MOEX_MODEL (not a MoE model?)"; return
|
|
fi
|
|
local thr=$(( (4 * n_exp + n_used - 1) / n_used )) # tokens per ubatch where the skip starts
|
|
local lines=$(( (2 * thr + 39) / 40 + 2 )) # ~45 tokens per line -> >= 2x the threshold
|
|
local body
|
|
body="{\"prompt\":\"$(moex_prompt "$lines")\",\"n_predict\":64,$GREEDY,\"cache_prompt\":false}"
|
|
local knob=0; grep -qa LLAMA_MOE_IDS_SKIP "$BUILD/bin/libllama.so" 2>/dev/null && knob=1
|
|
log "moe-exact: n_expert $n_exp, n_expert_used $n_used -> skip at >= $thr tokens per ubatch; prompt of $lines lines; knob in candidate: $([ $knob = 1 ] && echo yes || echo no)"
|
|
|
|
# arm NAME BUILD ENV... : one server, one request; sets MX_CONTENT / MX_PN, returns 1 on failure
|
|
mx_arm() {
|
|
local name=$1 b=$2; shift 2
|
|
# shellcheck disable=SC2086
|
|
start_server "$name" "$b" "$PC" "$@" -- -m "$MOEX_MODEL" $MOEX_ARGS -fa 1 -lm none --parallel 1 --no-warmup
|
|
local p=$SPID code
|
|
if ! wait_health "$PC" "$p" 900; then fail "moe-exact/$name" "server did not start, see $S/$name.log"; stop_server "$p"; return 1; fi
|
|
code=$(complete "$PC" "$S/$name.json" "$body" 600)
|
|
stop_server "$p"
|
|
MX_CONTENT=$(json_str content "$S/$name.json"); MX_PN=$(json_num prompt_n "$S/$name.json")
|
|
[ "$code" = 200 ] && [ -n "$MX_CONTENT" ] || { fail "moe-exact/$name" "HTTP $code, empty output; see $S/$name.log"; return 1; }
|
|
[ "$RC" -eq 0 ] || log "warning: $name exited $RC"
|
|
}
|
|
local c_cand c_ref c_off pn
|
|
if [ $knob = 1 ]; then mx_arm X-cand "$BUILD" LLAMA_MOE_IDS_SKIP=log || return
|
|
else mx_arm X-cand "$BUILD" || return; fi
|
|
c_cand=$MX_CONTENT; pn=${MX_PN:-0}
|
|
if [ "$pn" -lt "$thr" ]; then
|
|
fail moe-exact/prompt "prompt_n $pn < threshold $thr: the skip never ran, nothing was tested"; return
|
|
fi
|
|
mx_arm X-ref "$REF" || return
|
|
c_ref=$MX_CONTENT
|
|
check moe-exact/identical-ref $([ "$c_cand" = "$c_ref" ] && echo 0 || echo 1) \
|
|
"prompt_n $pn (>= $thr), 64 greedy tokens: candidate $([ "$c_cand" = "$c_ref" ] && echo == || echo '!=') reference ('$(echo "$c_cand" | cut -c1-60)')"
|
|
if [ $knob = 1 ]; then
|
|
mx_arm X-cand-noskip "$BUILD" LLAMA_MOE_IDS_SKIP=0 || return
|
|
c_off=$MX_CONTENT
|
|
check moe-exact/identical-noskip $([ "$c_cand" = "$c_off" ] && echo 0 || echo 1) \
|
|
"candidate with the skip (LLAMA_MOE_IDS_SKIP=log) $([ "$c_cand" = "$c_off" ] && echo == || echo '!=') candidate with the readback forced (=0)"
|
|
local n_skip n_small
|
|
n_skip=$(grep -c 'sched_copy_experts: ids readback skipped' "$S/X-cand.log")
|
|
n_small=$(sed -n 's/.*ids readback skipped, n_ids=\([0-9]*\) n_expert=\([0-9]*\).*/\1 \2/p' "$S/X-cand.log" | awk '$1 < 4 * $2' | wc -l)
|
|
check moe-exact/skip-engaged $(( n_skip > 0 && n_small == 0 ? 0 : 1 )) \
|
|
"$n_skip skips logged ($n_small below 4*n_expert), $(grep -c 'ids readback skipped' "$S/X-cand-noskip.log") with the skip forced off"
|
|
else
|
|
skip moe-exact/skip-engaged "candidate has no LLAMA_MOE_IDS_SKIP knob to observe it; by construction the prefill ubatch had $pn x $n_used = $((pn * n_used)) ids >= 4 x $n_exp, so the skip branch ran"
|
|
fi
|
|
local bad_lines; bad_lines=$(scan_logs "$S"/X-*.log)
|
|
check moe-exact/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) "${bad_lines:-clean}"
|
|
}
|
|
|
|
# ================================================================ 7. disk prompt cache
|
|
# --cache-disk-path: on shutdown each slot's state goes to a file; after a restart a prompt that starts
|
|
# with a saved prompt (or with one of its checkpoints, for a hybrid model) is restored from the file.
|
|
# Method: chat requests on the small hybrid model, greedy. A reference server that never restarts
|
|
# gives turn 2 (turn 1 + answer + a new question); the candidate saves turn 1 on SIGTERM, restarts
|
|
# and must give the same answer with the same prompt_n (only the suffix processed). CPU only
|
|
# (-dev none -ngl 0, no VRAM), plus the GPU path (-ngl 99, arbiter env, warm and cold) if VRAM allows.
|
|
DC_ARGS=(-m "$SMALL" -c 4096 -dev none -ngl 0 -fa 1 -t 4 --parallel 1 --no-warmup --reasoning off --no-mmproj)
|
|
DC_GPU=(-dev Vulkan0 -ngl 99)
|
|
DC_ENV=(LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$ARENA" LLAMA_SLEEP_EVICT_KV=1)
|
|
DC_PID=
|
|
dc_chat() { # PORT OUT MESSAGES-JSON -> HTTP code
|
|
curl -sS -m 300 -o "$2" -w '%{http_code}' -H 'Content-Type: application/json' \
|
|
-d "{\"messages\":$3,\"max_tokens\":32,$GREEDY}" "http://127.0.0.1:$1/v1/chat/completions" 2>>"$S/curl.err" || true
|
|
}
|
|
dc_msgs() { # USER1 [ANSWER1 USER2] -> messages JSON
|
|
if [ $# -eq 1 ]; then echo "[{\"role\":\"user\",\"content\":\"$1\"}]"
|
|
else echo "[{\"role\":\"user\",\"content\":\"$1\"},{\"role\":\"assistant\",\"content\":\"$2\"},{\"role\":\"user\",\"content\":\"$3\"}]"; fi
|
|
}
|
|
# dc_start NAME DIR ENV... -- ARGS... : DIR "" = no disk cache; sets DC_PID, returns 1 if not healthy
|
|
dc_start() {
|
|
local name=$1 dir=$2; shift 2
|
|
local envs=(); while [ "$1" != -- ]; do envs+=("$1"); shift; done; shift
|
|
start_server "$name" "$BUILD" "$PC" "${envs[@]}" -- "${DC_ARGS[@]}" ${dir:+--cache-disk-path "$dir"} "$@"
|
|
DC_PID=$SPID
|
|
wait_health "$PC" "$DC_PID" 180
|
|
}
|
|
# dc_term : SIGTERM DC_PID; sets RC, DC_EXIT_MS and DC_FREE_MS (token.lock free, -1 if never held)
|
|
dc_term() {
|
|
local t0; t0=$(now_ms); DC_FREE_MS=-1; DC_EXIT_MS=-1
|
|
local held=0; [ -n "$(token_holder)" ] && held=1
|
|
kill -TERM "$DC_PID"
|
|
while [ $(( $(now_ms) - t0 )) -lt 30000 ]; do
|
|
[ $held = 1 ] && [ $DC_FREE_MS -lt 0 ] && [ -z "$(token_holder)" ] && DC_FREE_MS=$(( $(now_ms) - t0 ))
|
|
pid_alive "$DC_PID" || { DC_EXIT_MS=$(( $(now_ms) - t0 )); break; }
|
|
sleep 0.01
|
|
done
|
|
pid_alive "$DC_PID" && kill -KILL "$DC_PID"
|
|
wait "$DC_PID" 2>/dev/null; RC=$?
|
|
}
|
|
# dc_ref TAG ARGS... : turn 1 and turn 2 on one server that never restarts -> DC_C1, DC_REF2, DC_PN_REF, DC_N2, DC_MS_FULL
|
|
dc_ref() {
|
|
local tag=$1; shift
|
|
dc_start "$tag" "" -- "$@" || { fail "diskcache/$tag" "reference server did not start, see $S/$tag.log"; return 1; }
|
|
local code; code=$(dc_chat "$PC" "$S/$tag-1.json" "$(dc_msgs "$DC_U1")")
|
|
DC_C1=$(json_str content "$S/$tag-1.json")
|
|
[ "$code" = 200 ] && [ -n "$DC_C1" ] || { fail "diskcache/$tag" "turn 1: HTTP $code"; stop_server "$DC_PID"; return 1; }
|
|
DC_M2=$(dc_msgs "$DC_U1" "$DC_C1" "$DC_U2")
|
|
code=$(dc_chat "$PC" "$S/$tag-2.json" "$DC_M2")
|
|
DC_REF2=$(json_str content "$S/$tag-2.json"); DC_PN_REF=$(json_num prompt_n "$S/$tag-2.json"); DC_N2=$(json_num prompt_tokens "$S/$tag-2.json")
|
|
stop_server "$DC_PID"
|
|
# the same turn 2 on a fresh server: the full prefill the cache saves
|
|
dc_start "$tag-full" "" -- "$@" && dc_chat "$PC" "$S/$tag-full.json" "$DC_M2" > /dev/null
|
|
DC_MS_FULL=$(sed -n 's/.*"prompt_ms":\([0-9.]*\).*/\1/p' "$S/$tag-full.json" | head -1)
|
|
stop_server "$DC_PID"
|
|
[ "$code" = 200 ] && [ -n "$DC_REF2" ] && [ "${DC_PN_REF:-0}" -gt 0 ] || { fail "diskcache/$tag" "turn 2: HTTP $code"; return 1; }
|
|
log "diskcache: reference $tag: turn 2 prompt_n $DC_PN_REF of $DC_N2 tokens, a full prefill of turn 2 took ${DC_MS_FULL} ms"
|
|
}
|
|
# dc_restore TAG DIR COLD ENV... -- ARGS... : turn 1, SIGTERM (COLD=1: forced cold by a fake waiter first),
|
|
# restart, turn 2 -> restored from disk, prompt_n == reference, output == reference
|
|
dc_restore() {
|
|
local tag=$1 dir=$2 cold=$3; shift 3
|
|
local envs=(); while [ "$1" != -- ]; do envs+=("$1"); shift; done; shift
|
|
rm -rf "$dir"
|
|
dc_start "$tag-a" "$dir" "${envs[@]}" -- "$@" || { fail "diskcache/$tag" "server did not start, see $S/$tag-a.log"; return 1; }
|
|
local code c1 w=; code=$(dc_chat "$PC" "$S/$tag-a.json" "$(dc_msgs "$DC_U1")"); c1=$(json_str content "$S/$tag-a.json")
|
|
if [ "$cold" = 1 ]; then
|
|
( exec 9<"$ARENA/want.lock"; flock -s 9; exec sleep 120 ) & w=$!; track "$w"
|
|
wait_holder "" 10000 > /dev/null || log "warning: $tag-a did not go cold"
|
|
fi
|
|
local was_cold=no; [ "$cold" = 1 ] && [ -z "$(token_holder)" ] && grep -q 'server entering sleeping state' "$S/$tag-a.log" && was_cold=yes
|
|
dc_term
|
|
[ -n "$w" ] && { kill "$w" 2>/dev/null; wait "$w" 2>/dev/null; }
|
|
local rc1=$RC exit1=$DC_EXIT_MS free1=$DC_FREE_MS nf
|
|
nf=$(find "$dir" -name '*.lpcd' 2>/dev/null | wc -l)
|
|
local f1; f1=$(find "$dir" -name '*.lpcd' 2>/dev/null | head -1); [ -n "$f1" ] && cp "$f1" "$S/$tag-turn1.lpcd"
|
|
dc_start "$tag-b" "$dir" "${envs[@]}" -- "$@" || { fail "diskcache/$tag" "restart failed, see $S/$tag-b.log"; return 1; }
|
|
code=$(dc_chat "$PC" "$S/$tag-b.json" "$DC_M2")
|
|
local c2 pn ms nres; c2=$(json_str content "$S/$tag-b.json"); pn=$(json_num prompt_n "$S/$tag-b.json")
|
|
ms=$(sed -n 's/.*"prompt_ms":\([0-9.]*\).*/\1/p' "$S/$tag-b.json" | head -1)
|
|
nres=$(grep -c 'disk cache: restored' "$S/$tag-b.log")
|
|
local ok=1
|
|
[ "$code" = 200 ] && [ "$rc1" -eq 0 ] && [ "$nf" -eq 1 ] && [ "$nres" -eq 1 ] && [ "$c1" = "$DC_C1" ] && \
|
|
[ "${pn:-0}" = "$DC_PN_REF" ] && [ "${pn:-0}" -lt "${DC_N2:-0}" ] && [ "$c2" = "$DC_REF2" ] && ok=0
|
|
[ "$cold" = 1 ] && [ $was_cold != yes ] && ok=1
|
|
check "diskcache/$tag" $ok \
|
|
"$([ "$cold" = 1 ] && echo "cold before SIGTERM: $was_cold, ")exit $rc1 in ${exit1} ms$([ "$free1" -ge 0 ] && echo ", token free in ${free1} ms"), $nf file(s); restart: $nres restore(s), turn 2 prompt_n $pn (reference $DC_PN_REF) of $DC_N2 tokens in ${ms} ms (full prefill ${DC_MS_FULL} ms), output $([ "$c2" = "$DC_REF2" ] && echo == || echo '!=') reference"
|
|
log "diskcache: $tag: $(grep -h 'disk cache: \(copied\|saved\)' "$S/$tag-a.log" | sed 's/.*disk cache: //' | tr '\n' ';') $(grep -h 'disk cache: restored' "$S/$tag-b.log" | sed 's/.*disk cache: //')"
|
|
stop_server "$DC_PID"
|
|
}
|
|
run_diskcache() {
|
|
[ -f "$SMALL" ] || { fail diskcache "small model missing: $SMALL"; return; }
|
|
DC_U1="$(moex_prompt 12)"
|
|
DC_U2="Which town ships the fewest crates of salt each week?"
|
|
dc_ref dc-ref || return
|
|
|
|
# (1) save on SIGTERM, restore after a restart: plain, arbiter env warm, arbiter env cold
|
|
dc_restore restore "$S/dc" 0 -- || return
|
|
[ -f "$S/restore-turn1.lpcd" ] || { fail diskcache "no file saved on SIGTERM, see $S/restore-a.log"; return; }
|
|
dc_restore restore-arbiter "$S/dc-arb" 0 "${DC_ENV[@]}" --
|
|
dc_restore restore-cold "$S/dc-cold" 1 "${DC_ENV[@]}" --
|
|
# shutdown cost: the same warm arbiter shutdown without the cache
|
|
rm -rf "$S/dc-none"
|
|
dc_start dc-none "" "${DC_ENV[@]}" -- && dc_chat "$PC" "$S/dc-none.json" "$(dc_msgs "$DC_U1")" > /dev/null
|
|
dc_term
|
|
log "diskcache: warm arbiter shutdown without the cache: exit $RC in ${DC_EXIT_MS} ms, token free in ${DC_FREE_MS} ms"
|
|
|
|
# (2) no restore for a prompt that does not start with a saved one, or under another key
|
|
local code pn n nres
|
|
dc_start dc-np "$S/dc" -- || { fail diskcache/start "see $S/dc-np.log"; return; }
|
|
code=$(dc_chat "$PC" "$S/dc-np.json" "$(dc_msgs "Variant B. $DC_U1")")
|
|
pn=$(json_num prompt_n "$S/dc-np.json"); n=$(json_num prompt_tokens "$S/dc-np.json"); nres=$(grep -c 'disk cache: restored' "$S/dc-np.log")
|
|
check diskcache/no-prefix $(( code == 200 && nres == 0 && pn == n ? 0 : 1 )) "other conversation: HTTP $code, $nres restore(s), prompt_n $pn of $n"
|
|
stop_server "$DC_PID"
|
|
local tag args
|
|
ln -sf "$SMALL" "$S/other-path.gguf"
|
|
for tag in key-ctx key-ctk key-model; do
|
|
case $tag in
|
|
key-ctx) args=(-c 8192) ;;
|
|
key-ctk) args=(-ctk q8_0) ;;
|
|
key-model) args=(-m "$S/other-path.gguf") ;;
|
|
esac
|
|
rm -rf "$S/dc-$tag"; mkdir -p "$S/dc-$tag"; cp "$S/restore-turn1.lpcd" "$S/dc-$tag/"
|
|
dc_start "dc-$tag" "$S/dc-$tag" -- "${args[@]}" || { fail "diskcache/$tag" "see $S/dc-$tag.log"; continue; }
|
|
code=$(dc_chat "$PC" "$S/dc-$tag.json" "$DC_M2")
|
|
pn=$(json_num prompt_n "$S/dc-$tag.json"); n=$(json_num prompt_tokens "$S/dc-$tag.json"); nres=$(grep -c 'disk cache: restored' "$S/dc-$tag.log")
|
|
check "diskcache/$tag" $(( code == 200 && nres == 0 && pn == n ? 0 : 1 )) \
|
|
"${args[*]}: HTTP $code, $nres restore(s), prompt_n $pn of $n; index: $(grep -h 'disk cache: .*prompts of this model' "$S/dc-$tag.log" | sed 's/.*, \([0-9]* prompts\)/\1/')"
|
|
stop_server "$DC_PID"
|
|
done
|
|
|
|
# (3) corrupt and truncated files are removed, never used
|
|
local size; size=$(stat -c %s "$S/restore-turn1.lpcd")
|
|
rm -rf "$S/dc-bad"; mkdir -p "$S/dc-bad"
|
|
head -c $((size / 2)) "$S/restore-turn1.lpcd" > "$S/dc-bad/trunc.lpcd"
|
|
cp "$S/restore-turn1.lpcd" "$S/dc-bad/flip.lpcd"
|
|
printf '\377' | dd of="$S/dc-bad/flip.lpcd" bs=1 seek=$((size / 2)) conv=notrunc 2>/dev/null
|
|
dc_start dc-bad "$S/dc-bad" -- || { fail diskcache/corrupt "see $S/dc-bad.log"; return; }
|
|
local trunc_gone=no; [ -e "$S/dc-bad/trunc.lpcd" ] || trunc_gone=yes
|
|
code=$(dc_chat "$PC" "$S/dc-bad.json" "$DC_M2")
|
|
pn=$(json_num prompt_n "$S/dc-bad.json"); nres=$(grep -c 'disk cache: restored' "$S/dc-bad.log")
|
|
local flip_gone=no; [ -e "$S/dc-bad/flip.lpcd" ] || flip_gone=yes
|
|
check diskcache/corrupt $([ $trunc_gone = yes ] && [ $flip_gone = yes ] && [ "$code" = 200 ] && [ "$nres" -eq 0 ] && [ "$pn" = "$DC_N2" ] && [ "$(json_str content "$S/dc-bad.json")" = "$DC_REF2" ] && echo 0 || echo 1) \
|
|
"truncated file removed at startup: $trunc_gone, byte-flipped file removed at load: $flip_gone, $nres restore(s), prompt_n $pn of $DC_N2, output $([ "$(json_str content "$S/dc-bad.json")" = "$DC_REF2" ] && echo == || echo '!=') reference"
|
|
stop_server "$DC_PID"
|
|
|
|
# (3') LRU size limit: room for one file only, the second save evicts the first
|
|
local cap=$(( size * 3 / 2 / 1048576 + 1 )) files total
|
|
rm -rf "$S/dc-lru"
|
|
dc_start dc-lru1 "$S/dc-lru" -- --cache-disk-size "$cap" && dc_chat "$PC" "$S/dc-lru1.json" "$(dc_msgs "$DC_U1")" > /dev/null
|
|
stop_server "$DC_PID"
|
|
dc_start dc-lru2 "$S/dc-lru" -- --cache-disk-size "$cap" && dc_chat "$PC" "$S/dc-lru2.json" "$(dc_msgs "Variant B. $DC_U1")" > /dev/null
|
|
stop_server "$DC_PID"
|
|
files=$(find "$S/dc-lru" -name '*.lpcd' | wc -l); total=$(du -cb "$S"/dc-lru/*.lpcd 2>/dev/null | tail -1 | cut -f1)
|
|
check diskcache/lru $(( files == 1 && ${total:-0} <= cap * 1048576 && $(grep -c 'size limit, removing' "$S/dc-lru2.log") == 1 ? 0 : 1 )) \
|
|
"limit $cap MiB (file ~$((size / 1048576)) MiB): $files file(s), $((${total:-0} / 1048576)) MiB after two saves, $(grep -c 'size limit, removing' "$S/dc-lru2.log") eviction(s)"
|
|
|
|
# (3'') write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored -> EFBIG): logged, exit 0, nothing left, next start serves
|
|
rm -rf "$S/dc-full"
|
|
SRV_WRAP=(bash -c 'trap "" XFSZ; ulimit -f 1024; exec "$@"' _)
|
|
dc_start dc-full "$S/dc-full" -- && dc_chat "$PC" "$S/dc-full.json" "$(dc_msgs "$DC_U1")" > /dev/null
|
|
SRV_WRAP=()
|
|
dc_term
|
|
local left; left=$(find "$S/dc-full" -type f 2>/dev/null | wc -l)
|
|
dc_start dc-full2 "$S/dc-full" -- || { fail diskcache/write-error "restart failed, see $S/dc-full2.log"; return; }
|
|
code=$(dc_chat "$PC" "$S/dc-full2.json" "$DC_M2")
|
|
check diskcache/write-error $([ "$RC" -eq 0 ] && [ "$left" -eq 0 ] && grep -q 'disk cache: failed to write' "$S/dc-full.log" && [ "$code" = 200 ] && echo 0 || echo 1) \
|
|
"exit $RC in ${DC_EXIT_MS} ms, '$(grep -h 'disk cache: failed to write' "$S/dc-full.log" | sed 's/.*: \([^:]*\), not saved/\1/' | head -1)', $left file(s) left, next start HTTP $code"
|
|
stop_server "$DC_PID"
|
|
|
|
# (1') GPU: the state is saved from VRAM (warm) and from a host shadow (cold), restored into VRAM
|
|
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
|
|
if [ $free_mb -lt 1536 ]; then
|
|
skip diskcache/gpu "only $free_mb MiB VRAM free (< 1536)"
|
|
elif dc_ref dc-ref-gpu "${DC_GPU[@]}"; then
|
|
dc_restore gpu-warm "$S/dc-gpu" 0 "${DC_ENV[@]}" -- "${DC_GPU[@]}"
|
|
dc_restore gpu-cold "$S/dc-gpu" 1 "${DC_ENV[@]}" -- "${DC_GPU[@]}"
|
|
dc_start dc-none-gpu "" "${DC_ENV[@]}" -- "${DC_GPU[@]}" && dc_chat "$PC" "$S/dc-none-gpu.json" "$(dc_msgs "$DC_U1")" > /dev/null
|
|
dc_term
|
|
log "diskcache: warm GPU arbiter shutdown without the cache: exit $RC in ${DC_EXIT_MS} ms, token free in ${DC_FREE_MS} ms"
|
|
fi
|
|
|
|
local bad_lines; bad_lines=$(scan_logs "$S"/dc-*.log "$S"/restore*.log "$S"/gpu-*.log)
|
|
check diskcache/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) "${bad_lines:-clean}"
|
|
}
|
|
|
|
# ================================================================ main
|
|
if [ -n "$PERF_REPORT" ]; then
|
|
[ -r "$PERF_REPORT" ] || { echo "cannot read $PERF_REPORT" >&2; exit 2; }
|
|
perf_report "$PERF_REPORT"; exit 0
|
|
fi
|
|
[ $DO_OPS -eq 1 ] && run_ops
|
|
[ $DO_CONCAT -eq 1 ] && run_concat
|
|
[ $DO_ARB -eq 1 ] && run_arbiter
|
|
[ $DO_PERF -eq 1 ] && run_perf
|
|
[ $DO_MOEC -eq 1 ] && run_moecache
|
|
[ $DO_MOEX -eq 1 ] && run_moe_exact
|
|
[ $DO_DC -eq 1 ] && run_diskcache
|
|
exit 0
|