Production runs the 35B and gemma with an mmproj, and the first deploy of the disk prompt cache aborted on that path. Every case had used --no-mmproj. Now --no-mmproj is in DC_MM, separate from DC_ARGS, and two cases run with the 0.8B's mmproj (--no-mmproj-offload, arbiter env, LLAMA_MMPROJ_ONDEMAND=1), against their own reference: - restore-mmproj: turn 1 is saved on SIGTERM and restored after the restart, prompt_n equal to the reference, output identical; - mmproj-image: an image request (no restore, answer "red"), then the text chat (restored, identical), then an image again, then SIGTERM with media in the slot (no state copied, exit 0, the file count unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1189 lines
73 KiB
Bash
1189 lines
73 KiB
Bash
#!/usr/bin/env bash
|
|
# fork-validate.sh - validate a candidate build of the RX 580 llama.cpp fork against a reference build.
|
|
#
|
|
# Runs ON THE POD (Vulkan / RADV, RX 580). Prints PASS/FAIL/SKIP per check plus a summary; exits
|
|
# nonzero if any check FAILs. Only touches its own scratch dir under /tmp, its own scratch ports and
|
|
# processes it started itself (killed by explicit PID). Build dirs are only read.
|
|
#
|
|
# fork-validate.sh --build /root/arms/X/build --ref /root/llama.cpp/build [checks] [options]
|
|
#
|
|
# Checks (no check flag = --ops --concat --arbiter):
|
|
# --ops test-backend-ops on Vulkan0 for CONCAT, FLASH_ATTN_EXT, MUL_MAT, MUL_MAT_ID, GET_ROWS
|
|
# -> the fork's kernels and upstream's agree with the CPU reference; asserts the case count
|
|
# is > 0 because "-b Vulkan" (no index) silently runs nothing and still prints OK.
|
|
# --concat Vulkan concat tiled-transpose fast path (ggml_vk_concat, gate src_transposed_2d)
|
|
# -> it fires at prefill and NOT at decode on a real hybrid (delta-net) model; it is the
|
|
# fork's biggest prefill win, and taking it at decode once cost -17% decode.
|
|
# Needs a build carrying the GGML_VK_CONCAT_FASTPATH knob (=0 off, =log report);
|
|
# either --build has it or pass --concat-build DIR (candidate + the knob patch).
|
|
# --arbiter on-demand VRAM sharing (LLAMA_SLEEP_VRAM_ONLY ...) with two real servers on a scratch
|
|
# arena -> a cold server really gives its VRAM back (v-vram-release: the card-wide
|
|
# delta over a baseline taken before the scratch servers start must drop by >= 70% of
|
|
# what it added warm, and come back on restore; production's own DRM clients are
|
|
# re-read each time and the check SKIPs if they moved by > 64 MiB), handoffs never
|
|
# strand, release/restore is bit-exact, multimodal survives it,
|
|
# and every shutdown path (mid-release, mid-wait, mid-generation, cold, warm, kill -9 of
|
|
# a waiter) exits cleanly and hands the GPU token on. These are the past production
|
|
# incidents (stranded requests, wedged token, exit 134 on cold teardown).
|
|
# Then a-handoff-latency: ordinary handoff latency of --build vs --ref in the same
|
|
# session (fresh server pairs, blocks ref,cand,cand,ref x 16 handoffs) -> FAIL if the
|
|
# candidate median exceeds the reference median by more than max(25%, 200 ms). Two
|
|
# production deploys were lost to arbiter changes that only showed up here.
|
|
# Needs >= 1536 MiB free VRAM (FAILs otherwise); beside a warm gemma-26B only about
|
|
# 1.2 GiB is free, so then run it in the --perf swapdown window (about 2 minutes).
|
|
# --perf llama-bench, candidate vs reference, alternating ref,cand,cand,ref, on the 35B MoE
|
|
# -> no prefill regression > 5% at depth 0/16k/32k (FAIL), decode tg256@d4096
|
|
# reported; peak VRAM per arm (FAIL if > 95% of the card). Compares MEDIANS of every
|
|
# repetition sample pooled over a build's two arms (4 per prefill cell, 6 for decode),
|
|
# never means: CPU contention on the node produces single catastrophic samples.
|
|
# Needs llama-swap STOPPED (run it under /root/bench/moe/swapdown.sh); refuses to
|
|
# run if any other llama process is up. Samples are kept in <scratch>/perf.tsv.
|
|
# --moecache MODEL
|
|
# one arbiter server with --moe-cache-mib N on a MoE model, forced cold/warm twice
|
|
# -> greedy output identical before/after (the expert-cache SLRU is re-initialised on
|
|
# release; a stale SLRU maps experts to empty banks and produces garbage).
|
|
# --moe-exact MODEL
|
|
# the MoE routing-ids readback skip (F1, sched_copy_experts) is exact: MODEL (a MoE,
|
|
# e.g. the --perf model) with every expert on the host, greedy, one prompt of >= 2x
|
|
# the skip threshold ceil(4 * n_expert / n_expert_used) tokens (read from the GGUF;
|
|
# 128 on the 35B) -> candidate output byte-identical to the reference, and, if the
|
|
# candidate has the LLAMA_MOE_IDS_SKIP knob, to the candidate with the skip forced off
|
|
# (=0); =log proves the skip engaged. Without the knob skip-engaged is SKIPped with the
|
|
# batch-size argument. Needs the GPU to itself like --perf (same guard); about 3 model
|
|
# loads, so run it in the same swapdown window as --perf.
|
|
# --diskcache the disk prompt cache (--cache-disk-path) on the small hybrid model, CPU only (no VRAM;
|
|
# runs beside production), plus the GPU path when >= 1.5 GiB VRAM is free
|
|
# -> a slot state saved on SIGTERM (plain, arbiter warm, arbiter cold) is restored after a
|
|
# restart: only the suffix is processed (prompt_n == a server that never restarted) and
|
|
# the greedy answer is identical; no restore for another conversation or another model
|
|
# path / n_ctx / KV type; truncated and corrupt files are removed, never used; the size
|
|
# limit evicts the oldest file; a write error (RLIMIT_FSIZE) is logged and exit stays 0;
|
|
# with an mmproj loaded (as production runs) a text chat is restored the same way, and an
|
|
# image prompt is never restored or saved and does not crash.
|
|
#
|
|
# Options:
|
|
# --concat-build DIR build with the GGML_VK_CONCAT_FASTPATH knob for --concat (default: --build)
|
|
# --small MODEL small hybrid model for --concat/--arbiter (default: Qwen3.5-0.8B UD-Q4_K_XL)
|
|
# --mmproj FILE its multimodal projector (default: mmproj-F16 next to it)
|
|
# --perf-model MODEL model for --perf (default: Qwen3.6-35B-A3B UD-Q4_K_XL)
|
|
# --moecache-args S extra server args for --moecache (default: "-ngl 99 --n-cpu-moe 30 -c 4096")
|
|
# --moecache-mib N cache size for --moecache (default 2048; 256 never hits on a 26B and proves nothing)
|
|
# --port-base N scratch ports N..N+2 (default 8091)
|
|
# --keep keep the scratch dir even when everything passed (it is kept on any FAIL)
|
|
# --moe-exact-args S server args for --moe-exact (default: "-ngl 99 --n-cpu-moe 99 -c 4096 -b 2048
|
|
# -ub 2048 -t 6"; keep the experts on the host and -ub >= the prompt)
|
|
# --perf-report TSV only re-aggregate a saved <scratch>/perf.tsv of an earlier --perf run and
|
|
# apply the same gates (no GPU, no --build/--ref needed); runs nothing else
|
|
#
|
|
# Requirements: bash, coreutils, awk, sed, grep, curl, flock(1); gdb optional (makes the
|
|
# SIGTERM-during-release case deterministic; it is not in the pod image and is lost on a pod restart,
|
|
# then the case falls back to polling the log). No python/jq.
|
|
#
|
|
# Test only: FV_TEST_SHARED_GPU=1 skips the exclusive-GPU guard of --perf / --moe-exact, for running
|
|
# the script against stub binaries. Never set it for a real measurement.
|
|
|
|
set -u
|
|
|
|
# ---------------------------------------------------------------- defaults
|
|
HF=/root/.cache/huggingface/hub
|
|
SMALL_DIR=$HF/models--unsloth--Qwen3.5-0.8B-GGUF/snapshots/6ab461498e2023f6e3c1baea90a8f0fe38ab64d0
|
|
SMALL=$SMALL_DIR/Qwen3.5-0.8B-UD-Q4_K_XL.gguf
|
|
MMPROJ=
|
|
PERF_MODEL=$HF/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
|
|
MOECACHE_MODEL=
|
|
MOECACHE_ARGS="-ngl 99 --n-cpu-moe 30 -c 4096"
|
|
MOECACHE_MIB=2048
|
|
MOEX_MODEL=
|
|
MOEX_ARGS="-ngl 99 --n-cpu-moe 99 -c 4096 -b 2048 -ub 2048 -t 6"
|
|
BUILD= ; REF= ; CONCAT_BUILD=
|
|
PORT_BASE=8091
|
|
KEEP=0
|
|
DO_OPS=0 DO_CONCAT=0 DO_ARB=0 DO_PERF=0 DO_MOEC=0 DO_MOEX=0 DO_DC=0
|
|
PERF_REPORT=
|
|
|
|
usage() { awk 'NR > 1 && /^#/ {sub(/^# ?/, ""); print; next} NR > 1 {exit}' "$0"; exit 2; }
|
|
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--build) BUILD=$2; shift 2 ;;
|
|
--ref) REF=$2; shift 2 ;;
|
|
--ops) DO_OPS=1; shift ;;
|
|
--concat) DO_CONCAT=1; shift ;;
|
|
--arbiter) DO_ARB=1; shift ;;
|
|
--diskcache) DO_DC=1; shift ;;
|
|
--perf) DO_PERF=1; shift ;;
|
|
--moecache) DO_MOEC=1; MOECACHE_MODEL=$2; shift 2 ;;
|
|
--moecache-args) MOECACHE_ARGS=$2; shift 2 ;;
|
|
--moecache-mib) MOECACHE_MIB=$2; shift 2 ;;
|
|
--moe-exact) DO_MOEX=1; MOEX_MODEL=$2; shift 2 ;;
|
|
--moe-exact-args) MOEX_ARGS=$2; shift 2 ;;
|
|
--concat-build) CONCAT_BUILD=$2; shift 2 ;;
|
|
--small) SMALL=$2; shift 2 ;;
|
|
--mmproj) MMPROJ=$2; shift 2 ;;
|
|
--perf-model) PERF_MODEL=$2; shift 2 ;;
|
|
--port-base) PORT_BASE=$2; shift 2 ;;
|
|
--keep) KEEP=1; shift ;;
|
|
--perf-report) PERF_REPORT=$(readlink -f "$2"); shift 2 ;;
|
|
-h|--help) usage ;;
|
|
*) echo "unknown argument: $1" >&2; usage ;;
|
|
esac
|
|
done
|
|
[ -n "$PERF_REPORT" ] || { [ -n "$BUILD" ] && [ -n "$REF" ]; } || usage
|
|
if [ $((DO_OPS + DO_CONCAT + DO_ARB + DO_PERF + DO_MOEC + DO_MOEX + DO_DC)) -eq 0 ]; then
|
|
DO_OPS=1 DO_CONCAT=1 DO_ARB=1
|
|
fi
|
|
[ -n "$MMPROJ" ] || MMPROJ=$(dirname "$SMALL")/mmproj-F16.gguf
|
|
PA=$PORT_BASE PB=$((PORT_BASE + 1)) PC=$((PORT_BASE + 2))
|
|
|
|
# ---------------------------------------------------------------- reporting
|
|
RESULTS=()
|
|
NFAIL=0
|
|
ts() { date +%H:%M:%S; }
|
|
log() { echo "[$(ts)] $*"; }
|
|
pass() { RESULTS+=("PASS $1: $2"); echo "[$(ts)] PASS $1: $2"; }
|
|
fail() { RESULTS+=("FAIL $1: $2"); echo "[$(ts)] FAIL $1: $2"; NFAIL=$((NFAIL + 1)); }
|
|
skip() { RESULTS+=("SKIP $1: $2"); echo "[$(ts)] SKIP $1: $2"; }
|
|
# check NAME CONDITION-RESULT(0/1) MESSAGE
|
|
check() { if [ "$2" -eq 0 ]; then pass "$1" "$3"; else fail "$1" "$3"; fi; }
|
|
now_ms() { date +%s%3N; }
|
|
|
|
# ---------------------------------------------------------------- scratch + cleanup
|
|
S=$(mktemp -d /tmp/fork-validate.XXXXXX)
|
|
ARENA=$S/arena
|
|
mkdir -p "$ARENA"
|
|
cd "$S" || exit 2 # anything written to a relative path lands in the scratch dir
|
|
PIDS=() # "pid:starttime" of every background process we start (starttime guards PID reuse)
|
|
|
|
pid_alive() { # alive and not a zombie
|
|
[ -r "/proc/$1/stat" ] || return 1
|
|
local st; st=$(sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | awk '{print $1}') || return 1
|
|
[ -n "$st" ] && [ "$st" != Z ]
|
|
}
|
|
starttime() { sed 's/.*) //' "/proc/$1/stat" 2>/dev/null | awk '{print $20}'; }
|
|
track() { PIDS+=("$1:$(starttime "$1")"); }
|
|
ours() { # "pid:starttime" -> pid if that exact process is still alive
|
|
local p=${1%%:*} t=${1#*:}
|
|
pid_alive "$p" && [ "$(starttime "$p")" = "$t" ] && echo "$p"
|
|
}
|
|
|
|
cleanup() {
|
|
local e p
|
|
for e in "${PIDS[@]}"; do p=$(ours "$e") && kill -TERM "$p" 2>/dev/null; done
|
|
local i=0
|
|
while [ $i -lt 50 ]; do
|
|
local any=0
|
|
for e in "${PIDS[@]}"; do ours "$e" > /dev/null && any=1; done
|
|
[ $any -eq 0 ] && break
|
|
sleep 0.1; i=$((i + 1))
|
|
done
|
|
for e in "${PIDS[@]}"; do p=$(ours "$e") && kill -KILL "$p" 2>/dev/null; done
|
|
wait 2>/dev/null
|
|
}
|
|
finish() {
|
|
local rc=$?
|
|
trap - EXIT INT TERM HUP
|
|
cleanup
|
|
echo
|
|
echo "==================== fork-validate summary ===================="
|
|
echo "candidate: $BUILD ($(git -C "$BUILD/.." rev-parse --short HEAD 2>/dev/null || echo '?'))"
|
|
echo "reference: $REF ($(git -C "$REF/.." rev-parse --short HEAD 2>/dev/null || echo '?'))"
|
|
local r; for r in "${RESULTS[@]}"; do echo "$r"; done
|
|
local np ns
|
|
np=$(printf '%s\n' "${RESULTS[@]}" | grep -c '^PASS')
|
|
ns=$(printf '%s\n' "${RESULTS[@]}" | grep -c '^SKIP')
|
|
echo "total: $np PASS, $NFAIL FAIL, $ns SKIP"
|
|
if [ $NFAIL -gt 0 ] || [ $KEEP -eq 1 ] || [ $rc -ne 0 ]; then
|
|
echo "scratch kept: $S"
|
|
else
|
|
rm -rf "$S"
|
|
fi
|
|
[ $NFAIL -gt 0 ] && exit 1
|
|
exit $rc
|
|
}
|
|
trap finish EXIT
|
|
trap 'echo "interrupted"; NFAIL=$((NFAIL + 1)); exit 130' INT TERM HUP
|
|
|
|
# ---------------------------------------------------------------- environment sanity
|
|
for v in $(env | sed -n 's/^\(GGML_[A-Za-z0-9_]*\|LLAMA_\(SLEEP\|VRAM\|MMPROJ\|ARG\|CACHE\|LOG\|MOE\)[A-Za-z0-9_]*\)=.*/\1/p'); do
|
|
log "unsetting inherited $v"; unset "$v"
|
|
done
|
|
[ -n "$PERF_REPORT" ] || for d in "$BUILD" "$REF"; do
|
|
[ -x "$d/bin/llama-server" ] || { echo "no llama-server in $d/bin" >&2; exit 2; }
|
|
done
|
|
case "$ARENA" in /dev/shm/llama-vram*) echo "refusing the production arena" >&2; exit 2 ;; esac
|
|
VRAM_USED=$(ls /sys/class/drm/card*/device/mem_info_vram_used 2>/dev/null | head -1)
|
|
VRAM_TOTAL=$(ls /sys/class/drm/card*/device/mem_info_vram_total 2>/dev/null | head -1)
|
|
vram_mb() { echo $(( $(cat "$VRAM_USED") / 1048576 )); }
|
|
port_busy() { # LISTEN socket on the port (v4 or v6)
|
|
local hex; hex=$(printf '%04X' "$1")
|
|
awk -v h=":$hex" '$4=="0A" && substr($2, length($2)-4)==h {f=1} END {exit !f}' /proc/net/tcp /proc/net/tcp6 2>/dev/null
|
|
}
|
|
[ -n "$PERF_REPORT" ] || for p in $PA $PB $PC; do
|
|
port_busy "$p" && { echo "scratch port $p is in use" >&2; exit 2; }
|
|
done
|
|
log "scratch dir $S, ports $PA/$PB/$PC, VRAM used $(vram_mb) MiB"
|
|
|
|
# ---------------------------------------------------------------- helpers: servers
|
|
# start_server NAME BUILD PORT ENV-ASSIGNMENTS... -- SERVER-ARGS... (sets SPID, log at $S/NAME.log)
|
|
# SRV_WRAP: optional command prefix that execs its arguments (the PID stays the server's)
|
|
SRV_WRAP=()
|
|
start_server() {
|
|
local name=$1 build=$2 port=$3; shift 3
|
|
local envs=()
|
|
while [ $# -gt 0 ] && [ "$1" != -- ]; do envs+=("$1"); shift; done
|
|
shift
|
|
"${SRV_WRAP[@]}" env LD_LIBRARY_PATH="$build/bin" "${envs[@]}" "$build/bin/llama-server" \
|
|
--host 127.0.0.1 --port "$port" "$@" >> "$S/$name.log" 2>&1 &
|
|
SPID=$!
|
|
track "$SPID"
|
|
echo "=== start pid $SPID $(date +%T.%N)" >> "$S/$name.log.events"
|
|
}
|
|
# wait_health PORT PID TIMEOUT_S
|
|
wait_health() {
|
|
local t0; t0=$(now_ms)
|
|
while [ $(( $(now_ms) - t0 )) -lt $(( $3 * 1000 )) ]; do
|
|
pid_alive "$2" || return 1
|
|
[ "$(curl -s -m 2 -o /dev/null -w '%{http_code}' "http://127.0.0.1:$1/health")" = 200 ] && return 0
|
|
sleep 0.2
|
|
done
|
|
return 1
|
|
}
|
|
# wait_exit PID TIMEOUT_MS -> prints elapsed ms, returns 1 on timeout. Then reap with `wait PID`.
|
|
wait_exit() {
|
|
local t0; t0=$(now_ms)
|
|
while pid_alive "$1"; do
|
|
[ $(( $(now_ms) - t0 )) -ge "$2" ] && { echo $(( $(now_ms) - t0 )); return 1; }
|
|
sleep 0.02
|
|
done
|
|
echo $(( $(now_ms) - t0 ))
|
|
}
|
|
# stop_server PID -> SIGTERM, wait up to 20 s, sets RC and EXIT_MS
|
|
stop_server() {
|
|
kill -TERM "$1" 2>/dev/null
|
|
EXIT_MS=$(wait_exit "$1" 20000) || { kill -KILL "$1" 2>/dev/null; }
|
|
wait "$1" 2>/dev/null; RC=$?
|
|
}
|
|
|
|
# ---------------------------------------------------------------- helpers: locks (/proc/locks)
|
|
lock_id() { # maj:min:inode as /proc/locks prints it (hex dev, decimal inode)
|
|
local d; d=$(( 16#$(stat -c %D "$1") ))
|
|
printf '%02x:%02x:%s' $(( (d >> 8) & 0xfff )) $(( (d & 0xff) | ((d >> 12) & 0xfff00) )) "$(stat -c %i "$1")"
|
|
}
|
|
lock_pids() { # FILE WRITE|READ -> PIDs holding a flock on it (granted, not blocked)
|
|
[ -e "$1" ] || return 0
|
|
awk -v id="$(lock_id "$1")" -v m="$2" '$2=="FLOCK" && $4==m && $6==id {print $5}' /proc/locks
|
|
}
|
|
token_holder() { lock_pids "$ARENA/token.lock" WRITE | head -1; }
|
|
holder_num() { local h; h=$(token_holder); echo "${h:-0}"; } # for arithmetic: 0 = nobody
|
|
# wait_holder PID TIMEOUT_MS : until PID holds token.lock ("" = until nobody does)
|
|
wait_holder() {
|
|
local t0; t0=$(now_ms)
|
|
while [ "$(token_holder)" != "$1" ]; do
|
|
[ $(( $(now_ms) - t0 )) -ge "$2" ] && return 1
|
|
sleep 0.01
|
|
done
|
|
echo $(( $(now_ms) - t0 ))
|
|
}
|
|
wait_want() { # PID TIMEOUT_MS : until PID holds want.lock shared (registered waiter)
|
|
local t0; t0=$(now_ms)
|
|
until lock_pids "$ARENA/want.lock" READ | grep -qx "$1"; do
|
|
[ $(( $(now_ms) - t0 )) -ge "$2" ] && return 1
|
|
sleep 0.01
|
|
done
|
|
}
|
|
|
|
# ---------------------------------------------------------------- helpers: VRAM attribution
|
|
# mem_info_vram_used is card-wide and includes production. Production's own share is read from the
|
|
# DRM fdinfo of every process this script did not start (drm-memory-vram, deduplicated by
|
|
# drm-client-id; summing fds double-counts shared buffers), so a production model that warms or
|
|
# cools during a measurement is detected instead of being attributed to the scratch servers.
|
|
drm_vram_kib() { # PID... -> KiB of VRAM held by the DRM clients of these processes
|
|
local p fs=()
|
|
for p in "$@"; do [ -d "/proc/$p/fdinfo" ] && fs+=("/proc/$p/fdinfo/"*); done
|
|
[ ${#fs[@]} -gt 0 ] || { echo 0; return; }
|
|
cat "${fs[@]}" 2>/dev/null | awk '/^pos:/ {c = ""} /^drm-client-id:/ {c = $2}
|
|
/^drm-memory-vram:/ && c != "" {v = $2; if ($3 == "MiB") v *= 1024; m[c] = v}
|
|
END {for (k in m) s += m[k]; print s + 0}'
|
|
}
|
|
prod_vram_mb() { # MiB of VRAM held by every process this script did not start
|
|
local p e mine=" " others=()
|
|
for e in "${PIDS[@]}"; do mine+="${e%%:*} "; done
|
|
for p in /proc/[0-9]*; do
|
|
p=${p#/proc/}
|
|
case "$mine" in *" $p "*) ;; *) others+=("$p") ;; esac
|
|
done
|
|
echo $(( $(drm_vram_kib "${others[@]}") / 1024 ))
|
|
}
|
|
|
|
# ---------------------------------------------------------------- helpers: requests
|
|
json_str() { sed -n 's/.*"'"$1"'":"\(\([^"\\]\|\\.\)*\)".*/\1/p' "$2" | head -1; }
|
|
json_num() { grep -o "\"$1\":[0-9]*" "$2" | head -1 | cut -d: -f2; }
|
|
# complete PORT OUTFILE JSON-BODY [TIMEOUT_S] -> prints HTTP code ("000" on timeout/refused)
|
|
complete() {
|
|
curl -sS -m "${4:-60}" -o "$2" -w '%{http_code}' -H 'Content-Type: application/json' \
|
|
-d "$3" "http://127.0.0.1:$1/completion" 2>>"$S/curl.err" || true
|
|
}
|
|
PROMPT='The history of the printing press began in'
|
|
GREEDY="\"temperature\":0,\"top_k\":1,\"seed\":42"
|
|
greedy_body() { # N_PREDICT CACHE_PROMPT [PROMPT]
|
|
echo "{\"prompt\":\"${3:-$PROMPT}\",\"n_predict\":$1,$GREEDY,\"cache_prompt\":$2}"
|
|
}
|
|
LONG_BODY="{\"prompt\":\"Write a very long story about a lighthouse keeper.\",\"n_predict\":4000,\"ignore_eos\":true,$GREEDY,\"stream\":true}"
|
|
# stream_bg PORT OUTFILE BODY -> starts a streaming request, sets CPID
|
|
stream_bg() {
|
|
curl -sS -N -m 300 -H 'Content-Type: application/json' -d "$3" \
|
|
"http://127.0.0.1:$1/completion" > "$2" 2>&1 &
|
|
CPID=$!; track "$CPID"
|
|
}
|
|
wait_tokens() { # FILE N TIMEOUT_MS : until N SSE data lines arrived
|
|
local t0; t0=$(now_ms)
|
|
while [ "$(grep -c '^data: ' "$1" 2>/dev/null)" -lt "$2" ]; do
|
|
[ $(( $(now_ms) - t0 )) -ge "$3" ] && return 1
|
|
sleep 0.05
|
|
done
|
|
}
|
|
scan_logs() { # NAME-PATTERN... -> prints offending lines
|
|
grep -h -n -E 'GGML_ASSERT|Assertion|terminate called|Segmentation fault|core dumped|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST|vk::.*Error|GGML_ABORT|fatal error' "$@" 2>/dev/null | head -20
|
|
}
|
|
|
|
# ================================================================ 1. ops
|
|
run_ops() {
|
|
local op out passed total
|
|
for op in CONCAT FLASH_ATTN_EXT MUL_MAT MUL_MAT_ID GET_ROWS; do
|
|
out=$S/ops-$op.log
|
|
log "ops: test-backend-ops -b Vulkan0 -o $op"
|
|
( cd "$S" && LD_LIBRARY_PATH="$BUILD/bin" timeout 1800 "$BUILD/bin/test-backend-ops" -b Vulkan0 -o "$op" ) \
|
|
> "$out" 2>&1
|
|
local rc=$?
|
|
# strip ANSI colours; "N/M tests passed" is printed once per tested backend
|
|
read -r passed total < <(sed 's/\x1b\[[0-9;]*m//g' "$out" | sed -n 's/^ *\([0-9]*\)\/\([0-9]*\) tests passed.*/\1 \2/p' | head -1)
|
|
passed=${passed:-0}; total=${total:-0}
|
|
if [ $rc -eq 0 ] && [ "$total" -gt 0 ] && [ "$passed" -eq "$total" ] \
|
|
&& sed 's/\x1b\[[0-9;]*m//g' "$out" | grep -q 'Backend Vulkan0: OK'; then
|
|
pass "ops/$op" "$passed/$total cases on Vulkan0"
|
|
else
|
|
fail "ops/$op" "rc=$rc, $passed/$total passed (0 cases = wrong backend name); see $out"
|
|
fi
|
|
done
|
|
}
|
|
|
|
# ================================================================ 2. concat fast path
|
|
# Method: llama-bench on the small hybrid model (qwen35: delta-net layers concat the conv state with
|
|
# the transposed qkv in build_conv_state). (1) with the knob =log, count fast-path dispatches in a
|
|
# prefill-only run (must be > 0, all with >= 32 tokens) and a decode-only run (must be 0, while the
|
|
# perf logger proves CONCAT nodes did execute); (2) GGML_VK_PERF_LOGGER CONCAT time at prefill: the
|
|
# unmodified candidate must match knob-on and be >= 2x faster than knob-off, which ties the
|
|
# candidate binary itself (no knob) to the fast path.
|
|
concat_us() { # LOG -> "count avg_us" over all perf-logger CONCAT lines
|
|
awk '/^CONCAT: / {n += $2; t += $(NF-1)} END {if (n) printf "%d %.2f\n", n, t/n; else print "0 0"}' "$1"
|
|
}
|
|
run_concat() {
|
|
local tb=$CONCAT_BUILD
|
|
[ -n "$tb" ] || tb=$BUILD
|
|
if ! grep -qa GGML_VK_CONCAT_FASTPATH "$tb/bin/libggml-vulkan.so" 2>/dev/null; then
|
|
fail concat "no GGML_VK_CONCAT_FASTPATH knob in $tb (build the candidate + knob patch, pass --concat-build)"
|
|
return
|
|
fi
|
|
[ -f "$SMALL" ] || { fail concat "small model missing: $SMALL"; return; }
|
|
local common=(-m "$SMALL" -ngl 99 -fa 1 -lm none -t 6)
|
|
local pp=(-p 512 -n 0 -b 512 -ub 512 -r 3)
|
|
local tg=(-p 0 -n 32 -r 1)
|
|
lb() { # BUILD OUT ENV... -- ARGS
|
|
local b=$1 out=$2; shift 2
|
|
local envs=(); while [ "$1" != -- ]; do envs+=("$1"); shift; done; shift
|
|
env LD_LIBRARY_PATH="$b/bin" GGML_VK_PERF_LOGGER=1 "${envs[@]}" "$b/bin/llama-bench" "${common[@]}" "$@" \
|
|
> "$out" 2>&1
|
|
}
|
|
log "concat: llama-bench prefill/decode arms (perf logger on)"
|
|
lb "$tb" "$S/cc-pp-log.log" GGML_VK_CONCAT_FASTPATH=log -- "${pp[@]}"
|
|
lb "$tb" "$S/cc-tg-log.log" GGML_VK_CONCAT_FASTPATH=log -- "${tg[@]}"
|
|
lb "$tb" "$S/cc-pp-off.log" GGML_VK_CONCAT_FASTPATH=0 -- "${pp[@]}"
|
|
lb "$BUILD" "$S/cc-pp-cand.log" -- "${pp[@]}"
|
|
lb "$tb" "$S/cc-pp-off2.log" GGML_VK_CONCAT_FASTPATH=0 -- "${pp[@]}"
|
|
lb "$BUILD" "$S/cc-pp-cand2.log" -- "${pp[@]}"
|
|
cat "$S/cc-pp-off2.log" >> "$S/cc-pp-off.log"; cat "$S/cc-pp-cand2.log" >> "$S/cc-pp-cand.log"
|
|
|
|
local n_pp n_pp_small n_tg n_tg_concat
|
|
n_pp=$(grep -c 'transpose fast path' "$S/cc-pp-log.log")
|
|
n_pp_small=$(sed -n 's/.*transpose fast path, transposed ne0=\([0-9]*\).*/\1/p' "$S/cc-pp-log.log" | awk '$1 < 32' | wc -l)
|
|
n_tg=$(grep -c 'transpose fast path' "$S/cc-tg-log.log")
|
|
read -r n_tg_concat _ < <(concat_us "$S/cc-tg-log.log")
|
|
local ne0s; ne0s=$(sed -n 's/.*transposed ne0=\([0-9]*\).*/\1/p' "$S/cc-pp-log.log" | sort -n | uniq -c | awk '{printf "%sx%s ", $1, $2}')
|
|
check concat/prefill-fires $(( n_pp > 0 && n_pp_small == 0 ? 0 : 1 )) \
|
|
"$n_pp fast-path dispatches in the prefill run (transposed ne0: ${ne0s:-none})"
|
|
check concat/decode-skips $(( n_tg == 0 && n_tg_concat > 0 ? 0 : 1 )) \
|
|
"$n_tg fast-path dispatches over $n_tg_concat executed CONCAT nodes in the decode run"
|
|
|
|
local c_on c_off c_cand on off cand
|
|
read -r c_on on < <(concat_us "$S/cc-pp-log.log")
|
|
read -r c_off off < <(concat_us "$S/cc-pp-off.log")
|
|
read -r c_cand cand < <(concat_us "$S/cc-pp-cand.log")
|
|
local ok
|
|
ok=$(awk -v on="$on" -v off="$off" -v c="$cand" 'BEGIN {print (c > 0 && on > 0 && off/c >= 2 && c/on < 1.5 && on/c < 1.5) ? 0 : 1}')
|
|
check concat/candidate-timing "$ok" \
|
|
"prefill CONCAT avg: candidate ${cand} us, knob-on ${on} us, knob-off ${off} us (off/cand $(awk -v a="$off" -v b="$cand" 'BEGIN {printf "%.1f", (b > 0) ? a / b : 0}')x)"
|
|
local t_on t_off t_cand
|
|
t_on=$(grep -E '\| +pp512 ' "$S/cc-pp-log.log" | awk -F'|' '{print $(NF-1)}' | awk '{print $1}' | head -1)
|
|
t_off=$(grep -E '\| +pp512 ' "$S/cc-pp-off.log" | awk -F'|' '{print $(NF-1)}' | awk '{print $1}' | tr '\n' ' ')
|
|
t_cand=$(grep -E '\| +pp512 ' "$S/cc-pp-cand.log" | awk -F'|' '{print $(NF-1)}' | awk '{print $1}' | tr '\n' ' ')
|
|
log "concat: pp512 t/s (perf logger on, so absolute speed is not representative): on $t_on | off $t_off | candidate $t_cand"
|
|
}
|
|
|
|
# ================================================================ 3. arbiter
|
|
ARB_ENV=(LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$ARENA" LLAMA_SLEEP_EVICT_KV=1 LLAMA_MMPROJ_ONDEMAND=1)
|
|
ARB_ARGS=(-m "$SMALL" -c 8192 -ngl 99 -fa 1 -lm none --parallel 1 --no-warmup --reasoning off)
|
|
A_PID= ; B_PID=
|
|
start_a() { start_server A "$BUILD" "$PA" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --mmproj "$MMPROJ"; A_PID=$SPID; }
|
|
start_b() { start_server B "$BUILD" "$PB" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --no-mmproj; B_PID=$SPID; }
|
|
sleeps() { grep -c 'server entering sleeping state' "$1"; }
|
|
|
|
make_red_ppm() { # 128x128 solid red, binary PPM (stb_image reads it; no image tools needed)
|
|
local f=$S/red.ppm i
|
|
printf '\377\000\000' > "$S/px"
|
|
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do cat "$S/px" "$S/px" > "$S/px2"; mv "$S/px2" "$S/px"; done
|
|
{ printf 'P6\n128 128\n255\n'; cat "$S/px"; } > "$f"
|
|
base64 -w0 "$f"
|
|
}
|
|
|
|
run_arbiter() {
|
|
[ -f "$SMALL" ] && [ -f "$MMPROJ" ] || { fail arbiter "model/mmproj missing: $SMALL $MMPROJ"; return; }
|
|
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
|
|
if [ $free_mb -lt 1536 ]; then
|
|
fail arbiter "only $free_mb MiB VRAM free (< 1536): run inside the swapdown window"; return
|
|
fi
|
|
# card-wide and production VRAM before any scratch server exists (see prod_vram_mb)
|
|
local v0 p0; v0=$(vram_mb); p0=$(prod_vram_mb)
|
|
log "arbiter: $free_mb MiB VRAM free (card $v0 MiB used, production DRM clients $p0 MiB); starting A (mmproj) on :$PA"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/start "A did not become healthy, see $S/A.log"; return; }
|
|
|
|
# no-handoff reference: A has held the token since its load and never released
|
|
local code ref_a
|
|
code=$(complete "$PA" "$S/ref-a.json" "$(greedy_body 48 false)")
|
|
ref_a=$(json_str content "$S/ref-a.json")
|
|
[ "$code" = 200 ] && [ -n "$ref_a" ] || { fail arbiter/start "reference request on A: HTTP $code"; return; }
|
|
[ "$(token_holder)" = "$A_PID" ] || { fail arbiter/start "A serves but does not hold token.lock"; return; }
|
|
|
|
# ---- (v) VRAM really drops when A goes cold, and comes back when it warms again
|
|
# A is alone on the scratch arena; a fake waiter on want.lock makes it release. Its footprint is
|
|
# the card-wide delta against the baseline taken before A started; production's DRM clients are
|
|
# re-read at every point, and if they moved the check is SKIPped (it cannot attribute the delta).
|
|
local v_warm p_warm v_cold p_cold v_back p_back f_warm f_cold w k=0
|
|
sleep 0.3
|
|
v_warm=$(vram_mb); p_warm=$(prod_vram_mb); f_warm=$(( $(drm_vram_kib "$A_PID") / 1024 ))
|
|
local added=$(( v_warm - v0 )) need
|
|
need=$(( added * 7 / 10 ))
|
|
( exec 9<"$ARENA/want.lock"; flock -s 9; exec sleep 120 ) & w=$!; track "$w"
|
|
if ! wait_holder "" 10000 > /dev/null; then
|
|
fail arbiter/v-vram-release "A did not drop the token for a waiter within 10 s (holder $(token_holder))"
|
|
else
|
|
# the release frees before it unlocks; give the driver a moment to return the pages
|
|
while [ $(( v_warm - $(vram_mb) )) -lt $need ] && [ $k -lt 30 ]; do sleep 0.1; k=$((k + 1)); done
|
|
fi
|
|
v_cold=$(vram_mb); p_cold=$(prod_vram_mb); f_cold=$(( $(drm_vram_kib "$A_PID") / 1024 ))
|
|
kill "$w" 2>/dev/null; wait "$w" 2>/dev/null
|
|
code=$(complete "$PA" "$S/v-a.json" "$(greedy_body 48 false)")
|
|
sleep 0.3
|
|
v_back=$(vram_mb); p_back=$(prod_vram_mb)
|
|
local pmove; pmove=$(printf '%s\n' "$p_warm" "$p_cold" "$p_back" | awk -v b="$p0" '{d = $1 - b; if (d < 0) d = -d; if (d > m) m = d} END {print m + 0}')
|
|
local vmsg="card-wide over baseline: warm +${added} MiB, cold +$((v_cold - v0)) MiB (dropped $((v_warm - v_cold)), need >= ${need} = 70%), restored +$((v_back - v0)) MiB; A's own DRM clients warm ${f_warm} / cold ${f_cold} MiB; production ${p0} -> ${p_warm}/${p_cold}/${p_back} MiB"
|
|
if [ "$pmove" -gt 64 ]; then
|
|
skip arbiter/v-vram-release "production VRAM moved by ${pmove} MiB during the measurement, cannot attribute: $vmsg"
|
|
elif [ "$added" -lt 64 ]; then
|
|
fail arbiter/v-vram-release "A added only ${added} MiB when warm (measurement broken?): $vmsg"
|
|
else
|
|
check arbiter/v-vram-release $(( v_warm - v_cold >= need ? 0 : 1 )) "$vmsg"
|
|
check arbiter/v-vram-restore $(( code == 200 && v_back - v0 >= need && $(holder_num) == A_PID ? 0 : 1 )) \
|
|
"HTTP $code, A holds the token: $([ "$(token_holder)" = "$A_PID" ] && echo yes || echo NO), restored +$((v_back - v0)) MiB (need >= ${need}), output $([ "$(json_str content "$S/v-a.json")" = "$ref_a" ] && echo identical || echo DIFFERENT)"
|
|
[ "$(json_str content "$S/v-a.json")" = "$ref_a" ] || fail arbiter/v-vram-output "output after the release/restore differs from the reference"
|
|
fi
|
|
|
|
log "arbiter: starting B (text-only) on :$PB"
|
|
start_b
|
|
wait_health "$PB" "$B_PID" 180 || { fail arbiter/start "B did not become healthy, see $S/B.log"; return; }
|
|
pass arbiter/start "A=$A_PID B=$B_PID up, reference output ${#ref_a} chars"
|
|
|
|
# ---- (a)+(b) alternating handoffs
|
|
local i port pid name bad=0 stranded=0 diff_a=0 diff_b=0 ref_b= lat=() holder_bad=0
|
|
for i in $(seq 0 30); do
|
|
if [ $((i % 2)) -eq 0 ]; then port=$PB pid=$B_PID name=B; else port=$PA pid=$A_PID name=A; fi
|
|
local t0 t1 out=$S/h$i.json
|
|
t0=$(now_ms)
|
|
code=$(complete "$port" "$out" "$(greedy_body 48 false)" 60)
|
|
t1=$(now_ms)
|
|
[ $i -gt 0 ] && lat+=($((t1 - t0)))
|
|
if [ "$code" = 000 ]; then stranded=$((stranded + 1)); continue; fi
|
|
[ "$code" != 200 ] && { bad=$((bad + 1)); continue; }
|
|
local c; c=$(json_str content "$out")
|
|
if [ $name = A ]; then [ "$c" = "$ref_a" ] || diff_a=$((diff_a + 1))
|
|
else if [ -z "$ref_b" ]; then ref_b=$c; elif [ "$c" != "$ref_b" ]; then diff_b=$((diff_b + 1)); fi; fi
|
|
[ "$(token_holder)" = "$pid" ] || holder_bad=$((holder_bad + 1))
|
|
done
|
|
local med max
|
|
med=$(printf '%s\n' "${lat[@]}" | sort -n | awk '{a[NR]=$1} END {print a[int((NR+1)/2)]}')
|
|
max=$(printf '%s\n' "${lat[@]}" | sort -n | tail -1)
|
|
check arbiter/a-handoffs $(( stranded == 0 && bad == 0 && holder_bad == 0 ? 0 : 1 )) \
|
|
"30 handoffs: $stranded stranded, $bad non-200, $holder_bad served-without-token; request latency median ${med} ms, max ${max} ms"
|
|
check arbiter/b-greedy-identical $(( diff_a == 0 && diff_b == 0 ? 0 : 1 )) \
|
|
"A: 15/15 == no-handoff reference ($diff_a differ); B: $diff_b of 15 differ from its first; A==B: $([ "$ref_a" = "$ref_b" ] && echo yes || echo no)"
|
|
|
|
# ---- (b') KV / prompt cache survives an evict + restore (LLAMA_SLEEP_EVICT_KV=1)
|
|
local p1 p2 c1 c2
|
|
code=$(complete "$PA" "$S/kv1.json" "$(greedy_body 32 true 'Explain in detail how a refrigerator moves heat from inside to outside')")
|
|
complete "$PB" "$S/kv-b.json" "$(greedy_body 8 false)" > /dev/null
|
|
[ "$(token_holder)" = "$B_PID" ] || log "warning: B does not hold the token before the KV re-check"
|
|
complete "$PA" "$S/kv2.json" "$(greedy_body 32 true 'Explain in detail how a refrigerator moves heat from inside to outside')" > /dev/null
|
|
p1=$(json_num prompt_n "$S/kv1.json"); p2=$(json_num prompt_n "$S/kv2.json")
|
|
c1=$(json_str content "$S/kv1.json"); c2=$(json_str content "$S/kv2.json")
|
|
check arbiter/b-kv-survives $(( ${p2:-99} < ${p1:-0} && ${#c1} > 0 ? 0 : 1 )) \
|
|
"prompt_n $p1 -> $p2 after evict/restore (cache reused), output $([ "$c1" = "$c2" ] && echo identical || echo DIFFERENT)"
|
|
[ "$c1" = "$c2" ] || fail arbiter/b-kv-output "cached-prompt output changed across evict/restore"
|
|
|
|
# ---- (c) image after release/restore
|
|
complete "$PB" "$S/c-b.json" "$(greedy_body 8 false)" > /dev/null # A goes cold
|
|
local cold_ok=1; [ "$(token_holder)" = "$B_PID" ] && cold_ok=0
|
|
local img; img=$(make_red_ppm)
|
|
printf '{"messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"data:image/x-portable-pixmap;base64,%s"}},{"type":"text","text":"What is the main color of this image? Answer with one word."}]}],"max_tokens":16,%s}' \
|
|
"$img" "$GREEDY" > "$S/c-body.json"
|
|
code=$(curl -sS -m 120 -o "$S/c-a.json" -w '%{http_code}' -H 'Content-Type: application/json' \
|
|
-d @"$S/c-body.json" "http://127.0.0.1:$PA/v1/chat/completions" 2>>"$S/curl.err")
|
|
local ans; ans=$(json_str content "$S/c-a.json")
|
|
check arbiter/c-image-after-restore $(( cold_ok == 0 && ${#code} > 0 && code == 200 ? 0 : 1 )) \
|
|
"A cold before: $([ $cold_ok -eq 0 ] && echo yes || echo NO), HTTP $code, answer: '$ans'"
|
|
echo "$ans" | grep -qi red || fail arbiter/c-image-answer "expected 'red', got '$ans'"
|
|
# text right after the image path (encoder out, weights back)
|
|
code=$(complete "$PA" "$S/c-a2.json" "$(greedy_body 48 false)")
|
|
[ "$code" = 200 ] && [ "$(json_str content "$S/c-a2.json")" = "$ref_a" ] \
|
|
|| fail arbiter/c-text-after-image "HTTP $code, output differs from reference after the image request"
|
|
|
|
# ---- (g) kill -9 a waiter that holds want.lock -> the holder stops yielding
|
|
stream_bg "$PA" "$S/g-a.sse" "$LONG_BODY"; local gcl=$CPID
|
|
if ! wait_tokens "$S/g-a.sse" 3 30000; then fail arbiter/g-kill9-waiter "A did not start streaming"; else
|
|
local s_before; s_before=$(sleeps "$S/A.log")
|
|
complete "$PB" "$S/g-b.json" "$(greedy_body 8 false)" 60 > /dev/null &
|
|
local gbc=$!; track "$gbc"
|
|
if ! wait_want "$B_PID" 10000; then
|
|
fail arbiter/g-kill9-waiter "B never registered on want.lock (setup)"
|
|
else
|
|
kill -KILL "$B_PID"; wait "$B_PID" 2>/dev/null
|
|
kill "$gcl" 2>/dev/null; wait "$gcl" 2>/dev/null # end A's generation (client abort)
|
|
sleep 2
|
|
local wl; wl=$(lock_pids "$ARENA/want.lock" READ | wc -l)
|
|
check arbiter/g-kill9-waiter $(( $(holder_num) == A_PID && $(sleeps "$S/A.log") == s_before && wl == 0 ? 0 : 1 )) \
|
|
"after kill -9 of waiting B: want.lock readers $wl, token holder $(token_holder) (A=$A_PID), A sleeps +$(( $(sleeps "$S/A.log") - s_before ))"
|
|
fi
|
|
wait "$gbc" 2>/dev/null
|
|
fi
|
|
log "arbiter: restarting B"
|
|
start_b
|
|
wait_health "$PB" "$B_PID" 180 || { fail arbiter/restart "B did not come back"; return; }
|
|
|
|
# ---- (e) SIGTERM while A waits for the token -> exits fast, never restores
|
|
stream_bg "$PB" "$S/e-b.sse" "$LONG_BODY"; local ecl=$CPID
|
|
wait_tokens "$S/e-b.sse" 3 30000 || { fail arbiter/e-term-waiting "B did not start streaming"; }
|
|
complete "$PA" "$S/e-a.json" "$(greedy_body 8 false)" 30 > /dev/null &
|
|
local eac=$!; track "$eac"
|
|
if wait_want "$A_PID" 10000; then
|
|
sleep 0.3
|
|
local t0 ms took_token=0
|
|
t0=$(now_ms); kill -TERM "$A_PID"
|
|
while pid_alive "$A_PID"; do
|
|
[ "$(token_holder)" = "$A_PID" ] && took_token=1
|
|
[ $(( $(now_ms) - t0 )) -gt 10000 ] && break
|
|
sleep 0.01
|
|
done
|
|
ms=$(( $(now_ms) - t0 )); wait "$A_PID"; RC=$?
|
|
check arbiter/e-term-waiting $(( RC == 0 && ms <= 1500 && took_token == 0 && $(holder_num) == B_PID ? 0 : 1 )) \
|
|
"exit $RC in ${ms} ms, A took the token: $([ $took_token = 1 ] && echo YES || echo no), holder now $(token_holder) (B=$B_PID), log: $(grep -c 'no longer waiting for the GPU token' "$S/A.log") abort line(s)"
|
|
else
|
|
fail arbiter/e-term-waiting "A never registered on want.lock (setup)"
|
|
stop_server "$A_PID"
|
|
fi
|
|
kill "$ecl" 2>/dev/null; wait "$ecl" "$eac" 2>/dev/null
|
|
log "arbiter: restarting A"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
|
|
|
|
# ---- (d) SIGTERM during A's VRAM release -> exit 0 within ~2 s, B gets the token
|
|
# A is warm (fresh load, first release also captures the host shadow: the longest release)
|
|
local how sig_ms
|
|
if command -v gdb > /dev/null; then
|
|
how=gdb
|
|
gdb -p "$A_PID" -batch -nx -ex 'set pagination off' -ex 'handle all nostop noprint pass' \
|
|
-ex 'break llama_context_release_device' -ex 'continue' \
|
|
-ex "shell date +%s%3N > $S/d.sig" -ex 'delete' -ex 'signal SIGTERM' > "$S/d-gdb.log" 2>&1 &
|
|
local gpid=$!; track "$gpid"
|
|
local k=0; until grep -q '^Continuing' "$S/d-gdb.log" || [ $k -gt 300 ]; do sleep 0.1; k=$((k + 1)); done
|
|
sleep 0.5
|
|
else
|
|
how=log-poll
|
|
fi
|
|
local s0; s0=$(sleeps "$S/A.log")
|
|
complete "$PB" "$S/d-b.json" "$(greedy_body 8 false)" 60 > "$S/d-b.code" &
|
|
local dbc=$!; track "$dbc"
|
|
if [ $how = log-poll ]; then
|
|
local k=0; until [ "$(sleeps "$S/A.log")" -gt "$s0" ] || [ $k -gt 2000 ]; do sleep 0.005; k=$((k + 1)); done
|
|
now_ms > "$S/d.sig"; kill -TERM "$A_PID"
|
|
else
|
|
local k=0; until [ -s "$S/d.sig" ] || [ $k -gt 600 ]; do sleep 0.05; k=$((k + 1)); done
|
|
fi
|
|
if [ ! -s "$S/d.sig" ]; then
|
|
fail arbiter/d-term-in-release "release breakpoint never hit ($how), see $S/d-gdb.log"
|
|
kill "$gpid" 2>/dev/null; stop_server "$A_PID"
|
|
else
|
|
while pid_alive "$A_PID" && [ $(( $(now_ms) - $(cat "$S/d.sig") )) -lt 20000 ]; do sleep 0.01; done
|
|
sig_ms=$(( $(now_ms) - $(cat "$S/d.sig") ))
|
|
wait "$A_PID"; RC=$?
|
|
wait "$dbc"
|
|
local dcode; dcode=$(cat "$S/d-b.code")
|
|
sleep 0.2
|
|
check arbiter/d-term-in-release $(( RC == 0 && sig_ms <= 2500 && dcode == 200 && $(holder_num) == B_PID ? 0 : 1 )) \
|
|
"SIGTERM inside llama_context_release_device ($how): A exit $RC in ${sig_ms} ms, B HTTP $dcode, holder $(token_holder) (B=$B_PID)"
|
|
fi
|
|
log "arbiter: restarting A"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
|
|
|
|
# ---- (f) SIGTERM mid-generation (streaming)
|
|
stream_bg "$PA" "$S/f-a.sse" "$LONG_BODY"; local fcl=$CPID
|
|
if ! wait_tokens "$S/f-a.sse" 10 30000; then fail arbiter/f-term-streaming "A did not start streaming"; else
|
|
local t0 free_ms=-1 cl_ms=-1 exit_ms=-1
|
|
t0=$(now_ms); kill -TERM "$A_PID"
|
|
while [ $(( $(now_ms) - t0 )) -lt 10000 ]; do
|
|
[ $free_ms -lt 0 ] && [ -z "$(token_holder)" ] && free_ms=$(( $(now_ms) - t0 ))
|
|
[ $cl_ms -lt 0 ] && ! pid_alive "$fcl" && cl_ms=$(( $(now_ms) - t0 ))
|
|
[ $exit_ms -lt 0 ] && ! pid_alive "$A_PID" && exit_ms=$(( $(now_ms) - t0 ))
|
|
[ $free_ms -ge 0 ] && [ $cl_ms -ge 0 ] && [ $exit_ms -ge 0 ] && break
|
|
sleep 0.01
|
|
done
|
|
wait "$A_PID"; RC=$?; wait "$fcl" 2>/dev/null
|
|
local errev; errev=$(grep -c '"error"' "$S/f-a.sse")
|
|
check arbiter/f-term-streaming $(( RC == 0 && errev > 0 && cl_ms >= 0 && free_ms >= 0 && free_ms <= 2500 && exit_ms >= 0 ? 0 : 1 )) \
|
|
"A exit $RC in ${exit_ms} ms, token free in ${free_ms} ms, client closed in ${cl_ms} ms with $errev error event(s) after $(grep -c '^data: ' "$S/f-a.sse") data lines"
|
|
fi
|
|
log "arbiter: restarting A"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
|
|
|
|
# ---- (h) clean shutdown warm and cold
|
|
[ "$(token_holder)" = "$A_PID" ] || log "warning: A is not warm before the warm-shutdown case"
|
|
stop_server "$A_PID"
|
|
check arbiter/h-shutdown-warm $(( RC == 0 ? 0 : 1 )) "warm A: exit $RC in ${EXIT_MS} ms"
|
|
start_a
|
|
wait_health "$PA" "$A_PID" 180 || { fail arbiter/restart "A did not come back"; return; }
|
|
complete "$PB" "$S/h-b.json" "$(greedy_body 8 false)" > /dev/null # A goes cold
|
|
local acold=no; [ "$(token_holder)" = "$B_PID" ] && acold=yes
|
|
stop_server "$A_PID"
|
|
check arbiter/h-shutdown-cold $([ $RC -eq 0 ] && [ $acold = yes ] && echo 0 || echo 1) \
|
|
"cold A (B holds token: $acold): exit $RC in ${EXIT_MS} ms"
|
|
stop_server "$B_PID"
|
|
check arbiter/h-shutdown-warm-b $(( RC == 0 ? 0 : 1 )) "warm B: exit $RC in ${EXIT_MS} ms"
|
|
|
|
local bad_lines; bad_lines=$(scan_logs "$S/A.log" "$S/B.log")
|
|
check arbiter/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) \
|
|
"$([ -z "$bad_lines" ] && echo 'no asserts / Vulkan errors in A/B logs' || echo "$bad_lines" | head -3 | tr '\n' ' ')"
|
|
|
|
run_handoff_ab
|
|
}
|
|
|
|
# handoff_block BUILD TAG N : a fresh pair of BUILD on the scratch arena (A with mmproj, B text-only,
|
|
# the same config as case a), one warm-up request on A, then N+1 alternating requests starting with
|
|
# B. Request 0 is not a handoff (B holds the token after its load); the latency of requests 1..N is
|
|
# appended to $S/lat-TAG.txt. HB_BAD counts setup failures, non-200s and answers without the token.
|
|
handoff_block() {
|
|
local build=$1 tag=$2 n=$3 a b i port pid code t0 t1
|
|
HB_BAD=0
|
|
: > "$S/lat-$tag.txt"
|
|
start_server "L$tag-A" "$build" "$PA" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --mmproj "$MMPROJ"; a=$SPID
|
|
if ! wait_health "$PA" "$a" 180; then HB_BAD=1; stop_server "$a"; return 1; fi
|
|
complete "$PA" "$S/L$tag-warm.json" "$(greedy_body 8 false)" > /dev/null
|
|
start_server "L$tag-B" "$build" "$PB" "${ARB_ENV[@]}" -- "${ARB_ARGS[@]}" --no-mmproj; b=$SPID
|
|
if ! wait_health "$PB" "$b" 180; then HB_BAD=1; stop_server "$b"; stop_server "$a"; return 1; fi
|
|
for i in $(seq 0 "$n"); do
|
|
if [ $((i % 2)) -eq 0 ]; then port=$PB pid=$b; else port=$PA pid=$a; fi
|
|
t0=$(now_ms)
|
|
code=$(complete "$port" "$S/L$tag-$i.json" "$(greedy_body 48 false)" 60)
|
|
t1=$(now_ms)
|
|
if [ "$code" != 200 ] || [ "$(token_holder)" != "$pid" ]; then HB_BAD=$((HB_BAD + 1)); continue; fi
|
|
[ "$i" -gt 0 ] && echo $((t1 - t0)) >> "$S/lat-$tag.txt"
|
|
done
|
|
stop_server "$b"; stop_server "$a"
|
|
}
|
|
|
|
# ---- (a') ordinary handoff latency, candidate vs reference, in one session
|
|
# Two production deploys were lost to arbiter changes that passed the pathological cases but made
|
|
# every ordinary handoff slower. Blocks run ref, cand, cand, ref (a fixed order once gave the later
|
|
# slot about 9%); each build's median is over all its handoffs. FAIL if the candidate median exceeds
|
|
# the reference median by more than max(25%, 200 ms): small absolute numbers are noisy.
|
|
run_handoff_ab() {
|
|
local n=16 blk k=0 bad_r=0 bad_c=0 tag
|
|
log "arbiter: handoff latency A/B, blocks ref,cand,cand,ref x $n handoffs"
|
|
: > "$S/lat-ref.txt"; : > "$S/lat-cand.txt"
|
|
for blk in ref cand cand ref; do
|
|
k=$((k + 1)); tag=$k-$blk
|
|
local b; [ $blk = ref ] && b=$REF || b=$BUILD
|
|
handoff_block "$b" "$tag" "$n"
|
|
if [ $blk = ref ]; then bad_r=$((bad_r + HB_BAD)); else bad_c=$((bad_c + HB_BAD)); fi
|
|
cat "$S/lat-$tag.txt" >> "$S/lat-$blk.txt"
|
|
log "arbiter: block $tag median $(median < "$S/lat-$tag.txt") ms over $(wc -l < "$S/lat-$tag.txt") handoffs, $HB_BAD bad"
|
|
done
|
|
local mr mc nr nc maxr maxc limit
|
|
mr=$(median < "$S/lat-ref.txt"); mc=$(median < "$S/lat-cand.txt")
|
|
nr=$(wc -l < "$S/lat-ref.txt"); nc=$(wc -l < "$S/lat-cand.txt")
|
|
maxr=$(sort -n "$S/lat-ref.txt" | tail -1); maxc=$(sort -n "$S/lat-cand.txt" | tail -1)
|
|
if [ -z "$mr" ] || [ -z "$mc" ] || [ $bad_r -gt 0 ] || [ $bad_c -gt 0 ]; then
|
|
fail arbiter/a-handoff-latency "blocks incomplete: reference $nr handoffs ($bad_r bad), candidate $nc ($bad_c bad); see $S/L*-*.log"
|
|
return
|
|
fi
|
|
limit=$(awk -v r="$mr" 'BEGIN {d = r * 0.25; if (d < 200) d = 200; printf "%.0f", r + d}')
|
|
check arbiter/a-handoff-latency "$(awk -v c="$mc" -v l="$limit" 'BEGIN {print (c > l) ? 1 : 0}')" \
|
|
"median handoff latency candidate ${mc} ms (n $nc, max $maxc) vs reference ${mr} ms (n $nr, max $maxr); limit ${limit} ms = ref + max(25%, 200 ms)"
|
|
local bad_lines; bad_lines=$(scan_logs "$S"/L*-A.log "$S"/L*-B.log)
|
|
[ -z "$bad_lines" ] || fail arbiter/a-handoff-latency-logs "$(echo "$bad_lines" | head -3 | tr '\n' ' ')"
|
|
}
|
|
|
|
# ================================================================ 4. perf
|
|
vram_sampler() { # OUTFILE : max of mem_info_vram_used every 0.25 s, written continuously
|
|
local m=0 v
|
|
while :; do v=$(cat "$VRAM_USED"); [ "$v" -gt "$m" ] && { m=$v; echo "$m" > "$1"; }; sleep 0.25; done
|
|
}
|
|
# median of the numbers on stdin (one per line); empty input -> empty output
|
|
median() { sort -g | awk '{a[NR] = $1} END {if (NR) printf "%.2f", (NR % 2) ? a[(NR + 1) / 2] : (a[NR / 2] + a[NR / 2 + 1]) / 2}'; }
|
|
bench_samples() { # JSONL (llama-bench -o jsonl) -> "test<TAB>t/s", one row per repetition sample
|
|
sed -n 's/.*"n_prompt": *\([0-9]*\),.*"n_gen": *\([0-9]*\),.*"n_depth": *\([0-9]*\),.*"samples_ts": *\[\([^]]*\)\].*/\1 \2 \3 \4/p' "$1" |
|
|
awk '{t = ($1 > 0) ? "pp" $1 : "tg" $2; if ($3 > 0) t = t " @ d" $3
|
|
for (i = 4; i <= NF; i++) {v = $i; gsub(/,/, "", v); if (v != "") print t "\t" v}}'
|
|
}
|
|
run_perf() {
|
|
gpu_exclusive perf || return
|
|
[ -f "$PERF_MODEL" ] || { fail perf "model missing: $PERF_MODEL"; return; }
|
|
[ -x "$REF/bin/llama-bench" ] && [ -x "$BUILD/bin/llama-bench" ] || { fail perf "llama-bench missing"; return; }
|
|
local common=(-m "$PERF_MODEL" -t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -o jsonl -oe md)
|
|
local arm k=0 base; base=$(vram_mb)
|
|
log "perf: idle VRAM $base MiB; arms ref,cand,cand,ref"
|
|
for arm in ref cand cand ref; do
|
|
k=$((k + 1))
|
|
local b; [ $arm = ref ] && b=$REF || b=$BUILD
|
|
local tag=$k-$arm
|
|
vram_sampler "$S/perf-$tag.vram" & local vs=$!; track "$vs"
|
|
# stdout: one JSON line per test with every repetition sample; stderr: the markdown table + logs
|
|
log "perf: arm $tag pp sweep"
|
|
LD_LIBRARY_PATH="$b/bin" "$b/bin/llama-bench" "${common[@]}" -p 2048 -n 0 -d 0,16384,32768 -r 2 \
|
|
> "$S/perf-$tag-pp.jsonl" 2> "$S/perf-$tag-pp.log"
|
|
log "perf: arm $tag tg"
|
|
LD_LIBRARY_PATH="$b/bin" "$b/bin/llama-bench" "${common[@]}" -p 0 -n 256 -d 4096 -r 3 \
|
|
> "$S/perf-$tag-tg.jsonl" 2> "$S/perf-$tag-tg.log"
|
|
kill "$vs" 2>/dev/null; wait "$vs" 2>/dev/null
|
|
{ bench_samples "$S/perf-$tag-pp.jsonl"; bench_samples "$S/perf-$tag-tg.jsonl"; } | sed "s/^/$arm\t/" >> "$S/perf.tsv"
|
|
echo "$arm $(cat "$S/perf-$tag.vram" 2>/dev/null || echo 0)" >> "$S/perf-vram.txt"
|
|
done
|
|
perf_report "$S/perf.tsv"
|
|
local total; total=$(cat "$VRAM_TOTAL")
|
|
local vr vc
|
|
vr=$(awk '$1=="ref" && $2>m {m=$2} END {print m+0}' "$S/perf-vram.txt")
|
|
vc=$(awk '$1=="cand" && $2>m {m=$2} END {print m+0}' "$S/perf-vram.txt")
|
|
check perf/peak-vram $(( vc * 100 <= total * 95 ? 0 : 1 )) \
|
|
"peak VRAM reference $((vr / 1048576)) MiB, candidate $((vc / 1048576)) MiB of $((total / 1048576)) MiB (limit 95%)"
|
|
}
|
|
# perf_report TSV : "arm<TAB>test<TAB>t/s" rows, one per llama-bench repetition sample (arm = ref|cand).
|
|
# Gates on the MEDIAN of all samples of a build (pooled over its arms): a single catastrophic
|
|
# sample (CPU contention from the other pods on the node) moves a mean, not a median.
|
|
perf_report() {
|
|
local tsv=$1
|
|
echo
|
|
echo "perf: median t/s over every repetition sample of both arms (order ref,cand,cand,ref)"
|
|
printf '%-18s %24s %24s %8s\n' test "reference med (min-max, n)" "candidate med (min-max, n)" delta
|
|
local tests t r c d rs cs
|
|
tests=$(awk -F'\t' '{print $2}' "$tsv" | awk '!s[$0]++')
|
|
local IFS_=$IFS; IFS=$'\n'
|
|
for t in $tests; do
|
|
r=$(awk -F'\t' -v t="$t" '$1=="ref" && $2==t {print $3}' "$tsv" | median)
|
|
c=$(awk -F'\t' -v t="$t" '$1=="cand" && $2==t {print $3}' "$tsv" | median)
|
|
rs=$(awk -F'\t' -v t="$t" '$1=="ref" && $2==t {n++; if (n == 1 || $3 < lo) lo = $3; if (n == 1 || $3 > hi) hi = $3} END {if (n) printf "%.1f-%.1f, %d", lo, hi, n}' "$tsv")
|
|
cs=$(awk -F'\t' -v t="$t" '$1=="cand" && $2==t {n++; if (n == 1 || $3 < lo) lo = $3; if (n == 1 || $3 > hi) hi = $3} END {if (n) printf "%.1f-%.1f, %d", lo, hi, n}' "$tsv")
|
|
d=$(awk -v r="$r" -v c="$c" 'BEGIN {if (r > 0) printf "%+.1f%%", (c/r - 1)*100}')
|
|
printf '%-18s %24s %24s %8s\n' "$t" "$r ($rs)" "$c ($cs)" "$d"
|
|
case "$t" in pp*)
|
|
if [ -z "$r" ] || [ -z "$c" ] || [ "$(awk -v r="$r" -v c="$c" 'BEGIN {print (c < 0.95*r)}')" = 1 ]; then
|
|
fail "perf/$t" "candidate median $c vs reference median $r t/s ($d, limit -5%)"
|
|
else
|
|
pass "perf/$t" "candidate median $c vs reference median $r t/s ($d)"
|
|
fi ;;
|
|
tg*) RESULTS+=("INFO perf/$t: candidate median $c vs reference median $r t/s ($d; decode is CPU-bound and noisy here, not gated)") ;;
|
|
esac
|
|
done
|
|
IFS=$IFS_
|
|
local n_tests; n_tests=$(echo "$tests" | grep -c '^pp')
|
|
[ "$n_tests" -ge 3 ] || fail perf/rows "expected 3 pp rows, got $n_tests (see $S/perf-*-pp.log)"
|
|
}
|
|
|
|
# ================================================================ 5. MoE cache release/restore smoke
|
|
run_moecache() {
|
|
[ -f "$MOECACHE_MODEL" ] || { fail moecache "model missing: $MOECACHE_MODEL"; return; }
|
|
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
|
|
log "moecache: $free_mb MiB VRAM free; server on :$PC with --moe-cache-mib $MOECACHE_MIB"
|
|
local arena=$S/arena-moec; mkdir -p "$arena"
|
|
# -lv 4: the cache size and hit/miss stats are only logged at that verbosity
|
|
# shellcheck disable=SC2086
|
|
start_server M "$BUILD" "$PC" LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$arena" LLAMA_SLEEP_EVICT_KV=1 -- \
|
|
-m "$MOECACHE_MODEL" $MOECACHE_ARGS -fa 1 -lm none --parallel 1 --no-warmup --reasoning off -lv 4 \
|
|
--moe-cache-mib "$MOECACHE_MIB"
|
|
local mp=$SPID
|
|
wait_health "$PC" "$mp" 600 || { fail moecache "server did not start, see $S/M.log"; return; }
|
|
# chat endpoint: an instruct model fed a raw prompt can loop, which says nothing about the cache
|
|
local body="{\"messages\":[{\"role\":\"user\",\"content\":\"What are the three primary colors of light? Answer in one short sentence.\"}],\"max_tokens\":64,$GREEDY}"
|
|
local outs=() i code w
|
|
for i in 0 1 2; do
|
|
code=$(curl -sS -m 300 -o "$S/m$i.json" -w '%{http_code}' -H 'Content-Type: application/json' \
|
|
-d "$body" "http://127.0.0.1:$PC/v1/chat/completions" 2>>"$S/curl.err")
|
|
outs+=("$(json_str content "$S/m$i.json")")
|
|
[ "$code" = 200 ] || { fail moecache "request $i: HTTP $code"; stop_server "$mp"; return; }
|
|
[ $i -eq 2 ] && break
|
|
# force a release: a fake waiter holds want.lock shared until the server drops the token
|
|
( exec 9<"$arena/want.lock"; flock -s 9; exec sleep 120 ) & w=$!; track "$w"
|
|
local k=0; until [ -z "$(awk -v id="$(lock_id "$arena/token.lock")" '$2=="FLOCK" && $4=="WRITE" && $6==id' /proc/locks)" ] || [ $k -gt 600 ]; do sleep 0.05; k=$((k + 1)); done
|
|
log "moecache: cycle $((i + 1)) cold, VRAM $(vram_mb) MiB"
|
|
kill "$w"; wait "$w" 2>/dev/null
|
|
done
|
|
stop_server "$mp"
|
|
local size hits rel res
|
|
size=$(sed -n 's/.*MoE cache size = *\([0-9.]*\) MiB.*/\1/p' "$S/M.log" | head -1)
|
|
hits=$(sed -n 's/.*llama_moe_cache: .* hits = \([0-9]*\),.*/\1/p' "$S/M.log" | awk '{s += $1} END {print s+0}')
|
|
rel=$(grep -c 'released .* MiB of MoE cache' "$S/M.log"); res=$(grep -c 'restored .* MiB of MoE cache' "$S/M.log")
|
|
check moecache/active $(( hits > 0 ? 0 : 1 )) \
|
|
"cache ${size:-?} MiB, $hits hits (0 hits = cache too small to ever reuse an expert, so a stale SLRU would go unnoticed), released ${rel}x restored ${res}x"
|
|
local same=1; [ "${outs[0]}" = "${outs[1]}" ] && [ "${outs[1]}" = "${outs[2]}" ] && [ -n "${outs[0]}" ] && same=0
|
|
check moecache/identical $same "3 greedy answers across 2 cold/warm cycles $([ $same = 0 ] && echo identical || echo DIFFER): '$(echo "${outs[0]}" | cut -c1-80)'"
|
|
local coh=1; echo "${outs[2]}" | grep -qi red && echo "${outs[2]}" | grep -qi green && echo "${outs[2]}" | grep -qi blue && coh=0
|
|
check moecache/coherent $coh "answer after the 2nd restore names red, green and blue"
|
|
check moecache/shutdown $(( RC == 0 ? 0 : 1 )) "exit $RC"
|
|
local bad_lines; bad_lines=$(scan_logs "$S/M.log")
|
|
check moecache/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) "${bad_lines:-clean}"
|
|
}
|
|
|
|
# ================================================================ 6. MoE routing-ids readback skip (F1) exactness
|
|
# sched_copy_experts skips the ids readback when a split's ids tensor has n_tokens * n_expert_used >=
|
|
# 4 * n_expert entries, i.e. for any ubatch of at least ceil(4 * n_expert / n_expert_used) tokens
|
|
# (128 on a 256-expert / 8-used model). Method: the model with every MoE layer's experts on the host
|
|
# (so the expert-copy callback runs), greedy, one prompt of >= 2x that many tokens in a single
|
|
# ubatch: the prefill takes the skip and every decode step takes the readback. The candidate's
|
|
# output must be byte-identical to the reference's and, when the build has the LLAMA_MOE_IDS_SKIP
|
|
# knob, to the candidate with the skip forced off (=0), which proves the skip itself is exact even
|
|
# when the reference build carries it too; =log then shows the skip engaged.
|
|
gguf_u32() { # FILE KEY-SUFFIX -> uint32 value of the first GGUF metadata key ending in KEY-SUFFIX
|
|
local off t
|
|
off=$(head -c 67108864 "$1" | grep -abo -m1 -F "$2" | head -1 | cut -d: -f1)
|
|
[ -n "$off" ] || return 1
|
|
off=$((off + ${#2})) # key string -> uint32 value type -> value
|
|
t=$(od -An -tu4 -j "$off" -N4 "$1" | tr -d ' ')
|
|
[ "$t" = 4 ] || [ "$t" = 5 ] || return 1 # GGUF_TYPE_UINT32 / INT32
|
|
od -An -tu4 -j $((off + 4)) -N4 "$1" | tr -d ' '
|
|
}
|
|
gpu_exclusive() { # CHECK-NAME -> FAILs it and returns 1 unless nothing else can use the GPU
|
|
[ "${FV_TEST_SHARED_GPU:-}" = 1 ] && { log "FV_TEST_SHARED_GPU=1: skipping the exclusive-GPU guard (script testing only)"; return 0; }
|
|
if supervisorctl -c /root/supervisord.conf status llama-swap 2>/dev/null | grep -q RUNNING; then
|
|
fail "$1" "llama-swap is RUNNING; run under /root/bench/moe/swapdown.sh"; return 1
|
|
fi
|
|
local others; others=$(ps -eo pid=,comm= | awk '$2 ~ /^llama-(server|bench|cli)$/' | wc -l)
|
|
[ "$others" -eq 0 ] || { fail "$1" "$others other llama processes are running"; return 1; }
|
|
}
|
|
moex_prompt() { # N_LINES -> a deterministic prose-like prompt (JSON-safe: no quotes or backslashes)
|
|
local i
|
|
for i in $(seq 1 "$1"); do
|
|
printf 'Entry %d: the river town of Station%d ships %d barrels of grain and %d crates of salt each week to the harbour, and its council meets on day %d of the month. ' \
|
|
"$i" "$i" $(( (i * 37) % 101 + 3 )) $(( (i * 53) % 89 + 2 )) $(( (i * 7) % 28 + 1 ))
|
|
done
|
|
printf 'Question: which town ships the most barrels of grain each week? Answer:'
|
|
}
|
|
run_moe_exact() {
|
|
gpu_exclusive moe-exact || return
|
|
[ -f "$MOEX_MODEL" ] || { fail moe-exact "model missing: $MOEX_MODEL"; return; }
|
|
local n_exp n_used
|
|
n_exp=$(gguf_u32 "$MOEX_MODEL" .expert_count); n_used=$(gguf_u32 "$MOEX_MODEL" .expert_used_count)
|
|
if [ -z "$n_exp" ] || [ -z "$n_used" ] || [ "$n_exp" -eq 0 ] || [ "$n_used" -eq 0 ]; then
|
|
fail moe-exact "no expert_count / expert_used_count in $MOEX_MODEL (not a MoE model?)"; return
|
|
fi
|
|
local thr=$(( (4 * n_exp + n_used - 1) / n_used )) # tokens per ubatch where the skip starts
|
|
local lines=$(( (2 * thr + 39) / 40 + 2 )) # ~45 tokens per line -> >= 2x the threshold
|
|
local body
|
|
body="{\"prompt\":\"$(moex_prompt "$lines")\",\"n_predict\":64,$GREEDY,\"cache_prompt\":false}"
|
|
local knob=0; grep -qa LLAMA_MOE_IDS_SKIP "$BUILD/bin/libllama.so" 2>/dev/null && knob=1
|
|
log "moe-exact: n_expert $n_exp, n_expert_used $n_used -> skip at >= $thr tokens per ubatch; prompt of $lines lines; knob in candidate: $([ $knob = 1 ] && echo yes || echo no)"
|
|
|
|
# arm NAME BUILD ENV... : one server, one request; sets MX_CONTENT / MX_PN, returns 1 on failure
|
|
mx_arm() {
|
|
local name=$1 b=$2; shift 2
|
|
# shellcheck disable=SC2086
|
|
start_server "$name" "$b" "$PC" "$@" -- -m "$MOEX_MODEL" $MOEX_ARGS -fa 1 -lm none --parallel 1 --no-warmup
|
|
local p=$SPID code
|
|
if ! wait_health "$PC" "$p" 900; then fail "moe-exact/$name" "server did not start, see $S/$name.log"; stop_server "$p"; return 1; fi
|
|
code=$(complete "$PC" "$S/$name.json" "$body" 600)
|
|
stop_server "$p"
|
|
MX_CONTENT=$(json_str content "$S/$name.json"); MX_PN=$(json_num prompt_n "$S/$name.json")
|
|
[ "$code" = 200 ] && [ -n "$MX_CONTENT" ] || { fail "moe-exact/$name" "HTTP $code, empty output; see $S/$name.log"; return 1; }
|
|
[ "$RC" -eq 0 ] || log "warning: $name exited $RC"
|
|
}
|
|
local c_cand c_ref c_off pn
|
|
if [ $knob = 1 ]; then mx_arm X-cand "$BUILD" LLAMA_MOE_IDS_SKIP=log || return
|
|
else mx_arm X-cand "$BUILD" || return; fi
|
|
c_cand=$MX_CONTENT; pn=${MX_PN:-0}
|
|
if [ "$pn" -lt "$thr" ]; then
|
|
fail moe-exact/prompt "prompt_n $pn < threshold $thr: the skip never ran, nothing was tested"; return
|
|
fi
|
|
mx_arm X-ref "$REF" || return
|
|
c_ref=$MX_CONTENT
|
|
check moe-exact/identical-ref $([ "$c_cand" = "$c_ref" ] && echo 0 || echo 1) \
|
|
"prompt_n $pn (>= $thr), 64 greedy tokens: candidate $([ "$c_cand" = "$c_ref" ] && echo == || echo '!=') reference ('$(echo "$c_cand" | cut -c1-60)')"
|
|
if [ $knob = 1 ]; then
|
|
mx_arm X-cand-noskip "$BUILD" LLAMA_MOE_IDS_SKIP=0 || return
|
|
c_off=$MX_CONTENT
|
|
check moe-exact/identical-noskip $([ "$c_cand" = "$c_off" ] && echo 0 || echo 1) \
|
|
"candidate with the skip (LLAMA_MOE_IDS_SKIP=log) $([ "$c_cand" = "$c_off" ] && echo == || echo '!=') candidate with the readback forced (=0)"
|
|
local n_skip n_small
|
|
n_skip=$(grep -c 'sched_copy_experts: ids readback skipped' "$S/X-cand.log")
|
|
n_small=$(sed -n 's/.*ids readback skipped, n_ids=\([0-9]*\) n_expert=\([0-9]*\).*/\1 \2/p' "$S/X-cand.log" | awk '$1 < 4 * $2' | wc -l)
|
|
check moe-exact/skip-engaged $(( n_skip > 0 && n_small == 0 ? 0 : 1 )) \
|
|
"$n_skip skips logged ($n_small below 4*n_expert), $(grep -c 'ids readback skipped' "$S/X-cand-noskip.log") with the skip forced off"
|
|
else
|
|
skip moe-exact/skip-engaged "candidate has no LLAMA_MOE_IDS_SKIP knob to observe it; by construction the prefill ubatch had $pn x $n_used = $((pn * n_used)) ids >= 4 x $n_exp, so the skip branch ran"
|
|
fi
|
|
local bad_lines; bad_lines=$(scan_logs "$S"/X-*.log)
|
|
check moe-exact/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) "${bad_lines:-clean}"
|
|
}
|
|
|
|
# ================================================================ 7. disk prompt cache
|
|
# --cache-disk-path: on shutdown each slot's state goes to a file; after a restart a prompt that starts
|
|
# with a saved prompt (or with one of its checkpoints, for a hybrid model) is restored from the file.
|
|
# Method: chat requests on the small hybrid model, greedy. A reference server that never restarts
|
|
# gives turn 2 (turn 1 + answer + a new question); the candidate saves turn 1 on SIGTERM, restarts
|
|
# and must give the same answer with the same prompt_n (only the suffix processed). CPU only
|
|
# (-dev none -ngl 0, no VRAM), plus the GPU path (-ngl 99, arbiter env, warm and cold) if VRAM allows.
|
|
DC_ARGS=(-m "$SMALL" -c 4096 -dev none -ngl 0 -fa 1 -t 4 --parallel 1 --no-warmup --reasoning off)
|
|
DC_MM=(--no-mmproj) # the mmproj case sets --mmproj
|
|
DC_GPU=(-dev Vulkan0 -ngl 99)
|
|
DC_ENV=(LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$ARENA" LLAMA_SLEEP_EVICT_KV=1)
|
|
DC_PID=
|
|
dc_chat() { # PORT OUT MESSAGES-JSON -> HTTP code
|
|
curl -sS -m 300 -o "$2" -w '%{http_code}' -H 'Content-Type: application/json' \
|
|
-d "{\"messages\":$3,\"max_tokens\":32,$GREEDY}" "http://127.0.0.1:$1/v1/chat/completions" 2>>"$S/curl.err" || true
|
|
}
|
|
dc_msgs() { # USER1 [ANSWER1 USER2] -> messages JSON
|
|
if [ $# -eq 1 ]; then echo "[{\"role\":\"user\",\"content\":\"$1\"}]"
|
|
else echo "[{\"role\":\"user\",\"content\":\"$1\"},{\"role\":\"assistant\",\"content\":\"$2\"},{\"role\":\"user\",\"content\":\"$3\"}]"; fi
|
|
}
|
|
# dc_start NAME DIR ENV... -- ARGS... : DIR "" = no disk cache; sets DC_PID, returns 1 if not healthy
|
|
dc_start() {
|
|
local name=$1 dir=$2; shift 2
|
|
local envs=(); while [ "$1" != -- ]; do envs+=("$1"); shift; done; shift
|
|
start_server "$name" "$BUILD" "$PC" "${envs[@]}" -- "${DC_ARGS[@]}" "${DC_MM[@]}" ${dir:+--cache-disk-path "$dir"} "$@"
|
|
DC_PID=$SPID
|
|
wait_health "$PC" "$DC_PID" 180
|
|
}
|
|
# dc_term : SIGTERM DC_PID; sets RC, DC_EXIT_MS and DC_FREE_MS (token.lock free, -1 if never held)
|
|
dc_term() {
|
|
local t0; t0=$(now_ms); DC_FREE_MS=-1; DC_EXIT_MS=-1
|
|
local held=0; [ -n "$(token_holder)" ] && held=1
|
|
kill -TERM "$DC_PID"
|
|
while [ $(( $(now_ms) - t0 )) -lt 30000 ]; do
|
|
[ $held = 1 ] && [ $DC_FREE_MS -lt 0 ] && [ -z "$(token_holder)" ] && DC_FREE_MS=$(( $(now_ms) - t0 ))
|
|
pid_alive "$DC_PID" || { DC_EXIT_MS=$(( $(now_ms) - t0 )); break; }
|
|
sleep 0.01
|
|
done
|
|
pid_alive "$DC_PID" && kill -KILL "$DC_PID"
|
|
wait "$DC_PID" 2>/dev/null; RC=$?
|
|
}
|
|
# dc_ref TAG ARGS... : turn 1 and turn 2 on one server that never restarts -> DC_C1, DC_REF2, DC_PN_REF, DC_N2, DC_MS_FULL
|
|
dc_ref() {
|
|
local tag=$1; shift
|
|
dc_start "$tag" "" -- "$@" || { fail "diskcache/$tag" "reference server did not start, see $S/$tag.log"; return 1; }
|
|
local code; code=$(dc_chat "$PC" "$S/$tag-1.json" "$(dc_msgs "$DC_U1")")
|
|
DC_C1=$(json_str content "$S/$tag-1.json")
|
|
[ "$code" = 200 ] && [ -n "$DC_C1" ] || { fail "diskcache/$tag" "turn 1: HTTP $code"; stop_server "$DC_PID"; return 1; }
|
|
DC_M2=$(dc_msgs "$DC_U1" "$DC_C1" "$DC_U2")
|
|
code=$(dc_chat "$PC" "$S/$tag-2.json" "$DC_M2")
|
|
DC_REF2=$(json_str content "$S/$tag-2.json"); DC_PN_REF=$(json_num prompt_n "$S/$tag-2.json"); DC_N2=$(json_num prompt_tokens "$S/$tag-2.json")
|
|
stop_server "$DC_PID"
|
|
# the same turn 2 on a fresh server: the full prefill the cache saves
|
|
dc_start "$tag-full" "" -- "$@" && dc_chat "$PC" "$S/$tag-full.json" "$DC_M2" > /dev/null
|
|
DC_MS_FULL=$(sed -n 's/.*"prompt_ms":\([0-9.]*\).*/\1/p' "$S/$tag-full.json" | head -1)
|
|
stop_server "$DC_PID"
|
|
[ "$code" = 200 ] && [ -n "$DC_REF2" ] && [ "${DC_PN_REF:-0}" -gt 0 ] || { fail "diskcache/$tag" "turn 2: HTTP $code"; return 1; }
|
|
log "diskcache: reference $tag: turn 2 prompt_n $DC_PN_REF of $DC_N2 tokens, a full prefill of turn 2 took ${DC_MS_FULL} ms"
|
|
}
|
|
# dc_restore TAG DIR COLD ENV... -- ARGS... : turn 1, SIGTERM (COLD=1: forced cold by a fake waiter first),
|
|
# restart, turn 2 -> restored from disk, prompt_n == reference, output == reference
|
|
dc_restore() {
|
|
local tag=$1 dir=$2 cold=$3; shift 3
|
|
local envs=(); while [ "$1" != -- ]; do envs+=("$1"); shift; done; shift
|
|
rm -rf "$dir"
|
|
dc_start "$tag-a" "$dir" "${envs[@]}" -- "$@" || { fail "diskcache/$tag" "server did not start, see $S/$tag-a.log"; return 1; }
|
|
local code c1 w=; code=$(dc_chat "$PC" "$S/$tag-a.json" "$(dc_msgs "$DC_U1")"); c1=$(json_str content "$S/$tag-a.json")
|
|
if [ "$cold" = 1 ]; then
|
|
( exec 9<"$ARENA/want.lock"; flock -s 9; exec sleep 120 ) & w=$!; track "$w"
|
|
wait_holder "" 10000 > /dev/null || log "warning: $tag-a did not go cold"
|
|
fi
|
|
local was_cold=no; [ "$cold" = 1 ] && [ -z "$(token_holder)" ] && grep -q 'server entering sleeping state' "$S/$tag-a.log" && was_cold=yes
|
|
dc_term
|
|
[ -n "$w" ] && { kill "$w" 2>/dev/null; wait "$w" 2>/dev/null; }
|
|
local rc1=$RC exit1=$DC_EXIT_MS free1=$DC_FREE_MS nf
|
|
nf=$(find "$dir" -name '*.lpcd' 2>/dev/null | wc -l)
|
|
local f1; f1=$(find "$dir" -name '*.lpcd' 2>/dev/null | head -1); [ -n "$f1" ] && cp "$f1" "$S/$tag-turn1.lpcd"
|
|
dc_start "$tag-b" "$dir" "${envs[@]}" -- "$@" || { fail "diskcache/$tag" "restart failed, see $S/$tag-b.log"; return 1; }
|
|
code=$(dc_chat "$PC" "$S/$tag-b.json" "$DC_M2")
|
|
local c2 pn ms nres; c2=$(json_str content "$S/$tag-b.json"); pn=$(json_num prompt_n "$S/$tag-b.json")
|
|
ms=$(sed -n 's/.*"prompt_ms":\([0-9.]*\).*/\1/p' "$S/$tag-b.json" | head -1)
|
|
nres=$(grep -c 'disk cache: restored' "$S/$tag-b.log")
|
|
local ok=1
|
|
[ "$code" = 200 ] && [ "$rc1" -eq 0 ] && [ "$nf" -eq 1 ] && [ "$nres" -eq 1 ] && [ "$c1" = "$DC_C1" ] && \
|
|
[ "${pn:-0}" = "$DC_PN_REF" ] && [ "${pn:-0}" -lt "${DC_N2:-0}" ] && [ "$c2" = "$DC_REF2" ] && ok=0
|
|
[ "$cold" = 1 ] && [ $was_cold != yes ] && ok=1
|
|
check "diskcache/$tag" $ok \
|
|
"$([ "$cold" = 1 ] && echo "cold before SIGTERM: $was_cold, ")exit $rc1 in ${exit1} ms$([ "$free1" -ge 0 ] && echo ", token free in ${free1} ms"), $nf file(s); restart: $nres restore(s), turn 2 prompt_n $pn (reference $DC_PN_REF) of $DC_N2 tokens in ${ms} ms (full prefill ${DC_MS_FULL} ms), output $([ "$c2" = "$DC_REF2" ] && echo == || echo '!=') reference"
|
|
log "diskcache: $tag: $(grep -h 'disk cache: \(copied\|saved\)' "$S/$tag-a.log" | sed 's/.*disk cache: //' | tr '\n' ';') $(grep -h 'disk cache: restored' "$S/$tag-b.log" | sed 's/.*disk cache: //')"
|
|
stop_server "$DC_PID"
|
|
}
|
|
run_diskcache() {
|
|
[ -f "$SMALL" ] || { fail diskcache "small model missing: $SMALL"; return; }
|
|
DC_U1="$(moex_prompt 12)"
|
|
DC_U2="Which town ships the fewest crates of salt each week?"
|
|
dc_ref dc-ref || return
|
|
|
|
# (1) save on SIGTERM, restore after a restart: plain, arbiter env warm, arbiter env cold
|
|
dc_restore restore "$S/dc" 0 -- || return
|
|
[ -f "$S/restore-turn1.lpcd" ] || { fail diskcache "no file saved on SIGTERM, see $S/restore-a.log"; return; }
|
|
dc_restore restore-arbiter "$S/dc-arb" 0 "${DC_ENV[@]}" --
|
|
dc_restore restore-cold "$S/dc-cold" 1 "${DC_ENV[@]}" --
|
|
# shutdown cost: the same warm arbiter shutdown without the cache
|
|
rm -rf "$S/dc-none"
|
|
dc_start dc-none "" "${DC_ENV[@]}" -- && dc_chat "$PC" "$S/dc-none.json" "$(dc_msgs "$DC_U1")" > /dev/null
|
|
dc_term
|
|
log "diskcache: warm arbiter shutdown without the cache: exit $RC in ${DC_EXIT_MS} ms, token free in ${DC_FREE_MS} ms"
|
|
|
|
# (2) no restore for a prompt that does not start with a saved one, or under another key
|
|
local code pn n nres
|
|
dc_start dc-np "$S/dc" -- || { fail diskcache/start "see $S/dc-np.log"; return; }
|
|
code=$(dc_chat "$PC" "$S/dc-np.json" "$(dc_msgs "Variant B. $DC_U1")")
|
|
pn=$(json_num prompt_n "$S/dc-np.json"); n=$(json_num prompt_tokens "$S/dc-np.json"); nres=$(grep -c 'disk cache: restored' "$S/dc-np.log")
|
|
check diskcache/no-prefix $(( code == 200 && nres == 0 && pn == n ? 0 : 1 )) "other conversation: HTTP $code, $nres restore(s), prompt_n $pn of $n"
|
|
stop_server "$DC_PID"
|
|
local tag args
|
|
ln -sf "$SMALL" "$S/other-path.gguf"
|
|
for tag in key-ctx key-ctk key-model; do
|
|
case $tag in
|
|
key-ctx) args=(-c 8192) ;;
|
|
key-ctk) args=(-ctk q8_0) ;;
|
|
key-model) args=(-m "$S/other-path.gguf") ;;
|
|
esac
|
|
rm -rf "$S/dc-$tag"; mkdir -p "$S/dc-$tag"; cp "$S/restore-turn1.lpcd" "$S/dc-$tag/"
|
|
dc_start "dc-$tag" "$S/dc-$tag" -- "${args[@]}" || { fail "diskcache/$tag" "see $S/dc-$tag.log"; continue; }
|
|
code=$(dc_chat "$PC" "$S/dc-$tag.json" "$DC_M2")
|
|
pn=$(json_num prompt_n "$S/dc-$tag.json"); n=$(json_num prompt_tokens "$S/dc-$tag.json"); nres=$(grep -c 'disk cache: restored' "$S/dc-$tag.log")
|
|
check "diskcache/$tag" $(( code == 200 && nres == 0 && pn == n ? 0 : 1 )) \
|
|
"${args[*]}: HTTP $code, $nres restore(s), prompt_n $pn of $n; index: $(grep -h 'disk cache: .*prompts of this model' "$S/dc-$tag.log" | sed 's/.*, \([0-9]* prompts\)/\1/')"
|
|
stop_server "$DC_PID"
|
|
done
|
|
|
|
# (3) corrupt and truncated files are removed, never used
|
|
local size; size=$(stat -c %s "$S/restore-turn1.lpcd")
|
|
rm -rf "$S/dc-bad"; mkdir -p "$S/dc-bad"
|
|
head -c $((size / 2)) "$S/restore-turn1.lpcd" > "$S/dc-bad/trunc.lpcd"
|
|
cp "$S/restore-turn1.lpcd" "$S/dc-bad/flip.lpcd"
|
|
printf '\377' | dd of="$S/dc-bad/flip.lpcd" bs=1 seek=$((size / 2)) conv=notrunc 2>/dev/null
|
|
dc_start dc-bad "$S/dc-bad" -- || { fail diskcache/corrupt "see $S/dc-bad.log"; return; }
|
|
local trunc_gone=no; [ -e "$S/dc-bad/trunc.lpcd" ] || trunc_gone=yes
|
|
code=$(dc_chat "$PC" "$S/dc-bad.json" "$DC_M2")
|
|
pn=$(json_num prompt_n "$S/dc-bad.json"); nres=$(grep -c 'disk cache: restored' "$S/dc-bad.log")
|
|
local flip_gone=no; [ -e "$S/dc-bad/flip.lpcd" ] || flip_gone=yes
|
|
check diskcache/corrupt $([ $trunc_gone = yes ] && [ $flip_gone = yes ] && [ "$code" = 200 ] && [ "$nres" -eq 0 ] && [ "$pn" = "$DC_N2" ] && [ "$(json_str content "$S/dc-bad.json")" = "$DC_REF2" ] && echo 0 || echo 1) \
|
|
"truncated file removed at startup: $trunc_gone, byte-flipped file removed at load: $flip_gone, $nres restore(s), prompt_n $pn of $DC_N2, output $([ "$(json_str content "$S/dc-bad.json")" = "$DC_REF2" ] && echo == || echo '!=') reference"
|
|
stop_server "$DC_PID"
|
|
|
|
# (3') LRU size limit: room for one file only, the second save evicts the first
|
|
local cap=$(( size * 3 / 2 / 1048576 + 1 )) files total
|
|
rm -rf "$S/dc-lru"
|
|
dc_start dc-lru1 "$S/dc-lru" -- --cache-disk-size "$cap" && dc_chat "$PC" "$S/dc-lru1.json" "$(dc_msgs "$DC_U1")" > /dev/null
|
|
stop_server "$DC_PID"
|
|
dc_start dc-lru2 "$S/dc-lru" -- --cache-disk-size "$cap" && dc_chat "$PC" "$S/dc-lru2.json" "$(dc_msgs "Variant B. $DC_U1")" > /dev/null
|
|
stop_server "$DC_PID"
|
|
files=$(find "$S/dc-lru" -name '*.lpcd' | wc -l); total=$(du -cb "$S"/dc-lru/*.lpcd 2>/dev/null | tail -1 | cut -f1)
|
|
check diskcache/lru $(( files == 1 && ${total:-0} <= cap * 1048576 && $(grep -c 'size limit, removing' "$S/dc-lru2.log") == 1 ? 0 : 1 )) \
|
|
"limit $cap MiB (file ~$((size / 1048576)) MiB): $files file(s), $((${total:-0} / 1048576)) MiB after two saves, $(grep -c 'size limit, removing' "$S/dc-lru2.log") eviction(s)"
|
|
|
|
# (3'') write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored -> EFBIG): logged, exit 0, nothing left, next start serves
|
|
rm -rf "$S/dc-full"
|
|
SRV_WRAP=(bash -c 'trap "" XFSZ; ulimit -f 1024; exec "$@"' _)
|
|
dc_start dc-full "$S/dc-full" -- && dc_chat "$PC" "$S/dc-full.json" "$(dc_msgs "$DC_U1")" > /dev/null
|
|
SRV_WRAP=()
|
|
dc_term
|
|
local left; left=$(find "$S/dc-full" -type f 2>/dev/null | wc -l)
|
|
dc_start dc-full2 "$S/dc-full" -- || { fail diskcache/write-error "restart failed, see $S/dc-full2.log"; return; }
|
|
code=$(dc_chat "$PC" "$S/dc-full2.json" "$DC_M2")
|
|
check diskcache/write-error $([ "$RC" -eq 0 ] && [ "$left" -eq 0 ] && grep -q 'disk cache: failed to write' "$S/dc-full.log" && [ "$code" = 200 ] && echo 0 || echo 1) \
|
|
"exit $RC in ${DC_EXIT_MS} ms, '$(grep -h 'disk cache: failed to write' "$S/dc-full.log" | sed 's/.*: \([^:]*\), not saved/\1/' | head -1)', $left file(s) left, next start HTTP $code"
|
|
stop_server "$DC_PID"
|
|
|
|
# (4) with an mmproj loaded, as the production 35B and gemma-26B run: a text-only chat is saved and
|
|
# restored (server_tokens::get_tokens() asserts in that mode); a prompt with an image is never
|
|
# restored or saved and does not crash. Own reference: the CPU references above stay as they are.
|
|
if [ ! -f "$MMPROJ" ]; then
|
|
skip diskcache/mmproj "mmproj missing: $MMPROJ"
|
|
else
|
|
DC_MM=(--mmproj "$MMPROJ" --no-mmproj-offload)
|
|
local DC_ENV_MM=("${DC_ENV[@]}" LLAMA_MMPROJ_ONDEMAND=1)
|
|
if dc_ref dc-ref-mm; then
|
|
dc_restore restore-mmproj "$S/dc-mm" 0 "${DC_ENV_MM[@]}" --
|
|
local img ans1 ans2 c2 nres1 ncopy
|
|
img=$(make_red_ppm)
|
|
printf '{"messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"data:image/x-portable-pixmap;base64,%s"}},{"type":"text","text":"What is the main color of this image? Answer with one word."}]}],"max_tokens":16,%s}' \
|
|
"$img" "$GREEDY" > "$S/dc-img-body.json"
|
|
rm -rf "$S/dc-img"; mkdir -p "$S/dc-img"; cp "$S/restore-mmproj-turn1.lpcd" "$S/dc-img/" 2>/dev/null
|
|
if dc_start dc-img "$S/dc-img" "${DC_ENV_MM[@]}" --; then
|
|
# image first (no restore), then the text chat (restored), then an image again: the slot holds media at the SIGTERM
|
|
code=$(curl -sS -m 300 -o "$S/dc-img1.json" -w '%{http_code}' -H 'Content-Type: application/json' -d @"$S/dc-img-body.json" "http://127.0.0.1:$PC/v1/chat/completions" 2>>"$S/curl.err")
|
|
ans1=$(json_str content "$S/dc-img1.json"); nres1=$(grep -c 'disk cache: restored' "$S/dc-img.log")
|
|
dc_chat "$PC" "$S/dc-img2.json" "$DC_M2" > /dev/null; c2=$(json_str content "$S/dc-img2.json")
|
|
local code3; code3=$(curl -sS -m 300 -o "$S/dc-img3.json" -w '%{http_code}' -H 'Content-Type: application/json' -d @"$S/dc-img-body.json" "http://127.0.0.1:$PC/v1/chat/completions" 2>>"$S/curl.err")
|
|
ans2=$(json_str content "$S/dc-img3.json")
|
|
dc_term
|
|
ncopy=$(grep -c 'disk cache: copied' "$S/dc-img.log")
|
|
local nf; nf=$(find "$S/dc-img" -name '*.lpcd' | wc -l)
|
|
check diskcache/mmproj-image $([ "$code" = 200 ] && [ "$code3" = 200 ] && [ "$nres1" -eq 0 ] && [ "$(grep -c 'disk cache: restored' "$S/dc-img.log")" -eq 1 ] && \
|
|
[ "$c2" = "$DC_REF2" ] && [ "$RC" -eq 0 ] && [ "$ncopy" -eq 0 ] && [ "$nf" -eq 1 ] && echo "$ans1$ans2" | grep -qi red && echo 0 || echo 1) \
|
|
"image: HTTP $code '$ans1', $nres1 restore(s); text chat after it: $(grep -c 'disk cache: restored' "$S/dc-img.log") restore(s), output $([ "$c2" = "$DC_REF2" ] && echo == || echo '!=') reference; image again: HTTP $code3 '$ans2'; SIGTERM with media in the slot: exit $RC, $ncopy state(s) copied, $nf file(s)"
|
|
else
|
|
fail diskcache/mmproj-image "server did not start, see $S/dc-img.log"
|
|
fi
|
|
fi
|
|
DC_MM=(--no-mmproj)
|
|
fi
|
|
|
|
# (1') GPU: the state is saved from VRAM (warm) and from a host shadow (cold), restored into VRAM
|
|
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
|
|
if [ $free_mb -lt 1536 ]; then
|
|
skip diskcache/gpu "only $free_mb MiB VRAM free (< 1536)"
|
|
elif dc_ref dc-ref-gpu "${DC_GPU[@]}"; then
|
|
dc_restore gpu-warm "$S/dc-gpu" 0 "${DC_ENV[@]}" -- "${DC_GPU[@]}"
|
|
dc_restore gpu-cold "$S/dc-gpu" 1 "${DC_ENV[@]}" -- "${DC_GPU[@]}"
|
|
dc_start dc-none-gpu "" "${DC_ENV[@]}" -- "${DC_GPU[@]}" && dc_chat "$PC" "$S/dc-none-gpu.json" "$(dc_msgs "$DC_U1")" > /dev/null
|
|
dc_term
|
|
log "diskcache: warm GPU arbiter shutdown without the cache: exit $RC in ${DC_EXIT_MS} ms, token free in ${DC_FREE_MS} ms"
|
|
fi
|
|
|
|
local bad_lines; bad_lines=$(scan_logs "$S"/dc-*.log "$S"/restore*.log "$S"/gpu-*.log)
|
|
check diskcache/log-scan $([ -z "$bad_lines" ] && echo 0 || echo 1) "${bad_lines:-clean}"
|
|
}
|
|
|
|
# ================================================================ main
|
|
if [ -n "$PERF_REPORT" ]; then
|
|
[ -r "$PERF_REPORT" ] || { echo "cannot read $PERF_REPORT" >&2; exit 2; }
|
|
perf_report "$PERF_REPORT"; exit 0
|
|
fi
|
|
[ $DO_OPS -eq 1 ] && run_ops
|
|
[ $DO_CONCAT -eq 1 ] && run_concat
|
|
[ $DO_ARB -eq 1 ] && run_arbiter
|
|
[ $DO_PERF -eq 1 ] && run_perf
|
|
[ $DO_MOEC -eq 1 ] && run_moecache
|
|
[ $DO_MOEX -eq 1 ] && run_moe_exact
|
|
[ $DO_DC -eq 1 ] && run_diskcache
|
|
exit 0
|