diff --git a/FORK.md b/FORK.md index a3d2a88f7..d34b3e1e0 100644 --- a/FORK.md +++ b/FORK.md @@ -6,12 +6,22 @@ which also holds the operations reference (pod access, files, recovery). Status at the time of writing (2026-10-25): -- Branch `fable5/vram-ondemand-rebase-20261024`: upstream `3f03b1772` plus 27 fork commits. -- Production runs `6c480f7e7` (the first 25 commits). The 2 commits after it, `e085c347d` (concat - env knob) and `149a8cb0a` (`scripts/fork-validate.sh`), are validation-only and were not deployed. +- Branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772`. The fork commits are + `git log --oneline origin/master..HEAD`; that list includes documentation and script commits + (this file among them), so count it rather than trusting a number written here. Section 2 maps + every commit to a feature. +- Production runs `6c480f7e7`. Everything after it on the branch is validation tooling, env-gated + A/B knobs, comment fixes and documentation; none of it was deployed. The knobs and the comment + fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation. - Commit hashes below refer to that branch. They change on every rebase; find a commit again by - its subject line (`git log --oneline origin/master..HEAD`). The commit messages are detailed and - are the primary record of each mechanism and measurement; read them with `git show`. + its subject line. The commit messages are detailed and are the primary record of each mechanism + and measurement; read them with `git show`. +- Commit trailers (owner's convention): this fork is private and never contributes upstream. + Agent-written commits end with `Co-Authored-By: Claude ` (some + older ones say `Assisted-by: Claude`). AGENTS.md asks agents for `Assisted-by:` instead of + `Co-authored-by:` and forbids AI-written commit messages and PR descriptions; those are upstream + llama.cpp's contribution rules and do not apply to commits that stay in this fork. Anything meant + for upstream must be written by the owner under those rules. ## Contents @@ -95,6 +105,17 @@ Oldest first. "Feature" points at the section below. | 6c480f7e7 | server: release the GPU token promptly on shutdown | F7 | | e085c347d | ggml-vulkan: env knob for the concat transpose fast path | F3 | | 149a8cb0a | scripts: add fork-validate.sh, post-rebase validation of the fork | F9 | +| bfbaf15b2 | docs: FORK.md and rebase runbook for the RX 580 fork | docs | +| f5c0d8f28 | server: fix stale VRAM-arbiter comments and the sleep log line | F6 | +| f664b0f20 | llama: env knob for the routing-ids readback skip | F1 | +| 62d442e7b | scripts: rx580-bench - decode at tg256 depth 4096, not tg32 | F9 | +| 2512290f9 | scripts: fork-validate.sh - gate --perf on medians, not means | F9 | +| 80e662a2a | scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 | +| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 | +| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 | + +Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code. +Everything from `e085c347d` down is after the production commit `6c480f7e7`. --- @@ -145,10 +166,17 @@ resolution would have resurrected the deleted scheduler block. semantics around negative ids. - If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one. -**Validation.** No dedicated check. `fork-validate.sh --ops` covers the MUL_MAT_ID kernel (not this -callback); `--perf` catches a prefill regression; `--moecache` exercises the cache path through this -callback. After a rebase that touches this function, also compare greedy output against the reference -build on the 35B. +**Validation.** `fork-validate.sh --moe-exact MODEL` (needs the GPU to itself, run it in the +`--perf` swapdown window with the 35B): reads `expert_count`/`expert_used_count` from the GGUF, +computes the threshold ceil(4 * n_expert / n_expert_used) tokens per ubatch (128 on the 35B), and +compares 64 greedy tokens after a prompt of about twice that, every expert on the host: +candidate == reference (`identical-ref`), and with the `LLAMA_MOE_IDS_SKIP` knob (f664b0f20; +`=0` forces the readback, `=log` reports each skip on stderr) candidate == candidate with the skip +forced off (`identical-noskip`, which proves the skip exact even when the reference build carries +it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). Not yet run on the GPU +(added after the 20261024 deploy; only dry-run against stub servers). Also `--ops` (the MUL_MAT_ID +kernel, not this callback), `--perf` (prefill regression) and `--moecache` (the cache path through +this callback). ### F2. GCN flash-attention mask_opt gate (and the shmem-staging note) @@ -200,9 +228,19 @@ Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 19 **Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`: `ggml_vk_concat` (gate lambda `src_transposed_2d` and the `s1_trans`/`s0_trans` checks) and `ggml_vk_concat_transpose_fastpath`. -The graph pattern comes from `llm_build_delta_net_base::build_conv_state` in -`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next); qwen4exp's `build_conv_state_at` uses -the same pattern. Commits 3df719e1e (fast path), 5e429d4a9 (decode gate), e085c347d (env knob). +The graph pattern that matters here comes from `llm_build_delta_net_base::build_conv_state` in +`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next). Commits 3df719e1e (fast path), +5e429d4a9 (decode gate), e085c347d (env knob). + +Other graphs that build `ggml_concat(state, ggml_transpose(x), 0)` (found with +`git grep -n "ggml_concat.*ggml_transpose" src/models/` plus the two-line form in delta-net-base and +lfm-base): `qwen4exp.cpp` (`build_conv_state_at`), `bailingmoe3.cpp`, `clef.cpp`, `kimi-linear.cpp`, +`kimi-k3.cpp`, `glm5-next.cpp`, `mamba-base.cpp` (two sites, the Mamba/Mamba2 conv and the hybrids +built on it), `plamo2.cpp`, and `lfm-base.cpp` (causal short conv; the transpose is assigned to `bx` +first). The fast path takes any of them when the gate holds (transposed source a plain 2D transpose +of a contiguous f32/f16 tensor, one sequence, `ne[0] >= 32`), so a change to the gate or the shader +affects all of them. Only the delta-net path (Qwen3.5/3.6) has been measured and validated; the +`--concat` check uses qwen35. **How it works.** For a concat along dim 0 where one source is stored transposed (`nb[1] == type_size`, `nb[0] == ne[1] * type_size`) and the other source and dst are contiguous, it issues two @@ -389,9 +427,13 @@ Measured footprints: **Validation.** `fork-validate.sh --arbiter`: `b-greedy-identical` (15 cycles each side equal to a no-handoff reference), `b-kv-survives` (prompt_n drops after evict/restore, output identical), `c-image-after-restore`. `--moecache MODEL` (needs a cache large enough to hit; 256 MiB on gemma-26B -had 0 hits and proves nothing, 2048 MiB had 6651 hits). The script does not check that VRAM actually -drops on release; check that by hand with `cat /sys/class/drm/card*/device/mem_info_vram_used` -before and after forcing a release. +had 0 hits and proves nothing, 2048 MiB had 6651 hits). `--arbiter` `v-vram-release` / +`v-vram-restore` check that VRAM really drops: a baseline of `mem_info_vram_used` before any scratch +server, then server A warm and alone, forced cold by a fake waiter on want.lock: the card-wide delta +must drop by >= 70% of what A added and come back after the next request. Production's own share is +re-read from the DRM fdinfo of every process the script did not start; if it moved by > 64 MiB the +check SKIPs. First run (2026-10-25, 0.8B with mmproj, ctx 8192, `LLAMA_SLEEP_EVICT_KV=1`): warm ++546 MiB, cold +4 MiB, restored +546 MiB. ### F5. On-demand mmproj (multimodal encoder) @@ -400,9 +442,11 @@ before and after forcing a release. 35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered correctly, text after the image correct. -**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream renamed it from -`process_mtmd_chunk`), the `mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`, -`vram_go_cold`, `vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198. +**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream `9871df591` split the +encode out of `process_mtmd_chunk`; `encode_mtmd_chunk` is called from `process_mtmd_chunk` and +directly from the batched-embedding path in the slot update, so the restore covers both), the +`mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`, `vram_go_cold`, +`vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198. **How it works.** After load the encoder is released (shadow captured). Before `mtmd_batch_encode`, `mtmd_restore_device(mctx)`; if that fails (out of VRAM) and on-demand is set, it @@ -432,7 +476,9 @@ and text right after must equal the reference). Post-deploy: one image request t (holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production: 60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last -`fork-validate.sh` run: 30 handoffs, median 680 ms, max 1231 ms. +`fork-validate.sh` run before the 20261024 deploy: 30 handoffs, median 680 ms, max 1231 ms (the +2026-10-25 run of the same build read 1050 ms, max 1562 ms: handoff latency moves with node load, +which is why it is now A/B'd against the reference in one session). **Where.** `tools/server/server-context.cpp`: `vram_arena_open`, `vram_acquire_for_load`, `vram_share_init`, `vram_should_yield`, `vram_want_acquire`, `vram_want_release`, `vram_take_token`, @@ -472,8 +518,27 @@ Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f. queue; the idle `wait_for` only leaves the loop on a real task or termination. **Invariants that must hold.** -- Resident iff holding `token.lock`. `vram_cold = false` is set only AFTER the restore, so the loop - cannot give the token away while still bringing weights up. +- Resident iff holding `token.lock`; `vram_cold == false` means both. `vram_ensure_warm()` sets + `vram_cold = false` only AFTER the restore. What that order protects today, reasoned from the code: + the only thing that gives the token away is `vram_should_yield()`, called from `should_sleep()` on + the loop thread. `vram_ensure_warm()` also runs only on the loop thread, synchronously: from the + wake callback (`handle_sleeping_state(false)`, called inside `start_loop()` with `mutex_tasks` + held) and before decode in the slot update path (outside `yield_to_queue()`). So `should_sleep()` + cannot run between the token acquire and the end of the restore, whichever order the two lines + are in. The measured failure (attempt 1 below: handoffs of 11m20s and 3m57s) happened under the + retired doorbell design, where a separate warden thread read the cold flag to decide whether to + honour or discard rings, so an early `false` let it accept stale rings during the restore. Under + the inline want-lock probe the order is defensive: it keeps the flag's meaning exact for any future + reader off the loop thread, and keeps a release out of a half-restored model if the restore or the + probe ever moves (for example into `yield_to_queue()`). Either way, do not change it without an A/B + of ordinary handoff latency (`fork-validate.sh --arbiter`, `a-handoff-latency`). +- `vram_acquire_for_load()` sets `vram_cold = false` as soon as it holds the token, before the upload. + Nothing reads the flag during the load today (the loop is not running yet and `on_should_yield` is + only registered in `vram_share_init()`, which sets it again), so the early set is redundant now. + It records "holding the token" at the moment it becomes true, because the opposite error is the + dangerous one: a model that is warm and holds the token while `vram_cold` says cold never yields + (`vram_should_yield()` returns false for a cold model), which strands every waiter, and its next + wake would try to restore weights that were never released. - Release only happens from the idle branch of `start_loop()`, on the loop thread, never inside `yield_to_queue()` or while a slot is generating: a holder finishes its current generation before it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both). @@ -501,9 +566,10 @@ Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f. blocked flock held `mutex_tasks`, so the whole server stopped answering while `/health` still returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by the second day). -- Failed fix attempt 1: moving `vram_cold = false` before the restore made the holder honour stale - rings and hand the token back 2-5 s after taking it without serving. Handoffs took 11m20s and 3m57s - against 5 s and 20 s before. Rolled back. +- Failed fix attempt 1 (doorbell design): moving `vram_cold = false` before the restore made the + holder's warden honour stale rings and hand the token back 2-5 s after taking it without serving. + Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. See the first invariant + for what this means under the current design. - Failed fix attempt 2: a re-ring loop (`LOCK_NB` + re-ring every ~1 s). Queued rings from an already satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become @@ -537,7 +603,10 @@ Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f. **Validation.** `fork-validate.sh --arbiter` with two real servers on a scratch arena: `a-handoffs` (30 alternating, 0 stranded, 0 non-200, the server that answered holds `token.lock`), -`b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams: +`a-handoff-latency` (fresh server pairs of `--build` and `--ref` in the same session, blocks +ref, cand, cand, ref x 16 handoffs; FAIL if the candidate median exceeds the reference median by more +than max(25%, 200 ms); first run 2026-10-25, reb1024 against itself beside production: 1050.5 vs +1082.5 ms), `v-vram-*` (F4), `b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams: want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in F7 and a log scan for asserts and Vulkan errors. In production, `/proc/locks` must show the warm model holding WRITE on `token.lock`. @@ -569,6 +638,16 @@ first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/serv - After `start_loop()` returns, `vram_share_exit()` frees all device buffers with `llama_context_discard_device()` (no host shadows), releases the encoder, and only then unlocks `token.lock`. +- What llama-swap (v239 on the pod) does when a child exits on its own (crash, OOM kill, external + signal), verified 2026-10-25 with a scratch llama-swap instance and a CPU-only child, for both + `kill -9` and an external SIGTERM: it logs + `[WARN] group: running exited: [] upstream exited unexpectedly`, does not restart it + eagerly (no new process in the 2 s after the kill), and starts a new process on the next request + for that model (new PID, health check, HTTP 200). The production log since the last pod restart has + no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers + the restart pays a full model load (about 26 s for the 35B with `-lm none`), and under the arbiter + the new process takes the token through `vram_acquire_for_load()` like any load. A process that + wedges instead of exiting (the incidents below) is never restarted: `/health` keeps answering. **Invariants that must hold.** - Nothing that takes `mutex_tasks` runs in signal context. @@ -648,39 +727,59 @@ required, so a rebase can be validated by someone with no context. **Where.** `scripts/rx580-bench/` (`run.sh` workstation wrapper; `bench.sh` pod orchestrator; `ppbench.py` corpus prefill through llama-server; `lbsweep.sh` llama-bench sweep; `summarize.py`; `README.md` with methodology and reference values). `scripts/fork-validate.sh`. Commits 4c4486f8e, -54a7f534b, 149a8cb0a. +54a7f534b, 149a8cb0a, 62d442e7b, 2512290f9, 80e662a2a, ceab1a91f, e5fc474b8. **How it works.** - `rx580-bench`: stops llama-swap, waits for an idle GPU, runs fixed slices of `pan-tadeusz.txt` (4096/16384/32768 tokens, byte-identical across builds) through `/completion` with - `cache_prompt: false`, plus a llama-bench sweep, interleaved across arms and rounds; restarts - llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`. -- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]`: - runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under - /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets inherited - `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check flag is - `--ops --concat --arbiter`. See the header of the script for what each check proves. + `cache_prompt: false`, plus a llama-bench sweep (prefill pp2048/pp8192 at depth 0, decode + tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and + rounds; restarts llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`. +- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL] + [--moe-exact MODEL]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a + scratch dir under /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets + inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check + flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline. + See the header of the script for what each check proves. +- Which phase needs what: `--ops --concat --arbiter` run beside production (`--arbiter` refuses + below 1.5 GiB free VRAM); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the + first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share + the card either). -**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` refuses, and any -other llama process blocks it). `--perf` gates prefill at -5% per depth and peak VRAM at 95% of the -card; decode is reported, not gated (CPU-bound and noisy here). +**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24, +production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s, +pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. The corpus +harness and older numbers are in `scripts/rx580-bench/README.md`. A fresh run far outside these +means something else is wrong (another GPU user, thermal state, a bad build) before it means a +regression or a win. + +**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` and `--moe-exact` +refuse, and any other llama process blocks them). `--perf` gates prefill at -5% per depth on the +MEDIAN of all repetition samples of each build (never a mean: CPU contention produces single +catastrophic samples), and peak VRAM at 95% of the card; decode is reported, not gated (CPU-bound and +noisy here). **Failure modes and incidents.** Upstream #28334 removed `--mmap`/`--no-mmap`/`--mlock` and llama-bench's `-mmp`: the server refused to start and llama-bench silently produced no rows until 54a7f534b switched to `-lm none`. The perf and MoE-cache checks leave tens of GB of model files in page cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024 -deploy (see the runbook). +deploy (see the runbook). That pod restart also emptied `/tmp` (every earlier `fork-validate.*` +scratch dir with its raw samples is gone) and removed gdb, which is not in the pod image; `--arbiter` +case `d-term-in-release` then falls back to polling the log. **Rebase hazards.** CLI flag renames in `common/arg.cpp` and `tools/llama-bench`; log line texts the script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`, `no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`, -`llama_moe_cache: .* hits =`, `N/M tests passed`, `Backend Vulkan0: OK`); the llama-bench markdown -table layout. +`llama_moe_cache: .* hits =`, `sched_copy_experts: ids readback skipped`, `N/M tests passed`, +`Backend Vulkan0: OK`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and +markdown layout; the GGUF keys `.expert_count` / `.expert_used_count`; the DRM fdinfo +keys `drm-client-id` / `drm-memory-vram`. -**Validation.** Known gaps of the script: it does not check that VRAM drops on release; peak VRAM is -recorded per arm, not per test (one candidate arm once showed an unexplained 1.6 GiB spike that did -not repeat); the SIGTERM-in-release breakpoint is at function entry, not after the host-shadow -capture; nothing covers F1 directly. +**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test (one candidate +arm once showed an unexplained 1.6 GiB spike that did not repeat); the SIGTERM-in-release breakpoint +is at function entry, not after the host-shadow capture; `--moe-exact` and the median-gated `--perf` +have not run on the GPU yet (tested offline only, see the runbook); the `v-vram-release` SKIP path +(production moving during the measurement) has not been exercised. --- @@ -827,11 +926,12 @@ Machine-level: null buffer would crash. Only the mmproj path handles a failed restore. - Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4). - Router mode can spin at shutdown (F7); not used here. -- Two code comments are stale: the arbiter block comment in `server-context.cpp` says the KV cache is - never evicted (it is, with `LLAMA_SLEEP_EVICT_KV`), and `vram_go_cold` says the encoder is rebuilt by - `vram_ensure_warm()` (it is restored only at encode time). -- The README fork section says "the two code changes below are auto-on"; the fork now has more code - changes than that. Treat this file as the complete list. +- The stale arbiter comments (KV "never evicted", encoder "rebuilt by `vram_ensure_warm()`", lazy + wake, `process_mtmd_chunk`) and the "keeping KV cache" sleep log line were fixed in f5c0d8f28, + after the deployed build: production still logs "keeping KV cache" while it evicts the KV. +- The new checks (`--moe-exact`, `a-handoff-latency`, `v-vram-*`, median `--perf`) and the + `LLAMA_MOE_IDS_SKIP` knob were written after the 20261024 deploy; the knob is syntax-checked only + (not built), and only `--arbiter` has run on the GPU. Run the full set at the next rebase. - Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context. Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.