docs: FORK.md - review fixes, new checks, llama-swap restart behaviour
- status: no hardcoded fork-commit count (the list includes docs and scripts; count git log origin/master..HEAD); what is after the production commit; the commit-trailer convention of this private fork vs upstream's AGENTS.md rules - commit map: the commits after the production commit - F1: validated by the new --moe-exact check and the LLAMA_MOE_IDS_SKIP knob - F3: every graph that builds concat(state, transpose(x), 0) (bailingmoe3, clef, kimi-linear, kimi-k3, glm5-next, mamba-base, plamo2, lfm-base, qwen4exp besides delta-net); only the delta-net path is measured - F4/F6: the new v-vram-* and a-handoff-latency checks with first results - F5: encode_mtmd_chunk was split out of process_mtmd_chunk, not renamed - F6: the vram_cold ordering rule restated from the code (the measured failure was under the doorbell design; with the inline probe should_sleep cannot run inside the restore; keep it, and A/B handoff latency before any change) and why vram_acquire_for_load sets the flag early - F7: what llama-swap does when a child dies, verified with a scratch llama-swap: "upstream exited unexpectedly", restart on the next request - F9: new flags, phases, median gating, 2026-10-24 reference values, the /tmp and gdb loss on the pod restart, new grep targets and gaps - section 7: stale comments fixed, what is not yet run on the GPU Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
384a7a1bec
commit
c829418367
1 file changed
+148
-48
@@ -6,12 +6,22 @@ which also holds the operations reference (pod access, files, recovery).
|
||||
|
||||
Status at the time of writing (2026-10-25):
|
||||
|
||||
- Branch `fable5/vram-ondemand-rebase-20261024`: upstream `3f03b1772` plus 27 fork commits.
|
||||
- Production runs `6c480f7e7` (the first 25 commits). The 2 commits after it, `e085c347d` (concat
|
||||
env knob) and `149a8cb0a` (`scripts/fork-validate.sh`), are validation-only and were not deployed.
|
||||
- Branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772`. The fork commits are
|
||||
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
|
||||
(this file among them), so count it rather than trusting a number written here. Section 2 maps
|
||||
every commit to a feature.
|
||||
- Production runs `6c480f7e7`. Everything after it on the branch is validation tooling, env-gated
|
||||
A/B knobs, comment fixes and documentation; none of it was deployed. The knobs and the comment
|
||||
fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation.
|
||||
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
|
||||
its subject line (`git log --oneline origin/master..HEAD`). The commit messages are detailed and
|
||||
are the primary record of each mechanism and measurement; read them with `git show`.
|
||||
its subject line. The commit messages are detailed and are the primary record of each mechanism
|
||||
and measurement; read them with `git show`.
|
||||
- Commit trailers (owner's convention): this fork is private and never contributes upstream.
|
||||
Agent-written commits end with `Co-Authored-By: Claude <model> <noreply@anthropic.com>` (some
|
||||
older ones say `Assisted-by: Claude`). AGENTS.md asks agents for `Assisted-by:` instead of
|
||||
`Co-authored-by:` and forbids AI-written commit messages and PR descriptions; those are upstream
|
||||
llama.cpp's contribution rules and do not apply to commits that stay in this fork. Anything meant
|
||||
for upstream must be written by the owner under those rules.
|
||||
|
||||
## Contents
|
||||
|
||||
@@ -95,6 +105,17 @@ Oldest first. "Feature" points at the section below.
|
||||
| 6c480f7e7 | server: release the GPU token promptly on shutdown | F7 |
|
||||
| e085c347d | ggml-vulkan: env knob for the concat transpose fast path | F3 |
|
||||
| 149a8cb0a | scripts: add fork-validate.sh, post-rebase validation of the fork | F9 |
|
||||
| bfbaf15b2 | docs: FORK.md and rebase runbook for the RX 580 fork | docs |
|
||||
| f5c0d8f28 | server: fix stale VRAM-arbiter comments and the sleep log line | F6 |
|
||||
| f664b0f20 | llama: env knob for the routing-ids readback skip | F1 |
|
||||
| 62d442e7b | scripts: rx580-bench - decode at tg256 depth 4096, not tg32 | F9 |
|
||||
| 2512290f9 | scripts: fork-validate.sh - gate --perf on medians, not means | F9 |
|
||||
| 80e662a2a | scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 |
|
||||
| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
|
||||
| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
|
||||
|
||||
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
||||
Everything from `e085c347d` down is after the production commit `6c480f7e7`.
|
||||
|
||||
---
|
||||
|
||||
@@ -145,10 +166,17 @@ resolution would have resurrected the deleted scheduler block.
|
||||
semantics around negative ids.
|
||||
- If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.
|
||||
|
||||
**Validation.** No dedicated check. `fork-validate.sh --ops` covers the MUL_MAT_ID kernel (not this
|
||||
callback); `--perf` catches a prefill regression; `--moecache` exercises the cache path through this
|
||||
callback. After a rebase that touches this function, also compare greedy output against the reference
|
||||
build on the 35B.
|
||||
**Validation.** `fork-validate.sh --moe-exact MODEL` (needs the GPU to itself, run it in the
|
||||
`--perf` swapdown window with the 35B): reads `expert_count`/`expert_used_count` from the GGUF,
|
||||
computes the threshold ceil(4 * n_expert / n_expert_used) tokens per ubatch (128 on the 35B), and
|
||||
compares 64 greedy tokens after a prompt of about twice that, every expert on the host:
|
||||
candidate == reference (`identical-ref`), and with the `LLAMA_MOE_IDS_SKIP` knob (f664b0f20;
|
||||
`=0` forces the readback, `=log` reports each skip on stderr) candidate == candidate with the skip
|
||||
forced off (`identical-noskip`, which proves the skip exact even when the reference build carries
|
||||
it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). Not yet run on the GPU
|
||||
(added after the 20261024 deploy; only dry-run against stub servers). Also `--ops` (the MUL_MAT_ID
|
||||
kernel, not this callback), `--perf` (prefill regression) and `--moecache` (the cache path through
|
||||
this callback).
|
||||
|
||||
### F2. GCN flash-attention mask_opt gate (and the shmem-staging note)
|
||||
|
||||
@@ -200,9 +228,19 @@ Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 19
|
||||
|
||||
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`: `ggml_vk_concat` (gate lambda
|
||||
`src_transposed_2d` and the `s1_trans`/`s0_trans` checks) and `ggml_vk_concat_transpose_fastpath`.
|
||||
The graph pattern comes from `llm_build_delta_net_base::build_conv_state` in
|
||||
`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next); qwen4exp's `build_conv_state_at` uses
|
||||
the same pattern. Commits 3df719e1e (fast path), 5e429d4a9 (decode gate), e085c347d (env knob).
|
||||
The graph pattern that matters here comes from `llm_build_delta_net_base::build_conv_state` in
|
||||
`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next). Commits 3df719e1e (fast path),
|
||||
5e429d4a9 (decode gate), e085c347d (env knob).
|
||||
|
||||
Other graphs that build `ggml_concat(state, ggml_transpose(x), 0)` (found with
|
||||
`git grep -n "ggml_concat.*ggml_transpose" src/models/` plus the two-line form in delta-net-base and
|
||||
lfm-base): `qwen4exp.cpp` (`build_conv_state_at`), `bailingmoe3.cpp`, `clef.cpp`, `kimi-linear.cpp`,
|
||||
`kimi-k3.cpp`, `glm5-next.cpp`, `mamba-base.cpp` (two sites, the Mamba/Mamba2 conv and the hybrids
|
||||
built on it), `plamo2.cpp`, and `lfm-base.cpp` (causal short conv; the transpose is assigned to `bx`
|
||||
first). The fast path takes any of them when the gate holds (transposed source a plain 2D transpose
|
||||
of a contiguous f32/f16 tensor, one sequence, `ne[0] >= 32`), so a change to the gate or the shader
|
||||
affects all of them. Only the delta-net path (Qwen3.5/3.6) has been measured and validated; the
|
||||
`--concat` check uses qwen35.
|
||||
|
||||
**How it works.** For a concat along dim 0 where one source is stored transposed (`nb[1] ==
|
||||
type_size`, `nb[0] == ne[1] * type_size`) and the other source and dst are contiguous, it issues two
|
||||
@@ -389,9 +427,13 @@ Measured footprints:
|
||||
**Validation.** `fork-validate.sh --arbiter`: `b-greedy-identical` (15 cycles each side equal to a
|
||||
no-handoff reference), `b-kv-survives` (prompt_n drops after evict/restore, output identical),
|
||||
`c-image-after-restore`. `--moecache MODEL` (needs a cache large enough to hit; 256 MiB on gemma-26B
|
||||
had 0 hits and proves nothing, 2048 MiB had 6651 hits). The script does not check that VRAM actually
|
||||
drops on release; check that by hand with `cat /sys/class/drm/card*/device/mem_info_vram_used`
|
||||
before and after forcing a release.
|
||||
had 0 hits and proves nothing, 2048 MiB had 6651 hits). `--arbiter` `v-vram-release` /
|
||||
`v-vram-restore` check that VRAM really drops: a baseline of `mem_info_vram_used` before any scratch
|
||||
server, then server A warm and alone, forced cold by a fake waiter on want.lock: the card-wide delta
|
||||
must drop by >= 70% of what A added and come back after the next request. Production's own share is
|
||||
re-read from the DRM fdinfo of every process the script did not start; if it moved by > 64 MiB the
|
||||
check SKIPs. First run (2026-10-25, 0.8B with mmproj, ctx 8192, `LLAMA_SLEEP_EVICT_KV=1`): warm
|
||||
+546 MiB, cold +4 MiB, restored +546 MiB.
|
||||
|
||||
### F5. On-demand mmproj (multimodal encoder)
|
||||
|
||||
@@ -400,9 +442,11 @@ before and after forcing a release.
|
||||
35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered
|
||||
correctly, text after the image correct.
|
||||
|
||||
**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream renamed it from
|
||||
`process_mtmd_chunk`), the `mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`,
|
||||
`vram_go_cold`, `vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
|
||||
**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream `9871df591` split the
|
||||
encode out of `process_mtmd_chunk`; `encode_mtmd_chunk` is called from `process_mtmd_chunk` and
|
||||
directly from the batched-embedding path in the slot update, so the restore covers both), the
|
||||
`mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`, `vram_go_cold`,
|
||||
`vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
|
||||
|
||||
**How it works.** After load the encoder is released (shadow captured). Before
|
||||
`mtmd_batch_encode`, `mtmd_restore_device(mctx)`; if that fails (out of VRAM) and on-demand is set, it
|
||||
@@ -432,7 +476,9 @@ and text right after must equal the reference). Post-deploy: one image request t
|
||||
(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production:
|
||||
60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between
|
||||
the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last
|
||||
`fork-validate.sh` run: 30 handoffs, median 680 ms, max 1231 ms.
|
||||
`fork-validate.sh` run before the 20261024 deploy: 30 handoffs, median 680 ms, max 1231 ms (the
|
||||
2026-10-25 run of the same build read 1050 ms, max 1562 ms: handoff latency moves with node load,
|
||||
which is why it is now A/B'd against the reference in one session).
|
||||
|
||||
**Where.** `tools/server/server-context.cpp`: `vram_arena_open`, `vram_acquire_for_load`,
|
||||
`vram_share_init`, `vram_should_yield`, `vram_want_acquire`, `vram_want_release`, `vram_take_token`,
|
||||
@@ -472,8 +518,27 @@ Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
|
||||
queue; the idle `wait_for` only leaves the loop on a real task or termination.
|
||||
|
||||
**Invariants that must hold.**
|
||||
- Resident iff holding `token.lock`. `vram_cold = false` is set only AFTER the restore, so the loop
|
||||
cannot give the token away while still bringing weights up.
|
||||
- Resident iff holding `token.lock`; `vram_cold == false` means both. `vram_ensure_warm()` sets
|
||||
`vram_cold = false` only AFTER the restore. What that order protects today, reasoned from the code:
|
||||
the only thing that gives the token away is `vram_should_yield()`, called from `should_sleep()` on
|
||||
the loop thread. `vram_ensure_warm()` also runs only on the loop thread, synchronously: from the
|
||||
wake callback (`handle_sleeping_state(false)`, called inside `start_loop()` with `mutex_tasks`
|
||||
held) and before decode in the slot update path (outside `yield_to_queue()`). So `should_sleep()`
|
||||
cannot run between the token acquire and the end of the restore, whichever order the two lines
|
||||
are in. The measured failure (attempt 1 below: handoffs of 11m20s and 3m57s) happened under the
|
||||
retired doorbell design, where a separate warden thread read the cold flag to decide whether to
|
||||
honour or discard rings, so an early `false` let it accept stale rings during the restore. Under
|
||||
the inline want-lock probe the order is defensive: it keeps the flag's meaning exact for any future
|
||||
reader off the loop thread, and keeps a release out of a half-restored model if the restore or the
|
||||
probe ever moves (for example into `yield_to_queue()`). Either way, do not change it without an A/B
|
||||
of ordinary handoff latency (`fork-validate.sh --arbiter`, `a-handoff-latency`).
|
||||
- `vram_acquire_for_load()` sets `vram_cold = false` as soon as it holds the token, before the upload.
|
||||
Nothing reads the flag during the load today (the loop is not running yet and `on_should_yield` is
|
||||
only registered in `vram_share_init()`, which sets it again), so the early set is redundant now.
|
||||
It records "holding the token" at the moment it becomes true, because the opposite error is the
|
||||
dangerous one: a model that is warm and holds the token while `vram_cold` says cold never yields
|
||||
(`vram_should_yield()` returns false for a cold model), which strands every waiter, and its next
|
||||
wake would try to restore weights that were never released.
|
||||
- Release only happens from the idle branch of `start_loop()`, on the loop thread, never inside
|
||||
`yield_to_queue()` or while a slot is generating: a holder finishes its current generation before
|
||||
it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both).
|
||||
@@ -501,9 +566,10 @@ Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
|
||||
blocked flock held `mutex_tasks`, so the whole server stopped answering while `/health` still
|
||||
returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by
|
||||
the second day).
|
||||
- Failed fix attempt 1: moving `vram_cold = false` before the restore made the holder honour stale
|
||||
rings and hand the token back 2-5 s after taking it without serving. Handoffs took 11m20s and 3m57s
|
||||
against 5 s and 20 s before. Rolled back.
|
||||
- Failed fix attempt 1 (doorbell design): moving `vram_cold = false` before the restore made the
|
||||
holder's warden honour stale rings and hand the token back 2-5 s after taking it without serving.
|
||||
Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. See the first invariant
|
||||
for what this means under the current design.
|
||||
- Failed fix attempt 2: a re-ring loop (`LOCK_NB` + re-ring every ~1 s). Queued rings from an already
|
||||
satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s
|
||||
baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become
|
||||
@@ -537,7 +603,10 @@ Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
|
||||
|
||||
**Validation.** `fork-validate.sh --arbiter` with two real servers on a scratch arena:
|
||||
`a-handoffs` (30 alternating, 0 stranded, 0 non-200, the server that answered holds `token.lock`),
|
||||
`b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams:
|
||||
`a-handoff-latency` (fresh server pairs of `--build` and `--ref` in the same session, blocks
|
||||
ref, cand, cand, ref x 16 handoffs; FAIL if the candidate median exceeds the reference median by more
|
||||
than max(25%, 200 ms); first run 2026-10-25, reb1024 against itself beside production: 1050.5 vs
|
||||
1082.5 ms), `v-vram-*` (F4), `b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams:
|
||||
want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in
|
||||
F7 and a log scan for asserts and Vulkan errors. In production, `/proc/locks` must show the warm
|
||||
model holding WRITE on `token.lock`.
|
||||
@@ -569,6 +638,16 @@ first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/serv
|
||||
- After `start_loop()` returns, `vram_share_exit()` frees all device buffers with
|
||||
`llama_context_discard_device()` (no host shadows), releases the encoder, and only then unlocks
|
||||
`token.lock`.
|
||||
- What llama-swap (v239 on the pod) does when a child exits on its own (crash, OOM kill, external
|
||||
signal), verified 2026-10-25 with a scratch llama-swap instance and a CPU-only child, for both
|
||||
`kill -9` and an external SIGTERM: it logs
|
||||
`[WARN] group: running <model> exited: [<model>] upstream exited unexpectedly`, does not restart it
|
||||
eagerly (no new process in the 2 s after the kill), and starts a new process on the next request
|
||||
for that model (new PID, health check, HTTP 200). The production log since the last pod restart has
|
||||
no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers
|
||||
the restart pays a full model load (about 26 s for the 35B with `-lm none`), and under the arbiter
|
||||
the new process takes the token through `vram_acquire_for_load()` like any load. A process that
|
||||
wedges instead of exiting (the incidents below) is never restarted: `/health` keeps answering.
|
||||
|
||||
**Invariants that must hold.**
|
||||
- Nothing that takes `mutex_tasks` runs in signal context.
|
||||
@@ -648,39 +727,59 @@ required, so a rebase can be validated by someone with no context.
|
||||
**Where.** `scripts/rx580-bench/` (`run.sh` workstation wrapper; `bench.sh` pod orchestrator;
|
||||
`ppbench.py` corpus prefill through llama-server; `lbsweep.sh` llama-bench sweep; `summarize.py`;
|
||||
`README.md` with methodology and reference values). `scripts/fork-validate.sh`. Commits 4c4486f8e,
|
||||
54a7f534b, 149a8cb0a.
|
||||
54a7f534b, 149a8cb0a, 62d442e7b, 2512290f9, 80e662a2a, ceab1a91f, e5fc474b8.
|
||||
|
||||
**How it works.**
|
||||
- `rx580-bench`: stops llama-swap, waits for an idle GPU, runs fixed slices of `pan-tadeusz.txt`
|
||||
(4096/16384/32768 tokens, byte-identical across builds) through `/completion` with
|
||||
`cache_prompt: false`, plus a llama-bench sweep, interleaved across arms and rounds; restarts
|
||||
llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
|
||||
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]`:
|
||||
runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under
|
||||
/tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets inherited
|
||||
`GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check flag is
|
||||
`--ops --concat --arbiter`. See the header of the script for what each check proves.
|
||||
`cache_prompt: false`, plus a llama-bench sweep (prefill pp2048/pp8192 at depth 0, decode
|
||||
tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and
|
||||
rounds; restarts llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
|
||||
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]
|
||||
[--moe-exact MODEL]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a
|
||||
scratch dir under /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets
|
||||
inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check
|
||||
flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline.
|
||||
See the header of the script for what each check proves.
|
||||
- Which phase needs what: `--ops --concat --arbiter` run beside production (`--arbiter` refuses
|
||||
below 1.5 GiB free VRAM); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
|
||||
first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share
|
||||
the card either).
|
||||
|
||||
**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` refuses, and any
|
||||
other llama process blocks it). `--perf` gates prefill at -5% per depth and peak VRAM at 95% of the
|
||||
card; decode is reported, not gated (CPU-bound and noisy here).
|
||||
**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24,
|
||||
production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s,
|
||||
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. The corpus
|
||||
harness and older numbers are in `scripts/rx580-bench/README.md`. A fresh run far outside these
|
||||
means something else is wrong (another GPU user, thermal state, a bad build) before it means a
|
||||
regression or a win.
|
||||
|
||||
**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` and `--moe-exact`
|
||||
refuse, and any other llama process blocks them). `--perf` gates prefill at -5% per depth on the
|
||||
MEDIAN of all repetition samples of each build (never a mean: CPU contention produces single
|
||||
catastrophic samples), and peak VRAM at 95% of the card; decode is reported, not gated (CPU-bound and
|
||||
noisy here).
|
||||
|
||||
**Failure modes and incidents.** Upstream #28334 removed `--mmap`/`--no-mmap`/`--mlock` and
|
||||
llama-bench's `-mmp`: the server refused to start and llama-bench silently produced no rows until
|
||||
54a7f534b switched to `-lm none`. The perf and MoE-cache checks leave tens of GB of model files in page
|
||||
cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024
|
||||
deploy (see the runbook).
|
||||
deploy (see the runbook). That pod restart also emptied `/tmp` (every earlier `fork-validate.*`
|
||||
scratch dir with its raw samples is gone) and removed gdb, which is not in the pod image; `--arbiter`
|
||||
case `d-term-in-release` then falls back to polling the log.
|
||||
|
||||
**Rebase hazards.** CLI flag renames in `common/arg.cpp` and `tools/llama-bench`; log line texts the
|
||||
script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`,
|
||||
`no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`,
|
||||
`llama_moe_cache: .* hits =`, `N/M tests passed`, `Backend Vulkan0: OK`); the llama-bench markdown
|
||||
table layout.
|
||||
`llama_moe_cache: .* hits =`, `sched_copy_experts: ids readback skipped`, `N/M tests passed`,
|
||||
`Backend Vulkan0: OK`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and
|
||||
markdown layout; the GGUF keys `<arch>.expert_count` / `<arch>.expert_used_count`; the DRM fdinfo
|
||||
keys `drm-client-id` / `drm-memory-vram`.
|
||||
|
||||
**Validation.** Known gaps of the script: it does not check that VRAM drops on release; peak VRAM is
|
||||
recorded per arm, not per test (one candidate arm once showed an unexplained 1.6 GiB spike that did
|
||||
not repeat); the SIGTERM-in-release breakpoint is at function entry, not after the host-shadow
|
||||
capture; nothing covers F1 directly.
|
||||
**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test (one candidate
|
||||
arm once showed an unexplained 1.6 GiB spike that did not repeat); the SIGTERM-in-release breakpoint
|
||||
is at function entry, not after the host-shadow capture; `--moe-exact` and the median-gated `--perf`
|
||||
have not run on the GPU yet (tested offline only, see the runbook); the `v-vram-release` SKIP path
|
||||
(production moving during the measurement) has not been exercised.
|
||||
|
||||
---
|
||||
|
||||
@@ -827,11 +926,12 @@ Machine-level:
|
||||
null buffer would crash. Only the mmproj path handles a failed restore.
|
||||
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
|
||||
- Router mode can spin at shutdown (F7); not used here.
|
||||
- Two code comments are stale: the arbiter block comment in `server-context.cpp` says the KV cache is
|
||||
never evicted (it is, with `LLAMA_SLEEP_EVICT_KV`), and `vram_go_cold` says the encoder is rebuilt by
|
||||
`vram_ensure_warm()` (it is restored only at encode time).
|
||||
- The README fork section says "the two code changes below are auto-on"; the fork now has more code
|
||||
changes than that. Treat this file as the complete list.
|
||||
- The stale arbiter comments (KV "never evicted", encoder "rebuilt by `vram_ensure_warm()`", lazy
|
||||
wake, `process_mtmd_chunk`) and the "keeping KV cache" sleep log line were fixed in f5c0d8f28,
|
||||
after the deployed build: production still logs "keeping KV cache" while it evicts the KV.
|
||||
- The new checks (`--moe-exact`, `a-handoff-latency`, `v-vram-*`, median `--perf`) and the
|
||||
`LLAMA_MOE_IDS_SKIP` knob were written after the 20261024 deploy; the knob is syntax-checked only
|
||||
(not built), and only `--arbiter` has run on the GPU. Run the full set at the next rebase.
|
||||
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
|
||||
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
|
||||
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
|
||||
Reference in new issue
Block a user