docs: FORK.md - review fixes, new checks, llama-swap restart behaviour

- status: no hardcoded fork-commit count (the list includes docs and
  scripts; count git log origin/master..HEAD); what is after the production
  commit; the commit-trailer convention of this private fork vs upstream's
  AGENTS.md rules
- commit map: the commits after the production commit
- F1: validated by the new --moe-exact check and the LLAMA_MOE_IDS_SKIP knob
- F3: every graph that builds concat(state, transpose(x), 0) (bailingmoe3,
  clef, kimi-linear, kimi-k3, glm5-next, mamba-base, plamo2, lfm-base,
  qwen4exp besides delta-net); only the delta-net path is measured
- F4/F6: the new v-vram-* and a-handoff-latency checks with first results
- F5: encode_mtmd_chunk was split out of process_mtmd_chunk, not renamed
- F6: the vram_cold ordering rule restated from the code (the measured
  failure was under the doorbell design; with the inline probe should_sleep
  cannot run inside the restore; keep it, and A/B handoff latency before any
  change) and why vram_acquire_for_load sets the flag early
- F7: what llama-swap does when a child dies, verified with a scratch
  llama-swap: "upstream exited unexpectedly", restart on the next request
- F9: new flags, phases, median gating, 2026-10-24 reference values, the
  /tmp and gdb loss on the pod restart, new grep targets and gaps
- section 7: stale comments fixed, what is not yet run on the GPU

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-11-21 19:39:51 +01:00
1 parent 9537f885e0
commit 4e455b79df
1 file changed
+148 -48
+148 -48
View File
@@ -6,12 +6,22 @@ which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-10-25):
- Branch `fable5/vram-ondemand-rebase-20261024`: upstream `3f03b1772` plus 27 fork commits.
- Production runs `6c480f7e7` (the first 25 commits). The 2 commits after it, `e085c347d` (concat
env knob) and `149a8cb0a` (`scripts/fork-validate.sh`), are validation-only and were not deployed.
- Branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772`. The fork commits are
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
(this file among them), so count it rather than trusting a number written here. Section 2 maps
every commit to a feature.
- Production runs `6c480f7e7`. Everything after it on the branch is validation tooling, env-gated
A/B knobs, comment fixes and documentation; none of it was deployed. The knobs and the comment
fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation.
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line (`git log --oneline origin/master..HEAD`). The commit messages are detailed and
are the primary record of each mechanism and measurement; read them with `git show`.
its subject line. The commit messages are detailed and are the primary record of each mechanism
and measurement; read them with `git show`.
- Commit trailers (owner's convention): this fork is private and never contributes upstream.
Agent-written commits end with `Co-Authored-By: Claude <model> <noreply@anthropic.com>` (some
older ones say `Assisted-by: Claude`). AGENTS.md asks agents for `Assisted-by:` instead of
`Co-authored-by:` and forbids AI-written commit messages and PR descriptions; those are upstream
llama.cpp's contribution rules and do not apply to commits that stay in this fork. Anything meant
for upstream must be written by the owner under those rules.
## Contents
@@ -95,6 +105,17 @@ Oldest first. "Feature" points at the section below.
| 6c480f7e7 | server: release the GPU token promptly on shutdown | F7 |
| e085c347d | ggml-vulkan: env knob for the concat transpose fast path | F3 |
| 149a8cb0a | scripts: add fork-validate.sh, post-rebase validation of the fork | F9 |
| bfbaf15b2 | docs: FORK.md and rebase runbook for the RX 580 fork | docs |
| f5c0d8f28 | server: fix stale VRAM-arbiter comments and the sleep log line | F6 |
| f664b0f20 | llama: env knob for the routing-ids readback skip | F1 |
| 62d442e7b | scripts: rx580-bench - decode at tg256 depth 4096, not tg32 | F9 |
| 2512290f9 | scripts: fork-validate.sh - gate --perf on medians, not means | F9 |
| 80e662a2a | scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 |
| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything from `e085c347d` down is after the production commit `6c480f7e7`.
---
@@ -145,10 +166,17 @@ resolution would have resurrected the deleted scheduler block.
semantics around negative ids.
- If upstream moves expert copying again, port the threshold to the new place; do not resurrect the old one.
**Validation.** No dedicated check. `fork-validate.sh --ops` covers the MUL_MAT_ID kernel (not this
callback); `--perf` catches a prefill regression; `--moecache` exercises the cache path through this
callback. After a rebase that touches this function, also compare greedy output against the reference
build on the 35B.
**Validation.** `fork-validate.sh --moe-exact MODEL` (needs the GPU to itself, run it in the
`--perf` swapdown window with the 35B): reads `expert_count`/`expert_used_count` from the GGUF,
computes the threshold ceil(4 * n_expert / n_expert_used) tokens per ubatch (128 on the 35B), and
compares 64 greedy tokens after a prompt of about twice that, every expert on the host:
candidate == reference (`identical-ref`), and with the `LLAMA_MOE_IDS_SKIP` knob (f664b0f20;
`=0` forces the readback, `=log` reports each skip on stderr) candidate == candidate with the skip
forced off (`identical-noskip`, which proves the skip exact even when the reference build carries
it) and the log shows skips, all at >= 4 * n_expert ids (`skip-engaged`). Not yet run on the GPU
(added after the 20261024 deploy; only dry-run against stub servers). Also `--ops` (the MUL_MAT_ID
kernel, not this callback), `--perf` (prefill regression) and `--moecache` (the cache path through
this callback).
### F2. GCN flash-attention mask_opt gate (and the shmem-staging note)
@@ -200,9 +228,19 @@ Qwen3.5-0.8B the prefill CONCAT average was 369 us with the fast path against 19
**Where.** `ggml/src/ggml-vulkan/ggml-vulkan.cpp`: `ggml_vk_concat` (gate lambda
`src_transposed_2d` and the `s1_trans`/`s0_trans` checks) and `ggml_vk_concat_transpose_fastpath`.
The graph pattern comes from `llm_build_delta_net_base::build_conv_state` in
`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next); qwen4exp's `build_conv_state_at` uses
the same pattern. Commits 3df719e1e (fast path), 5e429d4a9 (decode gate), e085c347d (env knob).
The graph pattern that matters here comes from `llm_build_delta_net_base::build_conv_state` in
`src/models/delta-net-base.cpp` (qwen35, qwen35moe, qwen3next). Commits 3df719e1e (fast path),
5e429d4a9 (decode gate), e085c347d (env knob).
Other graphs that build `ggml_concat(state, ggml_transpose(x), 0)` (found with
`git grep -n "ggml_concat.*ggml_transpose" src/models/` plus the two-line form in delta-net-base and
lfm-base): `qwen4exp.cpp` (`build_conv_state_at`), `bailingmoe3.cpp`, `clef.cpp`, `kimi-linear.cpp`,
`kimi-k3.cpp`, `glm5-next.cpp`, `mamba-base.cpp` (two sites, the Mamba/Mamba2 conv and the hybrids
built on it), `plamo2.cpp`, and `lfm-base.cpp` (causal short conv; the transpose is assigned to `bx`
first). The fast path takes any of them when the gate holds (transposed source a plain 2D transpose
of a contiguous f32/f16 tensor, one sequence, `ne[0] >= 32`), so a change to the gate or the shader
affects all of them. Only the delta-net path (Qwen3.5/3.6) has been measured and validated; the
`--concat` check uses qwen35.
**How it works.** For a concat along dim 0 where one source is stored transposed (`nb[1] ==
type_size`, `nb[0] == ne[1] * type_size`) and the other source and dst are contiguous, it issues two
@@ -389,9 +427,13 @@ Measured footprints:
**Validation.** `fork-validate.sh --arbiter`: `b-greedy-identical` (15 cycles each side equal to a
no-handoff reference), `b-kv-survives` (prompt_n drops after evict/restore, output identical),
`c-image-after-restore`. `--moecache MODEL` (needs a cache large enough to hit; 256 MiB on gemma-26B
had 0 hits and proves nothing, 2048 MiB had 6651 hits). The script does not check that VRAM actually
drops on release; check that by hand with `cat /sys/class/drm/card*/device/mem_info_vram_used`
before and after forcing a release.
had 0 hits and proves nothing, 2048 MiB had 6651 hits). `--arbiter` `v-vram-release` /
`v-vram-restore` check that VRAM really drops: a baseline of `mem_info_vram_used` before any scratch
server, then server A warm and alone, forced cold by a fake waiter on want.lock: the card-wide delta
must drop by >= 70% of what A added and come back after the next request. Production's own share is
re-read from the DRM fdinfo of every process the script did not start; if it moved by > 64 MiB the
check SKIPs. First run (2026-10-25, 0.8B with mmproj, ctx 8192, `LLAMA_SLEEP_EVICT_KV=1`): warm
+546 MiB, cold +4 MiB, restored +546 MiB.
### F5. On-demand mmproj (multimodal encoder)
@@ -400,9 +442,11 @@ before and after forcing a release.
35B: warm text-only VRAM 6321 -> 5459 MB (about 860 MB freed for KV and compute), image answered
correctly, text after the image correct.
**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream renamed it from
`process_mtmd_chunk`), the `mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`,
`vram_go_cold`, `vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
**Where.** `tools/server/server-context.cpp`: `encode_mtmd_chunk` (upstream `9871df591` split the
encode out of `process_mtmd_chunk`; `encode_mtmd_chunk` is called from `process_mtmd_chunk` and
directly from the batched-embedding path in the slot update, so the restore covers both), the
`mtmd_release_device(mctx)` after `mtmd_init_from_file` in `load_model`, `vram_go_cold`,
`vram_share_exit`. Library side in F4. Commits 0a0f3ab6d, 5e99ac198.
**How it works.** After load the encoder is released (shadow captured). Before
`mtmd_batch_encode`, `mtmd_restore_device(mctx)`; if that fails (out of VRAM) and on-demand is set, it
@@ -432,7 +476,9 @@ and text right after must equal the reference). Post-deploy: one image request t
(holds its device buffers) at a time, and a cold one warms on demand. Handoff measured off production:
60 alternating handoffs between two small-model servers, 0 stranded, median 0.309 s. In production 4 handoffs between
the 35B and the 4B had a median of 4.357 s, most of it the 35B's own prefill. The last
`fork-validate.sh` run: 30 handoffs, median 680 ms, max 1231 ms.
`fork-validate.sh` run before the 20261024 deploy: 30 handoffs, median 680 ms, max 1231 ms (the
2026-10-25 run of the same build read 1050 ms, max 1562 ms: handoff latency moves with node load,
which is why it is now A/B'd against the reference in one session).
**Where.** `tools/server/server-context.cpp`: `vram_arena_open`, `vram_acquire_for_load`,
`vram_share_init`, `vram_should_yield`, `vram_want_acquire`, `vram_want_release`, `vram_take_token`,
@@ -472,8 +518,27 @@ Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
queue; the idle `wait_for` only leaves the loop on a real task or termination.
**Invariants that must hold.**
- Resident iff holding `token.lock`. `vram_cold = false` is set only AFTER the restore, so the loop
cannot give the token away while still bringing weights up.
- Resident iff holding `token.lock`; `vram_cold == false` means both. `vram_ensure_warm()` sets
`vram_cold = false` only AFTER the restore. What that order protects today, reasoned from the code:
the only thing that gives the token away is `vram_should_yield()`, called from `should_sleep()` on
the loop thread. `vram_ensure_warm()` also runs only on the loop thread, synchronously: from the
wake callback (`handle_sleeping_state(false)`, called inside `start_loop()` with `mutex_tasks`
held) and before decode in the slot update path (outside `yield_to_queue()`). So `should_sleep()`
cannot run between the token acquire and the end of the restore, whichever order the two lines
are in. The measured failure (attempt 1 below: handoffs of 11m20s and 3m57s) happened under the
retired doorbell design, where a separate warden thread read the cold flag to decide whether to
honour or discard rings, so an early `false` let it accept stale rings during the restore. Under
the inline want-lock probe the order is defensive: it keeps the flag's meaning exact for any future
reader off the loop thread, and keeps a release out of a half-restored model if the restore or the
probe ever moves (for example into `yield_to_queue()`). Either way, do not change it without an A/B
of ordinary handoff latency (`fork-validate.sh --arbiter`, `a-handoff-latency`).
- `vram_acquire_for_load()` sets `vram_cold = false` as soon as it holds the token, before the upload.
Nothing reads the flag during the load today (the loop is not running yet and `on_should_yield` is
only registered in `vram_share_init()`, which sets it again), so the early set is redundant now.
It records "holding the token" at the moment it becomes true, because the opposite error is the
dangerous one: a model that is warm and holds the token while `vram_cold` says cold never yields
(`vram_should_yield()` returns false for a cold model), which strands every waiter, and its next
wake would try to restore weights that were never released.
- Release only happens from the idle branch of `start_loop()`, on the loop thread, never inside
`yield_to_queue()` or while a slot is generating: a holder finishes its current generation before
it yields, and a waiter has no timeout (decoding without the token risks an OOM that kills both).
@@ -501,9 +566,10 @@ Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
blocked flock held `mutex_tasks`, so the whole server stopped answering while `/health` still
returned 200, and llama-swap never restarted it. Doorbell files also accumulated as litter (37 by
the second day).
- Failed fix attempt 1: moving `vram_cold = false` before the restore made the holder honour stale
rings and hand the token back 2-5 s after taking it without serving. Handoffs took 11m20s and 3m57s
against 5 s and 20 s before. Rolled back.
- Failed fix attempt 1 (doorbell design): moving `vram_cold = false` before the restore made the
holder's warden honour stale rings and hand the token back 2-5 s after taking it without serving.
Handoffs took 11m20s and 3m57s against 5 s and 20 s before. Rolled back. See the first invariant
for what this means under the current design.
- Failed fix attempt 2: a re-ring loop (`LOCK_NB` + re-ring every ~1 s). Queued rings from an already
satisfied waiter caused spurious releases; one handoff hung over 11 minutes against a 3.0-7.7 s
baseline. Lesson: a ring carries no notion of "still wanted"; the mechanism had to become
@@ -537,7 +603,10 @@ Commits 481be8b28, 958257cd6, b9df854de, 2feb6aeb2, 878d1338f.
**Validation.** `fork-validate.sh --arbiter` with two real servers on a scratch arena:
`a-handoffs` (30 alternating, 0 stranded, 0 non-200, the server that answered holds `token.lock`),
`b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams:
`a-handoff-latency` (fresh server pairs of `--build` and `--ref` in the same session, blocks
ref, cand, cand, ref x 16 handoffs; FAIL if the candidate median exceeds the reference median by more
than max(25%, 200 ms); first run 2026-10-25, reb1024 against itself beside production: 1050.5 vs
1082.5 ms), `v-vram-*` (F4), `b-*`, `c-*`, `g-kill9-waiter` (kill -9 a waiter registered on want.lock while the holder streams:
want.lock readers drop to 0, the holder keeps the token, no extra sleeps), plus the shutdown cases in
F7 and a log scan for asserts and Vulkan errors. In production, `/proc/locks` must show the warm
model holding WRITE on `token.lock`.
@@ -569,6 +638,16 @@ first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/serv
- After `start_loop()` returns, `vram_share_exit()` frees all device buffers with
`llama_context_discard_device()` (no host shadows), releases the encoder, and only then unlocks
`token.lock`.
- What llama-swap (v239 on the pod) does when a child exits on its own (crash, OOM kill, external
signal), verified 2026-10-25 with a scratch llama-swap instance and a CPU-only child, for both
`kill -9` and an external SIGTERM: it logs
`[WARN] group: running <model> exited: [<model>] upstream exited unexpectedly`, does not restart it
eagerly (no new process in the 2 s after the kill), and starts a new process on the next request
for that model (new PID, health check, HTTP 200). The production log since the last pod restart has
no such event. Consequences: the dead process's RAM prompt cache is gone, the request that triggers
the restart pays a full model load (about 26 s for the 35B with `-lm none`), and under the arbiter
the new process takes the token through `vram_acquire_for_load()` like any load. A process that
wedges instead of exiting (the incidents below) is never restarted: `/health` keeps answering.
**Invariants that must hold.**
- Nothing that takes `mutex_tasks` runs in signal context.
@@ -648,39 +727,59 @@ required, so a rebase can be validated by someone with no context.
**Where.** `scripts/rx580-bench/` (`run.sh` workstation wrapper; `bench.sh` pod orchestrator;
`ppbench.py` corpus prefill through llama-server; `lbsweep.sh` llama-bench sweep; `summarize.py`;
`README.md` with methodology and reference values). `scripts/fork-validate.sh`. Commits 4c4486f8e,
54a7f534b, 149a8cb0a.
54a7f534b, 149a8cb0a, 62d442e7b, 2512290f9, 80e662a2a, ceab1a91f, e5fc474b8.
**How it works.**
- `rx580-bench`: stops llama-swap, waits for an idle GPU, runs fixed slices of `pan-tadeusz.txt`
(4096/16384/32768 tokens, byte-identical across builds) through `/completion` with
`cache_prompt: false`, plus a llama-bench sweep, interleaved across arms and rounds; restarts
llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]`:
runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under
/tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets inherited
`GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check flag is
`--ops --concat --arbiter`. See the header of the script for what each check proves.
`cache_prompt: false`, plus a llama-bench sweep (prefill pp2048/pp8192 at depth 0, decode
tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and
rounds; restarts llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]
[--moe-exact MODEL]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a
scratch dir under /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets
inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check
flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline.
See the header of the script for what each check proves.
- Which phase needs what: `--ops --concat --arbiter` run beside production (`--arbiter` refuses
below 1.5 GiB free VRAM); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share
the card either).
**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` refuses, and any
other llama process blocks it). `--perf` gates prefill at -5% per depth and peak VRAM at 95% of the
card; decode is reported, not gated (CPU-bound and noisy here).
**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24,
production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s,
pp2048@d16384 about 201-206, pp2048@d32768 about 158-160, tg256@d4096 about 17.8. The corpus
harness and older numbers are in `scripts/rx580-bench/README.md`. A fresh run far outside these
means something else is wrong (another GPU user, thermal state, a bad build) before it means a
regression or a win.
**Invariants that must hold.** Never benchmark with llama-swap running (`--perf` and `--moe-exact`
refuse, and any other llama process blocks them). `--perf` gates prefill at -5% per depth on the
MEDIAN of all repetition samples of each build (never a mean: CPU contention produces single
catastrophic samples), and peak VRAM at 95% of the card; decode is reported, not gated (CPU-bound and
noisy here).
**Failure modes and incidents.** Upstream #28334 removed `--mmap`/`--no-mmap`/`--mlock` and
llama-bench's `-mmp`: the server refused to start and llama-bench silently produced no rows until
54a7f534b switched to `-lm none`. The perf and MoE-cache checks leave tens of GB of model files in page
cache charged to the pod; that was the likely trigger of a pod OOM kill during the first 20261024
deploy (see the runbook).
deploy (see the runbook). That pod restart also emptied `/tmp` (every earlier `fork-validate.*`
scratch dir with its raw samples is gone) and removed gdb, which is not in the pod image; `--arbiter`
case `d-term-in-release` then falls back to polling the log.
**Rebase hazards.** CLI flag renames in `common/arg.cpp` and `tools/llama-bench`; log line texts the
script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`,
`no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`,
`llama_moe_cache: .* hits =`, `N/M tests passed`, `Backend Vulkan0: OK`); the llama-bench markdown
table layout.
`llama_moe_cache: .* hits =`, `sched_copy_experts: ids readback skipped`, `N/M tests passed`,
`Backend Vulkan0: OK`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and
markdown layout; the GGUF keys `<arch>.expert_count` / `<arch>.expert_used_count`; the DRM fdinfo
keys `drm-client-id` / `drm-memory-vram`.
**Validation.** Known gaps of the script: it does not check that VRAM drops on release; peak VRAM is
recorded per arm, not per test (one candidate arm once showed an unexplained 1.6 GiB spike that did
not repeat); the SIGTERM-in-release breakpoint is at function entry, not after the host-shadow
capture; nothing covers F1 directly.
**Validation.** Known gaps of the script: peak VRAM is recorded per arm, not per test (one candidate
arm once showed an unexplained 1.6 GiB spike that did not repeat); the SIGTERM-in-release breakpoint
is at function entry, not after the host-shadow capture; `--moe-exact` and the median-gated `--perf`
have not run on the GPU yet (tested offline only, see the runbook); the `v-vram-release` SKIP path
(production moving during the measurement) has not been exercised.
---
@@ -827,11 +926,12 @@ Machine-level:
null buffer would crash. Only the mmproj path handles a failed restore.
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
- Router mode can spin at shutdown (F7); not used here.
- Two code comments are stale: the arbiter block comment in `server-context.cpp` says the KV cache is
never evicted (it is, with `LLAMA_SLEEP_EVICT_KV`), and `vram_go_cold` says the encoder is rebuilt by
`vram_ensure_warm()` (it is restored only at encode time).
- The README fork section says "the two code changes below are auto-on"; the fork now has more code
changes than that. Treat this file as the complete list.
- The stale arbiter comments (KV "never evicted", encoder "rebuilt by `vram_ensure_warm()`", lazy
wake, `process_mtmd_chunk`) and the "keeping KV cache" sleep log line were fixed in f5c0d8f28,
after the deployed build: production still logs "keeping KV cache" while it evicts the KV.
- The new checks (`--moe-exact`, `a-handoff-latency`, `v-vram-*`, median `--perf`) and the
`LLAMA_MOE_IDS_SKIP` knob were written after the 20261024 deploy; the knob is syntax-checked only
(not built), and only `--arbiter` has run on the GPU. Run the full set at the next rebase.
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.