docs: FORK.md/runbook after the 3621025e8 deploy - F12 measurements, Inkling ub1024 + ctx 65536 and the offload knob, nothink finding, pointer 14

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-11-22 20:38:33 +01:00
1 parent 01812e8b02
commit 1a19c3bbba
2 files changed
+85 -23

No files matched your search

+55 -16
View File
@@ -4,14 +4,17 @@ Maintainer documentation for this private fork. Read it before you rebase, chang
anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md),
which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-11-21):
Status at the time of writing (2026-11-22):
- Production runs `3bf6b55b3` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits), deployed 2026-11-21 19:54, validated with the full `fork-validate.sh` set
(phase 1 39 PASS, phase 2 9 PASS, plus the 35B disk-cache legs; runbook section 6). Rollback
pointer `/root/DEPLOY-ROLLBACK-commit-13.txt` = `adc2aa3ec`; next free pointer 14.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12676-3bf6b55b3`) and changes
with each rebase. The hand ports are listed in the runbook (section 3, "2026-11-21").
- Production runs `3621025e8` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12), deployed
2026-11-22 03:52 with the Inkling retune (section 5). Validated with the full `fork-validate.sh`
set (phase 2 47 PASS, 0 FAIL; runbook section 6). Rollback pointer
`/root/DEPLOY-ROLLBACK-commit-14.txt` = `3bf6b55b3` (build copy `/root/arms/build-3bf6b55b3`,
config backup `/root/config.yaml.bak-20261122-inkling-ub1024-ctx65536`); next free pointer 15.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12680-3621025e8`) and changes
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
(section 3, "2026-11-21").
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
`adc2aa3ec` until 2026-11-21), kept untouched as the safety net. The fork commits are
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
@@ -1122,8 +1125,33 @@ without `808fd1c4e`; base `3bf6b55b3` vs base + both PRs; logs `/root/arms/pr-ou
gemma-26B 249.4 both.
- test-backend-ops on Vulkan0: SCALE 15/15, FLASH_ATTN_EXT 5693/5693.
`808fd1c4e` (the PR head since 2026-11-22) changes the table layout from `[n_head, n_bins, n_tokens]`
to `[n_bins, n_head, n_tokens]`, which removes one permute+cont from the Inkling graph; it was
validated with `tests/test-flash-attn-indexed-bias` and test-backend-ops, not re-benchmarked.
to `[n_bins, n_head, n_tokens]`, which removes one permute+cont from the Inkling graph.
Measured with the carried head (2026-11-22, candidate `3621025e8` vs production `3bf6b55b3`,
production's Inkling command line with the nothink sampling, GPU to itself; driver
`/root/arms/carry1122-ink-direct.py`, logs `/root/arms/carry1122-p2.log`, `-p3.log`):
| build, args | 8.2K-token prefill | 13.9K | 28.4K | process VRAM / GTT after load |
|---|---|---|---|---|
| production, ub512, ctx 65536 | 32.7 t/s | 31.6 | 29.5 | 7209 / 28113 MiB |
| carry, ub1024, ctx 131072 | 43.7 | 42.5 | 42.5 | 7983 / 21835 MiB |
| carry, ub1024, ctx 65536 | 46.8 | 46.9 | 46.0 | 7368 / 19629 MiB |
- Device memory wanted (`-lv 4` breakdown): production ub512/65536 19424 MiB (compute 15866); carry
ub1024/131072 13388 MiB (model 1266, context 4419, compute 7702), host compute 798 MiB. Everything
above the 8 GB card lands in GTT (system RAM over PCIe): about 5.2 GB at 131072. That costs about
8% prefill against 65536, so production runs ub1024 with ctx 65536 (section 5) until the loop-layer
experts move into VRAM, which needs that headroom.
- Answers: a ~160-token first request on a fresh server (upstream #32778, all-'?' output) was
correct on every build. Pan Tadeusz fragments of 8.2K and 13.9K tokens gave coherent, correct
summaries on every build (upstream #32755 did not reproduce). The nothink entry
(`--reasoning-effort none --reasoning-budget 0`) returns repeated
`<|message_model|><|content_thinking|><|end_message|>` as content, with no tool call, for an image
request, a tool call and a 28.4K-token weather log ("which station is in Record 5?"). This happens
on every build, production `3bf6b55b3` with ub512 included, with and without
`GGML_OP_OFFLOAD_MIN_BATCH=64`, so the carry did not cause it. The think entry gets all three
right on both builds (image "Red", `get_weather({"city":"Warsaw"})`, 28.4K log "65" at 45.9 t/s
in production). Short text prompts work on the nothink entry. Open: fix the nothink macro.
**Rebase hazards.**
- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry
@@ -1140,7 +1168,9 @@ validated with `tests/test-flash-attn-indexed-bias` and test-backend-ops, not re
**Validation.** `test-backend-ops -b Vulkan0 -o SCALE` and `-o FLASH_ATTN_EXT` (check the case
count), and the PR's own test `tests/test-flash-attn-indexed-bias` (build target
`test-flash-attn-indexed-bias`; it runs every device, prints one line per case it ran and skips
unsupported cases silently, so count the `Vulkan0:` lines; the Q4_0 K/V cases run on the CPU only).
unsupported cases silently, so count the `Vulkan0:` lines: 16 f16 cases on the RX 580, max error
about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rejects indexed bias
(the graph then uses the dense mask), so it prints no CPU lines).
Then an Inkling request through llama-swap.
---
@@ -1192,7 +1222,7 @@ layers.
## 5. Configuration that matters
`/root/config.yaml` (llama-swap) and `/root/supervisord.conf` on the pod are the source of truth;
this section explains the choices. Values as of the 20261024 deploy.
this section explains the choices. Values as of the 2026-11-22 deploy.
Qwen 35B-A3B MoE entries (macro `moe_offload_args`, applied to all 35B-A3B variants):
`--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1`,
@@ -1232,10 +1262,15 @@ Other models:
matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m`
(the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it,
`--n-cpu-moe 24` (all experts on host), `-t 6`, f16 KV, `--parallel 1`, the disk prompt cache.
Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 131072`
(before: `-b 512 -ub 512 --ctx-size 65536`, because the dense per-head bias mask needed about
12.7 GB of host compute buffer at 128k). Its nothink macro uses `--reasoning-effort none
--reasoning-budget 0`.
Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 65536`
(before: `-b 512 -ub 512`; the dense per-head bias mask needed about 12.7 GB of host compute
buffer at 128k). 131072 loads and works with the carry, but overcommits about 5.2 GB of device
memory into GTT and is about 8% slower at prefill (F12). Both Inkling entries set
`env: ["GGML_OP_OFFLOAD_MIN_BATCH=64"]` (llama-swap per-model env): ubatches of 32-63 tokens keep
MUL_MAT_ID on the CPU. The looped layers share their expert tensors, and below F1's readback-skip
threshold (4 x 128 experts / 8 used = 64 tokens) only loop pass 0's routed experts would be
uploaded (the mechanism suspected behind upstream #32755). Its nothink macro uses
`--reasoning-effort none --reasoning-budget 0`.
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
`/root/supervisord.conf`:
@@ -1249,7 +1284,7 @@ Environment, set once for all of llama-swap in `[program:llama-swap] environment
Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`,
`GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`.
`--cache-disk-path` (F11) is not set yet: it is not deployed; the recommended flags are in F11.
`--cache-disk-path /root/prompt-cache --cache-disk-size 4096` (F11) is set in the `moe_offload_args`, `moe_offload_gemma` and `inkling_args` macros.
Machine-level:
- GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted
@@ -1310,6 +1345,10 @@ Machine-level:
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
null buffer would crash. Only the mmproj path handles a failed restore.
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
- Inkling (F12, section 5): the nothink entry returns repeated chat-structure tokens instead of an
answer for image requests, tool calls and long prompts (the think entry is fine); the config runs
ctx 65536 because 131072 overcommits about 5.2 GB of device memory into GTT; upstream #32755 /
#32778 did not reproduce here. Drop the F12 carries when upstream merges #32735 and #32740.
- Open after the 2026-11-21 rebase: is the fork's concat fast path (F3) now redundant with upstream's
own tiled concat (#30591, gated in the fork with `ne0 < 32` and the `GGML_VK_CONCAT_FASTPATH` knob)?
Benchmark it. Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no
+30 -7
View File
@@ -303,6 +303,11 @@ Add `--arbiter` to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus
warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run
`--arbiter` in the phase-2 swapdown window instead (about 2 minutes more downtime), as below.
The same applies to `--concat`'s `candidate-timing` gate: with the card nearly full (2026-11-22: 956
MiB free beside a warm Inkling) every concat arm ran about 2x slower and the fast path showed no
gain (off/cand 0.9x, FAIL); in the swapdown window the same build passed with 5.1x. A phase-1
`candidate-timing` FAIL with little free VRAM is not a verdict: rerun `--concat` in phase 2.
**Page cache before and after phase 2.** The `--ref` build may lack F10 (FORK.md), and `--perf`
loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the
eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check
@@ -343,12 +348,19 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- Last full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
- Last full run (2026-11-22, candidate `3621025e8` = `3bf6b55b3` + the F12 carries, vs reference
`3bf6b55b3`; logs `/root/arms/carry1122-p1.log`, `-p2.log`): phase 1 `--ops --concat --diskcache`
20 PASS, 1 FAIL (`concat/candidate-timing` beside a full card, see above), 1 SKIP (diskcache GPU
leg); phase 2 `--concat --arbiter --diskcache --perf --moe-exact --moecache` 47 PASS, 0 FAIL, 0
SKIP. Perf medians ref -> cand: pp2048 286.97 -> 286.32, d16384 208.31 -> 208.21, d32768 162.25 ->
162.37; handoff median ref 696 / cand 700 ms; peak VRAM ref 7771 / cand 5502 MiB. Plus SCALE
15/15 and `test-flash-attn-indexed-bias` (FORK.md F12). Window about 30 minutes.
- Previous full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
9 PASS plus the 35B disk-cache legs (`q35.py`, mmproj loaded). Perf medians ref -> cand: pp2048
259.39 -> 286.61 (+10.5%), d16384 199.26 -> 207.63 (+4.2%), d32768 156.61 -> 157.51 (+0.6%),
tg256@d4096 9.17 -> 8.85 t/s (noisy, not gated). Handoff median: candidate 692 ms, reference
1068 ms. Downtime: about 42 minutes of validation and about 7 minutes of deploy.
- Previous full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs
- Earlier full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs
`/root/arms/dep-2fc9f9c35-fv1.log` and `-fv2.log`): phase 1 `--ops --concat` 8 PASS, phase 2
`--arbiter --perf --moe-exact --moecache` 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048
274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff
@@ -392,8 +404,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 13 (holds adc2aa3ec)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-14.txt
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 14 (holds 3bf6b55b3)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-15.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3):
@@ -530,6 +542,17 @@ on `reasoning_content` and returns an empty `content`; for one, raise `max_token
- OOM kills, in the pod: `dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'`. The
timestamps are seconds since boot; compare them with the first field of `/proc/uptime` to see
whether any falls in the deploy window (2026-10-25: the newest was about 19 h old).
8. Inkling (FORK.md section 5, F12): judge the answer text, not only HTTP 200. The first request
after a fresh load should be a prompt of about 150 tokens (upstream #32778 gives all-'?' output
there); then a long prompt (Pan Tadeusz, 8K and 14K tokens, upstream #32755). Image and tool
requests go to the think entry: the nothink entry returns repeated
`<|message_model|><|content_thinking|><|end_message|>` for them on every build (FORK.md F12).
Check that the env of the running process has `GGML_OP_OFFLOAD_MIN_BATCH=64`
(`tr '\0' '\n' < /proc/<pid>/environ`). The memory breakdown table is printed only at `-lv 4`,
which production does not set, so it is not in the llama-swap log: read it from a scratch server
started with production's command line plus `-lv 4`. Drivers used on 2026-11-22 (pod only):
`/root/arms/carry1122-post.py` (through llama-swap) and `/root/arms/carry1122-ink-direct.py`
(scratch server, GPU to itself).
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
@@ -578,9 +601,9 @@ chain in the exec shell: it can die with the exec session halfway, leaving llama
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
Pointer files 2 to 13 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, and commit-12 holds
`2fc9f9c35`, and commit-13 holds `adc2aa3ec`, the rollback target for the current production
`3bf6b55b3` (next free: 14). The formats vary
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, commit-12 holds
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, and commit-14 holds `3bf6b55b3`, the rollback target
for the current production `3621025e8` (next free: 15). The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.