From 1a19c3bbba35ebd54be7f496bb86743e2f41792c Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Sun, 22 Nov 2026 05:38:11 +0100 Subject: [PATCH] docs: FORK.md/runbook after the 3621025e8 deploy - F12 measurements, Inkling ub1024 + ctx 65536 and the offload knob, nothink finding, pointer 14 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ --- FORK.md | 71 ++++++++++++++++++++++++++++--------- docs/fork-rebase-runbook.md | 37 +++++++++++++++---- 2 files changed, 85 insertions(+), 23 deletions(-) diff --git a/FORK.md b/FORK.md index de71003f0..433830d46 100644 --- a/FORK.md +++ b/FORK.md @@ -4,14 +4,17 @@ Maintainer documentation for this private fork. Read it before you rebase, chang anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md), which also holds the operations reference (pod access, files, recovery). -Status at the time of writing (2026-11-21): +Status at the time of writing (2026-11-22): -- Production runs `3bf6b55b3` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`, - 757 upstream commits), deployed 2026-11-21 19:54, validated with the full `fork-validate.sh` set - (phase 1 39 PASS, phase 2 9 PASS, plus the 35B disk-cache legs; runbook section 6). Rollback - pointer `/root/DEPLOY-ROLLBACK-commit-13.txt` = `adc2aa3ec`; next free pointer 14. - `system_fingerprint` is `b-` (currently `b12676-3bf6b55b3`) and changes - with each rebase. The hand ports are listed in the runbook (section 3, "2026-11-21"). +- Production runs `3621025e8` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`, + 757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12), deployed + 2026-11-22 03:52 with the Inkling retune (section 5). Validated with the full `fork-validate.sh` + set (phase 2 47 PASS, 0 FAIL; runbook section 6). Rollback pointer + `/root/DEPLOY-ROLLBACK-commit-14.txt` = `3bf6b55b3` (build copy `/root/arms/build-3bf6b55b3`, + config backup `/root/config.yaml.bak-20261122-inkling-ub1024-ctx65536`); next free pointer 15. + `system_fingerprint` is `b-` (currently `b12680-3621025e8`) and changes + with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook + (section 3, "2026-11-21"). - Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production `adc2aa3ec` until 2026-11-21), kept untouched as the safety net. The fork commits are `git log --oneline origin/master..HEAD`; that list includes documentation and script commits @@ -1122,8 +1125,33 @@ without `808fd1c4e`; base `3bf6b55b3` vs base + both PRs; logs `/root/arms/pr-ou gemma-26B 249.4 both. - test-backend-ops on Vulkan0: SCALE 15/15, FLASH_ATTN_EXT 5693/5693. `808fd1c4e` (the PR head since 2026-11-22) changes the table layout from `[n_head, n_bins, n_tokens]` -to `[n_bins, n_head, n_tokens]`, which removes one permute+cont from the Inkling graph; it was -validated with `tests/test-flash-attn-indexed-bias` and test-backend-ops, not re-benchmarked. +to `[n_bins, n_head, n_tokens]`, which removes one permute+cont from the Inkling graph. + +Measured with the carried head (2026-11-22, candidate `3621025e8` vs production `3bf6b55b3`, +production's Inkling command line with the nothink sampling, GPU to itself; driver +`/root/arms/carry1122-ink-direct.py`, logs `/root/arms/carry1122-p2.log`, `-p3.log`): + +| build, args | 8.2K-token prefill | 13.9K | 28.4K | process VRAM / GTT after load | +|---|---|---|---|---| +| production, ub512, ctx 65536 | 32.7 t/s | 31.6 | 29.5 | 7209 / 28113 MiB | +| carry, ub1024, ctx 131072 | 43.7 | 42.5 | 42.5 | 7983 / 21835 MiB | +| carry, ub1024, ctx 65536 | 46.8 | 46.9 | 46.0 | 7368 / 19629 MiB | + +- Device memory wanted (`-lv 4` breakdown): production ub512/65536 19424 MiB (compute 15866); carry + ub1024/131072 13388 MiB (model 1266, context 4419, compute 7702), host compute 798 MiB. Everything + above the 8 GB card lands in GTT (system RAM over PCIe): about 5.2 GB at 131072. That costs about + 8% prefill against 65536, so production runs ub1024 with ctx 65536 (section 5) until the loop-layer + experts move into VRAM, which needs that headroom. +- Answers: a ~160-token first request on a fresh server (upstream #32778, all-'?' output) was + correct on every build. Pan Tadeusz fragments of 8.2K and 13.9K tokens gave coherent, correct + summaries on every build (upstream #32755 did not reproduce). The nothink entry + (`--reasoning-effort none --reasoning-budget 0`) returns repeated + `<|message_model|><|content_thinking|><|end_message|>` as content, with no tool call, for an image + request, a tool call and a 28.4K-token weather log ("which station is in Record 5?"). This happens + on every build, production `3bf6b55b3` with ub512 included, with and without + `GGML_OP_OFFLOAD_MIN_BATCH=64`, so the carry did not cause it. The think entry gets all three + right on both builds (image "Red", `get_weather({"city":"Warsaw"})`, 28.4K log "65" at 45.9 t/s + in production). Short text prompts work on the nothink entry. Open: fix the nothink macro. **Rebase hazards.** - At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry @@ -1140,7 +1168,9 @@ validated with `tests/test-flash-attn-indexed-bias` and test-backend-ops, not re **Validation.** `test-backend-ops -b Vulkan0 -o SCALE` and `-o FLASH_ATTN_EXT` (check the case count), and the PR's own test `tests/test-flash-attn-indexed-bias` (build target `test-flash-attn-indexed-bias`; it runs every device, prints one line per case it ran and skips -unsupported cases silently, so count the `Vulkan0:` lines; the Q4_0 K/V cases run on the CPU only). +unsupported cases silently, so count the `Vulkan0:` lines: 16 f16 cases on the RX 580, max error +about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rejects indexed bias +(the graph then uses the dense mask), so it prints no CPU lines). Then an Inkling request through llama-swap. --- @@ -1192,7 +1222,7 @@ layers. ## 5. Configuration that matters `/root/config.yaml` (llama-swap) and `/root/supervisord.conf` on the pod are the source of truth; -this section explains the choices. Values as of the 20261024 deploy. +this section explains the choices. Values as of the 2026-11-22 deploy. Qwen 35B-A3B MoE entries (macro `moe_offload_args`, applied to all 35B-A3B variants): `--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1`, @@ -1232,10 +1262,15 @@ Other models: matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m` (the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it, `--n-cpu-moe 24` (all experts on host), `-t 6`, f16 KV, `--parallel 1`, the disk prompt cache. - Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 131072` - (before: `-b 512 -ub 512 --ctx-size 65536`, because the dense per-head bias mask needed about - 12.7 GB of host compute buffer at 128k). Its nothink macro uses `--reasoning-effort none - --reasoning-budget 0`. + Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 65536` + (before: `-b 512 -ub 512`; the dense per-head bias mask needed about 12.7 GB of host compute + buffer at 128k). 131072 loads and works with the carry, but overcommits about 5.2 GB of device + memory into GTT and is about 8% slower at prefill (F12). Both Inkling entries set + `env: ["GGML_OP_OFFLOAD_MIN_BATCH=64"]` (llama-swap per-model env): ubatches of 32-63 tokens keep + MUL_MAT_ID on the CPU. The looped layers share their expert tensors, and below F1's readback-skip + threshold (4 x 128 experts / 8 used = 64 tokens) only loop pass 0's routed experts would be + uploaded (the mechanism suspected behind upstream #32755). Its nothink macro uses + `--reasoning-effort none --reasoning-budget 0`. Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of `/root/supervisord.conf`: @@ -1249,7 +1284,7 @@ Environment, set once for all of llama-swap in `[program:llama-swap] environment Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`, `GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`. -`--cache-disk-path` (F11) is not set yet: it is not deployed; the recommended flags are in F11. +`--cache-disk-path /root/prompt-cache --cache-disk-size 4096` (F11) is set in the `moe_offload_args`, `moe_offload_gemma` and `inkling_args` macros. Machine-level: - GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted @@ -1310,6 +1345,10 @@ Machine-level: `device_resident()` stays false, decode retries the restore on every call, and a graph reading a null buffer would crash. Only the mmproj path handles a failed restore. - Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4). +- Inkling (F12, section 5): the nothink entry returns repeated chat-structure tokens instead of an + answer for image requests, tool calls and long prompts (the think entry is fine); the config runs + ctx 65536 because 131072 overcommits about 5.2 GB of device memory into GTT; upstream #32755 / + #32778 did not reproduce here. Drop the F12 carries when upstream merges #32735 and #32740. - Open after the 2026-11-21 rebase: is the fork's concat fast path (F3) now redundant with upstream's own tiled concat (#30591, gated in the fork with `ne0 < 32` and the `GGML_VK_CONCAT_FASTPATH` knob)? Benchmark it. Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md index 17b107ff1..ce41fb003 100644 --- a/docs/fork-rebase-runbook.md +++ b/docs/fork-rebase-runbook.md @@ -303,6 +303,11 @@ Add `--arbiter` to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run `--arbiter` in the phase-2 swapdown window instead (about 2 minutes more downtime), as below. +The same applies to `--concat`'s `candidate-timing` gate: with the card nearly full (2026-11-22: 956 +MiB free beside a warm Inkling) every concat arm ran about 2x slower and the fast path showed no +gain (off/cand 0.9x, FAIL); in the swapdown window the same build passed with 5.1x. A phase-1 +`candidate-timing` FAIL with little free VRAM is not a verdict: rerun `--concat` in phase 2. + **Page cache before and after phase 2.** The `--ref` build may lack F10 (FORK.md), and `--perf` loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check @@ -343,12 +348,19 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f `/perf.tsv`, and `fork-validate.sh --perf-report ` re-aggregates them offline. A pod restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24 scratch dirs were lost that way). -- Last full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2 +- Last full run (2026-11-22, candidate `3621025e8` = `3bf6b55b3` + the F12 carries, vs reference + `3bf6b55b3`; logs `/root/arms/carry1122-p1.log`, `-p2.log`): phase 1 `--ops --concat --diskcache` + 20 PASS, 1 FAIL (`concat/candidate-timing` beside a full card, see above), 1 SKIP (diskcache GPU + leg); phase 2 `--concat --arbiter --diskcache --perf --moe-exact --moecache` 47 PASS, 0 FAIL, 0 + SKIP. Perf medians ref -> cand: pp2048 286.97 -> 286.32, d16384 208.31 -> 208.21, d32768 162.25 -> + 162.37; handoff median ref 696 / cand 700 ms; peak VRAM ref 7771 / cand 5502 MiB. Plus SCALE + 15/15 and `test-flash-attn-indexed-bias` (FORK.md F12). Window about 30 minutes. +- Previous full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2 9 PASS plus the 35B disk-cache legs (`q35.py`, mmproj loaded). Perf medians ref -> cand: pp2048 259.39 -> 286.61 (+10.5%), d16384 199.26 -> 207.63 (+4.2%), d32768 156.61 -> 157.51 (+0.6%), tg256@d4096 9.17 -> 8.85 t/s (noisy, not gated). Handoff median: candidate 692 ms, reference 1068 ms. Downtime: about 42 minutes of validation and about 7 minutes of deploy. -- Previous full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs +- Earlier full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs `/root/arms/dep-2fc9f9c35-fv1.log` and `-fv2.log`): phase 1 `--ops --concat` 8 PASS, phase 2 `--arbiter --perf --moe-exact --moecache` 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048 274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff @@ -392,8 +404,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with 1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9): ```sh - ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 13 (holds adc2aa3ec) - git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-14.txt + ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 14 (holds 3bf6b55b3) + git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-15.txt git -C /root/llama.cpp status --short # must be clean ``` 2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3): @@ -530,6 +542,17 @@ on `reasoning_content` and returns an empty `content`; for one, raise `max_token - OOM kills, in the pod: `dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'`. The timestamps are seconds since boot; compare them with the first field of `/proc/uptime` to see whether any falls in the deploy window (2026-10-25: the newest was about 19 h old). +8. Inkling (FORK.md section 5, F12): judge the answer text, not only HTTP 200. The first request + after a fresh load should be a prompt of about 150 tokens (upstream #32778 gives all-'?' output + there); then a long prompt (Pan Tadeusz, 8K and 14K tokens, upstream #32755). Image and tool + requests go to the think entry: the nothink entry returns repeated + `<|message_model|><|content_thinking|><|end_message|>` for them on every build (FORK.md F12). + Check that the env of the running process has `GGML_OP_OFFLOAD_MIN_BATCH=64` + (`tr '\0' '\n' < /proc//environ`). The memory breakdown table is printed only at `-lv 4`, + which production does not set, so it is not in the llama-swap log: read it from a scratch server + started with production's command line plus `-lv 4`. Drivers used on 2026-11-22 (pod only): + `/root/arms/carry1122-post.py` (through llama-swap) and `/root/arms/carry1122-ink-direct.py` + (scratch server, GPU to itself). A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks` shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child @@ -578,9 +601,9 @@ chain in the exec shell: it can die with the exec session halfway, leaving llama `/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`. Pointer files 2 to 13 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`, -commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, and commit-12 holds -`2fc9f9c35`, and commit-13 holds `adc2aa3ec`, the rollback target for the current production -`3bf6b55b3` (next free: 14). The formats vary +commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, commit-12 holds +`2fc9f9c35`, commit-13 holds `adc2aa3ec`, and commit-14 holds `3bf6b55b3`, the rollback target +for the current production `3621025e8` (next free: 15). The formats vary (commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.