docs: FORK.md/runbook after the 3621025e8 deploy - F12 measurements, Inkling ub1024 + ctx 65536 and the offload knob, nothink finding, pointer 14
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
1 parent
01812e8b02
commit
1a19c3bbba
2 files changed
+85
-23
No files matched your search
@@ -4,14 +4,17 @@ Maintainer documentation for this private fork. Read it before you rebase, chang
|
||||
anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md),
|
||||
which also holds the operations reference (pod access, files, recovery).
|
||||
|
||||
Status at the time of writing (2026-11-21):
|
||||
Status at the time of writing (2026-11-22):
|
||||
|
||||
- Production runs `3bf6b55b3` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
||||
757 upstream commits), deployed 2026-11-21 19:54, validated with the full `fork-validate.sh` set
|
||||
(phase 1 39 PASS, phase 2 9 PASS, plus the 35B disk-cache legs; runbook section 6). Rollback
|
||||
pointer `/root/DEPLOY-ROLLBACK-commit-13.txt` = `adc2aa3ec`; next free pointer 14.
|
||||
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12676-3bf6b55b3`) and changes
|
||||
with each rebase. The hand ports are listed in the runbook (section 3, "2026-11-21").
|
||||
- Production runs `3621025e8` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
||||
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12), deployed
|
||||
2026-11-22 03:52 with the Inkling retune (section 5). Validated with the full `fork-validate.sh`
|
||||
set (phase 2 47 PASS, 0 FAIL; runbook section 6). Rollback pointer
|
||||
`/root/DEPLOY-ROLLBACK-commit-14.txt` = `3bf6b55b3` (build copy `/root/arms/build-3bf6b55b3`,
|
||||
config backup `/root/config.yaml.bak-20261122-inkling-ub1024-ctx65536`); next free pointer 15.
|
||||
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12680-3621025e8`) and changes
|
||||
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
|
||||
(section 3, "2026-11-21").
|
||||
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
|
||||
`adc2aa3ec` until 2026-11-21), kept untouched as the safety net. The fork commits are
|
||||
`git log --oneline origin/master..HEAD`; that list includes documentation and script commits
|
||||
@@ -1122,8 +1125,33 @@ without `808fd1c4e`; base `3bf6b55b3` vs base + both PRs; logs `/root/arms/pr-ou
|
||||
gemma-26B 249.4 both.
|
||||
- test-backend-ops on Vulkan0: SCALE 15/15, FLASH_ATTN_EXT 5693/5693.
|
||||
`808fd1c4e` (the PR head since 2026-11-22) changes the table layout from `[n_head, n_bins, n_tokens]`
|
||||
to `[n_bins, n_head, n_tokens]`, which removes one permute+cont from the Inkling graph; it was
|
||||
validated with `tests/test-flash-attn-indexed-bias` and test-backend-ops, not re-benchmarked.
|
||||
to `[n_bins, n_head, n_tokens]`, which removes one permute+cont from the Inkling graph.
|
||||
|
||||
Measured with the carried head (2026-11-22, candidate `3621025e8` vs production `3bf6b55b3`,
|
||||
production's Inkling command line with the nothink sampling, GPU to itself; driver
|
||||
`/root/arms/carry1122-ink-direct.py`, logs `/root/arms/carry1122-p2.log`, `-p3.log`):
|
||||
|
||||
| build, args | 8.2K-token prefill | 13.9K | 28.4K | process VRAM / GTT after load |
|
||||
|---|---|---|---|---|
|
||||
| production, ub512, ctx 65536 | 32.7 t/s | 31.6 | 29.5 | 7209 / 28113 MiB |
|
||||
| carry, ub1024, ctx 131072 | 43.7 | 42.5 | 42.5 | 7983 / 21835 MiB |
|
||||
| carry, ub1024, ctx 65536 | 46.8 | 46.9 | 46.0 | 7368 / 19629 MiB |
|
||||
|
||||
- Device memory wanted (`-lv 4` breakdown): production ub512/65536 19424 MiB (compute 15866); carry
|
||||
ub1024/131072 13388 MiB (model 1266, context 4419, compute 7702), host compute 798 MiB. Everything
|
||||
above the 8 GB card lands in GTT (system RAM over PCIe): about 5.2 GB at 131072. That costs about
|
||||
8% prefill against 65536, so production runs ub1024 with ctx 65536 (section 5) until the loop-layer
|
||||
experts move into VRAM, which needs that headroom.
|
||||
- Answers: a ~160-token first request on a fresh server (upstream #32778, all-'?' output) was
|
||||
correct on every build. Pan Tadeusz fragments of 8.2K and 13.9K tokens gave coherent, correct
|
||||
summaries on every build (upstream #32755 did not reproduce). The nothink entry
|
||||
(`--reasoning-effort none --reasoning-budget 0`) returns repeated
|
||||
`<|message_model|><|content_thinking|><|end_message|>` as content, with no tool call, for an image
|
||||
request, a tool call and a 28.4K-token weather log ("which station is in Record 5?"). This happens
|
||||
on every build, production `3bf6b55b3` with ub512 included, with and without
|
||||
`GGML_OP_OFFLOAD_MIN_BATCH=64`, so the carry did not cause it. The think entry gets all three
|
||||
right on both builds (image "Red", `get_weather({"city":"Warsaw"})`, 28.4K log "65" at 45.9 t/s
|
||||
in production). Short text prompts work on the nothink entry. Open: fix the nothink macro.
|
||||
|
||||
**Rebase hazards.**
|
||||
- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry
|
||||
@@ -1140,7 +1168,9 @@ validated with `tests/test-flash-attn-indexed-bias` and test-backend-ops, not re
|
||||
**Validation.** `test-backend-ops -b Vulkan0 -o SCALE` and `-o FLASH_ATTN_EXT` (check the case
|
||||
count), and the PR's own test `tests/test-flash-attn-indexed-bias` (build target
|
||||
`test-flash-attn-indexed-bias`; it runs every device, prints one line per case it ran and skips
|
||||
unsupported cases silently, so count the `Vulkan0:` lines; the Q4_0 K/V cases run on the CPU only).
|
||||
unsupported cases silently, so count the `Vulkan0:` lines: 16 f16 cases on the RX 580, max error
|
||||
about 1e-4. The 4 Q4_0 K/V cases run on no backend here, and the CPU backend rejects indexed bias
|
||||
(the graph then uses the dense mask), so it prints no CPU lines).
|
||||
Then an Inkling request through llama-swap.
|
||||
|
||||
---
|
||||
@@ -1192,7 +1222,7 @@ layers.
|
||||
## 5. Configuration that matters
|
||||
|
||||
`/root/config.yaml` (llama-swap) and `/root/supervisord.conf` on the pod are the source of truth;
|
||||
this section explains the choices. Values as of the 20261024 deploy.
|
||||
this section explains the choices. Values as of the 2026-11-22 deploy.
|
||||
|
||||
Qwen 35B-A3B MoE entries (macro `moe_offload_args`, applied to all 35B-A3B variants):
|
||||
`--no-warmup --port ${PORT} -lm none -tb 12 -t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 --parallel 1`,
|
||||
@@ -1232,10 +1262,15 @@ Other models:
|
||||
matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m`
|
||||
(the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it,
|
||||
`--n-cpu-moe 24` (all experts on host), `-t 6`, f16 KV, `--parallel 1`, the disk prompt cache.
|
||||
Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 131072`
|
||||
(before: `-b 512 -ub 512 --ctx-size 65536`, because the dense per-head bias mask needed about
|
||||
12.7 GB of host compute buffer at 128k). Its nothink macro uses `--reasoning-effort none
|
||||
--reasoning-budget 0`.
|
||||
Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 65536`
|
||||
(before: `-b 512 -ub 512`; the dense per-head bias mask needed about 12.7 GB of host compute
|
||||
buffer at 128k). 131072 loads and works with the carry, but overcommits about 5.2 GB of device
|
||||
memory into GTT and is about 8% slower at prefill (F12). Both Inkling entries set
|
||||
`env: ["GGML_OP_OFFLOAD_MIN_BATCH=64"]` (llama-swap per-model env): ubatches of 32-63 tokens keep
|
||||
MUL_MAT_ID on the CPU. The looped layers share their expert tensors, and below F1's readback-skip
|
||||
threshold (4 x 128 experts / 8 used = 64 tokens) only loop pass 0's routed experts would be
|
||||
uploaded (the mechanism suspected behind upstream #32755). Its nothink macro uses
|
||||
`--reasoning-effort none --reasoning-budget 0`.
|
||||
|
||||
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
|
||||
`/root/supervisord.conf`:
|
||||
@@ -1249,7 +1284,7 @@ Environment, set once for all of llama-swap in `[program:llama-swap] environment
|
||||
|
||||
Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`,
|
||||
`GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`.
|
||||
`--cache-disk-path` (F11) is not set yet: it is not deployed; the recommended flags are in F11.
|
||||
`--cache-disk-path /root/prompt-cache --cache-disk-size 4096` (F11) is set in the `moe_offload_args`, `moe_offload_gemma` and `inkling_args` macros.
|
||||
|
||||
Machine-level:
|
||||
- GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted
|
||||
@@ -1310,6 +1345,10 @@ Machine-level:
|
||||
`device_resident()` stays false, decode retries the restore on every call, and a graph reading a
|
||||
null buffer would crash. Only the mmproj path handles a failed restore.
|
||||
- Memory types without a `release_device_buffers` override keep their KV under `evict_kv` (F4).
|
||||
- Inkling (F12, section 5): the nothink entry returns repeated chat-structure tokens instead of an
|
||||
answer for image requests, tool calls and long prompts (the think entry is fine); the config runs
|
||||
ctx 65536 because 131072 overcommits about 5.2 GB of device memory into GTT; upstream #32755 /
|
||||
#32778 did not reproduce here. Drop the F12 carries when upstream merges #32735 and #32740.
|
||||
- Open after the 2026-11-21 rebase: is the fork's concat fast path (F3) now redundant with upstream's
|
||||
own tiled concat (#30591, gated in the fork with `ne0 < 32` and the `GGML_VK_CONCAT_FASTPATH` knob)?
|
||||
Benchmark it. Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no
|
||||
|
||||
@@ -303,6 +303,11 @@ Add `--arbiter` to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus
|
||||
warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run
|
||||
`--arbiter` in the phase-2 swapdown window instead (about 2 minutes more downtime), as below.
|
||||
|
||||
The same applies to `--concat`'s `candidate-timing` gate: with the card nearly full (2026-11-22: 956
|
||||
MiB free beside a warm Inkling) every concat arm ran about 2x slower and the fast path showed no
|
||||
gain (off/cand 0.9x, FAIL); in the swapdown window the same build passed with 5.1x. A phase-1
|
||||
`candidate-timing` FAIL with little free VRAM is not a verdict: rerun `--concat` in phase 2.
|
||||
|
||||
**Page cache before and after phase 2.** The `--ref` build may lack F10 (FORK.md), and `--perf`
|
||||
loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the
|
||||
eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check
|
||||
@@ -343,12 +348,19 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
|
||||
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
|
||||
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
|
||||
scratch dirs were lost that way).
|
||||
- Last full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
|
||||
- Last full run (2026-11-22, candidate `3621025e8` = `3bf6b55b3` + the F12 carries, vs reference
|
||||
`3bf6b55b3`; logs `/root/arms/carry1122-p1.log`, `-p2.log`): phase 1 `--ops --concat --diskcache`
|
||||
20 PASS, 1 FAIL (`concat/candidate-timing` beside a full card, see above), 1 SKIP (diskcache GPU
|
||||
leg); phase 2 `--concat --arbiter --diskcache --perf --moe-exact --moecache` 47 PASS, 0 FAIL, 0
|
||||
SKIP. Perf medians ref -> cand: pp2048 286.97 -> 286.32, d16384 208.31 -> 208.21, d32768 162.25 ->
|
||||
162.37; handoff median ref 696 / cand 700 ms; peak VRAM ref 7771 / cand 5502 MiB. Plus SCALE
|
||||
15/15 and `test-flash-attn-indexed-bias` (FORK.md F12). Window about 30 minutes.
|
||||
- Previous full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
|
||||
9 PASS plus the 35B disk-cache legs (`q35.py`, mmproj loaded). Perf medians ref -> cand: pp2048
|
||||
259.39 -> 286.61 (+10.5%), d16384 199.26 -> 207.63 (+4.2%), d32768 156.61 -> 157.51 (+0.6%),
|
||||
tg256@d4096 9.17 -> 8.85 t/s (noisy, not gated). Handoff median: candidate 692 ms, reference
|
||||
1068 ms. Downtime: about 42 minutes of validation and about 7 minutes of deploy.
|
||||
- Previous full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs
|
||||
- Earlier full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs
|
||||
`/root/arms/dep-2fc9f9c35-fv1.log` and `-fv2.log`): phase 1 `--ops --concat` 8 PASS, phase 2
|
||||
`--arbiter --perf --moe-exact --moecache` 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048
|
||||
274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff
|
||||
@@ -392,8 +404,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
|
||||
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
|
||||
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
|
||||
```sh
|
||||
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 13 (holds adc2aa3ec)
|
||||
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-14.txt
|
||||
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 14 (holds 3bf6b55b3)
|
||||
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-15.txt
|
||||
git -C /root/llama.cpp status --short # must be clean
|
||||
```
|
||||
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3):
|
||||
@@ -530,6 +542,17 @@ on `reasoning_content` and returns an empty `content`; for one, raise `max_token
|
||||
- OOM kills, in the pod: `dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'`. The
|
||||
timestamps are seconds since boot; compare them with the first field of `/proc/uptime` to see
|
||||
whether any falls in the deploy window (2026-10-25: the newest was about 19 h old).
|
||||
8. Inkling (FORK.md section 5, F12): judge the answer text, not only HTTP 200. The first request
|
||||
after a fresh load should be a prompt of about 150 tokens (upstream #32778 gives all-'?' output
|
||||
there); then a long prompt (Pan Tadeusz, 8K and 14K tokens, upstream #32755). Image and tool
|
||||
requests go to the think entry: the nothink entry returns repeated
|
||||
`<|message_model|><|content_thinking|><|end_message|>` for them on every build (FORK.md F12).
|
||||
Check that the env of the running process has `GGML_OP_OFFLOAD_MIN_BATCH=64`
|
||||
(`tr '\0' '\n' < /proc/<pid>/environ`). The memory breakdown table is printed only at `-lv 4`,
|
||||
which production does not set, so it is not in the llama-swap log: read it from a scratch server
|
||||
started with production's command line plus `-lv 4`. Drivers used on 2026-11-22 (pod only):
|
||||
`/root/arms/carry1122-post.py` (through llama-swap) and `/root/arms/carry1122-ink-direct.py`
|
||||
(scratch server, GPU to itself).
|
||||
|
||||
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
|
||||
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
|
||||
@@ -578,9 +601,9 @@ chain in the exec shell: it can die with the exec session halfway, leaving llama
|
||||
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
|
||||
|
||||
Pointer files 2 to 13 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
|
||||
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, and commit-12 holds
|
||||
`2fc9f9c35`, and commit-13 holds `adc2aa3ec`, the rollback target for the current production
|
||||
`3bf6b55b3` (next free: 14). The formats vary
|
||||
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, commit-12 holds
|
||||
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, and commit-14 holds `3bf6b55b3`, the rollback target
|
||||
for the current production `3621025e8` (next free: 15). The formats vary
|
||||
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
|
||||
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
|
||||
|
||||
|
||||
Reference in new issue
Block a user