diff --git a/FORK.md b/FORK.md index b969764f0..b844fcc69 100644 --- a/FORK.md +++ b/FORK.md @@ -4,19 +4,20 @@ Maintainer documentation for this private fork. Read it before you rebase, chang anything. The rebase and deploy procedure is in [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md), which also holds the operations reference (pod access, files, recovery). -Status at the time of writing (2026-10-25): +Status at the time of writing (2026-11-21): - Branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772`. The fork commits are `git log --oneline origin/master..HEAD`; that list includes documentation and script commits (this file among them), so count it rather than trusting a number written here. Section 2 maps every commit to a feature. -- Production runs `2fc9f9c35` (deployed 2026-10-25 13:01 UTC, `system_fingerprint` - `b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the - previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook - section 6). Any later commit that touches compiled code needs the same validation before a redeploy. -- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on - top. A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); production stays on - `2fc9f9c35`. The fix is on the branch; the 35B run with the production command line is still due. +- Production runs `adc2aa3ec` (system_fingerprint `b11906-adc2aa3ec`; rollback pointer + `/root/DEPLOY-ROLLBACK-commit-12.txt` = `2fc9f9c35`, the previous production; next free pointer 13), + with the disk prompt cache ON (F11). `2fc9f9c35` was validated with the full `fork-validate.sh` set + (37 PASS, 0 FAIL, runbook section 6). Any later commit that touches compiled code needs the same + validation before a redeploy. +- Branch `fable5/disk-prompt-cache` added F11 (disk prompt cache) and a CPU_REPACK release fix (F4). + A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); the fix (adc2aa3ec) passed + the 35B run with the production command line on 2026-11-21 and is deployed. - Commit hashes below refer to that branch. They change on every rebase; find a commit again by its subject line. The commit messages are detailed and are the primary record of each mechanism and measurement; read them with `git show`. @@ -1039,8 +1040,11 @@ Swapdown window (2026-10-25, started by the owner, `/root/arms/diskcache-window. incident above; `--cache-disk-path` on a scratch dir, arbiter env, an 18760-token pan-tadeusz chat; log `/root/arms/diskcache-test/q35.log`). `q35.py` now uses the exact production entry (`-hf`, so the mmproj loads, nothing extra). That run (`/root/arms/diskcache-window3.sh`, about 8 minutes under -swapdown, which also covers the same-reuse-point identity and an image request on the 35B) is -ready and not run: +swapdown, which also covers the same-reuse-point identity and an image request on the 35B) was run +2026-11-21 on adc2aa3ec (log `/root/arms/diskcache-test/q35-mm.log`) and passed all legs: 18.8k-token +chat, prefill 84.6 s -> 4.7 s for turn 2 after the restart with identical output; save 557 MB in +1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image answered and not saved. The +table below is the earlier `--no-mmproj` run: | leg | result | |---|---| @@ -1060,10 +1064,9 @@ Two anomalies in leg c, both explained: cold save and a second restore. Z == X, text and top-3 logprobs bit-identical, on CPU and on the GPU (`-ngl 99`); turn 2 and turn 2 again also agree there. The same check on the 35B needs the GPU to itself: `/root/arms/diskcache-window3.sh` (production command line, reference "turn 2 again" against - leg c; about 8 minutes under swapdown) is ready and not run. It supersedes `diskcache-window2.sh`, - whose `cmp.py` runs with `--no-mmproj`. - Until it runs, the 35B difference is attributed to the reuse point (a different batch split on the - GPU), not proven on that model. + leg c; about 8 minutes under swapdown). It ran 2026-11-21 and passed (same-reuse-point identity: + identical output), so the difference above is the reuse point. It supersedes + `diskcache-window2.sh`, whose `cmp.py` runs with `--no-mmproj`. - Exit peak RSS +3055 MiB and token free in 1881 ms (leg a: +430 MiB / 328 ms): a test harness bug. The driver's fake waiter was `flock -s want.lock sleep 120`; killing flock left `sleep` holding want.lock (flock(1) keeps the lock fd open in its child). So c had a permanent waiter, slept right @@ -1072,9 +1075,12 @@ Two anomalies in leg c, both explained: arrived during it. The cache's own save then ran as a cold save (copy 305 ms, no extra copy). The driver now holds the lock in the process it kills, like `fork-validate.sh` does. -Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache ---cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is -created on first start. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still +Configuration (applied in production 2026-11-21): `--cache-disk-path /root/prompt-cache +--cache-disk-size 4096` appended to the `moe_offload_args` and `moe_offload_gemma` macros in +`/root/config.yaml` (backup `/root/config.yaml.bak-diskcache-20261121`). The directory is created on +first start. Verified in production: a 35B 4.6k-token chat was saved on eviction (278.6 MiB, copy +126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms), and turn 2 prefilled +only 32 tokens; a gemma image request with the flags on works and is not cached. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still applies. Models only pick up the flags when they restart. --- @@ -1242,10 +1248,10 @@ Machine-level: Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them. - F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate; its memory effect is verified in production. -- F11 (disk prompt cache): the same-reuse-point identity check (F11, Validation) has not run on the - 35B, only on the 0.8B (CPU and GPU), and no 35B run has used production's command line (mmproj - loaded) yet. `/root/arms/diskcache-window3.sh` covers both (about 8 minutes of downtime). Do not - deploy `--cache-disk-path` before it passes. +- F11 (disk prompt cache): the same-reuse-point identity check and the production-command-line (mmproj + loaded) 35B run both passed on 2026-11-21 (`/root/arms/diskcache-window3.sh`, F11 Validation), and + the cache is on in production. Remaining gap: only the 35B and gemma were checked in production; + other models gain the flag only if they use those macros. - A SIGTERM during an arbiter release with `LLAMA_SLEEP_EVICT_KV` (F4, not F11) pays that release's host shadow of the full KV allocation (about 2.6 GB for the 35B at ctx 131072) and its D2H time (token free in about 1.9 s) before the exit. The same is true of every cold 35B, which holds that diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md index b7d5691be..8c4d757ff 100644 --- a/docs/fork-rebase-runbook.md +++ b/docs/fork-rebase-runbook.md @@ -184,6 +184,7 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c ' git -C /root/llama.cpp fetch origin "$B" git -C /root/llama.cpp worktree add "$W" FETCH_HEAD cd "$W" + export HF_UI_VERSION=b11906 # production's stamp: cat /root/llama.cpp/build/tools/ui/.ui-stamp cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8 ' @@ -196,8 +197,12 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c ' - Do not overlap a build with a large benchmark (host OOM). Disk is tight: `/root` had about 13 GB free (97% full) on 2026-10-25, so check `df -h /root` first. Each build tree is about 700 MB; remove scratch worktrees you no longer need with `git -C /root/llama.cpp worktree remove