diff --git a/FORK.md b/FORK.md index 82b710e60..c9ea436a6 100644 --- a/FORK.md +++ b/FORK.md @@ -14,6 +14,8 @@ Status at the time of writing (2026-10-25): `b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook section 6). Any later commit that touches compiled code needs the same validation before a redeploy. +- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on + top. It is not deployed and has not run on the GPU yet (F11, Validation). - Commit hashes below refer to that branch. They change on every rebase; find a commit again by its subject line. The commit messages are detailed and are the primary record of each mechanism and measurement; read them with `git show`. @@ -39,6 +41,7 @@ Status at the time of writing (2026-10-25): - F8 Cold-teardown guards - F9 Benchmark harness and validation script - F10 Page-cache drop after a copying load + - F11 Disk prompt cache (save slots on shutdown, restore after a restart) 4. Dead ends 5. Configuration that matters 6. Measurement methodology and traps @@ -116,10 +119,15 @@ Oldest first. "Feature" points at the section below. | ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 | | e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 | | cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 | +| f24c3582c | llama: do not release CPU extra buffers (CPU_REPACK) on device release | F4 | +| dd851fd0c | llama: read a released KV/recurrent state from its host shadow | F4, F11 | +| 2874dcb10 | server: disk prompt cache, save slots on shutdown and restore after a restart | F11 | +| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 | Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code. Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it); -`6c480f7e7` was the previous production commit. +`6c480f7e7` was the previous production commit. The last four rows are on `fable5/disk-prompt-cache` +only (hashes of that branch), not deployed. --- @@ -404,6 +412,12 @@ Measured footprints: `GGML_ASSERT(buf != NULL && "tensor buffer not set")`: `update_slots()` creates an SWA checkpoint that reads the KV via `ggml_backend_tensor_get` before decode. Fixed by restoring on wake (2feb6aeb2). - The recurrent state was left resident (default no-op), 201 MB on the 4B, until 9857a45f0. +- A model with weights on the CPU device (`-dev none` / `-ngl 0`) segfaulted (exit 139) in + `release_device_weights()` the first time the arbiter put it to sleep: CPU_REPACK buffers are not + host buffers, so `load_tensors` marked them releasable, and the shadow capture called their NULL + `get_tensor`. It also crashes production `2fc9f9c35`, but no production arbiter model has CPU-device + weights. Fixed in f24c3582c: only buffers on a non-CPU device are releasable. Found by the + `--diskcache` check (F11), whose arbiter cases run CPU only. - The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed resident in a cold model until bf2c1a699 / ad27d4921. - The MoE cache was not released at all before 7ac2014a8. Reallocating its banks without @@ -660,7 +674,8 @@ first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/serv - The token is released only after this process's VRAM is free (the next model may start uploading the moment the lock drops). - A shutting-down server never restores weights. -- SIGTERM in any state exits 0 within about 2 s and frees the token. +- SIGTERM in any state exits 0 within about 2 s and frees the token. With the disk prompt cache (F11) + the exit also writes each slot's state: about 1 s more per GB of state, after the token is free. **Failure modes and incidents.** - SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside @@ -743,7 +758,7 @@ required, so a rebase can be validated by someone with no context. tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and rounds; restarts llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`. - `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL] - [--moe-exact MODEL]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a + [--moe-exact MODEL] [--diskcache]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a scratch dir under /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline. @@ -752,7 +767,8 @@ required, so a rebase can be validated by someone with no context. of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs in the swapdown window, about 2 minutes); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share - the card either). + the card either). `--diskcache` runs beside production (CPU only); its GPU leg SKIPs below 1536 MiB + of free VRAM. **Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24, production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s, @@ -781,7 +797,8 @@ case `d-term-in-release` then falls back to polling the log. script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`, `no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`, `llama_moe_cache: .* hits =`, `sched_copy_experts: ids readback skipped`, `N/M tests passed`, -`Backend Vulkan0: OK`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and +`Backend Vulkan0: OK`, `disk cache: restored`, `disk cache: copied`/`saved`, `prompts of this model`, +`size limit, removing`, `disk cache: failed to write`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and markdown layout; the GGUF keys `.expert_count` / `.expert_used_count`; the DRM fdinfo keys `drm-client-id` / `drm-memory-vram`. @@ -873,6 +890,145 @@ tensors stay cached. The GPU upload path and a 35B load were first exercised by deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B request took 34 s including load and answer). +### F11. Disk prompt cache (save slots on shutdown, restore after a restart) + +**What and why.** Only one large model fits in RAM. Switching models in OpenWebUI makes llama-swap +SIGTERM the previous one, and switching back re-prefills the whole conversation, because the RAM +prompt cache died with the process (F4's reason for the arbiter). A 30k-token chat costs about 150 s +on the 35B at about 200 t/s. With `--cache-disk-path DIR`, each slot with a prompt is saved to DIR on +shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the +file, so only the suffix is processed. Measured on Qwen3.5-0.8B CPU only: a 12k-token turn 2 went +from a 72 s prefill to 1.2 s, of which the read took 307 ms and the restore 18 ms. Off by default. +Upstream PR #28092 (`--cache-disk`) was the reference for ideas, not ported (1700 lines). It +livelocked the kernel on AMD + Vulkan by DMA-ing GPU state into a MAP_SHARED mmap'd file, and it never +saved the slot in use on shutdown, which is the main case here. + +**Where.** +- `tools/server/server-task.h`/`.cpp`: `server_prompt_disk_entry`, `server_prompt_disk_cache` + (`n_reuse`, `find`, `has`, `load`, `save`, `remove`); file IO helpers `disk_cache_io`, + `disk_cache_open`. +- `tools/server/server-context.cpp`: `DISK_CACHE_MIN_TOKENS`, `disk_cache_key`, + `disk_cache_collect`, `disk_cache_flush`, `disk_cache_load`; the init in `load_model` after the RAM + cache; the `disk_cache_load` call at the end of `get_available_slot`; `server_context::start_loop` + (collect, `vram_share_exit`, flush). +- `common/arg.cpp`, `common/common.h`: `--cache-disk-path` (`cache_disk_path`), `--cache-disk-size` + (`cache_disk_mib`, default 4096, 0 = no limit). +- Library (cold save): `llama_context_restore_memory_host()` (`include/llama.h`, + `llama_context::restore_memory_host`); `llama_memory_i::restore_device_buffers(bool to_host)` in + `llama-memory.h`, `llama-kv-cache`, `llama-memory-recurrent`, `-hybrid`, `-kv-cache-iswa`; the helper + `llama_memory_wrap_host()` in `llama-memory.cpp`. +- Commits dd851fd0c, 2874dcb10, 2c6c517ba (and f24c3582c, F4). + +**How it works.** +- Save, after `start_loop()`'s loop returns. First `disk_cache_collect()` copies the state of every + slot with at least 256 text tokens to host memory (`llama_state_seq_get_data_ext`, flags NONE) + while the device buffers still exist. Then `vram_share_exit()` frees the VRAM and releases the GPU + token as before. Then `disk_cache_flush()` writes the files. The token release therefore waits + only for the device-to-host copy, never for disk IO. +- A slot is skipped when its KV does not cover its token list (`seq_pos_max + 1 != n_tokens`: the last + batch was not decoded, a shutdown during a token wait), when it holds media, and when the same + token list is already on disk. +- Cold server: with `LLAMA_SLEEP_EVICT_KV` the KV and recurrent state live only in host shadows (F4) + and their tensors have no buffer. `llama_context_restore_memory_host()` wraps each shadow as a + CPU_Mapped buffer (`ggml_backend_cpu_buffer_from_ptr`) and points the tensors into it, so there is + no copy, no allocation and no device use. It sets `device_discarded`. Without evict_kv the KV is still + in VRAM and is read directly. +- File, `/-.lpcd`: a 48-byte header (magic `LPCD`, version, key hash, + n_tokens, n_ckpt, payload size, payload hash), then 8-byte padded sections: tokens, checkpoint table, + state size, state, the state of each checkpoint. The hash is FNV-1a over the 64-bit words of the + payload. Write: plain fwrite to `.tmp`, fdatasync, `POSIX_FADV_DONTNEED`, rename. Read: plain fread, + then DONTNEED. There is no mmap anywhere. +- Index: at startup, from the headers, tokens and checkpoint tables of the files of this key. A bad + magic, a file size that differs from the header, or an inconsistent table means the file is removed. + Other keys and other versions are counted and left alone. A `.tmp` older than 1 h is removed. +- Key: the build commit, the model path + size + mtime, LoRA adapters and control vectors, n_ctx, + n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, K/V types, flash-attn type, and the RoPE/YaRN + parameters. +- Restore: at the end of `get_available_slot()`, after the RAM cache, only while warm, with + `cache_prompt` on and no media tokens. The file that reuses the most of the new prompt is loaded if + it reuses at least 256 more tokens than the slot holds. The slot's prompt goes to the RAM cache + first. Then the slot is cleared and the state is set; a failure clears the slot and removes the file. +- Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt, + or, for a recurrent or SWA model, a saved checkpoint at or below the common prefix can be used. A + plain-KV model can be cut back to any prefix. Exact prefix alone would almost never fire: the Qwen + template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is not a + prefix of the next OpenWebUI turn. Checkpoints are exact snapshots, and `update_slots()` rolls back + to one exactly as it does for the RAM cache; the recurrent state itself is never cut back. + Restored checkpoints get `id_task = -1`, so one that fails to load means a re-prefill, not an abort. +- Size limit: before a write, the least recently used `*.lpcd` files of every model in DIR are removed + until the new file fits. A restore updates the file's mtime. Entries the new prompt extends are + removed after the write and do not count against the limit. A write also needs its size + 1 GiB + free on the file system, so the cache never fills the PVC. +- Disabled with speculative decoding, with a warning: the draft state is not saved. + +**Invariants that must hold.** +- Never restore while cold or without the GPU token (`disk_cache_load` returns when `vram_cold`). + `get_available_slot()` runs on the loop thread after the wake, like the RAM cache's `prompt_save`. +- Copy the state before `vram_share_exit()` (it discards the device buffers) and write after it (the + token is not held for disk IO). +- Never serve a file that fails the size or hash check. Never mmap the file: a DMA into a shared + mapping is what livelocked upstream's version on AMD. +- A write is atomic (tmp + rename), and any IO error leaves the server serving and the exit clean. +- The key must cover everything that changes the state layout or the KV values. Add a new + state-changing parameter to `disk_cache_key()`. +- `llama_context_restore_memory_host()` is for exit only; the wrapped buffers point into the shadows, + which must outlive them (both are members of the same memory object). + +**Failure modes and incidents.** +- The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself + (CPU_REPACK, F4), not in the save. +- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full + KV allocation at exit (about 2.7 GB for the 35B at ctx 131072) on a node that has had an + OOM-controller kill. Replaced by the in-place wrap. The 32-byte alignment that + `cpu_buffer_from_ptr` asserts is met by rounding the base down; those leading bytes are never read. +- Writes to `/tmp` (node overlay) stalled once: 5.8 s for 64.5 MiB, against 171 ms normally. +- Costs to know about. llama-swap v239 waits for the old process to exit before it starts the next + model, and SIGKILLs a child still there 30 s after SIGTERM (measured 2026-10-25 with a scratch + llama-swap and dummy upstreams: term to exit 4.0 s, next start 2 ms later; a child that ignored SIGTERM + was replaced at +30.0 s). So the exit write adds directly to a model switch: about 1 s per GB on the + `/root` PVC (0.66 GB/s with fdatasync). A save cut off by the SIGKILL leaves only a `.tmp`. +- Only slots are saved. A conversation that sits only in the RAM cache when the process stops is lost. +- Every deploy empties the cache (the key holds the build commit), and so does any change to the + model file or the flags in the key. + +**Rebase hazards.** +- `server_context::start_loop` and `vram_share_exit` ordering (F7); `get_available_slot` and the RAM + cache (`prompt_save`, `prompt_load`, `server_prompt_cache_state`); `server_prompt` and + `common_prompt_checkpoint` fields (new fields must be saved, or the format version bumped). +- The checkpoint rollback in `update_slots` (`pos_min_thold`, `TAG_PROMPT_LOGITS`), which decides how + much of a restored file is reused. +- The llama state format (`state_write`/`state_read` of the memory types). The key holds the build + commit, so a format change cannot load an old file; it only empties the cache. +- `llama_memory_i::restore_device_buffers` and the shadow layout of F4 (`device_buffer_shadow`): + `llama_memory_wrap_host` relies on the compact iteration-order shadow. +- Flag names in `common/arg.cpp`. + +**Validation.** `fork-validate.sh --diskcache` (2c6c517ba), beside production. The model is the small +hybrid, CPU only (`-dev none -ngl 0`), with chat requests and greedy decoding, and a reference server +that never restarts. Cases: `restore`, `restore-arbiter` and `restore-cold` (turn 1, SIGTERM, restart, +turn 2 restored; prompt_n equal to the reference; identical answer), `no-prefix`, `key-ctx`, +`key-ctk`, `key-model`, `corrupt` (truncated and byte-flipped), `lru`, `write-error` (RLIMIT_FSIZE, +SIGXFSZ ignored), `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free. First run +(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219 +MiB free). + +| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill | +|---|---|---|---|---|---|---| +| 573-token chat | 64.5 MiB, 2 ckpt | 18 ms | 171 ms | 314 ms (token free 56 ms vs 28) | 166 + 3 ms | prompt_n 58 of 596, 462 vs 2913 ms | +| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms | +| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s | + +Not run yet: the GPU leg, the 35B, and `--arbiter` on this build. The swapdown window script +`/root/arms/diskcache-window.sh` and the 35B driver `/root/arms/diskcache-test/q35.py` are on the pod +(runbook section 6). Estimate for the 35B: 20 KiB of KV per token (10 attention layers, 2 KV heads, +head_dim 256, f16), so about 600 MiB for 30k tokens, plus the recurrent state and its checkpoints; +about 1-1.5 GB per file, about 1.5-2.5 s more per model switch, against about 150 s of re-prefill. + +Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache +--cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is +created on first start. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still +applies. Models only pick up the flags when they restart. + --- ## 4. Dead ends @@ -970,6 +1126,7 @@ Environment, set once for all of llama-swap in `[program:llama-swap] environment Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`, `GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`. +`--cache-disk-path` (F11) is not set yet: it is not deployed; the recommended flags are in F11. Machine-level: - GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted @@ -1037,3 +1194,8 @@ Machine-level: Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them. - F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate; its memory effect is verified in production. +- F11 (disk prompt cache) has run CPU only. Not yet run: its GPU leg (a save from VRAM and from a + released state, a restore into VRAM), a 35B measurement, and `--arbiter` on the candidate. The + 35B state size and exit time are estimates. The window script is ready (runbook section 6). +- F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key + holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB. diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md index 30e2ebc2e..4106603eb 100644 --- a/docs/fork-rebase-runbook.md +++ b/docs/fork-rebase-runbook.md @@ -267,6 +267,13 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f - Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a regression gate, not a measurement. +- Disk prompt cache (FORK.md F11, from `fable5/disk-prompt-cache`): add `--diskcache` to phase 1. It + is CPU only and runs beside production (about 2 minutes). Its GPU leg SKIPs below 1536 MiB of free + VRAM; then run `--diskcache` again in the phase-2 window. A 35B measurement driver is on the pod: + `/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the + production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script + `/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs + under swapdown; it has not been run yet. On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs. @@ -350,6 +357,14 @@ trees if in doubt. from the first chat response (section 8): `system_fingerprint` is `b-` (2026-10-25: `b11906-2fc9f9c35`). The first 35B request after that deploy took 34 s including the model load and the answer. +8. **Disk prompt cache** (FORK.md F11), only once `--cache-disk-path` is in `/root/config.yaml`: + - Every deploy empties the cache: the key holds the build commit, so the previous build's files are + never restored. They stay in the directory as "other files" until the size limit evicts them. + To free the space at once, `rm /root/prompt-cache/*.lpcd` while llama-swap is stopped (step 4). + - Check `/root` free space (`df -h /root`; 11-13 GB on 2026-10-25). The cap (`--cache-disk-size`) + plus a 1 GiB floor bounds the cache, but builds and model downloads share the PVC. + - Stopping llama-swap takes longer: each large model writes its slot state on SIGTERM, about 1 s + per GB of state, before it exits (FORK.md F7, F11). ## 8. Post-deploy checks @@ -385,6 +400,10 @@ on `reasoning_content` and returns an empty `content`; for one, raise `max_token evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed, releasing the GPU token`, with no kill escalation. 4. Text to gemma and to the 4B after the eviction; `holder` correct each time. + With the disk prompt cache on (F11): send the 35B a chat of at least 256 tokens before step 3. + Its eviction must log `disk cache: saved the prompt of N tokens`. Then send the same chat plus one + new message to the 35B again: the log shows `disk cache: restored`, and `timings.prompt_n` is + only the new suffix. 5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931). 6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for `GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none. @@ -481,6 +500,8 @@ Files (all under `/root`, a persistent volume): - `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program). - `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`). - `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers. +- `/root/prompt-cache/` (recommended, not yet configured): disk prompt cache files `*.lpcd` of every + model with `--cache-disk-path` (FORK.md F11). Safe to empty while llama-swap is stopped. - `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/` directory from the old design may remain). Never point a test at it.