docs: FORK.md F11 disk prompt cache, the CPU_REPACK release fix; runbook notes

F11 in the per-feature format: design, invariants, the measured costs (CPU
only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at
30 s). Also: the recommended config (not applied), the commit map entries, the
F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache,
the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run
yet). Runbook: --diskcache in phase 1, the pod window script, the deploy
notes (every deploy empties the cache, /root free space, a slower stop), a
post-deploy check, and the cache dir.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 19:00:21 +01:00
1 parent 2c6c517ba5
commit 18691e0090
2 files changed
+188 -5

No files matched your search

+167 -5
View File
@@ -14,6 +14,8 @@ Status at the time of writing (2026-10-25):
`b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
top. It is not deployed and has not run on the GPU yet (F11, Validation).
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line. The commit messages are detailed and are the primary record of each mechanism
and measurement; read them with `git show`.
@@ -39,6 +41,7 @@ Status at the time of writing (2026-10-25):
- F8 Cold-teardown guards
- F9 Benchmark harness and validation script
- F10 Page-cache drop after a copying load
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -116,10 +119,15 @@ Oldest first. "Feature" points at the section below.
| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 |
| f24c3582c | llama: do not release CPU extra buffers (CPU_REPACK) on device release | F4 |
| dd851fd0c | llama: read a released KV/recurrent state from its host shadow | F4, F11 |
| 2874dcb10 | server: disk prompt cache, save slots on shutdown and restore after a restart | F11 |
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
`6c480f7e7` was the previous production commit.
`6c480f7e7` was the previous production commit. The last four rows are on `fable5/disk-prompt-cache`
only (hashes of that branch), not deployed.
---
@@ -404,6 +412,12 @@ Measured footprints:
`GGML_ASSERT(buf != NULL && "tensor buffer not set")`: `update_slots()` creates an SWA checkpoint
that reads the KV via `ggml_backend_tensor_get` before decode. Fixed by restoring on wake (2feb6aeb2).
- The recurrent state was left resident (default no-op), 201 MB on the 4B, until 9857a45f0.
- A model with weights on the CPU device (`-dev none` / `-ngl 0`) segfaulted (exit 139) in
`release_device_weights()` the first time the arbiter put it to sleep: CPU_REPACK buffers are not
host buffers, so `load_tensors` marked them releasable, and the shadow capture called their NULL
`get_tensor`. It also crashes production `2fc9f9c35`, but no production arbiter model has CPU-device
weights. Fixed in f24c3582c: only buffers on a non-CPU device are releasable. Found by the
`--diskcache` check (F11), whose arbiter cases run CPU only.
- The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed
resident in a cold model until bf2c1a699 / ad27d4921.
- The MoE cache was not released at all before 7ac2014a8. Reallocating its banks without
@@ -660,7 +674,8 @@ first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/serv
- The token is released only after this process's VRAM is free (the next model may start uploading
the moment the lock drops).
- A shutting-down server never restores weights.
- SIGTERM in any state exits 0 within about 2 s and frees the token.
- SIGTERM in any state exits 0 within about 2 s and frees the token. With the disk prompt cache (F11)
the exit also writes each slot's state: about 1 s more per GB of state, after the token is free.
**Failure modes and incidents.**
- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside
@@ -743,7 +758,7 @@ required, so a rebase can be validated by someone with no context.
tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and
rounds; restarts llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]
[--moe-exact MODEL]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a
[--moe-exact MODEL] [--diskcache]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a
scratch dir under /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets
inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check
flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline.
@@ -752,7 +767,8 @@ required, so a rebase can be validated by someone with no context.
of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs
in the swapdown window, about 2 minutes); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share
the card either).
the card either). `--diskcache` runs beside production (CPU only); its GPU leg SKIPs below 1536 MiB
of free VRAM.
**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24,
production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s,
@@ -781,7 +797,8 @@ case `d-term-in-release` then falls back to polling the log.
script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`,
`no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`,
`llama_moe_cache: .* hits =`, `sched_copy_experts: ids readback skipped`, `N/M tests passed`,
`Backend Vulkan0: OK`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and
`Backend Vulkan0: OK`, `disk cache: restored`, `disk cache: copied`/`saved`, `prompts of this model`,
`size limit, removing`, `disk cache: failed to write`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and
markdown layout; the GGUF keys `<arch>.expert_count` / `<arch>.expert_used_count`; the DRM fdinfo
keys `drm-client-id` / `drm-memory-vram`.
@@ -873,6 +890,145 @@ tensors stay cached. The GPU upload path and a 35B load were first exercised by
deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B
request took 34 s including load and answer).
### F11. Disk prompt cache (save slots on shutdown, restore after a restart)
**What and why.** Only one large model fits in RAM. Switching models in OpenWebUI makes llama-swap
SIGTERM the previous one, and switching back re-prefills the whole conversation, because the RAM
prompt cache died with the process (F4's reason for the arbiter). A 30k-token chat costs about 150 s
on the 35B at about 200 t/s. With `--cache-disk-path DIR`, each slot with a prompt is saved to DIR on
shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the
file, so only the suffix is processed. Measured on Qwen3.5-0.8B CPU only: a 12k-token turn 2 went
from a 72 s prefill to 1.2 s, of which the read took 307 ms and the restore 18 ms. Off by default.
Upstream PR #28092 (`--cache-disk`) was the reference for ideas, not ported (1700 lines). It
livelocked the kernel on AMD + Vulkan by DMA-ing GPU state into a MAP_SHARED mmap'd file, and it never
saved the slot in use on shutdown, which is the main case here.
**Where.**
- `tools/server/server-task.h`/`.cpp`: `server_prompt_disk_entry`, `server_prompt_disk_cache`
(`n_reuse`, `find`, `has`, `load`, `save`, `remove`); file IO helpers `disk_cache_io`,
`disk_cache_open`.
- `tools/server/server-context.cpp`: `DISK_CACHE_MIN_TOKENS`, `disk_cache_key`,
`disk_cache_collect`, `disk_cache_flush`, `disk_cache_load`; the init in `load_model` after the RAM
cache; the `disk_cache_load` call at the end of `get_available_slot`; `server_context::start_loop`
(collect, `vram_share_exit`, flush).
- `common/arg.cpp`, `common/common.h`: `--cache-disk-path` (`cache_disk_path`), `--cache-disk-size`
(`cache_disk_mib`, default 4096, 0 = no limit).
- Library (cold save): `llama_context_restore_memory_host()` (`include/llama.h`,
`llama_context::restore_memory_host`); `llama_memory_i::restore_device_buffers(bool to_host)` in
`llama-memory.h`, `llama-kv-cache`, `llama-memory-recurrent`, `-hybrid`, `-kv-cache-iswa`; the helper
`llama_memory_wrap_host()` in `llama-memory.cpp`.
- Commits dd851fd0c, 2874dcb10, 2c6c517ba (and f24c3582c, F4).
**How it works.**
- Save, after `start_loop()`'s loop returns. First `disk_cache_collect()` copies the state of every
slot with at least 256 text tokens to host memory (`llama_state_seq_get_data_ext`, flags NONE)
while the device buffers still exist. Then `vram_share_exit()` frees the VRAM and releases the GPU
token as before. Then `disk_cache_flush()` writes the files. The token release therefore waits
only for the device-to-host copy, never for disk IO.
- A slot is skipped when its KV does not cover its token list (`seq_pos_max + 1 != n_tokens`: the last
batch was not decoded, a shutdown during a token wait), when it holds media, and when the same
token list is already on disk.
- Cold server: with `LLAMA_SLEEP_EVICT_KV` the KV and recurrent state live only in host shadows (F4)
and their tensors have no buffer. `llama_context_restore_memory_host()` wraps each shadow as a
CPU_Mapped buffer (`ggml_backend_cpu_buffer_from_ptr`) and points the tensors into it, so there is
no copy, no allocation and no device use. It sets `device_discarded`. Without evict_kv the KV is still
in VRAM and is read directly.
- File, `<dir>/<key hash>-<token hash>.lpcd`: a 48-byte header (magic `LPCD`, version, key hash,
n_tokens, n_ckpt, payload size, payload hash), then 8-byte padded sections: tokens, checkpoint table,
state size, state, the state of each checkpoint. The hash is FNV-1a over the 64-bit words of the
payload. Write: plain fwrite to `.tmp`, fdatasync, `POSIX_FADV_DONTNEED`, rename. Read: plain fread,
then DONTNEED. There is no mmap anywhere.
- Index: at startup, from the headers, tokens and checkpoint tables of the files of this key. A bad
magic, a file size that differs from the header, or an inconsistent table means the file is removed.
Other keys and other versions are counted and left alone. A `.tmp` older than 1 h is removed.
- Key: the build commit, the model path + size + mtime, LoRA adapters and control vectors, n_ctx,
n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, K/V types, flash-attn type, and the RoPE/YaRN
parameters.
- Restore: at the end of `get_available_slot()`, after the RAM cache, only while warm, with
`cache_prompt` on and no media tokens. The file that reuses the most of the new prompt is loaded if
it reuses at least 256 more tokens than the slot holds. The slot's prompt goes to the RAM cache
first. Then the slot is cleared and the state is set; a failure clears the slot and removes the file.
- Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt,
or, for a recurrent or SWA model, a saved checkpoint at or below the common prefix can be used. A
plain-KV model can be cut back to any prefix. Exact prefix alone would almost never fire: the Qwen
template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is not a
prefix of the next OpenWebUI turn. Checkpoints are exact snapshots, and `update_slots()` rolls back
to one exactly as it does for the RAM cache; the recurrent state itself is never cut back.
Restored checkpoints get `id_task = -1`, so one that fails to load means a re-prefill, not an abort.
- Size limit: before a write, the least recently used `*.lpcd` files of every model in DIR are removed
until the new file fits. A restore updates the file's mtime. Entries the new prompt extends are
removed after the write and do not count against the limit. A write also needs its size + 1 GiB
free on the file system, so the cache never fills the PVC.
- Disabled with speculative decoding, with a warning: the draft state is not saved.
**Invariants that must hold.**
- Never restore while cold or without the GPU token (`disk_cache_load` returns when `vram_cold`).
`get_available_slot()` runs on the loop thread after the wake, like the RAM cache's `prompt_save`.
- Copy the state before `vram_share_exit()` (it discards the device buffers) and write after it (the
token is not held for disk IO).
- Never serve a file that fails the size or hash check. Never mmap the file: a DMA into a shared
mapping is what livelocked upstream's version on AMD.
- A write is atomic (tmp + rename), and any IO error leaves the server serving and the exit clean.
- The key must cover everything that changes the state layout or the KV values. Add a new
state-changing parameter to `disk_cache_key()`.
- `llama_context_restore_memory_host()` is for exit only; the wrapped buffers point into the shadows,
which must outlive them (both are members of the same memory object).
**Failure modes and incidents.**
- The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself
(CPU_REPACK, F4), not in the save.
- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full
KV allocation at exit (about 2.7 GB for the 35B at ctx 131072) on a node that has had an
OOM-controller kill. Replaced by the in-place wrap. The 32-byte alignment that
`cpu_buffer_from_ptr` asserts is met by rounding the base down; those leading bytes are never read.
- Writes to `/tmp` (node overlay) stalled once: 5.8 s for 64.5 MiB, against 171 ms normally.
- Costs to know about. llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a child still there 30 s after SIGTERM (measured 2026-10-25 with a scratch
llama-swap and dummy upstreams: term to exit 4.0 s, next start 2 ms later; a child that ignored SIGTERM
was replaced at +30.0 s). So the exit write adds directly to a model switch: about 1 s per GB on the
`/root` PVC (0.66 GB/s with fdatasync). A save cut off by the SIGKILL leaves only a `.tmp`.
- Only slots are saved. A conversation that sits only in the RAM cache when the process stops is lost.
- Every deploy empties the cache (the key holds the build commit), and so does any change to the
model file or the flags in the key.
**Rebase hazards.**
- `server_context::start_loop` and `vram_share_exit` ordering (F7); `get_available_slot` and the RAM
cache (`prompt_save`, `prompt_load`, `server_prompt_cache_state`); `server_prompt` and
`common_prompt_checkpoint` fields (new fields must be saved, or the format version bumped).
- The checkpoint rollback in `update_slots` (`pos_min_thold`, `TAG_PROMPT_LOGITS`), which decides how
much of a restored file is reused.
- The llama state format (`state_write`/`state_read` of the memory types). The key holds the build
commit, so a format change cannot load an old file; it only empties the cache.
- `llama_memory_i::restore_device_buffers` and the shadow layout of F4 (`device_buffer_shadow`):
`llama_memory_wrap_host` relies on the compact iteration-order shadow.
- Flag names in `common/arg.cpp`.
**Validation.** `fork-validate.sh --diskcache` (2c6c517ba), beside production. The model is the small
hybrid, CPU only (`-dev none -ngl 0`), with chat requests and greedy decoding, and a reference server
that never restarts. Cases: `restore`, `restore-arbiter` and `restore-cold` (turn 1, SIGTERM, restart,
turn 2 restored; prompt_n equal to the reference; identical answer), `no-prefix`, `key-ctx`,
`key-ctk`, `key-model`, `corrupt` (truncated and byte-flipped), `lru`, `write-error` (RLIMIT_FSIZE,
SIGXFSZ ignored), `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free. First run
(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219
MiB free).
| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill |
|---|---|---|---|---|---|---|
| 573-token chat | 64.5 MiB, 2 ckpt | 18 ms | 171 ms | 314 ms (token free 56 ms vs 28) | 166 + 3 ms | prompt_n 58 of 596, 462 vs 2913 ms |
| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms |
| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s |
Not run yet: the GPU leg, the 35B, and `--arbiter` on this build. The swapdown window script
`/root/arms/diskcache-window.sh` and the 35B driver `/root/arms/diskcache-test/q35.py` are on the pod
(runbook section 6). Estimate for the 35B: 20 KiB of KV per token (10 attention layers, 2 KV heads,
head_dim 256, f16), so about 600 MiB for 30k tokens, plus the recurrent state and its checkpoints;
about 1-1.5 GB per file, about 1.5-2.5 s more per model switch, against about 150 s of re-prefill.
Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache
--cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is
created on first start. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
applies. Models only pick up the flags when they restart.
---
## 4. Dead ends
@@ -970,6 +1126,7 @@ Environment, set once for all of llama-swap in `[program:llama-swap] environment
Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`,
`GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`.
`--cache-disk-path` (F11) is not set yet: it is not deployed; the recommended flags are in F11.
Machine-level:
- GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted
@@ -1037,3 +1194,8 @@ Machine-level:
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
is an estimate; its memory effect is verified in production.
- F11 (disk prompt cache) has run CPU only. Not yet run: its GPU leg (a save from VRAM and from a
released state, a restore into VRAM), a 35B measurement, and `--arbiter` on the candidate. The
35B state size and exit time are estimates. The window script is ready (runbook section 6).
- F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key
holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB.
+21
View File
@@ -267,6 +267,13 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
regression gate, not a measurement.
- Disk prompt cache (FORK.md F11, from `fable5/disk-prompt-cache`): add `--diskcache` to phase 1. It
is CPU only and runs beside production (about 2 minutes). Its GPU leg SKIPs below 1536 MiB of free
VRAM; then run `--diskcache` again in the phase-2 window. A 35B measurement driver is on the pod:
`/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the
production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
under swapdown; it has not been run yet.
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
@@ -350,6 +357,14 @@ trees if in doubt.
from the first chat response (section 8): `system_fingerprint` is `b<build>-<short sha>`
(2026-10-25: `b11906-2fc9f9c35`). The first 35B request after that deploy took 34 s including
the model load and the answer.
8. **Disk prompt cache** (FORK.md F11), only once `--cache-disk-path` is in `/root/config.yaml`:
- Every deploy empties the cache: the key holds the build commit, so the previous build's files are
never restored. They stay in the directory as "other files" until the size limit evicts them.
To free the space at once, `rm /root/prompt-cache/*.lpcd` while llama-swap is stopped (step 4).
- Check `/root` free space (`df -h /root`; 11-13 GB on 2026-10-25). The cap (`--cache-disk-size`)
plus a 1 GiB floor bounds the cache, but builds and model downloads share the PVC.
- Stopping llama-swap takes longer: each large model writes its slot state on SIGTERM, about 1 s
per GB of state, before it exits (FORK.md F7, F11).
## 8. Post-deploy checks
@@ -385,6 +400,10 @@ on `reasoning_content` and returns an empty `content`; for one, raise `max_token
evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed,
releasing the GPU token`, with no kill escalation.
4. Text to gemma and to the 4B after the eviction; `holder` correct each time.
With the disk prompt cache on (F11): send the 35B a chat of at least 256 tokens before step 3.
Its eviction must log `disk cache: saved the prompt of N tokens`. Then send the same chat plus one
new message to the 35B again: the log shows `disk cache: restored`, and `timings.prompt_n` is
only the new suffix.
5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931).
6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for
`GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none.
@@ -481,6 +500,8 @@ Files (all under `/root`, a persistent volume):
- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program).
- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`).
- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers.
- `/root/prompt-cache/` (recommended, not yet configured): disk prompt cache files `*.lpcd` of every
model with `--cache-disk-path` (FORK.md F11). Safe to empty while llama-swap is stopped.
- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/`
directory from the old design may remain). Never point a test at it.