docs: FORK.md F11 disk prompt cache, the CPU_REPACK release fix; runbook notes
F11 in the per-feature format: design, invariants, the measured costs (CPU only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at 30 s). Also: the recommended config (not applied), the commit map entries, the F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache, the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run yet). Runbook: --diskcache in phase 1, the pod window script, the deploy notes (every deploy empties the cache, /root free space, a slower stop), a post-deploy check, and the cache dir. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2c6c517ba5
commit
18691e0090
2 files changed
+188
-5
No files matched your search
@@ -14,6 +14,8 @@ Status at the time of writing (2026-10-25):
|
||||
`b11906-2fc9f9c35`; rollback pointer `/root/DEPLOY-ROLLBACK-commit-10.txt` = `6c480f7e7`, the
|
||||
previous production). It was validated with the full `fork-validate.sh` set (37 PASS, 0 FAIL, runbook
|
||||
section 6). Any later commit that touches compiled code needs the same validation before a redeploy.
|
||||
- Branch `fable5/disk-prompt-cache` adds F11 (disk prompt cache) and a CPU_REPACK release fix (F4) on
|
||||
top. It is not deployed and has not run on the GPU yet (F11, Validation).
|
||||
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
|
||||
its subject line. The commit messages are detailed and are the primary record of each mechanism
|
||||
and measurement; read them with `git show`.
|
||||
@@ -39,6 +41,7 @@ Status at the time of writing (2026-10-25):
|
||||
- F8 Cold-teardown guards
|
||||
- F9 Benchmark harness and validation script
|
||||
- F10 Page-cache drop after a copying load
|
||||
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
||||
4. Dead ends
|
||||
5. Configuration that matters
|
||||
6. Measurement methodology and traps
|
||||
@@ -116,10 +119,15 @@ Oldest first. "Feature" points at the section below.
|
||||
| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
|
||||
| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
|
||||
| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 |
|
||||
| f24c3582c | llama: do not release CPU extra buffers (CPU_REPACK) on device release | F4 |
|
||||
| dd851fd0c | llama: read a released KV/recurrent state from its host shadow | F4, F11 |
|
||||
| 2874dcb10 | server: disk prompt cache, save slots on shutdown and restore after a restart | F11 |
|
||||
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
|
||||
|
||||
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
||||
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
||||
`6c480f7e7` was the previous production commit.
|
||||
`6c480f7e7` was the previous production commit. The last four rows are on `fable5/disk-prompt-cache`
|
||||
only (hashes of that branch), not deployed.
|
||||
|
||||
---
|
||||
|
||||
@@ -404,6 +412,12 @@ Measured footprints:
|
||||
`GGML_ASSERT(buf != NULL && "tensor buffer not set")`: `update_slots()` creates an SWA checkpoint
|
||||
that reads the KV via `ggml_backend_tensor_get` before decode. Fixed by restoring on wake (2feb6aeb2).
|
||||
- The recurrent state was left resident (default no-op), 201 MB on the 4B, until 9857a45f0.
|
||||
- A model with weights on the CPU device (`-dev none` / `-ngl 0`) segfaulted (exit 139) in
|
||||
`release_device_weights()` the first time the arbiter put it to sleep: CPU_REPACK buffers are not
|
||||
host buffers, so `load_tensors` marked them releasable, and the shadow capture called their NULL
|
||||
`get_tensor`. It also crashes production `2fc9f9c35`, but no production arbiter model has CPU-device
|
||||
weights. Fixed in f24c3582c: only buffers on a non-CPU device are releasable. Found by the
|
||||
`--diskcache` check (F11), whose arbiter cases run CPU only.
|
||||
- The scheduler's compute buffer (about 700 MiB on gemma-26B) and the Vulkan prealloc buffers stayed
|
||||
resident in a cold model until bf2c1a699 / ad27d4921.
|
||||
- The MoE cache was not released at all before 7ac2014a8. Reallocating its banks without
|
||||
@@ -660,7 +674,8 @@ first), `vram_share_exit`, the destructor's `vram_share_shutdown()`. `tools/serv
|
||||
- The token is released only after this process's VRAM is free (the next model may start uploading
|
||||
the moment the lock drops).
|
||||
- A shutting-down server never restores weights.
|
||||
- SIGTERM in any state exits 0 within about 2 s and frees the token.
|
||||
- SIGTERM in any state exits 0 within about 2 s and frees the token. With the disk prompt cache (F11)
|
||||
the exit also writes each slot's state: about 1 s more per GB of state, after the token is free.
|
||||
|
||||
**Failure modes and incidents.**
|
||||
- SIGTERM self-deadlock (production incident, gemma-26B): the signal landed on the loop thread inside
|
||||
@@ -743,7 +758,7 @@ required, so a rebase can be validated by someone with no context.
|
||||
tg256@d4096; it used to be tg32, an invalid metric here, section 6), interleaved across arms and
|
||||
rounds; restarts llama-swap from an EXIT trap. Results append to `/root/bench/results.txt`.
|
||||
- `fork-validate.sh --build CAND --ref REF [--ops] [--concat] [--arbiter] [--perf] [--moecache MODEL]
|
||||
[--moe-exact MODEL]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a
|
||||
[--moe-exact MODEL] [--diskcache]`: runs on the pod, PASS/FAIL/SKIP per check, nonzero exit on any FAIL. Uses a
|
||||
scratch dir under /tmp, a scratch arena (refuses `/dev/shm/llama-vram*`), ports 8091-8093, unsets
|
||||
inherited `GGML_*`/`LLAMA_*` env, kills only PIDs it started (PID-reuse safe). Default with no check
|
||||
flag is `--ops --concat --arbiter`. `--perf-report TSV` re-aggregates a saved `perf.tsv` offline.
|
||||
@@ -752,7 +767,8 @@ required, so a rebase can be validated by someone with no context.
|
||||
of VRAM is free (it FAILs at once otherwise; with gemma-26B warm only about 1.2 GiB is, so it often runs
|
||||
in the swapdown window, about 2 minutes); `--perf`, `--moe-exact` and `--moecache` need llama-swap stopped (the
|
||||
first two refuse otherwise; `--moecache` does not check, but loads a MoE model and must not share
|
||||
the card either).
|
||||
the card either). `--diskcache` runs beside production (CPU only); its GPU leg SKIPs below 1536 MiB
|
||||
of free VRAM.
|
||||
|
||||
**Reference values** (Qwen3.6-35B-A3B UD-Q4_K_XL, production config, `--perf` on 2026-10-24,
|
||||
production `6c480f7e7` and the previous production `5751c977b`): pp2048 about 274-286 t/s,
|
||||
@@ -781,7 +797,8 @@ case `d-term-in-release` then falls back to polling the log.
|
||||
script greps for (`server entering sleeping state`, `transpose fast path, transposed ne0=`,
|
||||
`no longer waiting for the GPU token`, `released .* MiB of MoE cache`, `MoE cache size =`,
|
||||
`llama_moe_cache: .* hits =`, `sched_copy_experts: ids readback skipped`, `N/M tests passed`,
|
||||
`Backend Vulkan0: OK`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and
|
||||
`Backend Vulkan0: OK`, `disk cache: restored`, `disk cache: copied`/`saved`, `prompts of this model`,
|
||||
`size limit, removing`, `disk cache: failed to write`); llama-bench's jsonl fields (`n_prompt`, `n_gen`, `n_depth`, `samples_ts`) and
|
||||
markdown layout; the GGUF keys `<arch>.expert_count` / `<arch>.expert_used_count`; the DRM fdinfo
|
||||
keys `drm-client-id` / `drm-memory-vram`.
|
||||
|
||||
@@ -873,6 +890,145 @@ tensors stay cached. The GPU upload path and a 35B load were first exercised by
|
||||
deploy (page cache result above); the load-time cost on the 35B was not isolated (the first 35B
|
||||
request took 34 s including load and answer).
|
||||
|
||||
### F11. Disk prompt cache (save slots on shutdown, restore after a restart)
|
||||
|
||||
**What and why.** Only one large model fits in RAM. Switching models in OpenWebUI makes llama-swap
|
||||
SIGTERM the previous one, and switching back re-prefills the whole conversation, because the RAM
|
||||
prompt cache died with the process (F4's reason for the arbiter). A 30k-token chat costs about 150 s
|
||||
on the 35B at about 200 t/s. With `--cache-disk-path DIR`, each slot with a prompt is saved to DIR on
|
||||
shutdown. After the restart, a request whose prompt starts with a saved prompt is restored from the
|
||||
file, so only the suffix is processed. Measured on Qwen3.5-0.8B CPU only: a 12k-token turn 2 went
|
||||
from a 72 s prefill to 1.2 s, of which the read took 307 ms and the restore 18 ms. Off by default.
|
||||
Upstream PR #28092 (`--cache-disk`) was the reference for ideas, not ported (1700 lines). It
|
||||
livelocked the kernel on AMD + Vulkan by DMA-ing GPU state into a MAP_SHARED mmap'd file, and it never
|
||||
saved the slot in use on shutdown, which is the main case here.
|
||||
|
||||
**Where.**
|
||||
- `tools/server/server-task.h`/`.cpp`: `server_prompt_disk_entry`, `server_prompt_disk_cache`
|
||||
(`n_reuse`, `find`, `has`, `load`, `save`, `remove`); file IO helpers `disk_cache_io`,
|
||||
`disk_cache_open`.
|
||||
- `tools/server/server-context.cpp`: `DISK_CACHE_MIN_TOKENS`, `disk_cache_key`,
|
||||
`disk_cache_collect`, `disk_cache_flush`, `disk_cache_load`; the init in `load_model` after the RAM
|
||||
cache; the `disk_cache_load` call at the end of `get_available_slot`; `server_context::start_loop`
|
||||
(collect, `vram_share_exit`, flush).
|
||||
- `common/arg.cpp`, `common/common.h`: `--cache-disk-path` (`cache_disk_path`), `--cache-disk-size`
|
||||
(`cache_disk_mib`, default 4096, 0 = no limit).
|
||||
- Library (cold save): `llama_context_restore_memory_host()` (`include/llama.h`,
|
||||
`llama_context::restore_memory_host`); `llama_memory_i::restore_device_buffers(bool to_host)` in
|
||||
`llama-memory.h`, `llama-kv-cache`, `llama-memory-recurrent`, `-hybrid`, `-kv-cache-iswa`; the helper
|
||||
`llama_memory_wrap_host()` in `llama-memory.cpp`.
|
||||
- Commits dd851fd0c, 2874dcb10, 2c6c517ba (and f24c3582c, F4).
|
||||
|
||||
**How it works.**
|
||||
- Save, after `start_loop()`'s loop returns. First `disk_cache_collect()` copies the state of every
|
||||
slot with at least 256 text tokens to host memory (`llama_state_seq_get_data_ext`, flags NONE)
|
||||
while the device buffers still exist. Then `vram_share_exit()` frees the VRAM and releases the GPU
|
||||
token as before. Then `disk_cache_flush()` writes the files. The token release therefore waits
|
||||
only for the device-to-host copy, never for disk IO.
|
||||
- A slot is skipped when its KV does not cover its token list (`seq_pos_max + 1 != n_tokens`: the last
|
||||
batch was not decoded, a shutdown during a token wait), when it holds media, and when the same
|
||||
token list is already on disk.
|
||||
- Cold server: with `LLAMA_SLEEP_EVICT_KV` the KV and recurrent state live only in host shadows (F4)
|
||||
and their tensors have no buffer. `llama_context_restore_memory_host()` wraps each shadow as a
|
||||
CPU_Mapped buffer (`ggml_backend_cpu_buffer_from_ptr`) and points the tensors into it, so there is
|
||||
no copy, no allocation and no device use. It sets `device_discarded`. Without evict_kv the KV is still
|
||||
in VRAM and is read directly.
|
||||
- File, `<dir>/<key hash>-<token hash>.lpcd`: a 48-byte header (magic `LPCD`, version, key hash,
|
||||
n_tokens, n_ckpt, payload size, payload hash), then 8-byte padded sections: tokens, checkpoint table,
|
||||
state size, state, the state of each checkpoint. The hash is FNV-1a over the 64-bit words of the
|
||||
payload. Write: plain fwrite to `.tmp`, fdatasync, `POSIX_FADV_DONTNEED`, rename. Read: plain fread,
|
||||
then DONTNEED. There is no mmap anywhere.
|
||||
- Index: at startup, from the headers, tokens and checkpoint tables of the files of this key. A bad
|
||||
magic, a file size that differs from the header, or an inconsistent table means the file is removed.
|
||||
Other keys and other versions are counted and left alone. A `.tmp` older than 1 h is removed.
|
||||
- Key: the build commit, the model path + size + mtime, LoRA adapters and control vectors, n_ctx,
|
||||
n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, K/V types, flash-attn type, and the RoPE/YaRN
|
||||
parameters.
|
||||
- Restore: at the end of `get_available_slot()`, after the RAM cache, only while warm, with
|
||||
`cache_prompt` on and no media tokens. The file that reuses the most of the new prompt is loaded if
|
||||
it reuses at least 256 more tokens than the slot holds. The slot's prompt goes to the RAM cache
|
||||
first. Then the slot is cleared and the state is set; a failure clears the slot and removes the file.
|
||||
- Reusable means one of two things. Either the saved token list is an exact prefix of the new prompt,
|
||||
or, for a recurrent or SWA model, a saved checkpoint at or below the common prefix can be used. A
|
||||
plain-KV model can be cut back to any prefix. Exact prefix alone would almost never fire: the Qwen
|
||||
template rewrites the previous assistant turn (the reasoning is dropped), so the saved list is not a
|
||||
prefix of the next OpenWebUI turn. Checkpoints are exact snapshots, and `update_slots()` rolls back
|
||||
to one exactly as it does for the RAM cache; the recurrent state itself is never cut back.
|
||||
Restored checkpoints get `id_task = -1`, so one that fails to load means a re-prefill, not an abort.
|
||||
- Size limit: before a write, the least recently used `*.lpcd` files of every model in DIR are removed
|
||||
until the new file fits. A restore updates the file's mtime. Entries the new prompt extends are
|
||||
removed after the write and do not count against the limit. A write also needs its size + 1 GiB
|
||||
free on the file system, so the cache never fills the PVC.
|
||||
- Disabled with speculative decoding, with a warning: the draft state is not saved.
|
||||
|
||||
**Invariants that must hold.**
|
||||
- Never restore while cold or without the GPU token (`disk_cache_load` returns when `vram_cold`).
|
||||
`get_available_slot()` runs on the loop thread after the wake, like the RAM cache's `prompt_save`.
|
||||
- Copy the state before `vram_share_exit()` (it discards the device buffers) and write after it (the
|
||||
token is not held for disk IO).
|
||||
- Never serve a file that fails the size or hash check. Never mmap the file: a DMA into a shared
|
||||
mapping is what livelocked upstream's version on AMD.
|
||||
- A write is atomic (tmp + rename), and any IO error leaves the server serving and the exit clean.
|
||||
- The key must cover everything that changes the state layout or the KV values. Add a new
|
||||
state-changing parameter to `disk_cache_key()`.
|
||||
- `llama_context_restore_memory_host()` is for exit only; the wrapped buffers point into the shadows,
|
||||
which must outlive them (both are members of the same memory object).
|
||||
|
||||
**Failure modes and incidents.**
|
||||
- The first run of the cold case crashed (exit 139). The crash was in the CPU-only release itself
|
||||
(CPU_REPACK, F4), not in the save.
|
||||
- The first version of the cold save copied the shadow into a new CPU buffer: a second copy of the full
|
||||
KV allocation at exit (about 2.7 GB for the 35B at ctx 131072) on a node that has had an
|
||||
OOM-controller kill. Replaced by the in-place wrap. The 32-byte alignment that
|
||||
`cpu_buffer_from_ptr` asserts is met by rounding the base down; those leading bytes are never read.
|
||||
- Writes to `/tmp` (node overlay) stalled once: 5.8 s for 64.5 MiB, against 171 ms normally.
|
||||
- Costs to know about. llama-swap v239 waits for the old process to exit before it starts the next
|
||||
model, and SIGKILLs a child still there 30 s after SIGTERM (measured 2026-10-25 with a scratch
|
||||
llama-swap and dummy upstreams: term to exit 4.0 s, next start 2 ms later; a child that ignored SIGTERM
|
||||
was replaced at +30.0 s). So the exit write adds directly to a model switch: about 1 s per GB on the
|
||||
`/root` PVC (0.66 GB/s with fdatasync). A save cut off by the SIGKILL leaves only a `.tmp`.
|
||||
- Only slots are saved. A conversation that sits only in the RAM cache when the process stops is lost.
|
||||
- Every deploy empties the cache (the key holds the build commit), and so does any change to the
|
||||
model file or the flags in the key.
|
||||
|
||||
**Rebase hazards.**
|
||||
- `server_context::start_loop` and `vram_share_exit` ordering (F7); `get_available_slot` and the RAM
|
||||
cache (`prompt_save`, `prompt_load`, `server_prompt_cache_state`); `server_prompt` and
|
||||
`common_prompt_checkpoint` fields (new fields must be saved, or the format version bumped).
|
||||
- The checkpoint rollback in `update_slots` (`pos_min_thold`, `TAG_PROMPT_LOGITS`), which decides how
|
||||
much of a restored file is reused.
|
||||
- The llama state format (`state_write`/`state_read` of the memory types). The key holds the build
|
||||
commit, so a format change cannot load an old file; it only empties the cache.
|
||||
- `llama_memory_i::restore_device_buffers` and the shadow layout of F4 (`device_buffer_shadow`):
|
||||
`llama_memory_wrap_host` relies on the compact iteration-order shadow.
|
||||
- Flag names in `common/arg.cpp`.
|
||||
|
||||
**Validation.** `fork-validate.sh --diskcache` (2c6c517ba), beside production. The model is the small
|
||||
hybrid, CPU only (`-dev none -ngl 0`), with chat requests and greedy decoding, and a reference server
|
||||
that never restarts. Cases: `restore`, `restore-arbiter` and `restore-cold` (turn 1, SIGTERM, restart,
|
||||
turn 2 restored; prompt_n equal to the reference; identical answer), `no-prefix`, `key-ctx`,
|
||||
`key-ctk`, `key-model`, `corrupt` (truncated and byte-flipped), `lru`, `write-error` (RLIMIT_FSIZE,
|
||||
SIGXFSZ ignored), `log-scan`, and `gpu-warm`/`gpu-cold` when 1536 MiB of VRAM is free. First run
|
||||
(2026-10-25, beside production, candidate build of this branch): 11 PASS, 0 FAIL, 1 SKIP (GPU: 1219
|
||||
MiB free).
|
||||
|
||||
| case (Qwen3.5-0.8B, CPU) | state + checkpoints | copy | write | exit (no cache: 125 ms) | read + restore | turn 2 vs full prefill |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 573-token chat | 64.5 MiB, 2 ckpt | 18 ms | 171 ms | 314 ms (token free 56 ms vs 28) | 166 + 3 ms | prompt_n 58 of 596, 462 vs 2913 ms |
|
||||
| 2.9k tokens | 92 MiB | 37 ms | 170 ms | 376 ms | 157 + 6 ms | 35 tokens, 408 ms |
|
||||
| 12k tokens | 200.7 MiB | 110 ms | 350 ms | 641 ms, peak RSS +162 MiB | 307 + 18 ms | 35 tokens, 1.2 s vs 72 s |
|
||||
|
||||
Not run yet: the GPU leg, the 35B, and `--arbiter` on this build. The swapdown window script
|
||||
`/root/arms/diskcache-window.sh` and the 35B driver `/root/arms/diskcache-test/q35.py` are on the pod
|
||||
(runbook section 6). Estimate for the 35B: 20 KiB of KV per token (10 attention layers, 2 KV heads,
|
||||
head_dim 256, f16), so about 600 MiB for 30k tokens, plus the recurrent state and its checkpoints;
|
||||
about 1-1.5 GB per file, about 1.5-2.5 s more per model switch, against about 150 s of re-prefill.
|
||||
|
||||
Recommended configuration (not applied): append `--cache-disk-path /root/prompt-cache
|
||||
--cache-disk-size 4096` to the `moe_offload_args` and `moe_offload_gemma` macros. The directory is
|
||||
created on first start. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
|
||||
applies. Models only pick up the flags when they restart.
|
||||
|
||||
---
|
||||
|
||||
## 4. Dead ends
|
||||
@@ -970,6 +1126,7 @@ Environment, set once for all of llama-swap in `[program:llama-swap] environment
|
||||
|
||||
Not set, on purpose: `GGML_VK_DISABLE_GRAPH_OPTIMIZE`, `GGML_SCHED_PREFETCH_EXPERTS`,
|
||||
`GGML_VK_ASYNC_USE_TRANSFER_QUEUE`, `GGML_VK_CONCAT_FASTPATH`, `--moe-cache-mib`.
|
||||
`--cache-disk-path` (F11) is not set yet: it is not deployed; the recommended flags are in F11.
|
||||
|
||||
Machine-level:
|
||||
- GPU power cap fixed at 120 W by `/root/fan-control.sh` (a separate supervisord program; re-asserted
|
||||
@@ -1037,3 +1194,8 @@ Machine-level:
|
||||
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
|
||||
- F10's load-time cost on the 35B (synchronous page freeing, about 0.28 s per GiB on the test model)
|
||||
is an estimate; its memory effect is verified in production.
|
||||
- F11 (disk prompt cache) has run CPU only. Not yet run: its GPU leg (a save from VRAM and from a
|
||||
released state, a restore into VRAM), a 35B measurement, and `--arbiter` on the candidate. The
|
||||
35B state size and exit time are estimates. The window script is ready (runbook section 6).
|
||||
- F11 saves only the slots, not the RAM prompt cache entries, and every deploy empties it (the key
|
||||
holds the build commit). Its exit write delays a llama-swap model switch by about 1 s per GB.
|
||||
@@ -267,6 +267,13 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
|
||||
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
|
||||
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
|
||||
regression gate, not a measurement.
|
||||
- Disk prompt cache (FORK.md F11, from `fable5/disk-prompt-cache`): add `--diskcache` to phase 1. It
|
||||
is CPU only and runs beside production (about 2 minutes). Its GPU leg SKIPs below 1536 MiB of free
|
||||
VRAM; then run `--diskcache` again in the phase-2 window. A 35B measurement driver is on the pod:
|
||||
`/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the
|
||||
production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script
|
||||
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
|
||||
under swapdown; it has not been run yet.
|
||||
|
||||
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
|
||||
|
||||
@@ -350,6 +357,14 @@ trees if in doubt.
|
||||
from the first chat response (section 8): `system_fingerprint` is `b<build>-<short sha>`
|
||||
(2026-10-25: `b11906-2fc9f9c35`). The first 35B request after that deploy took 34 s including
|
||||
the model load and the answer.
|
||||
8. **Disk prompt cache** (FORK.md F11), only once `--cache-disk-path` is in `/root/config.yaml`:
|
||||
- Every deploy empties the cache: the key holds the build commit, so the previous build's files are
|
||||
never restored. They stay in the directory as "other files" until the size limit evicts them.
|
||||
To free the space at once, `rm /root/prompt-cache/*.lpcd` while llama-swap is stopped (step 4).
|
||||
- Check `/root` free space (`df -h /root`; 11-13 GB on 2026-10-25). The cap (`--cache-disk-size`)
|
||||
plus a 1 GiB floor bounds the cache, but builds and model downloads share the PVC.
|
||||
- Stopping llama-swap takes longer: each large model writes its slot state on SIGTERM, about 1 s
|
||||
per GB of state, before it exits (FORK.md F7, F11).
|
||||
|
||||
## 8. Post-deploy checks
|
||||
|
||||
@@ -385,6 +400,10 @@ on `reasoning_content` and returns an empty `content`; for one, raise `max_token
|
||||
evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed,
|
||||
releasing the GPU token`, with no kill escalation.
|
||||
4. Text to gemma and to the 4B after the eviction; `holder` correct each time.
|
||||
With the disk prompt cache on (F11): send the 35B a chat of at least 256 tokens before step 3.
|
||||
Its eviction must log `disk cache: saved the prompt of N tokens`. Then send the same chat plus one
|
||||
new message to the 35B again: the log shows `disk cache: restored`, and `timings.prompt_n` is
|
||||
only the new suffix.
|
||||
5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931).
|
||||
6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for
|
||||
`GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none.
|
||||
@@ -481,6 +500,8 @@ Files (all under `/root`, a persistent volume):
|
||||
- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program).
|
||||
- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`).
|
||||
- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers.
|
||||
- `/root/prompt-cache/` (recommended, not yet configured): disk prompt cache files `*.lpcd` of every
|
||||
model with `--cache-disk-path` (FORK.md F11). Safe to empty while llama-swap is stopped.
|
||||
- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/`
|
||||
directory from the old design may remain). Never point a test at it.
|
||||
|
||||
|
||||
Reference in new issue
Block a user