docs: FORK.md F10 page-cache drop, -lm dio dead end; runbook note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-25 13:14:08 +01:00
1 parent cce0ac1f09
commit 2fc9f9c351
2 files changed
+87 -1

No files matched your search

+85 -1
View File
@@ -11,7 +11,8 @@ Status at the time of writing (2026-10-25):
(this file among them), so count it rather than trusting a number written here. Section 2 maps
every commit to a feature.
- Production runs `6c480f7e7`. Everything after it on the branch is validation tooling, env-gated
A/B knobs, comment fixes and documentation; none of it was deployed. The knobs and the comment
A/B knobs, comment fixes, documentation and one loader behaviour change (F10, the page-cache
drop); none of it was deployed. The knobs and the comment
fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation.
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
its subject line. The commit messages are detailed and are the primary record of each mechanism
@@ -37,6 +38,7 @@ Status at the time of writing (2026-10-25):
- F7 Server shutdown path
- F8 Cold-teardown guards
- F9 Benchmark harness and validation script
- F10 Page-cache drop after a copying load
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -113,6 +115,7 @@ Oldest first. "Feature" points at the section below.
| 80e662a2a | scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 |
| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything from `e085c347d` down is after the production commit `6c480f7e7`.
@@ -781,6 +784,84 @@ is at function entry, not after the host-shadow capture; `--moe-exact` and the m
have not run on the GPU yet (tested offline only, see the runbook); the `v-vram-release` SKIP path
(production moving during the measurement) has not been exercised.
### F10. Page-cache drop after a copying load
**What and why.** With `-lm none` the loader read()s the GGUF into its own buffers, but the kernel
keeps the file in page cache afterwards, charged to the pod's cgroup: about 20 GB after a 35B load.
The pod is BestEffort on a shared node, and on 2026-10-24 the Talos OOM controller killed it right
after a model load under exactly that pressure (MemFree < 1 GB, file cache about 20 GB). Once the
data is copied those pages are pure waste, so the loader advises the kernel to drop them.
**Where.** `src/llama-mmap.h`/`.cpp`: `llama_file::drop_cache(offset, len)`.
`src/llama-model-loader.cpp`, `llama_model_loader::load_all_data`: the per-tensor drop at the end
of the copy (non-mapping) branch and the whole-file drop in the final cleanup
(`size_done >= size_data`). `tools/mtmd/clip.cpp`, `clip_model_loader::load_tensors`: whole-file drop
after the tensor reads. Commit cce0ac1f0.
**How it works.**
- `drop_cache()` is `posix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED)` under `#ifdef __linux__`
(no-op elsewhere); `len` 0 means to the end of the file. Errors are logged at debug level only.
- `load_all_data()` drops each tensor's range right after it is read (all three copy paths: straight
into a host buffer, the async pinned-upload chunks, the staging buffer), so the cache never grows
past about one tensor during the load. The final cleanup drops every file whole, which catches the
GGUF metadata and the partial pages at tensor edges that a range drop leaves.
- Both run only when `mappings.empty()`, i.e. the loader holds no mapping of any of its files: not
`mmap`, `mmap+mlock` or `auto`, and no lazy tensors (`-lzm`; with lazy tensors `init_mappings()`
maps every file even under `-lm none`). Split GGUFs are covered (every entry of `files`).
- clip.cpp always copies the encoder through an ifstream (no mmap path exists), so it opens the
file again after the reads and drops it whole, unless `no_alloc`.
- Choice: per-tensor plus a final drop rather than a post-load drop only, because the per-tensor
drop also removes the peak during the load (the incident window: anon and file cache grow together)
for three extra lines. It costs about 0.27 s more than a post-load-only drop on the test model
below. Direct IO (`-lm dio`, upstream's) was measured as the alternative and is not better (section 4).
**Invariants that must hold.**
- Never drop for a file this loader maps: a mapped page would just be faulted back in (the kernel
also skips pages that are mapped into a process, so a violation costs refaults, not correctness).
- Drop only after the bytes are in our own buffers. Nothing in the fork re-reads the GGUF after
load: the VRAM release/restore (F4) uses host shadows, not the file.
- The drop is a hint: a failure must not fail the load.
**Failure modes and incidents.** None yet (not deployed). Costs to know about:
- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a
5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few
seconds on a 26 s load.
- A reload of the same model can no longer hit a warm page cache (a warm-cache reload of the test
model took 4.6 s against 5.9 s cold). Under llama-swap a reload happens only after an eviction or
a crash.
- `TENSOR_DUPLICATED` tensors read the same range twice; the second read now comes from disk (small
tensors only).
**Rebase hazards.** The structure of `load_all_data()` (the `from_mapping` branch and its three copy
paths, the final cleanup block), `init_mappings()` creating mappings for lazy tensors, new load modes
in `llama_load_mode` (any new mode that maps must leave `mappings` non-empty), and the tensor-read
loop in clip.cpp `load_tensors`.
**Validation.** Not in `fork-validate.sh`. By hand on the pod with a scratch llama-server on a
scratch port, CPU only (`-dev none -ngl 0 --no-mmproj-offload`, no VRAM), cold cache (evict the test
files with python `os.posix_fadvise(..., DONTNEED)` first): page residency of the model and mmproj
files by `mincore` before and after the load (must be 0 after with `-lm none`, unchanged with
`-lm mmap`), the pod cgroup's `file`/`anon` in
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/memory.stat`, host `MemFree`, load time, and a greedy
request. First run (2026-10-25, gemma-4-E2B UD-Q4_K_XL 2.5 GiB + mmproj BF16 0.9 GiB, production
6c480f7e7 and the candidate with the drops disabled as controls, 4 rotated rounds):
| | drops off (control) | post-load only | per-tensor + final (committed) |
|---|---|---|---|
| model / mmproj resident after load | 2431-2444 / 940 MiB | 0 / 0 | 0 / 0 |
| peak model residency during load | 2.1-2.5 GiB | 2.1-2.4 GiB | 1.2-1.5 GiB (largest tensor) |
| pod cgroup `file` after load | +3.37 GiB | +0 | +0 |
| host MemFree after load | 0.6 GB | 2.5 GB | 2.5-2.8 GB |
| load time (median, production 5.85 s) | 5.87 s | 6.56 s | 6.82 s |
Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it counts clean page cache as
available. Greedy output byte-identical in all arms. `-lm mmap` control: the model file stays fully
resident and mapped (2498 MiB, `file_mapped` +2.5 GiB) after load and after a request, and only the
copied mmproj is dropped. `-lzm on` control: no drop (the file is mapped), 1252 MiB of copied
tensors stay cached. Not run: the GPU upload path (the per-tensor drop sits after all three copy
paths, so it is covered by construction, but the pinned async upload was not exercised: the card had
only 1.2 GiB free beside production), and a 35B load.
---
## 4. Dead ends
@@ -812,6 +893,7 @@ without a new reason.
| `--no-mmap`/`-lm none` for Laguna-S 118B | does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. |
| Laguna `--cache-reuse`, `--swa-full`, `-t 12`, KV quant | ruled out | `--cache-reuse` is force-disabled for iSWA when base and SWA sizes differ. |
| CPU-only task model (`-ngl 0`) to avoid handoffs | rejected | At `--ctx-size 8192` titles could face 8k tokens of CPU prefill. The arbiter is the answer. |
| `-lm dio` (direct IO) instead of the F10 page-cache drop | load +0.9 s on 3.4 GiB (6.89 vs 5.99 s), the same cost as the drop; 175 MiB of the model file and all of the mmproj stay cached | Synchronous 64 MiB O_DIRECT chunks with no readahead; the GGUF metadata goes through a separate buffered FILE and clip.cpp has no direct-IO path. Measured CPU only; reads into pinned Vulkan buffers may fall back to buffered IO (`EFAULT` or `EINVAL`). Not adopted. |
The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256
tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.
@@ -947,3 +1029,5 @@ Machine-level:
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
- F10 (page-cache drop) has not run on the GPU upload path or on a 35B load; its load-time cost there
(synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate.
+2
View File
@@ -277,6 +277,8 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6
EOF
grep -E 'MemAvailable|MemFree' /proc/meminfo
```
A build with F10 (FORK.md) drops the cache of every `-lm none` load by itself, but files cached
by mmap loads, benchmarks or older builds still need this step.
Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by
models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache
from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the