docs: FORK.md F10 page-cache drop, -lm dio dead end; runbook note
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
cce0ac1f09
commit
2fc9f9c351
2 files changed
+87
-1
No files matched your search
@@ -11,7 +11,8 @@ Status at the time of writing (2026-10-25):
|
||||
(this file among them), so count it rather than trusting a number written here. Section 2 maps
|
||||
every commit to a feature.
|
||||
- Production runs `6c480f7e7`. Everything after it on the branch is validation tooling, env-gated
|
||||
A/B knobs, comment fixes and documentation; none of it was deployed. The knobs and the comment
|
||||
A/B knobs, comment fixes, documentation and one loader behaviour change (F10, the page-cache
|
||||
drop); none of it was deployed. The knobs and the comment
|
||||
fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation.
|
||||
- Commit hashes below refer to that branch. They change on every rebase; find a commit again by
|
||||
its subject line. The commit messages are detailed and are the primary record of each mechanism
|
||||
@@ -37,6 +38,7 @@ Status at the time of writing (2026-10-25):
|
||||
- F7 Server shutdown path
|
||||
- F8 Cold-teardown guards
|
||||
- F9 Benchmark harness and validation script
|
||||
- F10 Page-cache drop after a copying load
|
||||
4. Dead ends
|
||||
5. Configuration that matters
|
||||
6. Measurement methodology and traps
|
||||
@@ -113,6 +115,7 @@ Oldest first. "Feature" points at the section below.
|
||||
| 80e662a2a | scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 |
|
||||
| ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 |
|
||||
| e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 |
|
||||
| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 |
|
||||
|
||||
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
||||
Everything from `e085c347d` down is after the production commit `6c480f7e7`.
|
||||
@@ -781,6 +784,84 @@ is at function entry, not after the host-shadow capture; `--moe-exact` and the m
|
||||
have not run on the GPU yet (tested offline only, see the runbook); the `v-vram-release` SKIP path
|
||||
(production moving during the measurement) has not been exercised.
|
||||
|
||||
### F10. Page-cache drop after a copying load
|
||||
|
||||
**What and why.** With `-lm none` the loader read()s the GGUF into its own buffers, but the kernel
|
||||
keeps the file in page cache afterwards, charged to the pod's cgroup: about 20 GB after a 35B load.
|
||||
The pod is BestEffort on a shared node, and on 2026-10-24 the Talos OOM controller killed it right
|
||||
after a model load under exactly that pressure (MemFree < 1 GB, file cache about 20 GB). Once the
|
||||
data is copied those pages are pure waste, so the loader advises the kernel to drop them.
|
||||
|
||||
**Where.** `src/llama-mmap.h`/`.cpp`: `llama_file::drop_cache(offset, len)`.
|
||||
`src/llama-model-loader.cpp`, `llama_model_loader::load_all_data`: the per-tensor drop at the end
|
||||
of the copy (non-mapping) branch and the whole-file drop in the final cleanup
|
||||
(`size_done >= size_data`). `tools/mtmd/clip.cpp`, `clip_model_loader::load_tensors`: whole-file drop
|
||||
after the tensor reads. Commit cce0ac1f0.
|
||||
|
||||
**How it works.**
|
||||
- `drop_cache()` is `posix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED)` under `#ifdef __linux__`
|
||||
(no-op elsewhere); `len` 0 means to the end of the file. Errors are logged at debug level only.
|
||||
- `load_all_data()` drops each tensor's range right after it is read (all three copy paths: straight
|
||||
into a host buffer, the async pinned-upload chunks, the staging buffer), so the cache never grows
|
||||
past about one tensor during the load. The final cleanup drops every file whole, which catches the
|
||||
GGUF metadata and the partial pages at tensor edges that a range drop leaves.
|
||||
- Both run only when `mappings.empty()`, i.e. the loader holds no mapping of any of its files: not
|
||||
`mmap`, `mmap+mlock` or `auto`, and no lazy tensors (`-lzm`; with lazy tensors `init_mappings()`
|
||||
maps every file even under `-lm none`). Split GGUFs are covered (every entry of `files`).
|
||||
- clip.cpp always copies the encoder through an ifstream (no mmap path exists), so it opens the
|
||||
file again after the reads and drops it whole, unless `no_alloc`.
|
||||
- Choice: per-tensor plus a final drop rather than a post-load drop only, because the per-tensor
|
||||
drop also removes the peak during the load (the incident window: anon and file cache grow together)
|
||||
for three extra lines. It costs about 0.27 s more than a post-load-only drop on the test model
|
||||
below. Direct IO (`-lm dio`, upstream's) was measured as the alternative and is not better (section 4).
|
||||
|
||||
**Invariants that must hold.**
|
||||
- Never drop for a file this loader maps: a mapped page would just be faulted back in (the kernel
|
||||
also skips pages that are mapped into a process, so a violation costs refaults, not correctness).
|
||||
- Drop only after the bytes are in our own buffers. Nothing in the fork re-reads the GGUF after
|
||||
load: the VRAM release/restore (F4) uses host shadows, not the file.
|
||||
- The drop is a hint: a failure must not fail the load.
|
||||
|
||||
**Failure modes and incidents.** None yet (not deployed). Costs to know about:
|
||||
- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a
|
||||
5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few
|
||||
seconds on a 26 s load.
|
||||
- A reload of the same model can no longer hit a warm page cache (a warm-cache reload of the test
|
||||
model took 4.6 s against 5.9 s cold). Under llama-swap a reload happens only after an eviction or
|
||||
a crash.
|
||||
- `TENSOR_DUPLICATED` tensors read the same range twice; the second read now comes from disk (small
|
||||
tensors only).
|
||||
|
||||
**Rebase hazards.** The structure of `load_all_data()` (the `from_mapping` branch and its three copy
|
||||
paths, the final cleanup block), `init_mappings()` creating mappings for lazy tensors, new load modes
|
||||
in `llama_load_mode` (any new mode that maps must leave `mappings` non-empty), and the tensor-read
|
||||
loop in clip.cpp `load_tensors`.
|
||||
|
||||
**Validation.** Not in `fork-validate.sh`. By hand on the pod with a scratch llama-server on a
|
||||
scratch port, CPU only (`-dev none -ngl 0 --no-mmproj-offload`, no VRAM), cold cache (evict the test
|
||||
files with python `os.posix_fadvise(..., DONTNEED)` first): page residency of the model and mmproj
|
||||
files by `mincore` before and after the load (must be 0 after with `-lm none`, unchanged with
|
||||
`-lm mmap`), the pod cgroup's `file`/`anon` in
|
||||
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/memory.stat`, host `MemFree`, load time, and a greedy
|
||||
request. First run (2026-10-25, gemma-4-E2B UD-Q4_K_XL 2.5 GiB + mmproj BF16 0.9 GiB, production
|
||||
6c480f7e7 and the candidate with the drops disabled as controls, 4 rotated rounds):
|
||||
|
||||
| | drops off (control) | post-load only | per-tensor + final (committed) |
|
||||
|---|---|---|---|
|
||||
| model / mmproj resident after load | 2431-2444 / 940 MiB | 0 / 0 | 0 / 0 |
|
||||
| peak model residency during load | 2.1-2.5 GiB | 2.1-2.4 GiB | 1.2-1.5 GiB (largest tensor) |
|
||||
| pod cgroup `file` after load | +3.37 GiB | +0 | +0 |
|
||||
| host MemFree after load | 0.6 GB | 2.5 GB | 2.5-2.8 GB |
|
||||
| load time (median, production 5.85 s) | 5.87 s | 6.56 s | 6.82 s |
|
||||
|
||||
Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it counts clean page cache as
|
||||
available. Greedy output byte-identical in all arms. `-lm mmap` control: the model file stays fully
|
||||
resident and mapped (2498 MiB, `file_mapped` +2.5 GiB) after load and after a request, and only the
|
||||
copied mmproj is dropped. `-lzm on` control: no drop (the file is mapped), 1252 MiB of copied
|
||||
tensors stay cached. Not run: the GPU upload path (the per-tensor drop sits after all three copy
|
||||
paths, so it is covered by construction, but the pinned async upload was not exercised: the card had
|
||||
only 1.2 GiB free beside production), and a 35B load.
|
||||
|
||||
---
|
||||
|
||||
## 4. Dead ends
|
||||
@@ -812,6 +893,7 @@ without a new reason.
|
||||
| `--no-mmap`/`-lm none` for Laguna-S 118B | does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. |
|
||||
| Laguna `--cache-reuse`, `--swa-full`, `-t 12`, KV quant | ruled out | `--cache-reuse` is force-disabled for iSWA when base and SWA sizes differ. |
|
||||
| CPU-only task model (`-ngl 0`) to avoid handoffs | rejected | At `--ctx-size 8192` titles could face 8k tokens of CPU prefill. The arbiter is the answer. |
|
||||
| `-lm dio` (direct IO) instead of the F10 page-cache drop | load +0.9 s on 3.4 GiB (6.89 vs 5.99 s), the same cost as the drop; 175 MiB of the model file and all of the mmproj stay cached | Synchronous 64 MiB O_DIRECT chunks with no readahead; the GGUF metadata goes through a separate buffered FILE and clip.cpp has no direct-IO path. Measured CPU only; reads into pinned Vulkan buffers may fall back to buffered IO (`EFAULT` or `EINVAL`). Not adopted. |
|
||||
|
||||
The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256
|
||||
tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch.
|
||||
@@ -947,3 +1029,5 @@ Machine-level:
|
||||
- Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new
|
||||
tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context.
|
||||
Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them.
|
||||
- F10 (page-cache drop) has not run on the GPU upload path or on a 35B load; its load-time cost there
|
||||
(synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate.
|
||||
@@ -277,6 +277,8 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6
|
||||
EOF
|
||||
grep -E 'MemAvailable|MemFree' /proc/meminfo
|
||||
```
|
||||
A build with F10 (FORK.md) drops the cache of every `-lm none` load by itself, but files cached
|
||||
by mmap loads, benchmarks or older builds still need this step.
|
||||
Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by
|
||||
models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache
|
||||
from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the
|
||||
|
||||
Reference in new issue
Block a user