diff --git a/FORK.md b/FORK.md index bf941c82f..d419f7452 100644 --- a/FORK.md +++ b/FORK.md @@ -11,7 +11,8 @@ Status at the time of writing (2026-10-25): (this file among them), so count it rather than trusting a number written here. Section 2 maps every commit to a feature. - Production runs `6c480f7e7`. Everything after it on the branch is validation tooling, env-gated - A/B knobs, comment fixes and documentation; none of it was deployed. The knobs and the comment + A/B knobs, comment fixes, documentation and one loader behaviour change (F10, the page-cache + drop); none of it was deployed. The knobs and the comment fix touch compiled code, so a redeploy of the tip is a code change and needs the usual validation. - Commit hashes below refer to that branch. They change on every rebase; find a commit again by its subject line. The commit messages are detailed and are the primary record of each mechanism @@ -37,6 +38,7 @@ Status at the time of writing (2026-10-25): - F7 Server shutdown path - F8 Cold-teardown guards - F9 Benchmark harness and validation script + - F10 Page-cache drop after a copying load 4. Dead ends 5. Configuration that matters 6. Measurement methodology and traps @@ -113,6 +115,7 @@ Oldest first. "Feature" points at the section below. | 80e662a2a | scripts: fork-validate.sh - A/B ordinary handoff latency against --ref | F6, F9 | | ceab1a91f | scripts: fork-validate.sh - check that a cold server gives its VRAM back | F4, F9 | | e5fc474b8 | scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip | F1, F9 | +| cce0ac1f0 | llama: drop the model file's page cache after a copying load | F10 | Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code. Everything from `e085c347d` down is after the production commit `6c480f7e7`. @@ -781,6 +784,84 @@ is at function entry, not after the host-shadow capture; `--moe-exact` and the m have not run on the GPU yet (tested offline only, see the runbook); the `v-vram-release` SKIP path (production moving during the measurement) has not been exercised. +### F10. Page-cache drop after a copying load + +**What and why.** With `-lm none` the loader read()s the GGUF into its own buffers, but the kernel +keeps the file in page cache afterwards, charged to the pod's cgroup: about 20 GB after a 35B load. +The pod is BestEffort on a shared node, and on 2026-10-24 the Talos OOM controller killed it right +after a model load under exactly that pressure (MemFree < 1 GB, file cache about 20 GB). Once the +data is copied those pages are pure waste, so the loader advises the kernel to drop them. + +**Where.** `src/llama-mmap.h`/`.cpp`: `llama_file::drop_cache(offset, len)`. +`src/llama-model-loader.cpp`, `llama_model_loader::load_all_data`: the per-tensor drop at the end +of the copy (non-mapping) branch and the whole-file drop in the final cleanup +(`size_done >= size_data`). `tools/mtmd/clip.cpp`, `clip_model_loader::load_tensors`: whole-file drop +after the tensor reads. Commit cce0ac1f0. + +**How it works.** +- `drop_cache()` is `posix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED)` under `#ifdef __linux__` + (no-op elsewhere); `len` 0 means to the end of the file. Errors are logged at debug level only. +- `load_all_data()` drops each tensor's range right after it is read (all three copy paths: straight + into a host buffer, the async pinned-upload chunks, the staging buffer), so the cache never grows + past about one tensor during the load. The final cleanup drops every file whole, which catches the + GGUF metadata and the partial pages at tensor edges that a range drop leaves. +- Both run only when `mappings.empty()`, i.e. the loader holds no mapping of any of its files: not + `mmap`, `mmap+mlock` or `auto`, and no lazy tensors (`-lzm`; with lazy tensors `init_mappings()` + maps every file even under `-lm none`). Split GGUFs are covered (every entry of `files`). +- clip.cpp always copies the encoder through an ifstream (no mmap path exists), so it opens the + file again after the reads and drops it whole, unless `no_alloc`. +- Choice: per-tensor plus a final drop rather than a post-load drop only, because the per-tensor + drop also removes the peak during the load (the incident window: anon and file cache grow together) + for three extra lines. It costs about 0.27 s more than a post-load-only drop on the test model + below. Direct IO (`-lm dio`, upstream's) was measured as the alternative and is not better (section 4). + +**Invariants that must hold.** +- Never drop for a file this loader maps: a mapped page would just be faulted back in (the kernel + also skips pages that are mapped into a process, so a violation costs refaults, not correctness). +- Drop only after the bytes are in our own buffers. Nothing in the fork re-reads the GGUF after + load: the VRAM release/restore (F4) uses host shadows, not the file. +- The drop is a hint: a failure must not fail the load. + +**Failure modes and incidents.** None yet (not deployed). Costs to know about: +- Load time rises by the synchronous page freeing that reclaim would otherwise do later: +0.95 s on a + 5.9 s load of 3.4 GiB (gemma-4-E2B plus mmproj). Unmeasured on the 35B; by size it would be a few + seconds on a 26 s load. +- A reload of the same model can no longer hit a warm page cache (a warm-cache reload of the test + model took 4.6 s against 5.9 s cold). Under llama-swap a reload happens only after an eviction or + a crash. +- `TENSOR_DUPLICATED` tensors read the same range twice; the second read now comes from disk (small + tensors only). + +**Rebase hazards.** The structure of `load_all_data()` (the `from_mapping` branch and its three copy +paths, the final cleanup block), `init_mappings()` creating mappings for lazy tensors, new load modes +in `llama_load_mode` (any new mode that maps must leave `mappings` non-empty), and the tensor-read +loop in clip.cpp `load_tensors`. + +**Validation.** Not in `fork-validate.sh`. By hand on the pod with a scratch llama-server on a +scratch port, CPU only (`-dev none -ngl 0 --no-mmproj-offload`, no VRAM), cold cache (evict the test +files with python `os.posix_fadvise(..., DONTNEED)` first): page residency of the model and mmproj +files by `mincore` before and after the load (must be 0 after with `-lm none`, unchanged with +`-lm mmap`), the pod cgroup's `file`/`anon` in +`/sys/fs/cgroup/kubepods/besteffort/pod/memory.stat`, host `MemFree`, load time, and a greedy +request. First run (2026-10-25, gemma-4-E2B UD-Q4_K_XL 2.5 GiB + mmproj BF16 0.9 GiB, production +6c480f7e7 and the candidate with the drops disabled as controls, 4 rotated rounds): + +| | drops off (control) | post-load only | per-tensor + final (committed) | +|---|---|---|---| +| model / mmproj resident after load | 2431-2444 / 940 MiB | 0 / 0 | 0 / 0 | +| peak model residency during load | 2.1-2.5 GiB | 2.1-2.4 GiB | 1.2-1.5 GiB (largest tensor) | +| pod cgroup `file` after load | +3.37 GiB | +0 | +0 | +| host MemFree after load | 0.6 GB | 2.5 GB | 2.5-2.8 GB | +| load time (median, production 5.85 s) | 5.87 s | 6.56 s | 6.82 s | + +Anon was +3.75 GiB in every arm. Host MemAvailable cannot show the win: it counts clean page cache as +available. Greedy output byte-identical in all arms. `-lm mmap` control: the model file stays fully +resident and mapped (2498 MiB, `file_mapped` +2.5 GiB) after load and after a request, and only the +copied mmproj is dropped. `-lzm on` control: no drop (the file is mapped), 1252 MiB of copied +tensors stay cached. Not run: the GPU upload path (the per-tensor drop sits after all three copy +paths, so it is covered by construction, but the pinned async upload was not exercised: the card had +only 1.2 GiB free beside production), and a 35B load. + --- ## 4. Dead ends @@ -812,6 +893,7 @@ without a new reason. | `--no-mmap`/`-lm none` for Laguna-S 118B | does not load, OOM-killed the node 3 times | Pinned host buffer is charged against the GTT limit (~31.4 GiB); experts are 30.7 GiB. Laguna uses mmap. | | Laguna `--cache-reuse`, `--swa-full`, `-t 12`, KV quant | ruled out | `--cache-reuse` is force-disabled for iSWA when base and SWA sizes differ. | | CPU-only task model (`-ngl 0`) to avoid handoffs | rejected | At `--ctx-size 8192` titles could face 8k tokens of CPU prefill. The arbiter is the answer. | +| `-lm dio` (direct IO) instead of the F10 page-cache drop | load +0.9 s on 3.4 GiB (6.89 vs 5.99 s), the same cost as the drop; 175 MiB of the model file and all of the mmproj stay cached | Synchronous 64 MiB O_DIRECT chunks with no readahead; the GGUF metadata goes through a separate buffered FILE and clip.cpp has no direct-IO path. Measured CPU only; reads into pinned Vulkan buffers may fall back to buffered IO (`EFAULT` or `EINVAL`). Not adopted. | The one live prefill lead: a partial trailing ubatch is expensive (ub4096 at p8448 = 2 full + 256 tail measured 258.4 vs 296.4 at p8192, -12.8%), consistent with a fixed expert-upload cost per ubatch. @@ -947,3 +1029,5 @@ Machine-level: - Not reported upstream: the `wait_until_no_sleep()` lost wakeup and the sleep wait that ignored new tasks (both upstream bugs fixed in 878d1338f), and the graph-optimize alias cost at tiny context. Repository rules forbid agent-written upstream issues or PRs; the owner would have to write them. +- F10 (page-cache drop) has not run on the GPU upload path or on a 35B load; its load-time cost there + (synchronous page freeing, about 0.28 s per GiB on the test model) is an estimate. diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md index 108b395cc..b4e586830 100644 --- a/docs/fork-rebase-runbook.md +++ b/docs/fork-rebase-runbook.md @@ -277,6 +277,8 @@ Deploy exactly the validated commit. Downtime for the 2026-10-24 redeploy was 6 EOF grep -E 'MemAvailable|MemFree' /proc/meminfo ``` + A build with F10 (FORK.md) drops the cache of every `-lm none` load by itself, but files cached + by mmap loads, benchmarks or older builds still need this step. Proceed only with MemAvailable >= 25 GB once the running models are gone (after step 4; RAM held by models llama-swap is about to stop does not count). In 2026-10-24 this took node-wide file cache from 15.6 to 4.9 GB and MemAvailable to 45.8 GB. Note that `/sys/fs/cgroup/memory.stat` inside the