HF_UI_VERSION must use the tag after the '|' in .ui-stamp; bundle path
for agents told not to push; build backup and copy-based rollback;
status, pointers and last-run record for 3bf6b55b3; q35.py build path
note; disk-cache restore test wording; open questions.
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort
with the production mmproj), the fix, the new mmproj cases (15 PASS), and
that the earlier 35B numbers were measured with --no-mmproj. The 35B run with
production's exact command line (diskcache-window3.sh) is ready, not run, and
a precondition for deploying --cache-disk-path.
Runbook:
- tests must use production's exact model command line, mmproj included;
- the validated build has all three targets;
- the deploy mtime list includes libllama-server-impl.so and
libllama-common.so (llama-server is a stub);
- deploy and rollback steps run as one setsid-wrapped script on the pod,
never as a backgrounded cat && sh -n && setsid chain from kubectl exec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.
Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.
Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F11 in the per-feature format: design, invariants, the measured costs (CPU
only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at
30 s). Also: the recommended config (not applied), the commit map entries, the
F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache,
the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run
yet). Runbook: --diskcache in phase 1, the pod window script, the deploy
notes (every deploy empties the cache, /root free space, a slower stop), a
post-deploy check, and the cache dir.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Feedback from the 2026-10-25 deploy, which used only these docs:
- status: production is 2fc9f9c35; F10 deployed and verified on a 35B
GPU load; --moe-exact and median --perf have run on the GPU; last
full fork-validate run (37 PASS) and its timings; the first-candidate
-arm VRAM spike seen twice is an open question.
- runbook: a no-rebase redeploy path; --arbiter beside production only
if 1536 MiB VRAM is free, else in the swapdown window; full --moecache
model path; page-cache eviction before and after phase 2; deploy
steps reordered so the memory check follows stopping llama-swap;
in-place rebuild and system_fingerprint check; mtime check through the
versioned symlinks; rollback pointer numbering and one format; disk
space; nothink alias and a PPM image helper for post-deploy requests;
power1_cap via sysfs; restart and OOM checks; harmless UI warning.
- fork-validate.sh header: note the --arbiter VRAM requirement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 1 now covers the VRAM release/restore check and the handoff latency
A/B against --ref (which must be the production build); phase 2 adds
--moe-exact on the 35B and keeps its scratch dir. Explains median gating and
--perf-report, that a pod restart empties /tmp, that the manual checks are
now automated, and that gdb is not in the pod image (lost in the 2026-10-24
restart). States the commit trailer convention and what llama-swap does with
a child that exits vs one that wedges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FORK.md documents every fork feature (what and why with measured numbers,
where, how it works, invariants, past failures, rebase hazards, validation),
the dead ends measured on this hardware, the production configuration and
the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation,
deploy, rollback and GPU-recovery procedure plus an operations reference.
The README fork section links both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>