Commit Graph
12 Commits
Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 01812e8b02 docs: FORK.md F12 temporary upstream carries (#32735, #32740), Inkling config; runbook carry procedure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Sonnet d2143f1e47 docs: runbook/FORK.md fixes from the 2026-11-21 rebase, validate and deploy cycle
HF_UI_VERSION must use the tag after the '|' in .ui-stamp; bundle path
for agents told not to push; build backup and copy-based rollback;
status, pointers and last-run record for 3bf6b55b3; q35.py build path
note; disk-cache restore test wording; open questions.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Sonnet 5.5 372f14e0b2 docs: runbook and FORK.md status for the 2026-11-21 rebase onto e6d624b9a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Sonnet 86bf601db6 docs: runbook/FORK.md after the adc2aa3ec deploy
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 92abb2d89b docs: F11 mmproj incident and fix; runbook - production command line, build targets, detached scripts
FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort
with the production mmproj), the fix, the new mmproj cases (15 PASS), and
that the earlier 35B numbers were measured with --no-mmproj. The 35B run with
production's exact command line (diskcache-window3.sh) is ready, not run, and
a precondition for deploying --cache-disk-path.

Runbook:
- tests must use production's exact model command line, mmproj included;
- the validated build has all three targets;
- the deploy mtime list includes libllama-server-impl.so and
  libllama-common.so (llama-server is a stub);
- deploy and rollback steps run as one setsid-wrapped script on the pod,
  never as a backgrounded cat && sh -n && setsid chain from kubectl exec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 1505d6a52b docs: FORK.md F11 - 35B results, the two leg-c anomalies explained
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.

Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.

Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 3d862b1357 docs: FORK.md F11 disk prompt cache, the CPU_REPACK release fix; runbook notes
F11 in the per-feature format: design, invariants, the measured costs (CPU
only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at
30 s). Also: the recommended config (not applied), the commit map entries, the
F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache,
the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run
yet). Runbook: --diskcache in phase 1, the pod window script, the deploy
notes (every deploy empties the cache, /root free space, a slower stop), a
post-deploy check, and the cache dir.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 80777eb831 docs: FORK.md and runbook after the 2fc9f9c35 deploy
Feedback from the 2026-10-25 deploy, which used only these docs:
- status: production is 2fc9f9c35; F10 deployed and verified on a 35B
  GPU load; --moe-exact and median --perf have run on the GPU; last
  full fork-validate run (37 PASS) and its timings; the first-candidate
  -arm VRAM spike seen twice is an open question.
- runbook: a no-rebase redeploy path; --arbiter beside production only
  if 1536 MiB VRAM is free, else in the swapdown window; full --moecache
  model path; page-cache eviction before and after phase 2; deploy
  steps reordered so the memory check follows stopping llama-swap;
  in-place rebuild and system_fingerprint check; mtime check through the
  versioned symlinks; rollback pointer numbering and one format; disk
  space; nothink alias and a PPM image helper for post-deploy requests;
  power1_cap via sysfs; restart and OOM checks; harmless UI warning.
- fork-validate.sh header: note the --arbiter VRAM requirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 e5762e0ec2 docs: FORK.md F10 page-cache drop, -lm dio dead end; runbook note
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 2edc33c430 docs(readme): short fork summary, FORK.md is the source of truth
The README fork section still said "the two code changes below are auto-on"
(the fork has three prefill changes plus the VRAM sharing and server fixes)
and recommended --n-cpu-moe 28 on the 35B, which FORK.md section 4 records as
worse at depth and unable to hold a long context. Replace it with a short
accurate summary that points to FORK.md, the runbook and the scripts. The
Laguna time profile, the iSWA KV note and Laguna's --parallel 1 saving were
only in the README and move to FORK.md sections 4 and 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 c829418367 docs: FORK.md - review fixes, new checks, llama-swap restart behaviour
- status: no hardcoded fork-commit count (the list includes docs and
  scripts; count git log origin/master..HEAD); what is after the production
  commit; the commit-trailer convention of this private fork vs upstream's
  AGENTS.md rules
- commit map: the commits after the production commit
- F1: validated by the new --moe-exact check and the LLAMA_MOE_IDS_SKIP knob
- F3: every graph that builds concat(state, transpose(x), 0) (bailingmoe3,
  clef, kimi-linear, kimi-k3, glm5-next, mamba-base, plamo2, lfm-base,
  qwen4exp besides delta-net); only the delta-net path is measured
- F4/F6: the new v-vram-* and a-handoff-latency checks with first results
- F5: encode_mtmd_chunk was split out of process_mtmd_chunk, not renamed
- F6: the vram_cold ordering rule restated from the code (the measured
  failure was under the doorbell design; with the inline probe should_sleep
  cannot run inside the restore; keep it, and A/B handoff latency before any
  change) and why vram_acquire_for_load sets the flag early
- F7: what llama-swap does when a child dies, verified with a scratch
  llama-swap: "upstream exited unexpectedly", restart on the next request
- F9: new flags, phases, median gating, 2026-10-24 reference values, the
  /tmp and gdb loss on the pod restart, new grep targets and gaps
- section 7: stale comments fixed, what is not yet run on the GPU

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 4ed4e58cf5 docs: FORK.md and rebase runbook for the RX 580 fork
FORK.md documents every fork feature (what and why with measured numbers,
where, how it works, invariants, past failures, rebase hazards, validation),
the dead ends measured on this hardware, the production configuration and
the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation,
deploy, rollback and GPU-recovery procedure plus an operations reference.
The README fork section links both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00