From 3621025e8f72f1d25efeb67aa66e4d9da2f2c3f8 Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Sun, 22 Nov 2026 02:33:14 +0100 Subject: [PATCH] docs: FORK.md F12 temporary upstream carries (#32735, #32740), Inkling config; runbook carry procedure Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ --- FORK.md | 64 +++++++++++++++++++++++++++++++++++++ docs/fork-rebase-runbook.md | 17 +++++++++- 2 files changed, 80 insertions(+), 1 deletion(-) diff --git a/FORK.md b/FORK.md index b62a4189f..de71003f0 100644 --- a/FORK.md +++ b/FORK.md @@ -49,6 +49,7 @@ Status at the time of writing (2026-11-21): - F9 Benchmark harness and validation script - F10 Page-cache drop after a copying load - F11 Disk prompt cache (save slots on shutdown, restore after a restart) + - F12 Temporary upstream carries (open upstream PRs, drop when merged) 4. Dead ends 5. Configuration that matters 6. Measurement methodology and traps @@ -130,6 +131,8 @@ Oldest first. "Feature" points at the section below. | dd851fd0c | llama: read a released KV/recurrent state from its host shadow | F4, F11 | | 2874dcb10 | server: disk prompt cache, save slots on shutdown and restore after a restart | F11 | | 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 | +| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 | +| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 | Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code. Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it); @@ -1088,6 +1091,58 @@ first start. Verified in production: a 35B 4.6k-token chat was saved on eviction only 32 tokens; a gemma image request with the flags on works and is not cached. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still applies. Models only pick up the flags when they restart. +### F12. Temporary upstream carries (open upstream PRs, drop when merged) + +**What and why.** Open upstream PRs carried unchanged until upstream merges them. Each carry is one +commit whose subject starts with `upstream-carry #:`; the author is the PR author, and the body +names the PR URL and the PR head that was carried. List them with +`git log --oneline --grep '^upstream-carry' origin/master..HEAD`. They are not fork features: never +edit them in place, re-fetch the PR instead. + +Carried since 2026-11-22 (both on upstream `e6d624b9a`, for Inkling-2-Mini, section 5): +- #32735 `vulkan : support non-contiguous src in SCALE` (head `0ea4ffd00`): the Vulkan backend runs + SCALE on a non-contiguous source instead of rejecting it. +- #32740 `ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan` (head + `808fd1c4e`, four PR commits squashed): Inkling's per-query bias table (`[n_bins, n_head, + n_tokens]` plus an I32 bin index per key) is applied inside the flash-attention kernel. Before it, + the graph gathered the bias into a dense per-head mask (`n_kv x n_tokens x n_head`, F32, added to + the KQ mask and cast to F16) and passed that to FA, and the Inkling layers could not use the + `kv_rows` cell lists (dense masks only). Vulkan + binds the table and the indices in the two mask_opt descriptor slots, so `use_mask_opt` gets + `!bias` (F2). Vulkan supports it only for F16 K/V, head size 64 or 128, with subgroup shuffle and + vote (RX 580 RADV has both; Inkling is f16 KV, head size 128). + +Measured on the RX 580 (2026-11-21, the previous PR head: `1fa93f902`, `d29f9d489`, `ff91e4f84`, +without `808fd1c4e`; base `3bf6b55b3` vs base + both PRs; logs `/root/arms/pr-out/` on the pod): +- Inkling pp2048 at ub1024: 91.6 -> 110.5 t/s. pp2048 at depth 16384 (ub512): 48.4 -> 73.0 t/s. +- Inkling compute buffers at ctx 131072, ub512: Vulkan0 12918 -> 7173 MiB, Vulkan_Host + 12729 -> 398 MiB. At ctx 65536 the base build already reserved 15866 MiB of Vulkan0 compute (more + than the card; the driver places the excess in GTT). With the carry: 6981 MiB. +- No regression elsewhere: Qwen 35B pp2048 286.5 -> 287.9, pp2048 at 16k 208.0 -> 208.5; + gemma-26B 249.4 both. +- test-backend-ops on Vulkan0: SCALE 15/15, FLASH_ATTN_EXT 5693/5693. +`808fd1c4e` (the PR head since 2026-11-22) changes the table layout from `[n_head, n_bins, n_tokens]` +to `[n_bins, n_head, n_tokens]`, which removes one permute+cont from the Inkling graph; it was +validated with `tests/test-flash-attn-indexed-bias` and test-backend-ops, not re-benchmarked. + +**Rebase hazards.** +- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry + commit; still open -> compare `git diff ` and re-carry the current + head if it moved; closed unmerged -> decide with the owner (Inkling's prefill and memory depend on + #32740, section 5). +- #32740 conflicts with F2 on `use_mask_opt` in `ggml_vk_flash_attn`. The resolution: upstream's + `!use_sparse`, the PR's `!bias &&`, the fork's GCN `HSK >= 256 || HSV >= 256` and its comment. + When the PR is merged and the carry dropped, the same line conflicts again when F2 is re-applied + on top; resolve it the same way. +- #32740 changes `llama_kv_cache::set_input_relative_pos` (new `rows` argument) and the + supports_op of every backend; a fork change in those places must follow it. + +**Validation.** `test-backend-ops -b Vulkan0 -o SCALE` and `-o FLASH_ATTN_EXT` (check the case +count), and the PR's own test `tests/test-flash-attn-indexed-bias` (build target +`test-flash-attn-indexed-bias`; it runs every device, prints one line per case it ran and skips +unsupported cases silently, so count the `Vulkan0:` lines; the Q4_0 K/V cases run on the CPU only). +Then an Inkling request through llama-swap. + --- ## 4. Dead ends @@ -1172,6 +1227,15 @@ Other models: the task model killed it on every title request and forced full re-prefills. `--parallel 1` frees 223 MiB of VRAM at 64k context there (4 auto slots size the SWA cache `n_swa * n_seq_max + n_ubatch`, 4096 cells instead of 2560). +- Inkling-2-Mini (26B MoE, `inkling2`, 24 layers with the middle 12 looped 3x, 24 heads / 8 KV + heads, head size 128; macro `inkling_args`, entries `unsloth/Inkling-2-Mini-GGUF[-nothink]:UD-Q4_K_XL`, + matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m` + (the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it, + `--n-cpu-moe 24` (all experts on host), `-t 6`, f16 KV, `--parallel 1`, the disk prompt cache. + Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 131072` + (before: `-b 512 -ub 512 --ctx-size 65536`, because the dense per-head bias mask needed about + 12.7 GB of host compute buffer at 128k). Its nothink macro uses `--reasoning-effort none + --reasoning-budget 0`. Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of `/root/supervisord.conf`: diff --git a/docs/fork-rebase-runbook.md b/docs/fork-rebase-runbook.md index 804e5ce83..17b107ff1 100644 --- a/docs/fork-rebase-runbook.md +++ b/docs/fork-rebase-runbook.md @@ -58,6 +58,21 @@ git log --oneline "$OLD_BASE"..origin/master -- \ tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/ ``` +**Temporary upstream carries (FORK.md F12)** come first: `git log --oneline --grep '^upstream-carry' +"$OLD_BASE"..HEAD`. For each: merged into the new upstream -> do not cherry-pick it (check that the +merged version is the carried head or newer); still open -> re-fetch the PR head and re-carry it if +it moved; closed -> ask the owner. To carry a PR (2026-11-22: #32735, #32740): +```sh +git fetch https://github.com/ggml-org/llama.cpp pull//head:pr- +git cherry-pick $(git merge-base pr- HEAD)..pr- # the PR's commits, oldest first; resolve by hand +# squash them into one commit with the PR author, subject "upstream-carry #: ", and a +# body with the PR URL, the carried head and any conflict resolution (git commit-tree on the final tree) +diff <(git diff $(git merge-base pr- HEAD) pr- | grep '^[+-]') <(git diff HEAD~1 HEAD | grep '^[+-]') +``` +The last diff shows only the lines you changed while resolving conflicts (for #32740: none, the F2 +line differs only in context). Validate a carry like any compiled change (sections 5-6), plus the +PR's own tests. + For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past decisions: - 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional `-ncmoe` (all measured @@ -168,7 +183,7 @@ Open questions from that rebase: **Semantic re-checks to do for every rebase, clean or not** (details in FORK.md): - F1: `sched_copy_experts` and its `st` struct unchanged in meaning. -- F2: `use_mask_opt` still contains upstream's terms plus our `>= 256`. +- F2: `use_mask_opt` still contains upstream's terms plus our `>= 256` (and `!bias &&` from #32740, carried as F12 or merged upstream). - F3: `build_conv_state` still emits `ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0)` with `qkv_mixed` contiguous `[ch, n_seq_tokens, n_seqs]`; pipeline names; `types_ok` still excludes quantized types.