docs: FORK.md F12 temporary upstream carries (#32735, #32740), Inkling config; runbook carry procedure

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-11-22 20:38:33 +01:00
1 parent 66b3c3f72f
commit 01812e8b02
2 files changed
+80 -1

No files matched your search

+64
View File
@@ -49,6 +49,7 @@ Status at the time of writing (2026-11-21):
- F9 Benchmark harness and validation script
- F10 Page-cache drop after a copying load
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -130,6 +131,8 @@ Oldest first. "Feature" points at the section below.
| dd851fd0c | llama: read a released KV/recurrent state from its host shadow | F4, F11 |
| 2874dcb10 | server: disk prompt cache, save slots on shutdown and restore after a restart | F11 |
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1088,6 +1091,58 @@ first start. Verified in production: a 35B 4.6k-token chat was saved on eviction
only 32 tokens; a gemma image request with the flags on works and is not cached. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
applies. Models only pick up the flags when they restart.
### F12. Temporary upstream carries (open upstream PRs, drop when merged)
**What and why.** Open upstream PRs carried unchanged until upstream merges them. Each carry is one
commit whose subject starts with `upstream-carry #<PR>:`; the author is the PR author, and the body
names the PR URL and the PR head that was carried. List them with
`git log --oneline --grep '^upstream-carry' origin/master..HEAD`. They are not fork features: never
edit them in place, re-fetch the PR instead.
Carried since 2026-11-22 (both on upstream `e6d624b9a`, for Inkling-2-Mini, section 5):
- #32735 `vulkan : support non-contiguous src in SCALE` (head `0ea4ffd00`): the Vulkan backend runs
SCALE on a non-contiguous source instead of rejecting it.
- #32740 `ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan` (head
`808fd1c4e`, four PR commits squashed): Inkling's per-query bias table (`[n_bins, n_head,
n_tokens]` plus an I32 bin index per key) is applied inside the flash-attention kernel. Before it,
the graph gathered the bias into a dense per-head mask (`n_kv x n_tokens x n_head`, F32, added to
the KQ mask and cast to F16) and passed that to FA, and the Inkling layers could not use the
`kv_rows` cell lists (dense masks only). Vulkan
binds the table and the indices in the two mask_opt descriptor slots, so `use_mask_opt` gets
`!bias` (F2). Vulkan supports it only for F16 K/V, head size 64 or 128, with subgroup shuffle and
vote (RX 580 RADV has both; Inkling is f16 KV, head size 128).
Measured on the RX 580 (2026-11-21, the previous PR head: `1fa93f902`, `d29f9d489`, `ff91e4f84`,
without `808fd1c4e`; base `3bf6b55b3` vs base + both PRs; logs `/root/arms/pr-out/` on the pod):
- Inkling pp2048 at ub1024: 91.6 -> 110.5 t/s. pp2048 at depth 16384 (ub512): 48.4 -> 73.0 t/s.
- Inkling compute buffers at ctx 131072, ub512: Vulkan0 12918 -> 7173 MiB, Vulkan_Host
12729 -> 398 MiB. At ctx 65536 the base build already reserved 15866 MiB of Vulkan0 compute (more
than the card; the driver places the excess in GTT). With the carry: 6981 MiB.
- No regression elsewhere: Qwen 35B pp2048 286.5 -> 287.9, pp2048 at 16k 208.0 -> 208.5;
gemma-26B 249.4 both.
- test-backend-ops on Vulkan0: SCALE 15/15, FLASH_ATTN_EXT 5693/5693.
`808fd1c4e` (the PR head since 2026-11-22) changes the table layout from `[n_head, n_bins, n_tokens]`
to `[n_bins, n_head, n_tokens]`, which removes one permute+cont from the Inkling graph; it was
validated with `tests/test-flash-attn-indexed-bias` and test-backend-ops, not re-benchmarked.
**Rebase hazards.**
- At each rebase, check every carry first (runbook section 1): merged upstream -> drop the carry
commit; still open -> compare `git diff <carried-head> <current PR head>` and re-carry the current
head if it moved; closed unmerged -> decide with the owner (Inkling's prefill and memory depend on
#32740, section 5).
- #32740 conflicts with F2 on `use_mask_opt` in `ggml_vk_flash_attn`. The resolution: upstream's
`!use_sparse`, the PR's `!bias &&`, the fork's GCN `HSK >= 256 || HSV >= 256` and its comment.
When the PR is merged and the carry dropped, the same line conflicts again when F2 is re-applied
on top; resolve it the same way.
- #32740 changes `llama_kv_cache::set_input_relative_pos` (new `rows` argument) and the
supports_op of every backend; a fork change in those places must follow it.
**Validation.** `test-backend-ops -b Vulkan0 -o SCALE` and `-o FLASH_ATTN_EXT` (check the case
count), and the PR's own test `tests/test-flash-attn-indexed-bias` (build target
`test-flash-attn-indexed-bias`; it runs every device, prints one line per case it ran and skips
unsupported cases silently, so count the `Vulkan0:` lines; the Q4_0 K/V cases run on the CPU only).
Then an Inkling request through llama-swap.
---
## 4. Dead ends
@@ -1172,6 +1227,15 @@ Other models:
the task model killed it on every title request and forced full re-prefills. `--parallel 1` frees
223 MiB of VRAM at 64k context there (4 auto slots size the SWA cache `n_swa * n_seq_max +
n_ubatch`, 4096 cells instead of 2560).
- Inkling-2-Mini (26B MoE, `inkling2`, 24 layers with the middle 12 looped 3x, 24 heads / 8 KV
heads, head size 128; macro `inkling_args`, entries `unsloth/Inkling-2-Mini-GGUF[-nothink]:UD-Q4_K_XL`,
matrix vars `ink`/`inknt`): the mainline-layout GGUF in `/root/models/inkling/` passed with `-m`
(the repo-root GGUFs that `-hf` would pick are Unsloth's layout), `--mmproj` beside it,
`--n-cpu-moe 24` (all experts on host), `-t 6`, f16 KV, `--parallel 1`, the disk prompt cache.
Since the 2026-11-22 carry of upstream #32740 (F12): `-b 1024 -ub 1024 --ctx-size 131072`
(before: `-b 512 -ub 512 --ctx-size 65536`, because the dense per-head bias mask needed about
12.7 GB of host compute buffer at 128k). Its nothink macro uses `--reasoning-effort none
--reasoning-budget 0`.
Environment, set once for all of llama-swap in `[program:llama-swap] environment=` of
`/root/supervisord.conf`: