Compare commits

Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 9573fe0a2f docs: FORK.md F15, AVX2 NVFP4 x Q8_0 dot product (not deployed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 21:38:01 +01:00
LumpiastyandClaude Opus 5.5 78a204bb2f ggml-cpu: faster AVX2 NVFP4 x Q8_0 dot product (two blocks per iteration, vector UE4M3 scales)
The x86 AVX2 ggml_vec_dot_nvfp4_q8_0 did one block per iteration with four
scalar UE4M3 table lookups, two scalar FP16 conversions, four scalar muls and
broadcasts per 64 weights, plus two FMAs on one accumulator chain.

New AVX2 path, two blocks (128 weights) per iteration:
- E2M1 decode with 256-bit pshufb on the whole 32-byte qs, unpack + permute2x128
  into the q8_0 element order (low nibbles are elements 0-7 of a sub-block,
  high nibbles 8-15);
- the 8 sub-block sums reduced exactly in int16 (a sub-block sum is at most
  16*12*127) with two levels of hadd_epi16 and one madd, one cvt;
- the 8 UE4M3 scales decoded in one vector (bit manipulation, bit-exact with
  ggml_ue4m3_to_fp32 for all 256 bytes: x0.5, 0x7F -> 0, sign bit ignored,
  subnormals), the 4 q8_0 FP16 scales assembled in a GPR and converted with one
  vcvtph2ps; one mul and one FMA per iteration.
An odd trailing block uses the existing scalar loop. The AVX (non-AVX2) path
and the generic reference are unchanged.

tests: test-backend-ops perf cases at Inkling-2-Mini expert shapes (n_embd 2304,
gate_up 2304, down 1152, 128 experts / 8 used; n = 1, 8, 16, 32) for nvfp4,
mxfp4 and q4_0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 21:10:39 +01:00
LumpiastyandClaude Opus 5.5 2caae7e113 docs: FORK.md/runbook after the f3397cded deploy - F14 measured (35B TTFT A/B, batch layout, edit cost, disk cache), pointer 16
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:48:22 +01:00
LumpiastyandClaude Opus 5.5 f3397cded0 docs: FORK.md F14, user-message batch-break merge (not deployed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
LumpiastyandClaude Opus 5.5 510aa6e090 server: merge user-message batch breaks when the rest of the prompt fits in one ubatch
Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last
user message (and of earlier ones on a fresh prompt) so that a context
checkpoint lands there. On a hybrid/recurrent model with MoE experts on the
host, that turns a typical chat turn (previous answer + new user message, under
one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the
host experts again.

When the rest of the prompt from the current batch start fits in one ubatch,
skip the user-start break. The near-end rule still checkpoints the batch start
and the end-4 rule the prompt end, so a later prompt that diverges inside the
last user message re-processes less than one ubatch more, in the same ubatch.
Prompts longer than one ubatch, and batches after a media chunk, keep the
upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
5 changed files with 323 additions and 51 deletions

No files matched your search

+234 -8
View File
@@ -6,14 +6,14 @@ which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-11-22):
- Production runs `b13c41355` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, and F13, the
per-pass loop views for Inkling), deployed 2026-11-22 08:47 with the Inkling placement change
(section 5, F13). Validated with the full `fork-validate.sh` set in one window (52 PASS, 0 FAIL,
0 SKIP; runbook section 6). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-15.txt` = `3621025e8`
(build copy `/root/arms/build-3621025e8`, config backup `/root/config.yaml.bak-20261122-predeploy`;
restore both, the old build has no F13); next free pointer 16.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12683-b13c41355`) and changes
- Production runs `f3397cded` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, F13, the
per-pass loop views for Inkling, and F14, the user-message batch-break merge), deployed 2026-11-22
10:37 (F14 validated in its own window: 35B TTFT A/B, disk cache 15 PASS and the 35B legs; F14
measured). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-16.txt` = `b13c41355` (build copy
`/root/arms/build-b13c41355`; config unchanged by this deploy); next free pointer 17. The previous
pointer 15 = `3621025e8` needs its config backup `/root/config.yaml.bak-20261122-predeploy` too.
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12687-f3397cded`) and changes
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
(section 3, "2026-11-21").
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
@@ -55,6 +55,8 @@ Status at the time of writing (2026-11-22):
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
- F14 User-message batch-break merge (one ubatch per short turn)
- F15 AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -139,6 +141,8 @@ Oldest first. "Feature" points at the section below.
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
| 78a204bb2 | ggml-cpu: faster AVX2 NVFP4 x Q8_0 dot product (two blocks per iteration, vector UE4M3 scales) (branch `fable5/nvfp4-avx2`, not deployed) | F15 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1397,6 +1401,228 @@ task-model call (arbiter handoff, F4), identical at temperature 0.
---
### F14. User-message batch-break merge (one ubatch per short turn)
Branch `fable5/userstart-split` (rebased onto the F13 production tip `166ec22e2`), fast-forwarded into
the production branch on 2026-11-22 (deploy and results below). Knob: `LLAMA_USERSTART_MERGE=0`.
**What and why.** Upstream ends a prompt batch right before a user message so that a context
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
message is what a later request uses when it diverges inside that message (the user edits it, or the
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
template drops the previous answer's reasoning, which moves the divergence to just after
`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation
prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each
ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
`<|im_start|>user`, the detected delimiter).
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
Commit 510aa6e09.
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
the restored checkpoint at the same position, with the current task id, so the next request still
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
diverges inside that message restores the batch-start checkpoint instead and re-processes the
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
upstream behaviour everywhere.
**Invariants that must hold.**
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
divergence inside the last user message to less than `n_ubatch` tokens.
- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared
by `has_mtmd`); hence the `!has_mtmd` condition.
- It only changes where batches end, never which tokens are processed, so outputs differ from an
unmerged run only by batch-composition rounding.
**Measured (2026-11-22, pod, GPU to itself in one swapdown window, build `f3397cded`; driver
`/root/arms/usw_ab.py`, window script `/root/arms/usw-window.sh`, logs and `res.json` per arm in
`/root/arms/usw-ab/{M1,U1,U2,M2}/`).** Production 35B think entry (`q36t`: `${ctx_256k}
${qwen35_think_args} ${moe_offload_args}`, `-hf` pulls the mmproj, supervisord's environment, scratch
arena and `--cache-disk-path`, plus `-lv 4`), one fresh server per arm, arms M (merged, default) and U
(`LLAMA_USERSTART_MERGE=0`) in the order M U U M. Each chat: system prompt, 6 user turns from
`pt-chunk.txt` (150, 10, 40, 150, 600, 40 tokens), synthetic previous answers of 300, 800, 1200, 500,
1300 tokens, `max_tokens 8`, `temperature 0`, streamed; then the last user message edited in the
middle and resent. TTFT in seconds (mean of the two runs per arm; the single runs in brackets),
`prompt_n` was identical in all four runs on every turn:
| turn | user tokens | prompt_n | M TTFT | U TTFT | M - U |
|---|---|---|---|---|---|
| 1 (fresh prompt) | 150 | 196 | 3.99 (4.43, 3.55) | 6.08 (5.73, 6.42) | -2.1 |
| 2 | 10 | 327 | 4.11 (3.75, 4.47) | 7.66 (6.75, 8.57) | -3.6 |
| 3 | 40 | 869 | 6.34 (6.62, 6.06) | 9.34 (7.26, 11.42) | -3.0 |
| 4 | 150 | 1381 | 7.20 (7.30, 7.10) | 10.95 (12.31, 9.58) | -3.7 |
| 5 | 600 | 1130 | 8.82 (7.47, 10.17) | 9.80 (9.52, 10.07) | -1.0 |
| 6 | 40 | 1370 | 8.11 (8.01, 8.20) | 9.53 (9.33, 9.72) | -1.4 |
| edit of turn 6 | 40 | M 1374, U 66 | 7.94 (8.16, 7.71) | 3.65 (4.09, 3.20) | +4.3 |
- Batch layout (`prompt processing, n_tokens` and checkpoint lines at `-lv 4`): M runs every turn as
one batch `[restore point, end-4)` plus the tail, with checkpoints at the batch start (superseding
the restored one) and at end minus 4; U splits at the last user message (an extra checkpoint
there, for example 1323 between 515 and 1380 on turn 3) and on turn 1 also at the first user
message (system prompt, 23 tokens). Turn N+1 restores the end-minus-4 checkpoint of turn N in both
arms. No `forcing full prompt re-processing` in any run.
- The 10-token turn gained too, against the plan's "no change". U split it into 308 tokens (the
previous answer, on the GPU) and 19 (message, generation prompt and tail), and the 19-token part runs
the experts of all 40 layers on the CPU, which costs seconds rather than nothing at 6 threads
(inferred from the totals, not profiled). The 600-token turn gained
least (its two parts are 508 and 618 tokens, both on the GPU, so the split costs one extra
upload, about 1-2 s against noise of the same size).
- Edit of the last user message (the cost of the trade): M has no checkpoint at the user-message
start, restores the batch start and re-processes previous answer + message (1374 tokens, one
ubatch); U restores the user-start checkpoint and re-processes 66 tokens. +4.3 s per such edit.
Ordinary follow-up turns are far more frequent than edits of the last message in this use; a client
that injects per-request context into the last user message (RAG, web search) pays the same
as an edit on every turn, so set `LLAMA_USERSTART_MERGE=0` for such a client.
- Outputs: identical within each arm across runs; between arms turns 5 and 6 differ after a few
tokens (batch-composition rounding, as the invariant says).
- Disk cache: `fork-validate.sh --diskcache` (in the window, GPU leg included) 15 PASS, 0 FAIL, 0
SKIP. `diskcache-window3.sh` legs with production's 35B command line (`q35.py` copied to
`q35-usw.py` with the candidate build path; log `/root/arms/usw-q35-mm.log`): 18.8k-token chat,
turn 1 83.5 s, save 557 MB in 1.0 s, restart restore and turn 2 `prompt_n` 27 in 3.7 s, output ==
reference; cold save and restore identity (`prompt_n` 4, output == reference); image answered
'Red' and not saved. The saved state after turn 2 holds 4 checkpoints instead of 5 (the user-start
one is gone).
**Production (deployed 2026-11-22 10:37, `f3397cded`, config unchanged).** Deploy script
`/root/arms/dep-usw.sh` (log `dep-usw.log`): incremental in-place build, downtime 33 s. Post-deploy
checks through llama-swap (`/root/arms/post-usw.py`, `ink-usw.py`, `ink2-usw.py`, logs beside them):
`system_fingerprint` `b12687-f3397cded`; 35B 'Paris', then 4B / 35B handoffs twice (correct answers,
`holder` the answering PID each time, 2.9-6.3 s per switch); a 1140-token 35B chat, then a gemma-26B
image ('Red', the 35B exited by itself with `loop stopped, device memory freed` and `disk cache:
saved the prompt of 1157 tokens`), gemma text, 4B text, then the 35B chat plus one user message:
`disk cache: restored 1157 tokens`, `prompt_n` 30 (the suffix only). Inkling nothink: text correct
(it still spends about 40 tokens on reasoning, so a 96-token budget returns an empty content), image
'Red', a 9293-token Pan Tadeusz prompt at 82.0 t/s with the right answer (F13 measured 81.6). Children
bind llama-swap's ports. Log scan clean, `dmesg` clean (no ring timeout, GPU hang or OOM kill), pod
restarts 0 as before.
**Failure modes and incidents.** None in validation or the deploy. Known cost: an edit of the last
user message (or per-request context injected into it) re-processes the previous answer too, +4.3 s
measured above.
**Rebase hazards.**
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
`checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch
start is still checkpointed when the batch ends at prompt end minus 4.
**Validation plan (run 2026-11-22, results under Measured above).**
1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the
`created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch
`[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the
last user message.
2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded),
candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn
chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read
`prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens
(the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens.
3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit
the last user message and resend: merged re-processes previous answer + message (bounded by one
ubatch), unmerged only the message; no "forcing full prompt re-processing".
4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line):
restore after a restart, `prompt_n` and output identical to the reference built from the same
commit.
### F15. AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
Branch `fable5/nvfp4-avx2` (on the production branch tip `2caae7e11`). Not deployed. No knob: the
old kernel is gone from the AVX2 path (drop = revert the commit).
**What and why.** Inkling-2-Mini UD-Q4_K_XL stores the experts of 20 of its 24 layers as NVFP4
(ggml type 40: `block_nvfp4` = 4 UE4M3 scale bytes, one per 16 elements, then 32 bytes of E2M1
nibbles; within each 8-byte sub-block group the low nibbles are elements 0-7 and the high nibbles
8-15; values are the doubled E2M1 table `kvalues_fp4` and `ggml_ue4m3_to_fp32` returns the scale
times 0.5; the per-expert `ffn_*_exps.scale` tensor is a separate graph multiply). The host experts
(`-ot ...exps=CPU`) decode on the CPU through `ggml_vec_dot_nvfp4_q8_0`: the F13 profile had it at
21% of decode CPU time, plus 36% in the OpenMP barrier. CPU_REPACK has no NVFP4 path, so vec_dot is
what runs. Upstream's AVX2 loop did one block per iteration with 4 scalar UE4M3 table lookups, 2
scalar FP16 conversions, 4 scalar muls, broadcasts and 2 FMAs on one accumulator.
**Where.** `ggml/src/ggml-cpu/arch/x86/quants.c`: `ue4m3x8_to_fp32_half` and the `__AVX2__` branch
of `ggml_vec_dot_nvfp4_q8_0`. `tests/test-backend-ops.cpp`: perf cases at Inkling's expert shapes
(nvfp4, mxfp4, q4_0; `m=2304`, k 2304 and 1152, n 1/8/16/32, MUL_MAT and MUL_MAT_ID 128x8). Commit
`78a204bb2`.
**How it works.** Two blocks (128 weights) per iteration: one 32-byte load of `qs` per block, low and
high nibbles through a 256-bit `pshufb` of `kvalues_fp4`, `unpack{lo,hi}_epi64` + `permute2x128`
into the q8_0 element order, sign trick + `maddubs` against the two q8_0 blocks. The 8 sub-block sums
are reduced exactly in int16 (at most 16*12*127 = 24384) by two levels of `hadd_epi16` and one
`madd`, lanes `[b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]`. The 8 scale bytes are shuffled into the
same lane order and decoded in one vector: normal `(x & 0x7F) << 20` plus exponent bias 119 (bias 7
and the 0.5), subnormal `m * 2^-10` by blend, `0x7F` -> 0; bit-identical to `ggml_ue4m3_to_fp32` for
all 256 bytes (including the sign-bit quirk: 0xFF decodes to 240, as in the reference). The 4 q8_0
FP16 scales are packed in a GPR and converted with one `vcvtph2ps`. One mul + one FMA per
iteration. An odd last block takes the existing scalar tail. The AVX-only path and the generic
reference are untouched.
Tried and not kept (standalone harness `/root/arms/nvfp4-bench/nvfp4bench.c`, single core, min of
400 trials): 4 blocks per iteration with two accumulators (no change), `srlv`-based nibble split with
the q8_0 vector permuted instead (no change), `packs_epi32(madd)` instead of `hadd_epi16` (no
change), the scale decode via a denormal multiply (`as_float(x << 20) * 2^119`, -4% time but wrong
under DAZ, so not used). Ablation of the kept kernel: nibble decode + dot product about 60% of the
time, the reduction 17%, the scales 23%. Zen 2 runs it at about 2.7 IPC with FP pipe 2 the busiest
(`perf stat` `fpu_pipe_assignment`), not at the 4-pipe limit.
**Correctness.**
- Standalone: bit-exact scale decode for all 256 bytes and the lane order; random blocks (wild and
realistic scale bytes, n 64..2368, odd block counts) against the generic reference: worst error
3.7e-8 of the sum of |terms|.
- Linked against the built libraries (`vecdot_check.cpp`): 1400 rows of Gaussian weights at 4 scales
(1e-3 drives the UE4M3 scales into subnormals), quantized with the library, vs the exact dot of the
dequantized operands in double: new 5.5e-8, old 4.2e-8 (float rounding of the accumulation order).
- `test-quantize-fns`: nvfp4 dot product error 0.019774 in both builds (identical).
- `test-backend-ops -b CPU` cannot test a CPU kernel (it compares the CPU backend with itself).
**Measured (2026-11-22, pod, beside production, CPU-only scratch builds `-DGGML_VULKAN=OFF` in
`/root/arms/nvfp4-base` (`2caae7e11`) and `/root/arms/nvfp4`; logs in `/root/arms/nvfp4-bench/`).**
- Kernel, one pinned core, 64 rows L2-resident, min of 3000 trials, through the library's type
traits (`vecdot_perf.cpp`, `vp1.log`, 4 runs per arm, spread 0.2 ns): k=2304 129.0 -> 113.8 ns/row
(1.13x), k=1152 67.6 -> 59.5 (1.14x). Unchanged controls: q4_0 96.6 / 51.6, mxfp4 103.0 / 55.0 in
both builds. NVFP4 is still about 18% slower per row than q4_0.
- 6 threads streaming 512 MiB from DRAM (decode-like): every type, old or new, sits at the same
21-23 GB/s (best of 15; medians 11-21 GB/s under the co-tenants). At 6 threads the expert dot
product is DRAM-bound on this node, so the kernel speedup mostly turns into idle time, not decode
speed. A decode token streams about 1.3 GB of host NVFP4 experts at production's k=4 (8 experts x
36 layer passes x 35.8 MB).
- `test-backend-ops perf` at 6 threads is useless here: the unchanged q4_0 and mxfp4 cases varied
0.6x-1.7x between the two builds.
- llama-bench CPU-only decode, `-ngl 0 -t 6 -p 0`, arms B N N B, 6 rounds: rounds 1-2 tg32 x3 (`lb.log`): base 5.23 5.12 5.04 5.19, new 5.52 5.34 5.23 5.15 t/s,
medians 5.16 -> 5.29 (+2.5%); rounds 3-6 tg64 x5 (`lb2.log`): base 4.53 5.10 4.87 5.31 5.02 4.65 4.79
4.93, new 4.98 5.24 5.34 5.09 4.70 4.38 5.02 5.18, medians 4.90 -> 5.06 (+3.2%). The new build won 5
of the 6 rounds (mean of its two arms against the base pair). Below the 5% bar of section 6, so a
likely +2-3%, not proven. CPU-only is the upper bound of the effect: every layer's experts and
attention run on the CPU here; with production's k=4 the NVFP4 share of a token is smaller.
**What the GPU+CPU window should measure.** Production command line (`inkling_args`, k=4 `-ot`),
llama-bench `-lm none -ngl 99 -ot <k=4 pattern> -fa 1 -t 6 -b 1024 -ub 1024 -p 0 -n 256 -d 4096 -r 3`
(the F13 sweep's tg256 @ d4096 cell), base `2caae7e11` vs this branch, B N N B, medians (section 6).
Expected: decode +0-3% (DRAM-bound), prefill unchanged (prefill MUL_MAT_ID runs on the GPU for
ubatches >= 32). Optionally `perf record` the decode: `ggml_vec_dot_nvfp4_q8_0` should drop from 21%
by roughly a tenth and the barrier share rise. Then a greedy text request against the base build
(temperature 0): output may differ only by float rounding.
**How to drop it.** Revert `78a204bb2` (the kernel and the perf cases are independent hunks). Drop it
when upstream rewrites this AVX2 path.
**Rebase hazards.** Upstream changes to `block_nvfp4` (layout, scale semantics, the 0.5 convention in
`ggml_ue4m3_to_fp32`, the `0x7F`/sign handling TODO there) must be mirrored in
`ue4m3x8_to_fp32_half`; the standalone check (`nvfp4bench.c`) tests all 256 bytes against the
reference.
---
## 4. Dead ends
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
+12 -5
View File
@@ -348,6 +348,13 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- F14 run (2026-11-22, candidate `f3397cded` = `166ec22e2` (docs on `b13c41355`) + F14, server-only change; one
swapdown window, `/root/arms/usw-window.sh`, log `/root/arms/usw-window.log`, about 15 minutes):
the 35B TTFT A/B with and without `LLAMA_USERSTART_MERGE=0` (FORK.md F14, M U U M, production 35B
think command line with the mmproj), `--diskcache` 15 PASS, 0 FAIL, 0 SKIP, and the
`diskcache-window3.sh` 35B legs (all passed). The full `--perf`/`--arbiter` set was not rerun: the
change touches only the server prompt-batch loop. Deploy downtime 33 s (incremental in-place
build).
- Last full run (2026-11-22, candidate `b13c41355` = `3621025e8` + F13 vs reference `3621025e8`, one
swapdown window, `--ops --concat --diskcache --arbiter --perf --moe-exact --moecache`, log
`/root/arms/overnight/fv.log`): 52 PASS, 0 FAIL, 0 SKIP in 41 minutes. Perf medians ref -> cand:
@@ -411,8 +418,8 @@ Run the stop, build and start (steps 4-7) as one script on the pod, started with
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 15 (holds 3621025e8)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-16.txt
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 16 (holds b13c41355)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-17.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3):
@@ -613,10 +620,10 @@ chain in the exec shell: it can die with the exec session halfway, leaving llama
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
Pointer files 2 to 15 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
Pointer files 2 to 16 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, commit-12 holds
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, commit-14 holds `3bf6b55b3`, and commit-15 holds `3621025e8`, the rollback target
for the current production `b13c41355` (next free: 16). The formats vary
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, commit-14 holds `3bf6b55b3`, commit-15 holds `3621025e8`, and commit-16 holds `b13c41355`, the rollback target
for the current production `f3397cded` (next free: 17; build copy `/root/arms/build-b13c41355`). The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
+53 -37
View File
@@ -1083,6 +1083,29 @@ void ggml_vec_dot_mxfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
*s = sumf;
}
#if defined(__AVX2__)
// The 8 UE4M3 scales of two NVFP4 blocks as fp32, equal to GGML_CPU_UE4M3_TO_FP32 (x0.5, 0x7F -> 0, sign bit
// ignored, subnormals), in the lane order of the reduction in ggml_vec_dot_nvfp4_q8_0:
// [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
static inline __m256 ue4m3x8_to_fp32_half(const uint8_t * d0, const uint8_t * d1) {
uint32_t a, b;
memcpy(&a, d0, sizeof(a));
memcpy(&b, d1, sizeof(b));
const __m256i v = _mm256_set1_epi64x((int64_t) (((uint64_t) b << 32) | a));
const __m256i x = _mm256_shuffle_epi8(v, _mm256_setr_epi8(
0, -1, -1, -1, 2, -1, -1, -1, 4, -1, -1, -1, 6, -1, -1, -1,
1, -1, -1, -1, 3, -1, -1, -1, 5, -1, -1, -1, 7, -1, -1, -1));
const __m256i nan = _mm256_cmpeq_epi32(x, _mm256_set1_epi32(0x7F));
const __m256i xm = _mm256_and_si256(x, _mm256_set1_epi32(0x7F));
// normal: fp32 exponent e + 119 (bias 7, times 0.5), mantissa bits in place
const __m256 fn = _mm256_castsi256_ps(_mm256_add_epi32(_mm256_slli_epi32(xm, 20), _mm256_set1_epi32(119 << 23)));
// subnormal (e == 0): m * 2^-10
const __m256 fs = _mm256_mul_ps(_mm256_cvtepi32_ps(xm), _mm256_set1_ps(1.0f/1024.0f));
const __m256 f = _mm256_blendv_ps(fn, fs, _mm256_castsi256_ps(_mm256_cmpgt_epi32(_mm256_set1_epi32(8), xm)));
return _mm256_andnot_ps(_mm256_castsi256_ps(nan), f);
}
#endif
void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
assert(nrc == 1);
UNUSED(nrc);
@@ -1100,47 +1123,40 @@ void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
#if defined(__AVX2__)
const __m128i values128 = _mm_loadu_si128((const __m128i*)kvalues_fp4);
const __m128i m4b = _mm_set1_epi8(0x0f);
const __m256i mone = _mm256_set1_epi16(1);
const __m256i values = _mm256_broadcastsi128_si256(_mm_loadu_si128((const __m128i *) kvalues_fp4));
const __m256i m4b = _mm256_set1_epi8(0x0f);
const __m256i mone = _mm256_set1_epi16(1);
// two blocks per iteration: 8 sub-block sums, 8 scales and one FMA
__m256 accum = _mm256_setzero_ps();
for(; ib < nb; ib++){
for (; ib + 1 < nb; ib += 2) {
__m256i p[4];
for (int k = 0; k < 2; ++k) {
// 8 bytes per sub-block: low nibbles are elements 0-7, high nibbles 8-15
const __m256i q = _mm256_loadu_si256((const __m256i *) x[ib + k].qs);
const __m256i lo = _mm256_shuffle_epi8(values, _mm256_and_si256(q, m4b));
const __m256i hi = _mm256_shuffle_epi8(values, _mm256_and_si256(_mm256_srli_epi16(q, 4), m4b));
const __m256i a = _mm256_unpacklo_epi64(lo, hi); // [sub0 | sub2]
const __m256i b = _mm256_unpackhi_epi64(lo, hi); // [sub1 | sub3]
p[2*k + 0] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x20), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 0].qs));
p[2*k + 1] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x31), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 1].qs));
}
// a sub-block sum is at most 16*12*127, so it fits in int16:
// lanes [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
const __m256i h = _mm256_hadd_epi16(_mm256_hadd_epi16(p[0], p[1]), _mm256_hadd_epi16(p[2], p[3]));
const __m256 sums = _mm256_cvtepi32_ps(_mm256_madd_epi16(h, mone));
const __m128i q4bits_01 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 0));
const __m128i q4bits_23 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 16));
#if defined(__F16C__)
// assembled in a GPR: cheaper than four inserts
const uint64_t dyb = (uint64_t) y[2*ib + 0].d | ((uint64_t) y[2*ib + 1].d << 16) |
((uint64_t) y[2*ib + 2].d << 32) | ((uint64_t) y[2*ib + 3].d << 48);
const __m128 dy = _mm_cvtph_ps(_mm_cvtsi64_si128((int64_t) dyb));
#else
const __m128 dy = _mm_setr_ps(GGML_CPU_FP16_TO_FP32(y[2*ib + 0].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 1].d),
GGML_CPU_FP16_TO_FP32(y[2*ib + 2].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 3].d));
#endif
const __m256i q8_01 = _mm256_loadu_si256((const __m256i *)y[2*ib + 0].qs);
const __m256i q8_23 = _mm256_loadu_si256((const __m256i *)y[2*ib + 1].qs);
const __m128i q4_01_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_01, m4b));
const __m128i q4_01_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_01, 4), m4b));
const __m128i q4_23_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_23, m4b));
const __m128i q4_23_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_23, 4), m4b));
//reordering
const __m256i q4_01 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_01_lo,q4_01_hi), _mm_unpacklo_epi64(q4_01_lo,q4_01_hi));
const __m256i q4_23 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_23_lo,q4_23_hi),_mm_unpacklo_epi64(q4_23_lo,q4_23_hi));
const __m256i p01 = mul_add_epi8(q4_01,q8_01);
const __m256i p_1 = _mm256_madd_epi16(p01, mone);
const __m256i p23 = mul_add_epi8(q4_23,q8_23);
const __m256i p_2 = _mm256_madd_epi16(p23, mone);
const float dy0 = GGML_CPU_FP16_TO_FP32(y[2*ib].d);
const float dy1 = GGML_CPU_FP16_TO_FP32(y[2*ib+1].d);
const float s0 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[0]) * dy0;
const float s1 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[1]) * dy0;
const float s2 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[2]) * dy1;
const float s3 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[3]) * dy1;
const __m256 scales01 = _mm256_set_m128(_mm_set1_ps(s1), _mm_set1_ps(s0));
const __m256 scales23 = _mm256_set_m128(_mm_set1_ps(s3), _mm_set1_ps(s2));
accum = _mm256_fmadd_ps(scales01, _mm256_cvtepi32_ps(p_1), accum);
accum = _mm256_fmadd_ps(scales23, _mm256_cvtepi32_ps(p_2), accum);
accum = _mm256_fmadd_ps(_mm256_mul_ps(ue4m3x8_to_fp32_half(x[ib].d, x[ib + 1].d), _mm256_set_m128(dy, dy)), sums, accum);
}
sumf = hsum_float_8(accum);
+10
View File
@@ -14178,6 +14178,16 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_perf() {
}
}
// Inkling-2-Mini experts (NVFP4 in UD-Q4_K_XL): 128 experts, 8 used, n_embd 2304, gate_up 2x1152 fused, down 1152
for (int bs : {1, 8, 16, 32}) {
for (ggml_type type_a : {GGML_TYPE_NVFP4, GGML_TYPE_MXFP4, GGML_TYPE_Q4_0}) {
test_cases.emplace_back(new test_mul_mat(type_a, GGML_TYPE_F32, 2304, bs, 2304, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(type_a, GGML_TYPE_F32, 2304, bs, 1152, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat_id(type_a, GGML_TYPE_F32, 128, 8, false, 2304, bs, 2304));
test_cases.emplace_back(new test_mul_mat_id(type_a, GGML_TYPE_F32, 128, 8, false, 2304, bs, 1152));
}
}
for (int K : {3, 5}) {
for (int IC : {256, 2560}) {
for (int IW_IH : {32, 64, 256}) {
+14 -1
View File
@@ -4942,6 +4942,19 @@ private:
const int32_t n_decision_first = slot.task->type == SERVER_TASK_TYPE_DECISION ? slot.task->decision.pos_first() : -1;
// [fork F14] do not break at user-message starts when the rest of the prompt
// from here fits in one ubatch: the break turns one ubatch into two, and with
// MoE experts on the host each one of >= 32 tokens uploads the experts again.
// A checkpoint at this batch start (the near-end rule below) and at the end
// minus 4 still exist, so a later prompt that diverges inside the merged range
// re-processes less than one ubatch more. LLAMA_USERSTART_MERGE=0 disables.
static const bool userstart_merge = [] {
const char * env = getenv("LLAMA_USERSTART_MERGE");
return env == nullptr || std::string(env) != "0";
}();
const bool merge_user_starts = userstart_merge && !has_mtmd &&
slot.task->n_tokens() - slot.prompt.n_tokens() <= n_ubatch;
// add prompt tokens for processing in the current batch
while (slot.prompt.n_tokens() < slot.task->n_tokens() && batch.size() < n_batch) {
// get next token to process
@@ -4985,7 +4998,7 @@ private:
slot.prompt.tokens.push_back(cur_tok);
// break at the last user message, or at user messages at least min step past the last checkpoint
if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) {
if (do_checkpoint && !merge_user_starts && spans.is_user_start(slot.prompt.n_tokens())) {
const auto pos = slot.prompt.n_tokens();
const auto & checkpoints = slot.prompt.checkpoints;