|
|
|
@@ -6,14 +6,14 @@ which also holds the operations reference (pod access, files, recovery).
|
|
|
|
|
|
|
|
|
|
Status at the time of writing (2026-11-22):
|
|
|
|
|
|
|
|
|
|
- Production runs `b13c41355` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
|
|
|
|
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, and F13, the
|
|
|
|
|
per-pass loop views for Inkling), deployed 2026-11-22 08:47 with the Inkling placement change
|
|
|
|
|
(section 5, F13). Validated with the full `fork-validate.sh` set in one window (52 PASS, 0 FAIL,
|
|
|
|
|
0 SKIP; runbook section 6). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-15.txt` = `3621025e8`
|
|
|
|
|
(build copy `/root/arms/build-3621025e8`, config backup `/root/config.yaml.bak-20261122-predeploy`;
|
|
|
|
|
restore both, the old build has no F13); next free pointer 16.
|
|
|
|
|
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12683-b13c41355`) and changes
|
|
|
|
|
- Production runs `f3397cded` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
|
|
|
|
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, F13, the
|
|
|
|
|
per-pass loop views for Inkling, and F14, the user-message batch-break merge), deployed 2026-11-22
|
|
|
|
|
10:37 (F14 validated in its own window: 35B TTFT A/B, disk cache 15 PASS and the 35B legs; F14
|
|
|
|
|
measured). Rollback pointer `/root/DEPLOY-ROLLBACK-commit-16.txt` = `b13c41355` (build copy
|
|
|
|
|
`/root/arms/build-b13c41355`; config unchanged by this deploy); next free pointer 17. The previous
|
|
|
|
|
pointer 15 = `3621025e8` needs its config backup `/root/config.yaml.bak-20261122-predeploy` too.
|
|
|
|
|
`system_fingerprint` is `b<build number>-<short sha>` (currently `b12687-f3397cded`) and changes
|
|
|
|
|
with each rebase or carry. The hand ports of the 2026-11-21 rebase are listed in the runbook
|
|
|
|
|
(section 3, "2026-11-21").
|
|
|
|
|
- Previous branch `fable5/vram-ondemand-rebase-20261024` on upstream `3f03b1772` (production
|
|
|
|
@@ -55,6 +55,8 @@ Status at the time of writing (2026-11-22):
|
|
|
|
|
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
|
|
|
|
|
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
|
|
|
|
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
|
|
|
|
- F14 User-message batch-break merge (one ubatch per short turn)
|
|
|
|
|
- F15 AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
|
|
|
|
|
4. Dead ends
|
|
|
|
|
5. Configuration that matters
|
|
|
|
|
6. Measurement methodology and traps
|
|
|
|
@@ -139,6 +141,8 @@ Oldest first. "Feature" points at the section below.
|
|
|
|
|
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
|
|
|
|
|
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
|
|
|
|
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
|
|
|
|
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
|
|
|
|
|
| 78a204bb2 | ggml-cpu: faster AVX2 NVFP4 x Q8_0 dot product (two blocks per iteration, vector UE4M3 scales) (branch `fable5/nvfp4-avx2`, not deployed) | F15 |
|
|
|
|
|
|
|
|
|
|
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
|
|
|
|
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
|
|
|
@@ -1397,6 +1401,228 @@ task-model call (arbiter handoff, F4), identical at temperature 0.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### F14. User-message batch-break merge (one ubatch per short turn)
|
|
|
|
|
|
|
|
|
|
Branch `fable5/userstart-split` (rebased onto the F13 production tip `166ec22e2`), fast-forwarded into
|
|
|
|
|
the production branch on 2026-11-22 (deploy and results below). Knob: `LLAMA_USERSTART_MERGE=0`.
|
|
|
|
|
|
|
|
|
|
**What and why.** Upstream ends a prompt batch right before a user message so that a context
|
|
|
|
|
checkpoint lands there (#22929 for the last user message, #24176 for every user message, #25420
|
|
|
|
|
limited the earlier ones to `--checkpoint-min-step`). On a hybrid/recurrent model the recurrent state
|
|
|
|
|
cannot be cut back, so a checkpoint is the only way to reuse a prefix; the one at the last user
|
|
|
|
|
message is what a later request uses when it diverges inside that message (the user edits it, or the
|
|
|
|
|
client injected per-request context into it, RAG/web search). Ordinary turns do not need it: Qwen's
|
|
|
|
|
template drops the previous answer's reasoning, which moves the divergence to just after
|
|
|
|
|
`<|im_start|>assistant\n`, below the end-minus-4 checkpoint (#20288), and that one is what turn N+1
|
|
|
|
|
restores. The cost on this hardware: a turn processes `[previous answer | new user message + generation
|
|
|
|
|
prompt]`. With the break that is two ubatches instead of one, and with every expert on the host each
|
|
|
|
|
ubatch of >= 32 tokens (`GGML_OP_OFFLOAD_MIN_BATCH`) uploads the experts again: all of them from 128
|
|
|
|
|
tokens (F1), about 64-96% of them at 32-127 tokens (routing readback), about 2.8 s per full upload on
|
|
|
|
|
the 35B. The break also fires at the first user message of a fresh prompt (no checkpoints yet),
|
|
|
|
|
which splits off the system prompt. Tool results count as user messages (Qwen opens them with
|
|
|
|
|
`<|im_start|>user`, the detected delimiter).
|
|
|
|
|
|
|
|
|
|
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
|
|
|
|
|
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
|
|
|
|
|
Commit 510aa6e09.
|
|
|
|
|
|
|
|
|
|
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
|
|
|
|
|
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
|
|
|
|
|
iteration, the user-start break is skipped. The batch then runs to the end-minus-4 break: one ubatch
|
|
|
|
|
plus the 4-token tail, as for a prompt without user messages. Checkpoints left: the batch start
|
|
|
|
|
(the near-prompt-end rule; the batch ends at prompt end minus 4, so it always applies; it supersedes
|
|
|
|
|
the restored checkpoint at the same position, with the current task id, so the next request still
|
|
|
|
|
has it) and the prompt end minus 4. Lost: the one at the last user message. A later request that
|
|
|
|
|
diverges inside that message restores the batch-start checkpoint instead and re-processes the
|
|
|
|
|
previous answer too, less than one ubatch, in the same single ubatch, so no extra upload. Prompts
|
|
|
|
|
whose rest is longer than one ubatch keep the upstream breaks (there the break does not save the
|
|
|
|
|
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
|
|
|
|
|
upstream behaviour everywhere.
|
|
|
|
|
|
|
|
|
|
**Invariants that must hold.**
|
|
|
|
|
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
|
|
|
|
|
divergence inside the last user message to less than `n_ubatch` tokens.
|
|
|
|
|
- The batch-start checkpoint must still be created (near-prompt-end rule, `do_checkpoint` not cleared
|
|
|
|
|
by `has_mtmd`); hence the `!has_mtmd` condition.
|
|
|
|
|
- It only changes where batches end, never which tokens are processed, so outputs differ from an
|
|
|
|
|
unmerged run only by batch-composition rounding.
|
|
|
|
|
|
|
|
|
|
**Measured (2026-11-22, pod, GPU to itself in one swapdown window, build `f3397cded`; driver
|
|
|
|
|
`/root/arms/usw_ab.py`, window script `/root/arms/usw-window.sh`, logs and `res.json` per arm in
|
|
|
|
|
`/root/arms/usw-ab/{M1,U1,U2,M2}/`).** Production 35B think entry (`q36t`: `${ctx_256k}
|
|
|
|
|
${qwen35_think_args} ${moe_offload_args}`, `-hf` pulls the mmproj, supervisord's environment, scratch
|
|
|
|
|
arena and `--cache-disk-path`, plus `-lv 4`), one fresh server per arm, arms M (merged, default) and U
|
|
|
|
|
(`LLAMA_USERSTART_MERGE=0`) in the order M U U M. Each chat: system prompt, 6 user turns from
|
|
|
|
|
`pt-chunk.txt` (150, 10, 40, 150, 600, 40 tokens), synthetic previous answers of 300, 800, 1200, 500,
|
|
|
|
|
1300 tokens, `max_tokens 8`, `temperature 0`, streamed; then the last user message edited in the
|
|
|
|
|
middle and resent. TTFT in seconds (mean of the two runs per arm; the single runs in brackets),
|
|
|
|
|
`prompt_n` was identical in all four runs on every turn:
|
|
|
|
|
|
|
|
|
|
| turn | user tokens | prompt_n | M TTFT | U TTFT | M - U |
|
|
|
|
|
|---|---|---|---|---|---|
|
|
|
|
|
| 1 (fresh prompt) | 150 | 196 | 3.99 (4.43, 3.55) | 6.08 (5.73, 6.42) | -2.1 |
|
|
|
|
|
| 2 | 10 | 327 | 4.11 (3.75, 4.47) | 7.66 (6.75, 8.57) | -3.6 |
|
|
|
|
|
| 3 | 40 | 869 | 6.34 (6.62, 6.06) | 9.34 (7.26, 11.42) | -3.0 |
|
|
|
|
|
| 4 | 150 | 1381 | 7.20 (7.30, 7.10) | 10.95 (12.31, 9.58) | -3.7 |
|
|
|
|
|
| 5 | 600 | 1130 | 8.82 (7.47, 10.17) | 9.80 (9.52, 10.07) | -1.0 |
|
|
|
|
|
| 6 | 40 | 1370 | 8.11 (8.01, 8.20) | 9.53 (9.33, 9.72) | -1.4 |
|
|
|
|
|
| edit of turn 6 | 40 | M 1374, U 66 | 7.94 (8.16, 7.71) | 3.65 (4.09, 3.20) | +4.3 |
|
|
|
|
|
|
|
|
|
|
- Batch layout (`prompt processing, n_tokens` and checkpoint lines at `-lv 4`): M runs every turn as
|
|
|
|
|
one batch `[restore point, end-4)` plus the tail, with checkpoints at the batch start (superseding
|
|
|
|
|
the restored one) and at end minus 4; U splits at the last user message (an extra checkpoint
|
|
|
|
|
there, for example 1323 between 515 and 1380 on turn 3) and on turn 1 also at the first user
|
|
|
|
|
message (system prompt, 23 tokens). Turn N+1 restores the end-minus-4 checkpoint of turn N in both
|
|
|
|
|
arms. No `forcing full prompt re-processing` in any run.
|
|
|
|
|
- The 10-token turn gained too, against the plan's "no change". U split it into 308 tokens (the
|
|
|
|
|
previous answer, on the GPU) and 19 (message, generation prompt and tail), and the 19-token part runs
|
|
|
|
|
the experts of all 40 layers on the CPU, which costs seconds rather than nothing at 6 threads
|
|
|
|
|
(inferred from the totals, not profiled). The 600-token turn gained
|
|
|
|
|
least (its two parts are 508 and 618 tokens, both on the GPU, so the split costs one extra
|
|
|
|
|
upload, about 1-2 s against noise of the same size).
|
|
|
|
|
- Edit of the last user message (the cost of the trade): M has no checkpoint at the user-message
|
|
|
|
|
start, restores the batch start and re-processes previous answer + message (1374 tokens, one
|
|
|
|
|
ubatch); U restores the user-start checkpoint and re-processes 66 tokens. +4.3 s per such edit.
|
|
|
|
|
Ordinary follow-up turns are far more frequent than edits of the last message in this use; a client
|
|
|
|
|
that injects per-request context into the last user message (RAG, web search) pays the same
|
|
|
|
|
as an edit on every turn, so set `LLAMA_USERSTART_MERGE=0` for such a client.
|
|
|
|
|
- Outputs: identical within each arm across runs; between arms turns 5 and 6 differ after a few
|
|
|
|
|
tokens (batch-composition rounding, as the invariant says).
|
|
|
|
|
- Disk cache: `fork-validate.sh --diskcache` (in the window, GPU leg included) 15 PASS, 0 FAIL, 0
|
|
|
|
|
SKIP. `diskcache-window3.sh` legs with production's 35B command line (`q35.py` copied to
|
|
|
|
|
`q35-usw.py` with the candidate build path; log `/root/arms/usw-q35-mm.log`): 18.8k-token chat,
|
|
|
|
|
turn 1 83.5 s, save 557 MB in 1.0 s, restart restore and turn 2 `prompt_n` 27 in 3.7 s, output ==
|
|
|
|
|
reference; cold save and restore identity (`prompt_n` 4, output == reference); image answered
|
|
|
|
|
'Red' and not saved. The saved state after turn 2 holds 4 checkpoints instead of 5 (the user-start
|
|
|
|
|
one is gone).
|
|
|
|
|
|
|
|
|
|
**Production (deployed 2026-11-22 10:37, `f3397cded`, config unchanged).** Deploy script
|
|
|
|
|
`/root/arms/dep-usw.sh` (log `dep-usw.log`): incremental in-place build, downtime 33 s. Post-deploy
|
|
|
|
|
checks through llama-swap (`/root/arms/post-usw.py`, `ink-usw.py`, `ink2-usw.py`, logs beside them):
|
|
|
|
|
`system_fingerprint` `b12687-f3397cded`; 35B 'Paris', then 4B / 35B handoffs twice (correct answers,
|
|
|
|
|
`holder` the answering PID each time, 2.9-6.3 s per switch); a 1140-token 35B chat, then a gemma-26B
|
|
|
|
|
image ('Red', the 35B exited by itself with `loop stopped, device memory freed` and `disk cache:
|
|
|
|
|
saved the prompt of 1157 tokens`), gemma text, 4B text, then the 35B chat plus one user message:
|
|
|
|
|
`disk cache: restored 1157 tokens`, `prompt_n` 30 (the suffix only). Inkling nothink: text correct
|
|
|
|
|
(it still spends about 40 tokens on reasoning, so a 96-token budget returns an empty content), image
|
|
|
|
|
'Red', a 9293-token Pan Tadeusz prompt at 82.0 t/s with the right answer (F13 measured 81.6). Children
|
|
|
|
|
bind llama-swap's ports. Log scan clean, `dmesg` clean (no ring timeout, GPU hang or OOM kill), pod
|
|
|
|
|
restarts 0 as before.
|
|
|
|
|
|
|
|
|
|
**Failure modes and incidents.** None in validation or the deploy. Known cost: an edit of the last
|
|
|
|
|
user message (or per-request context injected into it) re-processes the previous answer too, +4.3 s
|
|
|
|
|
measured above.
|
|
|
|
|
|
|
|
|
|
**Rebase hazards.**
|
|
|
|
|
- Upstream keeps reworking this loop (#20288, #22929, #24176, #25420): the user-start break, the
|
|
|
|
|
`checkpoint_offsets` breaks and the near-end / min-step checkpoint rules. Re-check that the batch
|
|
|
|
|
start is still checkpointed when the batch ends at prompt end minus 4.
|
|
|
|
|
|
|
|
|
|
**Validation plan (run 2026-11-22, results under Measured above).**
|
|
|
|
|
1. Pre-check at trace log level (`SLT_TRC`; `-lv` high enough for `LOG_TRC`): per request, the
|
|
|
|
|
`created context checkpoint ... n_tokens = N` and `restored context checkpoint` lines. Merged: one batch
|
|
|
|
|
`[restore point, end-4)` plus `[end-4, end)`; `LLAMA_USERSTART_MERGE=0`: the extra break at the
|
|
|
|
|
last user message.
|
|
|
|
|
2. TTFT A/B through the production 35B entry (exact `moe_offload_args` command line, mmproj loaded),
|
|
|
|
|
candidate with and without `LLAMA_USERSTART_MERGE=0`, arms alternated in one run: scripted 6-turn
|
|
|
|
|
chats, user messages of about 10, 40, 150 and 600 tokens, previous answers 200-1500 tokens. Read
|
|
|
|
|
`prompt_n`/`prompt_ms` from the timings and TTFT from the stream. Expected: no change at 10 tokens
|
|
|
|
|
(the second part ran on the CPU), about -2 to -2.8 s per turn at 40+ tokens.
|
|
|
|
|
3. Reuse: turn N+1 must report the same `prompt_n` in both arms (restore from end-minus-4). Then edit
|
|
|
|
|
the last user message and resend: merged re-processes previous answer + message (bounded by one
|
|
|
|
|
ubatch), unmerged only the message; no "forcing full prompt re-processing".
|
|
|
|
|
4. Disk cache: `fork-validate.sh --diskcache` and `diskcache-window3.sh` (production command line):
|
|
|
|
|
restore after a restart, `prompt_n` and output identical to the reference built from the same
|
|
|
|
|
commit.
|
|
|
|
|
|
|
|
|
|
### F15. AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
|
|
|
|
|
|
|
|
|
|
Branch `fable5/nvfp4-avx2` (on the production branch tip `2caae7e11`). Not deployed. No knob: the
|
|
|
|
|
old kernel is gone from the AVX2 path (drop = revert the commit).
|
|
|
|
|
|
|
|
|
|
**What and why.** Inkling-2-Mini UD-Q4_K_XL stores the experts of 20 of its 24 layers as NVFP4
|
|
|
|
|
(ggml type 40: `block_nvfp4` = 4 UE4M3 scale bytes, one per 16 elements, then 32 bytes of E2M1
|
|
|
|
|
nibbles; within each 8-byte sub-block group the low nibbles are elements 0-7 and the high nibbles
|
|
|
|
|
8-15; values are the doubled E2M1 table `kvalues_fp4` and `ggml_ue4m3_to_fp32` returns the scale
|
|
|
|
|
times 0.5; the per-expert `ffn_*_exps.scale` tensor is a separate graph multiply). The host experts
|
|
|
|
|
(`-ot ...exps=CPU`) decode on the CPU through `ggml_vec_dot_nvfp4_q8_0`: the F13 profile had it at
|
|
|
|
|
21% of decode CPU time, plus 36% in the OpenMP barrier. CPU_REPACK has no NVFP4 path, so vec_dot is
|
|
|
|
|
what runs. Upstream's AVX2 loop did one block per iteration with 4 scalar UE4M3 table lookups, 2
|
|
|
|
|
scalar FP16 conversions, 4 scalar muls, broadcasts and 2 FMAs on one accumulator.
|
|
|
|
|
|
|
|
|
|
**Where.** `ggml/src/ggml-cpu/arch/x86/quants.c`: `ue4m3x8_to_fp32_half` and the `__AVX2__` branch
|
|
|
|
|
of `ggml_vec_dot_nvfp4_q8_0`. `tests/test-backend-ops.cpp`: perf cases at Inkling's expert shapes
|
|
|
|
|
(nvfp4, mxfp4, q4_0; `m=2304`, k 2304 and 1152, n 1/8/16/32, MUL_MAT and MUL_MAT_ID 128x8). Commit
|
|
|
|
|
`78a204bb2`.
|
|
|
|
|
|
|
|
|
|
**How it works.** Two blocks (128 weights) per iteration: one 32-byte load of `qs` per block, low and
|
|
|
|
|
high nibbles through a 256-bit `pshufb` of `kvalues_fp4`, `unpack{lo,hi}_epi64` + `permute2x128`
|
|
|
|
|
into the q8_0 element order, sign trick + `maddubs` against the two q8_0 blocks. The 8 sub-block sums
|
|
|
|
|
are reduced exactly in int16 (at most 16*12*127 = 24384) by two levels of `hadd_epi16` and one
|
|
|
|
|
`madd`, lanes `[b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]`. The 8 scale bytes are shuffled into the
|
|
|
|
|
same lane order and decoded in one vector: normal `(x & 0x7F) << 20` plus exponent bias 119 (bias 7
|
|
|
|
|
and the 0.5), subnormal `m * 2^-10` by blend, `0x7F` -> 0; bit-identical to `ggml_ue4m3_to_fp32` for
|
|
|
|
|
all 256 bytes (including the sign-bit quirk: 0xFF decodes to 240, as in the reference). The 4 q8_0
|
|
|
|
|
FP16 scales are packed in a GPR and converted with one `vcvtph2ps`. One mul + one FMA per
|
|
|
|
|
iteration. An odd last block takes the existing scalar tail. The AVX-only path and the generic
|
|
|
|
|
reference are untouched.
|
|
|
|
|
|
|
|
|
|
Tried and not kept (standalone harness `/root/arms/nvfp4-bench/nvfp4bench.c`, single core, min of
|
|
|
|
|
400 trials): 4 blocks per iteration with two accumulators (no change), `srlv`-based nibble split with
|
|
|
|
|
the q8_0 vector permuted instead (no change), `packs_epi32(madd)` instead of `hadd_epi16` (no
|
|
|
|
|
change), the scale decode via a denormal multiply (`as_float(x << 20) * 2^119`, -4% time but wrong
|
|
|
|
|
under DAZ, so not used). Ablation of the kept kernel: nibble decode + dot product about 60% of the
|
|
|
|
|
time, the reduction 17%, the scales 23%. Zen 2 runs it at about 2.7 IPC with FP pipe 2 the busiest
|
|
|
|
|
(`perf stat` `fpu_pipe_assignment`), not at the 4-pipe limit.
|
|
|
|
|
|
|
|
|
|
**Correctness.**
|
|
|
|
|
- Standalone: bit-exact scale decode for all 256 bytes and the lane order; random blocks (wild and
|
|
|
|
|
realistic scale bytes, n 64..2368, odd block counts) against the generic reference: worst error
|
|
|
|
|
3.7e-8 of the sum of |terms|.
|
|
|
|
|
- Linked against the built libraries (`vecdot_check.cpp`): 1400 rows of Gaussian weights at 4 scales
|
|
|
|
|
(1e-3 drives the UE4M3 scales into subnormals), quantized with the library, vs the exact dot of the
|
|
|
|
|
dequantized operands in double: new 5.5e-8, old 4.2e-8 (float rounding of the accumulation order).
|
|
|
|
|
- `test-quantize-fns`: nvfp4 dot product error 0.019774 in both builds (identical).
|
|
|
|
|
- `test-backend-ops -b CPU` cannot test a CPU kernel (it compares the CPU backend with itself).
|
|
|
|
|
|
|
|
|
|
**Measured (2026-11-22, pod, beside production, CPU-only scratch builds `-DGGML_VULKAN=OFF` in
|
|
|
|
|
`/root/arms/nvfp4-base` (`2caae7e11`) and `/root/arms/nvfp4`; logs in `/root/arms/nvfp4-bench/`).**
|
|
|
|
|
- Kernel, one pinned core, 64 rows L2-resident, min of 3000 trials, through the library's type
|
|
|
|
|
traits (`vecdot_perf.cpp`, `vp1.log`, 4 runs per arm, spread 0.2 ns): k=2304 129.0 -> 113.8 ns/row
|
|
|
|
|
(1.13x), k=1152 67.6 -> 59.5 (1.14x). Unchanged controls: q4_0 96.6 / 51.6, mxfp4 103.0 / 55.0 in
|
|
|
|
|
both builds. NVFP4 is still about 18% slower per row than q4_0.
|
|
|
|
|
- 6 threads streaming 512 MiB from DRAM (decode-like): every type, old or new, sits at the same
|
|
|
|
|
21-23 GB/s (best of 15; medians 11-21 GB/s under the co-tenants). At 6 threads the expert dot
|
|
|
|
|
product is DRAM-bound on this node, so the kernel speedup mostly turns into idle time, not decode
|
|
|
|
|
speed. A decode token streams about 1.3 GB of host NVFP4 experts at production's k=4 (8 experts x
|
|
|
|
|
36 layer passes x 35.8 MB).
|
|
|
|
|
- `test-backend-ops perf` at 6 threads is useless here: the unchanged q4_0 and mxfp4 cases varied
|
|
|
|
|
0.6x-1.7x between the two builds.
|
|
|
|
|
- llama-bench CPU-only decode, `-ngl 0 -t 6 -p 0`, arms B N N B, 6 rounds: rounds 1-2 tg32 x3 (`lb.log`): base 5.23 5.12 5.04 5.19, new 5.52 5.34 5.23 5.15 t/s,
|
|
|
|
|
medians 5.16 -> 5.29 (+2.5%); rounds 3-6 tg64 x5 (`lb2.log`): base 4.53 5.10 4.87 5.31 5.02 4.65 4.79
|
|
|
|
|
4.93, new 4.98 5.24 5.34 5.09 4.70 4.38 5.02 5.18, medians 4.90 -> 5.06 (+3.2%). The new build won 5
|
|
|
|
|
of the 6 rounds (mean of its two arms against the base pair). Below the 5% bar of section 6, so a
|
|
|
|
|
likely +2-3%, not proven. CPU-only is the upper bound of the effect: every layer's experts and
|
|
|
|
|
attention run on the CPU here; with production's k=4 the NVFP4 share of a token is smaller.
|
|
|
|
|
|
|
|
|
|
**What the GPU+CPU window should measure.** Production command line (`inkling_args`, k=4 `-ot`),
|
|
|
|
|
llama-bench `-lm none -ngl 99 -ot <k=4 pattern> -fa 1 -t 6 -b 1024 -ub 1024 -p 0 -n 256 -d 4096 -r 3`
|
|
|
|
|
(the F13 sweep's tg256 @ d4096 cell), base `2caae7e11` vs this branch, B N N B, medians (section 6).
|
|
|
|
|
Expected: decode +0-3% (DRAM-bound), prefill unchanged (prefill MUL_MAT_ID runs on the GPU for
|
|
|
|
|
ubatches >= 32). Optionally `perf record` the decode: `ggml_vec_dot_nvfp4_q8_0` should drop from 21%
|
|
|
|
|
by roughly a tenth and the barrier share rise. Then a greedy text request against the base build
|
|
|
|
|
(temperature 0): output may differ only by float rounding.
|
|
|
|
|
|
|
|
|
|
**How to drop it.** Revert `78a204bb2` (the kernel and the perf cases are independent hunks). Drop it
|
|
|
|
|
when upstream rewrites this AVX2 path.
|
|
|
|
|
|
|
|
|
|
**Rebase hazards.** Upstream changes to `block_nvfp4` (layout, scale semantics, the 0.5 convention in
|
|
|
|
|
`ggml_ue4m3_to_fp32`, the `0x7F`/sign handling TODO there) must be mirrored in
|
|
|
|
|
`ue4m3x8_to_fp32_half`; the standalone check (`nvfp4bench.c`) tests all 256 bytes against the
|
|
|
|
|
reference.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## 4. Dead ends
|
|
|
|
|
|
|
|
|
|
Measured on this hardware and rejected. Each entry has the number and the mechanism. Do not retry
|
|
|
|
|