Compare commits

Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 9573fe0a2f docs: FORK.md F15, AVX2 NVFP4 x Q8_0 dot product (not deployed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 21:38:01 +01:00
LumpiastyandClaude Opus 5.5 78a204bb2f ggml-cpu: faster AVX2 NVFP4 x Q8_0 dot product (two blocks per iteration, vector UE4M3 scales)
The x86 AVX2 ggml_vec_dot_nvfp4_q8_0 did one block per iteration with four
scalar UE4M3 table lookups, two scalar FP16 conversions, four scalar muls and
broadcasts per 64 weights, plus two FMAs on one accumulator chain.

New AVX2 path, two blocks (128 weights) per iteration:
- E2M1 decode with 256-bit pshufb on the whole 32-byte qs, unpack + permute2x128
  into the q8_0 element order (low nibbles are elements 0-7 of a sub-block,
  high nibbles 8-15);
- the 8 sub-block sums reduced exactly in int16 (a sub-block sum is at most
  16*12*127) with two levels of hadd_epi16 and one madd, one cvt;
- the 8 UE4M3 scales decoded in one vector (bit manipulation, bit-exact with
  ggml_ue4m3_to_fp32 for all 256 bytes: x0.5, 0x7F -> 0, sign bit ignored,
  subnormals), the 4 q8_0 FP16 scales assembled in a GPR and converted with one
  vcvtph2ps; one mul and one FMA per iteration.
An odd trailing block uses the existing scalar loop. The AVX (non-AVX2) path
and the generic reference are unchanged.

tests: test-backend-ops perf cases at Inkling-2-Mini expert shapes (n_embd 2304,
gate_up 2304, down 1152, 128 experts / 8 used; n = 1, 8, 16, 32) for nvfp4,
mxfp4 and q4_0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 21:10:39 +01:00
3 changed files with 151 additions and 37 deletions

No files matched your search

+88
View File
@@ -56,6 +56,7 @@ Status at the time of writing (2026-11-22):
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
- F14 User-message batch-break merge (one ubatch per short turn)
- F15 AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -141,6 +142,7 @@ Oldest first. "Feature" points at the section below.
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
| 78a204bb2 | ggml-cpu: faster AVX2 NVFP4 x Q8_0 dot product (two blocks per iteration, vector UE4M3 scales) (branch `fable5/nvfp4-avx2`, not deployed) | F15 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1533,6 +1535,92 @@ measured above.
restore after a restart, `prompt_n` and output identical to the reference built from the same
commit.
### F15. AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
Branch `fable5/nvfp4-avx2` (on the production branch tip `2caae7e11`). Not deployed. No knob: the
old kernel is gone from the AVX2 path (drop = revert the commit).
**What and why.** Inkling-2-Mini UD-Q4_K_XL stores the experts of 20 of its 24 layers as NVFP4
(ggml type 40: `block_nvfp4` = 4 UE4M3 scale bytes, one per 16 elements, then 32 bytes of E2M1
nibbles; within each 8-byte sub-block group the low nibbles are elements 0-7 and the high nibbles
8-15; values are the doubled E2M1 table `kvalues_fp4` and `ggml_ue4m3_to_fp32` returns the scale
times 0.5; the per-expert `ffn_*_exps.scale` tensor is a separate graph multiply). The host experts
(`-ot ...exps=CPU`) decode on the CPU through `ggml_vec_dot_nvfp4_q8_0`: the F13 profile had it at
21% of decode CPU time, plus 36% in the OpenMP barrier. CPU_REPACK has no NVFP4 path, so vec_dot is
what runs. Upstream's AVX2 loop did one block per iteration with 4 scalar UE4M3 table lookups, 2
scalar FP16 conversions, 4 scalar muls, broadcasts and 2 FMAs on one accumulator.
**Where.** `ggml/src/ggml-cpu/arch/x86/quants.c`: `ue4m3x8_to_fp32_half` and the `__AVX2__` branch
of `ggml_vec_dot_nvfp4_q8_0`. `tests/test-backend-ops.cpp`: perf cases at Inkling's expert shapes
(nvfp4, mxfp4, q4_0; `m=2304`, k 2304 and 1152, n 1/8/16/32, MUL_MAT and MUL_MAT_ID 128x8). Commit
`78a204bb2`.
**How it works.** Two blocks (128 weights) per iteration: one 32-byte load of `qs` per block, low and
high nibbles through a 256-bit `pshufb` of `kvalues_fp4`, `unpack{lo,hi}_epi64` + `permute2x128`
into the q8_0 element order, sign trick + `maddubs` against the two q8_0 blocks. The 8 sub-block sums
are reduced exactly in int16 (at most 16*12*127 = 24384) by two levels of `hadd_epi16` and one
`madd`, lanes `[b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]`. The 8 scale bytes are shuffled into the
same lane order and decoded in one vector: normal `(x & 0x7F) << 20` plus exponent bias 119 (bias 7
and the 0.5), subnormal `m * 2^-10` by blend, `0x7F` -> 0; bit-identical to `ggml_ue4m3_to_fp32` for
all 256 bytes (including the sign-bit quirk: 0xFF decodes to 240, as in the reference). The 4 q8_0
FP16 scales are packed in a GPR and converted with one `vcvtph2ps`. One mul + one FMA per
iteration. An odd last block takes the existing scalar tail. The AVX-only path and the generic
reference are untouched.
Tried and not kept (standalone harness `/root/arms/nvfp4-bench/nvfp4bench.c`, single core, min of
400 trials): 4 blocks per iteration with two accumulators (no change), `srlv`-based nibble split with
the q8_0 vector permuted instead (no change), `packs_epi32(madd)` instead of `hadd_epi16` (no
change), the scale decode via a denormal multiply (`as_float(x << 20) * 2^119`, -4% time but wrong
under DAZ, so not used). Ablation of the kept kernel: nibble decode + dot product about 60% of the
time, the reduction 17%, the scales 23%. Zen 2 runs it at about 2.7 IPC with FP pipe 2 the busiest
(`perf stat` `fpu_pipe_assignment`), not at the 4-pipe limit.
**Correctness.**
- Standalone: bit-exact scale decode for all 256 bytes and the lane order; random blocks (wild and
realistic scale bytes, n 64..2368, odd block counts) against the generic reference: worst error
3.7e-8 of the sum of |terms|.
- Linked against the built libraries (`vecdot_check.cpp`): 1400 rows of Gaussian weights at 4 scales
(1e-3 drives the UE4M3 scales into subnormals), quantized with the library, vs the exact dot of the
dequantized operands in double: new 5.5e-8, old 4.2e-8 (float rounding of the accumulation order).
- `test-quantize-fns`: nvfp4 dot product error 0.019774 in both builds (identical).
- `test-backend-ops -b CPU` cannot test a CPU kernel (it compares the CPU backend with itself).
**Measured (2026-11-22, pod, beside production, CPU-only scratch builds `-DGGML_VULKAN=OFF` in
`/root/arms/nvfp4-base` (`2caae7e11`) and `/root/arms/nvfp4`; logs in `/root/arms/nvfp4-bench/`).**
- Kernel, one pinned core, 64 rows L2-resident, min of 3000 trials, through the library's type
traits (`vecdot_perf.cpp`, `vp1.log`, 4 runs per arm, spread 0.2 ns): k=2304 129.0 -> 113.8 ns/row
(1.13x), k=1152 67.6 -> 59.5 (1.14x). Unchanged controls: q4_0 96.6 / 51.6, mxfp4 103.0 / 55.0 in
both builds. NVFP4 is still about 18% slower per row than q4_0.
- 6 threads streaming 512 MiB from DRAM (decode-like): every type, old or new, sits at the same
21-23 GB/s (best of 15; medians 11-21 GB/s under the co-tenants). At 6 threads the expert dot
product is DRAM-bound on this node, so the kernel speedup mostly turns into idle time, not decode
speed. A decode token streams about 1.3 GB of host NVFP4 experts at production's k=4 (8 experts x
36 layer passes x 35.8 MB).
- `test-backend-ops perf` at 6 threads is useless here: the unchanged q4_0 and mxfp4 cases varied
0.6x-1.7x between the two builds.
- llama-bench CPU-only decode, `-ngl 0 -t 6 -p 0`, arms B N N B, 6 rounds: rounds 1-2 tg32 x3 (`lb.log`): base 5.23 5.12 5.04 5.19, new 5.52 5.34 5.23 5.15 t/s,
medians 5.16 -> 5.29 (+2.5%); rounds 3-6 tg64 x5 (`lb2.log`): base 4.53 5.10 4.87 5.31 5.02 4.65 4.79
4.93, new 4.98 5.24 5.34 5.09 4.70 4.38 5.02 5.18, medians 4.90 -> 5.06 (+3.2%). The new build won 5
of the 6 rounds (mean of its two arms against the base pair). Below the 5% bar of section 6, so a
likely +2-3%, not proven. CPU-only is the upper bound of the effect: every layer's experts and
attention run on the CPU here; with production's k=4 the NVFP4 share of a token is smaller.
**What the GPU+CPU window should measure.** Production command line (`inkling_args`, k=4 `-ot`),
llama-bench `-lm none -ngl 99 -ot <k=4 pattern> -fa 1 -t 6 -b 1024 -ub 1024 -p 0 -n 256 -d 4096 -r 3`
(the F13 sweep's tg256 @ d4096 cell), base `2caae7e11` vs this branch, B N N B, medians (section 6).
Expected: decode +0-3% (DRAM-bound), prefill unchanged (prefill MUL_MAT_ID runs on the GPU for
ubatches >= 32). Optionally `perf record` the decode: `ggml_vec_dot_nvfp4_q8_0` should drop from 21%
by roughly a tenth and the barrier share rise. Then a greedy text request against the base build
(temperature 0): output may differ only by float rounding.
**How to drop it.** Revert `78a204bb2` (the kernel and the perf cases are independent hunks). Drop it
when upstream rewrites this AVX2 path.
**Rebase hazards.** Upstream changes to `block_nvfp4` (layout, scale semantics, the 0.5 convention in
`ggml_ue4m3_to_fp32`, the `0x7F`/sign handling TODO there) must be mirrored in
`ue4m3x8_to_fp32_half`; the standalone check (`nvfp4bench.c`) tests all 256 bytes against the
reference.
---
## 4. Dead ends
+53 -37
View File
@@ -1083,6 +1083,29 @@ void ggml_vec_dot_mxfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
*s = sumf;
}
#if defined(__AVX2__)
// The 8 UE4M3 scales of two NVFP4 blocks as fp32, equal to GGML_CPU_UE4M3_TO_FP32 (x0.5, 0x7F -> 0, sign bit
// ignored, subnormals), in the lane order of the reduction in ggml_vec_dot_nvfp4_q8_0:
// [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
static inline __m256 ue4m3x8_to_fp32_half(const uint8_t * d0, const uint8_t * d1) {
uint32_t a, b;
memcpy(&a, d0, sizeof(a));
memcpy(&b, d1, sizeof(b));
const __m256i v = _mm256_set1_epi64x((int64_t) (((uint64_t) b << 32) | a));
const __m256i x = _mm256_shuffle_epi8(v, _mm256_setr_epi8(
0, -1, -1, -1, 2, -1, -1, -1, 4, -1, -1, -1, 6, -1, -1, -1,
1, -1, -1, -1, 3, -1, -1, -1, 5, -1, -1, -1, 7, -1, -1, -1));
const __m256i nan = _mm256_cmpeq_epi32(x, _mm256_set1_epi32(0x7F));
const __m256i xm = _mm256_and_si256(x, _mm256_set1_epi32(0x7F));
// normal: fp32 exponent e + 119 (bias 7, times 0.5), mantissa bits in place
const __m256 fn = _mm256_castsi256_ps(_mm256_add_epi32(_mm256_slli_epi32(xm, 20), _mm256_set1_epi32(119 << 23)));
// subnormal (e == 0): m * 2^-10
const __m256 fs = _mm256_mul_ps(_mm256_cvtepi32_ps(xm), _mm256_set1_ps(1.0f/1024.0f));
const __m256 f = _mm256_blendv_ps(fn, fs, _mm256_castsi256_ps(_mm256_cmpgt_epi32(_mm256_set1_epi32(8), xm)));
return _mm256_andnot_ps(_mm256_castsi256_ps(nan), f);
}
#endif
void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
assert(nrc == 1);
UNUSED(nrc);
@@ -1100,47 +1123,40 @@ void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
#if defined(__AVX2__)
const __m128i values128 = _mm_loadu_si128((const __m128i*)kvalues_fp4);
const __m128i m4b = _mm_set1_epi8(0x0f);
const __m256i mone = _mm256_set1_epi16(1);
const __m256i values = _mm256_broadcastsi128_si256(_mm_loadu_si128((const __m128i *) kvalues_fp4));
const __m256i m4b = _mm256_set1_epi8(0x0f);
const __m256i mone = _mm256_set1_epi16(1);
// two blocks per iteration: 8 sub-block sums, 8 scales and one FMA
__m256 accum = _mm256_setzero_ps();
for(; ib < nb; ib++){
for (; ib + 1 < nb; ib += 2) {
__m256i p[4];
for (int k = 0; k < 2; ++k) {
// 8 bytes per sub-block: low nibbles are elements 0-7, high nibbles 8-15
const __m256i q = _mm256_loadu_si256((const __m256i *) x[ib + k].qs);
const __m256i lo = _mm256_shuffle_epi8(values, _mm256_and_si256(q, m4b));
const __m256i hi = _mm256_shuffle_epi8(values, _mm256_and_si256(_mm256_srli_epi16(q, 4), m4b));
const __m256i a = _mm256_unpacklo_epi64(lo, hi); // [sub0 | sub2]
const __m256i b = _mm256_unpackhi_epi64(lo, hi); // [sub1 | sub3]
p[2*k + 0] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x20), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 0].qs));
p[2*k + 1] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x31), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 1].qs));
}
// a sub-block sum is at most 16*12*127, so it fits in int16:
// lanes [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
const __m256i h = _mm256_hadd_epi16(_mm256_hadd_epi16(p[0], p[1]), _mm256_hadd_epi16(p[2], p[3]));
const __m256 sums = _mm256_cvtepi32_ps(_mm256_madd_epi16(h, mone));
const __m128i q4bits_01 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 0));
const __m128i q4bits_23 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 16));
#if defined(__F16C__)
// assembled in a GPR: cheaper than four inserts
const uint64_t dyb = (uint64_t) y[2*ib + 0].d | ((uint64_t) y[2*ib + 1].d << 16) |
((uint64_t) y[2*ib + 2].d << 32) | ((uint64_t) y[2*ib + 3].d << 48);
const __m128 dy = _mm_cvtph_ps(_mm_cvtsi64_si128((int64_t) dyb));
#else
const __m128 dy = _mm_setr_ps(GGML_CPU_FP16_TO_FP32(y[2*ib + 0].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 1].d),
GGML_CPU_FP16_TO_FP32(y[2*ib + 2].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 3].d));
#endif
const __m256i q8_01 = _mm256_loadu_si256((const __m256i *)y[2*ib + 0].qs);
const __m256i q8_23 = _mm256_loadu_si256((const __m256i *)y[2*ib + 1].qs);
const __m128i q4_01_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_01, m4b));
const __m128i q4_01_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_01, 4), m4b));
const __m128i q4_23_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_23, m4b));
const __m128i q4_23_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_23, 4), m4b));
//reordering
const __m256i q4_01 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_01_lo,q4_01_hi), _mm_unpacklo_epi64(q4_01_lo,q4_01_hi));
const __m256i q4_23 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_23_lo,q4_23_hi),_mm_unpacklo_epi64(q4_23_lo,q4_23_hi));
const __m256i p01 = mul_add_epi8(q4_01,q8_01);
const __m256i p_1 = _mm256_madd_epi16(p01, mone);
const __m256i p23 = mul_add_epi8(q4_23,q8_23);
const __m256i p_2 = _mm256_madd_epi16(p23, mone);
const float dy0 = GGML_CPU_FP16_TO_FP32(y[2*ib].d);
const float dy1 = GGML_CPU_FP16_TO_FP32(y[2*ib+1].d);
const float s0 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[0]) * dy0;
const float s1 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[1]) * dy0;
const float s2 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[2]) * dy1;
const float s3 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[3]) * dy1;
const __m256 scales01 = _mm256_set_m128(_mm_set1_ps(s1), _mm_set1_ps(s0));
const __m256 scales23 = _mm256_set_m128(_mm_set1_ps(s3), _mm_set1_ps(s2));
accum = _mm256_fmadd_ps(scales01, _mm256_cvtepi32_ps(p_1), accum);
accum = _mm256_fmadd_ps(scales23, _mm256_cvtepi32_ps(p_2), accum);
accum = _mm256_fmadd_ps(_mm256_mul_ps(ue4m3x8_to_fp32_half(x[ib].d, x[ib + 1].d), _mm256_set_m128(dy, dy)), sums, accum);
}
sumf = hsum_float_8(accum);
+10
View File
@@ -14178,6 +14178,16 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_perf() {
}
}
// Inkling-2-Mini experts (NVFP4 in UD-Q4_K_XL): 128 experts, 8 used, n_embd 2304, gate_up 2x1152 fused, down 1152
for (int bs : {1, 8, 16, 32}) {
for (ggml_type type_a : {GGML_TYPE_NVFP4, GGML_TYPE_MXFP4, GGML_TYPE_Q4_0}) {
test_cases.emplace_back(new test_mul_mat(type_a, GGML_TYPE_F32, 2304, bs, 2304, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(type_a, GGML_TYPE_F32, 2304, bs, 1152, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat_id(type_a, GGML_TYPE_F32, 128, 8, false, 2304, bs, 2304));
test_cases.emplace_back(new test_mul_mat_id(type_a, GGML_TYPE_F32, 128, 8, false, 2304, bs, 1152));
}
}
for (int K : {3, 5}) {
for (int IC : {256, 2560}) {
for (int IW_IH : {32, 64, 256}) {