Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9573fe0a2f | ||
|
|
78a204bb2f |
No files matched your search
@@ -56,6 +56,7 @@ Status at the time of writing (2026-11-22):
|
||||
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
||||
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
||||
- F14 User-message batch-break merge (one ubatch per short turn)
|
||||
- F15 AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
|
||||
4. Dead ends
|
||||
5. Configuration that matters
|
||||
6. Measurement methodology and traps
|
||||
@@ -141,6 +142,7 @@ Oldest first. "Feature" points at the section below.
|
||||
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
||||
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
||||
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
|
||||
| 78a204bb2 | ggml-cpu: faster AVX2 NVFP4 x Q8_0 dot product (two blocks per iteration, vector UE4M3 scales) (branch `fable5/nvfp4-avx2`, not deployed) | F15 |
|
||||
|
||||
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
||||
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
||||
@@ -1533,6 +1535,92 @@ measured above.
|
||||
restore after a restart, `prompt_n` and output identical to the reference built from the same
|
||||
commit.
|
||||
|
||||
### F15. AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
|
||||
|
||||
Branch `fable5/nvfp4-avx2` (on the production branch tip `2caae7e11`). Not deployed. No knob: the
|
||||
old kernel is gone from the AVX2 path (drop = revert the commit).
|
||||
|
||||
**What and why.** Inkling-2-Mini UD-Q4_K_XL stores the experts of 20 of its 24 layers as NVFP4
|
||||
(ggml type 40: `block_nvfp4` = 4 UE4M3 scale bytes, one per 16 elements, then 32 bytes of E2M1
|
||||
nibbles; within each 8-byte sub-block group the low nibbles are elements 0-7 and the high nibbles
|
||||
8-15; values are the doubled E2M1 table `kvalues_fp4` and `ggml_ue4m3_to_fp32` returns the scale
|
||||
times 0.5; the per-expert `ffn_*_exps.scale` tensor is a separate graph multiply). The host experts
|
||||
(`-ot ...exps=CPU`) decode on the CPU through `ggml_vec_dot_nvfp4_q8_0`: the F13 profile had it at
|
||||
21% of decode CPU time, plus 36% in the OpenMP barrier. CPU_REPACK has no NVFP4 path, so vec_dot is
|
||||
what runs. Upstream's AVX2 loop did one block per iteration with 4 scalar UE4M3 table lookups, 2
|
||||
scalar FP16 conversions, 4 scalar muls, broadcasts and 2 FMAs on one accumulator.
|
||||
|
||||
**Where.** `ggml/src/ggml-cpu/arch/x86/quants.c`: `ue4m3x8_to_fp32_half` and the `__AVX2__` branch
|
||||
of `ggml_vec_dot_nvfp4_q8_0`. `tests/test-backend-ops.cpp`: perf cases at Inkling's expert shapes
|
||||
(nvfp4, mxfp4, q4_0; `m=2304`, k 2304 and 1152, n 1/8/16/32, MUL_MAT and MUL_MAT_ID 128x8). Commit
|
||||
`78a204bb2`.
|
||||
|
||||
**How it works.** Two blocks (128 weights) per iteration: one 32-byte load of `qs` per block, low and
|
||||
high nibbles through a 256-bit `pshufb` of `kvalues_fp4`, `unpack{lo,hi}_epi64` + `permute2x128`
|
||||
into the q8_0 element order, sign trick + `maddubs` against the two q8_0 blocks. The 8 sub-block sums
|
||||
are reduced exactly in int16 (at most 16*12*127 = 24384) by two levels of `hadd_epi16` and one
|
||||
`madd`, lanes `[b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]`. The 8 scale bytes are shuffled into the
|
||||
same lane order and decoded in one vector: normal `(x & 0x7F) << 20` plus exponent bias 119 (bias 7
|
||||
and the 0.5), subnormal `m * 2^-10` by blend, `0x7F` -> 0; bit-identical to `ggml_ue4m3_to_fp32` for
|
||||
all 256 bytes (including the sign-bit quirk: 0xFF decodes to 240, as in the reference). The 4 q8_0
|
||||
FP16 scales are packed in a GPR and converted with one `vcvtph2ps`. One mul + one FMA per
|
||||
iteration. An odd last block takes the existing scalar tail. The AVX-only path and the generic
|
||||
reference are untouched.
|
||||
|
||||
Tried and not kept (standalone harness `/root/arms/nvfp4-bench/nvfp4bench.c`, single core, min of
|
||||
400 trials): 4 blocks per iteration with two accumulators (no change), `srlv`-based nibble split with
|
||||
the q8_0 vector permuted instead (no change), `packs_epi32(madd)` instead of `hadd_epi16` (no
|
||||
change), the scale decode via a denormal multiply (`as_float(x << 20) * 2^119`, -4% time but wrong
|
||||
under DAZ, so not used). Ablation of the kept kernel: nibble decode + dot product about 60% of the
|
||||
time, the reduction 17%, the scales 23%. Zen 2 runs it at about 2.7 IPC with FP pipe 2 the busiest
|
||||
(`perf stat` `fpu_pipe_assignment`), not at the 4-pipe limit.
|
||||
|
||||
**Correctness.**
|
||||
- Standalone: bit-exact scale decode for all 256 bytes and the lane order; random blocks (wild and
|
||||
realistic scale bytes, n 64..2368, odd block counts) against the generic reference: worst error
|
||||
3.7e-8 of the sum of |terms|.
|
||||
- Linked against the built libraries (`vecdot_check.cpp`): 1400 rows of Gaussian weights at 4 scales
|
||||
(1e-3 drives the UE4M3 scales into subnormals), quantized with the library, vs the exact dot of the
|
||||
dequantized operands in double: new 5.5e-8, old 4.2e-8 (float rounding of the accumulation order).
|
||||
- `test-quantize-fns`: nvfp4 dot product error 0.019774 in both builds (identical).
|
||||
- `test-backend-ops -b CPU` cannot test a CPU kernel (it compares the CPU backend with itself).
|
||||
|
||||
**Measured (2026-11-22, pod, beside production, CPU-only scratch builds `-DGGML_VULKAN=OFF` in
|
||||
`/root/arms/nvfp4-base` (`2caae7e11`) and `/root/arms/nvfp4`; logs in `/root/arms/nvfp4-bench/`).**
|
||||
- Kernel, one pinned core, 64 rows L2-resident, min of 3000 trials, through the library's type
|
||||
traits (`vecdot_perf.cpp`, `vp1.log`, 4 runs per arm, spread 0.2 ns): k=2304 129.0 -> 113.8 ns/row
|
||||
(1.13x), k=1152 67.6 -> 59.5 (1.14x). Unchanged controls: q4_0 96.6 / 51.6, mxfp4 103.0 / 55.0 in
|
||||
both builds. NVFP4 is still about 18% slower per row than q4_0.
|
||||
- 6 threads streaming 512 MiB from DRAM (decode-like): every type, old or new, sits at the same
|
||||
21-23 GB/s (best of 15; medians 11-21 GB/s under the co-tenants). At 6 threads the expert dot
|
||||
product is DRAM-bound on this node, so the kernel speedup mostly turns into idle time, not decode
|
||||
speed. A decode token streams about 1.3 GB of host NVFP4 experts at production's k=4 (8 experts x
|
||||
36 layer passes x 35.8 MB).
|
||||
- `test-backend-ops perf` at 6 threads is useless here: the unchanged q4_0 and mxfp4 cases varied
|
||||
0.6x-1.7x between the two builds.
|
||||
- llama-bench CPU-only decode, `-ngl 0 -t 6 -p 0`, arms B N N B, 6 rounds: rounds 1-2 tg32 x3 (`lb.log`): base 5.23 5.12 5.04 5.19, new 5.52 5.34 5.23 5.15 t/s,
|
||||
medians 5.16 -> 5.29 (+2.5%); rounds 3-6 tg64 x5 (`lb2.log`): base 4.53 5.10 4.87 5.31 5.02 4.65 4.79
|
||||
4.93, new 4.98 5.24 5.34 5.09 4.70 4.38 5.02 5.18, medians 4.90 -> 5.06 (+3.2%). The new build won 5
|
||||
of the 6 rounds (mean of its two arms against the base pair). Below the 5% bar of section 6, so a
|
||||
likely +2-3%, not proven. CPU-only is the upper bound of the effect: every layer's experts and
|
||||
attention run on the CPU here; with production's k=4 the NVFP4 share of a token is smaller.
|
||||
|
||||
**What the GPU+CPU window should measure.** Production command line (`inkling_args`, k=4 `-ot`),
|
||||
llama-bench `-lm none -ngl 99 -ot <k=4 pattern> -fa 1 -t 6 -b 1024 -ub 1024 -p 0 -n 256 -d 4096 -r 3`
|
||||
(the F13 sweep's tg256 @ d4096 cell), base `2caae7e11` vs this branch, B N N B, medians (section 6).
|
||||
Expected: decode +0-3% (DRAM-bound), prefill unchanged (prefill MUL_MAT_ID runs on the GPU for
|
||||
ubatches >= 32). Optionally `perf record` the decode: `ggml_vec_dot_nvfp4_q8_0` should drop from 21%
|
||||
by roughly a tenth and the barrier share rise. Then a greedy text request against the base build
|
||||
(temperature 0): output may differ only by float rounding.
|
||||
|
||||
**How to drop it.** Revert `78a204bb2` (the kernel and the perf cases are independent hunks). Drop it
|
||||
when upstream rewrites this AVX2 path.
|
||||
|
||||
**Rebase hazards.** Upstream changes to `block_nvfp4` (layout, scale semantics, the 0.5 convention in
|
||||
`ggml_ue4m3_to_fp32`, the `0x7F`/sign handling TODO there) must be mirrored in
|
||||
`ue4m3x8_to_fp32_half`; the standalone check (`nvfp4bench.c`) tests all 256 bytes against the
|
||||
reference.
|
||||
|
||||
---
|
||||
|
||||
## 4. Dead ends
|
||||
|
||||
@@ -1083,6 +1083,29 @@ void ggml_vec_dot_mxfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
|
||||
*s = sumf;
|
||||
}
|
||||
|
||||
#if defined(__AVX2__)
|
||||
// The 8 UE4M3 scales of two NVFP4 blocks as fp32, equal to GGML_CPU_UE4M3_TO_FP32 (x0.5, 0x7F -> 0, sign bit
|
||||
// ignored, subnormals), in the lane order of the reduction in ggml_vec_dot_nvfp4_q8_0:
|
||||
// [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
|
||||
static inline __m256 ue4m3x8_to_fp32_half(const uint8_t * d0, const uint8_t * d1) {
|
||||
uint32_t a, b;
|
||||
memcpy(&a, d0, sizeof(a));
|
||||
memcpy(&b, d1, sizeof(b));
|
||||
const __m256i v = _mm256_set1_epi64x((int64_t) (((uint64_t) b << 32) | a));
|
||||
const __m256i x = _mm256_shuffle_epi8(v, _mm256_setr_epi8(
|
||||
0, -1, -1, -1, 2, -1, -1, -1, 4, -1, -1, -1, 6, -1, -1, -1,
|
||||
1, -1, -1, -1, 3, -1, -1, -1, 5, -1, -1, -1, 7, -1, -1, -1));
|
||||
const __m256i nan = _mm256_cmpeq_epi32(x, _mm256_set1_epi32(0x7F));
|
||||
const __m256i xm = _mm256_and_si256(x, _mm256_set1_epi32(0x7F));
|
||||
// normal: fp32 exponent e + 119 (bias 7, times 0.5), mantissa bits in place
|
||||
const __m256 fn = _mm256_castsi256_ps(_mm256_add_epi32(_mm256_slli_epi32(xm, 20), _mm256_set1_epi32(119 << 23)));
|
||||
// subnormal (e == 0): m * 2^-10
|
||||
const __m256 fs = _mm256_mul_ps(_mm256_cvtepi32_ps(xm), _mm256_set1_ps(1.0f/1024.0f));
|
||||
const __m256 f = _mm256_blendv_ps(fn, fs, _mm256_castsi256_ps(_mm256_cmpgt_epi32(_mm256_set1_epi32(8), xm)));
|
||||
return _mm256_andnot_ps(_mm256_castsi256_ps(nan), f);
|
||||
}
|
||||
#endif
|
||||
|
||||
void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
|
||||
assert(nrc == 1);
|
||||
UNUSED(nrc);
|
||||
@@ -1100,47 +1123,40 @@ void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
|
||||
|
||||
#if defined(__AVX2__)
|
||||
|
||||
const __m128i values128 = _mm_loadu_si128((const __m128i*)kvalues_fp4);
|
||||
const __m128i m4b = _mm_set1_epi8(0x0f);
|
||||
const __m256i mone = _mm256_set1_epi16(1);
|
||||
const __m256i values = _mm256_broadcastsi128_si256(_mm_loadu_si128((const __m128i *) kvalues_fp4));
|
||||
const __m256i m4b = _mm256_set1_epi8(0x0f);
|
||||
const __m256i mone = _mm256_set1_epi16(1);
|
||||
|
||||
// two blocks per iteration: 8 sub-block sums, 8 scales and one FMA
|
||||
__m256 accum = _mm256_setzero_ps();
|
||||
for(; ib < nb; ib++){
|
||||
for (; ib + 1 < nb; ib += 2) {
|
||||
__m256i p[4];
|
||||
for (int k = 0; k < 2; ++k) {
|
||||
// 8 bytes per sub-block: low nibbles are elements 0-7, high nibbles 8-15
|
||||
const __m256i q = _mm256_loadu_si256((const __m256i *) x[ib + k].qs);
|
||||
const __m256i lo = _mm256_shuffle_epi8(values, _mm256_and_si256(q, m4b));
|
||||
const __m256i hi = _mm256_shuffle_epi8(values, _mm256_and_si256(_mm256_srli_epi16(q, 4), m4b));
|
||||
const __m256i a = _mm256_unpacklo_epi64(lo, hi); // [sub0 | sub2]
|
||||
const __m256i b = _mm256_unpackhi_epi64(lo, hi); // [sub1 | sub3]
|
||||
p[2*k + 0] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x20), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 0].qs));
|
||||
p[2*k + 1] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x31), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 1].qs));
|
||||
}
|
||||
// a sub-block sum is at most 16*12*127, so it fits in int16:
|
||||
// lanes [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
|
||||
const __m256i h = _mm256_hadd_epi16(_mm256_hadd_epi16(p[0], p[1]), _mm256_hadd_epi16(p[2], p[3]));
|
||||
const __m256 sums = _mm256_cvtepi32_ps(_mm256_madd_epi16(h, mone));
|
||||
|
||||
const __m128i q4bits_01 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 0));
|
||||
const __m128i q4bits_23 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 16));
|
||||
#if defined(__F16C__)
|
||||
// assembled in a GPR: cheaper than four inserts
|
||||
const uint64_t dyb = (uint64_t) y[2*ib + 0].d | ((uint64_t) y[2*ib + 1].d << 16) |
|
||||
((uint64_t) y[2*ib + 2].d << 32) | ((uint64_t) y[2*ib + 3].d << 48);
|
||||
const __m128 dy = _mm_cvtph_ps(_mm_cvtsi64_si128((int64_t) dyb));
|
||||
#else
|
||||
const __m128 dy = _mm_setr_ps(GGML_CPU_FP16_TO_FP32(y[2*ib + 0].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 1].d),
|
||||
GGML_CPU_FP16_TO_FP32(y[2*ib + 2].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 3].d));
|
||||
#endif
|
||||
|
||||
const __m256i q8_01 = _mm256_loadu_si256((const __m256i *)y[2*ib + 0].qs);
|
||||
const __m256i q8_23 = _mm256_loadu_si256((const __m256i *)y[2*ib + 1].qs);
|
||||
|
||||
const __m128i q4_01_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_01, m4b));
|
||||
const __m128i q4_01_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_01, 4), m4b));
|
||||
const __m128i q4_23_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_23, m4b));
|
||||
const __m128i q4_23_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_23, 4), m4b));
|
||||
|
||||
//reordering
|
||||
const __m256i q4_01 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_01_lo,q4_01_hi), _mm_unpacklo_epi64(q4_01_lo,q4_01_hi));
|
||||
const __m256i q4_23 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_23_lo,q4_23_hi),_mm_unpacklo_epi64(q4_23_lo,q4_23_hi));
|
||||
|
||||
const __m256i p01 = mul_add_epi8(q4_01,q8_01);
|
||||
const __m256i p_1 = _mm256_madd_epi16(p01, mone);
|
||||
|
||||
const __m256i p23 = mul_add_epi8(q4_23,q8_23);
|
||||
const __m256i p_2 = _mm256_madd_epi16(p23, mone);
|
||||
|
||||
const float dy0 = GGML_CPU_FP16_TO_FP32(y[2*ib].d);
|
||||
const float dy1 = GGML_CPU_FP16_TO_FP32(y[2*ib+1].d);
|
||||
|
||||
const float s0 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[0]) * dy0;
|
||||
const float s1 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[1]) * dy0;
|
||||
const float s2 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[2]) * dy1;
|
||||
const float s3 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[3]) * dy1;
|
||||
|
||||
const __m256 scales01 = _mm256_set_m128(_mm_set1_ps(s1), _mm_set1_ps(s0));
|
||||
const __m256 scales23 = _mm256_set_m128(_mm_set1_ps(s3), _mm_set1_ps(s2));
|
||||
|
||||
accum = _mm256_fmadd_ps(scales01, _mm256_cvtepi32_ps(p_1), accum);
|
||||
accum = _mm256_fmadd_ps(scales23, _mm256_cvtepi32_ps(p_2), accum);
|
||||
accum = _mm256_fmadd_ps(_mm256_mul_ps(ue4m3x8_to_fp32_half(x[ib].d, x[ib + 1].d), _mm256_set_m128(dy, dy)), sums, accum);
|
||||
}
|
||||
sumf = hsum_float_8(accum);
|
||||
|
||||
|
||||
@@ -14178,6 +14178,16 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_perf() {
|
||||
}
|
||||
}
|
||||
|
||||
// Inkling-2-Mini experts (NVFP4 in UD-Q4_K_XL): 128 experts, 8 used, n_embd 2304, gate_up 2x1152 fused, down 1152
|
||||
for (int bs : {1, 8, 16, 32}) {
|
||||
for (ggml_type type_a : {GGML_TYPE_NVFP4, GGML_TYPE_MXFP4, GGML_TYPE_Q4_0}) {
|
||||
test_cases.emplace_back(new test_mul_mat(type_a, GGML_TYPE_F32, 2304, bs, 2304, {1, 1}, {1, 1}));
|
||||
test_cases.emplace_back(new test_mul_mat(type_a, GGML_TYPE_F32, 2304, bs, 1152, {1, 1}, {1, 1}));
|
||||
test_cases.emplace_back(new test_mul_mat_id(type_a, GGML_TYPE_F32, 128, 8, false, 2304, bs, 2304));
|
||||
test_cases.emplace_back(new test_mul_mat_id(type_a, GGML_TYPE_F32, 128, 8, false, 2304, bs, 1152));
|
||||
}
|
||||
}
|
||||
|
||||
for (int K : {3, 5}) {
|
||||
for (int IC : {256, 2560}) {
|
||||
for (int IW_IH : {32, 64, 256}) {
|
||||
|
||||
Reference in new issue
Block a user