Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9573fe0a2f | ||
|
|
78a204bb2f |
No files matched your search
@@ -56,6 +56,7 @@ Status at the time of writing (2026-11-22):
|
|||||||
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
|
||||||
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
|
||||||
- F14 User-message batch-break merge (one ubatch per short turn)
|
- F14 User-message batch-break merge (one ubatch per short turn)
|
||||||
|
- F15 AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
|
||||||
4. Dead ends
|
4. Dead ends
|
||||||
5. Configuration that matters
|
5. Configuration that matters
|
||||||
6. Measurement methodology and traps
|
6. Measurement methodology and traps
|
||||||
@@ -141,6 +142,7 @@ Oldest first. "Feature" points at the section below.
|
|||||||
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
||||||
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
||||||
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
|
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
|
||||||
|
| 78a204bb2 | ggml-cpu: faster AVX2 NVFP4 x Q8_0 dot product (two blocks per iteration, vector UE4M3 scales) (branch `fable5/nvfp4-avx2`, not deployed) | F15 |
|
||||||
|
|
||||||
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
||||||
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
||||||
@@ -1533,6 +1535,92 @@ measured above.
|
|||||||
restore after a restart, `prompt_n` and output identical to the reference built from the same
|
restore after a restart, `prompt_n` and output identical to the reference built from the same
|
||||||
commit.
|
commit.
|
||||||
|
|
||||||
|
### F15. AVX2 NVFP4 x Q8_0 dot product (Inkling-2-Mini CPU experts)
|
||||||
|
|
||||||
|
Branch `fable5/nvfp4-avx2` (on the production branch tip `2caae7e11`). Not deployed. No knob: the
|
||||||
|
old kernel is gone from the AVX2 path (drop = revert the commit).
|
||||||
|
|
||||||
|
**What and why.** Inkling-2-Mini UD-Q4_K_XL stores the experts of 20 of its 24 layers as NVFP4
|
||||||
|
(ggml type 40: `block_nvfp4` = 4 UE4M3 scale bytes, one per 16 elements, then 32 bytes of E2M1
|
||||||
|
nibbles; within each 8-byte sub-block group the low nibbles are elements 0-7 and the high nibbles
|
||||||
|
8-15; values are the doubled E2M1 table `kvalues_fp4` and `ggml_ue4m3_to_fp32` returns the scale
|
||||||
|
times 0.5; the per-expert `ffn_*_exps.scale` tensor is a separate graph multiply). The host experts
|
||||||
|
(`-ot ...exps=CPU`) decode on the CPU through `ggml_vec_dot_nvfp4_q8_0`: the F13 profile had it at
|
||||||
|
21% of decode CPU time, plus 36% in the OpenMP barrier. CPU_REPACK has no NVFP4 path, so vec_dot is
|
||||||
|
what runs. Upstream's AVX2 loop did one block per iteration with 4 scalar UE4M3 table lookups, 2
|
||||||
|
scalar FP16 conversions, 4 scalar muls, broadcasts and 2 FMAs on one accumulator.
|
||||||
|
|
||||||
|
**Where.** `ggml/src/ggml-cpu/arch/x86/quants.c`: `ue4m3x8_to_fp32_half` and the `__AVX2__` branch
|
||||||
|
of `ggml_vec_dot_nvfp4_q8_0`. `tests/test-backend-ops.cpp`: perf cases at Inkling's expert shapes
|
||||||
|
(nvfp4, mxfp4, q4_0; `m=2304`, k 2304 and 1152, n 1/8/16/32, MUL_MAT and MUL_MAT_ID 128x8). Commit
|
||||||
|
`78a204bb2`.
|
||||||
|
|
||||||
|
**How it works.** Two blocks (128 weights) per iteration: one 32-byte load of `qs` per block, low and
|
||||||
|
high nibbles through a 256-bit `pshufb` of `kvalues_fp4`, `unpack{lo,hi}_epi64` + `permute2x128`
|
||||||
|
into the q8_0 element order, sign trick + `maddubs` against the two q8_0 blocks. The 8 sub-block sums
|
||||||
|
are reduced exactly in int16 (at most 16*12*127 = 24384) by two levels of `hadd_epi16` and one
|
||||||
|
`madd`, lanes `[b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]`. The 8 scale bytes are shuffled into the
|
||||||
|
same lane order and decoded in one vector: normal `(x & 0x7F) << 20` plus exponent bias 119 (bias 7
|
||||||
|
and the 0.5), subnormal `m * 2^-10` by blend, `0x7F` -> 0; bit-identical to `ggml_ue4m3_to_fp32` for
|
||||||
|
all 256 bytes (including the sign-bit quirk: 0xFF decodes to 240, as in the reference). The 4 q8_0
|
||||||
|
FP16 scales are packed in a GPR and converted with one `vcvtph2ps`. One mul + one FMA per
|
||||||
|
iteration. An odd last block takes the existing scalar tail. The AVX-only path and the generic
|
||||||
|
reference are untouched.
|
||||||
|
|
||||||
|
Tried and not kept (standalone harness `/root/arms/nvfp4-bench/nvfp4bench.c`, single core, min of
|
||||||
|
400 trials): 4 blocks per iteration with two accumulators (no change), `srlv`-based nibble split with
|
||||||
|
the q8_0 vector permuted instead (no change), `packs_epi32(madd)` instead of `hadd_epi16` (no
|
||||||
|
change), the scale decode via a denormal multiply (`as_float(x << 20) * 2^119`, -4% time but wrong
|
||||||
|
under DAZ, so not used). Ablation of the kept kernel: nibble decode + dot product about 60% of the
|
||||||
|
time, the reduction 17%, the scales 23%. Zen 2 runs it at about 2.7 IPC with FP pipe 2 the busiest
|
||||||
|
(`perf stat` `fpu_pipe_assignment`), not at the 4-pipe limit.
|
||||||
|
|
||||||
|
**Correctness.**
|
||||||
|
- Standalone: bit-exact scale decode for all 256 bytes and the lane order; random blocks (wild and
|
||||||
|
realistic scale bytes, n 64..2368, odd block counts) against the generic reference: worst error
|
||||||
|
3.7e-8 of the sum of |terms|.
|
||||||
|
- Linked against the built libraries (`vecdot_check.cpp`): 1400 rows of Gaussian weights at 4 scales
|
||||||
|
(1e-3 drives the UE4M3 scales into subnormals), quantized with the library, vs the exact dot of the
|
||||||
|
dequantized operands in double: new 5.5e-8, old 4.2e-8 (float rounding of the accumulation order).
|
||||||
|
- `test-quantize-fns`: nvfp4 dot product error 0.019774 in both builds (identical).
|
||||||
|
- `test-backend-ops -b CPU` cannot test a CPU kernel (it compares the CPU backend with itself).
|
||||||
|
|
||||||
|
**Measured (2026-11-22, pod, beside production, CPU-only scratch builds `-DGGML_VULKAN=OFF` in
|
||||||
|
`/root/arms/nvfp4-base` (`2caae7e11`) and `/root/arms/nvfp4`; logs in `/root/arms/nvfp4-bench/`).**
|
||||||
|
- Kernel, one pinned core, 64 rows L2-resident, min of 3000 trials, through the library's type
|
||||||
|
traits (`vecdot_perf.cpp`, `vp1.log`, 4 runs per arm, spread 0.2 ns): k=2304 129.0 -> 113.8 ns/row
|
||||||
|
(1.13x), k=1152 67.6 -> 59.5 (1.14x). Unchanged controls: q4_0 96.6 / 51.6, mxfp4 103.0 / 55.0 in
|
||||||
|
both builds. NVFP4 is still about 18% slower per row than q4_0.
|
||||||
|
- 6 threads streaming 512 MiB from DRAM (decode-like): every type, old or new, sits at the same
|
||||||
|
21-23 GB/s (best of 15; medians 11-21 GB/s under the co-tenants). At 6 threads the expert dot
|
||||||
|
product is DRAM-bound on this node, so the kernel speedup mostly turns into idle time, not decode
|
||||||
|
speed. A decode token streams about 1.3 GB of host NVFP4 experts at production's k=4 (8 experts x
|
||||||
|
36 layer passes x 35.8 MB).
|
||||||
|
- `test-backend-ops perf` at 6 threads is useless here: the unchanged q4_0 and mxfp4 cases varied
|
||||||
|
0.6x-1.7x between the two builds.
|
||||||
|
- llama-bench CPU-only decode, `-ngl 0 -t 6 -p 0`, arms B N N B, 6 rounds: rounds 1-2 tg32 x3 (`lb.log`): base 5.23 5.12 5.04 5.19, new 5.52 5.34 5.23 5.15 t/s,
|
||||||
|
medians 5.16 -> 5.29 (+2.5%); rounds 3-6 tg64 x5 (`lb2.log`): base 4.53 5.10 4.87 5.31 5.02 4.65 4.79
|
||||||
|
4.93, new 4.98 5.24 5.34 5.09 4.70 4.38 5.02 5.18, medians 4.90 -> 5.06 (+3.2%). The new build won 5
|
||||||
|
of the 6 rounds (mean of its two arms against the base pair). Below the 5% bar of section 6, so a
|
||||||
|
likely +2-3%, not proven. CPU-only is the upper bound of the effect: every layer's experts and
|
||||||
|
attention run on the CPU here; with production's k=4 the NVFP4 share of a token is smaller.
|
||||||
|
|
||||||
|
**What the GPU+CPU window should measure.** Production command line (`inkling_args`, k=4 `-ot`),
|
||||||
|
llama-bench `-lm none -ngl 99 -ot <k=4 pattern> -fa 1 -t 6 -b 1024 -ub 1024 -p 0 -n 256 -d 4096 -r 3`
|
||||||
|
(the F13 sweep's tg256 @ d4096 cell), base `2caae7e11` vs this branch, B N N B, medians (section 6).
|
||||||
|
Expected: decode +0-3% (DRAM-bound), prefill unchanged (prefill MUL_MAT_ID runs on the GPU for
|
||||||
|
ubatches >= 32). Optionally `perf record` the decode: `ggml_vec_dot_nvfp4_q8_0` should drop from 21%
|
||||||
|
by roughly a tenth and the barrier share rise. Then a greedy text request against the base build
|
||||||
|
(temperature 0): output may differ only by float rounding.
|
||||||
|
|
||||||
|
**How to drop it.** Revert `78a204bb2` (the kernel and the perf cases are independent hunks). Drop it
|
||||||
|
when upstream rewrites this AVX2 path.
|
||||||
|
|
||||||
|
**Rebase hazards.** Upstream changes to `block_nvfp4` (layout, scale semantics, the 0.5 convention in
|
||||||
|
`ggml_ue4m3_to_fp32`, the `0x7F`/sign handling TODO there) must be mirrored in
|
||||||
|
`ue4m3x8_to_fp32_half`; the standalone check (`nvfp4bench.c`) tests all 256 bytes against the
|
||||||
|
reference.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Dead ends
|
## 4. Dead ends
|
||||||
|
|||||||
@@ -1083,6 +1083,29 @@ void ggml_vec_dot_mxfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
|
|||||||
*s = sumf;
|
*s = sumf;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#if defined(__AVX2__)
|
||||||
|
// The 8 UE4M3 scales of two NVFP4 blocks as fp32, equal to GGML_CPU_UE4M3_TO_FP32 (x0.5, 0x7F -> 0, sign bit
|
||||||
|
// ignored, subnormals), in the lane order of the reduction in ggml_vec_dot_nvfp4_q8_0:
|
||||||
|
// [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
|
||||||
|
static inline __m256 ue4m3x8_to_fp32_half(const uint8_t * d0, const uint8_t * d1) {
|
||||||
|
uint32_t a, b;
|
||||||
|
memcpy(&a, d0, sizeof(a));
|
||||||
|
memcpy(&b, d1, sizeof(b));
|
||||||
|
const __m256i v = _mm256_set1_epi64x((int64_t) (((uint64_t) b << 32) | a));
|
||||||
|
const __m256i x = _mm256_shuffle_epi8(v, _mm256_setr_epi8(
|
||||||
|
0, -1, -1, -1, 2, -1, -1, -1, 4, -1, -1, -1, 6, -1, -1, -1,
|
||||||
|
1, -1, -1, -1, 3, -1, -1, -1, 5, -1, -1, -1, 7, -1, -1, -1));
|
||||||
|
const __m256i nan = _mm256_cmpeq_epi32(x, _mm256_set1_epi32(0x7F));
|
||||||
|
const __m256i xm = _mm256_and_si256(x, _mm256_set1_epi32(0x7F));
|
||||||
|
// normal: fp32 exponent e + 119 (bias 7, times 0.5), mantissa bits in place
|
||||||
|
const __m256 fn = _mm256_castsi256_ps(_mm256_add_epi32(_mm256_slli_epi32(xm, 20), _mm256_set1_epi32(119 << 23)));
|
||||||
|
// subnormal (e == 0): m * 2^-10
|
||||||
|
const __m256 fs = _mm256_mul_ps(_mm256_cvtepi32_ps(xm), _mm256_set1_ps(1.0f/1024.0f));
|
||||||
|
const __m256 f = _mm256_blendv_ps(fn, fs, _mm256_castsi256_ps(_mm256_cmpgt_epi32(_mm256_set1_epi32(8), xm)));
|
||||||
|
return _mm256_andnot_ps(_mm256_castsi256_ps(nan), f);
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
|
void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
|
||||||
assert(nrc == 1);
|
assert(nrc == 1);
|
||||||
UNUSED(nrc);
|
UNUSED(nrc);
|
||||||
@@ -1100,47 +1123,40 @@ void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
|
|||||||
|
|
||||||
#if defined(__AVX2__)
|
#if defined(__AVX2__)
|
||||||
|
|
||||||
const __m128i values128 = _mm_loadu_si128((const __m128i*)kvalues_fp4);
|
const __m256i values = _mm256_broadcastsi128_si256(_mm_loadu_si128((const __m128i *) kvalues_fp4));
|
||||||
const __m128i m4b = _mm_set1_epi8(0x0f);
|
const __m256i m4b = _mm256_set1_epi8(0x0f);
|
||||||
const __m256i mone = _mm256_set1_epi16(1);
|
const __m256i mone = _mm256_set1_epi16(1);
|
||||||
|
|
||||||
|
// two blocks per iteration: 8 sub-block sums, 8 scales and one FMA
|
||||||
__m256 accum = _mm256_setzero_ps();
|
__m256 accum = _mm256_setzero_ps();
|
||||||
for(; ib < nb; ib++){
|
for (; ib + 1 < nb; ib += 2) {
|
||||||
|
__m256i p[4];
|
||||||
|
for (int k = 0; k < 2; ++k) {
|
||||||
|
// 8 bytes per sub-block: low nibbles are elements 0-7, high nibbles 8-15
|
||||||
|
const __m256i q = _mm256_loadu_si256((const __m256i *) x[ib + k].qs);
|
||||||
|
const __m256i lo = _mm256_shuffle_epi8(values, _mm256_and_si256(q, m4b));
|
||||||
|
const __m256i hi = _mm256_shuffle_epi8(values, _mm256_and_si256(_mm256_srli_epi16(q, 4), m4b));
|
||||||
|
const __m256i a = _mm256_unpacklo_epi64(lo, hi); // [sub0 | sub2]
|
||||||
|
const __m256i b = _mm256_unpackhi_epi64(lo, hi); // [sub1 | sub3]
|
||||||
|
p[2*k + 0] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x20), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 0].qs));
|
||||||
|
p[2*k + 1] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x31), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 1].qs));
|
||||||
|
}
|
||||||
|
// a sub-block sum is at most 16*12*127, so it fits in int16:
|
||||||
|
// lanes [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
|
||||||
|
const __m256i h = _mm256_hadd_epi16(_mm256_hadd_epi16(p[0], p[1]), _mm256_hadd_epi16(p[2], p[3]));
|
||||||
|
const __m256 sums = _mm256_cvtepi32_ps(_mm256_madd_epi16(h, mone));
|
||||||
|
|
||||||
const __m128i q4bits_01 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 0));
|
#if defined(__F16C__)
|
||||||
const __m128i q4bits_23 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 16));
|
// assembled in a GPR: cheaper than four inserts
|
||||||
|
const uint64_t dyb = (uint64_t) y[2*ib + 0].d | ((uint64_t) y[2*ib + 1].d << 16) |
|
||||||
|
((uint64_t) y[2*ib + 2].d << 32) | ((uint64_t) y[2*ib + 3].d << 48);
|
||||||
|
const __m128 dy = _mm_cvtph_ps(_mm_cvtsi64_si128((int64_t) dyb));
|
||||||
|
#else
|
||||||
|
const __m128 dy = _mm_setr_ps(GGML_CPU_FP16_TO_FP32(y[2*ib + 0].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 1].d),
|
||||||
|
GGML_CPU_FP16_TO_FP32(y[2*ib + 2].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 3].d));
|
||||||
|
#endif
|
||||||
|
|
||||||
const __m256i q8_01 = _mm256_loadu_si256((const __m256i *)y[2*ib + 0].qs);
|
accum = _mm256_fmadd_ps(_mm256_mul_ps(ue4m3x8_to_fp32_half(x[ib].d, x[ib + 1].d), _mm256_set_m128(dy, dy)), sums, accum);
|
||||||
const __m256i q8_23 = _mm256_loadu_si256((const __m256i *)y[2*ib + 1].qs);
|
|
||||||
|
|
||||||
const __m128i q4_01_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_01, m4b));
|
|
||||||
const __m128i q4_01_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_01, 4), m4b));
|
|
||||||
const __m128i q4_23_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_23, m4b));
|
|
||||||
const __m128i q4_23_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_23, 4), m4b));
|
|
||||||
|
|
||||||
//reordering
|
|
||||||
const __m256i q4_01 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_01_lo,q4_01_hi), _mm_unpacklo_epi64(q4_01_lo,q4_01_hi));
|
|
||||||
const __m256i q4_23 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_23_lo,q4_23_hi),_mm_unpacklo_epi64(q4_23_lo,q4_23_hi));
|
|
||||||
|
|
||||||
const __m256i p01 = mul_add_epi8(q4_01,q8_01);
|
|
||||||
const __m256i p_1 = _mm256_madd_epi16(p01, mone);
|
|
||||||
|
|
||||||
const __m256i p23 = mul_add_epi8(q4_23,q8_23);
|
|
||||||
const __m256i p_2 = _mm256_madd_epi16(p23, mone);
|
|
||||||
|
|
||||||
const float dy0 = GGML_CPU_FP16_TO_FP32(y[2*ib].d);
|
|
||||||
const float dy1 = GGML_CPU_FP16_TO_FP32(y[2*ib+1].d);
|
|
||||||
|
|
||||||
const float s0 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[0]) * dy0;
|
|
||||||
const float s1 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[1]) * dy0;
|
|
||||||
const float s2 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[2]) * dy1;
|
|
||||||
const float s3 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[3]) * dy1;
|
|
||||||
|
|
||||||
const __m256 scales01 = _mm256_set_m128(_mm_set1_ps(s1), _mm_set1_ps(s0));
|
|
||||||
const __m256 scales23 = _mm256_set_m128(_mm_set1_ps(s3), _mm_set1_ps(s2));
|
|
||||||
|
|
||||||
accum = _mm256_fmadd_ps(scales01, _mm256_cvtepi32_ps(p_1), accum);
|
|
||||||
accum = _mm256_fmadd_ps(scales23, _mm256_cvtepi32_ps(p_2), accum);
|
|
||||||
}
|
}
|
||||||
sumf = hsum_float_8(accum);
|
sumf = hsum_float_8(accum);
|
||||||
|
|
||||||
|
|||||||
@@ -14178,6 +14178,16 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_perf() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Inkling-2-Mini experts (NVFP4 in UD-Q4_K_XL): 128 experts, 8 used, n_embd 2304, gate_up 2x1152 fused, down 1152
|
||||||
|
for (int bs : {1, 8, 16, 32}) {
|
||||||
|
for (ggml_type type_a : {GGML_TYPE_NVFP4, GGML_TYPE_MXFP4, GGML_TYPE_Q4_0}) {
|
||||||
|
test_cases.emplace_back(new test_mul_mat(type_a, GGML_TYPE_F32, 2304, bs, 2304, {1, 1}, {1, 1}));
|
||||||
|
test_cases.emplace_back(new test_mul_mat(type_a, GGML_TYPE_F32, 2304, bs, 1152, {1, 1}, {1, 1}));
|
||||||
|
test_cases.emplace_back(new test_mul_mat_id(type_a, GGML_TYPE_F32, 128, 8, false, 2304, bs, 2304));
|
||||||
|
test_cases.emplace_back(new test_mul_mat_id(type_a, GGML_TYPE_F32, 128, 8, false, 2304, bs, 1152));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
for (int K : {3, 5}) {
|
for (int K : {3, 5}) {
|
||||||
for (int IC : {256, 2560}) {
|
for (int IC : {256, 2560}) {
|
||||||
for (int IW_IH : {32, 64, 256}) {
|
for (int IW_IH : {32, 64, 256}) {
|
||||||
|
|||||||
Reference in new issue
Block a user