From 418fe97b5f01b11eeb89f30d8a0459433edbf1d2 Mon Sep 17 00:00:00 2001 From: cwriter Date: Sat, 21 Nov 2026 04:37:40 +0100 Subject: [PATCH] sycl: Improve Grouped XMX GEMM by avoiding SLM staging (#31428) --- docs/backend/SYCL.md | 1 + ggml/src/ggml-sycl/CMakeLists.txt | 14 +- ggml/src/ggml-sycl/common.hpp | 1 + ggml/src/ggml-sycl/fused-gemm-esimd.cpp | 1160 +++++++++++++++++++++++ ggml/src/ggml-sycl/fused-gemm-esimd.hpp | 33 + ggml/src/ggml-sycl/fused-gemm.cpp | 89 +- ggml/src/ggml-sycl/fused-gemm.hpp | 27 + ggml/src/ggml-sycl/ggml-sycl.cpp | 3 + 8 files changed, 1299 insertions(+), 29 deletions(-) create mode 100644 ggml/src/ggml-sycl/fused-gemm-esimd.cpp create mode 100644 ggml/src/ggml-sycl/fused-gemm-esimd.hpp diff --git a/docs/backend/SYCL.md b/docs/backend/SYCL.md index ca0b6d7bc..4a1d16aad 100644 --- a/docs/backend/SYCL.md +++ b/docs/backend/SYCL.md @@ -827,6 +827,7 @@ User can use the device management in [docs/multi-gpu.md](https://github.com/ggm | GGML_SYCL_XMX_MAX_COLS | non-negative integer, 80 (default) | Most activation columns (batch size) the `GGML_SYCL_ENABLE_XMX_ESIMD` kernels take; wider batches use the other mat-mul kernels. A larger value keeps XMX for bigger batches (as repeated launches), `0` never uses it. | | GGML_SYCL_XMX_GLU_MAX_COLS | 0 to 16, 16 (default) | Most columns the fused gate and up XMX kernel takes; wider batches run gate and up as separate mat-vecs. 16 is the most the kernel is built for, larger values are clamped to 16 and `0` turns the fused kernel off. | | GGML_SYCL_XMX_GATHER_TYPES | decimal bitmask, all bits set (default) | Weight formats that may use the XMX dequant-GEMM paths, which dequantize weights straight into the XMX tiles. This speeds up prompt processing of MoE models on GPUs with XMX units (Arc A- and B-series, Arc Pro, Data Center GPU Max), for example pp512 of Qwen3-30B-A3B UD-IQ3_XXS by about 50% on an Arc Pro B60. Bits:
* 1: IQ4_NL, 2: IQ3_S, 4: IQ4_XS, 8: IQ3_XXS, 16: IQ2_XXS, 32: IQ2_XS, 64: IQ2_S, 128: IQ1_S, 256: IQ1_M
* 512: Q8_0, 1024: Q4_K, 2048: Q5_K, 4096: Q6_K, 8192: Q2_K, 16384: Q3_K (MoE `MUL_MAT_ID` only)
Add values to combine them, for example `3` for IQ4_NL and IQ3_S; `0` disables the paths. A set bit does not force the path: batches of more than 64 tokens per expert use the library GEMM unless the device schedule width heuristic is overridden with `GGML_SYCL_MMID_SCHED=3`. Row lengths must be a multiple of 256 (32 for IQ4_NL and Q8_0). | +| GGML_SYCL_MMID_ESIMD | 0, 1 (default), 2 | The `MUL_MAT_ID` grouped XMX dequant-GEMM as an ESIMD kernel that dequantizes the weights straight into the `dpas` register operands, with no shared local memory image of them, from either weight layout (the per-expert reorder or the canonical blocks). q8_0, iq4_nl, iq4_xs, iq3_s, iq3_xxs and q2_K to q6_K. It stands in for the 8x16x16 sub-group-16 combination of f16 or bf16 (Xe2, Xe-HPC) and for the 8x8x16 sub-group-8 one (Xe-HPG: Arc A-series, ARL-H; there it works on 16 rows a thread and reads with 1D block loads, since that architecture has half the register file and no 2D block loads); while it is on, a grouped call whose weight type it covers is moved to that combination from any other shape of its type and sub-group size, whatever `GGML_SYCL_XMX_GATHER_SHAPES` allows, because the shape bits choose among `joint_matrix` tiles and the ESIMD kernel tiles on its own. A call it declines runs that combination's `joint_matrix` kernel. 0 uses the `joint_matrix` kernel, 1 the ESIMD one, 2 the ESIMD one summing K in the `joint_matrix` kernel's order (bit-identical output for f16; slower). An AOT build gets its 256-register file through `-ze-opt-enable-auto-large-GRF-mode` (set by the CMake build). Not built with `GGML_SYCL_SUPPORT_XMX=OFF`. | | GGML_SYCL_MMID_SCHED | decimal bitmask, 1 (default) | Build the grouped MoE routing and tile schedule on the GPU, avoiding the expert-ID host readback and CPU sort.
* 1: enable the device schedule.
* 2: override the fallback for models with few routed experts by disabling the performance heuristic that rejects batches averaging more than 64 routed rows per expert. This may reduce performance; type, shape, precision and device capability checks still apply.
Use `3` to enable the device schedule and override this heuristic, or `0` to use the host schedule. Bit 2 alone does not enable the device schedule. Higher bits are ignored. | | GGML_SYCL_XMX_GATHER_SHAPES | decimal bitmask, 255 (default) | XMX `joint_matrix` combinations the paths of `GGML_SYCL_XMX_GATHER_TYPES` may use; the operand type comes from `GGML_SYCL_DYNAMIC_PRECISION` and the best supported combination is picked automatically (logged as `fg_pick_combo`). Bits:
* Xe2, Xe3, Xe-HPC: 1: f16 8x16x16, 2: f16 16x16x16, 4: f16 32x64x16, 8: f16 32x64x32, 32: tf32 8x16x8, 64: bf16 8x16x16
* Xe-HPG (Arc A770, ARL-H): 16: f16 8x8x16, 128: bf16 8x8x16
Clear a bit to exclude a combination, or set a single bit to force one for testing. | | GGML_SYCL_DYNAMIC_PRECISION | `F16` (default with `GGML_SYCL_F16=ON`), `BF16`, `TF32` or `F32` (default otherwise) | Operand type of the XMX dequant-GEMM paths (`GGML_SYCL_XMX_GATHER_TYPES`); accumulation is always f32. `F16` is the fastest, but activations above 65504 overflow. `BF16` keeps the f32 range at a 7-bit mantissa, `TF32` keeps the range and the f16 mantissa but is about 30% slower and needs Xe2, Xe3 or Xe-HPC, and `F32` turns the XMX paths off. Ops that request a higher src1 precision ([TAG_GGML_PREC]) get it regardless of this setting. | diff --git a/ggml/src/ggml-sycl/CMakeLists.txt b/ggml/src/ggml-sycl/CMakeLists.txt index 8312a8305..0a5aede0b 100644 --- a/ggml/src/ggml-sycl/CMakeLists.txt +++ b/ggml/src/ggml-sycl/CMakeLists.txt @@ -218,6 +218,8 @@ endif() if (GGML_SYCL_TARGET STREQUAL "INTEL") add_compile_definitions(GGML_SYCL_WARP_SIZE=16) + # With AOT the option goes to ocloc inside -options (see GGML_SYCL_DEVICE_ARCH below): a bare + # -Xs hands it to ocloc as a command-line flag, which fails the device link. if (NOT GGML_SYCL_DEVICE_ARCH) target_link_options(ggml-sycl PRIVATE -Xs -ze-intel-greater-than-4GB-buffer-required) else() @@ -248,10 +250,18 @@ endif() if (GGML_SYCL_DEVICE_ARCH) message(STATUS "GGML_SYCL_DEVICE_ARCH=${GGML_SYCL_DEVICE_ARCH} (AOT via spir64_gen)") + # AOT (oneAPI 2026.2) ignores the grf_size<256> kernel property on ESIMD kernels and builds them + # with 128 registers, which made the ESIMD MoE GEMM 1.7-3.9x slower than under JIT; SPMD kernels + # keep honouring the property. A module-wide -doubleGRF also moves the ESIMD kernels that ask for + # 128 to 256, and a server built with it hung on its first request, so IGC picks the size per + # kernel from its register pressure instead (ESIMD kernels too; -ze-opt-large-grf-kernel does not + # reach them). + set(GGML_SYCL_AOT_BACKEND_OPTIONS + "-device ${GGML_SYCL_DEVICE_ARCH} -options '-ze-intel-greater-than-4GB-buffer-required -ze-opt-enable-auto-large-GRF-mode'") target_compile_options( ggml-sycl PRIVATE -fsycl-targets=spir64_gen - "SHELL:-Xsycl-target-backend=spir64_gen \"-device ${GGML_SYCL_DEVICE_ARCH} -options -ze-intel-greater-than-4GB-buffer-required\"" + "SHELL:-Xsycl-target-backend=spir64_gen \"${GGML_SYCL_AOT_BACKEND_OPTIONS}\"" ) # Pass through parallel job (process) count for parallelising the @@ -266,7 +276,7 @@ if (GGML_SYCL_DEVICE_ARCH) target_link_options( ggml-sycl PRIVATE -fsycl-targets=spir64_gen - "SHELL:-Xsycl-target-backend=spir64_gen \"-device ${GGML_SYCL_DEVICE_ARCH} -options -ze-intel-greater-than-4GB-buffer-required\"" + "SHELL:-Xsycl-target-backend=spir64_gen \"${GGML_SYCL_AOT_BACKEND_OPTIONS}\"" -fsycl-max-parallel-link-jobs=${GGML_SYCL_MAX_PARALLEL_LINK_JOBS} ) diff --git a/ggml/src/ggml-sycl/common.hpp b/ggml/src/ggml-sycl/common.hpp index 75e783d12..ae156acca 100644 --- a/ggml/src/ggml-sycl/common.hpp +++ b/ggml/src/ggml-sycl/common.hpp @@ -72,6 +72,7 @@ extern int g_ggml_sycl_debug; extern int g_ggml_sycl_enable_optimize; extern int g_ggml_sycl_enable_fusion; extern int g_ggml_sycl_enable_esimd; +extern int g_ggml_sycl_mmid_esimd; extern int g_ggml_sycl_mmvq_wide; extern int g_ggml_sycl_enable_xmx_esimd; extern int g_ggml_sycl_xmx_max_cols; diff --git a/ggml/src/ggml-sycl/fused-gemm-esimd.cpp b/ggml/src/ggml-sycl/fused-gemm-esimd.cpp new file mode 100644 index 000000000..37f87e377 --- /dev/null +++ b/ggml/src/ggml-sycl/fused-gemm-esimd.cpp @@ -0,0 +1,1160 @@ +#include "fused-gemm-esimd.hpp" + +#include +#include +#include + +// One thread owns MR rows of one expert's weight and one BN-wide tile of the schedule, over the +// whole K. Per k step (32 values) it loads the rows' stored bits, dequantizes them in registers to +// the f16 A operand of dpas (RC = 8 rows x 16 k, row major) and multiplies it with the packed B, +// which is already the dpas VNNI operand: [K/2][Npad][2] halves, NW columns a dpas. +// +// The joint_matrix kernel dequantizes into SLM and reads it back in the same sub-group, which is +// staging, not sharing; dpas takes registers, so that round trip and its barriers are gone here. +// Its K split over GGML_SYCL_GROUPED_GEMM_KSPLIT sub-groups is reproduced as sequential K ranges summed in +// the same ascending order as its tile_c reduction, and every format below rounds A exactly as its +// joint_matrix A stage does, so the output is the same bits. +// +// A weight format is a grouped_gemm_esimd_a with: raw, the stored bits of the MR rows for one k +// step; load(), which fetches them from the expert's slice in either layout (grouped_gemm_esimd::slice); and +// decode(), the f16 A of one row group. +// It reads the same fields as the joint_matrix A stage of its format. +// +// The kernel is built for one grouped_gemm_arch: the dpas width NW (the columns of B one dpas takes, the +// sub-group size of the joint_matrix combination it stands in for), the rows of one thread, and +// whether 2D block loads exist. Xe2 and Xe-HPC: 16 wide, 32 rows, 2D loads. Xe-HPG (Arc A, ARL-H): +// 8 wide, no 2D block messages, and 256 registers of 32 bytes, half of Xe2's file, so 16 rows: its +// 32-row accumulators alone would take half the registers. +static constexpr int GGML_SYCL_GROUPED_GEMM_KSPLIT = FG_KSPLIT; +static constexpr int GGML_SYCL_GROUPED_GEMM_BN = GGML_SYCL_FG_BN; +static constexpr int GROUPED_GEMM_MR = 32; // rows of one thread on Xe2: one 2D load covers them +static constexpr int GROUPED_GEMM_RC = 8; // rows of one dpas +static constexpr int GROUPED_GEMM_QK = 32; // values of one k step + +template struct grouped_gemm_arch { + static constexpr int NW = NW_; + static constexpr int MR = MR_; + static constexpr bool L2D = L2D_; +}; + +using grouped_gemm_xe2 = grouped_gemm_arch<16, GROUPED_GEMM_MR, true>; +using grouped_gemm_xehpg = grouped_gemm_arch<8, 16, false>; + +// An AOT build compiles each kernel only for the targets of its dpas width: GGML_SYCL_XMX_AOT_SG for +// spir64_gen, FG_AOT_SG (fused-gemm.hpp) for intel_gpu_* targets. A build without XMX (GGML_SYCL_SUPPORT_XMX +// off) compiles none of them +template static constexpr bool grouped_gemm_built() { +#if !defined(GGML_SYCL_SUPPORT_XMX) + return false; +#elif defined(GGML_SYCL_XMX_AOT_SG) + return A::NW == GGML_SYCL_XMX_AOT_SG; +#elif defined(FG_AOT_SG) + return A::NW == FG_AOT_SG; +#else + return true; +#endif +} + +namespace grouped_gemm_esimd { +using namespace sycl::ext::intel::esimd; + +// f(std::integral_constant) for I in [0, N): register selects at a runtime offset become +// indirect accesses, so loops whose index picks registers are spelled out at compile time +template struct static_for { + template static ESIMD_INLINE void run(Fn & f) { + if constexpr (I < N) { + f(std::integral_constant{}); + static_for::run(f); + } + } +}; + +// One field of a format's block. In the reorder (SoA) layout it is a stream of its own, `size` bytes +// a block, after `pre` bytes a block of the streams before it; in the canonical (AoS) layout it is +// `aos` bytes into the block. +struct field { + int pre; + int size; + int aos; +}; + +// Where one thread's rows sit in an expert's slice, in either layout. nblocks counts the stored blocks +// of the slice, bpr those of one row, bs is the size of one block. +template struct slice { + const uint8_t * x; + int M; + int bpr; + int nblocks; + int m0; + bool aos; + int bs; + + // per row, the byte offset from x of byte `inner` of field f of block ib + ESIMD_INLINE simd at(const field & f, const int ib, const int inner) const { + const int stride = aos ? bs : f.size; + simd o(0, (uint32_t) (bpr * stride)); + return o + (uint32_t) ((aos ? f.aos : nblocks * f.pre) + (m0 * bpr + ib) * stride + inner); + } + + // a field's bytes at offset `inner` start on a dword in every block (the AoS block and offset + // are multiples of 4); the SoA streams always do + ESIMD_INLINE bool dword_aligned(const field & f, const int inner) const { + return !aos || (bs % 4 == 0 && (f.aos + inner) % 4 == 0); + } + + ESIMD_INLINE simd u16(const field & f, const int ib, const int inner) const { + return gather((const uint16_t *) x, at(f, ib, inner)); + } + + ESIMD_INLINE simd u32(const field & f, const int ib, const int inner) const { + if (dword_aligned(f, inner)) { + return gather((const uint32_t *) x, at(f, ib, inner)); + } + return rows<1>(f, ib, inner); + } + + // W dwords of each of the MR rows from byte `inner` (a multiple of 4) of field f of block ib, as + // [row][W]. SoA with 2D block loads (L2D): one message over the stream. Otherwise the rows' dwords + // are gathered, as [W][row], at offsets rounded down to a dword (every field starts on an even + // byte; the SoA streams are dword aligned), shifted by 16 bits where an AoS row's block starts on + // 2 mod 4, and transposed with strided reads. + template ESIMD_INLINE simd rows(const field & f, const int ib, const int inner) const { + if constexpr (L2D) { + if (!aos) { + const unsigned pitch = (unsigned) (bpr * f.size); + return load_2d((const uint32_t *) (x + (size_t) nblocks * f.pre), pitch - 1, M - 1, + pitch - 1, (ib * f.size + inner) / 4, m0); + } + } + const simd o = at(f, ib, inner); + simd g; + if (dword_aligned(f, inner)) { + g = gather((const uint32_t *) x, o); + } else { + // per row: sh is 16 where the run starts on 2 mod 4, else 0; hm keeps the next dword's low + // half only where it is needed + const simd oa = o & ~3u; + const simd sh = (o & 2u) << 3; + const simd hm = 0u - (sh >> 4); + simd lo = gather((const uint32_t *) x, oa); + // the dword after the run holds its last 2 bytes where a row needs it; a row that does not + // reads its first dword again rather than past the field + const simd tail = gather((const uint32_t *) x, oa + ((4u * W) & hm)); + auto shift = [&](auto vc) SYCL_ESIMD_FUNCTION { + constexpr int v = decltype(vc)::value; + simd a = lo.template select(v * MR); + simd b; + if constexpr (v + 1 < W) { + b = lo.template select((v + 1) * MR); + } else { + b = tail; + } + g.template select(v * MR) = (a >> sh) | ((b << 16) & hm); + }; + static_for<0, W>::run(shift); + } + if constexpr (W == 1) { + return g; + } else { + simd t; + auto row = [&](auto rc) SYCL_ESIMD_FUNCTION { + constexpr int r = decltype(rc)::value; + t.template select(r * W) = g.template select(r); + }; + static_for<0, MR>::run(row); + return t; + } + } +}; + +// A decode returns its row group's A in the order the two dpas of a k step read it, [k half][row][16 +// k]: both iq formats store k and k + 16 in the same byte or grid dword half, so that order needs +// only reads with a stride, while [row][32 k] needed strided writes, which split into SIMD4. +using a_halves = simd; + +// f16 of the f32 product code * scale, each row's scale over its 16 values of a k half: the +// expression the joint_matrix stages evaluate before E::make rounds it +ESIMD_INLINE simd scale_half(simd v, + simd d) { + v *= d.template replicate_vs_w_hs(0); + return v; +} + +// The iq4 codebook by byte, in SLM: entry b holds the values of its low and its high nibble as the +// two 16-bit halves of one dword, as T (half, or int16 for an f32 product). A read-only table every +// lane indexes, which is what SLM is for: register-indirect iselect() over the same lanes cost +// 7.3 of 12.7 ms, a 16-entry SLM table per nibble about 0.4. +template ESIMD_INLINE void iq4_pair_table_to_slm() { + static_assert(sizeof(T) == 2, "two values a dword"); + simd lo; +#pragma unroll + for (int i = 0; i < 16; ++i) { + lo[i] = sycl::bit_cast((T) kvalues_iq4nl[i]); + } +#pragma unroll + for (int hi = 0; hi < 16; ++hi) { + slm_block_store(hi * 64, lo | ((uint32_t) sycl::bit_cast((T) kvalues_iq4nl[hi]) << 16)); + } +} + +// 16 packed bytes a row; byte j holds k = j in its low and k = 16 + j in its high nibble (the order +// of fg_decode_iq4_xs and of the iq4_nl stage). One gather per byte; the low halves are k half 0, +// the high halves k half 1, each [row][16]. +template +ESIMD_INLINE simd iq4_codes(simd q) { + simd off = q.template bit_cast_view(); + simd pr = slm_gather(off * 4); + auto pv = pr.template bit_cast_view(); + simd v; + v.template select(0) = pv.template select(0); + v.template select(GROUPED_GEMM_RC * 16) = pv.template select(1); + return v; +} +// A 2D surface: rows `pitch` bytes apart, 16-byte aligned and at least 64 bytes +constexpr bool surface_ok(const int pitch) { + return pitch % 16 == 0 && pitch >= 64; +} + +// byte i of each lane's dword, i a runtime index: a shift by a scalar, not a register pick +template ESIMD_INLINE simd byte_of(simd w, const int i) { + return (w >> (8 * i)) & 0xffu; +} + +// one of four per-lane dwords by a runtime index, as uniform branches rather than a register pick +template ESIMD_INLINE simd dword_of(simd * w, const int i) { + simd r = w[0]; + if (i == 1) { + r = w[1]; + } else if (i == 2) { + r = w[2]; + } else if (i == 3) { + r = w[3]; + } + return r; +} + +// a row group's 32 k-ordered code bytes a row as f32, in [k half][row][16 k] +ESIMD_INLINE simd byte_codes(simd q) { + auto bm = q.template bit_cast_view(); + simd v; + v.template select(0) = bm.template select(0, 0); + v.template select(GROUPED_GEMM_RC * 16) = + bm.template select(0, 16); + return v; +} + +// f16 of d * code - m in f32, each row's d and m over its 16 values of a k half: the expression of +// the K-quant stages with a min +ESIMD_INLINE simd scale_min_half(simd v, + simd d, + simd m) { + v = v * d.template replicate_vs_w_hs(0) - + m.template replicate_vs_w_hs(0); + return v; +} + +// the two k halves of a row group, the code bytes times one (d, m) per row and half +ESIMD_INLINE a_halves halves_dm(simd q, + simd d0, + simd m0, + simd d1, + simd m1, + const float bias = 0.0f) { + simd c = byte_codes(q); + if (bias != 0.0f) { + c -= bias; + } + a_halves v; + v.template select(0) = + scale_min_half(c.template select(0), d0, m0); + v.template select(GROUPED_GEMM_RC * 16) = + scale_min_half(c.template select(GROUPED_GEMM_RC * 16), d1, m1); + return v; +} + +// a row group's 32 values from grid dwords (byte b of entry e is k = 4e + b) and a sign dword per +// row (bit k negates k), times d: the expression of the iq3 stages +ESIMD_INLINE a_halves grid_signed(simd ge, + simd signs, + simd d) { + auto gm = ge.template bit_cast_view(); + simd sg = signs.template replicate_vs_w_hs(0); + simd k(0, 1); + k &= 15; + a_halves v; +#pragma unroll + for (int h = 0; h < 2; ++h) { + simd eh = gm.template select(0, 4 * h); + simd f = eh.template bit_cast_view(); + f.merge(-f, ((sg >> (k + 16 * h)) & 1) != 0); + v.template select(h * GROUPED_GEMM_RC * 16) = scale_half(f, d); + } + return v; +} +} // namespace grouped_gemm_esimd + +template struct grouped_gemm_esimd_a { + static constexpr bool supported = false; +}; + +// [qs 32][d], one block a k step +template struct grouped_gemm_esimd_a { + static constexpr bool supported = true; + static constexpr int qk = QK8_0; + static constexpr bool whole = true; // decode_all() for the whole tile + static constexpr int slm_bytes = 0; + static ESIMD_INLINE void init_slm() {} + struct raw { + grouped_gemm_esimd::simd q; + grouped_gemm_esimd::simd d; + }; + + static constexpr grouped_gemm_esimd::field QS = { 0, QK8_0, 2 }, D = { QK8_0, 2, 0 }; + static bool ok(const int K, const bool aos) { return K % qk == 0 && (aos || K >= 64); } + struct sb_t {}; + template static ESIMD_INLINE void load_sb(sb_t &, const Sl &, const int) {} + template static ESIMD_INLINE void load(raw & r, sb_t &, const Sl & s, const int kb, const int) { + r.q = s.template rows<8>(QS, kb, 0); + r.d = s.u16(D, kb, 0); + } + // all MR x 32 quants at once: int8 -> half, times the row's scale. q * d needs at most 19 + // significant bits, so the f32 product the joint_matrix stage rounds to half is exact and one + // correctly rounded half multiply gives the same bits. + static ESIMD_INLINE grouped_gemm_esimd::simd decode_all(raw & r) { + using namespace grouped_gemm_esimd; + simd av = r.q.template bit_cast_view(); + simd dh = r.d.template bit_cast_view(); + av *= dh.template replicate_vs_w_hs(0); + return av; + } +}; + +// [qs 16][d], one block a k step +template struct grouped_gemm_esimd_a { + static constexpr bool supported = true; + static constexpr int qk = QK4_NL; + static constexpr bool whole = false; + static constexpr int slm_bytes = 256 * sizeof(uint32_t); + + static ESIMD_INLINE void init_slm() { grouped_gemm_esimd::iq4_pair_table_to_slm(); } + struct raw { + grouped_gemm_esimd::simd q; + grouped_gemm_esimd::simd d; + }; + + static constexpr grouped_gemm_esimd::field QS = { 0, QK4_NL / 2, 2 }, D = { QK4_NL / 2, 2, 0 }; + static bool ok(const int K, const bool aos) { return K % qk == 0 && (aos || K / 2 >= 64); } + struct sb_t {}; + template static ESIMD_INLINE void load_sb(sb_t &, const Sl &, const int) {} + template static ESIMD_INLINE void load(raw & r, sb_t &, const Sl & s, const int kb, const int) { + using namespace grouped_gemm_esimd; + r.q = s.template rows<4>(QS, kb, 0); + simd d = s.u16(D, kb, 0); + r.d = d.template bit_cast_view(); + } + // the codebook values are integers below 128, exact in half, and d * value has at most 18 + // significant bits: one correctly rounded half multiply is the stage's f32 product rounded + template static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) { + using namespace grouped_gemm_esimd; + a_halves v = iq4_codes(r.q.template select(g * GROUPED_GEMM_RC * 4)); + simd dg = r.d.template select(g * GROUPED_GEMM_RC); + simd dr = dg.template replicate_vs_w_hs(0); + v.template select(0) *= dr; + v.template select(GROUPED_GEMM_RC * 16) *= dr; + return v; + } +}; + +// [qs 128][d 2][scales_h 2][scales_l 4] per 256 values; k step ib8 of a superblock is sub-block ib8 +template struct grouped_gemm_esimd_a { + static constexpr bool supported = true; + static constexpr int qk = QK_K; + static constexpr bool whole = false; + // int16 table and f32 products: d * (scale - 32) * value can need 24 significant bits, more + // than a half multiply keeps + static constexpr int slm_bytes = 256 * sizeof(uint32_t); + + static ESIMD_INLINE void init_slm() { grouped_gemm_esimd::iq4_pair_table_to_slm(); } + struct raw { + grouped_gemm_esimd::simd q; + grouped_gemm_esimd::simd d; + }; + struct sb_t { + grouped_gemm_esimd::simd d; + grouped_gemm_esimd::simd sh; // scales_h + grouped_gemm_esimd::simd sl; // scales_l, 4 bytes + }; + + static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 2, 8 }, D = { QK_K / 2, 2, 0 }, + SH = { QK_K / 2 + 2, 2, 2 }, SL = { QK_K / 2 + 4, 4, 4 }; + static bool ok(const int K, const bool) { return K % qk == 0; } + template static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) { + using namespace grouped_gemm_esimd; + simd dh = s.u16(D, ib, 0); + sb.d = simd(dh.template bit_cast_view()); + sb.sh = s.u16(SH, ib, 0); + sb.sl = s.u32(SL, ib, 0); + } + template static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) { + using namespace grouped_gemm_esimd; + r.q = s.template rows<4>(QS, kb / 8, 16 * j); + // d * (6-bit scale - 32), in f32 as the stage computes it + simd sc = simd(((sb.sl >> (4 * j)) & 0xf) | (((sb.sh >> (2 * j)) & 3) << 4)) - 32; + r.d = sb.d * simd(sc); + } + + template static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) { + using namespace grouped_gemm_esimd; + simd c = + iq4_codes(r.q.template select(g * GROUPED_GEMM_RC * 4)); + simd dg = r.d.template select(g * GROUPED_GEMM_RC); + a_halves v; + v.template select(0) = scale_half(c.template select(0), dg); + v.template select(GROUPED_GEMM_RC * 16) = + scale_half(c.template select(GROUPED_GEMM_RC * 16), dg); + return v; + } +}; + +// [qs 64][qh 8][signs 32][d 2, scales 4] per 256 values. Value k of sub-block ib8 is byte k % 4 of +// grid entry k / 4, the entry index being qs[k / 4] with bit 8 from qh bit k / 4, negated where bit +// k of the sub-block's sign dword is set; times d * (1 + 2 * 4-bit scale) +template struct grouped_gemm_esimd_a { + static constexpr bool supported = true; + static constexpr int qk = QK_K; + static constexpr bool whole = false; + // the 512-entry grid in SLM, read by every lane: the global gather of it cost 2.4x the kernel + static constexpr int slm_bytes = 512 * sizeof(uint32_t); + static ESIMD_INLINE void init_slm() { + using namespace grouped_gemm_esimd; +#pragma unroll + for (int i = 0; i < 512; i += 64) { + slm_block_store(i * 4, block_load(iq3s_grid + i)); + } + } + struct raw { + grouped_gemm_esimd::simd q; // [row][8 qs bytes] + grouped_gemm_esimd::simd qh; // the sub-block's qh byte + grouped_gemm_esimd::simd signs; // the sub-block's 32 sign bits + grouped_gemm_esimd::simd d; + }; + // d and the scale bytes are a block's, kept over its k steps; the sign dword and the qh byte of + // a k step are a 2D load each at that step's column, so no register is picked by a runtime index + struct sb_t { + grouped_gemm_esimd::simd d; + grouped_gemm_esimd::simd sc; // the 4 scale bytes + }; + // the SoA stream {d, scales} is d and scales of the block in AoS, which are not adjacent there: MS + // reads the scales at their SoA offset 2 in the stream, which is AoS offset 2 past 104 + static constexpr int META = QK_K / 4 + QK_K / 32 + QK_K / 8; + static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 4, 2 }, QH = { QK_K / 4, QK_K / 32, 2 + QK_K / 4 }, + SG = { QK_K / 4 + QK_K / 32, QK_K / 8, 2 + QK_K / 4 + QK_K / 32 }, + MD = { META, 2 + IQ3S_N_SCALE, 0 }, + MS = { META, 2 + IQ3S_N_SCALE, META }; + // the SoA qh and signs streams are 2D surfaces: their row pitches must be 16-byte aligned + static bool ok(const int K, const bool aos) { + return K % qk == 0 && (aos || ((K / QK_K * 8) % 16 == 0 && K / QK_K * 8 >= 64)); + } + template static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) { + using namespace grouped_gemm_esimd; + simd dh = s.u16(MD, ib, 0); + simd s0 = s.u16(MS, ib, 2); + simd s1 = s.u16(MS, ib, 4); + sb.d = simd(dh.template bit_cast_view()); + sb.sc = s0 | (s1 << 16); + } + template static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) { + using namespace grouped_gemm_esimd; + const int ib = kb / 8; + r.q = s.template rows<2>(QS, ib, 8 * j); + r.qh = (s.template rows<1>(QH, ib, 4 * (j / 4)) >> (8 * (j % 4))) & 0xffu; + r.signs = s.template rows<1>(SG, ib, 4 * j); + r.d = sb.d * simd(simd(1 + 2 * ((sb.sc >> (4 * j)) & 0xf))); + } + + template static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) { + using namespace grouped_gemm_esimd; + // the 8 grid entries of each of the row group's rows + simd qb = + r.q.template select(g * GROUPED_GEMM_RC * 2).template bit_cast_view(); + simd qr = r.qh.template select(g * GROUPED_GEMM_RC); + simd qh = qr.template replicate_vs_w_hs(0); + simd j(0, 1); + j &= 7; + simd idx = simd(qb) | (((qh >> j) & 1) << 8); + simd ge = slm_gather(idx * 4); + return grid_signed(ge, r.signs.template select(g * GROUPED_GEMM_RC), + r.d.template select(g * GROUPED_GEMM_RC)); + } +}; + +// The K-quants and iq3_xxs: a 256-value superblock, k step j its sub-block j. Their value bytes for +// a k step are one rectangle of a stream, in k order after a shift and a mask per dword; scales, +// mins and d are the block's and are loaded once per block. + +// [qs 128][scales 12][dm]: value l of sub-block j is nibble j & 1 of byte 32 (j / 2) + l, times +// dall * scale_j, minus dmin * min_j (6-bit scale and min, get_scale_min_k4) +template struct grouped_gemm_esimd_a { + static constexpr bool supported = true; + static constexpr int qk = QK_K; + static constexpr bool whole = false; + static constexpr int slm_bytes = 0; + static ESIMD_INLINE void init_slm() {} + struct raw { + grouped_gemm_esimd::simd q; + grouped_gemm_esimd::simd d, m; + }; + struct sb_t { + grouped_gemm_esimd::simd sc[3]; // the 12 scale bytes + grouped_gemm_esimd::simd dall, dmin; + }; + + static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 2, 16 }, SC = { QK_K / 2, K_SCALE_SIZE, 4 }, + DM = { QK_K / 2 + K_SCALE_SIZE, 4, 0 }; + + static bool ok(const int K, const bool aos) { + return K % qk == 0 && (aos || grouped_gemm_esimd::surface_ok(K / 2)); + } + + template + static ESIMD_INLINE void load_sb(sb_t & sb, + const Sl & s, + const int ib, + const grouped_gemm_esimd::field & sc = SC, + const grouped_gemm_esimd::field & dmf = DM) { + using namespace grouped_gemm_esimd; +#pragma unroll + for (int w = 0; w < 3; ++w) { + sb.sc[w] = s.u32(sc, ib, 4 * w); + } + simd dm = s.u32(dmf, ib, 0); + auto dh = dm.template bit_cast_view(); + sb.dall = simd(dh.template select(0)); + sb.dmin = simd(dh.template select(1)); + } + // get_scale_min_k4 for sub-block j over the lanes + static ESIMD_INLINE void scale_min(sb_t & sb, + const int j, + grouped_gemm_esimd::simd & d, + grouped_gemm_esimd::simd & m) { + using namespace grouped_gemm_esimd; + simd sc, mb; + if (j < 4) { + sc = byte_of(sb.sc[0], j) & 63; + mb = byte_of(sb.sc[1], j) & 63; + } else { + sc = (byte_of(sb.sc[2], j - 4) & 0xf) | ((byte_of(sb.sc[0], j - 4) >> 6) << 4); + mb = (byte_of(sb.sc[2], j - 4) >> 4) | ((byte_of(sb.sc[1], j - 4) >> 6) << 4); + } + d = sb.dall * simd(sc); + m = sb.dmin * simd(mb); + } + template static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) { + using namespace grouped_gemm_esimd; + r.q = (s.template rows<8>(QS, kb / 8, 32 * (j / 2)) >> (4 * (j & 1))) & 0x0F0F0F0Fu; + scale_min(sb, j, r.d, r.m); + } + + template static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) { + using namespace grouped_gemm_esimd; + simd d = r.d.template select(g * GROUPED_GEMM_RC), + m = r.m.template select(g * GROUPED_GEMM_RC); + return halves_dm(r.q.template select(g * GROUPED_GEMM_RC * 8), d, m, d, m); + } +}; + +// [qs 128][qh 32][scales 12][dm]: q4_K plus bit j of qh byte l as 16 +template struct grouped_gemm_esimd_a { + using q4 = grouped_gemm_esimd_a; + static constexpr bool supported = true; + static constexpr int qk = QK_K; + static constexpr bool whole = false; + static constexpr int slm_bytes = 0; + static ESIMD_INLINE void init_slm() {} + using raw = typename q4::raw; + using sb_t = typename q4::sb_t; + static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 2, 48 }, QH = { QK_K / 2, QK_K / 8, 16 }, + SC = { QK_K / 2 + QK_K / 8, K_SCALE_SIZE, 4 }, + DM = { QK_K / 2 + QK_K / 8 + K_SCALE_SIZE, 4, 0 }; + static bool ok(const int K, const bool aos) { + return K % qk == 0 && (aos || (grouped_gemm_esimd::surface_ok(K / 2) && grouped_gemm_esimd::surface_ok(K / 8))); + } + template static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) { q4::load_sb(sb, s, ib, SC, DM); } + template static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) { + using namespace grouped_gemm_esimd; + const int ib = kb / 8; + simd qh = s.template rows<8>(QH, ib, 0); + r.q = ((s.template rows<8>(QS, ib, 32 * (j / 2)) >> (4 * (j & 1))) & 0x0F0F0F0Fu) | + (((qh >> j) & 0x01010101u) << 4); + q4::scale_min(sb, j, r.d, r.m); + } + + template static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) { + return q4::template decode(r); + } +}; + +// [ql 128][qh 64][scales 16][d]: sub-block j is quarter r = j % 4 of half h = j / 4; value l is +// nibble r >> 1 of ql byte 64h + 32 (r & 1) + l with bits 2r, 2r+1 of qh byte 32h + l above it, +// minus 32, times d * the int8 scale 2j + l / 16 +template struct grouped_gemm_esimd_a { + static constexpr bool supported = true; + static constexpr int qk = QK_K; + static constexpr bool whole = false; + static constexpr int slm_bytes = 0; + static ESIMD_INLINE void init_slm() {} + struct raw { + grouped_gemm_esimd::simd q; + grouped_gemm_esimd::simd d0, d1; // per k half + }; + struct sb_t { + grouped_gemm_esimd::simd sc[4]; // the 16 int8 scales + grouped_gemm_esimd::simd d; + }; + + static constexpr grouped_gemm_esimd::field QL = { 0, QK_K / 2, 0 }, QH = { QK_K / 2, QK_K / 4, QK_K / 2 }, + SC = { QK_K / 2 + QK_K / 4, QK_K / 16, QK_K / 2 + QK_K / 4 }, + D = { QK_K / 2 + QK_K / 4 + QK_K / 16, 2, + QK_K / 2 + QK_K / 4 + QK_K / 16 }; + static bool ok(const int K, const bool aos) { + return K % qk == 0 && (aos || (grouped_gemm_esimd::surface_ok(K / 2) && grouped_gemm_esimd::surface_ok(K / 4))); + } + template static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) { + using namespace grouped_gemm_esimd; +#pragma unroll + for (int w = 0; w < 4; ++w) { + sb.sc[w] = s.u32(SC, ib, 4 * w); + } + simd dh = s.u16(D, ib, 0); + sb.d = simd(dh.template bit_cast_view()); + } + template static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) { + using namespace grouped_gemm_esimd; + const int ib = kb / 8, h = j / 4, q = j % 4; + simd ql = s.template rows<8>(QL, ib, 64 * h + 32 * (q & 1)); + simd qh = s.template rows<8>(QH, ib, 32 * h); + r.q = ((ql >> (4 * (q >> 1))) & 0x0F0F0F0Fu) | (((qh >> (2 * q)) & 0x03030303u) << 4); + // scales 2j and 2j + 1: bytes 2 (j % 2) and 2 (j % 2) + 1 of dword j / 2, sign extended + simd w = dword_of(sb.sc, j / 2); + simd s0 = simd(w << (24 - 16 * (j % 2))) >> 24; + simd s1 = simd(w << (16 - 16 * (j % 2))) >> 24; + r.d0 = sb.d * simd(s0); + r.d1 = sb.d * simd(s1); + } + + template static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) { + using namespace grouped_gemm_esimd; + simd z = 0.0f; + return halves_dm(r.q.template select(g * GROUPED_GEMM_RC * 8), + r.d0.template select(g * GROUPED_GEMM_RC), z, + r.d1.template select(g * GROUPED_GEMM_RC), z, 32.0f); + } +}; + +// [qs 64][scales 16][dm]: value l of sub-block j is bits 2 (j % 4) of byte 32 (j / 4) + l, times +// d * (scale & 15), minus dmin * (scale >> 4), scale byte 2j + l / 16 +template struct grouped_gemm_esimd_a { + static constexpr bool supported = true; + static constexpr int qk = QK_K; + static constexpr bool whole = false; + static constexpr int slm_bytes = 0; + static ESIMD_INLINE void init_slm() {} + struct raw { + grouped_gemm_esimd::simd q; + grouped_gemm_esimd::simd d0, m0, d1, m1; + }; + struct sb_t { + grouped_gemm_esimd::simd sc[4]; + grouped_gemm_esimd::simd d, dmin; + }; + + static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 4, QK_K / 16 }, SC = { QK_K / 4, QK_K / 16, 0 }, + DM = { QK_K / 4 + QK_K / 16, 4, QK_K / 4 + QK_K / 16 }; + + static bool ok(const int K, const bool aos) { + return K % qk == 0 && (aos || grouped_gemm_esimd::surface_ok(K / 4)); + } + template static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) { + using namespace grouped_gemm_esimd; +#pragma unroll + for (int w = 0; w < 4; ++w) { + sb.sc[w] = s.u32(SC, ib, 4 * w); + } + simd dm = s.u32(DM, ib, 0); + auto dh = dm.template bit_cast_view(); + sb.d = simd(dh.template select(0)); + sb.dmin = simd(dh.template select(1)); + } + template static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) { + using namespace grouped_gemm_esimd; + r.q = (s.template rows<8>(QS, kb / 8, 32 * (j / 4)) >> (2 * (j % 4))) & 0x03030303u; + simd w = dword_of(sb.sc, j / 2); + simd s0 = byte_of(w, 2 * (j % 2)), s1 = byte_of(w, 2 * (j % 2) + 1); + r.d0 = sb.d * simd(s0 & 0xf); + r.m0 = sb.dmin * simd(s0 >> 4); + r.d1 = sb.d * simd(s1 & 0xf); + r.m1 = sb.dmin * simd(s1 >> 4); + } + + template static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) { + using namespace grouped_gemm_esimd; + return halves_dm(r.q.template select(g * GROUPED_GEMM_RC * 8), + r.d0.template select(g * GROUPED_GEMM_RC), + r.m0.template select(g * GROUPED_GEMM_RC), + r.d1.template select(g * GROUPED_GEMM_RC), + r.m1.template select(g * GROUPED_GEMM_RC)); + } +}; + +// [qs 64][hmask 32][scales 12][d]: q2_K's 2 bits, minus 4 where bit j of hmask byte l is clear, +// times d * (6-bit scale - 32), scale s = 2j + l / 16 split over bytes s % 8 and 8 + s % 4 +template struct grouped_gemm_esimd_a { + static constexpr bool supported = true; + static constexpr int qk = QK_K; + static constexpr bool whole = false; + static constexpr int slm_bytes = 0; + static ESIMD_INLINE void init_slm() {} + struct raw { + grouped_gemm_esimd::simd q; + grouped_gemm_esimd::simd d0, d1; + }; + struct sb_t { + grouped_gemm_esimd::simd sc[4]; // 12 scale bytes, sc[3] unused + grouped_gemm_esimd::simd d; + }; + + static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 4, QK_K / 8 }, HM = { QK_K / 4, QK_K / 8, 0 }, + SC = { QK_K / 4 + QK_K / 8, 12, QK_K / 4 + QK_K / 8 }, + D = { QK_K / 4 + QK_K / 8 + 12, 2, QK_K / 4 + QK_K / 8 + 12 }; + static bool ok(const int K, const bool aos) { + return K % qk == 0 && (aos || (grouped_gemm_esimd::surface_ok(K / 4) && grouped_gemm_esimd::surface_ok(K / 8))); + } + template static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) { + using namespace grouped_gemm_esimd; +#pragma unroll + for (int w = 0; w < 3; ++w) { + sb.sc[w] = s.u32(SC, ib, 4 * w); + } + sb.sc[3] = 0; + simd dh = s.u16(D, ib, 0); + sb.d = simd(dh.template bit_cast_view()); + } + + static ESIMD_INLINE grouped_gemm_esimd::simd scale(sb_t & sb, const int sidx) { + using namespace grouped_gemm_esimd; + simd lo = (byte_of(dword_of(sb.sc, (sidx % 8) / 4), sidx % 4) >> (4 * (sidx / 8))) & 0xf; + simd hi = (byte_of(sb.sc[2], sidx % 4) >> (2 * (sidx / 4))) & 3; + return sb.d * simd(simd(lo | (hi << 4)) - 32); + } + template static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) { + using namespace grouped_gemm_esimd; + const int ib = kb / 8; + simd hm = s.template rows<8>(HM, ib, 0); + r.q = ((s.template rows<8>(QS, ib, 32 * (j / 4)) >> (2 * (j % 4))) & 0x03030303u) | + (((hm >> j) & 0x01010101u) << 2); + r.d0 = scale(sb, 2 * j); + r.d1 = scale(sb, 2 * j + 1); + } + + template static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) { + using namespace grouped_gemm_esimd; + simd z = 0.0f; + return halves_dm(r.q.template select(g * GROUPED_GEMM_RC * 8), + r.d0.template select(g * GROUPED_GEMM_RC), z, + r.d1.template select(g * GROUPED_GEMM_RC), z, 4.0f); + } +}; + +// [qs 96][d]: per sub-block j, 8 grid indices at 8j and an aux dword at 64 + 4j whose 4 x 7 bits +// are sign bytes of ksigns_iq2xs and whose top 4 bits scale d * (0.5 + s) * 0.5. ksigns_iq2xs[i] is +// i with the parity of its 7 bits as bit 7, so the sign bytes are computed, not looked up. +template struct grouped_gemm_esimd_a { + static constexpr bool supported = true; + static constexpr int qk = QK_K; + static constexpr bool whole = false; + // the 256-entry grid, a read-only table in SLM + static constexpr int slm_bytes = 256 * sizeof(uint32_t); + static ESIMD_INLINE void init_slm() { + using namespace grouped_gemm_esimd; +#pragma unroll + for (int i = 0; i < 256; i += 64) { + slm_block_store(i * 4, block_load(iq3xxs_grid + i)); + } + } + struct raw { + grouped_gemm_esimd::simd q; // [row][8 grid indices] + grouped_gemm_esimd::simd aux; + grouped_gemm_esimd::simd d; + }; + struct sb_t { + grouped_gemm_esimd::simd d; + }; + + static constexpr grouped_gemm_esimd::field QS = { 0, 3 * QK_K / 8, 2 }, D = { 3 * QK_K / 8, 2, 0 }; + + static bool ok(const int K, const bool aos) { + return K % qk == 0 && (aos || grouped_gemm_esimd::surface_ok(K / QK_K * 96)); + } + template static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) { + using namespace grouped_gemm_esimd; + simd dh = s.u16(D, ib, 0); + sb.d = simd(dh.template bit_cast_view()); + } + template static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) { + using namespace grouped_gemm_esimd; + const int ib = kb / 8; + r.q = s.template rows<2>(QS, ib, 8 * j); + r.aux = s.template rows<1>(QS, ib, 64 + 4 * j); + r.d = sb.d * (0.5f + simd(r.aux >> 28)) * 0.5f; + } + + template static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) { + using namespace grouped_gemm_esimd; + simd qb = + r.q.template select(g * GROUPED_GEMM_RC * 2).template bit_cast_view(); + simd ge = slm_gather(qb * 4); + simd ax = r.aux.template select(g * GROUPED_GEMM_RC); + simd sg = 0; +#pragma unroll + for (int il = 0; il < 4; ++il) { + simd si = (ax >> (7 * il)) & 127; + sg |= (si | ((cbit(si) & 1) << 7)) << (8 * il); + } + return grid_signed(ge, sg, r.d.template select(g * GROUPED_GEMM_RC)); + } +}; + +// T is the dpas operand type, sycl::half or bfloat16: the type B is packed in. A is decoded in half +// either way and converted for bf16 (see the double rounding note at the conversion). AOS: the weight +// is in the canonical block layout rather than the reorder (SoA) one. A: the grouped_gemm_arch built for. +template struct grouped_gemm_esimd_kernel { + using F = grouped_gemm_esimd_a; + + const char * src0_dd; + size_t expert_stride; + const ggml_sycl_grouped_gemm_tile * tiles; + const T * packed; + float * dst; + int M; + int Npad; + int K; + ggml_sycl_grouped_gemm_rows dst_rows; + + void operator()(sycl::nd_item<1> it) const SYCL_ESIMD_KERNEL { + if constexpr (grouped_gemm_built()) { + run(it); + } + } + + ESIMD_INLINE void run(sycl::nd_item<1> it) const { + using namespace sycl::ext::intel::esimd; + namespace xmx = sycl::ext::intel::esimd::xmx; + constexpr int MR = A::MR; + constexpr int NW = A::NW; + constexpr int RC = GROUPED_GEMM_RC; + constexpr int RG = MR / RC; // dpas row groups of this thread + constexpr int BN = GGML_SYCL_GROUPED_GEMM_BN; + constexpr int NT = BN / NW; + + if constexpr (F::slm_bytes > 0) { + // the format's lookup table; a work-group is one thread, so nothing else writes it + slm_init(); + F::init_slm(); + } + + const int mblocks = M / MR; + const int gid = it.get_global_id(0); + const int t = gid / mblocks; + const int m0 = (gid % mblocks) * MR; + + const ggml_sycl_grouped_gemm_tile tile = tiles[t]; + if (tile.n1 <= tile.n0) { + return; + } + const int b0 = t * BN; // the tile's packed-B columns + + // the slice of this expert + const int bpr = K / F::qk; + const grouped_gemm_esimd::slice sl = { + (const uint8_t *) src0_dd + (size_t) tile.expert * expert_stride, + M, + bpr, + M * bpr, + m0, + AOS, + (int) sizeof(block_q_t) + }; + const uint32_t * bp = (const uint32_t *) packed; + const unsigned bw = (unsigned) Npad * 4; // bytes of one packed B row + const int nstep = K / GROUPED_GEMM_QK; + + simd acc[RG][NT]; + simd total[KS > 1 ? RG : 1][KS > 1 ? NT : 1]; +#pragma unroll + for (int g = 0; g < RG; ++g) { +#pragma unroll + for (int nt = 0; nt < NT; ++nt) { + acc[g][nt] = 0.0f; + if constexpr (KS > 1) { + total[g][nt] = 0.0f; + } + } + } + + // The tile's route map, gathered once up front; the K loop hides its latency, and the epilogue + // below reads it at compile-time indices. + const int ncols = tile.n1 - tile.n0; + simd_mask cm = simd(0, 1) < ncols; + simd i2 = 0, r1 = 0; // r1: the dst row slot, i1 % ne1 + if (dst_rows.map) { + const int32_t * mp = (const int32_t *) (dst_rows.map + tile.n0); + simd mo(0, sizeof(mmid_row_mapping)); + simd i1 = gather(mp, mo, cm); + i2 = gather(mp + 1, mo, cm); + // once per thread in 32 bits: a 64-bit modulo per column is emulated, and at 20 columns a + // tile it cost more than the whole GEMM (7.2 vs 3.1 ms) + r1 = i1 % simd((int32_t) dst_rows.ne1); + } + + // One k step at a time: the format's stored bits for the MR rows, and the 16 k pairs x NW + // columns of packed B per dpas column tile, one 2D block message each (without 2D loads, one + // block load per packed row, cut into the tiles in registers). A second stage in + // flight spilled 7 KB of the 256 registers and ran 1.6x slower; the other threads of the + // XVE hide the load latency instead. + // A stored block's metadata (its sub-block scales) is loaded once per block. With KS > 1 the + // K split of the joint_matrix kernel is reproduced for bit-identical output (a second set of + // accumulators); KS == 1 sums the whole K in one, which is a different f32 rounding order and + // frees the 4 KB of registers the second set holds. + constexpr int KPB = F::qk / GROUPED_GEMM_QK; // k steps per stored block + int split = 0; + int split_end = KS > 1 ? nstep / KS : nstep; + typename F::sb_t sb; + for (int kb = 0; kb < nstep; ++kb) { + const int j = kb % KPB; + if (j == 0) { + F::load_sb(sb, sl, kb / KPB); + } + typename F::raw r; + F::load(r, sb, sl, kb, j); + simd b[NT]; // [16 k pairs][NW columns] each + if constexpr (A::L2D) { +#pragma unroll + for (int nt = 0; nt < NT; ++nt) { + b[nt] = load_2d(bp, bw - 1, K / 2 - 1, bw - 1, b0 + nt * NW, kb * 16); + } + } else { + simd bm; +#pragma unroll + for (int r = 0; r < 16; ++r) { + bm.template select(r * BN) = + block_load(bp + (size_t) (kb * 16 + r) * Npad + b0, overaligned_tag<16>{}); + } + auto bv = bm.template bit_cast_view(); +#pragma unroll + for (int nt = 0; nt < NT; ++nt) { + b[nt] = bv.template select<16, 1, NW, 1>(0, nt * NW); + } + } + simd all; + if constexpr (F::whole) { + all = F::decode_all(r); + } + // one row group of the thread, g a compile-time constant: with a runtime g the larger + // decodes were left rolled + auto row_group = [&](auto gc) SYCL_ESIMD_FUNCTION { + constexpr int g = decltype(gc)::value; + grouped_gemm_esimd::a_halves av; + if constexpr (F::whole) { + // [row][32 k] of the whole tile to this row group's [k half][row][16 k] + auto am = all.template bit_cast_view(); + av.template select(0) = am.template select(g * RC, 0); + av.template select(RC * 16) = am.template select(g * RC, 16); + } else { + av = F::template decode(r); + } +#pragma unroll + for (int h = 0; h < 2; ++h) { + // A of this k16 half: rows x k [16h, 16h+16), row major + simd ah = av.template select(h * RC * 16); + simd at; + if constexpr (std::is_same_v) { + at = ah; + } else { + // rounded twice (f32 to half in the decode, half to bf16 here), so A's rounding + // error is about twice the joint_matrix bf16 stage's, which rounds f32 once + at = simd(ah); + } +#pragma unroll + for (int nt = 0; nt < NT; ++nt) { + simd bh32 = b[nt].template select<8 * NW, 1>(h * 8 * NW); + simd bh = bh32.template bit_cast_view(); + acc[g][nt] = xmx::dpas<8, RC, float, float>(acc[g][nt], bh, at); + } + } + }; + grouped_gemm_esimd::static_for<0, RG>::run(row_group); + if constexpr (KS > 1) { + // the K split's range ends, in k steps; a range's partial joins the total when it closes + if (kb + 1 == split_end) { +#pragma unroll + for (int g = 0; g < RG; ++g) { +#pragma unroll + for (int nt = 0; nt < NT; ++nt) { + total[g][nt] += acc[g][nt]; + acc[g][nt] = 0.0f; + } + } + ++split; + split_end = ((split + 1) * nstep) / KS; + } + } + } + (void) split; + (void) split_end; + + // column c of the tile: MR consecutive floats of one dst row. Unrolled, so c indexes the + // accumulators at compile time; a runtime index sends them all to private memory. +#pragma unroll + for (int c = 0; c < BN; ++c) { + if (c >= ncols) { + continue; + } + simd col; +#pragma unroll + for (int g = 0; g < RG; ++g) { + if constexpr (KS > 1) { + col.template select(g * RC) = total[g][c / NW].template select(c % NW); + } else { + col.template select(g * RC) = acc[g][c / NW].template select(c % NW); + } + } + const int n = tile.n0 + c; + if (dst_rows.map) { + float * out = (float *) (dst_rows.base + (size_t) r1[c] * dst_rows.nb1 + i2[c] * dst_rows.nb2) + m0; + block_store(out, col, element_aligned_tag{}); + } else { + block_store(dst + (size_t) n * M + m0, col, element_aligned_tag{}); + } + } + } + + auto get(syclexp::properties_tag) const { return syclexp::properties{ sycl::ext::intel::experimental::grf_size<256> }; } +}; + +// the weights' surfaces: 2D block loads (SoA with L2D) want 64-byte aligned bases, the dword gathers 4 +static bool grouped_gemm_esimd_weights_ok(const char * w, const size_t expert_stride, const bool l2d) { + const size_t a = l2d ? 64 : 4; + return (uintptr_t) w % a == 0 && expert_stride % a == 0; +} + +// 0 off, 1 on (default), 2 on with the joint_matrix kernel's K-split order reproduced (bit-identical output) +static int ggml_sycl_grouped_gemm_esimd_mode() { + return g_ggml_sycl_mmid_esimd; +} + +bool ggml_sycl_grouped_gemm_esimd_enabled() { +#if defined(GGML_SYCL_SUPPORT_XMX) + return g_ggml_sycl_enable_esimd && ggml_sycl_grouped_gemm_esimd_mode() != 0; +#else + return false; +#endif +} + +template +static void grouped_gemm_esimd_launch(const char * src0_dd, + size_t expert_stride, + const ggml_sycl_grouped_gemm_tile * tiles, + const T * packed, + float * dst, + int M, + int Npad, + int K, + const ggml_sycl_grouped_gemm_rows & dst_rows, + int64_t n_tiles, + dpct::queue_ptr stream) { + const sycl::nd_range<1> range((size_t) n_tiles * (M / A::MR), 1); + if (ggml_sycl_grouped_gemm_esimd_mode() == 2) { + stream->parallel_for(range, grouped_gemm_esimd_kernel{ + src0_dd, expert_stride, tiles, packed, dst, M, Npad, K, dst_rows }); + } else { + stream->parallel_for(range, grouped_gemm_esimd_kernel{ + src0_dd, expert_stride, tiles, packed, dst, M, Npad, K, dst_rows }); + } +} + +template +static bool grouped_gemm_esimd_run(const char * src0_dd, + size_t expert_stride, + const ggml_sycl_grouped_gemm_tile * tiles, + const T * packed, + float * dst, + int M, + int Npad, + int K, + const ggml_sycl_grouped_gemm_rows & dst_rows, + bool reordered, + int64_t n_tiles, + dpct::queue_ptr stream) { + using F = grouped_gemm_esimd_a; + // whole row groups and stored blocks, every K-split range non-empty (also in mode 1, so both + // modes take the same shapes); B is read with 2D block loads, which want 64-byte aligned + // surfaces and pitches, or with 16-byte aligned block loads of whole tiles + if (M % A::MR != 0 || !F::ok(K, !reordered) || K < GROUPED_GEMM_QK * GGML_SYCL_GROUPED_GEMM_KSPLIT || + !grouped_gemm_esimd_weights_ok(src0_dd, expert_stride, reordered && A::L2D) || ((uintptr_t) packed % 64) != 0 || + Npad % 16 != 0) { + return false; + } + if (reordered) { + grouped_gemm_esimd_launch(src0_dd, expert_stride, tiles, packed, dst, M, Npad, K, + dst_rows, n_tiles, stream); + } else { + grouped_gemm_esimd_launch(src0_dd, expert_stride, tiles, packed, dst, M, Npad, K, + dst_rows, n_tiles, stream); + } + return true; +} + +template +bool ggml_sycl_grouped_gemm_esimd(const char * src0_dd, + size_t expert_stride, + const ggml_sycl_grouped_gemm_tile * tiles, + const T * packed, + float * dst, + int M, + int Npad, + int K, + const ggml_sycl_grouped_gemm_rows & dst_rows, + bool reordered, + int64_t n_tiles, + dpct::queue_ptr stream) { + static_assert(grouped_gemm_esimd_a::supported == ggml_sycl_grouped_gemm_esimd_has, + "the header's format list and the formats implemented here must agree"); + static_assert(NW == 8 || NW == 16, "the dpas widths of Xe-HPG and of Xe2 / Xe-HPC"); + if constexpr (NW == 16) { + return grouped_gemm_esimd_run(src0_dd, expert_stride, tiles, packed, dst, M, + Npad, K, dst_rows, reordered, n_tiles, stream); + } else { + return grouped_gemm_esimd_run(src0_dd, expert_stride, tiles, packed, dst, M, + Npad, K, dst_rows, reordered, n_tiles, stream); + } +} + +#define GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, T, NW) \ + template bool ggml_sycl_grouped_gemm_esimd( \ + const char *, size_t, const ggml_sycl_grouped_gemm_tile *, const T *, float *, int, int, int, \ + const ggml_sycl_grouped_gemm_rows &, bool, int64_t, dpct::queue_ptr); +#define GROUPED_GEMM_ESIMD_INSTANCE(block_q_t) \ + GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::half, 16) \ + GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::ext::oneapi::bfloat16, 16) \ + GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::half, 8) \ + GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::ext::oneapi::bfloat16, 8) +GROUPED_GEMM_ESIMD_INSTANCE(block_q8_0) +GROUPED_GEMM_ESIMD_INSTANCE(block_iq4_nl) +GROUPED_GEMM_ESIMD_INSTANCE(block_iq4_xs) +GROUPED_GEMM_ESIMD_INSTANCE(block_iq3_s) +GROUPED_GEMM_ESIMD_INSTANCE(block_iq3_xxs) +GROUPED_GEMM_ESIMD_INSTANCE(block_q3_K) +GROUPED_GEMM_ESIMD_INSTANCE(block_q2_K) +GROUPED_GEMM_ESIMD_INSTANCE(block_q6_K) +GROUPED_GEMM_ESIMD_INSTANCE(block_q5_K) +GROUPED_GEMM_ESIMD_INSTANCE(block_q4_K) diff --git a/ggml/src/ggml-sycl/fused-gemm-esimd.hpp b/ggml/src/ggml-sycl/fused-gemm-esimd.hpp new file mode 100644 index 000000000..029d19480 --- /dev/null +++ b/ggml/src/ggml-sycl/fused-gemm-esimd.hpp @@ -0,0 +1,33 @@ +#ifndef GGML_SYCL_FUSED_GEMM_ESIMD_HPP +#define GGML_SYCL_FUSED_GEMM_ESIMD_HPP + +#include "fused-gemm.hpp" + +// The grouped (MUL_MAT_ID) dequant GEMM, written with ESIMD so the dequantized weights go from +// registers straight into the dpas A operand: no SLM, no barrier. Same inputs and outputs as the +// joint_matrix grouped kernel of the 8 x NW x 16 sub-group-NW combination of T (sycl::half or +// bfloat16): NW = 16 on Xe2 and Xe-HPC, 8 on Xe-HPG (Arc A, ARL-H). The schedule's tiles, B packed as +// [K/2][Npad][2] T, and dst through dst_rows. The weight is in the reorder (SoA) layout if +// `reordered`, else in the canonical block layout. Returns false when the shape is not covered. +template +constexpr bool ggml_sycl_grouped_gemm_esimd_has = std::is_same_v || + std::is_same_v || + std::is_same_v || + std::is_same_v || + std::is_same_v || + std::is_same_v || + std::is_same_v || + std::is_same_v || + std::is_same_v || + std::is_same_v; + +template +bool ggml_sycl_grouped_gemm_esimd(const char * src0_dd, size_t expert_stride, const ggml_sycl_grouped_gemm_tile * tiles, + const T * packed, float * dst, int M, int Npad, int K, + const ggml_sycl_grouped_gemm_rows & dst_rows, bool reordered, int64_t n_tiles, + dpct::queue_ptr stream); + +// GGML_SYCL_MMID_ESIMD: 1 (default) takes the ESIMD kernel above where it applies, 0 never does +bool ggml_sycl_grouped_gemm_esimd_enabled(); + +#endif // GGML_SYCL_FUSED_GEMM_ESIMD_HPP diff --git a/ggml/src/ggml-sycl/fused-gemm.cpp b/ggml/src/ggml-sycl/fused-gemm.cpp index 803a994c0..d0525e3a0 100644 --- a/ggml/src/ggml-sycl/fused-gemm.cpp +++ b/ggml/src/ggml-sycl/fused-gemm.cpp @@ -1,4 +1,5 @@ #include "fused-gemm.hpp" +#include "fused-gemm-esimd.hpp" #include @@ -17,7 +18,6 @@ namespace mx = sycl::ext::oneapi::experimental::matrix; // superblock into steps of this width. The sub-groups of a work-group each walk their own K range // and are summed at the end. static constexpr int FG_BK = QK4_NL; -static constexpr int FG_KSPLIT = 4; // Element traits of one joint_matrix operand type. The A stage and the B pack compute in f32 and // convert once, in registers, when they write the element, so any type costs the same one pass. @@ -113,6 +113,15 @@ using fg_half = sycl::half; using fg_bf16 = sycl::ext::oneapi::bfloat16; using fg_tf32 = mx::precision::tf32; +// The combinations the ESIMD grouped GEMM (fused-gemm-esimd.cpp) stands in for: 8 x SG x 16 with f16 +// or bf16 operands, the shape of one dpas, on sub-group 16 (Xe2, Xe-HPC) or 8 (Xe-HPG). Its B is +// packed and tiled as theirs is. +template static constexpr bool fg_esimd_combo() { + return S::TM == 8 && S::TN == S::SG && S::TK == 16 && (S::SG == 16 || S::SG == 8) && + std::is_same_v && + (std::is_same_v || std::is_same_v); +} + // One bit of GGML_SYCL_XMX_GATHER_SHAPES per combination. Only combinations some device lists in // matrix_combinations are built (appendix of sycl_ext_oneapi_matrix and the runtime's own list). template static void fg_visit_combo(int idx, F && f); @@ -152,29 +161,7 @@ template static void fg_visit_combo(int idx, F && f) { // size, since IGC fails on the other ones. A JIT build keeps them all, but each combination lands in // its own device image (joint_matrix is an optional kernel feature) and only a combination the // device reports is launched, so the runtime never asks IGC for the others. -#if defined(__SYCL_DEVICE_ONLY__) -# if __SYCL_TARGET_INTEL_GPU_ACM_G10__ || __SYCL_TARGET_INTEL_GPU_ACM_G11__ || __SYCL_TARGET_INTEL_GPU_ACM_G12__ || \ - __SYCL_TARGET_INTEL_GPU_ARL_H__ -# define FG_AOT_SG 8 -# elif __SYCL_TARGET_INTEL_GPU_PVC__ || __SYCL_TARGET_INTEL_GPU_PVC_VG__ || __SYCL_TARGET_INTEL_GPU_BMG_G21__ || \ - __SYCL_TARGET_INTEL_GPU_BMG_G31__ || __SYCL_TARGET_INTEL_GPU_LNL_M__ || __SYCL_TARGET_INTEL_GPU_PTL_H__ || \ - __SYCL_TARGET_INTEL_GPU_PTL_U__ || __SYCL_TARGET_INTEL_GPU_WCL__ || __SYCL_TARGET_INTEL_GPU_NVL_S__ || \ - __SYCL_TARGET_INTEL_GPU_NVL_U__ || __SYCL_TARGET_INTEL_GPU_NVL_P__ -# define FG_AOT_SG 16 -# elif __SYCL_TARGET_INTEL_GPU_TGLLP__ || __SYCL_TARGET_INTEL_GPU_RKL__ || __SYCL_TARGET_INTEL_GPU_ADL_S__ || \ - __SYCL_TARGET_INTEL_GPU_ADL_P__ || __SYCL_TARGET_INTEL_GPU_ADL_N__ || __SYCL_TARGET_INTEL_GPU_DG1__ || \ - __SYCL_TARGET_INTEL_GPU_MTL_U__ || __SYCL_TARGET_INTEL_GPU_MTL_H__ -# define FG_AOT_SG 0 // no XMX -# endif -#endif - -template static constexpr bool fg_built() { -#if defined(FG_AOT_SG) - return SG == FG_AOT_SG; -#else - return true; -#endif -} +// (FG_AOT_SG is defined in fused-gemm.hpp, which the ESIMD kernels share) // Upper bound on the tile count when total_rows rows are routed to n_as experts: the worst case gives // each expert one row and fills whole tiles with the rest. The bound depends only on the shape, not @@ -991,6 +978,15 @@ static void grouped_dequant_gemm_launch(const char * src0_dd, const size_t exper float * dst, const int M, const int Npad, const int K, const int64_t n_tiles, const int64_t groups_m, dpct::queue_ptr stream, ggml_sycl_grouped_gemm_rows dst_rows = {}) { + if constexpr (fg_esimd_combo() && ggml_sycl_grouped_gemm_esimd_has) { + // the register-operand ESIMD kernel, either weight layout: no SLM A image (GGML_SYCL_MMID_ESIMD) + if (ggml_sycl_grouped_gemm_esimd_enabled() && + ggml_sycl_grouped_gemm_esimd(src0_dd, expert_stride, tiles_ptr, packed, + dst, M, Npad, K, dst_rows, reordered, + n_tiles, stream)) { + return; + } + } stream->submit([&](sycl::handler & cgh) { sycl::local_accessor tile_a(FG_KSPLIT * S::SG_ROWS * FG_BK, cgh); sycl::local_accessor tile_c(FG_KSPLIT * S::SG_ROWS * S::BN, cgh); @@ -1087,6 +1083,45 @@ template static bool fg_visit_type(ggml_type type, bool reordered, } } +// The ESIMD grouped GEMM loads A straight into its dpas operands, so the joint_matrix tile shape +// means nothing to it; it runs on the fg_esimd_combo of the picked operand type and sub-group size. A +// grouped call whose weight it covers is moved there from any other shape of that type and sub-group +// size, if the device lists it, whatever GGML_SYCL_XMX_GATHER_SHAPES allows: the shape bits choose +// among joint_matrix tiles. +// A call the ESIMD kernel then declines (a shape it does not take) runs that combination's joint_matrix +// kernel. +static int fg_esimd_remap(dpct::queue_ptr stream, int combo, ggml_type src0_type, bool reordered) { + if (combo < 0 || !ggml_sycl_grouped_gemm_esimd_enabled()) { + return combo; + } + bool covered = false; + fg_visit_type(src0_type, reordered, [&](auto tag) { + using T = decltype(tag); + covered = ggml_sycl_grouped_gemm_esimd_has; + }); + if (!covered) { + return combo; + } + int target = -1; + fg_visit_combo(combo, [&](auto s) { + using S = decltype(s); + if constexpr (std::is_same_v) { + for (int idx = 0; idx < FG_N_COMBOS; ++idx) { + fg_visit_combo(idx, [&](auto t) { + using U = decltype(t); + if constexpr (fg_esimd_combo() && U::SG == S::SG && std::is_same_v) { + target = idx; + } + }); + } + } + }); + if (target < 0 || target == combo || !(fg_device_combos(stream->get_device()) & (1 << target))) { + return combo; + } + return target; +} + template static bool fg_fused_run(ggml_type src0_type, bool src0_reordered, const void * src0, const void * src1, ggml_type src1_type, float * dst, int64_t M, int64_t N, int64_t K, int64_t ldd, ggml_sycl_pool & pool, @@ -1201,7 +1236,7 @@ bool ggml_sycl_grouped_dequant_gemm(ggml_type src0_type, bool reordered, const v if (!ggml_sycl_grouped_dequant_gemm_shape_ok(src0_type, M, K, total_rows, n_active)) { return false; } - const int combo = fg_pick_combo(stream, GGML_TYPE_F32, src1_prec); + const int combo = fg_esimd_remap(stream, fg_pick_combo(stream, GGML_TYPE_F32, src1_prec), src0_type, reordered); if (combo < 0) { return false; } @@ -1407,7 +1442,7 @@ bool ggml_sycl_grouped_dequant_gemm_device_supported(ggml_type src0_type, bool r if (grouped_gemm_max_tiles(total_rows, n_as, GGML_SYCL_FG_BN) > INT32_MAX / GGML_SYCL_FG_BN) { return false; } - const int combo = fg_pick_combo(stream, GGML_TYPE_F32, src1_prec); + const int combo = fg_esimd_remap(stream, fg_pick_combo(stream, GGML_TYPE_F32, src1_prec), src0_type, reordered); if (combo < 0) { return false; } @@ -1428,7 +1463,7 @@ bool ggml_sycl_grouped_dequant_gemm_device(ggml_type src0_type, bool reordered, n_tokens, M, K, stream)) { return false; } - const int combo = fg_pick_combo(stream, GGML_TYPE_F32, src1_prec); + const int combo = fg_esimd_remap(stream, fg_pick_combo(stream, GGML_TYPE_F32, src1_prec), src0_type, reordered); bool launched = false; fg_visit_combo(combo, [&](auto s) { launched = fg_grouped_device_run(src0_type, reordered, src0_base, expert_stride, src1, dst, diff --git a/ggml/src/ggml-sycl/fused-gemm.hpp b/ggml/src/ggml-sycl/fused-gemm.hpp index 0e807a2c0..e65e98fc8 100644 --- a/ggml/src/ggml-sycl/fused-gemm.hpp +++ b/ggml/src/ggml-sycl/fused-gemm.hpp @@ -13,7 +13,34 @@ struct ggml_sycl_grouped_gemm_rows { size_t nb2; }; +// AOT: the sub-group size of the XMX combinations of the target, 0 for a target without XMX. +// Undefined in a JIT build, which keeps every combination. +#if defined(__SYCL_DEVICE_ONLY__) +# if __SYCL_TARGET_INTEL_GPU_ACM_G10__ || __SYCL_TARGET_INTEL_GPU_ACM_G11__ || __SYCL_TARGET_INTEL_GPU_ACM_G12__ || \ + __SYCL_TARGET_INTEL_GPU_ARL_H__ +# define FG_AOT_SG 8 +# elif __SYCL_TARGET_INTEL_GPU_PVC__ || __SYCL_TARGET_INTEL_GPU_PVC_VG__ || __SYCL_TARGET_INTEL_GPU_BMG_G21__ || \ + __SYCL_TARGET_INTEL_GPU_BMG_G31__ || __SYCL_TARGET_INTEL_GPU_LNL_M__ || __SYCL_TARGET_INTEL_GPU_PTL_H__ || \ + __SYCL_TARGET_INTEL_GPU_PTL_U__ || __SYCL_TARGET_INTEL_GPU_WCL__ || __SYCL_TARGET_INTEL_GPU_NVL_S__ || \ + __SYCL_TARGET_INTEL_GPU_NVL_U__ || __SYCL_TARGET_INTEL_GPU_NVL_P__ +# define FG_AOT_SG 16 +# elif __SYCL_TARGET_INTEL_GPU_TGLLP__ || __SYCL_TARGET_INTEL_GPU_RKL__ || __SYCL_TARGET_INTEL_GPU_ADL_S__ || \ + __SYCL_TARGET_INTEL_GPU_ADL_P__ || __SYCL_TARGET_INTEL_GPU_ADL_N__ || __SYCL_TARGET_INTEL_GPU_DG1__ || \ + __SYCL_TARGET_INTEL_GPU_MTL_U__ || __SYCL_TARGET_INTEL_GPU_MTL_H__ +# define FG_AOT_SG 0 // no XMX +# endif +#endif + +template static constexpr bool fg_built() { +#if defined(FG_AOT_SG) + return SG == FG_AOT_SG; +#else + return true; +#endif +} + static constexpr int GGML_SYCL_FG_BN = 32; +static constexpr int FG_KSPLIT = 4; static constexpr int64_t GGML_SYCL_MMID_SCHED_MAX_EXPERTS = 4096; // Shape and type gates for the kernels below. Device capability is separate: it needs a queue to ask. diff --git a/ggml/src/ggml-sycl/ggml-sycl.cpp b/ggml/src/ggml-sycl/ggml-sycl.cpp index 9da436ced..5b87fa024 100644 --- a/ggml/src/ggml-sycl/ggml-sycl.cpp +++ b/ggml/src/ggml-sycl/ggml-sycl.cpp @@ -111,6 +111,7 @@ int g_ggml_sycl_memtrace_step = 64; int g_ggml_sycl_enable_vmm = 1; int g_ggml_sycl_enable_fusion = 1; int g_ggml_sycl_enable_esimd = 1; +int g_ggml_sycl_mmid_esimd = 1; int g_ggml_sycl_mmvq_wide = 1; int g_ggml_sycl_enable_xmx_esimd = 1; int g_ggml_sycl_xmx_max_cols = GGML_SYCL_XMX_MAX_COLS_DEFAULT; @@ -444,6 +445,7 @@ static void ggml_check_sycl() try { g_ggml_sycl_enable_vmm = ggml_sycl_get_env("GGML_SYCL_ENABLE_VMM", 1); g_ggml_sycl_enable_fusion = ggml_sycl_get_env("GGML_SYCL_ENABLE_FUSION", 1); g_ggml_sycl_enable_esimd = ggml_sycl_get_env("GGML_SYCL_ENABLE_ESIMD", 1); + g_ggml_sycl_mmid_esimd = ggml_sycl_get_env("GGML_SYCL_MMID_ESIMD", 1); g_ggml_sycl_mmvq_wide = ggml_sycl_get_env("GGML_SYCL_MMVQ_WIDE", 1); g_ggml_sycl_enable_xmx_esimd = ggml_sycl_get_env("GGML_SYCL_ENABLE_XMX_ESIMD", 1); g_ggml_sycl_xmx_max_cols = std::max(0, ggml_sycl_get_env("GGML_SYCL_XMX_MAX_COLS", GGML_SYCL_XMX_MAX_COLS_DEFAULT)); @@ -578,6 +580,7 @@ static void ggml_check_sycl() try { GGML_LOG_INFO(" GGML_SYCL_ENABLE_OPT: %d\n", g_ggml_sycl_enable_optimize); GGML_LOG_INFO(" GGML_SYCL_XMX_GATHER_TYPES: %d\n", g_ggml_sycl_xmx_gather_types); GGML_LOG_INFO(" GGML_SYCL_MMID_SCHED: %d\n", g_ggml_sycl_mmid_sched); + GGML_LOG_INFO(" GGML_SYCL_MMID_ESIMD: %d\n", g_ggml_sycl_mmid_esimd); GGML_LOG_INFO(" GGML_SYCL_XMX_GATHER_SHAPES: %d\n", g_ggml_sycl_xmx_gather_shapes); GGML_LOG_INFO(" GGML_SYCL_DYNAMIC_PRECISION: %s\n", ggml_sycl_dynamic_precision_names[g_ggml_sycl_dynamic_precision]);