Multi-column on Intel XMX matrix engines refactored (#29864)

* Multi-column on Intel XMX matrix engines refactored

Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).

* Fixed regression bug

real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4

* Removed GGML_TYPE_Q8_0 activation

* rewritten xmx_mul_mat

64-byte alignment and 16-byte pitch check on the activations

* Implemented fused gate/up kernel

One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.

* Removed the temporary XMX_NOGLU environment switch

* Q2_K min-sum optimization

Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group

* Added dispatch check and GRF sweep

tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.

* Renamed variables

* Added 2 environment variables

GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16):  the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.

* Update esimd.hpp

1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.

2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.

The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
This commit is contained in:
budiony authored and GitHub committed 2026-10-22 21:39:01 +08:00
1 parent d77256999f
commit d81d0d8fb9
8 files changed
+1143 -5

No files matched your search

+12
View File
@@ -12246,6 +12246,18 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
true, 16, 8, b, false, true, false));
}
// Gate and up on the SYCL XMX kernels: rows long enough for 2D block loads, a last tile of 8 rows, and
// columns on both sides of the 8 and 16 column tiles
for (ggml_type type : {GGML_TYPE_Q2_K, GGML_TYPE_Q3_K, GGML_TYPE_Q4_K, GGML_TYPE_Q5_K, GGML_TYPE_Q6_K,
GGML_TYPE_Q8_0}) {
for (ggml_glu_op glu_op : {GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU}) {
for (int64_t m_batch : {1, 2, 3, 5, 8, 9, 16, 17}) {
test_cases.emplace_back(new test_mul_mat_vec_fusion(type, glu_op, m_batch, 1000, 1024,
false, 16, 8, false, false, true, false, { 1, 1 }));
}
}
}
for (int64_t k : {96, 896, 1024, 2880}) {
for (ggml_glu_op glu_op : {GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU}) {
test_cases.emplace_back(new test_mul_mat_vec_fusion(GGML_TYPE_Q8_0, glu_op, 1, 32, k,