Multi-column on Intel XMX matrix engines refactored (#29864)
* Multi-column on Intel XMX matrix engines refactored Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0). * Fixed regression bug real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4 * Removed GGML_TYPE_Q8_0 activation * rewritten xmx_mul_mat 64-byte alignment and 16-byte pitch check on the activations * Implemented fused gate/up kernel One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types. * Removed the temporary XMX_NOGLU environment switch * Q2_K min-sum optimization Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group * Added dispatch check and GRF sweep tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%). Q6_K is unchanged. MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361. Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without. * Renamed variables * Added 2 environment variables GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it. * Update esimd.hpp 1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces. 2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once. The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
This commit is contained in:
1 parent
d77256999f
commit
d81d0d8fb9
8 files changed
+1143
-5
No files matched your search
@@ -12246,6 +12246,18 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
|
||||
true, 16, 8, b, false, true, false));
|
||||
}
|
||||
|
||||
// Gate and up on the SYCL XMX kernels: rows long enough for 2D block loads, a last tile of 8 rows, and
|
||||
// columns on both sides of the 8 and 16 column tiles
|
||||
for (ggml_type type : {GGML_TYPE_Q2_K, GGML_TYPE_Q3_K, GGML_TYPE_Q4_K, GGML_TYPE_Q5_K, GGML_TYPE_Q6_K,
|
||||
GGML_TYPE_Q8_0}) {
|
||||
for (ggml_glu_op glu_op : {GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU}) {
|
||||
for (int64_t m_batch : {1, 2, 3, 5, 8, 9, 16, 17}) {
|
||||
test_cases.emplace_back(new test_mul_mat_vec_fusion(type, glu_op, m_batch, 1000, 1024,
|
||||
false, 16, 8, false, false, true, false, { 1, 1 }));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (int64_t k : {96, 896, 1024, 2880}) {
|
||||
for (ggml_glu_op glu_op : {GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU}) {
|
||||
test_cases.emplace_back(new test_mul_mat_vec_fusion(GGML_TYPE_Q8_0, glu_op, 1, 32, k,
|
||||
|
||||
Reference in new issue
Block a user