sycl: Add device-side MoE scheduler (#30156)

* sycl: Add device-side MoE scheduler

Assisted-by: Codex

* Add width-heuristic override bit and Q2_K and Q3_K types

Assisted-by: Codex

* tests: cover skipped MUL_MAT_ID slots on the device MoE schedule

Assisted-by: Claude Opus 5.5

* sycl: keep graphs on for lazily reordered MUL_MAT and reject BN=64 combos on the device MoE schedule

The graph check disabled SYCL graphs until every MUL_MAT weight with a
reorder extra was reordered, which never happens for some (dense MXFP4,
the fused GLU plain path), so graphs stayed off and the check reran on
every compute. Upstream already reorders those during recording with
the async memory extension, so only MUL_MAT_ID weights are reordered
ahead of recording now.

ggml_sycl_grouped_dequant_gemm_device_supported() accepted any XMX
combo, but the device schedule only runs with BN == GGML_SYCL_FG_BN, so
a 32x64 combo (GGML_SYCL_XMX_GATHER_SHAPES=4 or 8) made the graph check
pass and the compute fall back to the host sync path while recording.

Assisted-by: Claude Opus 5.5

* sycl: keep the ggml_sycl_mul_mat_id_needs_sync name for the graph check

Assisted-by: Claude Opus 5.5

* fix: stop erroneously reordering host-offloaded tensors

With -ncmoe, prompt processing offloads the CPU-resident expert matmuls
to the GPU: the scheduler uploads the experts into a copy in the compute
buffer before every run. init_tensor gave that copy a reorder record, so
the MUL_MAT_ID gather path reordered every offloaded expert tensor on
every ubatch, and the next upload overwrote the result.

Qwen3-30B-A3B Q4_K_M, 1x B60, llama-perplexity -c 4096 -ub 2048
-ncmoe 48, 8 chunks, on master ac8e1694:
  master                        116 s, PPL 7.739
  master XMX_GATHER_TYPES=0      67 s, PPL 7.738
  with this change               67 s, PPL 7.729

Refs ggml-org/llama.cpp#31262

Assisted-by: Claude Opus 5.5

* Remove SYCL-specific test comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* sycl : name the MUL_MAT_ID path predicates after what they decide

ggml_sycl_mul_mat_id_mmvq_ok -> ggml_sycl_mul_mat_id_should_use_mmvq
ggml_sycl_mul_mat_id_mmvq_shape_ok -> ggml_sycl_mul_mat_id_mmvq_eligible
ggml_sycl_mul_mat_id_grouped_shape_ok -> ggml_sycl_mul_mat_id_device_sched_eligible
node_is_device_local -> graph_node_is_in_local_device
ggml_sycl_gg_rows -> ggml_sycl_grouped_gemm_rows
ggml_sycl_gg_tile -> ggml_sycl_grouped_gemm_tile (from #29245, renamed to match)

"eligible" is a necessary condition; the type and width checks follow
elsewhere. "should_use" means the path runs. No functional change.

Assisted-by: Claude Opus 5.5

* sycl : reorder Q3_K MoE experts and load them as vectors in the grouped GEMM

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: cwriter <cwriter@localhost>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit is contained in:
authored and GitHub committed 2026-11-10 09:42:25 +02:00
1 parent ae4893a092
commit 18c5a95e2b
8 files changed
+645 -81

No files matched your search

+18 -1
View File
@@ -11931,6 +11931,23 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
// the batches of a are padded, nb[2] is not a multiple of nb[1]
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F32, GGML_TYPE_F32, 8, 1, 64, {8, 1}, {1, 1}, {0, 1, 2, 3}, 0, 1, false, 16, 16));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F32, 8, 16, 64, {8, 1}, {1, 1}, {0, 1, 2, 3}, 0, 1, false, 16, 16));
// Ragged expert slices, strided routing IDs and broadcast activations.
for (ggml_type type : {GGML_TYPE_Q8_0, GGML_TYPE_IQ4_NL, GGML_TYPE_IQ3_S, GGML_TYPE_Q2_K, GGML_TYPE_Q3_K}) {
for (int n : {3, 17, 33}) {
for (bool broadcast : {false, true}) {
test_cases.emplace_back(new test_mul_mat_id(type, GGML_TYPE_F32, 37, 2, broadcast, 65, n, 256));
}
test_cases.emplace_back(new test_mul_mat_id(type, GGML_TYPE_F32, 37, 2, false, 65, n, 256, 0, false, true));
}
}
// Both experts receive every token
for (ggml_type type : {GGML_TYPE_Q2_K, GGML_TYPE_Q3_K, GGML_TYPE_Q4_K}) {
for (int n : {64, 65}) {
test_cases.emplace_back(new test_mul_mat_id(type, GGML_TYPE_F32, 2, 2, false, 33, n, 512));
}
}
// as is a view whose experts are strided by more rows than it uses
test_cases.emplace_back(new test_mul_mat_id(GGML_TYPE_F32, GGML_TYPE_F32, 4, 2, false, 8, 1, 64, 64));
test_cases.emplace_back(new test_mul_mat_id(GGML_TYPE_F16, GGML_TYPE_F32, 4, 2, false, 8, 16, 64, 64));
@@ -13465,7 +13482,7 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_perf() {
// qwen3-30b-a3b
for (int bs : {1, 4, 8, 32, 64, 128, 256, 512}) {
for (ggml_type type_a : {GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_Q4_0, GGML_TYPE_Q8_0, GGML_TYPE_Q4_K, GGML_TYPE_Q6_K, GGML_TYPE_IQ2_XS, GGML_TYPE_IQ4_XS}) {
for (ggml_type type_a : {GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_Q4_0, GGML_TYPE_Q8_0, GGML_TYPE_Q2_K, GGML_TYPE_Q3_K, GGML_TYPE_Q4_K, GGML_TYPE_Q6_K, GGML_TYPE_IQ2_XS, GGML_TYPE_IQ4_XS}) {
for (ggml_type type_b : {GGML_TYPE_F32}) {
test_cases.emplace_back(new test_mul_mat_id(type_a, type_b, 128, 8, false, 768, bs, 2048));
test_cases.emplace_back(new test_mul_mat_id_fusion(type_a, type_b, 128, 8, false, 768, bs, 2048, 1));