From c72cb424e115a430efc92a4f7347cd966af9e088 Mon Sep 17 00:00:00 2001 From: Lumpiasty Date: Fri, 9 Oct 2026 00:30:07 +0200 Subject: [PATCH] llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert Offloaded expert weights are uploaded per split, and to decide which experts to upload the copy callback reads the routing ids back to the host. The ids are produced on the same device we are about to upload to, so the readback forces a full pipeline flush - 47 of them per eval on a 48-layer MoE. Once the batch draws enough experts the readback stops telling us anything. At 2048 tokens x 10 experts over 256 experts every expert comes back used, so the used set is all true and the copy is a single whole-tensor range anyway. Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert used. Uploading an expert that no id selects cannot change the result, since mul_mat_id only reads the rows the ids point at, so this stays exact. With the MoE cache on, the all-used set still goes through moe_cache->copy_experts, which copies the cached experts device-to-device from the host slot maps and uploads the rest - still exact, and no readback is needed for it. Decode is unaffected: n_ids there is the number of experts per token, far below the threshold. The small-batch MoE cache slot path has its own readback and is not touched. The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor reuses the all-true set, which is what the threshold would decide again. Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch uses every expert"), whose target block in ggml_backend_sched_compute_splits was removed upstream (#29943) in favour of this copy callback. Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth 32k, +0.9% at 16k, tg32 +3%. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ --- src/llama-context.cpp | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/src/llama-context.cpp b/src/llama-context.cpp index ca84fc405..aa4d9931b 100644 --- a/src/llama-context.cpp +++ b/src/llama-context.cpp @@ -2803,16 +2803,30 @@ bool llama_context::sched_copy_experts(ggml_backend_t backend, const ggml_tensor const size_t expert_size = src->nb[2]; if (ids != st.ids || (int64_t) st.used.size() != n_expert) { - st.ids_data.resize(ggml_nbytes(ids)/sizeof(int32_t)); - ggml_backend_tensor_get_async(backend, ids, st.ids_data.data(), 0, ggml_nbytes(ids)); - ggml_backend_synchronize(backend); + // Reading the ids back forces a full pipeline flush, since the ids are + // produced on the same device we are about to upload to. Once the batch + // draws enough experts, essentially all of them come back used and the + // readback has bought nothing, so skip it and copy the whole tensor as one + // range. Copying an expert that no id selects cannot change the result: + // mul_mat_id only reads the rows the ids point at, so this stays exact. + // With the MoE cache on, cached experts are still copied device-to-device. + const bool all_experts_used = ggml_nelements(ids) >= 4*n_expert; - st.used.assign(n_expert, false); - for (int64_t i1 = 0; i1 < ids->ne[1]; i1++) { - for (int64_t i0 = 0; i0 < ids->ne[0]; i0++) { - const int32_t id = st.ids_data[i1*ids->nb[1]/sizeof(int32_t) + i0*ids->nb[0]/sizeof(int32_t)]; - GGML_ASSERT(id >= 0 && id < n_expert); - st.used[id] = true; + if (all_experts_used) { + st.ids_data.clear(); + st.used.assign(n_expert, true); + } else { + st.ids_data.resize(ggml_nbytes(ids)/sizeof(int32_t)); + ggml_backend_tensor_get_async(backend, ids, st.ids_data.data(), 0, ggml_nbytes(ids)); + ggml_backend_synchronize(backend); + + st.used.assign(n_expert, false); + for (int64_t i1 = 0; i1 < ids->ne[1]; i1++) { + for (int64_t i0 = 0; i0 < ids->ne[0]; i0++) { + const int32_t id = st.ids_data[i1*ids->nb[1]/sizeof(int32_t) + i0*ids->nb[0]/sizeof(int32_t)]; + GGML_ASSERT(id >= 0 && id < n_expert); + st.used[id] = true; + } } }