llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert

Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.

Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.

Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.

The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.

Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.

Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
This commit is contained in:
LumpiastyandClaude Opus 5.5 committed 2026-10-24 18:21:42 +02:00
1 parent 3f03b17724
commit c72cb424e1
1 file changed
+23 -9
+23 -9
View File
@@ -2803,16 +2803,30 @@ bool llama_context::sched_copy_experts(ggml_backend_t backend, const ggml_tensor
const size_t expert_size = src->nb[2];
if (ids != st.ids || (int64_t) st.used.size() != n_expert) {
st.ids_data.resize(ggml_nbytes(ids)/sizeof(int32_t));
ggml_backend_tensor_get_async(backend, ids, st.ids_data.data(), 0, ggml_nbytes(ids));
ggml_backend_synchronize(backend);
// Reading the ids back forces a full pipeline flush, since the ids are
// produced on the same device we are about to upload to. Once the batch
// draws enough experts, essentially all of them come back used and the
// readback has bought nothing, so skip it and copy the whole tensor as one
// range. Copying an expert that no id selects cannot change the result:
// mul_mat_id only reads the rows the ids point at, so this stays exact.
// With the MoE cache on, cached experts are still copied device-to-device.
const bool all_experts_used = ggml_nelements(ids) >= 4*n_expert;
st.used.assign(n_expert, false);
for (int64_t i1 = 0; i1 < ids->ne[1]; i1++) {
for (int64_t i0 = 0; i0 < ids->ne[0]; i0++) {
const int32_t id = st.ids_data[i1*ids->nb[1]/sizeof(int32_t) + i0*ids->nb[0]/sizeof(int32_t)];
GGML_ASSERT(id >= 0 && id < n_expert);
st.used[id] = true;
if (all_experts_used) {
st.ids_data.clear();
st.used.assign(n_expert, true);
} else {
st.ids_data.resize(ggml_nbytes(ids)/sizeof(int32_t));
ggml_backend_tensor_get_async(backend, ids, st.ids_data.data(), 0, ggml_nbytes(ids));
ggml_backend_synchronize(backend);
st.used.assign(n_expert, false);
for (int64_t i1 = 0; i1 < ids->ne[1]; i1++) {
for (int64_t i0 = 0; i0 < ids->ne[0]; i0++) {
const int32_t id = st.ids_data[i1*ids->nb[1]/sizeof(int32_t) + i0*ids->nb[0]/sizeof(int32_t)];
GGML_ASSERT(id >= 0 && id < n_expert);
st.used[id] = true;
}
}
}