Offloaded expert weights are uploaded per split, and to decide which experts to
upload the scheduler reads the routing ids back to the host. The ids are produced
on the same device we are about to upload to, so the readback forces a full
pipeline flush - 47 of them per eval on a 48-layer MoE.
Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
bitset is all ones and the copy is a single whole-tensor range anyway.
Skip the readback when the batch guarantees that. Uploading an expert that no id
selects cannot change the result, since mul_mat_id only reads the rows the ids
point at, so this stays exact. Decode is unaffected: n_ids there is the number of
experts per token, far below the threshold, so it keeps the bitset path.
Measured on an RX 580 (Vulkan, Polaris) running Laguna-S-2.1 118B IQ2_M with all
experts host-resident: pp2048 +4.7% at depth 32k, +0.9% at 16k, tg32 +3%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LumpiastyandClaude Opus 5
dbc449afd0