* metal : support scale-bias routing norm in the TOPK_MOE fusion The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias (sum + 1e-20) instead of the lower clamp used by the other models. Add norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a unit scale factor at the clamp position, pass an additive bias in the kernel args, and apply it as max(wt_sum, clamp) + bias so both variants share one kernel. Cover the new chains in test-backend-ops. Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB) * metal : fuse sqrt(softplus) MoE routing with a selection bias Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+ ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1 routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks selection-biased keys while the gathered weights and the norm stay unbiased, matching the reference graph. Dispatch hooks at the softplus node, cover the norm-bias combination in test-backend-ops and record the new deepseek4/deepseek41 fusion counters. Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB) * metal : fuse sigmoid MoE routing with a selection bias The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank sigmoid(logits) + exp_probs_b with an optional weights-norm, but only the softmax and sqrt(softplus) routing chains were fusable. - add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT + GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm tail; the selection-bias ADD is required, same as the sqrt(softplus) chains - generalize the gating function constant from a sqrt(softplus) bool to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and CUDA kernels - anchor the unary dispatch hook at sigmoid nodes too - test-backend-ops: 4 sigmoid routing cases with and without a weights scale - re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next, deepseek2/32, kimi, minimax, ...) now fuse their routing, and muse-spark prefill drops one MOE_REDUCE fire because the reordered graph allocates the routing weights inside the reduce output range, which the dst/input overlap check correctly declines Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB) * cont : remove low-signal comments * ocd : valign ggml_metal_fusions
Fusion baselines
Per-device baselines for test-fusion, one CSV per backend (e.g. MTL.csv). Rows are
arch,moe,mode,label,count. Regenerate a CSV whenever fusion patterns change.
Update a baseline
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON # enable the target backend
cmake --build build --config Release --target test-llama-archs --target test-fusion -j
rm -rf build-ci-models && mkdir -p build-ci-models
./build/bin/test-llama-archs -o build-ci-models
./build/bin/test-fusion --models build-ci-models --device MTL0 --record MTL.csv
Validate
./build/bin/test-fusion --models build-ci-models --device MTL0 --check MTL.csv
Non-zero exit means a row differs from the baseline. Use --model FILE to run a single
architecture. --check also reports baseline rows that the run no longer produces, so a fusion
that stops matching fails the run instead of silently dropping out of the CSV.