model: add Muse Spark Open (text, vision and audio) (#30616)
* model: add Muse Spark Open (text, vision and audio) Muse Spark Open is a latent space MoE: the residual stream stays at 8192 while the 256 experts live in 4096, with a projection down and up around them, top-8 sigmoid routing with a correction bias. Attention interleaves three sliding layers at a 2048 window with rope for one global layer without rope, with a sigmoid output gate and a weightless QK-norm. Residuals are norm preserving, their gate is folded at conversion time. Vision reuses the Muse Glimmer tower. Audio is a causal convolutional pyramid with no attention and no spectrogram, four stages folding 16, 6, 5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz. The quantizer keeps the latent projections and the attention gate at Q8_0, a little over 1% of the weights, since the whole residual stream passes through them and the low-bit types turned the output into a repeating loop. The model saver writes the post norm epsilon, and the arch is registered in test-llama-archs. Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Muse Spark: drop the redundant forward expand * Muse Spark: drop the redundant forward expand and tighten the comments * Update src/llama-quant.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update src/llama-graph.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
This commit is contained in:
36 files changed
+1290
-9
No files matched your search
+16
-3
@@ -2093,7 +2093,9 @@ ggml_tensor * llm_graph_context::build_moe_ffn(
|
||||
ggml_tensor * up_exps_s,
|
||||
ggml_tensor * gate_exps_s,
|
||||
ggml_tensor * down_exps_s,
|
||||
ggml_tensor * selected_experts_in) const {
|
||||
ggml_tensor * selected_experts_in,
|
||||
ggml_tensor * post_exp_norm,
|
||||
float post_exp_norm_eps) const {
|
||||
return build_moe_ffn(
|
||||
cur,
|
||||
gate_inp, /* gate_inp_b */ nullptr,
|
||||
@@ -2114,7 +2116,9 @@ ggml_tensor * llm_graph_context::build_moe_ffn(
|
||||
up_exps_s,
|
||||
gate_exps_s,
|
||||
down_exps_s,
|
||||
selected_experts_in
|
||||
selected_experts_in,
|
||||
post_exp_norm,
|
||||
post_exp_norm_eps
|
||||
);
|
||||
}
|
||||
|
||||
@@ -2142,7 +2146,9 @@ ggml_tensor * llm_graph_context::build_moe_ffn(
|
||||
ggml_tensor * up_exps_s,
|
||||
ggml_tensor * gate_exps_s,
|
||||
ggml_tensor * down_exps_s,
|
||||
ggml_tensor * selected_experts_in) const {
|
||||
ggml_tensor * selected_experts_in,
|
||||
ggml_tensor * post_exp_norm,
|
||||
float post_exp_norm_eps) const {
|
||||
const int64_t n_embd = cur->ne[0];
|
||||
const int64_t n_tokens = cur->ne[1];
|
||||
const bool weight_before_ffn = arch == LLM_ARCH_LLAMA4; // for llama4, we apply the sigmoid-ed weights before the FFN
|
||||
@@ -2448,6 +2454,13 @@ ggml_tensor * llm_graph_context::build_moe_ffn(
|
||||
cb(experts, "ffn_moe_down_biased", il);
|
||||
}
|
||||
|
||||
// an expert output can be scaled by a norm weight, then normalized, before the router weights apply
|
||||
if (post_exp_norm) {
|
||||
experts = ggml_mul(ctx0, experts, post_exp_norm);
|
||||
experts = ggml_rms_norm(ctx0, experts, post_exp_norm_eps);
|
||||
cb(experts, "ffn_moe_post_exp_norm", il);
|
||||
}
|
||||
|
||||
if (!weight_before_ffn) {
|
||||
experts = ggml_mul(ctx0, experts, weights);
|
||||
cb(experts, "ffn_moe_weighted", il);
|
||||
|
||||
Reference in new issue
Block a user