Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

model : BailingMoE3 Support (#26608)

Browse Source 
* Adding support for bailingmoe3

* Adds speculative decoding support

* Make BailingMoE3 safe gate metadata optional

* bailingmoe3: apply trained SwiGLU clamps

* common: fix Bailing V3 tool argument parsing

* llama-model-saver, instantiate float vector metadata writer

* bailingmoe3: support Q-LoRA (Ling-3.0-tiny)

Ling-3.0-flash sets q_lora_rank: None and projects Q directly, so the current
implementation loads a single ATTN_Q tensor. Ling-3.0-tiny sets q_lora_rank: 256
and routes Q through a LoRA bottleneck instead:

    q_a_proj -> q_a_layernorm -> q_b_proj

Conversion therefore failed with:

    ValueError: Can not map tensor 'model.layers.3.attention.q_a_layernorm.weight'

Add the missing path, mirroring the existing deepseek2 MLA implementation:

  * constants.py       - add ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM to BAILINGMOE3
  * tensor_mapping.py  - map model.layers.{bid}.attention.q_{a,b}_proj and
                         q_a_layernorm
  * conversion         - emit attention.q_lora_rank when the config has it
  * bailingmoe3.cpp    - read n_lora_q; create the Q-LoRA tensors and build Q
                         through the bottleneck when q_lora_rank > 0

Everything is gated on q_lora_rank > 0. Ling-3.0-flash's config has no
q_lora_rank, the converter only emits the key when present, hparams.n_lora_q
defaults to 0, and get_key(..., required=false) leaves the target untouched when
the key is absent - so flash keeps taking the existing direct-Q branch.

The LoRA path produces the same shape as the direct projection, so the
nope/rope split, RoPE application and wk_b absorption downstream are unchanged.

* small mtp change

* bailingmoe3: support separate MTP GGUF and Q-LoRA MTP

* gguf: remove duplicate add_kda_gate_lower_bound definition

---------

Co-authored-by: bloomer <bloomer@booper.brushtail.me>
Co-authored-by: Dyluhn <dylanranejohnston1@gmail.com>
This commit is contained in:
Toby
2026-08-17 09:49:49 +02:00
committed by GitHub
co-authored by bloomer Dyluhn
parent 4197155add
commit 3733366720
18 changed files with 938 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/llama-model-saver.cpp
+6 -2
View File
@@ -121,6 +121,7 @@ void llama_model_saver::add_kv(const enum llm_kv key, const Container & value, c
}
// instantiate for external usage:
template void llama_model_saver::add_kv<std::vector<uint32_t>>(const enum llm_kv, const std::vector<uint32_t> &, const bool);
template void llama_model_saver::add_kv<std::vector<float>>(const enum llm_kv, const std::vector<float> &, const bool);
void llama_model_saver::add_kv(const enum llm_kv key, const std::vector<std::string> & value) {
std::vector<const char *> tmp(value.size());
@@ -216,8 +217,10 @@ void llama_model_saver::add_kv_from_model() {
add_kv(LLM_KV_EXPERT_LATENT_LENGTH, hparams.n_expert_latent);
add_kv(LLM_KV_EXPERT_SHARED_FEED_FORWARD_LENGTH, hparams.n_ff_shexp);
add_kv(LLM_KV_EXPERT_CHUNK_FEED_FORWARD_LENGTH, hparams.n_ff_chexp);
add_kv(LLM_KV_SWIGLU_CLAMP_EXP, hparams.swiglu_clamp_exp);
add_kv(LLM_KV_SWIGLU_CLAMP_SHEXP, hparams.swiglu_clamp_shexp);
add_kv(LLM_KV_SWIGLU_CLAMP_EXP, std::vector<float>(
hparams.swiglu_clamp_exp.begin(), hparams.swiglu_clamp_exp.begin() + hparams.n_layer_all));
add_kv(LLM_KV_SWIGLU_CLAMP_SHEXP, std::vector<float>(
hparams.swiglu_clamp_shexp.begin(), hparams.swiglu_clamp_shexp.begin() + hparams.n_layer_all));
add_kv(LLM_KV_USE_PARALLEL_RESIDUAL, hparams.use_par_res);
// add_kv(LLM_KV_TENSOR_DATA_LAYOUT, ???);
add_kv(LLM_KV_EXPERT_COUNT, hparams.n_expert);
@@ -320,6 +323,7 @@ void llama_model_saver::add_kv_from_model() {
add_kv(LLM_KV_SSM_DT_B_C_RMS, hparams.ssm_dt_b_c_rms);
add_kv(LLM_KV_KDA_HEAD_DIM, hparams.n_embd_head_kda);
add_kv(LLM_KV_KDA_SAFE_GATE, hparams.kda_safe_gate);
add_kv(LLM_KV_KDA_GATE_LOWER_BOUND, hparams.kda_gate_lower_bound);
add_kv(LLM_KV_WKV_HEAD_SIZE, hparams.wkv_head_size);