* hparams: add per-layer n_ff_exp/n_expert_used arrays with scalar-or-array loading
G1/G2 infrastructure for variable-per-layer expert FFN size and top-k routing
(required for Puzzle-75B which has 5 distinct n_ff_exp values and 7 top-k values
across its 40 MoE layers).
Design: rename scalar members to _impl suffix (following existing convention),
add LLAMA_MAX_LAYERS arrays, add n_ff_exp(il)/n_expert_used(il) accessors with
scalar fallback. No new GGUF keys: reuses existing expert_feed_forward_length and
expert_used_count keys via get_key_or_arr (scalar -> broadcast, array -> per-layer).
- llama-hparams.h: n_ff_exp -> n_ff_exp_impl, n_expert_used -> n_expert_used_impl;
add n_ff_exp_arr / n_expert_used_arr arrays; add per-layer accessor declarations.
- llama-hparams.cpp: implement n_ff_exp(il) and n_expert_used(il); out-of-range
il returns impl safely (shared code, no abort).
- llama-model.cpp: central n_expert_used load changed to get_key_or_arr; derive
impl as max-of-array for validations and backward compat; zero both new arrays;
HunyuanVL override also zeroes n_expert_used_arr.
- llama-graph.cpp: aggregation loop in build_moe_ffn uses hparams.n_expert_used(il)
so per-layer top-k bounds the ggml_view loop correctly.
- All other files: mechanical rename hparams.n_{ff_exp,expert_used} -> *_impl.
Scalar arches are unaffected (broadcast fills all array slots with the single value).
(cherry picked from commit 269a81e03d66e1c353e1a203a0c03a03eb2b1a4e)
* nemotron-h: use per-layer n_ff_exp(il) and n_expert_used(il) at MoE call-sites
Load n_ff_exp via get_key_or_arr into hparams.n_ff_exp_arr in load_arch_hparams;
derive impl as max for existing uniform GGUFs.
In load_arch_tensors, compute n_ff_exp_i = hparams.n_ff_exp(i) with fallback to
n_ff(i)/n_expert_used(i) for GGUFs that omit expert_feed_forward_length.
In build_ffn_layer, pass hparams.n_expert_used(il) to build_moe_ffn so per-layer
top-k is used for expert routing selection.
All other nemotron-h behaviour (mamba2, attention, shared-exp, latent projection,
routed_scaling_factor, expert_weights_norm, sigmoid gating) is unchanged.
(cherry picked from commit b1878a101793cd4e59868ac72635a86ea694987c)
* arch/*.cpp + gguf-py: mechanical rename n_ff_exp->n_ff_exp_impl, n_expert_used->n_expert_used_impl
All non-nemotron arch files continue using the scalar impl member directly.
Behaviour is identical: the impl value is the broadcast value from the GGUF scalar.
gguf_writer: add_expert_feed_forward_length and add_expert_used_count now accept
int | Sequence[int], mirroring add_feed_forward_length, so converters can write
per-layer arrays with the same existing GGUF keys.
(cherry picked from commit 8f009f54bea5ef9a6a354123bd25e9d5ea2d5e03)
* convert: support NemotronHPuzzleForCausalLM (per-block MoE config)
Parse block_configs/mtp_block_configs into per-layer arrays (scalar-or-array
keys), append the MTP [attention, moe] sub-blocks as blk.88/blk.89 with
nextn tensors, accept the backbone.* prefix, and register the arch.
Also fix a pre-existing undeclared _experts attribute on NemotronHModel.
(cherry picked from commit d1a592f278336e78457454eb6c96bca917135f10)
* nemotron-h: distinguish Nemotron 3 Puzzle (75B.A9B) from Super (120B.A12B)
Both have 88 layers; the per-layer expert_used_count array (heterogeneous
for Puzzle, broadcast-uniform for Super) is the discriminator.
(cherry picked from commit f824e09dc812169589cf5662c92d149a4c18c30a)
* convert: accept the official Puzzle BF16 checkpoint's tensor naming
The officially distributed BF16 checkpoint (NVIDIA-Nemotron-Labs-3-Puzzle-
75B-A9B-BF16) names the trunk model.* (model.layers.*, model.embeddings,
model.norm_f) where the original release used the NemotronH-style
backbone.*, and spells the router bias e_score_correction_bias instead of
e_score_correction.bias. Normalize both at the top of
NemotronHPuzzleModel.modify_tensors so either checkpoint converts; every
tensor name in the official index (42683 keys, MTP head included) resolves
through the tensor map after normalization.
(cherry picked from commit 189b67fc2c9d50970416c94b3317a6e7baa49b03)
* laguna: use n_ff_exp_impl for the uniform-MoE FFN size
Laguna landed after this branch was cut and reads hparams.n_ff_exp as a
scalar. This series turns it into a per-layer array with an n_ff_exp(il)
accessor, so the three scalar reads no longer compile. Laguna is a
uniform MoE, so point them at the scalar fallback n_ff_exp_impl, same as
deepseek2/qwen3moe/gemma4 in this series. No behaviour change.
(cherry picked from commit dbedc9e19c50dca0acdfb402362e2707bee424ae)
* arch: extend the n_ff_exp/n_expert_used rename to archs added upstream
kimi-k3, dflash, bailingmoe3, deepseek4, granite-swa and the nemotron-h MTP
block still referenced the scalar fields by their old names. n_ff_exp and
n_expert_used are accessors now, so those reads no longer compile; point the
non-per-layer archs at the _impl scalars and use the indexed form where the
call site is per-layer.
* convert: keep Puzzle opted out of the NemotronH MTP export path
#26725 added MTP export to NemotronHModel, keyed on num_nextn_predict_layers.
Puzzle's config carries that key, but NemotronHPuzzleModel bypasses
NemotronHModel.__init__ (its per-block config needs a different setup), so
_mtp_bid was never assigned and modify_tensors raised AttributeError on any
mtp.* tensor. Puzzle's head is also laid out by mtp_block_configs, not the
mtp.layers.* form the base maps.
Set _mtp_bid to None, drop mtp.* in filter_tensors, and declare
supports_mtp_export = False so --mtp / --no-mtp fail at the CLI.
* llama: replace n_ff_exp/n_expert_used scalars with per-layer accessors
Follow-up to review feedback: the previous revision kept the scalar
hparams fields alongside the new per-layer arrays, which duplicated
state that get_key_or_arr already handles by broadcasting a scalar
value over every layer.
Drop both scalars and expose n_ff_exp(il) / n_expert_used(il) built
exactly like the existing n_head_kv(il) and n_ff(il) accessors: they
index the array and GGML_ABORT out of range, with il defaulting to 0
so genuinely uniform call sites stay a plain n_ff_exp().
Arch loaders now read both keys through get_key_or_arr over
n_layer_all, and the n_expert_used validation checks the maximum
across layers instead of a single field.
* llama: restore per-key required flags on the expert hparam reads
The scalar-to-array conversion passed required=false at every call site,
which silently made mandatory keys optional. Each read now carries the
same required flag it had before the conversion.
* DeepseekV4: fix rollback with multi-seq
* fix model loading
* make pending rollback single use
* only clear cache for seq_id for full load
* add assert for compress ratio
* make graph topology static
* pass true instead of flags in clear_compressed
* cont : clean-up + TODOs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
build_attn with the llm_graph_input_attn_k_iswa input was using the cached K
tensor itself as V. Create V as a view of K (the first v_cur->ne[0] elements
of each row), like the other K-only build_attn overloads.
The deepseek4 MTP call site now passes the kv tensor as v_cur.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* llama : make all KQ masks (except the lightning indexer one) f16 if FA is used and remove zero attention bias in DeepSeek V4
* llama : remove dead code that repeats unified raw_k cache for each stream in DeepSeek V4 - no longer needed as raw_k is always non-unified.
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* convert: add dsv4 conversion
* add basic setup
* add llm_graph_input_dsv4
* add save-load state
* add sinkhorn eps - correction by @fairydreaming
* add rope fix
* cleanup dead code
* fix bugs
* support pro model: added by @fairydreaming
* remove redundant V cache
* Chat template
* remove debugging leftovers
* Add mechanism for inlining templates based on architecture
* s/deepseek-v4-flash/deepseek4/g
* s/deepseek-v4-flash/deepseek4/g continued
* enable graph reuse
* enable FA
* fix test llama archs
* rename
* compatibility with antirez ds4 GGUFs
* simplified set_gguf_parameters() by calling super class method, replaced moe.score_func with expert_gating_func.
* reserve worst-case kv-cache
* revert max split inputs
* address review comments
* add padding to enable FA
* pad only the final value of plan.n_kv to 256
* remove built-in cpp chat template
* cont: remove cpp built-in template
* rm outdated test
* replace ggml_view_3d() with ggml_reshape_3d()
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* only support n_seq=1 for now
* remove unused var
* cont: remove unused var
* use scale bias
* use correct ptr for can_reuse
* remove gen-chat-inline-templates.py
* simplify graph reuse
* cont: cleanup
* remove unused inputs
* enable partial checkpointing
* add correct shape for kq_mask + set llama_model_n_swa to 0 for dsv4
* precompute source_idx + add comment about dummy write
* support multi-seq
* remove restored_trim_pos
* use split_equal when possible
* fix indent
* address review comments
* use LLM_KV
* fix ci
---------
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Xuan-Son Nguyen