model: add Kimi-K3.1-Pro text model (#31779)
* model: add Kimi-K3.1-Pro text model
K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.
Indexer:
- the selection is an additive mask over the whole cache, not a gather of
the top-k rows, so it reuses the existing sparse flash attention path;
off that path the mask stays correct but still costs a kq over the cache
- the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
because dsa_config declares no eps of its own
Graph and cache:
- new arch with four indexer tensors per DSA layer, and a hybrid memory
that keeps a K-only MLA cache for those layers, recurrent state for the
KDA layers, and the indexer key cache
- the graph node budget shares the K3 entry
Conversion:
- KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
- dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
plus the indexer LayerNorm eps; the layer list is 1-indexed like
linear_attn_config's, and the converter asserts that it covers exactly
the full-attention layers
- the routed experts stay MXFP4 and are repacked as in K3, not dequantized
- the vision tower and mm_projector are skipped: text only for now
- the arch key follows the checkpoint's own model_type ("kimi_k31") and the
loader registers KimiK31ForConditionalGeneration; the release name carries
a -Pro suffix that nothing in the config or the tensor names uses
Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.
Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.
* converter: avoid clutter in the k3 base
* narrow comment to match model
* update metal fusion expectations
* restore
* sync with PR 31733 refactoring
This commit is contained in:
1 parent
669204ee2b
commit
bcb0152876
15 files changed
+919
-35
No files matched your search
+17
-2
@@ -372,6 +372,8 @@ static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params
|
||||
return new llama_model_kimi_linear(params);
|
||||
case LLM_ARCH_KIMI_K3:
|
||||
return new llama_model_kimi_k3(params);
|
||||
case LLM_ARCH_KIMI_K31:
|
||||
return new llama_model_kimi_k31(params);
|
||||
case LLM_ARCH_GLM5_NEXT:
|
||||
return new llama_model_glm5_next(params);
|
||||
case LLM_ARCH_STEP35:
|
||||
@@ -3282,8 +3284,20 @@ llama_memory_i * llama_model::create_memory(const llama_memory_params & params,
|
||||
// only the sparse-attention architectures use llama_memory_hybrid_idx
|
||||
// a null filter_idx means the GGUF has no indexer tensors
|
||||
llama_memory_hybrid::layer_filter_cb filter_idx = nullptr;
|
||||
const bool needs_mem_idx = (arch == LLM_ARCH_QWEN4EXP);
|
||||
if (arch == LLM_ARCH_FALCON_H1) {
|
||||
const bool needs_mem_idx = (arch == LLM_ARCH_QWEN4EXP || arch == LLM_ARCH_KIMI_K31);
|
||||
if (arch == LLM_ARCH_KIMI_K31) {
|
||||
// DSA layers keep a K-only MLA cache plus a per-token indexer key cache,
|
||||
// the KDA layers stay recurrent
|
||||
filter_attn = [&](uint32_t il) {
|
||||
return il < hparams.n_layer() && !hparams.is_recr(il);
|
||||
};
|
||||
filter_recr = [&](uint32_t il) {
|
||||
return il < hparams.n_layer() && hparams.is_recr(il);
|
||||
};
|
||||
filter_idx = [&](uint32_t il) {
|
||||
return il < hparams.n_layer() && !hparams.is_recr(il) && hparams.is_indexer_full(il);
|
||||
};
|
||||
} else if (arch == LLM_ARCH_FALCON_H1) {
|
||||
filter_attn = [&](uint32_t) { return true; };
|
||||
filter_recr = [&](uint32_t) { return true; };
|
||||
} else if (arch == LLM_ARCH_NEMOTRON_H || arch == LLM_ARCH_NEMOTRON_H_MOE) {
|
||||
@@ -3660,6 +3674,7 @@ llama_rope_type llama_model_rope_type(const llama_model * model) {
|
||||
case LLM_ARCH_METIS16:
|
||||
case LLM_ARCH_KIMI_LINEAR:
|
||||
case LLM_ARCH_KIMI_K3:
|
||||
case LLM_ARCH_KIMI_K31:
|
||||
case LLM_ARCH_GLM5_NEXT:
|
||||
case LLM_ARCH_STARCODER:
|
||||
return LLAMA_ROPE_TYPE_NONE;
|
||||
|
||||
Reference in new issue
Block a user