* model: add Kimi-K3.1-Pro text model
K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.
Indexer:
- the selection is an additive mask over the whole cache, not a gather of
the top-k rows, so it reuses the existing sparse flash attention path;
off that path the mask stays correct but still costs a kq over the cache
- the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
because dsa_config declares no eps of its own
Graph and cache:
- new arch with four indexer tensors per DSA layer, and a hybrid memory
that keeps a K-only MLA cache for those layers, recurrent state for the
KDA layers, and the indexer key cache
- the graph node budget shares the K3 entry
Conversion:
- KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
- dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
plus the indexer LayerNorm eps; the layer list is 1-indexed like
linear_attn_config's, and the converter asserts that it covers exactly
the full-attention layers
- the routed experts stay MXFP4 and are repacked as in K3, not dequantized
- the vision tower and mm_projector are skipped: text only for now
- the arch key follows the checkpoint's own model_type ("kimi_k31") and the
loader registers KimiK31ForConditionalGeneration; the release name carries
a -Pro suffix that nothing in the config or the tensor names uses
Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.
Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.
* converter: avoid clutter in the k3 base
* narrow comment to match model
* update metal fusion expectations
* restore
* sync with PR 31733 refactoring
* feat : added ricursiveai ripple support
* fix : mandatiry SWA, scale through helper, comments back
* fix : metal
* model : make ricursive RoPE/NoPE explicit and validate the layer pattern
Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.
Assisted-by: Claude Opus 5.5
* model : make ricursive swiglu_clamp per-layer
Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.
Assisted-by: Claude Opus 5.5
* model : use the ricursive per-tensor scales and RoPE parameters
Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.
Assisted-by: Claude Opus 5.5
* convert : drop the ricursive chat_template.jinja override
The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.
Assisted-by: Claude Opus 5.5
* fix : remove NVFP4 checkpoint quantization block
* match #31733 refactor
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* meta: support GLM-5.3 next sm tensor
* glm5-next : cont q_absorbed for -sm tensor with non-unified KV
* meta : keep GGML_OP_ARANGE in ggml.h order
* glm5-next : split MLA heads in pairs for -sm tensor
The MLA cache has a single KV head, so the heads on a device are its GQA
ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3
devices the 16 heads of the test model were split 5/5/6 and it aborted.
* meta : keep the split for views inside one slab of the split axis
With a non-unified KV cache, build_attn_mha splits q into streams with a
view. For a permuted q, e.g. the absorbed query of glm5-next, that view
aborted with "view of permuted tensor not implemented". The split is now
kept if every other dim of the source and the view stays inside one slab
of the split axis. The per-device strides are only rescaled above it.
From tetherto/qvac-fabric-llm.cpp#353.
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
* glm5-next : drop the q_absorbed cont
The meta backend can now split the per-stream view of the permuted q, so
the cont is not needed for -sm tensor with a non-unified KV cache. With
the cont, the q of flash attention was not contiguous, and when FA ran on
the CPU (WebGPU) the meta get_tensor aborted.
---------
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
* WIP OCP FP8 E4M3 support
* Fix python type-checker
* Materialize input_scales also for FP8 when converting
* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4
1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
weights are reshaped
* Allow remaining FP8 tensors to be dequantized during conversion
* Allow tensor-strategy for FP8-only compressed-tensor ckpts
* Fix and test gguf-py behavior for FP8_E4M3
* Disallow QKV-fusion for FP8 temporarily
* Fix conversion of fused qkvz & FP8 for Qwen3-Next
* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws
* Emit warnings when not fusing FP8 during conversion
* More FP8 conversion fixes
* Add supported for strided copies to/from FP8 in CPU
* remove unneeded scratch buffer from FP8 cpu cpy/dup path
* add explanatory comments and TODOs
* Add dedicated encoder and decoder tests for FP8
* Whitelist ggml_op_get_rows in metal instead of blacklisting
required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list
https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512
* Fix more conversion bugs
* Defer weight materialization during conversion
* Blacklist fp8 in metal backend instead of white-list
* Adjust indents
* Fix lint + execute tests
* Add fast CPU kernel for F8_E4M3
* Conversion fixes
* Fix remaining gaps
* Enable tied weights with shared scales
* Make mandatory-F32 rules apply to FP8 during conversion
* Fix python type checks
https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512
* Fix kaleidai unit-test failures
https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150
* Relax numeric precision constraints for pre-check
* Leave fast-FP8 CPU on RISC-V for follow-up
We already exclude arm from the fast-path
* Allow output scale tensors to be located in any gguf shard
* Move test_conversion_fp8.py test execution to behavior-check.yaml
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it
* Install all requirements also on Windows for behavior-check
Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512
* Unmap lazy tensors before clean-up in Windows
Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512
* LF line endings
https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512
* Extract FP8 tests from llama-arch to its own test-fixture
See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221
* Update gguf-py/gguf/scripts/gguf_convert_endian.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* gguf-py : add Limite arch
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* convert : add Limite support
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* llama : add Limite support
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
---------
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B
Assisted-by: Claude Opus 5.5
* convert : keep the speech task templates out of the d1 GGUFs
Assisted-by: Claude Opus 5.5
* small refactoring: no_batching
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* tests : set the SWA rope base of the LFM3 test model
Assisted-by: Claude Opus 5.5
* llama : enable recurrent state rollback for LFM3
Assisted-by: Claude Opus 5.5
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>
* gguf-py : align the lfm3vl mapping comments
Assisted-by: Claude Opus 5.5
* llama : fix CRLF line endings from the review suggestions
Assisted-by: Claude Opus 5.5
* tests : add the LFM3 Metal fusion baseline
Assisted-by: Claude Opus 5.5
* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Every caller passes n_pad = 1 and the only two users clamp it to 256.
Drop the parameter, the members and the trivially true kv_size assert.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* model : add DeepSeek-V4.1-Flash (deepseek41)
* deepseek41 : remove MXFP8, convert the Engram table to BF16
* deepseek41 : compute the compressed-KV read layers once in the cache
* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le
* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather
* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer
* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle
* deepseek41 : keep il last in build_hc_mixes
* deepseek41 : sort the top-k rows before the sparse gather
* tests : add deepseek41 to the Metal fusion baseline
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Add Flint model (FlintForCausalLM). Assisted by Codex
* cleanup
* one more
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : add SenseNova U1 understanding support
Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.
- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
tensors and concatenating temporal and spatial Q/K normalization
weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.
Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.
* conversion : treat one-expert SenseNova U1 configs as dense
* mtmd : add SenseNova U1 vision support
Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.
The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.
* mtmd : fix SenseNova U1 vision token layout
* model : clean up SenseNova U1 RoPE metadata
Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.
Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].
* mtmd : simplify SenseNova U1 projector integration
Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.
Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.
* mtmd : fix SenseNova U1 image positions and attention
* fix spacing
* mtmd : restore generic test input chunks
* mtmd : restore SenseNova U1 integration after upstream merge
* Update conversion/sensenova_u1.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : remove SenseNova U1 BF16 activation casts
Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.
Assisted-by: Codex
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : add H2O-Lightning-4B decision model
Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b
(Qwen3.5-4B with a merged decision LoRA).
None of the existing types match this model. It answers with " A"
rather than "A", uses up to 255 labels, and was trained on Python
json.dumps text, so floats like 12.0 must render as 12.0.
The converter writes the decision metadata and the systemone template
from the model's serve_config.json and refuses a config that doesn't
match. Tests skip unless the tiny test model is available locally.
Assisted-by: Claude Opus 5.5
* rm redundant code
* put labels to gguf
---------
Co-authored-by: Jonathan C. McKinney <jon.mckinney@h2o.ai>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add Kolibri-1 (Aleph-Alpha) support
Kolibri-1 is a bilingual German/English MoE reasoning model with 78B
parameters and 3.46B active per token: 50 layers, 384 routed experts plus
one shared expert.
Attention interleaves four sliding layers at a 513 window with one full
layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE.
GQA runs 48 query heads against 4 KV heads at head_dim 128 with a
QK-RMSNorm, and every block carries four RMSNorms, so the attention and the
MLP output are normalized before they rejoin the residual stream. Routing
picks the experts from logits + expert_bias and weights them with the
sigmoid of the unbiased logits, without renorm.
The converter reuses AfmoeModel for the per-expert checkpoint layout and
the expert selection bias, and writes the shared expert and the per layer
sliding window pattern. The released FP8 checkpoint needs no special code,
it goes through the generic weight_scale_inv path. Three tensor names are
added to the mapping (post_attn_norm, post_ffn_norm,
moe.router.expert_bias) and the 128k vocabulary registers the qwen2
pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA
and the sliding window pattern.
Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash)
* Update src/models/kolibri1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Add fusion records for kolibri1
* Update src/models/kolibri1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model: add Muse Spark Open (text, vision and audio)
Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.
Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.
The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Muse Spark: drop the redundant forward expand
* Muse Spark: drop the redundant forward expand and tighten the comments
* Update src/llama-quant.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-graph.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Add Support to maion-coder arch
* iAdd Support to support Draft models
* Address comments
* Remove maion context shift function
---------
Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
When every device reports zero free memory the split points are
normalized by a zero sum, the NaN split sends layers to a device index
past the end and loading aborts with vector::_M_range_check.
* llama : do not map foreign ranges into a backend buffer
A buffer created from a host pointer pins everything that it wraps. When
several contexts share the same file (lazy tensors, MoE tensors kept on the
CPU with -cmoe), the single buffer that spans the first and the last tensor
of a context also wraps the ranges owned by the other contexts, pinning data
that the backend never reads.
For models with per-layer token embeddings this pins the whole lazy tensor
into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the
point of reading it lazily.
Instead, compute the file ranges of each context and subtract the ranges
owned by the other contexts, then create one buffer per range. Weights that
are mapped in more than one context (e.g. token_embd.weight) keep being part
of the range of every context that uses them.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : enable prefetch on macOS
The prefetch hint of the tensors marked as prefetchable is a no-op on macOS,
so their pages are faulted in one by one while the graph runs. madvise
(MADV_WILLNEED) behaves as expected here, so add the platform to the guard.
With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read
from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to
735 / 749 / 788 t/s for pp512 / pp2048 / pp4096.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : use a plain buffer list per context
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* graph: skip the LM head for pooled embeddings
A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.
The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.
* graph: drop the redundant t_embd expand
Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.
* graph: add build_logits() for the LM head output
Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.
* graph: call build_logits() before build_sampling()
Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.
* nit
* tests : consolidate the context state tests into test-llama-context
test-save-load-state, test-recurrent-state-rollback and
test-state-restore-fragmented check the same object from three places:
the KV cache and the recurrent state of a llama_context, saved, loaded,
copied, rolled back. Each had its own main, its own loop over the
models, its own copy of the token and the comparison helpers, and a run
of all three loaded the same models several times.
They are one binary now, and the case is the unit of the test:
tests/llama-context/test-case.h the interface a case implements
tests/llama-context/cases-kv.cpp the cases of the KV cache
tests/llama-context/cases-rs.cpp the cases of the recurrent state
tests/llama-context/test-util.cpp tokens, generation, comparison
tests/llama-context/test-runner.cpp the models of a case, on threads
tests/llama-context/test-log.cpp the log file of a model
A run is a table: a row per model, a column per case, a cell holds the
verdict of that pair. The models load on several threads (-j, --threads,
min(cores, 4) by default), the table is printed when it is complete and
the log of a model that failed is printed after it, in the order of the
rows. A case returns a verdict and the reason for it, and the runner is
the only one who reports - the test does not use tests/testing.h, a run
over 130 models is a matrix of verdicts and not a tree of subtests.
Staging no longer travels through the environment: llama_model_set_attn_rot
replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one
thread used to change for the models the other threads were loading, and
llama_memory_get_buffers is added for the cases that copy the buffers of
a memory. common_log_set_stdo keeps the console quiet while the table is
being built.
ggml_aligned_malloc allocates with posix_memalign when the thread
sanitizer is on. On macOS it allocates with the Mach VM, which the
sanitizer does not intercept, so it never sees a page range change hands
and reports the first write into recycled memory as a race - 30 of them
in this test, and every one of them false.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : run single-threaded by default
* cont : try to build the tests on all platforms
* cont : trim down some body fat
* cont : fix
* cont : fix the fix
* model : use exact GELU for ModernBERT encoders
Assisted-by: Codex
* model : keep tanh GELU aliases on ggml_geglu
Assisted-by: Claude Opus 5.5
* model : map gelu_python to ggml_geglu_erf
Assisted-by: Claude Opus 5.5
* model : support classifier_activation for rerankers
Assisted-by: Claude Opus 5.5
* model : map classifier gelu to gelu_erf and accept tanh
Assisted-by: Claude Opus 5.5
* model : default act_cls to tanh, ModernBERT falls back to gelu_erf
Assisted-by: Claude Opus 5.5
* model : add LiquidAI/d1-omni-600M decision model
Assisted-by: Claude Opus 5.5
* mtmd : keep conformer GLU sigmoid on CUDA
Assisted-by: Claude Opus 5.5
* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal
Assisted-by: Claude Opus 5.5
* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files
Assisted-by: Claude Opus 5.5
* llama: share the nextn tensor flags between models
Follow-up of the TODO in glm5-next: move the trunk-only and MTP-only
detection that each model copied into a nextn_flags helper of
llama_model_base. It probes the first trunk layer and the first NextN
layer, and adds TENSOR_SKIP when MTP is not loaded. qwen4exp probes
hc_attn_norm since it has no attn_norm.
deepseek4, nemotron-h, qwen35, qwen35moe, qwen3next and qwen4exp now
also accept a trunk-only file, like the other models.
* llama: avoid capturing structured bindings in the nextn flags
Lambdas that capture structured bindings need C++20, and GCC 15
rejects them under -Werror, so the models read the trunk and MTP
flags into plain variables.
* model: K2 Horizon gguf conversion code
* model: loading hparams and tensors in k2-horizon.cpp
* model: K2 Horizon compute graph
* model: K2 Horizon compute graph adjustment and registering tokenizers
* model: K2 Horizon chat template and accomodate safetensors naming
* unicode : add the K2-Horizon pre-tokenizer splitter
The K2-Horizon regex had no arm in unicode_regex_split_custom and fell through to the
general std::regex fallback, which fails two ways.
On MSVC std::regex rejects \p{...}, so no K2-Horizon GGUF loads on Windows at all:
llama-quantize, llama-imatrix and llama-perplexity all abort with
regex_error(error_escape) before a token is produced.
Where the fallback does compile it is still wrong. unicode_regex_split collapses each
codepoint to a single byte naming its Unicode category before matching, and U+200C/U+200D
are category Control, which has no entry in k_ucat_cpt, so both become the 0xD0 fallback
byte. The literal and alternatives in K2's regex can then never match and
every ZWNJ or ZWJ ends a letter run.
The splitter is the existing llama3 one with a single rule widened, since K2's regex
differs from llama3's only in that a letter run also takes marks, ZWNJ and ZWJ.
tests/test-unicode.cpp gains a case for this: it fails before the change with
[Amy] [ZWNJ khaham] and passes after with the run intact.
* tests: expand K2 Horizon unicode splitter coverage
* unicode: handle K2 Horizon case folding and empty input
Assisted-by: Codex
* jinja : support sequence indices in selectattr and rejectattr
Assisted-by: Codex
* model : add K2 Horizon dense and MoVA support
Includes the K2 Horizon implementation from ifm-ai/llama.cpp with converter, tensor-parallel and model save/reload fixes.
Assisted-by: Codex
* chat : support K2 Horizon reasoning and tool calls
Assisted-by: Codex
* conversion: remove obsolete K2 Aurora alias
Assisted-by: Codex
* k2-horizon: enforce response schemas and load YaRN betas
Constrain final JSON after reasoning, accept flexible JSON tool envelopes,
enforce XML dialects, and handle repeated or alternate thinking markers.
Load YaRN beta metadata instead of retaining the default values.
Add schema, streaming, continuation, and model reload regressions. Validate
CUDA and CPU builds and 0.9B, 4B, and MoVA conversation/tool round trips.
Assisted-by: Codex
* renaming template fixture
* adressing cisc follows ups
* desloppify the parser / adress aldehir comments
* clean test-chat
* remove fallback : model trained mostly on high anyway
* fix k2 attn_v_exp tn splitting and metal fusion baseline
* k2-horizon : forward expand views before sums
* k2-horizon: copy embds before group norm to fix TP
* disable tesnor parallelism
---------
Co-authored-by: Ryandito Diandaru <ryandito.diandaru@mbzuai.ac.ae>
Co-authored-by: WestWaters <mario.papaleo2013@gmail.com>
Co-authored-by: Natani L. Mayday <71436458+TaskPuppyNatani@users.noreply.github.com>
Co-authored-by: West <100190545+WestWaters@users.noreply.github.com>
Co-authored-by: aaryamonvikram <aaryamonvikram@gmail.com>
Co-authored-by: aaryamonvikram <96529820+aaryamonvikram@users.noreply.github.com>
* llama: llama_prefetch_rows
* llama: support row prefetch on Windows
Apply the Windows port contributed by @praneshgo unchanged.
Source: https://github.com/ggml-org/llama.cpp/pull/29599#issuecomment-5887721014
* avoid exposing llama-mmap in model code, route via llama-impl
* add windows check, only prefetch in lazy mode
* cont : clean-up
* cont : fix build
* cont : clarify padding token for gemma4
---------
Co-authored-by: Pranesh Gonegandla <pranesh.iitp@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx
* Add initial MTP support
* Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements.
* Review driven changes, remove env vars, protect tensors
* Strip MTP for initial PR
* Clean up after mtp strip
* Clean up after mtp strip
* Update speculative.cpp
* Update llama-context.h
* Clean up after mtp strip
* Fix tokenizer ignore merges
* Improve quantization protection selection
* Refactor mhc helpers, graph base
* Lint Fixes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Skip glm5-next in model saver, fix CRLF
* Skip glm5-next in sweep
* Remove T4 fallback
* Review cleanup
* Review suggestions
* Defer separate MTP gguf handling to MTP PR, drop filter
* Repad n_head_kv
* kpool init apply
* Order by descending score
* Drop guard
* read kpool from hparams, clarify kpool cache flags, remove kpool_build_state(nullptr)
* Add glm5-next support to model saver and add arch test fixture
* Review cleanup
* Kpool pooled caching clarify
* Add multi stream support
* Finish Rebase
* Sparse FA fir DSA prefill
* Const
* Update llama-model.cpp to fix rebase error
* gguf-py : merge tensor map entries for HC tensors
* model : use build_gdn_l2_norm in GLM5_NEXT implementation
* chore : remove trailing whitespace
* model : use new OP precision setting API in GLM5_NEXT implementation
* mtmd : use ggml_swiglu_clamp in GLM5V and apply the image token limit
The two clamps around swiglu_split are what ggml_swiglu_clamp already does,
so the clamp bounds collapse back to one value. GLM5V also never called
set_limit_image_tokens(), so --image-max-tokens had no effect.
Assisted-by: Claude Opus 5
(cherry picked from commit 46d18e12d422be4cc04a70e4a9a9e0168bb3d5b7)
* llama : keep the GLM5-Next k-pool layout across ubatches
The layout was rebuilt from a full cell scan on every ubatch. Pools are fixed
by the positions relative to the sequence's first one, so the layout now lives
on the memory and a ubatch only appends to it.
A sequence edit no longer stales every pooled key either, only the ones at or
after the edited position, which makes a tail seq_rm free. The pooling subgraph
is built unconditionally so the graph shape no longer changes every kpool
tokens, and the pool axis is folded into rows before soft_max, which otherwise
exceeds the CUDA gridDim.y limit past n_kv 262144.
Assisted-by: Claude Opus 5
(cherry picked from commit 5d1c40b93e17fddbf73b785efe43e0d02ccb3977)
* model : write the GLM5-Next recurrent rollback checkpoints
The conv state and the delta net state were only written to the live row, so a
rollback restored whatever the checkpoint rows happened to hold. Take the same
route as kimi-k3: build_recurrent_attn for the state, and write all K_rs conv
groups. That also drops a state view that assumed contiguous rows.
Enroll the arch in test-recurrent-state-rollback, which catches this under its
garbage-filled cache pass.
Assisted-by: Claude Opus 5
(cherry picked from commit 5ace37e86d5d448e83ef5dde5632c748185b18cd)
* llama: fix PR #27773 test-save-load-state restore failure
Clear the attention and indexer cache data after a failed hybrid state restore so restored NaNs cannot affect a later sequence.
Assisted-by: Codex
* llama: fix PR #27773 gpu-rocm graph reallocation
Reserve the full GLM5-Next pool capacity and dirty pool count. The gpu-rocm Test step aborts when n_new grows while the graph node count stays fixed; CUDA, Vulkan, Metal, and WebGPU checks report the same error.
Assisted-by: Codex
* llama : fix GLM5-Next k-pool layout staleness after edits and shared teardown
Two defects in the cross-ubatch k-pool layout added by the k-pool commit:
1. Wrong results. An edited sequence only rebuilt its pool layout when its cell
count changed, so if the first ubatch after an edit added back exactly as many
cells as were removed, the stale position-to-cell list survived. With a unified
cache and more than one sequence, where another sequence takes the freed cells,
the reused layout points at the wrong cells (CPU: large logit drift, CUDA: NaN).
Rebuild whenever the sequence is stale, not only on a size mismatch.
2. Slowdown. "shared" mode was assumed to end only with an edit that forces a
rebuild, but sharing also ends when the other sequence is removed. The survivor
kept shared = true, pinning cache_safe off and re-pooling every pool on every
ubatch (server trigger: n>1 completions with -kvu, via the seq_cp in
copy_state_to). In seq_rm, if the layout has shared cells, stale every sequence
so one rebuild re-derives sharing and cache_safe returns to 1.
Assisted-by: Claude Opus 5
* llama : fix build_attn_mha stream stride for non-contiguous q
build_attn_mha split the batch into streams with a stream stride of
q->nb[3]/n_stream. That only equals one stream's span, (ne[2]/n_stream)*nb[2],
when q is contiguous. GLM5-Next is nope-only, so it does not concat a rope part
and passes the permuted q_absorbed straight in, where nb[3] != ne[2]*nb[2]; the
stride was then n_head times too large and every stream s >= 1 read another
head's queries. Split-KV (-np N without --kv-unified) multi-stream prefill was
wrong for every stream past the first. Unified KV and decode were unaffected
(n_stream == 1, and decode takes the gather path). Other MLA models concat rope
so q is contiguous and the computed value is unchanged for them.
Compute the stride from the token dimension, which is identical for a
contiguous q.
Assisted-by: Claude Opus 5
* llama : re-derive GLM5-Next k-pool sharing on state_read/state_drop
The shared-cell teardown added to seq_rm (stale every sequence when the layout
has shared cells, so a survivor does not keep shared = true and pin cache_safe
off) was missing from the other paths that can free shared cells: state_read
and state_drop staled only the one sequence. Apply the same re-derivation there
and correct the comment that claimed sharing ends only via an edit or seq_rm.
Assisted-by: Claude Opus 5
* quant : drop duplicate GLM5-Next hc_ filter
The hc_ name filter was listed twice in the GLM5_NEXT protection block.
Assisted-by: Claude Opus 5
* glm5-next: scope K-pool cache access to indexed operations
* glm5-next: keep K-pool access in hybrid index memory
* glm5-next: keep mHC graph builders model-local
* glm5-next: mark only touched pools per ubatch
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
* model : support classifier_pooling for ModernBERT rerankers
Assisted-by: Claude Opus 5.5
* model : read classifier pooling type in load_hparams
Write classifier.pooling_type from _try_set_pooling_type whenever the
config has classifier_pooling, and read it in
llama_model_base::load_hparams. ModernBERT falls back to mean when it
is unspecified.
Assisted-by: Claude Opus 5.5
* conversion : only accept cls and mean for classifier_pooling
Assisted-by: Claude Opus 5.5
* model : rename classifier_pooling_type to pooling_type_cls
Assisted-by: Claude Opus 5.5