* mtmd: support hibiki realtime duplex
* fix and clean up
* fix long session quality
* fix ci
* tests : add the moshi fusion baseline for Metal
The Metal fusion check found 5 RMS_NORM+MUL fusions in the new moshi
arch and no baseline for them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* mtmd: drama3 voice cloning
The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3:
- conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2
- fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj,
builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2)
- drama3_gen tower: teacher forcing when the caller gives the codes of the next frame
- drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the
reference frames go through the backbone as codec rows and through the tower with their codes
- mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state
Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX
* fix correctness
* clean up
* rm redundant code
* fish_s2: rm unused import
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* jinja: impl scope-chain key lookup
* add test for call_statement lambda
* chat: add tts_dict for kokoro G2P template
* more testing
* value_float_t range check
* no json-in-meta, add llama_model_meta_arr_str/_n
* mtmd: add kokoro tts support
* detailed guide about G2P and text norm port
* refine G2P
* clean up
* migrate batch api
* nits
* Added support for Qwen3-TTS ICL/reference mode
Assisted-By: Claude Opus 5.5
* fix correctness
* attach transcription to bitmap
* nits
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* mtmd: support YuE2-3B
* improve speed
* add CFG
* fix streaming
* add cfg control from api
* check ctx size
* fix n_predict
* add flow_steps config
* nits
* fix batch api
* nits
* adapt
* mtmd: yue2 samples from its own saved logits
The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.
* mtmd: let the yue2 snake chain fuse
The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* convert : add Inkling2 text GGUF conversion
Assisted-by: Codex
* llama : implement Inkling2 text inference
Assisted-by: Codex
* tests : cover Inkling2 text support
Assisted-by: Codex
* conversion : dequantize packed Inkling2 experts in row chunks
Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.
Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.
Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.
Assisted-by: Codex
* convert : add Inkling2 vision mmproj support
Assisted-by: Codex
* mtmd : implement Inkling2 still-image encoder
Assisted-by: Codex
* inkling2 : scope image loading
Assisted-by: Codex
* inkling2 : enable flash attention with folded bias and sinks
Assisted-by: Codex
* cuda : route per-head flash attention masks through vec
Assisted-by: Codex
* mtmd : support Inkling2 temporal frame means
Assisted-by: Codex
* style : restore shared image comment
Assisted-by: Codex
* tests : enlarge Inkling2 context fixture
Assisted-by: Codex
* inkling2 : load per-layer expert feed-forward widths
Assisted-by: Codex
* tests : add Inkling2 Metal fusion baselines
Assisted-by: Codex
* inkling2 : group and align registry entries
Assisted-by: Fable
* convert : regenerate the Inkling2 tokenizer hash with the update script
Assisted-by: Fable
* conversion : revert the print_registered_models change
Assisted-by: Fable
* conversion : sort the Inkling2 model map entries
Assisted-by: Fable
* convert : map Inkling2 vision checkpoint names in the tensor table
Assisted-by: Fable
* convert : map Inkling2 text checkpoint names in the tensor table
Assisted-by: Fable
* tests : document the Inkling2 packed expert conversion test
Assisted-by: Fable
* tests : cover Inkling2 checkpoint and transformers tensor names
Assisted-by: Fable
* llama : place Inkling2 fields and list entries by file convention
Assisted-by: Fable
* gguf-py : align Inkling2 tensor mapping comments
Assisted-by: Fable
* mtmd : place Inkling2 entries by file convention
Assisted-by: Fable
* tests : place Inkling2 cases after the existing ones
Assisted-by: Fable
* cuda : index per-head flash attention masks in the tile and mma kernels
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : project the relative bias before the gather, drop arch branches
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* convert : require Inkling2 hparams, add inkling2 pre-tokenizer
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* mtmd : cap Inkling2 image size, drop video frame averaging
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : fix CI build, Meta device assert and Windows conversion test
Assisted-by: Claude
* tests : use the default sliding window for the Inkling2 test model
Assisted-by: Claude
* simplify mtmd part
* give _v arch for vision
* remove alpha channel handling, it should be a follow up
* explicit rule about audio + vision models
* add audio support
* rm audio
* rm gguf mtmd rope base
* address review comment
* nits
* do not normalize embd from mtmd
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* mtmd: require the MLP projector and patch embedding tensors
The loader marks the patch embedding and the LLaVA projector tensors
as optional, while the graphs and n_mmproj_embd() dereference them
unconditionally. A GGUF without them crashes on a null tensor during
the warmup.
Load the patch embedding as required for the vision encoders that
start with it, which is all of them except GEMMA3NV, DEEPSEEKOCR and
DEEPSEEKOCR2. Load mm.0 and mm.2 as required for MLP, and mm.0, mm.1,
mm.3 and mm.4 for the Yi-type MLP_NORM variant, selected by mm.3.
The graph no longer tests mm.2 and the yasa2 assert on the patch
embedding is covered by the loader.
* mtmd: keep the Yi-type llava TODO
The Yi-type variant is still detected by the presence of mm.3, the
TODO on that detection stays.
* mtmd: support gemma 4 input audio stream
* use 480ms step
* server: enable realtime embeddings only for audio generation
The realtime slot forced embeddings on, which marks every row of a
batch as an output. Only a model that generates audio reserves
n_batch outputs, so a chat model with an audio stream (Gemma 4)
aborted on its first batch of audio rows. The hidden state is only
read by the audio generation pipeline.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls
Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.
* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends
Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.
* tests: cover GGML_PREC_BF16 in the flash attention tests
The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.
* llama: request GGML_PREC_BF16 for the KQ multiplication
* cont : add TODO in clip.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* mtmd: support MageVL-Cowork model
* mtmd: keep magevl rope_freqs at model level and check its shape
The rope factors are shared by every layer, so they live in clip_model
instead of a pointer copied into each clip_layer. The loader now checks
that v.rope_freqs is F32 with d_head elements and that d_head splits
into the 4:6:6 pairs the graph expects.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* mtmd : add Nemotron 3.5 Transcribe support
Assisted-by: Claude
* convert : fix vocab of draft models for Nemotron-H targets
Assisted-by: Claude
* convert : reject non-BF16 Transcribe checkpoints
Assisted-by: Claude
* mtmd : make speaker FIFO predictions local
Assisted-by: Claude
* mtmd : normalize the mel in encode_windows
Assisted-by: Claude
* convert : map the generated speaker kernel
Assisted-by: Claude
* mtmd : validate Nemotron Transcribe mmproj shapes
Assisted-by: Claude
* common: bound control vector layer indices by the model layer count
A direction tensor layer index is parsed from the tensor name with no
upper bound, and the buffer size and write offset are computed in int.
A large index overflows both, so the accumulation writes before the
start of the buffer.
Pass the model layer count to the loader, reject indices above it like
the existing zero index check, and compute the sizes in size_t.
* mtmd: fill M-RoPE positions by whole merge tiles
The position loops of the Qwen-VL, GLM, PaddleOCR and Qwen2.5-VL
encoders hardcode a 2x2 merge inside a stride of n_merge, so a patch
grid that is not a multiple of the merge writes past the positions
buffer. Iterate over whole merge tiles with n_merge inner steps, which
bounds the writes by the patch count and gives the same positions on
aligned grids.
PaddleOCR also requires a merge aligned image at graph build, as the
Qwen-VL family already does.
* mtmd : add LFM3-VL and LFM3-Omni video input
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* mtmd : reserve the soundtrack command to avoid a GCC 11 warning
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* convert : read the LFM3-Audio config.json as released
A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).
Assisted-by: Claude Opus 5.5
* mtmd : disable flash attention for the LFM3-Audio depthformer
Assisted-by: Claude Opus 5.5
* tts : add a voice option for LFM3-Audio and LFM3-Omni
The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.
Assisted-by: Claude Opus 5.5
* clean up template handling
* convert : detect LFM3-Audio and LFM3-Omni by model_type
Assisted-by: Claude Opus 5.5
* move banned tokens --> logits bias --> mtmd_helper
* leave mtmd-cli unchanged
* no more template_asr_system
* left over
* llama : keep the LFM3-Audio encoder convs and codec unquantized
Assisted-by: Claude Opus 5.5
* common, mtmd : shorten comments
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* tests : set the SWA rope base of the LFM3 test model
Assisted-by: Claude Opus 5.5
* llama : enable recurrent state rollback for LFM3
Assisted-by: Claude Opus 5.5
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>
* gguf-py : align the lfm3vl mapping comments
Assisted-by: Claude Opus 5.5
* llama : fix CRLF line endings from the review suggestions
Assisted-by: Claude Opus 5.5
* tests : add the LFM3 Metal fusion baseline
Assisted-by: Claude Opus 5.5
* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* model : add DeepSeek-V4.1-Flash (deepseek41)
* deepseek41 : remove MXFP8, convert the Engram table to BF16
* deepseek41 : compute the compressed-KV read layers once in the cache
* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le
* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather
* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer
* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle
* deepseek41 : keep il last in build_hc_mixes
* deepseek41 : sort the top-k rows before the sparse gather
* tests : add deepseek41 to the Metal fusion baseline
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Add Soprano-80M and Soprano-1.1-80M (https://huggingface.co/ekwek/Soprano-1.1-80M)
as an audio generation model: a Qwen3 backbone plus a Vocos decoder that runs
as a gen-only mmproj.
- convert: detect Soprano checkpoints and export the backbone as qwen3 and the
decoder.pth Vocos weights as the mmproj
- vocab: add the soprano pre-tokenizer (lowercase, whitespace collapse,
per-digit split, [UNK] fallback)
- mtmd: add the Vocos graph and ISTFT, allow loading a gen-only mmproj for
Soprano and include the gen-audio context in mtmd_get_memory_usage
- mtmd-helper: add the Soprano pipeline; the decoder runs over the whole
utterance, so stream mode sends the audio in one chunk after flush()
- tts: document Soprano usage
Assisted-by: Codex
Assisted-by: Claude Opus 5.5
* model : add SenseNova U1 understanding support
Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.
- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
tensors and concatenating temporal and spatial Q/K normalization
weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.
Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.
* conversion : treat one-expert SenseNova U1 configs as dense
* mtmd : add SenseNova U1 vision support
Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.
The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.
* mtmd : fix SenseNova U1 vision token layout
* model : clean up SenseNova U1 RoPE metadata
Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.
Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].
* mtmd : simplify SenseNova U1 projector integration
Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.
Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.
* mtmd : fix SenseNova U1 image positions and attention
* fix spacing
* mtmd : restore generic test input chunks
* mtmd : restore SenseNova U1 integration after upstream merge
* Update conversion/sensenova_u1.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : remove SenseNova U1 BF16 activation casts
Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.
Assisted-by: Codex
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model: add Muse Spark Open (text, vision and audio)
Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.
Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.
The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Muse Spark: drop the redundant forward expand
* Muse Spark: drop the redundant forward expand and tighten the comments
* Update src/llama-quant.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-graph.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : add LiquidAI/d1-omni-600M decision model
Assisted-by: Claude Opus 5.5
* mtmd : keep conformer GLU sigmoid on CUDA
Assisted-by: Claude Opus 5.5
* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal
Assisted-by: Claude Opus 5.5
* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files
Assisted-by: Claude Opus 5.5
* Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx
* Add initial MTP support
* Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements.
* Review driven changes, remove env vars, protect tensors
* Strip MTP for initial PR
* Clean up after mtp strip
* Clean up after mtp strip
* Update speculative.cpp
* Update llama-context.h
* Clean up after mtp strip
* Fix tokenizer ignore merges
* Improve quantization protection selection
* Refactor mhc helpers, graph base
* Lint Fixes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Skip glm5-next in model saver, fix CRLF
* Skip glm5-next in sweep
* Remove T4 fallback
* Review cleanup
* Review suggestions
* Defer separate MTP gguf handling to MTP PR, drop filter
* Repad n_head_kv
* kpool init apply
* Order by descending score
* Drop guard
* read kpool from hparams, clarify kpool cache flags, remove kpool_build_state(nullptr)
* Add glm5-next support to model saver and add arch test fixture
* Review cleanup
* Kpool pooled caching clarify
* Add multi stream support
* Finish Rebase
* Sparse FA fir DSA prefill
* Const
* Update llama-model.cpp to fix rebase error
* gguf-py : merge tensor map entries for HC tensors
* model : use build_gdn_l2_norm in GLM5_NEXT implementation
* chore : remove trailing whitespace
* model : use new OP precision setting API in GLM5_NEXT implementation
* mtmd : use ggml_swiglu_clamp in GLM5V and apply the image token limit
The two clamps around swiglu_split are what ggml_swiglu_clamp already does,
so the clamp bounds collapse back to one value. GLM5V also never called
set_limit_image_tokens(), so --image-max-tokens had no effect.
Assisted-by: Claude Opus 5
(cherry picked from commit 46d18e12d422be4cc04a70e4a9a9e0168bb3d5b7)
* llama : keep the GLM5-Next k-pool layout across ubatches
The layout was rebuilt from a full cell scan on every ubatch. Pools are fixed
by the positions relative to the sequence's first one, so the layout now lives
on the memory and a ubatch only appends to it.
A sequence edit no longer stales every pooled key either, only the ones at or
after the edited position, which makes a tail seq_rm free. The pooling subgraph
is built unconditionally so the graph shape no longer changes every kpool
tokens, and the pool axis is folded into rows before soft_max, which otherwise
exceeds the CUDA gridDim.y limit past n_kv 262144.
Assisted-by: Claude Opus 5
(cherry picked from commit 5d1c40b93e17fddbf73b785efe43e0d02ccb3977)
* model : write the GLM5-Next recurrent rollback checkpoints
The conv state and the delta net state were only written to the live row, so a
rollback restored whatever the checkpoint rows happened to hold. Take the same
route as kimi-k3: build_recurrent_attn for the state, and write all K_rs conv
groups. That also drops a state view that assumed contiguous rows.
Enroll the arch in test-recurrent-state-rollback, which catches this under its
garbage-filled cache pass.
Assisted-by: Claude Opus 5
(cherry picked from commit 5ace37e86d5d448e83ef5dde5632c748185b18cd)
* llama: fix PR #27773 test-save-load-state restore failure
Clear the attention and indexer cache data after a failed hybrid state restore so restored NaNs cannot affect a later sequence.
Assisted-by: Codex
* llama: fix PR #27773 gpu-rocm graph reallocation
Reserve the full GLM5-Next pool capacity and dirty pool count. The gpu-rocm Test step aborts when n_new grows while the graph node count stays fixed; CUDA, Vulkan, Metal, and WebGPU checks report the same error.
Assisted-by: Codex
* llama : fix GLM5-Next k-pool layout staleness after edits and shared teardown
Two defects in the cross-ubatch k-pool layout added by the k-pool commit:
1. Wrong results. An edited sequence only rebuilt its pool layout when its cell
count changed, so if the first ubatch after an edit added back exactly as many
cells as were removed, the stale position-to-cell list survived. With a unified
cache and more than one sequence, where another sequence takes the freed cells,
the reused layout points at the wrong cells (CPU: large logit drift, CUDA: NaN).
Rebuild whenever the sequence is stale, not only on a size mismatch.
2. Slowdown. "shared" mode was assumed to end only with an edit that forces a
rebuild, but sharing also ends when the other sequence is removed. The survivor
kept shared = true, pinning cache_safe off and re-pooling every pool on every
ubatch (server trigger: n>1 completions with -kvu, via the seq_cp in
copy_state_to). In seq_rm, if the layout has shared cells, stale every sequence
so one rebuild re-derives sharing and cache_safe returns to 1.
Assisted-by: Claude Opus 5
* llama : fix build_attn_mha stream stride for non-contiguous q
build_attn_mha split the batch into streams with a stream stride of
q->nb[3]/n_stream. That only equals one stream's span, (ne[2]/n_stream)*nb[2],
when q is contiguous. GLM5-Next is nope-only, so it does not concat a rope part
and passes the permuted q_absorbed straight in, where nb[3] != ne[2]*nb[2]; the
stride was then n_head times too large and every stream s >= 1 read another
head's queries. Split-KV (-np N without --kv-unified) multi-stream prefill was
wrong for every stream past the first. Unified KV and decode were unaffected
(n_stream == 1, and decode takes the gather path). Other MLA models concat rope
so q is contiguous and the computed value is unchanged for them.
Compute the stride from the token dimension, which is identical for a
contiguous q.
Assisted-by: Claude Opus 5
* llama : re-derive GLM5-Next k-pool sharing on state_read/state_drop
The shared-cell teardown added to seq_rm (stale every sequence when the layout
has shared cells, so a survivor does not keep shared = true and pin cache_safe
off) was missing from the other paths that can free shared cells: state_read
and state_drop staled only the one sequence. Apply the same re-derivation there
and correct the comment that claimed sharing ends only via an edit or seq_rm.
Assisted-by: Claude Opus 5
* quant : drop duplicate GLM5-Next hc_ filter
The hc_ name filter was listed twice in the GLM5_NEXT protection block.
Assisted-by: Claude Opus 5
* glm5-next: scope K-pool cache access to indexed operations
* glm5-next: keep K-pool access in hybrid index memory
* glm5-next: keep mHC graph builders model-local
* glm5-next: mark only touched pools per ubatch
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
* models: pad on the left with ggml_pad_ext
The Parakeet, LFM2-Audio, Granite Speech and Gemma 4 audio encoders
build a left padding as a right pad followed by a roll, and DFlash2
concatenates a zero filled block in front of the previous tokens.
ggml_pad_ext does both in one node now that every backend supports a
left padding. The Gemma 4 audio embeddings are bit identical.
* models: skip the DFlash2 taps that only read padding
A tap at or past block_size shifts every row out of the block, so its
term is zero. The loop runs min(kernel_size, block_size) taps.
* model : fold Ling 3.0 VL into the BailingMoeV3 architecture
Assisted-by: Scout
* model : keep shared NORM rope list intact when gating bailingmoe3 on mrope sections
---------
Co-authored-by: aetherbird <aetherbird@users.noreply.github.com>