load_arch_hparams multiplies n_hrm_layers_per_stack, n_hrm_h_cycles and
(n_hrm_l_cycles + 1) into a uint32_t, then asserts the product equals
block_count.
Any factors whose true product is block_count + k*2^32 still pass the
assert; 1073741825 * 2 * 2 = 2^32 + 4 compares equal to block_count = 4.
load_arch_tensors iterates the same three values in int and writes past
the end of the block_count-sized layers vector.
* llama : give Inkling2 its own relative-position attention input
The Inkling2 graph built its flash attention mask from the hybrid iswa
input and passed it to build_attn through a new kq_mask argument. That
argument let a caller pair its own dense mask with the kv_rows of
llm_graph_input_attn_kv_iswa, whose mask follows the per-sequence cell
lists whenever kv_rows is enabled.
Add llm_graph_input_attn_rel_pos instead. It owns dense KQ masks and the
query-to-cell distances, fills both from the same cells, and its
build_attn overload adds the per-head bias to its own mask and never
passes kv_rows. The iswa build_attn is back to its previous signature;
its body moves to build_attn_iswa_impl, which both overloads call.
Inkling2 now folds the per-query sink into the bias with and without
flash attention, so build_attn_mha no longer needs the per-query sink
path.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* inkling2 : make the expert weight scale and final logit softcap optional
Load both keys as optional and skip them when they are zero, as other
models do, instead of rejecting zero at load.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : build kv_rows masks for the hybrid iswa memory
llm_graph_input_mem_hybrid_iswa checked its masks with
can_reuse_attn_mask, which expects kv_rows whenever they are enabled,
but built them dense. With a unified cache and flash attention the
check always failed, and hybrid iswa models (LFM2, LFM3, Inkling2)
never reused a graph. Build the masks with build_attn_inp_attn_mask,
like the plain iswa input.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : make the relative-position input a hybrid iswa input
Address review: llm_graph_input_mem_hybrid_iswa_rel derives from
llm_graph_input_mem_hybrid_iswa and owns the attention and recurrent
inputs, so Inkling2 builds one memory input instead of two. Its
attention input is built with dense masks, can_reuse checks them as
dense, and its build_attn asserts that the mask has no kv_rows (TODO:
a kv_rows variant). build_attn_iswa_impl is now a static helper.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : compose the hybrid iswa rel input, format inkling2
Address review: llm_graph_input_mem_hybrid_iswa_rel owns its attention
and recurrent inputs like the other composite inputs instead of deriving
from llm_graph_input_mem_hybrid_iswa.
Format src/models/inkling2.cpp like the other model sources: grouped
and aligned key and tensor loading, descriptive per-layer names, blank
lines and section comments between the steps, and cb() names for the
intermediate tensors. The graph is unchanged.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
n_mudd_taps comes from limite.mudd.tap_count, a scalar read separately
from the length-checked tap_indices array. mudd_tap_idx is a fixed
std::array<int32_t, 512> indexed at [i*n_taps] in both the tensor load
and the graph build, so a crafted GGUF with n_layer * n_mudd_taps > 512
reads past the array.
Reject that at hparams load; the one check covers both index sites.
* model:add Zaya1 8B (Zyphra)
- Added the support for Zaya.
- Fixed the SSM_CONV case, so the model runs on cuda without abort.
Please enter the commit message for your changes. Lines starting
* Made the following changes based on the bot reviewer feedback:
1) Added runtime asserts when loading wrong KV head size, unsupported conv kernel size.
2) Removed the static cast usage.
3) Fixed double load of tokenizer.
4) Updated res_scale_in to be input tensor.
* - Moved the res_scale_in back to layer 0 to fix the cuda bug
* i) Removed the EOS addition.Team agreed that updating Tokenizer is the right move.
ii) Fixed formatting
* Fixed the comment to make it clear
---------
Co-authored-by: varun <varun_subramanian@zyphra.com>
* jinja: impl scope-chain key lookup
* add test for call_statement lambda
* chat: add tts_dict for kokoro G2P template
* more testing
* value_float_t range check
* no json-in-meta, add llama_model_meta_arr_str/_n
* mtmd: add kokoro tts support
* detailed guide about G2P and text norm port
* refine G2P
* clean up
* migrate batch api
* nits
* fixed support for DFlash2 for Ripple 65B
* spec: Add NoPE handling to DFlash
(cherry picked from commit 89ea8e3e77e3359d17902a2db136022d5d419d88)
* spec: Fix NoPE not being picked up by DFlash
(cherry picked from commit 43393b95fda2ce7e7d243e2650bf7b1cadf1c732)
---------
Co-authored-by: James Pickering <james_pic@hotmail.com>
* lfm3 : put the normed tensor first in residual adds to enable norm fusion
Assisted-by: Claude Opus 5.5
* tests : update Metal fusion baseline for lfm3
Assisted-by: Claude Opus 5.5
* convert : add Inkling2 text GGUF conversion
Assisted-by: Codex
* llama : implement Inkling2 text inference
Assisted-by: Codex
* tests : cover Inkling2 text support
Assisted-by: Codex
* conversion : dequantize packed Inkling2 experts in row chunks
Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.
Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.
Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.
Assisted-by: Codex
* convert : add Inkling2 vision mmproj support
Assisted-by: Codex
* mtmd : implement Inkling2 still-image encoder
Assisted-by: Codex
* inkling2 : scope image loading
Assisted-by: Codex
* inkling2 : enable flash attention with folded bias and sinks
Assisted-by: Codex
* cuda : route per-head flash attention masks through vec
Assisted-by: Codex
* mtmd : support Inkling2 temporal frame means
Assisted-by: Codex
* style : restore shared image comment
Assisted-by: Codex
* tests : enlarge Inkling2 context fixture
Assisted-by: Codex
* inkling2 : load per-layer expert feed-forward widths
Assisted-by: Codex
* tests : add Inkling2 Metal fusion baselines
Assisted-by: Codex
* inkling2 : group and align registry entries
Assisted-by: Fable
* convert : regenerate the Inkling2 tokenizer hash with the update script
Assisted-by: Fable
* conversion : revert the print_registered_models change
Assisted-by: Fable
* conversion : sort the Inkling2 model map entries
Assisted-by: Fable
* convert : map Inkling2 vision checkpoint names in the tensor table
Assisted-by: Fable
* convert : map Inkling2 text checkpoint names in the tensor table
Assisted-by: Fable
* tests : document the Inkling2 packed expert conversion test
Assisted-by: Fable
* tests : cover Inkling2 checkpoint and transformers tensor names
Assisted-by: Fable
* llama : place Inkling2 fields and list entries by file convention
Assisted-by: Fable
* gguf-py : align Inkling2 tensor mapping comments
Assisted-by: Fable
* mtmd : place Inkling2 entries by file convention
Assisted-by: Fable
* tests : place Inkling2 cases after the existing ones
Assisted-by: Fable
* cuda : index per-head flash attention masks in the tile and mma kernels
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : project the relative bias before the gather, drop arch branches
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* convert : require Inkling2 hparams, add inkling2 pre-tokenizer
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* mtmd : cap Inkling2 image size, drop video frame averaging
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : fix CI build, Meta device assert and Windows conversion test
Assisted-by: Claude
* tests : use the default sliding window for the Inkling2 test model
Assisted-by: Claude
* simplify mtmd part
* give _v arch for vision
* remove alpha channel handling, it should be a follow up
* explicit rule about audio + vision models
* add audio support
* rm audio
* rm gguf mtmd rope base
* address review comment
* nits
* do not normalize embd from mtmd
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
The qwen2vl graph passed NULL for the ffn_up/gate/down scales, so the
per-tensor NVFP4 scales loaded by the generic pass were never applied
and the model produced garbage (PPL ~1e6 on CUDA, NaN on CPU).
Pass the scales to build_ffn like qwen2 does.
Fixes#32101
* dflash : move the DSpark markov head and backbone graphs to dspark.cpp
Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and
graph_mistral_large4 move verbatim. build_inp_target_features and
build_dspark_markov_head become static members of llama_model_dflash so
both files can use them.
* dflash : detect the head and the backbone once, split the loaders per file
load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2,
DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4).
The head, backbone and base layer loaders become member helpers in
dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph
dispatches on the backbone.
The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through
a helper in dflash2.cpp, right after their head tensors instead of in
the base layer loop; the other tensors keep their creation order. The
DFlash2 loader drops its copy of the conv group size check, which master
never reached.
* dflash : share the injection input, token embeddings and lm head lookups
build_inject_inp (target features + fc + encoder norm), get_tok_embd and
get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral
Large 4 graphs. build_inp_target_features is file-static again.
* dflash : split the DFlash decoder into a graph per head
graph_base creates the shared inputs and provides build_inject,
build_inp_tokens, build_layers and build_logits. graph_dflash
(dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and
graph_lilicorr (lilicorr.cpp) assemble them and add their head; the
selector, scorer and markov head builders move to the file of their
graph, and the DFlash2 local conv run by build_layers to dflash2.cpp.
build_arch_graph dispatches on the backbone, then on the head.
The op order and the tensor names are unchanged.
* speculative : split the DFlash driver into a class per head
common_speculative_impl_draft_dflash stays the DFlash implementation and
becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its
protected constructor only reads the draft metadata; init() then sets up
the contexts with the options of the draft (draft length, backend
sampling, device features, masked h_nextn). Each derived class overrides
draft() with the path of its head from the former draft(). DFlash2 also
overrides begin() to reseed its generator per request, and LiliCorr
overrides process() to store the context features of each injected
chunk, which its noise block carries as its state.
The factory keeps the current routing: draft-dspark picks DSpark,
draft-dflash picks DFlash2 or LiliCorr from the draft metadata and
DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs
the DSpark reader instead of a mix of both. sample_from_anchor is logged
by DSpark only.
* speculative : add the draft-dflash2 and draft-lilicorr types
draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts,
which draft-dflash keeps running too: it picks them from the draft
metadata. draft-dflash2 and draft-lilicorr must match the draft model,
draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a
DFlash draft is rejected, and a run uses at most one DFlash family type;
these errors leave speculative decoding uninitialized.
The type detected from the draft GGUF is now the concrete one, and
draft-dflash2 and draft-lilicorr download the dflash sidecar.
* docs : describe the DFlash2 and LiliCorr drafts
Add the DFlash2 and LiliCorr sections and the new types, and explain
that draft-dflash also runs DFlash2 and LiliCorr drafts.
* speculative : drop the unused DFlash is_mrope
Nothing reads it since the migration to common_batch (#29385), which
picks the number of M-RoPE position rows from the rope type of the draft.
* doc: update spec dec
* address review comments
- remove build_delta_net_chunking and build_delta_net_autoregressive
- always use the fused gated delta net
- make graph_max_nodes independent of the ubatch size
- drop the fused GDN probes and fused node tags
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls
Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.
* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends
Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.
* tests: cover GGML_PREC_BF16 in the flash attention tests
The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.
* llama: request GGML_PREC_BF16 for the KQ multiplication
* cont : add TODO in clip.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* model: add Kimi-K3.1-Pro text model
K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.
Indexer:
- the selection is an additive mask over the whole cache, not a gather of
the top-k rows, so it reuses the existing sparse flash attention path;
off that path the mask stays correct but still costs a kq over the cache
- the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
because dsa_config declares no eps of its own
Graph and cache:
- new arch with four indexer tensors per DSA layer, and a hybrid memory
that keeps a K-only MLA cache for those layers, recurrent state for the
KDA layers, and the indexer key cache
- the graph node budget shares the K3 entry
Conversion:
- KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
- dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
plus the indexer LayerNorm eps; the layer list is 1-indexed like
linear_attn_config's, and the converter asserts that it covers exactly
the full-attention layers
- the routed experts stay MXFP4 and are repacked as in K3, not dequantized
- the vision tower and mm_projector are skipped: text only for now
- the arch key follows the checkpoint's own model_type ("kimi_k31") and the
loader registers KimiK31ForConditionalGeneration; the release name carries
a -Pro suffix that nothing in the config or the tensor names uses
Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.
Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.
* converter: avoid clutter in the k3 base
* narrow comment to match model
* update metal fusion expectations
* restore
* sync with PR 31733 refactoring
* feat : added ricursiveai ripple support
* fix : mandatiry SWA, scale through helper, comments back
* fix : metal
* model : make ricursive RoPE/NoPE explicit and validate the layer pattern
Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.
Assisted-by: Claude Opus 5.5
* model : make ricursive swiglu_clamp per-layer
Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.
Assisted-by: Claude Opus 5.5
* model : use the ricursive per-tensor scales and RoPE parameters
Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.
Assisted-by: Claude Opus 5.5
* convert : drop the ricursive chat_template.jinja override
The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.
Assisted-by: Claude Opus 5.5
* fix : remove NVFP4 checkpoint quantization block
* match #31733 refactor
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* graph: store the conv state of every rollback arch in one set_rows
The rollback slots of the causal conv history were written with one
copy per slot in qwen4exp, glm5-next, kimi-k3, bailingmoe3, lfm and
mamba2, while delta_net already wrote them with a single set_rows.
build_conv_state_store now does this for all of them: one copy without
rollback, otherwise one set_rows of the overlapping windows into the
min(n, n_rs_seq + 1) slots a rollback can read.
The slot order {n_rs_seq, ..., 0} is computed once per graph and each
conv layer takes a view of its tail. Its size does not depend on the
ubatch, so the reserved graph covers every ubatch shape, and the graph
input that held it is gone.
A model that is not causal never rolls back, so the context clamps
n_rs_seq to 0 for it, like for an arch without rollback support.
* nits: compute the conv slot order in the graph and keep the n_head name of bailingmoe3
* graph: build the conv slot order in each conv layer
The slot order {K-1, ..., 0} of the conv state store was one tensor
shared by every conv layer of the graph, built on the device of the
first one, so each layer on another device copied the view of its
tail. Each conv layer now builds its own order next to its set_rows,
and llm_graph_input_rs no longer holds it.
This commit adds nvfp4 scales to qwen2's build_ffn call which are
currently passed as NULL.
The motivation for this is that a model converted to NVFP4 will
currently crash because the scales are loaded but never applied in
the FFN:
```console
llama-completion: llama.cpp/src/llama-sampler.cpp:1218:
void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*):
Assertion `found' failed.
```
The attention already applies its scales but ffn is missing them.
* ggml: bound views by the bytes their strides reach
A view was checked against its source with the size of a contiguous
tensor of the same shape, computed before its strides were set. Views
whose rows overlap failed that check although they stay inside their
source, and views with wider strides passed it while reaching past
the end. The view functions now pass their strides to the tensor
constructor, which checks the bytes they actually reach, like
ggml_nbytes. The conv windows of delta_net are created with all their
rows at once instead of being widened after the check.
* ggml: assert that a strided view starts from contiguous elements
The view functions set nb[0] to the type size, so the source must have
contiguous elements along dim 0 for the view to read them.
* Optimizing Gemma4 graph for prompt processing
TODO:
- Currently complete microbatch needs to contain tokens which does not require the logits; need to optimize this further
- There are a few more operations that are performed after the kv cache is calculated, need to safely optimize them.
* Not filtering the rows when they are unmasked
* Cropping out the attention mask too.
* gemma-4: improved pruning, mask cropping and PLE scaling
Prune before the first KV-shared layer with the single-stream, causal, non-embedding safety gates.
Use cropped masks only during shared-layer attention.
Preserve requested layer-input embeddings in the KV prefix.
Remove the PLE contiguous materialization and fold equivalent PLE scaling into RMSNorm parameters/weights.
* spec-mtp: Enable masked h_nextn extraction only when the draft context shares the target KV cache
This lets Gemma 4 prune unused prompt rows before its shared layers so that we keep the full perf P.P gains with MTP.
Avoid extracting target hidden states for output-free prompt chunks in that shared-MTP path.
When a batch has a selected final token, extract only that row unless the whole verification batch is selected.
Keep ordinary MTP and classic speculative decoding behavior unchanged: they still use unmasked h_nextn because the draft
context must catch up on all target rows.
speculative-simple needs an update to select the final prompt row only for shared-KV MTP,
so the optimization works there without changing other speculative modes.
And it looks like the server already selects its final prompt row for first-token sampling.
* spec-mtp: make sure to not drop first draft round
* gemma4: enable pruning based on static cparams.kv_unified/n_seq_max=1
* gemma4-mtp: simplify prunning logic and avoid graph branches/re-allocs
* gemma4: update pruning gates to use only n_seq_max==1, kv_unified can now be used for multi-seq
* spec-simple: revert spec-simple changes for now
* gemma4: remove causal_attn from pruning gate (last source of reallocs)
* cont : adapt to llm_attn_mask changes
* cont : add TODO
---------
Co-authored-by: Meet Amin <meetamin378@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
build_conv_state_at makes a contiguous copy of the conv input tail
before writing it into the recurrent state, so every GDN layer issues
two device copies per generated token where one is enough. On
Qwen3.8-Flash-Next (36 GDN layers) that is 72 copies per token, now 36.
Speculative decoding adds one more per rollback slot. ggml_cpy reads
the strided view directly, as the shared build_conv_state already does,
so the extra copy is dropped.
* delta_net: access conv state via views
* fix sm tensor
* delta_net: fill the conv slot index on the host
The rollback slot index of the conv windows is a constant per graph
shape. It is now an I32 input of the recurrent memory filled on the
host, instead of an arange and an argsort computed on the device, and
the meta backend no longer needs a mirrored split for ARANGE.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
The dspark block mask is now derived at input-fill time: llm_attn_mask::block_uniform
fills every token of a sequence with the mask row of its first token, so the
graph-level reshape/repeat in dflash is gone and cnv always aliases raw.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : use llm_attn_mask and kv_rows for unified attention
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont : optimize `get_seq_cells`
* llama : build the kv_rows lists from per-sequence cell bitmaps
Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.
Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).
* cont : make get_kv_rows_p_min private
* cont : minor metal
* llama : use kv_rows for every ubatch of a unified cache
A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.
Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.
* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The duplicated output looks up token_embd in its buffer context, so
token_embd must exist first. With the old order a load where input and
output share a buffer type (CPU-only, -ngl 0) held the table twice:
gemma-4-E2B-it Q4_K_M, no mmap, CPU model buffer 3163.73 -> 2947.73 MiB.