* opencl: use the multi-column q6_K vocab head GEMV for 2..4 tokens on X2E
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q6K_MC3 to GGML_OPENCL_Q6_K_MC3
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q40/Q41/Q4K_MC3 to GGML_OPENCL_Q4_0/Q4_1/Q4_K_MC3
GGML_OPENCL_Q4K_MC3_DQ and GGML_OPENCL_Q4K_MC3_LDS become GGML_OPENCL_Q4_K_MC3_DQ
and GGML_OPENCL_Q4_K_MC3_LDS.
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: remove unnecessary debug probe
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: clean up
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hex-repack: move repacking/tiling into graph-optimize
* hex-tiling: simplify tiling rules and remove obsolete helpers
* hex-ops: use a hash for de-duplicating tensor descriptors
* hex-bufs: improve tensor hash
* hex-ops: accept empty (zero-elements) ops
* hex-bufs: enable hostbuf by default and reject tensors that do not fit into max_vmem
* hex-bufs: improve error message and update readme
* hex-bufs: make sure to sync flush opbatch in all sync set_tensor paths
* feat : added ricursiveai ripple support
* fix : mandatiry SWA, scale through helper, comments back
* fix : metal
* model : make ricursive RoPE/NoPE explicit and validate the layer pattern
Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.
Assisted-by: Claude Opus 5.5
* model : make ricursive swiglu_clamp per-layer
Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.
Assisted-by: Claude Opus 5.5
* model : use the ricursive per-tensor scales and RoPE parameters
Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.
Assisted-by: Claude Opus 5.5
* convert : drop the ricursive chat_template.jinja override
The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.
Assisted-by: Claude Opus 5.5
* fix : remove NVFP4 checkpoint quantization block
* match #31733 refactor
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* RDNA3/4 Vulkan Large-Tile Split-K Optimization
* Add single-operator performance test cases
* Update the policy to use driver-reported LDS usage
* Add comments
* Use the simpler split_k rule from review
Assisted-by: GitHub Copilot
* tests : drop the split_k test cases
* vulkan : simplify the RDNA3/4 split_k condition
* metal : support scale-bias routing norm in the TOPK_MOE fusion
The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sqrt(softplus) MoE routing with a selection bias
Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sigmoid MoE routing with a selection bias
The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.
- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
tail; the selection-bias ADD is required, same as the sqrt(softplus)
chains
- generalize the gating function constant from a sqrt(softplus) bool
to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
deepseek2/32, kimi, minimax, ...) now fuse their routing, and
muse-spark prefill drops one MOE_REDUCE fire because the reordered
graph allocates the routing weights inside the reduce output range,
which the dst/input overlap check correctly declines
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : remove low-signal comments
* ocd : valign ggml_metal_fusions
* CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a)
* Addressing reviewers comments, mma.cuh and physical warp size
* CUDA/HIP: single mma.cuh matrix-core lightning indexer ::CDNA::
* Support RDNA3
* CUDA/HIP: stream heads through the mma lightning indexer kernel
Process 16 heads at a time with double-buffered Q staging instead of keeping
all heads resident in shared memory, and drop the GGML_INDEXER_NO_MFMA env.
* enable for CUDA
* CUDA: Remove wmma lightning-indexer, also run head 32 through mma kernel
* CUDA: Nv mma dosent transpose accumulator
---------
Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* vulkan: add F16 input to the FWHT
The Vulkan FWHT accepts F32 input only. This adds an F16 source variant of the
shader, for both the subgroup and the shared-memory paths, so the kernel reads
an F16 source directly instead of requiring a converted copy.
The F16 shader needs shader-float16, so the pipelines are only created when the
device reports it. A null pipeline makes ggml_vk_can_use_fwht fall back, which
is the same path an unsupported width already takes.
supports_op and the dispatch both already route through ggml_vk_can_use_fwht,
so there is a single place asking whether an op qualifies; nothing to
consolidate here the way the CUDA/Metal ports needed.
test-backend-ops on an M5 Pro (MoltenVK, fp16: 1): MUL_MAT 1128/1128,
including all 24 Hadamard cases (18 existing F32, 6 new F16).
Assisted-by: Codex
* vulkan: FWHT kernels for block widths above 512
The Vulkan FWHT covers block widths 64 to 512; wider blocks fall back to the
dense matmul. Extend the width list to 8192 and pick the shader per width: the
subgroup shader keeps n/subgroup_size values per invocation, so widths above
512 (or a subgroup narrow enough to exceed 64 values per invocation) use the
shared-memory shader with one row per workgroup and a 256-wide block. The rows
per workgroup become a specialization constant so the dispatch matches.
Pipelines for a width are only created when the device's shared-memory and
workgroup-invocation limits allow it; otherwise that width keeps the fallback.
This is the Vulkan counterpart to #29100 (CUDA). Stacked on #29101; the last
commit is the one to review.
Assisted-by: Codex
* checkpoints: reduce overhead
Assisted-by: Codex
* rebase
Adapt the slot save/restore checkpoint appendix from #26004 to the
shared_ptr checkpoint list and the common_state_data buffers.
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: cwriter <cwriter@localhost>
* ggml : support `truncate=false` for YaRN
YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.
Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.
* support CUDA backend
* add tests
* move RoPE helpers to internal header
* format
* conversion : resolve quantized_layers from source tensor names
This commit changes how the per layer NVFP4 quant_algo are resolved from
hf_quant_config.json quantized_layers.
The keys in quantized_layers are module paths in the source checkpoint,
and these can be either be modules like 'lm_head' or parent modules
like 'layers.0.ffn.experts'. Currently these keys are passed through
filter_tensors, but they are not tensor names. So a filter_tensors that
keeps an explicit list of tensor names, like Nemotron's with --mtp
would drop them. That means there was no `_nvfp4_layer_algo entry`, and
`_tag_prec_a4` fell back to `_nvfp4_global_algo`.
This commit records the mapping from source tensor name to filtered
tensor name in index_tensors. Then in prepare_tensors each source
tensor is resolved to its closest quantized_layers parent, and
the algo is stored under the filtered tensor name, which is what
_tag_prec_a4 later looks up.
Converting Nemotron 3.5 Lightning NVFP4 with --mtp now produces:
```console
general.tensor_extra.name = ['output.weight']
general.tensor_extra.prec_a4 = [False]
```
whereas on master no general.tensor_extra fields are written. For the
non-MTP conversion, the general.tensor_extra fields are identical to
master.
Also manually verified (using pdb) that this works for
DeepSeek-V4.1-Flash-1017-NVFP4.
Refs: https://github.com/ggml-org/llama.cpp/pull/31602#issuecomment-6603337403
* conversion : shorten comment
* conversion : also remove '.bias' suffixes
Prevent malformed realtime session updates from throwing an uncaught
exception and terminating the server
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* fit : account for skipped NextN layers when setting n_gpu_layers
Without MTP, fit counts n_layer + 1 layers while the model loader counts
n_layer_all + 1 for n_gpu_layers, including the NextN layers which hold no
weights. A fit result that offloads all layers then leaves layer 0 on the CPU
("offloaded 65/66 layers" for Qwen3.8-27B on two GPUs).
Add the skipped NextN layers to n_gpu_layers and to the tensor split of the
device that holds the output layer so the loader assigns the layers that
fit measured.
Fixes#31531
* fit : pass hp_ngl_skip by reference like the other hparams
* fit : add hp_ngl_skip to common_get_device_memory_data
* graph: store the conv state of every rollback arch in one set_rows
The rollback slots of the causal conv history were written with one
copy per slot in qwen4exp, glm5-next, kimi-k3, bailingmoe3, lfm and
mamba2, while delta_net already wrote them with a single set_rows.
build_conv_state_store now does this for all of them: one copy without
rollback, otherwise one set_rows of the overlapping windows into the
min(n, n_rs_seq + 1) slots a rollback can read.
The slot order {n_rs_seq, ..., 0} is computed once per graph and each
conv layer takes a view of its tail. Its size does not depend on the
ubatch, so the reserved graph covers every ubatch shape, and the graph
input that held it is gone.
A model that is not causal never rolls back, so the context clamps
n_rs_seq to 0 for it, like for an arch without rollback support.
* nits: compute the conv slot order in the graph and keep the n_head name of bailingmoe3
* graph: build the conv slot order in each conv layer
The slot order {K-1, ..., 0} of the conv state store was one tensor
shared by every conv layer of the graph, built on the device of the
first one, so each layer on another device copied the view of its
tail. Each conv layer now builds its own order next to its set_rows,
and llm_graph_input_rs no longer holds it.
* tests: fix CI flakes of TOPK_MOE caused by near ties
With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.
Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.
* tests: build the TOPK_MOE logits by inverting the gating function
Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.
The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
The first command buffer takes the first 10% of the graph nodes and the
rest is split evenly between the extra command buffers. A fusion cannot
span two command buffers, so a boundary that falls inside a fusable
group runs that group unfused, and the output then depends on the node
count of the graph.
Move each boundary to the end of the fusable group it falls into, so
the encoders fuse the same nodes as with a single command buffer.
This commit adds nvfp4 scales to qwen2's build_ffn call which are
currently passed as NULL.
The motivation for this is that a model converted to NVFP4 will
currently crash because the scales are loaded but never applied in
the FFN:
```console
llama-completion: llama.cpp/src/llama-sampler.cpp:1218:
void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*):
Assertion `found' failed.
```
The attention already applies its scales but ffn is missing them.
* opencl: use the small-N f32 GEMV by default on X2E
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: use the q5_K multi-column verify GEMV by default on X2E
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* cuda: fuse NVFP4 gate/up and GLU in MMQ
* cuda: honor the requested src1 precision in fused gate/up MMQ
* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels
* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming
* cuda: check gate_scale before loading the fused NVFP4 gate scale
* llama : fix on-device state copy sizes for quantized KV caches
The on-device sequence state copies (LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) computed
their element count as size / ggml_element_size(). For a quantized type such as
q8_0 or q4_0 that is the number of blocks, not elements, so only part of each
range was copied: the restored state was wrong, or the copy crashed.
Compute the count as size / ggml_type_size() * ggml_blck_size() for the save,
the restore and the copy between differently chunked ranges, and assert that
each range covers whole blocks.
* llama : drop the one-line comments
* llama : drop the redundant cast in the state copy size
* server : realtime: send an error event instead of crashing on exceptions
* server : route websocket handlers through ex_wrapper, remove the realtime test
* server : realtime: keep the previous tools when the chat template rejects new ones
* server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapper
* server : realtime: clarify the tool schema check comment