* sampling : use greedy selection for eligible temperature-zero chains
Assisted-by: OpenAI Codex
* Apply suggestion from @ggerganov
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sampling: simplify zero-temperature greedy eligibility
Allow the same greedy selection on CPU and grammar/reasoning-budget paths.
Keep distribution sampling for dynamic temperature and requested probabilities.
Cover the common sampler selection and probability behavior in the existing sampler tests.
Assisted-by: OpenAI Codex
* sampling : use greedy selection after final top-k with k=1
Assisted-by: OpenAI Codex
* cont : clean-up
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* convert : Fix token configuration for PLaMo-3
The reasoning and tool calling tags in PLaMo-3 consist of three tokens
each. For example, for reasoning:
* reasoning start: `<|plamo:begin_`, `think`, `:plamo|>`
* reasoning end: `<|plamo:end_`, `think`, `:plamo|>`
Registering `<|plamo:begin_`, `<|plamo:end_`, and `:plamo|>` as
`USER_DEFINED` so that they are parsed correctly.
Also PLaMo-3 models use <|plamo:tag|> as EOT, while PLaMo-2 models
use <|plamo:op|>.
Take EOT token as a parameter and look it up so that PLaMo-2 and
PLaMo-3 can use their appropriate ones.
* use NORMAL instead of USER_DEFINED
* fix mixed different model GPUs issue
* Update ggml/src/ggml-sycl/ggml-sycl.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sycl: accelerate GLM MLA prefill with MKL flash attention
GLM-4.7 Flash uses an MLA shape with 576-wide Q/K heads, a 512-wide
V head, GQA 20, and F16 KV. The SYCL dispatcher rejects this shape
because the normal MKL flash-attention gate requires matching K/V
widths and caps the head dimension at 512, so prompt processing falls
back to the substantially slower TILE kernel.
Admit only the validated 576/576/512, GQA-20 F16 shape to the existing
MKL pipeline. Keep all other mismatched K/V shapes on their current
fallback paths.
Handle GLM's V cache as a narrower strided view of K rows. Select the
strided F16 descriptor when row stride is padded, and alias K/V
dequantization buffers only when their logical widths match. Restrict
the stride exception to a real V view sharing K's row stride.
Add the exact 576/512, GQA-20 prompt-path backend test.
On an Intel Arc Pro B70 at master e613ef2, pp8192 improves from
432.80 to 1292.29 tok/s (2.99x, +198.6%). tg256 remains unchanged
within noise at 45.67 versus 45.65 tok/s. The exact MLA test passes
and debug output confirms MKL dispatch.
* sycl: store MKL flash attention scores in F16
Keep the QK GEMM output in F16 instead of F32. The online softmax still
converts each score to F32 for its max, exponent, and sum, so the
per-element math is unchanged apart from score rounding, and the F32
matrix was being written only to be consumed as F16 probabilities.
The F32 score matrix is the largest flash-attention intermediate on this
path; storing it as F16 halves its size and traffic. This builds on the
coalesced softmax loads from 1aa2954bd, which read each score row
cooperatively, so the smaller dtype pays off.
Measured on an Intel Arc Pro B70 with the dispatch from the previous
commit, -ngl 999 -b 4096 -ub 1024 -ctk f16 -ctv f16 -fa on:
pp8192 1583.9 -> 1657.1 tok/s (+4.6%)
pp64000 610.0 -> 684.5 tok/s (+12.2%)
pp131072 ~354 -> 402.1 tok/s (+13.5%)
tg256 at 8k context is unchanged (32.64), and the FLASH_ATTN_EXT suite
shows no new failures. The exact GLM MLA backend cases pass against CPU.
Adjust the ~354 baseline figure if you prefer citing only measured pairs (the 131k dispatch-only point came from the equivalent maintained build). Optionally add Assisted-by: <tool name> per the contribution guidelines since AI contributed to the change.
* Revert "sycl: store MKL flash attention scores in F16"
This reverts commit 265f9748164dc88e02678425b75f520ccd528b32.
* hex-cpy: replace more paths with dma and simplify l2flush
* hex-cpy: use DMA in all sametype paths
* hex-cpy: rewrite the rest of the copy paths (diff type) to use dma
* hex-concat: use dma for multi-dev path which also removes the need for l2-line alignment
* hex-concat: proper support for mdev splitting
* hex-concat: cleanup ctx and kern params usage
* hex-cpy: cleanup contex and remove left-over non-dma checks
* hex-cpy: clean dma_cpy naming
* hex-cpy: proper kernel params and kernel selection
* hex-build: resolve left-over rebase conflicts
* hex-cpy: update dev guide to clarify 128 byte alignment requirement
* hex-concat: make sure we go through mdev barrier
* hex-concat: make sure to flush dma-queue
* hex-cpy/concat: cleanup kparams and vtcm layout handling
* hex-cpy: remove dead check for contig (routed to diff kernel) and update comments
* hex-dev: update developer guide based on latest changes
* hex-cpy: safe skip of noop copies
* hex-dup: route DUP to CPY
The XIELU CUDA kernel template is already generic over the element
type; only the F32/F16 type assertion and the else-if dispatch were
missing. Add the nv_bfloat16 branch to the launcher, and drop the
temporary supports_op gate in ggml-cuda.cu that rejected BF16+XIELU.
test-backend-ops gains two BF16 cases ([10,5,4,3] and [512,16,1,1]).
docs/ops/CUDA.csv and docs/ops.md are regenerated; the F32 xIELU row
flips from no to yes as well, i.e. the previous record was stale.
Tested:
- Mac CPU: xIELU F32/F16/BF16, 6/6
- Mac Metal: existing F32/F16, 4/4; BF16 still unsupported
- RTX 4090 CUDA: xIELU F32/F16/BF16, 6/6
- RTX 4090 CUDA BF16-only: 2/2
- git diff --check passes
* vocab : implement PLaMo-3 tokenizer pre-segmentation
The PLaMo-3 tokenizer inserts hard boundaries before running the Unigram
DP, around <|plamo:...|>-looking text, and around runs of at least 4
identical characters or 2 spaces. Without them llama.cpp tokenizes code
indentation and repeated punctuation differently from the reference.
Reproduce the two re.sub() passes in llm_tokenizer_plamo2 by encoding
each segment independently.
* add vocab type "plamo3"
* Update src/llama-vocab.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* misc change
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model: K2 Horizon gguf conversion code
* model: loading hparams and tensors in k2-horizon.cpp
* model: K2 Horizon compute graph
* model: K2 Horizon compute graph adjustment and registering tokenizers
* model: K2 Horizon chat template and accomodate safetensors naming
* unicode : add the K2-Horizon pre-tokenizer splitter
The K2-Horizon regex had no arm in unicode_regex_split_custom and fell through to the
general std::regex fallback, which fails two ways.
On MSVC std::regex rejects \p{...}, so no K2-Horizon GGUF loads on Windows at all:
llama-quantize, llama-imatrix and llama-perplexity all abort with
regex_error(error_escape) before a token is produced.
Where the fallback does compile it is still wrong. unicode_regex_split collapses each
codepoint to a single byte naming its Unicode category before matching, and U+200C/U+200D
are category Control, which has no entry in k_ucat_cpt, so both become the 0xD0 fallback
byte. The literal and alternatives in K2's regex can then never match and
every ZWNJ or ZWJ ends a letter run.
The splitter is the existing llama3 one with a single rule widened, since K2's regex
differs from llama3's only in that a letter run also takes marks, ZWNJ and ZWJ.
tests/test-unicode.cpp gains a case for this: it fails before the change with
[Amy] [ZWNJ khaham] and passes after with the run intact.
* tests: expand K2 Horizon unicode splitter coverage
* unicode: handle K2 Horizon case folding and empty input
Assisted-by: Codex
* jinja : support sequence indices in selectattr and rejectattr
Assisted-by: Codex
* model : add K2 Horizon dense and MoVA support
Includes the K2 Horizon implementation from ifm-ai/llama.cpp with converter, tensor-parallel and model save/reload fixes.
Assisted-by: Codex
* chat : support K2 Horizon reasoning and tool calls
Assisted-by: Codex
* conversion: remove obsolete K2 Aurora alias
Assisted-by: Codex
* k2-horizon: enforce response schemas and load YaRN betas
Constrain final JSON after reasoning, accept flexible JSON tool envelopes,
enforce XML dialects, and handle repeated or alternate thinking markers.
Load YaRN beta metadata instead of retaining the default values.
Add schema, streaming, continuation, and model reload regressions. Validate
CUDA and CPU builds and 0.9B, 4B, and MoVA conversation/tool round trips.
Assisted-by: Codex
* renaming template fixture
* adressing cisc follows ups
* desloppify the parser / adress aldehir comments
* clean test-chat
* remove fallback : model trained mostly on high anyway
* fix k2 attn_v_exp tn splitting and metal fusion baseline
* k2-horizon : forward expand views before sums
* k2-horizon: copy embds before group norm to fix TP
* disable tesnor parallelism
---------
Co-authored-by: Ryandito Diandaru <ryandito.diandaru@mbzuai.ac.ae>
Co-authored-by: WestWaters <mario.papaleo2013@gmail.com>
Co-authored-by: Natani L. Mayday <71436458+TaskPuppyNatani@users.noreply.github.com>
Co-authored-by: West <100190545+WestWaters@users.noreply.github.com>
Co-authored-by: aaryamonvikram <aaryamonvikram@gmail.com>
Co-authored-by: aaryamonvikram <96529820+aaryamonvikram@users.noreply.github.com>
The gather path attended over the selected latents with a plain
matmul and softmax. It only ran with n_ubatch <= 16, and the flash
attention backends now skip the masked rows through n_kv_max, so
the scatter path covers every case.
Drop the gather flag, the gathered attention branch and
gather_mla_rows. set_input_kpool always maps padding to the n_kv
sentinel, and the slot mask becomes sel_mask since only the
scatter reads it.
* ggml: fix CLAMP on non-contiguous views (CPU, CUDA)
CUDA clamped ggml_nelements values flat and ignored the view strides.
CPU addressed row j as j*nb01 and ignored nb02/nb03. Both now follow the
strides of dims 1..3; CUDA supports_op requires contiguous rows, like
Metal. test_clamp gains a non-contiguous view case.
* cuda: clamp kernel uses fastdiv for the view strides
A 1.0 loader (e.g. Android 8.1) has no vkEnumerateInstanceVersion, so
backend init called a null pointer. Treat it like any loader under 1.2.
Fixes#29871.
Assisted-by: Claude Opus 5.5
* mimo2 : always emit h_nextn
the other nextn-capable models set it unconditionally
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* models : consolidate nextn row cropping into shared helpers
- replace the duplicated crop conditions and the per-model flags (narrow_early,
crop_before_ffn, crop_last_layer, emit_h_nextn) with two helpers on llm_graph_context:
crop_before_nextn() / crop_after_nextn()
- models that only tested embeddings_nextn_masked now share the same condition, so they
crop the last layer before the nextn capture whenever extraction is off
- t_h_nextn is now set unconditionally in mimo2, qwen4exp and deepseek4 (as in the other
nextn-capable models); host-side reads stay gated by cparams.embeddings_nextn
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
The speculative MTP init enables NextN extraction on the target and draft
contexts after both were created and their schedulers reserved. With
unmasked extraction the trunk graph keeps every token through the last
layer instead of cropping to the output rows, so the first decode
reallocates to that batch's shape and the next, wider batch trips
GGML_SCHED_DEBUG_REALLOC. Invalidate the reserve when the flags change so
the next compute re-reserves with the new graph shape.
Assisted-by: Claude
* ggml: refactor selective expert copying to user code
* tests: enroll two models into selective expert copy test
* tests: use deepseek2 as test model
* improve comment in ggml-backend.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont: fix whitespace
* cont : better comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml-openvino: skip unselected graph branches and support DUP
Upstream #29622 adds a mixed token/embd branch to every input
embedding graph through ggml_build_forward_select(). Its nodes are
not flagged for compute, but the backend translated them anyway,
and the DUP in that branch was unsupported, so the scheduler split
the graph and passed the embeddings across the split with a fixed
token count. The first single-token decode then failed
(test-thread-safety on CPU and GPU).
Build the OV model from the compute nodes only, and translate a
same-type contiguous DUP like CONT so the graph stays on one backend.
* ggml-openvino: make inp_scale_rows token dim dynamic
#29622 also moves the per-token embedding scale (gemma3, gemma3n,
gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim
and pad it per chunk on the static (NPU) path.
* ggml-openvino: skip GPU MUL_MAT op tests with unbound Q4_1/Q4_K weights
Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU
plugin fails to compile that form for some row counts with "clFinish,
error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on
the MUL_MAT cases added in #29869 (e.g. m=1000, n=2, k=1024). Model
weights use a u4 zero point and are not affected.
Report these cases as unsupported on GPU until the plugin is fixed.
Op tests check support before allocating, so the check matches unbound
weights only; model loading probes with a dummy buffer and keeps its
weights on the GPU.
* ggml-openvino: create FILL in the output type
translate_fill always built an f32 constant, so an f16 FILL produced
f32 data and the copy back overran the f16 output buffer. Use the
output type for the constant.
* ggml-openvino: reject CONCAT with a quantized type
Quantized inputs are dequantized when translated, so the backend cannot
write a quantized CONCAT output. Report it as unsupported, as for CPY
to a quantized type.
* ggml-openvino: handle the single recurrent state gather of build_rs
#29856 changed build_rs to gather all recurrent states with one GET_ROWS
on the s_copy leaf and take the ubatch and extra states as views of it.
The stateful path matched only the previous form, a GET_ROWS per view of
s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU
("is_axis_valid(axis, r)" in a Concat).
For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the
active-state gather, keep the rank-4 layout of reshapes that read a view
of it, and map the copy of the empty extra-state view to the single-slot
remainder writeback. Do not warn about the dynamic dim of empty views.
* openvino: align eltwise operand ranks to work around a GPU-plugin defect
* openvino: match the MoE fusion on the rank-3 stateful graph
* ggml-openvino: do not unsqueeze an RMS norm output in AlignEltwiseOperandRanks
The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract
whose operand ranks differ. In gemma-3 the lower-rank operand of the
post-attention residual add is the norm output, and unsqueezing it makes
the GPU plugin compute the layer wrongly: gemma-3 returns empty answers
on GPU with stateful execution.
Skip the rewrite when the lower-rank operand is an RMS norm output.
* docs : update OpenVINO validated models
---------
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
* llama: re-pool each shared k-pool rep once
With shared cells every pool is re-pooled, and since the pooled keys
are always scattered, the pools a seq_cp shares between sequences
wrote the same rep row from several scatter entries, a data race on
the CPU backend. Mark each rep once: the sharing sequences read the
same row through pool_cells.
* llama: assert whole-sequence seq_cp in the hybrid idx memory
The recurrent state is always copied whole whatever the range, and a
k-pool cell shared by a partial copy could carry two pool groupings
with a single pooled row. Every caller copies whole sequences, so
reject partial ranges instead of supporting them.
* llama: drop the k-pool cache_safe mode
With whole-sequence seq_cp, sequences sharing cells share their pools
too, so the pooled row of a shared rep is valid for all of them. Mark
each rep once in every ubatch instead of re-pooling everything while
cells are shared, which removes the sharing scan and the stale-all
workarounds in seq_rm, state_read and state_drop. seq_cp now only
stales the destination.
* hexagon: head-parallel flash_attn partitioning for row-split multicore
In row-split mode each core computes its output row shard of every
MUL_MAT, but flash_attn was previously partitioning by Q tokens
(flat qrow split) instead of by heads. This forced every core to
read the full KV cache (all n_kv_heads), negating the memory
bandwidth benefit of multicore on flash_attn.
Change both HMX and HVX flash_attn kernels to partition by KV heads
when n_kv_heads is divisible by n_cores: core i processes heads
[i*n_kv_heads/N, (i+1)*n_kv_heads/N) exclusively, reading only its
head shard of the KV cache. Falls back to the original token-block
split when n_kv_heads % n_cores != 0 (e.g. Gemma-4 with 2 KV heads
on 4 cores).
Controlled by GGML_HEXAGON_FA_HEAD_SPLIT (default 1 = on).
The flag is packed into bit 1 of the existing is_dst_fp32 kparams
byte to stay within the 128-byte kernel_params blob limit.
Measured gains at 4c row-split (PP t/s, ubatch=1024):
Qwen3-0.6B: 6977 -> 11026 (+58%)
llama-3.2-3B: 3717 -> 5522 (+49%)
Qwen3.5-4B: 2739 -> 2855 (+4%)
Gemma-4 MoE: no change (MoE FFN dominates, fallback path)
TG is unchanged (flash_attn is a small fraction of decode time
relative to the matmul+barrier cost per layer).
* hex-fa: cleanup kern_params and head-split selection
* hex-fa: add -fa-head-split option to run.py
* hex-mdev: update matmul solver to account for reduced work in row-split scenarios
* hex-mmid: better work splitting by expers in multi-dev scenarios
* hex-fa: update HMX gating based on the model/n-hvx/ctx-len sweep
* hex-fa: precompute softcap/scale on the host
* hexagon: flatten matmul into 2d to use HMX in multi-sequence
* hex-mm: cleanup kparams and use collapse to 3/4D -> 2D mapping
* hex-mm: fix typo in collapse fallback
* hex-mm: another pass at consistent naming for act tensors
* hex-mm: add support for colapsing dims in fused matmuls
* hex-build: fix WoS build errors
* hex-mm: make sure to enforce dst stride in can_collapse
* hex-fa: add a onliner commit for head-split check
* hex-fa: remove unused local head_split var
* hex-fa: tighten up can_split checks
* hex-mm: update unfused paths to use act instead src1
* hex-mm: make sure to check all dsts for splitting
* hexagon: fix the second weight chunk address in the batched HMX matmul prologue
* hexagon: F16 activation and ragged N in the HMX matmul
* hex-mm: tighten the ragged/split checks in mdev cases
* hex-mm: enable MM fusion for F16 activations
* hex-mm: pass tiled sizes to the solver in fused paths
* hex-mmid: remove scalar divs from expert mapping loops
* hex-mmid: proper cacheline safety enforcement for mdev splits
* hex-mm: improve solver for mdev split scanarios and tail handling
* hex-mm: remove redundant checks
* hex-mm: fix fused HMX MUL_MAT_NX drops the final partial tile for quantized weights
* hex-mm: better handling of ragged shapes (removes scalar memset of vtcm)
---------
Co-authored-by: ebateni <ebateni@qti.qualcomm.com>
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>