* chat: allow tool calls inside reasoning blocks for Ministral-3 and Qwen3
Fixes#29248Fixes#20837
* chat: address review feedback for Ministral-3 reasoning end
- Remove <function= from Qwen3 (intentionally not used outside outdated coder models)
- Ministral-3: use choice between [/THINK] and peek([TOOL_CALLS]) instead of optional
* chat: add [TOOL_CALLS] to Ministral-3 thinking_end_tags
* chat: simplify Ministral-3 reasoning end choice
* chat: restore peek lookahead for Ministral-3 tool call start
* chat: keep [TOOL_CALLS] only in parser, not in thinking_end_tags
* cont : revert tool call tag removal and update tests
---------
Co-authored-by: Alen Požun <kukirin-ui@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
The thought channel is normally closed with <channel|>, but the model
may open it and go straight to a tool call without closing it. Stop the
reasoning at <channel|> OR <|tool_call>, so an unclosed thought does
not swallow the following tool call.
Fixes: #32231
Assisted-by: Qwen 3.8 27b
The qwen2vl graph passed NULL for the ffn_up/gate/down scales, so the
per-tensor NVFP4 scales loaded by the generic pass were never applied
and the model produced garbage (PPL ~1e6 on CUDA, NaN on CPU).
Pass the scales to build_ffn like qwen2 does.
Fixes#32101
* mtmd : allow custom video decoder
* also adapt to wllama requirements
* clean up
* add test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* dflash : move the DSpark markov head and backbone graphs to dspark.cpp
Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and
graph_mistral_large4 move verbatim. build_inp_target_features and
build_dspark_markov_head become static members of llama_model_dflash so
both files can use them.
* dflash : detect the head and the backbone once, split the loaders per file
load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2,
DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4).
The head, backbone and base layer loaders become member helpers in
dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph
dispatches on the backbone.
The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through
a helper in dflash2.cpp, right after their head tensors instead of in
the base layer loop; the other tensors keep their creation order. The
DFlash2 loader drops its copy of the conv group size check, which master
never reached.
* dflash : share the injection input, token embeddings and lm head lookups
build_inject_inp (target features + fc + encoder norm), get_tok_embd and
get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral
Large 4 graphs. build_inp_target_features is file-static again.
* dflash : split the DFlash decoder into a graph per head
graph_base creates the shared inputs and provides build_inject,
build_inp_tokens, build_layers and build_logits. graph_dflash
(dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and
graph_lilicorr (lilicorr.cpp) assemble them and add their head; the
selector, scorer and markov head builders move to the file of their
graph, and the DFlash2 local conv run by build_layers to dflash2.cpp.
build_arch_graph dispatches on the backbone, then on the head.
The op order and the tensor names are unchanged.
* speculative : split the DFlash driver into a class per head
common_speculative_impl_draft_dflash stays the DFlash implementation and
becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its
protected constructor only reads the draft metadata; init() then sets up
the contexts with the options of the draft (draft length, backend
sampling, device features, masked h_nextn). Each derived class overrides
draft() with the path of its head from the former draft(). DFlash2 also
overrides begin() to reseed its generator per request, and LiliCorr
overrides process() to store the context features of each injected
chunk, which its noise block carries as its state.
The factory keeps the current routing: draft-dspark picks DSpark,
draft-dflash picks DFlash2 or LiliCorr from the draft metadata and
DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs
the DSpark reader instead of a mix of both. sample_from_anchor is logged
by DSpark only.
* speculative : add the draft-dflash2 and draft-lilicorr types
draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts,
which draft-dflash keeps running too: it picks them from the draft
metadata. draft-dflash2 and draft-lilicorr must match the draft model,
draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a
DFlash draft is rejected, and a run uses at most one DFlash family type;
these errors leave speculative decoding uninitialized.
The type detected from the draft GGUF is now the concrete one, and
draft-dflash2 and draft-lilicorr download the dflash sidecar.
* docs : describe the DFlash2 and LiliCorr drafts
Add the DFlash2 and LiliCorr sections and the new types, and explain
that draft-dflash also runs DFlash2 and LiliCorr drafts.
* speculative : drop the unused DFlash is_mrope
Nothing reads it since the migration to common_batch (#29385), which
picks the number of M-RoPE position rows from the rope type of the draft.
* doc: update spec dec
* address review comments
* requirements : support Python 3.15
torch 2.11.0, numpy 2.4, matplotlib 3.10, pandas 2.3, pillow 11.3 and
cffi 1.16 have no Python 3.15 wheels, so the CI runner on 3.15 cannot
install the requirements.
Python 3.15 gets the first releases that ship 3.15 wheels through
version markers, and every version below 3.15 stays as is.
pyproject.toml now allows 3.15 and picks torch 2.15.0 there.
* requirements : apply review suggestions
numpy 2.5.4 from Python 3.14, matplotlib 3.11.2 and pandas 3.0.6 from
3.11, cffi 2.1.1 and pillow 12.3.0 for every version. cffi 1.16.0 has
no 3.14 wheel and fails to build there, so this also fixes the
tokenizer test requirements on 3.14.
* hex-bufs: combine queue and fence buffers
* hex-bufs: use n_threads for queue sizing
* hex-mmap: optimize buffer stacking at the host to avoid iova fragmentation
* hex-mmap: add simply retry for spurios mmap failures
* hex-mmap: use eager mode for extended mmap retries
* vulkan: add TOP_K_MASK
Assisted-by: Claude Opus 5.5
* vulkan: fix the TOP_K_MASK barrier and use the common offset helper
Assisted-by: Claude Opus 5.5
* vulkan: declare ggml_vk_top_k_mask in the common header
Assisted-by: Claude Opus 5.5
* spec : skip p_min for sampled MTP and DFlash2 drafts
Under rejection sampling the acceptance at a position is 1 - TV(p, q),
which the draft's top probability does not tell, so the p_min cutoff
dropped draft tokens the target would have accepted. Skip it when
MTP or DFlash2 samples the draft. draft-simple keeps it: each extra
draft token costs a full draft model pass.
* spec : replace the sampled-draft p_min skip with a warning
p_min behaves as on master again for every drafter and sampling mode.
Instead, log a warning when --spec-draft-p-min > 0 is used with
--spec-draft-sampling probabilistic, suggesting p_min 0: under
rejection sampling the draft's top probability does not tell how much
of the draft the target accepts. The cut is not skipped because with
MTP at long drafts it can still pay off.
* spec : warn about p_min only for drafters that use it
The n-gram drafters ignore p_min, and DSpark uses it as the cutoff of its confidence head, so the warning is skipped when only these are configured.
---------
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* grammar : limit the number of alternatives
* grammar : introduce an AST for grammar parsing
* grammar : revert AST refactor and alternatives limit
This reverts commits:
- 84f3b57c4d6c482e7187a30735d16b02de3784fa
- 393a707ebb7e6f798abdb8af34cb1bc102f4aaa6
* grammar: optimize big grammar matching by using left-factorization
* grammar : preserve character class boundaries during left factoring
* grammar : prevent infinite expansion during left factoring
* grammar : use ASCII characters in comments
* grammar : test empty suffixes after left factoring
* grammar : deduplicate left factoring logic
* grammar : add comments
* grammar : use number parameters and clarify RSS output
* grammar : clarify character class test names
* grammar : remove performance benchmark
* grammar : share immutable rules
---------
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
The kernel launches one block per row and 256-wide tile, so the
{2, 16777217} test case asks for 16777217 blocks of 256 threads. HIP
refuses a grid dimension above 2^32 threads and the ROCm jobs fail
with "invalid configuration argument".
The host now launches the grid in chunks of at most UINT32_MAX / 256
blocks and passes the chunk offset to the kernel. Real sizes fit in a
single chunk and keep their timings.
This commit removes the llama version test-cmake and instead uses the
version that is supplied by CMAKE_PREFIX_PATH.
The motivation for this is to avoid issues when bumping major versions.
The D=256 vec flash attention kernel with F8_E4M3 K and V uses 257
VGPRs (255 + 2 spilled) on gfx908, one over the limit of the check.
The spill is a perf issue only, like the Q4_0 D=256 vec kernel
already in the ignore list.
Both variants of the kernel join the ignore list.
[no ci]
* server: read a small temp file in the read_file tool test
test_tools_builtin_read_file read its own file, which grows with every
added test. On Windows the CRLF checkout puts it at 16664 bytes, over
the 16384 byte read limit, and the tool refuses it.
The test now writes a small file of its own and checks the exact
content read back.
* server: use a temp cwd in the x-tool-cwd tool test
test_tools_builtin_cwd_header read test_tools_builtin.py through the
x-tool-cwd header, which hits the same 16384 byte read limit on the
Windows CRLF checkout, and wrote its shell marker into the source tree.
The test now uses a temp directory as cwd, reads a small file from it
and checks the shell marker lands there.
The vision tests download their images from huggingface.co, which
resolves to a private address on the runners inside the HF network.
The media URL check refuses it with "host is not a public address"
and the server-cuda job fails.
The vision fixture enables media_allow_private_urls. The private
address check keeps its own coverage in
test_vision_image_url_private_address.
The same-origin tests over the LAN address block forever on the Metal
runners: the request to the host's own LAN address never returns and
has no timeout, so the job runs until the 6h limit.
get_lan_ip() now probes /health on that address with a short timeout
and returns None when the host cannot reach it, which skips the test.
The test requests themselves use the default request timeout so a
hang fails the test instead of stalling the job.
* fix(hexagon): make host build work with clang-cl on Windows ARM64
Two CMake fixes for building ggml-hexagon natively on Windows-on-ARM64
with x64-hosted clang-cl (validated on Snapdragon X Elite, discussion in
qualcomm/GenieX#1390):
- Some clang drivers (e.g. x64-hosted toolchains used on ARM64) don't
predefine the arch macro, which makes winnt.h fail with "No Target
Architecture". Define _M_ARM64/_ARM64_ explicitly for the host TUs.
- The Hexagon SDK's incs/stddef include (AEEStdDef.h and friends) poisons
the MSVC header chain when it leaks onto dependent TUs' include paths
via htp_iface's PUBLIC includes. Keep it PRIVATE to the interface stub
and system-level for the backend's own TUs (which include <AEEStdErr.h>
directly).
Originally validated 2026-08-26 as reported in the issue; applies
cleanly to 15759e0.
* hex-cmake: trim down excessive comments
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* llama : remove deprecated `llama_set_warmup`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : remove remaining deprecated API
Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* fit-params : remove stale `llama-ext.h` include
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : document `llama_model_dflash_selector_top_k`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : promote the NextN / speculative API from `llama-ext.h`
Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : polish the NextN API promotion
Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : prune deprecated API
* cont : bump RPC version
* cont : minor
* ggml : remove outdated documentation from ggml.h
The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:
- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`
The block also ended with 5 empty `TODO` sections.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)
* rpc : bump RPC protocol major version
The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.
The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: losewayy <1134285612@qq.com>