* dflash : move the DSpark markov head and backbone graphs to dspark.cpp
Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and
graph_mistral_large4 move verbatim. build_inp_target_features and
build_dspark_markov_head become static members of llama_model_dflash so
both files can use them.
* dflash : detect the head and the backbone once, split the loaders per file
load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2,
DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4).
The head, backbone and base layer loaders become member helpers in
dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph
dispatches on the backbone.
The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through
a helper in dflash2.cpp, right after their head tensors instead of in
the base layer loop; the other tensors keep their creation order. The
DFlash2 loader drops its copy of the conv group size check, which master
never reached.
* dflash : share the injection input, token embeddings and lm head lookups
build_inject_inp (target features + fc + encoder norm), get_tok_embd and
get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral
Large 4 graphs. build_inp_target_features is file-static again.
* dflash : split the DFlash decoder into a graph per head
graph_base creates the shared inputs and provides build_inject,
build_inp_tokens, build_layers and build_logits. graph_dflash
(dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and
graph_lilicorr (lilicorr.cpp) assemble them and add their head; the
selector, scorer and markov head builders move to the file of their
graph, and the DFlash2 local conv run by build_layers to dflash2.cpp.
build_arch_graph dispatches on the backbone, then on the head.
The op order and the tensor names are unchanged.
* speculative : split the DFlash driver into a class per head
common_speculative_impl_draft_dflash stays the DFlash implementation and
becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its
protected constructor only reads the draft metadata; init() then sets up
the contexts with the options of the draft (draft length, backend
sampling, device features, masked h_nextn). Each derived class overrides
draft() with the path of its head from the former draft(). DFlash2 also
overrides begin() to reseed its generator per request, and LiliCorr
overrides process() to store the context features of each injected
chunk, which its noise block carries as its state.
The factory keeps the current routing: draft-dspark picks DSpark,
draft-dflash picks DFlash2 or LiliCorr from the draft metadata and
DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs
the DSpark reader instead of a mix of both. sample_from_anchor is logged
by DSpark only.
* speculative : add the draft-dflash2 and draft-lilicorr types
draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts,
which draft-dflash keeps running too: it picks them from the draft
metadata. draft-dflash2 and draft-lilicorr must match the draft model,
draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a
DFlash draft is rejected, and a run uses at most one DFlash family type;
these errors leave speculative decoding uninitialized.
The type detected from the draft GGUF is now the concrete one, and
draft-dflash2 and draft-lilicorr download the dflash sidecar.
* docs : describe the DFlash2 and LiliCorr drafts
Add the DFlash2 and LiliCorr sections and the new types, and explain
that draft-dflash also runs DFlash2 and LiliCorr drafts.
* speculative : drop the unused DFlash is_mrope
Nothing reads it since the migration to common_batch (#29385), which
picks the number of M-RoPE position rows from the rope type of the draft.
* doc: update spec dec
* address review comments
* requirements : support Python 3.15
torch 2.11.0, numpy 2.4, matplotlib 3.10, pandas 2.3, pillow 11.3 and
cffi 1.16 have no Python 3.15 wheels, so the CI runner on 3.15 cannot
install the requirements.
Python 3.15 gets the first releases that ship 3.15 wheels through
version markers, and every version below 3.15 stays as is.
pyproject.toml now allows 3.15 and picks torch 2.15.0 there.
* requirements : apply review suggestions
numpy 2.5.4 from Python 3.14, matplotlib 3.11.2 and pandas 3.0.6 from
3.11, cffi 2.1.1 and pillow 12.3.0 for every version. cffi 1.16.0 has
no 3.14 wheel and fails to build there, so this also fixes the
tokenizer test requirements on 3.14.
* hex-bufs: combine queue and fence buffers
* hex-bufs: use n_threads for queue sizing
* hex-mmap: optimize buffer stacking at the host to avoid iova fragmentation
* hex-mmap: add simply retry for spurios mmap failures
* hex-mmap: use eager mode for extended mmap retries
* vulkan: add TOP_K_MASK
Assisted-by: Claude Opus 5.5
* vulkan: fix the TOP_K_MASK barrier and use the common offset helper
Assisted-by: Claude Opus 5.5
* vulkan: declare ggml_vk_top_k_mask in the common header
Assisted-by: Claude Opus 5.5
* spec : skip p_min for sampled MTP and DFlash2 drafts
Under rejection sampling the acceptance at a position is 1 - TV(p, q),
which the draft's top probability does not tell, so the p_min cutoff
dropped draft tokens the target would have accepted. Skip it when
MTP or DFlash2 samples the draft. draft-simple keeps it: each extra
draft token costs a full draft model pass.
* spec : replace the sampled-draft p_min skip with a warning
p_min behaves as on master again for every drafter and sampling mode.
Instead, log a warning when --spec-draft-p-min > 0 is used with
--spec-draft-sampling probabilistic, suggesting p_min 0: under
rejection sampling the draft's top probability does not tell how much
of the draft the target accepts. The cut is not skipped because with
MTP at long drafts it can still pay off.
* spec : warn about p_min only for drafters that use it
The n-gram drafters ignore p_min, and DSpark uses it as the cutoff of its confidence head, so the warning is skipped when only these are configured.
---------
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* grammar : limit the number of alternatives
* grammar : introduce an AST for grammar parsing
* grammar : revert AST refactor and alternatives limit
This reverts commits:
- 84f3b57c4d6c482e7187a30735d16b02de3784fa
- 393a707ebb7e6f798abdb8af34cb1bc102f4aaa6
* grammar: optimize big grammar matching by using left-factorization
* grammar : preserve character class boundaries during left factoring
* grammar : prevent infinite expansion during left factoring
* grammar : use ASCII characters in comments
* grammar : test empty suffixes after left factoring
* grammar : deduplicate left factoring logic
* grammar : add comments
* grammar : use number parameters and clarify RSS output
* grammar : clarify character class test names
* grammar : remove performance benchmark
* grammar : share immutable rules
---------
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
The kernel launches one block per row and 256-wide tile, so the
{2, 16777217} test case asks for 16777217 blocks of 256 threads. HIP
refuses a grid dimension above 2^32 threads and the ROCm jobs fail
with "invalid configuration argument".
The host now launches the grid in chunks of at most UINT32_MAX / 256
blocks and passes the chunk offset to the kernel. Real sizes fit in a
single chunk and keep their timings.
This commit removes the llama version test-cmake and instead uses the
version that is supplied by CMAKE_PREFIX_PATH.
The motivation for this is to avoid issues when bumping major versions.
The D=256 vec flash attention kernel with F8_E4M3 K and V uses 257
VGPRs (255 + 2 spilled) on gfx908, one over the limit of the check.
The spill is a perf issue only, like the Q4_0 D=256 vec kernel
already in the ignore list.
Both variants of the kernel join the ignore list.
[no ci]
* server: read a small temp file in the read_file tool test
test_tools_builtin_read_file read its own file, which grows with every
added test. On Windows the CRLF checkout puts it at 16664 bytes, over
the 16384 byte read limit, and the tool refuses it.
The test now writes a small file of its own and checks the exact
content read back.
* server: use a temp cwd in the x-tool-cwd tool test
test_tools_builtin_cwd_header read test_tools_builtin.py through the
x-tool-cwd header, which hits the same 16384 byte read limit on the
Windows CRLF checkout, and wrote its shell marker into the source tree.
The test now uses a temp directory as cwd, reads a small file from it
and checks the shell marker lands there.
The vision tests download their images from huggingface.co, which
resolves to a private address on the runners inside the HF network.
The media URL check refuses it with "host is not a public address"
and the server-cuda job fails.
The vision fixture enables media_allow_private_urls. The private
address check keeps its own coverage in
test_vision_image_url_private_address.
The same-origin tests over the LAN address block forever on the Metal
runners: the request to the host's own LAN address never returns and
has no timeout, so the job runs until the 6h limit.
get_lan_ip() now probes /health on that address with a short timeout
and returns None when the host cannot reach it, which skips the test.
The test requests themselves use the default request timeout so a
hang fails the test instead of stalling the job.
* fix(hexagon): make host build work with clang-cl on Windows ARM64
Two CMake fixes for building ggml-hexagon natively on Windows-on-ARM64
with x64-hosted clang-cl (validated on Snapdragon X Elite, discussion in
qualcomm/GenieX#1390):
- Some clang drivers (e.g. x64-hosted toolchains used on ARM64) don't
predefine the arch macro, which makes winnt.h fail with "No Target
Architecture". Define _M_ARM64/_ARM64_ explicitly for the host TUs.
- The Hexagon SDK's incs/stddef include (AEEStdDef.h and friends) poisons
the MSVC header chain when it leaks onto dependent TUs' include paths
via htp_iface's PUBLIC includes. Keep it PRIVATE to the interface stub
and system-level for the backend's own TUs (which include <AEEStdErr.h>
directly).
Originally validated 2026-08-26 as reported in the issue; applies
cleanly to 15759e0.
* hex-cmake: trim down excessive comments
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* llama : remove deprecated `llama_set_warmup`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : remove remaining deprecated API
Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* fit-params : remove stale `llama-ext.h` include
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : document `llama_model_dflash_selector_top_k`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : promote the NextN / speculative API from `llama-ext.h`
Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : polish the NextN API promotion
Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : prune deprecated API
* cont : bump RPC version
* cont : minor
* ggml : remove outdated documentation from ggml.h
The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:
- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`
The block also ended with 5 empty `TODO` sections.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)
* rpc : bump RPC protocol major version
The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.
The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: losewayy <1134285612@qq.com>
* CUDA: few-row MMQ for 2-24 src1 columns
New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.
* nit fixes
* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat
* few-row quantize: DS4 sum from the quantized values, as in #31247
* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build
* few-row MMQ: dispatch order MMVQ, few-row, MMQ
With --sleep-idle-seconds, a handler passes wait_until_no_sleep()
while the server is awake, then keeps using the server state, for
example to parse the body and tokenize. If the idle timer expires in
that window, the main loop enters sleep and the callbacks destroy the
model: the handler reads freed vocab memory, and a task posted after
that stays in the queue with nobody awake to process it.
Count the requests that passed wait_until_no_sleep() and are not done
yet, and do not enter sleep while the count is positive. The response
generator releases its count in its destructor, so every return path
of a handler gives it back.
Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
HunyuanVL resamples its position embedding on the host into a float
buffer sized for F32, then reads ggml_nbytes() of the tensor into it.
The graph never uses the tensor, so any type loads, and an F64 tensor
overflows the buffer at the first image request.
Require the position embedding to be present and F32 when loading the
HunyuanVL tensors.
The MiniCPM-V resampler ends with build_mm(mm_model_proj), whose output
width is ne[1], while n_mmproj_embd() returns ne[0]. Both are equal in
real models, the projection is square. A non square projection passes
the text model n_embd check and only aborts at the encode, on the
output buffer size check.
Return ne[1], so the width matches the graph output.
* mtmd: require the MLP projector and patch embedding tensors
The loader marks the patch embedding and the LLaVA projector tensors
as optional, while the graphs and n_mmproj_embd() dereference them
unconditionally. A GGUF without them crashes on a null tensor during
the warmup.
Load the patch embedding as required for the vision encoders that
start with it, which is all of them except GEMMA3NV, DEEPSEEKOCR and
DEEPSEEKOCR2. Load mm.0 and mm.2 as required for MLP, and mm.0, mm.1,
mm.3 and mm.4 for the Yi-type MLP_NORM variant, selected by mm.3.
The graph no longer tests mm.2 and the yasa2 assert on the patch
embedding is covered by the loader.
* mtmd: keep the Yi-type llava TODO
The Yi-type variant is still detected by the presence of mm.3, the
TODO on that detection stays.
The Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Ling3-VL graphs build their
input with a temporal merge that assumes a 2x2 patch grid, and the
Qwen2-VL merger reshapes by 4. Any other spatial_merge_size from the
GGUF reaches a GGML_ASSERT at the first image request and aborts the
server.
Reject a spatial_merge_size other than 2 when loading the hparams.
The SpacemiT compute case was commented out, so the op stayed on the scalar CPU path. Type dispatch stays with the other ops, and use_ref keeps the reference path on the generic CPU kernel.
Assisted-by: Codex
Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
* vulkan : Intel customized FA shader for prefill usage
* vulkan : adding check of max_bias/logit_softcap for Intel FA kernel path
* vulkan : add m16n16k16/SIMD32 support for Xe FA prefill shader
* vulkan : Intel FA shader and host refinement, with accuracy issues fixes
* vulkan : Limit the SLM usage to 48 KiB for Intel prefill FA shader
* vulkan : Update Intel FA shaders to handle cases with large head dimension K, unaligned Q head stride and multi head mask cases
* vulkan : Intel FA shader and host refinement to align with existing design, extreme cases handling, shape check relaxing.
* vulkan : Intel FA shader and host update to support q as tensor view, unaligned batch stride q. Add env variable for enforcing XE FA path for other vendor GPUs with 16x16x16 CM1 and SIMD32 support
* vulkan : Inte FA shader and host update to fix the potential SLM over 48 KiB issue, enhance the buffer access robust and add asymmetric qk/pv ration support.
* vulkan : Inte FA shader and host update to add SLM 48 KiB size check, remove unnecessary extensions and finetune the unroll/dont_unroll loops
* vulkan : Intel FA shader and host update to add SLM size check function, use real device max SLM size in shader specialization constants.
* vulkan : Intel FA shader and host update to consolidate host helper function, remove redundant buffer sync, bypass unused pipeline and buffer, fix subgroup shmem conflict and add negative case handling.
* vulkan : Intel FA host update to correct get_xe_fa_tuning_params() function log message, and add workgroupMemoryExplicitLayout/workgroupMemoryExplicitLayout16BitAccess prerequisite feature check.