Commit Graph
1379 Commits
Author SHA1 Message Date
eddiechen 955c4e8ee5 server : return 400 when grammar and ignore_eos are on (#32377)
* sampling : fix crash when both ignore_eos and grammar are on

* server : rework, throw 400 when grammar and ignore_eos are on

* sampling : move the check into the sampler (review suggestion)

* tests : remove redundant tests
2026-11-17 17:03:21 +01:00
anujj 8777c0c833 Title: CUDA: few-row MMQ for MUL_MAT_ID (#32065)
* CUDA: few-row MMQ for MUL_MAT_ID

* CUDA: few-row MMQ for MUL_MAT_ID from 2 tokens on DGX Spark

* CUDA: few-row MUL_MAT_ID: one launcher, tighter pair count, more tests

* CUDA: few-row MUL_MAT_ID: dispatch order MMVQ, few-row, MMQ and review fixes
2026-11-17 21:01:40 +05:30
Giorgio Sarno ebc9fb933e CUDA: use block reductions for whole-row pool_1d (#31855)
* CUDA: use block reductions for whole-row pool_1d

Assisted-by: Claude Code

* CUDA: template pool1d row kernel by ggml_op_pool
2026-11-17 15:51:15 +01:00
08a9b35893 llama : keep the k-pool layout prefix on sequence edits (#32137)
* llama : keep the k-pool layout prefix on sequence edits

Assisted-by: Claude Code

* llama : address review of the k-pool layout update

* Append and edit share one branch, using stale > sq.pos_min.
* LLAMA_KPOOL_VERIFY=1 checks every update against a full rebuild. The scan moved into kpool_scan() so the check can reuse it.
* The test turns the check on for its rollback context, skips -sm tensor, and drops kimi-k31.
* Two comments that were no longer true are fixed.

Assisted-by: Claude Code

* llama : check the k-pool layout update in debug builds instead of LLAMA_KPOOL_VERIFY

* llama : remove the k-pool layout verification

* llama : inline the k-pool scan again

* remove dangling new line

* do not search twice

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-17 22:37:25 +08:00
Aman Gupta d3064fddce convert : fix FP8 qwen4exp checkpoints (n-gram table scale, block-scaled FP8) (#32476)
* convert : fix FP8 n-gram table export for qwen4exp

* convert : dequantize block-scaled FP8 weight_scale_inv in ModelOpt checkpoints
2026-11-17 22:25:01 +08:00
Xuan-Son Nguyen d92ea9bb12 mtmd: support Minimax-Music3 (#32435)
* mtmd: support Minimax-Music3

* fix and clean up
2026-11-17 12:53:19 +01:00
Xuan-Son Nguyen 8d1599fca1 chat: add tts_dict for kokoro G2P template (#32397)
* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n
2026-11-17 12:52:50 +01:00
Neo Zhang dde9de82cb sycl : Fix FA mkl issue: a mask broadcast over heads or batches (#31536)
* fix the SYCL fa with mkl error of a mask broadcast over heads or batches; add argument cache-ram to start-svr.sh

* add argument cache-ram to win-start-svr.bat

* fix format issue, remove --cache-ram in script: move to another PR
2026-11-17 10:50:32 +02:00
Nikhil Jain 140a279874 webgpu : Implement timestep embedding (#31695)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* resolve merge conflicts

* save state

* save kernel

* add kernel back in

* fix merge issues with kernels

* updates

* update tests

* change invocation count and workgroup to be u32

* clang format

* clang format

* update docs

* format

* remove extra file
2026-11-17 10:41:37 +02:00
Pascal e8fbcad855 tests: print the exception message in test-llama-context (#32428)
A case that throws only reported "unhandled exception", which hides
the cause of the intermittent failures on the Vulkan Apple runner.
2026-11-16 21:53:32 +01:00
Aldehir Rojas 4ddf2b9470 chat : implement inkling parser (#32381) 2026-11-16 19:48:24 +01:00
Rudransh Shekhar 7804be0634 arg : allow up to llama_max_devices() configs for --tensor-split and --fit-target (#13293) (#31435) 2026-11-16 17:07:45 +01:00
Xuan-Son Nguyen ef6bc6f367 jinja: impl scope-chain key lookup (#32396)
* jinja: impl scope-chain key lookup

* add test for call_statement lambda
2026-11-16 15:48:46 +01:00
Niklas Wenzel 62a6d3a302 llama : migrate to llama_batch_ext (#32356)
* llama : migrate to llama_batch_ext

* mtmd : migrate zonos3-mini code

Added in https://github.com/ggml-org/llama.cpp/pull/32020
2026-11-16 15:16:12 +01:00
Jeff Bolz ef3848f9d7 vulkan: Support split_k for conv3d. (#29929)
This tries to use the 256x128 shape (coopmat2-only) with split_k when
it appears profitable. It reuses the mul_mat split_k reduce shader.
2026-11-16 15:25:12 +02:00
Xuan-Son Nguyen e1925b3695 jinja: support json sort_keys=true (#32366) 2026-11-16 12:28:19 +01:00
PascalandRuben Ortlam adf554d239 vulkan: make the radix top-k tie selection deterministic (#31931)
* vulkan: make the radix top-k tie selection deterministic

The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.

Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.

* optimize shader for AMD, Intel and Nvidia

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-16 11:41:44 +01:00
liminfei-amd f477c1b90a cuda : fix MUL_MAT_ID abort with odd src0 columns on AMD (#32385)
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.

Fixes #30576

Assisted-by: GitHub Copilot

Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
2026-11-16 17:53:06 +08:00
Titaniumtown 21fa9c043f kv-cells: avoid an O(n_cells) scan per sequence removal (#32380) 2026-11-16 10:52:09 +01:00
Pascal a946eb541a server: answer reasoning_effort levels the template rejects with 400 (#32332)
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.

Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.

Also trim the output_config.effort test to a single level.
2026-11-16 06:55:56 +01:00
Xuan-Son NguyenandAldehir Rojas b4c0e00a4f jinja: add 4 missing features (#32352)
* jinja: add 4 missing features

* apply suggestion

Co-authored-by: Aldehir Rojas <hello@alde.dev>

* fix error msg

---------

Co-authored-by: Aldehir Rojas <hello@alde.dev>
2026-11-16 00:40:44 +01:00
Xuan-Son Nguyen 7ae3c6b231 mtmd: add zonos3-mini TTS model (#32020)
* zonos3 init

* add audio gen pipeline

* fix string_format

* add default voice as gguf metadata

* better ref

* support voice clone

* add delivery params

* optimize speed

* nits

* nits 2

* add text normalization

* nits
2026-11-15 19:44:44 +01:00
Xuan Son Nguyen f4d0673917 Revert "jinja: add 4 missing features"
This reverts commit a681a53b60.
2026-11-15 19:35:31 +01:00
Xuan Son Nguyen a681a53b60 jinja: add 4 missing features 2026-11-15 19:33:06 +01:00
Roshan Kumar 56fd8a375f server : ignore empty stop strings (#32093) 2026-11-15 17:23:05 +01:00
Xuan-Son Nguyen dd099a6bf3 chat: fix inkling 2 begin/end of image tokens placement (#32294)
* chat: fix inkling 2 begin/end of image tokens placement

* add test
2026-11-15 09:41:44 +01:00
Xuan-Son Nguyen 429460dc0e server: refactor, add server_slot_sched, prompt running hash (#32154)
* server: refactor, add server_slot_sched, prompt running hash

* use sha256 hash
2026-11-15 09:37:17 +01:00
Ed Addario 5a4afd8077 tests: Fail test-backend-ops and exit with error status 1 when missing and/or unsupported types and/or unsupported backends are provided in the command line (#30279)
* Fail tests on missing or unsupported type

* Fix output

* Remove whitespace

* Fail if no tests ran regardless of backend support

* Revert unnecessary cosmetic change
2026-11-15 08:26:23 +01:00
Tarek Dakhran e574a17b06 model : metal : workaround fusion for LFM3 (#32296)
* lfm3 : put the normed tensor first in residual adds to enable norm fusion

Assisted-by: Claude Opus 5.5

* tests : update Metal fusion baseline for lfm3

Assisted-by: Claude Opus 5.5
2026-11-15 01:05:20 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5c008e0008 feat: add Inkling 2 support (text + vision) (#32217)
* convert : add Inkling2 text GGUF conversion

Assisted-by: Codex

* llama : implement Inkling2 text inference

Assisted-by: Codex

* tests : cover Inkling2 text support

Assisted-by: Codex

* conversion : dequantize packed Inkling2 experts in row chunks

Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.

Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.

Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.

Assisted-by: Codex

* convert : add Inkling2 vision mmproj support

Assisted-by: Codex

* mtmd : implement Inkling2 still-image encoder

Assisted-by: Codex

* inkling2 : scope image loading

Assisted-by: Codex

* inkling2 : enable flash attention with folded bias and sinks

Assisted-by: Codex

* cuda : route per-head flash attention masks through vec

Assisted-by: Codex

* mtmd : support Inkling2 temporal frame means

Assisted-by: Codex

* style : restore shared image comment

Assisted-by: Codex

* tests : enlarge Inkling2 context fixture

Assisted-by: Codex

* inkling2 : load per-layer expert feed-forward widths

Assisted-by: Codex

* tests : add Inkling2 Metal fusion baselines

Assisted-by: Codex

* inkling2 : group and align registry entries

Assisted-by: Fable

* convert : regenerate the Inkling2 tokenizer hash with the update script

Assisted-by: Fable

* conversion : revert the print_registered_models change

Assisted-by: Fable

* conversion : sort the Inkling2 model map entries

Assisted-by: Fable

* convert : map Inkling2 vision checkpoint names in the tensor table

Assisted-by: Fable

* convert : map Inkling2 text checkpoint names in the tensor table

Assisted-by: Fable

* tests : document the Inkling2 packed expert conversion test

Assisted-by: Fable

* tests : cover Inkling2 checkpoint and transformers tensor names

Assisted-by: Fable

* llama : place Inkling2 fields and list entries by file convention

Assisted-by: Fable

* gguf-py : align Inkling2 tensor mapping comments

Assisted-by: Fable

* mtmd : place Inkling2 entries by file convention

Assisted-by: Fable

* tests : place Inkling2 cases after the existing ones

Assisted-by: Fable

* cuda : index per-head flash attention masks in the tile and mma kernels

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : project the relative bias before the gather, drop arch branches

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* convert : require Inkling2 hparams, add inkling2 pre-tokenizer

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* mtmd : cap Inkling2 image size, drop video frame averaging

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : fix CI build, Meta device assert and Windows conversion test

Assisted-by: Claude

* tests : use the default sliding window for the Inkling2 test model

Assisted-by: Claude

* simplify mtmd part

* give _v arch for vision

* remove alpha channel handling, it should be a follow up

* explicit rule about audio + vision models

* add audio support

* rm audio

* rm gguf mtmd rope base

* address review comment

* nits

* do not normalize embd from mtmd

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-15 00:51:31 +01:00
Pranav DhinakarandMax Krasnyansky dca216ad63 hexagon: add TIMESTEP_EMBEDDING HVX kernel (#32228)
* hexagon: add TIMESTEP_EMBEDDING HVX kernel

* hex-timestep: add dma, improve kparams, and work-splits, and tracing

* timestep: add testcases that cover WAN2.2

* hex-timestep: relax the gates to cont_rows

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-14 15:03:31 -08:00
Jeff BolzandRuben Ortlam d20426bbbe vulkan: optimize matrix multiplication for narrow N (#31990)
* vulkan: optimize matrix multiplication for narrow N

Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.

* Attempt to handle wave64

* add n=16 tile for cm1

* apply RDNA split_k heuristic to narrow_n generally

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-14 23:36:05 +01:00
Tarek Dakhran 0cc522e45d chat : allow undeclared tool calls in LFM2 tool-call grammar (#32164)
* chat : honor OpenAI tool strict flag in LFM2 tool-call grammar


* chat : drop strict, always allow undeclared LFM2 tool calls
2026-11-14 23:34:18 +01:00
3c311796f2 chat : allow tool calls inside reasoning blocks for Ministral-3 (#32271)
* chat: allow tool calls inside reasoning blocks for Ministral-3 and Qwen3

Fixes #29248
Fixes #20837

* chat: address review feedback for Ministral-3 reasoning end

- Remove <function= from Qwen3 (intentionally not used outside outdated coder models)
- Ministral-3: use choice between [/THINK] and peek([TOOL_CALLS]) instead of optional

* chat: add [TOOL_CALLS] to Ministral-3 thinking_end_tags

* chat: simplify Ministral-3 reasoning end choice

* chat: restore peek lookahead for Ministral-3 tool call start

* chat: keep [TOOL_CALLS] only in parser, not in thinking_end_tags

* cont : revert tool call tag removal and update tests

---------

Co-authored-by: Alen Požun <kukirin-ui@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
2026-11-14 23:33:35 +01:00
Cristian Adam 1c2adc75f6 chat : allow Gemma4 tool call during thought (#32232)
The thought channel is normally closed with <channel|>, but the model
may open it and go straight to a tool call without closing it. Stop the
reasoning at <channel|> OR <|tool_call>, so an unclosed thought does
not swallow the following tool call.

Fixes: #32231

Assisted-by: Qwen 3.8 27b
2026-11-14 23:32:19 +01:00
miyan a956a852c9 Fix ctz64 on 32bit (#32273) 2026-11-14 19:06:52 +01:00
Jhen-Jie HongandXuan Son Nguyen e3483e40b0 mtmd : allow custom video decoder (#32188)
* mtmd : allow custom video decoder

* also adapt to wllama requirements

* clean up

* add test

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-14 14:12:07 +01:00
HaoJun ZHANGandXuan Son Nguyen 4f2187e73d jinja : reduce template rendering overhead (#32023)
* jinja : reduce value allocation and lookup overhead

Assisted-by: GPT-6 Astra

* jinja : address rendering review feedback

Assisted-by: GPT-6 Astra

* jinja : test multipart strings and simplify lookup

Assisted-by: GPT-6 Astra

* fix "undefined hint"

* fix divisibleby 0

* clean up

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-14 13:34:20 +01:00
Hera 0cce7580b8 mtmd : avoid padding-only Whisper audio chunks (#30403)
Assisted-by: Claude Opus 5
2026-11-14 13:33:59 +01:00
Ehsan BateniandMax Krasnyansky 61dbf49228 Hexagon: add support for IQ2_XXS and NVFP4 (#31954)
* ggml-hexagon: IQ2_XXS and NVFP4 tiled repack layouts

* ggml-hexagon: IQ2_XXS HVX matmul kernel

* ggml-hexagon: NVFP4 HVX matmul kernel

* ggml-hexagon: NVFP4 per-tensor scale

* ggml-hexagon: HMX paths for IQ2_XXS and NVFP4

* ggml-hexagon: supports_op coverage for NVFP4 and IQ2_XXS

* fix HMX IQ2_XXS dequant scale underflow in fp16

* hexagon: IQ2_XXS 2-gather tile decode with compact grid, small-weight tests

* IQ2_XXS vpopcount sign parity and gather pipelining

* pipeline IQ2_XXS HMX dequant gathers and add large_w tests

* hex-iq2-xxs: rebase and cleanup

Remove unused helper functions.
Add lut_size to the overall vtcm required size.
Trim down (way way down) the new tests.

* hex-iq2-xxs: actually trim down the tests

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 20:57:05 -08:00
Todor BoinovskiandMax Krasnyansky ded522f830 hexagon: add support for SSM_SCAN op (#31921)
* hexagon: add SSM_SCAN Op support

* hex-ssm-scan: fix multi-device spliting

* hex-ssm-scan: remove redundant checks and make kernel use host config

* hex-ssm-scan: remove direct ddr access

* hex-ssm-scan: remove DMA ordering

* hex-ops: update supported ops

* hex-ssm-scan: rewrite the dma pipeline to match other ops (dbl.buf, etc)

* hex-ssm-scan: remove all fallbacks, only dma/vtcm paths

* hex-ssm-scan: add kernel_type to kparams and remove redundant recomputes

* hex-ssm-scan: avoid stalls in the overall pipeline (dma flush between chunks)

* hex-ssm-scan: vectorize softplus

* hex-ssm-scan: optimize hvx loops

* hex-ssm-scan: add HMX support

* hex-ssm-scan: add support for GGML_HEXAGON_SSM_SELECT

* hex-ssm-scan: cleanup kernel names

* hex-ssm-scan: simplify mdev checks

* hex-mdev: improve mdev row-split solver

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 12:27:51 -08:00
anujj 31953bcf74 ggml : add BF16 support to RMS_NORM (CPU, CUDA) (#32135)
* ggml : add BF16 support to RMS_NORM (CPU, CUDA)

* cuda : use one helper for the F32/BF16 RMS_NORM launches
2026-11-13 15:41:17 +02:00
o7siandPiotr Wilkin 108f73adfe grammar : optimize matching with left factoring (#29557)
* grammar : limit the number of alternatives

* grammar : introduce an AST for grammar parsing

* grammar : revert AST refactor and alternatives limit

This reverts commits:
- 84f3b57c4d6c482e7187a30735d16b02de3784fa
- 393a707ebb7e6f798abdb8af34cb1bc102f4aaa6

* grammar: optimize big grammar matching by using left-factorization

* grammar : preserve character class boundaries during left factoring

* grammar : prevent infinite expansion during left factoring

* grammar : use ASCII characters in comments

* grammar : test empty suffixes after left factoring

* grammar : deduplicate left factoring logic

* grammar : add comments

* grammar : use number parameters and clarify RSS output

* grammar : clarify character class test names

* grammar : remove performance benchmark

* grammar : share immutable rules

---------

Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
2026-11-13 13:52:21 +02:00
Awss 5676dfbe01 chat : accept </ifm|think> as K2 Horizon closer at medium/low effort (#30335) 2026-11-13 00:11:58 +01:00
Georgi Gerganov 8ca5a8008b llama : prune (#30565)
* llama : remove deprecated `llama_set_warmup`

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : remove remaining deprecated API

Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* fit-params : remove stale `llama-ext.h` include

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : document `llama_model_dflash_selector_top_k`

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : promote the NextN / speculative API from `llama-ext.h`

Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* llama : polish the NextN API promotion

Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-12 20:59:09 +02:00
Georgi Gerganovandlosewayy 29dbbad2b1 ggml : prune (#29801)
* ggml : prune deprecated API

* cont : bump RPC version

* cont : minor

* ggml : remove outdated documentation from ggml.h

The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:

- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`

The block also ended with 5 empty `TODO` sections.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)

* rpc : bump RPC protocol major version

The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.

The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

---------

Co-authored-by: losewayy <1134285612@qq.com>
2026-11-12 20:58:24 +02:00
anujj 0fb8f15770 Title: CUDA: few-row MMQ for BS 2-24 (#31533)
* CUDA: few-row MMQ for 2-24 src1 columns

New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.

* nit fixes

* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat

* few-row quantize: DS4 sum from the quantized values, as in #31247

* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build

* few-row MMQ: dispatch order MMVQ, few-row, MMQ
2026-11-12 21:12:45 +08:00
Konrad Moren 0d69255c52 CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints (#26048)
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back

* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ

* ggml-cuda: Tests for the fused scale

* ggml-cuda: reuse existing graph to test fusion in mmq epiloque

* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency

* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions

* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
2026-11-12 10:28:36 +01:00
Aldehir Rojas 93cf736a8d chat : refactor gemma4 to build loose schema rules and clean up (#31915)
* chat : build a schema-shaped PEG parser for Gemma 4 tool arguments

* chat : enforce only the key structure of Gemma 4 tool arguments

* chat : remove old gemma4 workarounds

* cont : add node.as<>() to simplify static casts

* cont : add depth_guard and use numerical rule names

* cont : clean up

* cont : extract gemma 4 parsers

* cont : clean up

* cont : clean up

* cont : remove dynamic cast
2026-11-12 09:44:42 +02:00
Xuan-Son Nguyen 664f031f49 ci: move test models to ggml-org-dev (#32096)
* ci: move test models to ggml-org-dev

* fix test spec
2026-11-11 21:17:16 +01:00