Commit Graph
1401 Commits
Author SHA1 Message Date
Pedro Cuenca 29b52d398b model : Inkling 2 Mini (#32694)
* Conversion

* Separate model_type

* Add gate_rank to allow building from metadata while testing

* Base model

* Loop graph

* Support the new is_varloop attribute

* Adopt naming scheme from https://github.com/ggml-org/llama.cpp/pull/32588

* Rename to varying
2026-11-20 23:01:07 +01:00
Xuan-Son NguyenandPascal 9197945004 convert: qwen3-asr: put "language ..." output into thinking block (#32571)
* convert: qwen3-asr: put "language ..." output into thinking block

* convert: qwen3-asr: read the chat template from models/templates

The converter embedded its own copy of Qwen3-ASR.jinja. Read the
bundled file instead, as the Kimi-K3 and Kokoro converters do, so
the GGUF and test-chat share a single template.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-20 20:54:57 +01:00
Tarek Dakhran a83d68b562 metal : allow the norm chain in either operand of the fused MUL/ADD (#32300)
Assisted-by: Claude Opus 5.5
2026-11-20 19:55:44 +01:00
Xuan-Son Nguyen 98709fd82b server: add more decision tests (#32578)
* server: add more decision tests

* fix ci

* also test audio
2026-11-20 18:45:31 +01:00
nicholas-mainardi-domyn 55d323507d model: add Domyn Edge 4B Model (#32559)
* feat: add domyn-edge model

* Generate tokenizer hash for conversion + fix tests

* Address changes according to code-review skill

* catch ValueError in convert_hf_to_gguf_update.py

* renegerate fusion test baseline + fix model naming style
2026-11-20 09:51:41 +02:00
Nikhil Jain d7d1b632da webgpu : fix 2D indexing for copy kernel (#32576)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* resolve merge conflicts

* Add 2d indexing to copy kernel

* move test
2026-11-20 09:45:44 +02:00
Ruben OrtlamandJeff Bolz 16f80bd0c0 vulkan: chunked GATED_DELTA_NET implementation (#30207)
* vulkan: add cm2 chunked GATED_DELTA_NET shader

* fix shmem bank conflict bottleneck

* optimize further

* add coopmat maintenance1 variant

* optimize for RDNA4

* add gmem version RDNA4

* optimize for Ampere

* fix nvidia shmem overuse

* fix cm2 validation errors

* improve cm2 perf

* improve cm1 perf

* clean up

* enable by default

* show cm maintenance1 marker in device info

* properly check for cm_m1 features

* add tests

* add v32 mode, improve chunked condition

* allow other subgroup sizes

* vulkan: specialize the GDN CM2 workgroup size

Assisted-by: Codex

* vulkan: coopmat2 GDN implementation

* work around Nvidia shader compiler crash

* remove shared shader cm2 code

---------

Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
2026-11-20 09:38:14 +02:00
Leon 2d8d003937 webgpu : add COL2IM_1D support (#31708) 2026-11-20 09:36:58 +02:00
Johannes Gäßler b94959a64d CUDA: implement support for duplicate expert IDs (#32505)
implement support for duplicate expert IDs
2026-11-19 23:21:24 +01:00
anujj 7fce5439fe ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA) (#32387)
* ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA)

* cuda : SET_ROWS from BF16 into quantized types and F8_E4M3

* cuda : fuse BF16 rope into a BF16 cache

* tests : trim the BF16 rope cases to one per code path
2026-11-19 23:43:55 +05:30
Bhargav Krish c7468107ef cuda: Scatter kernel for set_n (#32489)
* cuda: add scatter for batched tensor writes

* update test case to mirror get_n test case

* save restore timing logs

* refactor code into batch-copy.cu file
2026-11-19 20:14:47 +05:30
ac830a9cbb ggml : add backend scheduler tests (#27258)
* Adds initial scheduler stress tests

* Adds new sleep op to simplify scheduler testing

* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321

* Adds experimental vulkan GGML_OP_SLEEP

* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.

* Cleanup: Removes CUDA specific comments and nvtx-markers

* Fix: forgot to add sleep specific test case

* Fix oversights identified by CI

* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior

* Fix: Early exit in CPU-only settings (cause for graviton CI fails)

* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds

* Exclude ACCEL devices from new tests due to incompatibility

* Fixes vulkan test failure in llvmpipe

* Adds asserts, improves comments and some checks

* Fixes nitpicks in white-space and variable naming

* Adds comments that certain vk drivers could time OP_SLEEP incorrectly

* Drops limitiation to CPU/GPU in test-sleep

* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)

* Implements feedback for OP_SLEEP to work in place (no copy)

* Apply non-code suggestions  from JohannesGaesslers code review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail

* Make clang-tidy happy

* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)

* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name

* Drops unnecessary shader_clock.comp test and guards

* Replaces sched_ prefixes with helper_ prefix

* Moves test suite to new RAII types for better memory management

* Adds sleep option to test_chain_all_backends

* Adds sleep option to first-run arm of the y-shaped test

* Fix switchcase order for legibility in stress_test_dag

* Simplfies stress_test_linked_list via double for-loop

* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)

* Fix RPC Proto patch due to GGML_OPS bump

* Adds ggml-cpp.h treatment to test-sleep.cpp

* Adds TODO pointers for subsequent additional tests.

* Apply suggestion to ggml.h

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)

* Apply batched suggestions from gg code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Completes callsite change of gg-suggestion

* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-19 16:40:13 +02:00
Jeff Bolz 573aa89de6 vulkan: Implement stream-k MUL_MAT (#28528)
* vulkan: Implement stream-k MUL_MAT

Stream-k tries to split chunks of 256 K elements evenly across all SMs.
Workgroups that finish a tile write to dst, others write to scratch, and
there's a resolve shader similar to split-k.

The shader code is implemented for scalar+cm1+cm2, but only currently enabled
for cm2. The heuristic for enabling on coopmat2 is pretty simple due to the
large tile size (one workgroup per SM), but would be more complex for the tile
sizes being used for cm1, and it doesn't make sense for me to tune that on
NVIDIA devices.

Assisted-by: Codex

* address review feedback
2026-11-19 08:14:14 +02:00
Aman Gupta d4f5c26049 CUDA: use 64-bit dst offsets in mul_mat_q (#32374)
The MMQ kernels computed the dst tile offset (jt*J*stride_col_dst) in
32-bit int. For a dst with more than 2^31 elements, column tiles that
start past INT_MAX/nrows wrap around and write out of bounds.

This happens with an all-logits output head, e.g. llama-perplexity
-c 32768 -b 32768 -ub 16384 on a 248320-vocab model: the q8_0 output
matmul has a 248320 x 16384 dst (4.07e9 elements) and aborts with an
illegal memory access.

Widen the offsets in mul_mat_q and the stream-k fixup to int64_t, and
add a (disabled by default, ~9 GB) test-backend-ops case.

Assisted-by: Claude Opus 5.5
2026-11-19 08:11:44 +02:00
mahounairoandAlexey Kopytko 0a76757469 llama : fix sticky NaN in recurrent state (#31913)
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-19 08:08:12 +09:00
Aman Gupta ff6b8d1761 ggml-cpu : use all threads for large GET_ROWS (#32514) 2026-11-18 23:39:30 +08:00
varunandvarun 98d67c8d53 model:add Zaya1 8B (Zyphra) (#32071)
* model:add Zaya1 8B (Zyphra)
- Added the support for Zaya.
- Fixed the SSM_CONV case, so the model runs on cuda without abort.

 Please enter the commit message for your changes. Lines starting

* Made the following changes based on the bot reviewer feedback:
1) Added runtime asserts when loading wrong KV head size, unsupported conv kernel size.
2) Removed the static cast usage.
3) Fixed double load of tokenizer.
4) Updated res_scale_in to be input tensor.

* - Moved the res_scale_in back to layer 0 to fix the cuda bug

* i)  Removed the EOS addition.Team agreed that updating Tokenizer is the right move.
ii) Fixed formatting

* Fixed the comment to make it clear

---------

Co-authored-by: varun <varun_subramanian@zyphra.com>
2026-11-18 15:46:49 +01:00
losewayy 03ec5b4326 cuda : fuse NORM + MUL (+ ADD) (#32454)
Assisted-by: Codex
2026-11-18 12:43:52 +02:00
Ruben Ortlam 43bd7a2a42 vulkan: add bfloat16 support to various ops (#32202)
* vulkan: expand bf16 support for unary and binary ops and rms_norm

* cpu: add bf16 support for scale, clamp, leaky_relu, swiglu_oai, swiglu_clamp, tri and diag

* fix sycl fill supports_op condition

* restore CUDA op order
2026-11-18 11:26:03 +01:00
Xuan-Son NguyenandPascal 52975342ff server: slot_sched: fix lora and recurrent (#32434)
* server: fix lora and recurrent

* tests: run the hybrid SWA checkpoint test with one token per ubatch

With and without the cache, the prompt was not split into the same
ubatches, and the generated tokens are decoded one at a time while the
uncached run processes them in a batch. On CUDA this moves the top
logprobs of the tiny model by up to 0.03 and swaps near ties, so the
test failed while the cache behaved as expected.

With one token per ubatch both runs go through the same kernels and
the output is compared exactly.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-18 00:12:20 +01:00
Xuan-Son Nguyen 3574df84b7 mtmd: support tiny-voca (#32427)
* mtmd: support tiny-voca

* improve G2P
2026-11-18 00:04:15 +01:00
Xuan-Son Nguyen 307e3f1551 mtmd: add kokoro tts support (#32413)
* jinja: impl scope-chain key lookup

* add test for call_statement lambda

* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n

* mtmd: add kokoro tts support

* detailed guide about G2P and text norm port

* refine G2P

* clean up

* migrate batch api

* nits
2026-11-17 22:25:44 +01:00
eddiechen 955c4e8ee5 server : return 400 when grammar and ignore_eos are on (#32377)
* sampling : fix crash when both ignore_eos and grammar are on

* server : rework, throw 400 when grammar and ignore_eos are on

* sampling : move the check into the sampler (review suggestion)

* tests : remove redundant tests
2026-11-17 17:03:21 +01:00
anujj 8777c0c833 Title: CUDA: few-row MMQ for MUL_MAT_ID (#32065)
* CUDA: few-row MMQ for MUL_MAT_ID

* CUDA: few-row MMQ for MUL_MAT_ID from 2 tokens on DGX Spark

* CUDA: few-row MUL_MAT_ID: one launcher, tighter pair count, more tests

* CUDA: few-row MUL_MAT_ID: dispatch order MMVQ, few-row, MMQ and review fixes
2026-11-17 21:01:40 +05:30
Giorgio Sarno ebc9fb933e CUDA: use block reductions for whole-row pool_1d (#31855)
* CUDA: use block reductions for whole-row pool_1d

Assisted-by: Claude Code

* CUDA: template pool1d row kernel by ggml_op_pool
2026-11-17 15:51:15 +01:00
08a9b35893 llama : keep the k-pool layout prefix on sequence edits (#32137)
* llama : keep the k-pool layout prefix on sequence edits

Assisted-by: Claude Code

* llama : address review of the k-pool layout update

* Append and edit share one branch, using stale > sq.pos_min.
* LLAMA_KPOOL_VERIFY=1 checks every update against a full rebuild. The scan moved into kpool_scan() so the check can reuse it.
* The test turns the check on for its rollback context, skips -sm tensor, and drops kimi-k31.
* Two comments that were no longer true are fixed.

Assisted-by: Claude Code

* llama : check the k-pool layout update in debug builds instead of LLAMA_KPOOL_VERIFY

* llama : remove the k-pool layout verification

* llama : inline the k-pool scan again

* remove dangling new line

* do not search twice

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-17 22:37:25 +08:00
Aman Gupta d3064fddce convert : fix FP8 qwen4exp checkpoints (n-gram table scale, block-scaled FP8) (#32476)
* convert : fix FP8 n-gram table export for qwen4exp

* convert : dequantize block-scaled FP8 weight_scale_inv in ModelOpt checkpoints
2026-11-17 22:25:01 +08:00
Xuan-Son Nguyen d92ea9bb12 mtmd: support Minimax-Music3 (#32435)
* mtmd: support Minimax-Music3

* fix and clean up
2026-11-17 12:53:19 +01:00
Xuan-Son Nguyen 8d1599fca1 chat: add tts_dict for kokoro G2P template (#32397)
* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n
2026-11-17 12:52:50 +01:00
Neo Zhang dde9de82cb sycl : Fix FA mkl issue: a mask broadcast over heads or batches (#31536)
* fix the SYCL fa with mkl error of a mask broadcast over heads or batches; add argument cache-ram to start-svr.sh

* add argument cache-ram to win-start-svr.bat

* fix format issue, remove --cache-ram in script: move to another PR
2026-11-17 10:50:32 +02:00
Nikhil Jain 140a279874 webgpu : Implement timestep embedding (#31695)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* resolve merge conflicts

* save state

* save kernel

* add kernel back in

* fix merge issues with kernels

* updates

* update tests

* change invocation count and workgroup to be u32

* clang format

* clang format

* update docs

* format

* remove extra file
2026-11-17 10:41:37 +02:00
Pascal e8fbcad855 tests: print the exception message in test-llama-context (#32428)
A case that throws only reported "unhandled exception", which hides
the cause of the intermittent failures on the Vulkan Apple runner.
2026-11-16 21:53:32 +01:00
Aldehir Rojas 4ddf2b9470 chat : implement inkling parser (#32381) 2026-11-16 19:48:24 +01:00
Rudransh Shekhar 7804be0634 arg : allow up to llama_max_devices() configs for --tensor-split and --fit-target (#13293) (#31435) 2026-11-16 17:07:45 +01:00
Xuan-Son Nguyen ef6bc6f367 jinja: impl scope-chain key lookup (#32396)
* jinja: impl scope-chain key lookup

* add test for call_statement lambda
2026-11-16 15:48:46 +01:00
Niklas Wenzel 62a6d3a302 llama : migrate to llama_batch_ext (#32356)
* llama : migrate to llama_batch_ext

* mtmd : migrate zonos3-mini code

Added in https://github.com/ggml-org/llama.cpp/pull/32020
2026-11-16 15:16:12 +01:00
Jeff Bolz ef3848f9d7 vulkan: Support split_k for conv3d. (#29929)
This tries to use the 256x128 shape (coopmat2-only) with split_k when
it appears profitable. It reuses the mul_mat split_k reduce shader.
2026-11-16 15:25:12 +02:00
Xuan-Son Nguyen e1925b3695 jinja: support json sort_keys=true (#32366) 2026-11-16 12:28:19 +01:00
PascalandRuben Ortlam adf554d239 vulkan: make the radix top-k tie selection deterministic (#31931)
* vulkan: make the radix top-k tie selection deterministic

The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.

Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.

* optimize shader for AMD, Intel and Nvidia

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-16 11:41:44 +01:00
liminfei-amd f477c1b90a cuda : fix MUL_MAT_ID abort with odd src0 columns on AMD (#32385)
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.

Fixes #30576

Assisted-by: GitHub Copilot

Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
2026-11-16 17:53:06 +08:00
Titaniumtown 21fa9c043f kv-cells: avoid an O(n_cells) scan per sequence removal (#32380) 2026-11-16 10:52:09 +01:00
Pascal a946eb541a server: answer reasoning_effort levels the template rejects with 400 (#32332)
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.

Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.

Also trim the output_config.effort test to a single level.
2026-11-16 06:55:56 +01:00
Xuan-Son NguyenandAldehir Rojas b4c0e00a4f jinja: add 4 missing features (#32352)
* jinja: add 4 missing features

* apply suggestion

Co-authored-by: Aldehir Rojas <hello@alde.dev>

* fix error msg

---------

Co-authored-by: Aldehir Rojas <hello@alde.dev>
2026-11-16 00:40:44 +01:00
Xuan-Son Nguyen 7ae3c6b231 mtmd: add zonos3-mini TTS model (#32020)
* zonos3 init

* add audio gen pipeline

* fix string_format

* add default voice as gguf metadata

* better ref

* support voice clone

* add delivery params

* optimize speed

* nits

* nits 2

* add text normalization

* nits
2026-11-15 19:44:44 +01:00
Xuan Son Nguyen f4d0673917 Revert "jinja: add 4 missing features"
This reverts commit a681a53b60.
2026-11-15 19:35:31 +01:00
Xuan Son Nguyen a681a53b60 jinja: add 4 missing features 2026-11-15 19:33:06 +01:00
Roshan Kumar 56fd8a375f server : ignore empty stop strings (#32093) 2026-11-15 17:23:05 +01:00
Xuan-Son Nguyen dd099a6bf3 chat: fix inkling 2 begin/end of image tokens placement (#32294)
* chat: fix inkling 2 begin/end of image tokens placement

* add test
2026-11-15 09:41:44 +01:00
Xuan-Son Nguyen 429460dc0e server: refactor, add server_slot_sched, prompt running hash (#32154)
* server: refactor, add server_slot_sched, prompt running hash

* use sha256 hash
2026-11-15 09:37:17 +01:00
Ed Addario 5a4afd8077 tests: Fail test-backend-ops and exit with error status 1 when missing and/or unsupported types and/or unsupported backends are provided in the command line (#30279)
* Fail tests on missing or unsupported type

* Fix output

* Remove whitespace

* Fail if no tests ran regardless of backend support

* Revert unnecessary cosmetic change
2026-11-15 08:26:23 +01:00