Commit Graph
12643 Commits
Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 2ca30e5f35 llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.

Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.

Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.

The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.

Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.

Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:31 +01:00
Piotr Wilkin (ilintar) 5bacb7c868 llama : copy all experts of a host weight shared by several nodes (#32807)
* llama : copy all experts of a host weight shared by several nodes

The scheduler copies a host weight to the device once per graph, so a
partial expert copy made for the first MUL_MAT_ID is stale for the
others (e.g. the shared layers of inkling2_loop with --n-cpu-moe).

Assisted-by: Claude Opus 5.5

* llama : copy all experts of a host weight shared by several nodes

Assisted-by: Codex
2026-11-22 20:04:21 +01:00
Max S. d2bceda3de vocab: enable ignore_merges for inkling2 (#32799) 2026-11-22 19:52:33 +01:00
Adrien Gallouët 56eea9e4b0 server : add offset pagination to grep_search tool (#32792)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-22 19:00:05 +01:00
Pedro Cuenca e14a2d3a63 skill: Pay attention to tokenizer flags (#32819) 2026-11-22 18:59:17 +01:00
Aldehir Rojas 1014775c8a chat : implement token aware parsing (#29307) 2026-11-22 11:36:03 -06:00
Xuan-Son Nguyen 0b51bb2fed server: support /models/sse on single-model mode (#32398)
* server: support /models/sse on single-model mode

* nits

* nits
2026-11-22 17:52:56 +01:00
chessMan 8d60873dfb ggml/gguf : avoid creating invalid enum values from untrusted file data (#32641)
* ggml/gguf : avoid creating invalid enum values from untrusted file data

* removing the tests
2026-11-22 13:40:10 +02:00
gianni-corandJohannes Gäßler ad50351f50 ggml : add SIGMOID backward pass (#32126)
* ggml : add SIGMOID backward pass

Sigmoid is used in Qwen3.5 hybrid models and had no backward pass.
Bump RPC protocol patch version for the new op.

Signed-off-by: gianni-cor <gianfranco.cordella@tether.io>

* replace assert with ggml_assert

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* ggml : keep sigmoid backward checks in release builds

Assisted-by: Codex

* ggml : preserve existing operation IDs when adding SIGMOID_BACK

Assisted-by: Codex

---------

Signed-off-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-11-22 13:33:13 +02:00
Michael 42f6a58fd1 chat : trigger lazy grammar on bare JSON tool calls (#32497)
Models like Llama 3.x emit tool calls as plain {"name": ...} JSON without a start marker, so with tool_choice=auto no grammar was produced at all. Use a pattern trigger on the opening of the call object instead.

Assisted-by: Claude Code
2026-11-22 12:19:04 +01:00
Alexey Kopytko 9c10c94e16 ggml : fix nonzero clear on WebGPU and non-UMA devices (#32727) 2026-11-22 20:10:42 +09:00
Eddie Zhou 8cb0d98ccd HIP: fix double halving of NVFP4 scales on CDNA3 (#32790)
Fixes #32789
2026-11-22 12:06:12 +01:00
Alexey Kopytko e300919cc7 webgpu : update GDN supports_op check (#32782) 2026-11-22 17:08:21 +09:00
Alexey Kopytko 5d28d64bf2 ggml-webgpu : Update to a recent version of Dawn (#32779)
Dawn v20260908.214631 doesn't compile with emsdk latest, that is 7.0.1.
2026-11-22 17:07:48 +09:00
shaofeiqi 1fb8a26960 opencl: add coopmat GEMM for Adreno 850 (Q4_0, Q5_0, Q8_0, Q4_K, Q5_K, Q6_K) (#32686)
* opencl: add Q4_0 coopmat GEMM for Adreno 850

Assisted-by: Claude Opus 5.5

* opencl: add Q5_0 coopmat GEMM for Adreno 850

* opencl: add Q8_0 coopmat GEMM for Adreno 850

* opencl: add Q4_K coopmat GEMM for Adreno 850

* opencl: add Q5_K coopmat GEMM for Adreno 850

* opencl: add Q6_K coopmat GEMM for Adreno 850
2026-11-22 00:01:14 -08:00
Revertron 8a97fa98ab vulkan : support non-contiguous src in SCALE (#32735)
Assisted-By: Opus 5.5
2026-11-22 07:28:46 +01:00
Aman Gupta 225a5345bb server : fit non-causal batches in one ubatch (#32713) 2026-11-22 11:25:13 +08:00
Alexey Kopytko b052049276 vulkan : fix memset for unaligned ranges for non-UMA devices (#32725)
See https://github.com/ggml-org/llama.cpp/pull/32574
2026-11-22 10:11:39 +09:00
Xuan-Son Nguyen cbb1c1c50b model: fix no_realloc for clef and vllm-decision2 (#32754) 2026-11-22 00:35:43 +01:00
Aldehir Rojas afe77e18fa chat : pass reasoning_effort = none if supported (#32763) 2026-11-21 15:27:25 -06:00
saiph e6d624b9a4 mtmd : merge Parakeet rel-pos views to avoid OOB (#32722)
Assisted-by: Grok Bot
2026-11-21 19:05:02 +01:00
Xuan-Son Nguyen 19bc41dbb4 model: apply .scale tensors automatically + fix correctness (#32669)
* model: apply .scale tensors automatically

* rm build_moe_ffn "_s" overload

* use llama_prec_policy

* fix ci

* build_lora_mm_id: rm w_s
2026-11-21 16:30:01 +01:00
Frosty40andTitaniumtown e50c254a22 sycl: bound CONV_TRANSPOSE_1D input iteration (#31109)
Used angelX to optimize SYCL/drama 3 coordination, results are as follows

Each output reads only inputs with i*s0 <= idx < i*s0 + K, in the same ascending order as before.

Arc Pro B70, oneAPI 2026.1, test-backend-ops perf, base 9b56ccf00 vs patch, median of 3 same-boot A/B/A rounds, A/A 1.000-1.001:

| input [ne0, ne1] | kernel [K, OC, IC] | s0 | base us | patch us | speedup |
|---|---|---|---|---|---|
| [16, 1536] | [16, 768, 1536] | 8 | 3910.57 | 1348.59 | 2.90x |
| [128, 768] | [16, 384, 768] | 8 | 37735.00 | 1665.26 | 22.66x |
| [1024, 384] | [8, 192, 384] | 4 | 270371.75 | 1471.74 | 183.71x |
| [4096, 192] | [4, 96, 192] | 2 | 534614.63 | 752.74 | 710.23x |

Shapes are the drama 3 codec decoder upsampling stack, added as perf cases. Chained 4-layer stack at 1/4/24 input frames: base 66.3/847.4/28638.5 ms, patch 2.3/6.7/30.7 ms. Kernel numbers, not end-to-end TTS latency.

Correctness: test-backend-ops -o CONV_TRANSPOSE_1D 116/116 on base and patch, and 116/116 rebased on 3f03b1772. Seeded output dump (7.46 MB) byte-identical to base.

Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857). With only those cases excluded: 20145/20173 on base and patch, and every patch-run failure also fails on base.

Co-authored-by: Titaniumtown <titaniumtown@proton.me>
2026-11-21 16:20:29 +02:00
Kazutoshi, Akao 660be5506b ggml : improvements to the memory access pattern in ggml_compute_forward_mul_mat_one_chunk (#30846) 2026-11-21 16:16:40 +02:00
Xuan-Son NguyenandPascal c622f623fc mtmd: support kyutai/hibiki realtime full-duplex (#32491)
* mtmd: support hibiki realtime duplex

* fix and clean up

* fix long session quality

* fix ci

* tests : add the moshi fusion baseline for Metal

The Metal fusion check found 5 RMS_NORM+MUL fusions in the new moshi
arch and no baseline for them.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-21 14:25:19 +01:00
Titaniumtown c661033349 sycl: add IQ4_NL and IQ4_XS weight reorder (#31998) 2026-11-21 15:11:14 +02:00
Titaniumtown 44c20b1c0a sycl: column-blocked gated_delta_net kernel (#32456) 2026-11-21 14:38:00 +02:00
Aleksander Grygier 80325823d4 ui: open the context gauge panel on click as a popover (#32685)
* ui: open the context gauge panel on click as a popover

Assisted-by: Claude (Opus 5.5 Medium)

* ui: give the context gauge trigger the ghost button hover

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-21 13:21:47 +01:00
Clemens Wasser a818a46f3d sycl: add fused ADD, RMS_NORM, MUL and vectorize rms norm (#31753)
* sycl: vectorize contiguous rms norm

* sycl: add fused ADD, RMS_NORM, MUL

* tests: test fused ADD, RMS_NORM, MUL at decode widths
2026-11-21 14:20:22 +02:00
Johannes Gäßler b019be0c86 CUDA: abort on UB in fastdiv (#32698) 2026-11-21 13:46:59 +02:00
Ruben Ortlam 362ac92974 vulkan: unroll FA coopmat1 array due to a driver bug on AMD Windows (#32466)
* vulkan: unroll FA coopmat1 array due to a driver bug on AMD Windows

* simplify code to make it clearer
2026-11-21 13:46:20 +02:00
KaiFelixBennettandRuben Ortlam fb7090df56 vulkan: improve RDNA4 MMVQ performance for Q3_K, Q5_K and IQ4_XS (#31804)
* vulkan: keep default MMVQ rows for Q3_K, Q5_K, IQ4_XS on RDNA4 AMD proprietary driver

Assisted-by: Claude Opus 5.5

* clean up mmvq shader load logic

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-21 13:41:55 +02:00
Daniel Han 690b87d881 model : inkling2_loop share the loop slots after the scale pass (#32704)
The loop slots copied the physical layers at the end of load_arch_tensors,
before the generic pass attaches the optional .scale tensors, so the later
passes through the core ran NVFP4 experts without their per-expert scales.
Add an optional post_arch_tensors hook that runs after the scale pass and
share the slots there.
2026-11-21 12:10:38 +01:00
Sigbjørn Skjæret 838d942438 model : additional shortcut checks for OOB in hrm_text (#32720)
[no ci]
2026-11-21 12:10:04 +01:00
Aaron Teo 3da11de1f1 ggml-zdnn: fix ztensor load race condition (#32279)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-21 13:05:23 +02:00
shalinib-ibm 93cb37ea7b ggml-cpu: fix nc aligment in tinyBLAS_PPC_Q0 (#32659)
Fixes https://github.com/ggml-org/llama.cpp/issues/31680

The matmul_tiled() implementation in tinyBLAS_Q0_PPC requires nc to be a multiple of 8 to perform matrix multiplication correctly.

The previous alignment logic contained a special case for n == 4, which allowed n_aligned to be 4. Consequently, the tiled path could be selected with nc = 4, violating the kernel's tile-width requirement.

This caused incorrect matrix multiplication results, observed as garbled inference output when running  Flint Q4_K_M and Gemma 3 27B Instruct.
2026-11-21 12:58:49 +02:00
Aaron Teo 1fdda89a4d ci: switch ppc64le runners to ubuntu 26.04 (#32679)
* ci: switch ppc64le runners to ubuntu 26.04

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: rm problem matchers experiment

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-21 10:54:00 +01:00
Georgi Gerganov e412c6f468 llama : remove fused lightning indexer gating (#32650)
The fused lightning indexer is always used now, so drop the cparams flags, the auto probe and the non-fused fallbacks.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-21 10:26:55 +02:00
Xuan-Son Nguyen 40b7a546de ci: temporary disable no realloc case for mtmd (#32708) 2026-11-21 10:02:26 +02:00
Daniel Kuts 46eb1742db llama-grammar: Fix integer overflow (#32551)
* llama-grammar: Fix integer overflow

* Check incorrect values at parsing and bail out nicely

* fix style
2026-11-21 07:00:30 +01:00
Aman Gupta 6792507b7f llama : score only the k-pools of the ubatch sequences in QSA (#32519) 2026-11-21 13:22:36 +08:00
cwriter 418fe97b5f sycl: Improve Grouped XMX GEMM by avoiding SLM staging (#31428) 2026-11-20 22:37:40 -05:00
Ashutosh Singh 7476e47261 docs: openvino backend details, fix SYCL docs (#30967) 2026-11-20 22:08:53 -05:00
Pedro Cuenca 29b52d398b model : Inkling 2 Mini (#32694)
* Conversion

* Separate model_type

* Add gate_rank to allow building from metadata while testing

* Base model

* Loop graph

* Support the new is_varloop attribute

* Adopt naming scheme from https://github.com/ggml-org/llama.cpp/pull/32588

* Rename to varying
2026-11-20 23:01:07 +01:00
Pelayo Garcia BárcenaandJohannes Gäßler e3c3f666a4 hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1) (#32389)
* hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1)

* hip: write the RDNA1 MMVQ values explicitly instead of recursing into the generic table

* hip: list the RDNA1 MMVQ table before RDNA2 in the enum

* Update ggml/src/ggml-cuda/mmvq.cu

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-11-20 22:52:19 +01:00
Ravi Panchumarthy b2e659ec24 ggml-openvino: update openvino CI runs triggers (#32593)
* ci : run the OpenVINO GPU job only for OpenVINO changes

The self-hosted OpenVINO GPU job ran for most PRs and every master push
on a shared runner, so it queued for hours.

- ci-self-hosted-openvino.yml (GPU): runs only for changes to
  ggml/src/ggml-openvino.
- build-openvino.yml: keeps its triggers (OpenVINO changes on PRs, every
  source change pushed to master). The Linux job runs on a GitHub-hosted
  runner and tests the OpenVINO CPU device only.

* address review comments

* ci : limit the OpenVINO PR CMake trigger to ggml

* ci : drop the OpenVINO Ubuntu job

* ci : trigger OpenVINO PR runs on tests/CMakeLists.txt
2026-11-20 22:51:49 +01:00
Xuan-Son NguyenandPascal 9197945004 convert: qwen3-asr: put "language ..." output into thinking block (#32571)
* convert: qwen3-asr: put "language ..." output into thinking block

* convert: qwen3-asr: read the chat template from models/templates

The converter embedded its own copy of Qwen3-ASR.jinja. Read the
bundled file instead, as the Kimi-K3 and Kokoro converters do, so
the GGUF and test-chat share a single template.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-20 20:54:57 +01:00
Tarek Dakhran a83d68b562 metal : allow the norm chain in either operand of the fused MUL/ADD (#32300)
Assisted-by: Claude Opus 5.5
2026-11-20 19:55:44 +01:00
Aleksander Grygier e46069c409 ui: fix the flaky reset check of the MermaidPreview Flowchart story (#32675)
Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 19:43:24 +01:00
Aleksander Grygier 427a836734 ui: stage Llama Voice's sources for CMake and Docker UI builds (#32680)
Llama Chat compiles @ggml-org/llama-voice from ui/lib/voice/src, but the
CMake stage and the Docker UI stages copied only ui/lib/ui, so their npm
build could not resolve the voice imports. Stage ui/lib/voice/src next to
Llama UI and track it as an input of the CMake build.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 19:42:49 +01:00