Commit Graph
12168 Commits
Author SHA1 Message Date
Alessandro de Oliveira Faria (A.K.A.CABELO) d9cf1870be vendor: update BoringSSL to 0.20261105.0 (#31757)
[no ci]
2026-11-06 09:52:16 +01:00
Johannes Gäßler 240d68fcfe CUDA: fix Maxwell build (#31787) 2026-11-06 09:06:27 +01:00
Adrien Gallouët bbc2e4b657 common : restrict HF token to huggingface.co hosts (#31730)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-06 08:43:33 +01:00
Shawn GuandLi He 4b77dee052 opencl: add q6_k multi column tiled GEMV for speculative verification and refactoring (#31758)
* opencl: use the multi-column q6_K vocab head GEMV for 2..4 tokens on X2E

Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5

* opencl: rename GGML_OPENCL_Q6K_MC3 to GGML_OPENCL_Q6_K_MC3

Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5

* opencl: rename GGML_OPENCL_Q40/Q41/Q4K_MC3 to GGML_OPENCL_Q4_0/Q4_1/Q4_K_MC3

GGML_OPENCL_Q4K_MC3_DQ and GGML_OPENCL_Q4K_MC3_LDS become GGML_OPENCL_Q4_K_MC3_DQ
and GGML_OPENCL_Q4_K_MC3_LDS.

Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5

* opencl: remove unnecessary debug probe

Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5

* opencl: clean up

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-05 21:11:07 -08:00
Max Krasnyansky 67107ac34f hexagon: Implement set_tensor_n and improve repacking/tiling robustness (#31744)
* hex-repack: move repacking/tiling into graph-optimize

* hex-tiling: simplify tiling rules and remove obsolete helpers

* hex-ops: use a hash for de-duplicating tensor descriptors

* hex-bufs: improve tensor hash

* hex-ops: accept empty (zero-elements) ops

* hex-bufs: enable hostbuf by default and reject tensors that do not fit into max_vmem

* hex-bufs: improve error message and update readme

* hex-bufs: make sure to sync flush opbatch in all sync set_tensor paths
2026-11-05 18:15:58 -08:00
Xuan-Son Nguyen e14a779719 mtmd: enable cmake unity build (#31760) 2026-11-06 00:16:13 +01:00
Sigbjørn Skjæret d38bea3422 tests : drop u8path() (#31755)
[no ci]
2026-11-05 23:06:51 +01:00
Eve 186eab1cb8 ci: fix build-cmake-pkg ccache (#31705) 2026-11-05 21:16:56 +00:00
Xuan-Son Nguyen 6023692b1e llama: fix kv shifting, add tests, add per-model build_rope (#31683)
* llama: fix kv shifting and add tests

* llama: add graph build_rope (#31693)

* llama: add graph build_rope

* improve

* migrate new models

* update review skill

* fix ci

* add --kv-shift to CI
2026-11-05 21:10:36 +01:00
Xuan-Son Nguyen b1c902f026 llama, mtmd: (chatterbox) add use_classifier_free_guidance as gguf metadata (#31698)
* llama, mtmd: add use_cfg as gguf metadata

* use_classifier_free_guidance
2026-11-05 20:09:33 +01:00
Sigbjørn Skjæret e8df20d238 models : recursive fix (#31749) 2026-11-05 20:06:55 +01:00
fecdc81eec model : add Ripple (ricursiveai) (#31692)
* feat : added ricursiveai ripple support

* fix : mandatiry SWA, scale through helper, comments back

* fix : metal

* model : make ricursive RoPE/NoPE explicit and validate the layer pattern

Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.

Assisted-by: Claude Opus 5.5

* model : make ricursive swiglu_clamp per-layer

Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.

Assisted-by: Claude Opus 5.5

* model : use the ricursive per-tensor scales and RoPE parameters

Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.

Assisted-by: Claude Opus 5.5

* convert : drop the ricursive chat_template.jinja override

The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.

Assisted-by: Claude Opus 5.5

* fix : remove NVFP4 checkpoint quantization block

* match #31733 refactor

---------

Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 19:55:15 +01:00
45b6604b20 model: Add support for the xing4_0 (#31589)
* model: add support for Xing4.0

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* fix: rename model file to xing4-0.cpp to match CI expectations

* model: rename xing4_0.cpp to xing4-0.cpp

* lint: fix E128 continuation indent in conversion/xing.py

* tests: add xing4_0 fusion baseline for Metal

* reduce comments

---------

Co-authored-by: shixq7 <shixq7@chinatelecom.cn>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 17:47:27 +01:00
Nick Towle e30ad80fd7 ui: prevent a hidden sidebar from intercepting clicks (#31712)
* ui: prevent a hidden sidebar from intercepting clicks

* ui: also fix desktop sidebar
2026-11-05 15:44:34 +01:00
Xuan-Son Nguyen e0f96a1a12 llama: refactoring: move model to class member of llm_graph_context (#31733) 2026-11-05 15:41:03 +01:00
Adrien Gallouët 468fde90e8 server : simplify windows MCP command resolution (#31728)
* server : simplify windows MCP command resolution

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* common : remove utf8_to_wstring()

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 15:34:54 +01:00
Adrien Gallouët 1416f76aa1 vendor : update cpp-httplib to 0.65.0 (#31731)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 15:34:18 +01:00
Zheyuan Chen 10d4846a57 ggml-cuda: sanitize post-bias NaNs in MoE selection (#31005)
Fixes #28370
2026-11-05 13:50:16 +01:00
Jeff Bolz 61d4472344 vulkan: Implement test-fusion interface (#30769) 2026-11-05 13:31:44 +01:00
ManuelPDMandDukeNukem-dev 016c1d914c CUDA: use highest compiled arch in get_mmvq_mmid_max_batch (#31679)
Co-authored-by: DukeNukem-dev <54203477+DukeNukem-dev@users.noreply.github.com>
2026-11-05 20:18:38 +08:00
Alex-JP-93 da3416d807 vulkan: RDNA3/4 Large-Tile Split-K Optimization (#30280)
* RDNA3/4 Vulkan Large-Tile Split-K Optimization

* Add single-operator performance test cases

* Update the policy to use driver-reported LDS usage

* Add comments

* Use the simpler split_k rule from review

Assisted-by: GitHub Copilot

* tests : drop the split_k test cases

* vulkan : simplify the RDNA3/4 split_k condition
2026-11-05 13:11:07 +01:00
Sigbjørn Skjæret cc78da97c0 gguf-py : external chat templates have precedence over tokenizer_config (#31717) 2026-11-05 13:04:48 +01:00
Georgi Gerganov ea40a3a1e1 metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (#31615)
* metal : support scale-bias routing norm in the TOPK_MOE fusion

The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sqrt(softplus) MoE routing with a selection bias

Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sigmoid MoE routing with a selection bias

The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.

- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
  GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
  tail; the selection-bias ADD is required, same as the sqrt(softplus)
  chains
- generalize the gating function constant from a sqrt(softplus) bool
  to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
  scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
  CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
  weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
  deepseek2/32, kimi, minimax, ...) now fuse their routing, and
  muse-spark prefill drops one MOE_REDUCE fire because the reordered
  graph allocates the routing weights inside the reduce output range,
  which the dst/input overlap check correctly declines

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* cont : remove low-signal comments

* ocd : valign ggml_metal_fusions
2026-11-05 13:52:56 +02:00
15e50990bb CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a) (#29050)
* CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a)

* Addressing reviewers comments, mma.cuh and physical warp size

* CUDA/HIP: single mma.cuh matrix-core lightning indexer ::CDNA::

* Support RDNA3

* CUDA/HIP: stream heads through the mma lightning indexer kernel

Process 16 heads at a time with double-buffered Q staging instead of keeping
all heads resident in shared memory, and drop the GGML_INDEXER_NO_MFMA env.

* enable for CUDA

* CUDA: Remove wmma lightning-indexer, also run head 32 through mma kernel

* CUDA: Nv mma dosent transpose accumulator

---------

Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-11-05 11:44:15 +01:00
Johannes Gäßler 69ee2ecc01 CUDA: linearize MoE data layout for MMQ/MMF (#31047)
* CUDA: linearize MoE data layout for MMQ/MMF

* Apply MMQ tile size optimization more generally

* Fix F16 performance regression
2026-11-05 12:33:25 +02:00
bri-prism 2bc46baa92 vulkan: FWHT kernels for block widths above 512 (#29772)
* vulkan: add F16 input to the FWHT

The Vulkan FWHT accepts F32 input only. This adds an F16 source variant of the
shader, for both the subgroup and the shared-memory paths, so the kernel reads
an F16 source directly instead of requiring a converted copy.

The F16 shader needs shader-float16, so the pipelines are only created when the
device reports it. A null pipeline makes ggml_vk_can_use_fwht fall back, which
is the same path an unsupported width already takes.

supports_op and the dispatch both already route through ggml_vk_can_use_fwht,
so there is a single place asking whether an op qualifies; nothing to
consolidate here the way the CUDA/Metal ports needed.

test-backend-ops on an M5 Pro (MoltenVK, fp16: 1): MUL_MAT 1128/1128,
including all 24 Hadamard cases (18 existing F32, 6 new F16).

Assisted-by: Codex

* vulkan: FWHT kernels for block widths above 512

The Vulkan FWHT covers block widths 64 to 512; wider blocks fall back to the
dense matmul. Extend the width list to 8192 and pick the shader per width: the
subgroup shader keeps n/subgroup_size values per invocation, so widths above
512 (or a subgroup narrow enough to exceed 64 values per invocation) use the
shared-memory shader with one row per workgroup and a 256-wide block. The rows
per workgroup become a specialization constant so the dispatch matches.

Pipelines for a width are only created when the device's shared-memory and
workgroup-invocation limits allow it; otherwise that width keeps the fallback.

This is the Vulkan counterpart to #29100 (CUDA). Stacked on #29101; the last
commit is the one to review.

Assisted-by: Codex
2026-11-05 12:03:44 +02:00
Nisbenz 3b83d12b08 vulkan : check combined K+V f16 scratch size in FA dequant path (#30487)
Fixes #30048

Assisted-by: Claude
2026-11-05 12:02:39 +02:00
cwriterandcwriter e152332a15 llama-server: Use reference-counted checkpoints instead of deep-copies (#30137)
* checkpoints: reduce overhead

Assisted-by: Codex

* rebase

Adapt the slot save/restore checkpoint appendix from #26004 to the
shared_ptr checkpoint list and the common_state_data buffers.

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: cwriter <cwriter@localhost>
2026-11-05 11:02:36 +01:00
Takamichi Ohara 1fa7f1c488 ggml : support truncate=false for YaRN (#30206)
* ggml : support `truncate=false` for YaRN

YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.

Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.

* support CUDA backend

* add tests

* move RoPE helpers to internal header

* format
2026-11-05 12:02:05 +02:00
Daniel Bevenius 9c9074e14e conversion : resolve quantized_layers from source tensor names (#31611)
* conversion : resolve quantized_layers from source tensor names

This commit changes how the per layer NVFP4 quant_algo are resolved from
hf_quant_config.json quantized_layers.

The keys in quantized_layers are module paths in the source checkpoint,
and these can be either be modules like 'lm_head' or parent modules
like 'layers.0.ffn.experts'. Currently these keys are passed through
filter_tensors, but they are not tensor names. So a filter_tensors that
keeps an explicit list of tensor names, like Nemotron's with --mtp
would drop them. That means there was no `_nvfp4_layer_algo entry`, and
`_tag_prec_a4` fell back to `_nvfp4_global_algo`.

This commit records the mapping from source tensor name to filtered
tensor name in index_tensors. Then in prepare_tensors each source
tensor is resolved to its closest quantized_layers parent, and
the algo is stored under the filtered tensor name, which is what
_tag_prec_a4 later looks up.

Converting Nemotron 3.5 Lightning NVFP4 with --mtp now produces:
```console
    general.tensor_extra.name = ['output.weight']
    general.tensor_extra.prec_a4 = [False]
```
whereas on master no general.tensor_extra fields are written. For the
non-MTP conversion, the general.tensor_extra fields are identical to
master.

Also manually verified (using pdb) that this works for
DeepSeek-V4.1-Flash-1017-NVFP4.

Refs: https://github.com/ggml-org/llama.cpp/pull/31602#issuecomment-6603337403

* conversion : shorten comment

* conversion : also remove '.bias' suffixes
2026-11-05 10:05:06 +01:00
Hoang Vien DuyandGeorgi Gerganov f7b41cca4f metal : use mul_mm for 32 <= K < 64 (#30982)
* metal : use mul_mm for 32 <= K < 64

* metal : add bf16 x bf16 mul mat kernel

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-05 10:58:41 +02:00
Adrien Gallouët c55b094880 server : reject non-object session updates (#31714)
Prevent malformed realtime session updates from throwing an uncaught
exception and terminating the server

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 09:38:08 +01:00
Aman Gupta 2c192444c9 fit : account for skipped NextN layers when setting n_gpu_layers (#31569)
* fit : account for skipped NextN layers when setting n_gpu_layers

Without MTP, fit counts n_layer + 1 layers while the model loader counts
n_layer_all + 1 for n_gpu_layers, including the NextN layers which hold no
weights. A fit result that offloads all layers then leaves layer 0 on the CPU
("offloaded 65/66 layers" for Qwen3.8-27B on two GPUs).

Add the skipped NextN layers to n_gpu_layers and to the tensor split of the
device that holds the output layer so the loader assigns the layers that
fit measured.

Fixes #31531

* fit : pass hp_ngl_skip by reference like the other hparams

* fit : add hp_ngl_skip to common_get_device_memory_data
2026-11-05 10:31:48 +02:00
Leon 553d2c2e7e webgpu : add MEAN support (#31591)
Assisted-by: Codex
2026-11-05 10:29:14 +02:00
Pascal 040ce199a1 graph: store the MTP conv snapshots of every recurrent model in one set_rows (#31660)
* graph: store the conv state of every rollback arch in one set_rows

The rollback slots of the causal conv history were written with one
copy per slot in qwen4exp, glm5-next, kimi-k3, bailingmoe3, lfm and
mamba2, while delta_net already wrote them with a single set_rows.
build_conv_state_store now does this for all of them: one copy without
rollback, otherwise one set_rows of the overlapping windows into the
min(n, n_rs_seq + 1) slots a rollback can read.

The slot order {n_rs_seq, ..., 0} is computed once per graph and each
conv layer takes a view of its tail. Its size does not depend on the
ubatch, so the reserved graph covers every ubatch shape, and the graph
input that held it is gone.

A model that is not causal never rolls back, so the context clamps
n_rs_seq to 0 for it, like for an arch without rollback support.

* nits: compute the conv slot order in the graph and keep the n_head name of bailingmoe3

* graph: build the conv slot order in each conv layer

The slot order {K-1, ..., 0} of the conv state store was one tensor
shared by every conv layer of the graph, built on the device of the
first one, so each layer on another device copied the view of its
tail. Each conv layer now builds its own order next to its set_rows,
and llm_graph_input_rs no longer holds it.
2026-11-05 09:28:03 +01:00
Georgi Gerganov d4c345d6c4 ci : remove unused model generation (#31412)
* ci : re-enable test-llama-archs on Windows

* cont : remove unused test
2026-11-05 10:25:04 +02:00
Adrien Gallouët 2ada83079a common : drop u8path() and use generic_string() (#31681)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 08:37:59 +01:00
Adrien Gallouët 5a081c0a70 server : drop u8path() and use generic_string() (#31682)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 08:37:13 +01:00
Adrien Gallouët 7e42b95f23 rpc : align code with common.cpp (#31684)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 08:36:51 +01:00
Alex-JP-93 3259ab6e5d vulkan: Re-enable large coopmat matmul configuration on AMD proprietary driver 26.10.1+ (#31587) 2026-11-05 07:40:39 +01:00
Pascal a01dff908b tests: fix CI flakes of TOPK_MOE caused by near ties (#31426)
* tests: fix CI flakes of TOPK_MOE caused by near ties

With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.

Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.

* tests: build the TOPK_MOE logits by inverting the gating function

Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.

The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
2026-11-05 07:31:48 +01:00
Pascal 44f91a02b2 metal: keep fusable node groups inside one command buffer (#31688)
The first command buffer takes the first 10% of the graph nodes and the
rest is split evenly between the extra command buffers. A fusion cannot
span two command buffers, so a boundary that falls inside a fusable
group runs that group unfused, and the output then depends on the node
count of the graph.

Move each boundary to the end of the fusable group it falls into, so
the encoders fuse the same nodes as with a single command buffer.
2026-11-05 08:21:13 +02:00
Daniel Bevenius 330f0189c9 qwen2 : add nvfp4 scales to build_ffn (#31657)
This commit adds nvfp4 scales to qwen2's build_ffn call which are
currently passed as NULL.

The motivation for this is that a model converted to NVFP4 will
currently crash because the scales are loaded but never applied in
the FFN:
```console
llama-completion: llama.cpp/src/llama-sampler.cpp:1218:
void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*):
Assertion `found' failed.
```
The attention already applies its scales but ffn is missing them.
2026-11-05 06:29:50 +01:00
shaofeiqiandLi He 7d516e25ea opencl: enable f32 and q5_k multi column GEMV by default for Adreno X2 (#31694)
* opencl: use the small-N f32 GEMV by default on X2E

Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: use the q5_K multi-column verify GEMV by default on X2E

Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-04 21:22:41 -08:00
shaofeiqiandLi He 8bed7cfb99 opencl: support non-contig sigmoid (#31687)
Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-04 19:18:24 -08:00
shaofeiqiandLi He 8415bd7bb3 opencl: add DSV4_HC_PRE and DSV4_HC_POST (#31623)
* opencl: add DSV4_HC_PRE and DSV4_HC_POST

Co-authored-by: Shafei Qi <shaoqi@qti.qualcomm.com>

* opencl: improve dsv4_hc_post

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-04 19:14:11 -08:00
anujj 5ce06fc71f Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ (#31296)
* cuda: fuse NVFP4 gate/up and GLU in MMQ

* cuda: honor the requested src1 precision in fused gate/up MMQ

* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels

* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming

* cuda: check gate_scale before loading the fused NVFP4 gate scale
2026-11-05 10:47:33 +08:00
Hongqiang Wang e56f006ed6 llama : fix on-device state copy sizes for quantized KV caches (#30541)
* llama : fix on-device state copy sizes for quantized KV caches

The on-device sequence state copies (LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) computed
their element count as size / ggml_element_size(). For a quantized type such as
q8_0 or q4_0 that is the number of blocks, not elements, so only part of each
range was copied: the restored state was wrong, or the copy crashed.

Compute the count as size / ggml_type_size() * ggml_blck_size() for the save,
the restore and the copy between differently chunked ranges, and assert that
each range covers whole blocks.

* llama : drop the one-line comments

* llama : drop the redundant cast in the state copy size
2026-11-05 10:31:01 +08:00
Eve 3de52ac388 ci: run self hosted cpu jobs on hf or github (#31439) 2026-11-04 22:34:01 +00:00
Tom Vaucourt 6f1e654370 server : realtime: send an error event instead of crashing on exceptions (#31632)
* server : realtime: send an error event instead of crashing on exceptions

* server : route websocket handlers through ex_wrapper, remove the realtime test

* server : realtime: keep the previous tools when the chat template rejects new ones

* server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapper

* server : realtime: clarify the tool schema check comment
2026-11-04 21:34:58 +01:00