Commit Graph
1319 Commits
Author SHA1 Message Date
Ramzi aa75ff3239 chat : fix Muse Spark ignoring response_format json_schema (#32017) 2026-11-10 23:30:14 +01:00
Sigbjørn Skjæret 582eab7186 ci : bump ty to 0.0.100 (#32030)
* bump ty to 0.0.100

* rename rules

* type fixes

* type fix

* proper type fix
2026-11-10 21:54:14 +01:00
Oliver Simons 202c1a229f llama : Add FP8 KV-cache support (#28392)
* Add FP8 KV-cache support

* Add tests for scaled, FP8 KV-cache to test-backend-ops

* Fix get_*_scale mapping and dequnatize FP8 also before rope shift

* Normalizes cache-scale names independently of projection layout.

While fused KQV is currently impractival for non-shared scales, this is
an improvement for future use regardless

* Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs

* Update comment wording

* Move E2E FP8 KV cache tests to test-llama-archs

They belong there as they are not just a test for ggml's backend ops

* Increase support surface, remove seed

* make scaled kv generic

* Fix CI failings for test-kv-cache

https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392

* CI: execute kv-cache-scale tests

* exclude fp8-kv tests from OpenVINO

* Fix linting errors of test_kv_cache_scales

* Disable kv-ffload=false + multi-GPU tests for FP8

See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413

* Adjust test label for test-kv-cache.cpp

* Fix conflict-resolution mistakes in test-kv-cache.cpp

https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392

* Move KV tests to test-fp8.cpp

https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4410518158

* Remove tmp file for test-kv-cache-scales.py on Windows as well

* Fix fp8 kv cache test by population kv_source_impl

* Exclude models that consume KV cache directly

* Remove obsolete comment

* Allow BF16 scales as well for KV cache

* Rename *_cache_scale to *_cache_s

https://github.com/ggml-org/llama.cpp/pull/28392#discussion_r4517194999

* Move `is_type_scaled()` to `llama-kv-cache.cpp`
2026-11-10 18:08:32 +02:00
Georgi Gerganov 494d0f47bd ggml : add MUL_MAT src1 precision tests and gating (#31896) 2026-11-10 17:03:27 +02:00
chinmaykamjith001 45c8ea8374 common/json-schema : fix allOf with plain types (e.g. integer) generating an empty object grammar (#31834)
* common/json-schema : fix allOf with plain types (e.g. integer) generating an empty object grammar

* Fix empty range error and incorporated bot review feedback
2026-11-10 09:57:35 +02:00
Vladimir Sotnikov c90aa996ba common : escape control characters in tool argument strings (#31937)
normalize_quotes_to_json copies raw control characters verbatim while
converting pythonic tool arguments to JSON, so calls like
python(code='a<LF>b') produce invalid JSON in the arguments returned to
clients; replaying that history then fails hard. Escape C0 control
characters inside quoted strings via escape_json_string_inner.
2026-11-10 09:56:53 +02:00
18c5a95e2b sycl: Add device-side MoE scheduler (#30156)
* sycl: Add device-side MoE scheduler

Assisted-by: Codex

* Add width-heuristic override bit and Q2_K and Q3_K types

Assisted-by: Codex

* tests: cover skipped MUL_MAT_ID slots on the device MoE schedule

Assisted-by: Claude Opus 5.5

* sycl: keep graphs on for lazily reordered MUL_MAT and reject BN=64 combos on the device MoE schedule

The graph check disabled SYCL graphs until every MUL_MAT weight with a
reorder extra was reordered, which never happens for some (dense MXFP4,
the fused GLU plain path), so graphs stayed off and the check reran on
every compute. Upstream already reorders those during recording with
the async memory extension, so only MUL_MAT_ID weights are reordered
ahead of recording now.

ggml_sycl_grouped_dequant_gemm_device_supported() accepted any XMX
combo, but the device schedule only runs with BN == GGML_SYCL_FG_BN, so
a 32x64 combo (GGML_SYCL_XMX_GATHER_SHAPES=4 or 8) made the graph check
pass and the compute fall back to the host sync path while recording.

Assisted-by: Claude Opus 5.5

* sycl: keep the ggml_sycl_mul_mat_id_needs_sync name for the graph check

Assisted-by: Claude Opus 5.5

* fix: stop erroneously reordering host-offloaded tensors

With -ncmoe, prompt processing offloads the CPU-resident expert matmuls
to the GPU: the scheduler uploads the experts into a copy in the compute
buffer before every run. init_tensor gave that copy a reorder record, so
the MUL_MAT_ID gather path reordered every offloaded expert tensor on
every ubatch, and the next upload overwrote the result.

Qwen3-30B-A3B Q4_K_M, 1x B60, llama-perplexity -c 4096 -ub 2048
-ncmoe 48, 8 chunks, on master ac8e1694:
  master                        116 s, PPL 7.739
  master XMX_GATHER_TYPES=0      67 s, PPL 7.738
  with this change               67 s, PPL 7.729

Refs ggml-org/llama.cpp#31262

Assisted-by: Claude Opus 5.5

* Remove SYCL-specific test comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* sycl : name the MUL_MAT_ID path predicates after what they decide

ggml_sycl_mul_mat_id_mmvq_ok -> ggml_sycl_mul_mat_id_should_use_mmvq
ggml_sycl_mul_mat_id_mmvq_shape_ok -> ggml_sycl_mul_mat_id_mmvq_eligible
ggml_sycl_mul_mat_id_grouped_shape_ok -> ggml_sycl_mul_mat_id_device_sched_eligible
node_is_device_local -> graph_node_is_in_local_device
ggml_sycl_gg_rows -> ggml_sycl_grouped_gemm_rows
ggml_sycl_gg_tile -> ggml_sycl_grouped_gemm_tile (from #29245, renamed to match)

"eligible" is a necessary condition; the type and width checks follow
elsewhere. "should_use" means the path runs. No functional change.

Assisted-by: Claude Opus 5.5

* sycl : reorder Q3_K MoE experts and load them as vectors in the grouped GEMM

Assisted-by: Claude Opus 5.5

---------

Co-authored-by: cwriter <cwriter@localhost>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-10 09:42:25 +02:00
ae4893a092 vulkan: add tiled-transpose concat shader for Qwen models (#30591)
* vulkan: tiled transpose for dim-0 concat with a transposed source

The delta-net conv-state path does ggml_transpose() straight into a dim-0
ggml_concat(), so the generic concat kernel reads src1 with the transposed
stride, one element per thread, and neighbouring lanes land on different cache
lines. Stage a 32x32 tile through shared memory instead, which keeps both the
load and the store coalesced. 33 columns pads away the bank conflicts.

Gated narrowly: dim 0, 2D, unquantized 4-byte elements, contiguous src0/dst
rows, and src1 actually transposed. Everything else falls through to the
existing kernel.

Qwen3.6-35B-A3B on gfx1151/RADV, 30 gated-delta-net layers: the CONCAT op goes
11877 us -> 957 us, and pp2048 at ub2048 goes 1224 -> 1628 t/s. Generated text
is unchanged.

The approach follows the tiled transposed concat Gaetan Puleo wrote for the HIP
backend in his Strix Halo fork.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* enable by default, add smaller shader sizes

* clean up comments

* improve shader

* Add non-contiguous concat tests

Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>

* handle x or y axis overflow

---------

Co-authored-by: Nathan Wilson <nath.flagman151@passmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
2026-11-10 09:41:45 +02:00
figkcoci-debugandfigkcoci-debug 6122b566b0 ggml-rpc : fail gracefully on unreachable endpoints and failed handshakes (#30415)
* ggml-rpc: fail gracefully on unreachable endpoints and failed handshakes

rpc_dispatcher::start() aborted the whole process (GGML_ABORT -> SIGABRT) when
the endpoint could not be parsed, the transport failed to initialise, the socket
could not connect, or the HELLO handshake failed. A typo in --rpc or a
temporarily down node killed the host application, and the reason went to a log
stream the host may never see.

Make the connect-time failure reportable instead of fatal: start() returns bool
(LOG_ERROR + false; the dispatcher thread only starts after a successful
handshake), negotiate_hello() returns false on a half-open socket,
get_dispatcher() returns nullptr and does not cache failed attempts, and every
consumer treats nullptr as "endpoint did not answer" with the conservative
behaviour of its domain (allocation -> nullptr, alloc-size -> lower bound,
device memory -> 0/0, events -> no-op, device count -> 0, and add_server()
already maps 0 devices to nullptr).

Mid-stream failures keep the existing RPC_STATUS_ASSERT by design (fail fast
against silent corruption) and are intentionally out of scope.

Field motivation: on Android (llama.rn + phone RPC cluster) an unreachable
worker crashed the app; the crash buffer showed
abort <- ggml_abort <- rpc_dispatcher::start <- ggml_backend_rpc_add_server.

* tests : add a regression test for unreachable rpc endpoints

Calls ggml_backend_rpc_add_server with a socket that nothing listens on and with a
malformed endpoint, and expects nullptr from both. Without the fix in this branch
the first call aborts the process, so the test cannot even fail cleanly.

* ggml-rpc remove low-value comments

* tests : drop the internal device-count check after the rpc registry refactor

---------

Co-authored-by: figkcoci-debug <figkcoci-debug@users.noreply.github.com>
2026-11-10 09:40:35 +02:00
Ramzi 3b4d6ab3f7 chat : don't use lazy grammar when response_format is set with tools (#31956) 2026-11-10 01:44:07 +01:00
Matt Thompson 711f9f4c11 server: responses api: support conversion of text.format.json_schema to completions (#26145) 2026-11-10 00:43:38 +01:00
Xuan-Son NguyenandPascal 08f945ffb5 server: add oai-compat /v1/audio/speech endpoint (#31763)
* server: add oai-compat /v1/audio/speech endpoint

* add e2e pytest

* server: share the s16 rounding, type speech form fields

The WAV writer of the audio helper truncates while pcm_f32_to_s16
rounds, so wav and pcm differ by one LSB. Both now round.

pcm_f32_to_s16 takes a pointer and a count and converts the TTS
bytes without an intermediate copy.

/tts and /v1/audio/speech parse their body with parse_tts_body, so
multipart fields such as speed get their type before the speech
conversion checks them.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-09 23:58:52 +01:00
Xuan-Son Nguyen 3d94ffe452 mtmd: add mtmd_helper_resampler (#31964)
* mtmd: add mtmd_helper_resampler

* update docs

* add test

* MA_MAX_FILTER_ORDER
2026-11-09 23:04:39 +01:00
Piotr Wilkin (ilintar)andXuan Son Nguyen 5f490d7bfe feat: external PLE file loading (#31185)
* model : allow loading external PLE files

Assisted-by: Codex

* convert : add separate PLE exports

Assisted-by: Codex

* tests : fix PLE export test lint

Assisted-by: Codex

* model : make --ple an alias of --shard, allow multiple shard locations

Reuse the multi-shard loading path for external PLE files instead of a
separate option.

* --ple is an alias of --shard: both take a comma separated list and may
  be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
  auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
  exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
  with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
  common_model_load() uses llama_model_load_from_splits for two or more
  paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
  memory measurement load every shard

Assisted-by: GPT-6-Astra

* rm -isnm

* make sure to dedup paths

* rm test_convert_ple

* address comments

* allow ple shard to override tensor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-09 19:41:01 +01:00
Julius TischbeinandOliver Simons 9d340ec341 Windows unbuffered model load (#26014)
* Windows unbuffered io

* Fix read len calculation

* Add test and model load timing

* chunked fallback read + removing filepointer semantic divergence between linux and windows

* tests : remove unused mmap test arguments

Assisted-by: Codex

* remove timing, set file pointer after chunk read, test refinement

* Update tests/test-llama-mmap.cpp

Co-authored-by: Oliver Simons <osimons@nvidia.com>

* Update src/llama-mmap.cpp

Co-authored-by: Oliver Simons <osimons@nvidia.com>

* test refinement

Assisted-by: Codex

* remove duplicate constexpr

Assisted-by: Codex

* adding label behavior to tests

Assisted-by: Codex

---------

Co-authored-by: Oliver Simons <osimons@nvidia.com>
2026-11-09 19:22:08 +01:00
Georgi Gerganov d628a98f09 tests : keep dummy ffn.act scales away from zero (#31940)
The dummy-model generator fills every tensor from a plain normal pool.
`blk.N.ffn.act.scales` is a divisor in the mpt graph, so with seed
1224489258 it draws |v| ~ 1e-6, and `ffn_act = gelu(...) / scales` reaches
2.7e5. Metal stages both operands of `kernel_mul_mm_f32_f32` as half, so
those activations overflow to inf, the norm downstream turns the inf into
nan and the logits come out nan, while the CPU reference stays in f32 and
stays finite. Both NMSE values then become nan and the check fails.

Give `ffn.act` the same treatment as `rope_factors` and keep its magnitude
away from zero.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-11-09 14:54:27 +02:00
losewayy 4b46d96538 metal : make ggml_tri select instead of scaling by the comparison (#30711)
* metal: make ggml_tri select instead of scaling by the comparison

The kernel multiplied by the 0/1 comparison, which turns a +-inf outside the triangle into NaN, while the CPU kernel writes a plain zero there. Select, like the CPU does.

Adds test_tri_special, which puts +-inf into the cells that have to be discarded, for all four triangle types. It fails against the previous kernel and passes with this change.

* Just trim down the repetitive comments.
2026-11-09 13:31:16 +02:00
Pascal 6590d25a32 tests: bound the test_rope tolerance by the f32 angle resolution (#31874)
The rotation angle of a position p is rounded in f32 to about
p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach
it through different rounding sequences (cumulative product versus
pow). With positions up to 8192 this puts the NMSE floor between
backends near 8e-8, so the default 1e-7 threshold fails at random on
Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for
n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
2026-11-09 09:31:13 +01:00
Xuan-Son Nguyen 7d7f028753 mtmd: refactoring: remove clip.h, consolidate code into mtmd (#31865) 2026-11-08 15:49:06 +01:00
57d04820df ggml : update CUDA/Metal radix Top-K to make tied selections deterministic (#31084)
* ggml : fix for the radix TOP_K thread timing bug

* make the key an ordered 64-bit pair

* a rewrite without shift-skip logic

* fix casts

* warp-contiguous layout

* reduce the scope removing vulkan+metal changes

* turn it off for Metal/ROCm, note about cub::DeviceTopK

* fix the hip build

* skip Vulkan as well

* popc lambda from fattn

* do block_cols math in int64_t

* make it allowlist bc SYCL looks non deterministic too

* ggml : make Metal radix Top-K tied selections deterministic

The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.

Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.

This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* tests : enable deterministic top-k check on Metal, add radix-path tie cases

Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.

Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* ggml : use coalesced accesses in the Metal radix Top-K compaction

The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.

Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* tests : remove the extra top-k tie and perf cases

The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal: scan the radix bins and warp totals with a simdgroup

The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.

* tests: enable the deterministic top-k check on ROCm

HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.

* cuda: remove the redundant radix top-k counter reset

Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-08 16:42:57 +02:00
sargentmajordandSargentmajord 9982004913 ggml-cuda : add d_state=64 fast path to SSM_SCAN kernel (#31818)
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.

Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).

Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.

Assisted-by: Claude Sonnet 5

Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
2026-11-08 15:48:20 +02:00
LeonandMasashi Yoshimura a691b23458 webgpu : add COUNT_EQUAL support (#31578)
* webgpu : add COUNT_EQUAL support

Assisted-by: Codex

* webgpu : reduce small COUNT_EQUAL inputs in one dispatch

Assisted-by: Codex

* webgpu : simplify COUNT_EQUAL and reuse existing tests

* webgpu : move COUNT_EQUAL pipeline cache declaration

* Update ggml/src/ggml-webgpu/ggml-webgpu.cpp

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>

* tests : simplify COUNT_EQUAL coverage

---------

Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>
2026-11-08 11:50:09 +02:00
PascalandGeorgi Gerganov d6be065572 llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (#31851)
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls

Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.

* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends

Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.

* tests: cover GGML_PREC_BF16 in the flash attention tests

The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.

* llama: request GGML_PREC_BF16 for the KQ multiplication

* cont : add TODO in clip.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-08 08:37:46 +01:00
Xuan-Son Nguyen a2654ec431 mtmd: move log-mel computation to cgraph (#31808)
* mtmd: move log-mel computation to cgraph

* rm stray file

* also include source in mtmd_audio_track

* rm stray file

* rm mtmd_audio_stft_op

* add test
2026-11-07 23:53:11 +01:00
Nikhil Jain cf659f240b webgpu : Implement group norm kernel (#31490)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* Add WebGPU group_norm kernel

* clean up

* resolve merge conflicts

* parameterize data type; clean up cmake changes

* add nullptr check

* clean up webkit experiments

* more cleanup

* more cleanup

* clang format

* Remove unused feature check for WebGPU

* clean up supports_op condition and update documentation

* add perf tests for group_norm

* clean up doc with script and undo change in test-backend-ops

* Do 2d dispatch of workgroups

* Do 2d dispatch of workgroups

* Unstage unrelated changes

* undo docs churn

* add new test case

* undo small change

* undo small change
2026-11-07 10:01:27 +02:00
usrlocalben bcb0152876 model: add Kimi-K3.1-Pro text model (#31779)
* model: add Kimi-K3.1-Pro text model

K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.

Indexer:
  - the selection is an additive mask over the whole cache, not a gather of
    the top-k rows, so it reuses the existing sparse flash attention path;
    off that path the mask stays correct but still costs a kq over the cache
  - the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
    because dsa_config declares no eps of its own

Graph and cache:
  - new arch with four indexer tensors per DSA layer, and a hybrid memory
    that keeps a K-only MLA cache for those layers, recurrent state for the
    KDA layers, and the indexer key cache
  - the graph node budget shares the K3 entry

Conversion:
  - KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
  - dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
    plus the indexer LayerNorm eps; the layer list is 1-indexed like
    linear_attn_config's, and the converter asserts that it covers exactly
    the full-attention layers
  - the routed experts stay MXFP4 and are repacked as in K3, not dequantized
  - the vision tower and mm_projector are skipped: text only for now
  - the arch key follows the checkpoint's own model_type ("kimi_k31") and the
    loader registers KimiK31ForConditionalGeneration; the release name carries
    a -Pro suffix that nothing in the config or the tensor names uses

Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.

Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.

* converter: avoid clutter in the k3 base

* narrow comment to match model

* update metal fusion expectations

* restore

* sync with PR 31733 refactoring
2026-11-06 23:42:46 +01:00
669204ee2b hexagon: conv_2d optimizations and support for conv_3d (#31814)
* hexagon: Generalization of CONV_2D implementation for CONV_3D

- New op HTP_OP_CONV_3D is wired through htp-ops.h, main.c (shares op_conv_2d) and the op remap in htp-kparams.cpp.
- conv_3d runs on the existing HMX tiled (implicit GEMM) conv2d path. KD and KH are folded into one virtual kernel row (KD*KH), and output rows are counted as N*OD*OH.
- htp_kparams_solve_conv3d checks the conv_3d layout and calls the tiled solver htp_kparams_conv2d_tiled, which takes an is3d flag.
- The tiled kernel takes the sizes and strides that differ between conv_2d and conv_3d from a struct htp_conv_geom. Only the input stage unfolds depth.
- 256-entry DMA queue can no longer overflow

Assisted-by: Claude Code

* hexagon: modify checking to eliminate divide-by-zero hazards

* hexagon: conv2d tiled: compute the conv_3d unfold divisors on the host

The KH and OH fastdiv values used to unfold conv_3d kh into (kd, kh) and
oh into (od, oh) were built per op on the device, each with a 64-bit
divide. Pass them in the kernel params instead; they fit in the space the
window variant already reserves, so the struct size does not change.

Assisted-by: Claude Code

* hexagon : preserve fold scheduling in conv2d for small size

hexagon : support arbitrary positive conv2d strides
Support positive horizontal strides for WHCN and CWHN inputs. Preserve
load/shuffle paths for strides 1/2 and use gather for larger strides.

hexagon : improve conv2d tile size selection

Prefer the largest feasible output-channel block and derive spatial
chunk capacity from VTCM usage. Compare capacity and thread-aligned
candidates using per-thread work and chunk overhead.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-conv: fix compiler warning

---------

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-06 11:51:57 -08:00
Georgi Gerganov 8820b088de tests : qualify isnan and signbit with std:: (#31791)
* tests : qualify isnan and signbit with std::

The unqualified calls rely on the C math.h names also being present in the
global namespace, which is not the case for the NetBSD 11.99.9/x86_64 gcc
14.4.0 build reported in the issue.  Prefix the calls with std:: to select
the standard C++ overloads declared by <cmath>.

ref: https://github.com/ggml-org/ggml/issues/1681

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : qualify C math calls with std::

Follow-up to the isnan / signbit fix: make the remaining math calls in
test-quantize-fns.cpp consistent with the std:: isfinite / max / numeric_limits
already used in the same file.

The float variants are overloads in C++, not distinct names, so the mapping is
fabsf -> std::fabs, copysignf -> std::copysign, nextafterf -> std::nextafter -
not std::fabsf & friends, which do not exist in libstdc++ and would break the
gcc build.  NAN and INFINITY are macros and are left unqualified.

Only behavioural difference: array_rmse() fed a double accumulator to sqrtf(),
now std::sqrt() keeps the computation in double.  The result is more accurate
and differs from the previous value by at most 1.4e-7 relative, three orders of
magnitude below the tightest tolerance in this test.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-11-06 14:53:04 +02:00
Georgi Gerganov 9a1d03c7ca tests : override hardware_concurrency in CI (#31797)
The self-hosted CPU runners are containers with 8 CPUs, but
std::thread::hardware_concurrency reports the cores of the host. The
table mode of test-llama-context sizes its pool from that count, so it
runs 4 models x 8 threads on an 8-CPU job and hangs.

Add TEST_ARG_HARDWARE_CONCURRENCY and TEST_ARG_PARALLEL_WORKERS as the
defaults for the auto-selected counts of test-llama-context and of
test-backend-ops, and set them to 8 and 4 for cpu-x64-high-perf.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-11-06 14:52:26 +02:00
Sigbjørn Skjæret d38bea3422 tests : drop u8path() (#31755)
[no ci]
2026-11-05 23:06:51 +01:00
Xuan-Son Nguyen 6023692b1e llama: fix kv shifting, add tests, add per-model build_rope (#31683)
* llama: fix kv shifting and add tests

* llama: add graph build_rope (#31693)

* llama: add graph build_rope

* improve

* migrate new models

* update review skill

* fix ci

* add --kv-shift to CI
2026-11-05 21:10:36 +01:00
fecdc81eec model : add Ripple (ricursiveai) (#31692)
* feat : added ricursiveai ripple support

* fix : mandatiry SWA, scale through helper, comments back

* fix : metal

* model : make ricursive RoPE/NoPE explicit and validate the layer pattern

Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.

Assisted-by: Claude Opus 5.5

* model : make ricursive swiglu_clamp per-layer

Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.

Assisted-by: Claude Opus 5.5

* model : use the ricursive per-tensor scales and RoPE parameters

Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.

Assisted-by: Claude Opus 5.5

* convert : drop the ricursive chat_template.jinja override

The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.

Assisted-by: Claude Opus 5.5

* fix : remove NVFP4 checkpoint quantization block

* match #31733 refactor

---------

Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 19:55:15 +01:00
45b6604b20 model: Add support for the xing4_0 (#31589)
* model: add support for Xing4.0

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/xing4_0.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* fix: rename model file to xing4-0.cpp to match CI expectations

* model: rename xing4_0.cpp to xing4-0.cpp

* lint: fix E128 continuation indent in conversion/xing.py

* tests: add xing4_0 fusion baseline for Metal

* reduce comments

---------

Co-authored-by: shixq7 <shixq7@chinatelecom.cn>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-05 17:47:27 +01:00
Georgi Gerganov ea40a3a1e1 metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (#31615)
* metal : support scale-bias routing norm in the TOPK_MOE fusion

The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sqrt(softplus) MoE routing with a selection bias

Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* metal : fuse sigmoid MoE routing with a selection bias

The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.

- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
  GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
  tail; the selection-bias ADD is required, same as the sqrt(softplus)
  chains
- generalize the gating function constant from a sqrt(softplus) bool
  to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
  scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
  CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
  weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
  deepseek2/32, kimi, minimax, ...) now fuse their routing, and
  muse-spark prefill drops one MOE_REDUCE fire because the reordered
  graph allocates the routing weights inside the reduce output range,
  which the dst/input overlap check correctly declines

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* cont : remove low-signal comments

* ocd : valign ggml_metal_fusions
2026-11-05 13:52:56 +02:00
Takamichi Ohara 1fa7f1c488 ggml : support truncate=false for YaRN (#30206)
* ggml : support `truncate=false` for YaRN

YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.

Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.

* support CUDA backend

* add tests

* move RoPE helpers to internal header

* format
2026-11-05 12:02:05 +02:00
Hoang Vien DuyandGeorgi Gerganov f7b41cca4f metal : use mul_mm for 32 <= K < 64 (#30982)
* metal : use mul_mm for 32 <= K < 64

* metal : add bf16 x bf16 mul mat kernel

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-05 10:58:41 +02:00
Leon 553d2c2e7e webgpu : add MEAN support (#31591)
Assisted-by: Codex
2026-11-05 10:29:14 +02:00
Pascal a01dff908b tests: fix CI flakes of TOPK_MOE caused by near ties (#31426)
* tests: fix CI flakes of TOPK_MOE caused by near ties

With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.

Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.

* tests: build the TOPK_MOE logits by inverting the gating function

Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.

The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
2026-11-05 07:31:48 +01:00
anujj 5ce06fc71f Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ (#31296)
* cuda: fuse NVFP4 gate/up and GLU in MMQ

* cuda: honor the requested src1 precision in fused gate/up MMQ

* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels

* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming

* cuda: check gate_scale before loading the fused NVFP4 gate scale
2026-11-05 10:47:33 +08:00
Georgi Gerganov 4c4ca5b188 common : do not allow _ in sidecar (#31639) 2026-11-04 10:32:45 +02:00
Georgi GerganovandPascal 4b29d02629 common : fix -hf tag matching inside longer quant names (#31620)
* common : fix -hf tag matching inside longer quant names

":F16" picked a BF16 file when the repo had both, since "BF16"
contains "F16". The same applied to ":Q4_XS" vs "IQ4_XS". The tag
now has to start at a token boundary in the path.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* common : accept underscore as -hf tag boundary

Some repos separate the quant with an underscore, e.g.
stepfun-ai/Step-3.5-Flash-GGUF-Q4_K_S ships
step3p5_flash_Q4_K_S-00001-of-00012.gguf, which :Q4_K_S no longer
matched. Add _ to the token boundary and pin the tag and default
lookup cases.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-04 09:22:54 +02:00
Todor BoinovskiandMax Krasnyansky bee23322f1 hexagon: add LIGHTNING_INDEXER support (#31289)
* hexagon: initial support for LIGHTNING_INDEXER op

* hex-lght-idxr: hmx for prefill

* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute

* hex-lght-idxr: pipelined hmx path for decode and prefill

* hex-lid: add support for K (f32)

* hex-lid: add support for K (f32)

* hex-lid: add support for q4_0

* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills

* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel

* hex-lit: update dma pipeline to use canonical dbl buffering in all paths

* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths

* hex-lit: add support for dims collapse

* hex-lit: make kparams constistent with other ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 21:17:30 -08:00
Xuan-Son Nguyen 149f5641b9 mtmd: accurate support for qwen3-vl video prompt format (#31619) 2026-11-03 23:57:26 +01:00
Todor Boinovski 7071ba378a hexagon: add TOP_K_MASK op (#31552)
* hexagon: add TOP_K_MASK op

* Simplify Hexagon TOP_K_MASK kernel and do not duplicate host configuration
2026-11-03 12:04:03 -08:00
Todor BoinovskiandMax Krasnyansky 20a51f24a8 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
* hexagon: add support for DSV4_HC_PRE

* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization

* hex-dsv4-hc-pre: double-buffered DMA for prefill

* hex-dsv4-hc-pre: refactor DMA block setup

* hex-dsv4-hc-pre: precompute kernel params

* hex-dsv4-hc-pre: support for n_hc != 4

* hexagon: add support for DSV4_HC_POST

* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block

* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks

* hex-run: add support for test-llama-archs tool

* hex-dsv4: refactoring and optimizations

Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.

Tested with test-llama-archs -a qwen4ext and test-backend-ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 08:26:18 -08:00
Giulianno 9309bf1bcf model : Add support for Metis-1.6 models (#31134)
Assisted-by: Claude
2026-11-03 16:24:45 +01:00
Bhargav Krish 740251a920 cuda : gather batched tensor reads into mapped host memory (PR to master) (#31391)
* cuda: gather batched tensor reads into mapped host memory

Assisted-by: Codex

* add comments

Assisted-by: Codex

* simplify the implementation based on checks that exist earlier in the flow

* simplify batched tensor gather indexing

* add conditional dispatch

* address comments - make host registration raii, better test mindful of ci time, assert instead of return false

* cuda : gather small reads and use pinned DMA for large reads

* cuda : use std::vector for host gather descriptors

* cuda : add 8-byte gather copies, generalize tensor test and log registration failures

* common buffer for descriptors

* simplify stream sync + increase to 128k descriptor pinned buf
2026-11-03 15:31:50 +05:30
Aman GuptaandPratik Narola 259d91709d meta: support GLM-5.3 next sm tensor (#30499)
* meta: support GLM-5.3 next sm tensor

* glm5-next : cont q_absorbed for -sm tensor with non-unified KV

* meta : keep GGML_OP_ARANGE in ggml.h order

* glm5-next : split MLA heads in pairs for -sm tensor

The MLA cache has a single KV head, so the heads on a device are its GQA
ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3
devices the 16 heads of the test model were split 5/5/6 and it aborted.

* meta : keep the split for views inside one slab of the split axis

With a non-unified KV cache, build_attn_mha splits q into streams with a
view. For a permuted q, e.g. the absorbed query of glm5-next, that view
aborted with "view of permuted tensor not implemented". The split is now
kept if every other dim of the source and the view stays inside one slab
of the split axis. The per-device strides are only rescaled above it.

From tetherto/qvac-fabric-llm.cpp#353.

Co-authored-by: Pratik Narola <pratik.narola@tether.io>

* glm5-next : drop the q_absorbed cont

The meta backend can now split the per-stream view of the permuted q, so
the cont is not needed for -sm tensor with a non-unified KV cache. With
the cont, the q of flash attention was not contiguous, and when FA ran on
the CPU (WebGPU) the meta get_tensor aborted.

---------

Co-authored-by: Pratik Narola <pratik.narola@tether.io>
2026-11-03 10:51:10 +02:00
frogscooper 4402e65401 chat : allow constraint type after gpt-oss final channel (#31556) 2026-11-03 07:55:11 +02:00
Leon 713319e284 webgpu: add ARANGE support (#31455) 2026-11-03 07:54:00 +02:00