Commit Graph
11694 Commits
Author SHA1 Message Date
Aaron Teo b84ea5cf5e convert : speed up model endian conversion and add more datatype support (#30647)
* convert: speed up model endian conversion

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* convert: explicitly state noop

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* convert: add missing data type support

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-16 10:18:29 +02:00
Michael Taylor a30fb4924d [SYCL]: fix memory errors in mul_mat, split buffer, host pool (#29889) 2026-10-16 03:28:50 -04:00
Sh1m13 495c5805ce gguf-py : add numpy Q1_0 and Q2_0 de/quantization support (#30630)
* gguf-py : add numpy Q1_0 and Q2_0 de/quantization support

* cont: added q1_0 / q2_0 to the test list removing the small test

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:26:14 +02:00
uvos 623c16393c CUDA: HIP: rms_norm: add variants that load float4 at a time and store x in registers (#29720)
* CUDA: HIP: rms_norm: load float4 at a time and store x in registers

* Add template paramter comments

* use ggml_cuda_is_aligned for src

* nits

* also use ggml_cuda_is_aligned for src in fused variants

* fix nits
2026-10-16 10:07:34 +03:00
hayder 13b8048920 common : read ngram cache parts by reference instead of copying (#30061) 2026-10-16 10:06:32 +03:00
FINAL-Bench 0f4376268f convert : support text-only checkpoints in pplx-decider (#30575)
* pplx-decider-text-only

convert : support text-only checkpoints in pplx-decider

* Apply suggestions from code review

* convert : remove unused import
2026-10-16 08:56:00 +02:00
Trivalik 3dbcb7fca5 cpu : reduce AVX2 tiled accumulators together (#30296)
Assisted-by: Codex
2026-10-16 09:48:23 +03:00
Aaron Teo c804754529 ggml-cpu: refactor s309x repack helper functions (#30588)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-10-16 09:45:35 +03:00
Neo Zhang c0083e3052 sycl : reduce the build threads to half (#30236) 2026-10-16 09:44:24 +03:00
Titaniumtown 13de11e78f sycl: wide stores for the q4_K/q5_K weight dequant (#29696) 2026-10-16 09:43:07 +03:00
Georgi Gerganov 88aaf325a7 tests : catch vanished rows in the fusion baseline (#30618)
* tests : update the Metal fusion baseline

Re-record the MTL fusion baseline with the current build: 276 rows over 131
dummy models.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : fail test-fusion when a baseline row stops matching

The report was built from the counters the run produced, and the baseline was
only ever used as a lookup table, so a fusion that stopped matching produced no
row and was never compared. qwen4exp and mellum lost their fusions this way
while the Fusion / metal job stayed green.

Report a baseline row that the run does not reproduce as a regression. Scope
the pass to the arches that were actually run so --model keeps working.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : never write inline comments unless the user asks

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : keep the mellum dummy model softmax-gated

mellum reads expert_gating_func from the GGUF since #30578, while the fixture
writes sigmoid for every MoE arch, so the Mellum dummy model stopped building
the softmax MoE routing fusion. Write softmax for mellum, matching the arch
default and the GGUFs that conversion/mellum.py produces.

Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : use emoji in the AI stats disclosure

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : list the disclosure model after the hardware

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : use monochrome symbols in the AI stats disclosure

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* tests : add muse-spark to the Metal fusion baseline

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-16 09:35:19 +03:00
Ruben Ortlam a8d9a7e1e5 vulkan: fix gpu-visible host memory issues (#30468)
* vulkan: fix log spam if host pointer import fails

* exclude MoltenVK from shared memory import

* remove unreachable code, assert instead. Shorten warnings

* disable host memory buffer sharing on AMD due to amdgpu perf issue
2026-10-16 08:29:23 +02:00
Karthik VijayanandKarthik Vijayan f12aaa344c Fix failing RocM tests (#30611)
Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
2026-10-16 12:46:55 +08:00
Mikhail Podvitskii 5e03a1ff45 graph : add precision handling for MELLUM architecture (#30584) 2026-10-16 11:32:26 +08:00
Max Krasnyansky b797b9ca81 hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round

Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors

* hex-get-rows: vectorize raw k-quant ukernels

* hex-binary: refactor dma-loop thread functions (no need for macros, etc)

* hex-quant: move/cleanup flat and tiled dequant ukernels

* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights

* hex-get-rows: remove redundant cast

* hex-repack: proper filtering for BF16 that were leaking through during the tests

* hex-repack: streamling tiling rules (broken MM for now)

* hex-repack: fix mmid after tiling updates

* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars

* hex-mm: fix valid_rows calc in various kernels

* hex-mm: remove unused hvx_mm_nx_2d_flat

* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16

* hex-mm: another pass at hvx-tiled f16 matmul

* hex-get-rows: optimize small src1

* hex-get-rows: one more pass at recovering get-rows perf regression

* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py

* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul

* hex-mm: fold tiled_f16 checks into the general flow

* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op

* hex-quant: proper tail handling in new dequant ukernels

* hex-get-rows: remove unused code and cleanup support checks

* hex-mm: cleanup ne1 padding/tiling checks

* hex-mm: cleanup fusion checks for tiled tensors

* hex-repack: add a little can_repack helper to reduce duplication

* hex-repack: never repack host buffers

* hex-repack: factor out repack functions into separate .h and .cpp

* hex-tensor: cleanup tensor ifaces for tiled/padded sizes

* hex-repack: update k-quant tiling comments and cleanup #defines

* hex-dcache: factor out l2/dcache dirty tracking and flushing

* hex-tensor: update docs and remove #defs duplication

* hex-tensor: remove unused/incorrect get_data interface

* hex-get-rows: cleanup vtcm layout build

* tests: disable huge MM tests

* hex-dcache: add missing header
2026-10-15 20:30:11 -07:00
Aman Gupta 6afa9a3134 llama : share the k-pool input views between layers (#30613)
The scheduler uploads each view of a host input on its own, so the per-layer reshape of new_pool_idxs (and of sel_mask in glm5-next) was copied once per layer.
2026-10-16 11:14:36 +08:00
Gaurav Garg b6334e37e6 Limit max CUDA pinned host buffer size to 4 GB (#30449) 2026-10-16 08:20:26 +05:30
Frosty40 e707790c73 sycl: support reordered Q8_0 MoE weights (#29337)
Preserve the existing MoE accumulation order.

Assisted-by: Codex
Assisted-by: Angel
2026-10-15 21:59:38 -04:00
Neo ZhangandSigbjørn Skjæret 43d57fb8ca [SYCL] Enable ssl in build script and CI (#30456)
* enable OPENSSL=ON as default in build script, update it in CI and dockerfile

* change to LLAMA_BUILD_BORINGSSL on windows

* refactor

* rm Download UI build

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 21:58:07 -04:00
Xuan-Son Nguyen 2851b3bcbf mtmd: add mtmd_audio_stream + support voxtral-realtime (#30629)
* mtmd: add mtmd_audio_stream + support voxtral-realtime

* address review
2026-10-16 01:46:42 +02:00
Nigel JonesandGabe Goodhart 59d5fad7c1 convert : fix Granite Switch block_count for MultiSwitch checkpoints (#30621)
* convert : fix Granite Switch block_count for MultiSwitch checkpoints

Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>

* Update granite.py

Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>

---------

Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>
2026-10-16 00:41:01 +02:00
Matt Rutkowski e287bd8729 convert : fix regression in mapping of Granite speech tensors (#30628)
Signed-off-by: Matt Rutkowski <mrutkows@us.ibm.com>
2026-10-16 00:31:14 +02:00
Said Taghadouini 2f729cfa4c mtmd: lightonocr does not require longest edge to always be 1540, native resolution is the default (#30600)
Assisted-by: Codex
2026-10-15 23:53:02 +02:00
ba3c05f71d model: add Muse Spark Open (text, vision and audio) (#30616)
* model: add Muse Spark Open (text, vision and audio)

Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.

Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.

The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Muse Spark: drop the redundant forward expand

* Muse Spark: drop the redundant forward expand and tighten the comments

* Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/llama-graph.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 23:27:12 +02:00
Georgi Gerganov b76b13379d pi : expand AI disclosure with hardware and stats (#30615)
* pi : expand AI disclosure with hardware and stats

Prefer PI_HARDWARE for the hardware field, spell out the disclosure block
template (model link, harness, hardware, stats) and source [STATS] from
~/.pi/stats.sh. Record the hardware in the Assisted-by commit tag.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

* pi : drop square brackets from placeholders

Brackets inside a markdown link text break the link on GitHub. Describe the
disclosure bullets in words instead of embedding a bracketed template.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-15 20:32:52 +03:00
Georgi Gerganov 620d2225cc llama.cpp : bump version to 0.7.0 (#30603) 2026-10-15 16:48:22 +03:00
Kartik Gulia cd0a5b6a18 models : add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support (#30520)
* add support

* remove unneeded bos fix

* add video flag

* fix flake error

* remove video-specific budgeting

* convert : fix lilicorr-dflash bid clash

* spec : enforce lilicorr n_tokens condition

* spec : fix lilicorr walk

[no release]
2026-10-15 16:34:56 +03:00
Georgi Gerganov a11b6530b5 scripts : add ci-status.sh to list active CI runs (#30586)
Small helper around the GitHub Actions API: lists the workflow runs that are not
finished yet (in_progress, queued, pending, waiting, requested), grouped by status.

Supports selecting runs by status, by branch (server-side, to reduce API traffic)
or by a regex on the branch name, and can cancel the selected runs via --kill.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-15 16:28:09 +03:00
Pascal 61ad80bf93 server: fix the off-by-two in the SWA reuse threshold (#30580)
The first evaluated token attends to the positions above
pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the
whole window, but the reuse threshold asked for two more cells.
A saved SWA state keeps only the cells inside its window, so a
state read back from a slot file or from the RAM prompt cache was
always rejected and the slot rolled back to a checkpoint.

The SWA threshold now matches the window, while a state without SWA
keeps the exact bound. The test harness gains override_kv, used to
give the SWA test model a window smaller than the test context.

[no release]
2026-10-15 16:25:19 +03:00
Mikhail PodvitskiiandSigbjørn Skjæret a6b91d8c5e model : Mellum - read MoE gating params from GGUF, unify RoPE (#30578)
* model : mellum - read MoE gating params from GGUF, unify RoPE

- read/write expert_weights_norm and expert_gating_func instead of hardcoded values (defaults unchanged)
- use per-layer rope params via get_rope_freq_base/scale
- check n_expert once, outside the layer loop

* Update conversion/mellum.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

[no release]
2026-10-15 16:24:47 +03:00
Pascal ea736c1d5c server: save the draft state in the slot file (#30574)
The slot file held the target context and the context checkpoints,
but never the draft context, so a restored slot drafted with an
empty or stale draft KV until the cache refilled.

The file now ends with the live draft state, the full draft context
and the speculative state, in the same appendix style as the
checkpoints. A draft state that does not fit the draft context is
dropped and the draft starts empty, while older files and older
servers keep working.

[no release]
2026-10-15 16:24:19 +03:00
Pascal b0d239164b server: prefer empty slots and keep states overwritten by child tasks (#30567)
Slot selection takes free slots by id or by last use only, so child
tasks often land on a slot holding a parent state while empty slots
are idle, and overwrite it without saving it to the prompt cache.

Free slots are now ordered empty first, then least recently used, for
both the LRU fallback and the child slots, and a child task saves the
state it overwrites. With 4 grouped decision requests in flight and
12 states in rotation, prompt cache reads go from 5 to 36.

[no release]
2026-10-15 16:24:01 +03:00
Georgi Gerganov c0cb53ac4e sync : ggml 2026-10-15 16:20:14 +03:00
Georgi Gerganov 4ed1c8507e ggml : bump version to 0.27.0 2026-10-15 16:20:14 +03:00
Sigbjørn Skjæret 711780ffc7 ci : bump ty to 0.0.88 (#30587)
* bump ty to 0.0.88

* arange -> linspace.tolist() to pass expected type
2026-10-15 14:15:55 +02:00
Aleksander Grygier fd55cb98fe fix(ui): take the agentic turns default from DEFAULT_AGENTIC_CONFIG (#30582)
The registry hard-coded 10 for AGENTIC_MAX_TURNS while DEFAULT_AGENTIC_CONFIG
said 100. The registry seeds SETTING_CONFIG_DEFAULT, so the loop's
`|| DEFAULT_AGENTIC_CONFIG.maxTurns` fallback never fires and users got 10
turns. Reference the shared constant, as the adjacent MCP timeout does.

Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
2026-10-15 12:36:06 +02:00
Karthik VijayanandKarthik Vijayan 3e4dfa8f39 models : add maion-coder (#29778)
* Add Support to maion-coder arch

* iAdd Support to support Draft models

* Address comments

* Remove maion context shift function

---------

Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
2026-10-15 12:18:46 +03:00
William Tan 08698c81ed jinja : improve context copy constructor performance (#30562)
Assisted-by: Codex
2026-10-15 11:00:57 +02:00
Yuri Khrustalev 82e1e2fd84 jinja : write nan and inf in tojson as json.dumps does (#30483)
tojson wrote nan, inf and -inf for these floats, as values and as dict
keys; json.dumps writes NaN, Infinity and -Infinity. The string and safe
filters of a float keep nan and inf, as Python's str() does, so they no
longer use the tojson function. float_to_json is renamed float_repr, as
it now gives str(); float_json gives the json.dumps text.

Also say in a comment why a last digit 9 is not bumped at a power of 2.
2026-10-15 10:49:40 +02:00
losewayy 47f5a64ae1 vulkan : fix concurrent first device init (#29766)
* vulkan : fix concurrent first device init

* tests : load models in parallel in test-thread-safety

Loading them serially never exercised concurrent first backend init, so the crash reported in #29764 was invisible to this test.

* vulkan : create fence before retaining device in buffer context
2026-10-15 10:37:35 +02:00
Neo Zhang e36df60470 sycl : replace oneAPI package by DLE+oneDNN package (#30131)
* replace oneAPI package by DLE+oneDNN package

* support to install oneDNN in windows, in release and docker

* for internal test

* remove test code

* Update intel.Dockerfile
2026-10-15 09:26:01 +02:00
lhezandHongqiang Wang 9166ceaf16 opencl: fix OOB tail access for adreno noshuffle gemv kernels (#30421)
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-10-15 10:21:18 +03:00
William Tan 2130ca8476 jinja : fix loop body variable scoping (#30533)
Assisted-by: Codex
2026-10-15 09:14:01 +02:00
Adrien Gallouët 23f83e056d ggml-cpu : optimize pad_f32 (#30526)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-15 09:13:18 +02:00
Simone IacominoandSigbjørn Skjæret 2ce20d22fd unicode : reject overlong UTF-8 and out-of-range codepoints (#30485)
* unicode : reject overlong UTF-8 and out-of-range codepoints

* Update tests/test-unicode.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* tests: document byte-wise UTF-8 error recovery

* break out of first test on failure

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-15 10:12:56 +03:00
Adrien Gallouët 68ac765726 llama : split llama_prefetch() into merge and prefetch steps (#30532)
* llama : split llama_prefetch() into merge and prefetch steps

Merge the ranges first, then call the system functions.

This fixes -Wunreachable-code-return on Windows.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Use const_cast

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-15 09:12:45 +02:00
lhez 77c1f39929 opencl: apply flash attn softcap before mask (#30546) 2026-10-15 10:11:51 +03:00
Nathan WilsonandRuben Ortlam 733f57a999 vulkan : copy strided f16 KV once on AMD so reads spread across memory channels (#27703)
* vulkan : copy strided f16 KV once so reads spread across memory channels

Assisted-by: Claude (Opus 4.8 / Opus 5)

* tests : cover the f16 KV copy to scratch

Assisted-by: Claude (Opus 4.8 / Opus 5)

* vulkan : gate the f16 KV scratch copy to AMD

Assisted-by: Claude (Opus 4.8 / Opus 5)

* vulkan : reuse the generic copy shader for the f16 KV scratch fill

The f16 clause needs only a strided copy, not a dequant, so drop the
dedicated shader and dispatch the existing copy pipeline instead. The
copy is described in source memory order so the strided access lands on
the writes, which stream, instead of the reads, which would concentrate
on a few memory channels.

Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5)

* vulkan : extend the strided KV scratch copy to bf16

Assisted-by: Claude Opus 5

* vulkan : clean up comments

Assisted-by: Claude Opus 5

* consolidate copy shaders

* disable on RDNA4

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-10-15 10:09:02 +03:00
cqderekandravi9 5a638aedb7 cpu: add F16 SWIGLU_OAI reference implementation and validation (#29200)
* cpu: add F16 SWIGLU_OAI reference implementation and F16 activation op test coverage

Adds ggml_compute_forward_swiglu_oai_f16() to the CPU backend, mirroring
the existing F32 path, and extends test-backend-ops with F16 test cases
for SWIGLU_OAI so downstream backends(e.g. Hexagon) can be CPU-diffed
for correctness.

* ggml-openvino: fix f16 SWIGLU_OAI type mismatch in alpha/one constants

translate_glu_swiglu_oai creates alpha_const and one as f32 constants
unconditionally, so f16 inputs hit a type mismatch in Swish/Add on GPU
("Swish inputs must have the same type but they are: f16 and f32").
translate_glu_swiglu_clamp already upcasts to f32 and converts back at
the end; apply the same pattern here.

---------

Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
2026-10-15 10:07:45 +03:00
Ethan Guo 6e66cb5d4e metal : add BF16 variant of the tensor API flash attention kernel (#30491)
* metal : add BF16 variant of the tensor API flash attention kernel

* cont : move tensor FA kernels to kernels/fa_tensor.metal
2026-10-15 09:39:47 +03:00