Commit Graph
12110 Commits
Author SHA1 Message Date
Georgi Gerganov cd959bae99 llama.cpp : bump version to 0.9.0 (#31646)
* llama.cpp : bump version to 0.9.0

* authors : update
2026-11-04 14:06:26 +02:00
Georgi Gerganov 13ec8f614c sync : ggml 2026-11-04 12:00:37 +02:00
Georgi Gerganov cca14f64fe ggml : bump version to 0.29.0 (ggml/1680)
* ggml : bump version to 0.29.0

* authors : update
2026-11-04 12:00:37 +02:00
Aman Gupta e853ec9a8c llama : follow ctx_other devices and flash attention in Gemma4 assistant (#31635)
* llama : follow ctx_other devices and flash attention in Gemma4 assistant

* llama : include <algorithm> for std::any_of
2026-11-04 11:23:25 +02:00
Kirill A. Korinsky feaf800060 ggml : Gate RVV code on the target extension (#31607)
The RISC-V vector intrinsic specification requires __riscv_v_intrinsic
to be defined when the compiler supports the API, even when the vector
extension is not selected. Clang therefore defines it for rv64gc,
causing ggml to enter RVV branches where vector types are rejected
because the required ISA extensions are absent.

Use __riscv_v for optional RVV implementations; this macro represents
the selected target ISA. Retain __riscv_v_intrinsic checks in SpaceMIT
sources, where they validate mandatory compiler API support rather than
select optional vector code.

Closes: https://github.com/ggml-org/ggml/issues/1535
2026-11-04 11:11:52 +02:00
Georgi Gerganov 4c4ca5b188 common : do not allow _ in sidecar (#31639) 2026-11-04 10:32:45 +02:00
EugeoSynthesisThirtyTwoandYour Name 162e2bb2e4 ggml : fix bf16 (#31609)
Co-authored-by: Your Name <you@example.com>
2026-11-04 10:10:22 +02:00
lhez 6900fb5ac8 opencl: improve q2_0 GEMV for Adreno (#31577)
* opencl: improve q2_0 decoding and small f32 matmuls

Assisted-by: Codex

* opencl: optimize small f32 matmuls on Adreno 8xx

Assisted-by: Codex
2026-11-04 10:07:14 +02:00
Pascal bbb02c93e9 ggml: bound views by the bytes their strides reach (#31584)
* ggml: bound views by the bytes their strides reach

A view was checked against its source with the size of a contiguous
tensor of the same shape, computed before its strides were set. Views
whose rows overlap failed that check although they stay inside their
source, and views with wider strides passed it while reaching past
the end. The view functions now pass their strides to the tensor
constructor, which checks the bytes they actually reach, like
ggml_nbytes. The conv windows of delta_net are created with all their
rows at once instead of being widened after the check.

* ggml: assert that a strided view starts from contiguous elements

The view functions set nb[0] to the type size, so the source must have
contiguous elements along dim 0 for the view to read them.
2026-11-04 09:47:11 +02:00
Georgi Gerganov 960b1d51b7 ggml : write host-buffer tensor data directly in gguf writer (#31144)
The gguf writer staged each tensor into a temporary vector before
writing it. For host-buffer tensors the data pointer is directly
writable, so skip the staging copy and write it straight to the file.
This avoids the per-tensor allocation, zero-init and memcpy.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
2026-11-04 09:29:26 +02:00
Georgi GerganovandPascal 4b29d02629 common : fix -hf tag matching inside longer quant names (#31620)
* common : fix -hf tag matching inside longer quant names

":F16" picked a BF16 file when the repo had both, since "BF16"
contains "F16". The same applied to ":Q4_XS" vs "IQ4_XS". The tag
now has to start at a token boundary in the path.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* common : accept underscore as -hf tag boundary

Some repos separate the quant with an underscore, e.g.
stepfun-ai/Step-3.5-Flash-GGUF-Q4_K_S ships
step3p5_flash_Q4_K_S-00001-of-00012.gguf, which :Q4_K_S no longer
matched. Add _ to the token boundary and pin the tag and default
lookup cases.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-04 09:22:54 +02:00
Todor BoinovskiandMax Krasnyansky bee23322f1 hexagon: add LIGHTNING_INDEXER support (#31289)
* hexagon: initial support for LIGHTNING_INDEXER op

* hex-lght-idxr: hmx for prefill

* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute

* hex-lght-idxr: pipelined hmx path for decode and prefill

* hex-lid: add support for K (f32)

* hex-lid: add support for K (f32)

* hex-lid: add support for q4_0

* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills

* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel

* hex-lit: update dma pipeline to use canonical dbl buffering in all paths

* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths

* hex-lit: add support for dims collapse

* hex-lit: make kparams constistent with other ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 21:17:30 -08:00
Xuan-Son Nguyen 149f5641b9 mtmd: accurate support for qwen3-vl video prompt format (#31619) 2026-11-03 23:57:26 +01:00
b97627967c model, mtmd: add chatterbox TTS support (#30993)
* add chatterbox TTS support

Co-authored-by: Pascal <admin@serveurperso.com>

* add built-in voice support

* rm redundant code

* clean up

* clean up 2

* Apply batched suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* improve

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-03 23:47:11 +01:00
Todor Boinovski 7071ba378a hexagon: add TOP_K_MASK op (#31552)
* hexagon: add TOP_K_MASK op

* Simplify Hexagon TOP_K_MASK kernel and do not duplicate host configuration
2026-11-03 12:04:03 -08:00
6cb3e672e8 gemma4: graph optimizations (massive gains on pp for E2B and E4B) (#31356)
* Optimizing Gemma4 graph for prompt processing

TODO:
	- Currently complete microbatch needs to contain tokens which does not require the logits; need to optimize this further
	- There are a few more operations that are performed after the kv cache is calculated, need to safely optimize them.

* Not filtering the rows when they are unmasked

* Cropping out the attention mask too.

* gemma-4: improved pruning, mask cropping and PLE scaling

Prune before the first KV-shared layer with the single-stream, causal, non-embedding safety gates.
Use cropped masks only during shared-layer attention.
Preserve requested layer-input embeddings in the KV prefix.
Remove the PLE contiguous materialization and fold equivalent PLE scaling into RMSNorm parameters/weights.

* spec-mtp: Enable masked h_nextn extraction only when the draft context shares the target KV cache

This lets Gemma 4 prune unused prompt rows before its shared layers so that we keep the full perf P.P gains with MTP.

Avoid extracting target hidden states for output-free prompt chunks in that shared-MTP path.
When a batch has a selected final token, extract only that row unless the whole verification batch is selected.

Keep ordinary MTP and classic speculative decoding behavior unchanged: they still use unmasked h_nextn because the draft
context must catch up on all target rows.

speculative-simple needs an update to select the final prompt row only for shared-KV MTP,
so the optimization works there without changing other speculative modes.
And it looks like the server already selects its final prompt row for first-token sampling.

* spec-mtp: make sure to not drop first draft round

* gemma4: enable pruning based on static cparams.kv_unified/n_seq_max=1

* gemma4-mtp: simplify prunning logic and avoid graph branches/re-allocs

* gemma4: update pruning gates to use only n_seq_max==1, kv_unified can now be used for multi-seq

* spec-simple: revert spec-simple changes for now

* gemma4: remove causal_attn from pruning gate (last source of reallocs)

* cont : adapt to llm_attn_mask changes

* cont : add TODO

---------

Co-authored-by: Meet Amin <meetamin378@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-03 20:30:37 +02:00
Zack LiandMax Krasnyansky 6b9c0cfcba hexagon: faster TOP_K for the backend sampler (#31279)
* hexagon: faster TOP_K for the backend sampler (skip tiles, then block scan)

Skip the bitonic sort for tiles whose max cannot reach the running k-th
best value, then replace the tile sort with top_k_scan(): reject 256-float
blocks with one HVX max-reduce and insert only surviving elements into a
sorted k-entry list. Used for k in 2..64 and rows wider than 1024, in both
the single-row and multi-row paths.

Ties keep the first-seen index. NaN is treated as -INF.

* hex-top-k: eliminate scalar reads from vtcm

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 10:06:28 -08:00
Aleksander Grygier d1c4e5bec1 ui: run local browser tests in installed Chrome with its sandbox on (#31596)
Playwright's bundled Chromium lags behind Chrome security releases
(1.56.1 ships 141, and even 1.64.1 ships 156.0.8078.4, before the
156.0.8078.73 fixes), and Playwright launches it with --no-sandbox by
default. Outside CI, use the installed, auto-updated Chrome with the
sandbox enabled. CI keeps the bundled Chromium, so the self-hosted
runner's preinstalled 1.56.1 browsers still work.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-03 19:02:54 +01:00
Todor BoinovskiandMax Krasnyansky 20a51f24a8 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
* hexagon: add support for DSV4_HC_PRE

* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization

* hex-dsv4-hc-pre: double-buffered DMA for prefill

* hex-dsv4-hc-pre: refactor DMA block setup

* hex-dsv4-hc-pre: precompute kernel params

* hex-dsv4-hc-pre: support for n_hc != 4

* hexagon: add support for DSV4_HC_POST

* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block

* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks

* hex-run: add support for test-llama-archs tool

* hex-dsv4: refactoring and optimizations

Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.

Tested with test-llama-archs -a qwen4ext and test-backend-ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 08:26:18 -08:00
randomizedandzouyongxing 0f35d6d7b0 spec : save and restore MTP hidden state (#30141)
* speculative : save and restore MTP hidden state

Assisted-by: Codex

* tests : remove MTP state test

Assisted-by: Codex

* speculative : simplify MTP checkpoint state

Save only the boundary hidden row; verification rows are rebuilt on the supported restore paths.

Assisted-by: GPT-6

---------

Co-authored-by: zouyongxing <zouyongxing@bytedance.com>
2026-11-03 16:53:57 +01:00
Aaron Teo a464960ed3 ggml-zdnn: fix fattn test failure (#31583) 2026-11-03 23:51:07 +08:00
Giulianno 9309bf1bcf model : Add support for Metis-1.6 models (#31134)
Assisted-by: Claude
2026-11-03 16:24:45 +01:00
Tarek Dakhran d1044bc827 model : enable NVFP4 LFM3 ggufs (#31608) 2026-11-03 15:36:44 +01:00
Prabhsimran Singh 28b812d6cb mtmd : add Nemotron 3.5 Transcribe support (#31471)
* mtmd : add Nemotron 3.5 Transcribe support

Assisted-by: Claude

* convert : fix vocab of draft models for Nemotron-H targets

Assisted-by: Claude

* convert : reject non-BF16 Transcribe checkpoints

Assisted-by: Claude

* mtmd : make speaker FIFO predictions local

Assisted-by: Claude

* mtmd : normalize the mel in encode_windows

Assisted-by: Claude

* convert : map the generated speaker kernel

Assisted-by: Claude

* mtmd : validate Nemotron Transcribe mmproj shapes

Assisted-by: Claude
2026-11-03 14:27:05 +01:00
Pascal 749db7f034 qwen4exp: write the conv state tail without an intermediate cont (#31443)
build_conv_state_at makes a contiguous copy of the conv input tail
before writing it into the recurrent state, so every GDN layer issues
two device copies per generated token where one is enough. On
Qwen3.8-Flash-Next (36 GDN layers) that is 72 copies per token, now 36.
Speculative decoding adds one more per rollback slot. ggml_cpy reads
the strided view directly, as the shared build_conv_state already does,
so the extra copy is dropped.
2026-11-03 13:47:23 +02:00
Aleksander Grygier 0943ca5472 ui: add Playwright harness for recording demo videos (#31318)
* ui: add Playwright harness for recording demo videos

Scripted walkthroughs under tools/ui/demo are recorded with Playwright and
exported as H.264 mp4 for pull requests (demo/pr) and docs (demo/docs).
Fixture MCP servers and seeded settings keep recordings deterministic, a
drawn cursor and captions make the flow readable, and DEMO_UI_DIR points
the recording at another checkout such as a PR worktree.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record demo scripts from PR descriptions and a configurable storage

Demo scripts no longer live in the repo. `npm run demo:pr -- <number>`
records the `ts demo` block from a PR description against the PR head,
uploads the mp4, a GIF preview and the script to DEMO_STORAGE, and links
the preview in a `## Demo` section of the PR description.
`npm run demo:docs` records the docs scripts kept in DEMO_STORAGE.
DEMO_STORAGE (hf:// buckets or file://) and the public URL are read
from tools/ui/.env.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: import generated conversations into demo recordings

Demos can now start with existing chats: `test.use({ traces })` imports
llama.app JSONL traces through Settings > Import/Export before the video
starts. Built-in traces are generated deterministically, other names are
downloaded from DEMO_TRACES_DATASET, and `demo:traces` pushes the built-in
ones there. Adds `demo.scroll` for wheel scrolling.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep demo captions clear of the chat form and isolate concurrent runs

Captions move to the top of the frame, since at the bottom they covered
the chat form a demo is often about. Each recording run gets its own
Playwright output folder, so a run starting elsewhere no longer clears
another run's video before it is saved.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record demos at 2x and 60 fps with human-like timing

Recording goes through Chromium's screencast instead of Playwright's
video, which encoded VP8 at about 1 Mbit/s at 1x. Frames are captured at
the device pixel ratio (DEMO_SCALE, default 2) and encoded into a 60 fps
H.264 mp4, starting once the page has settled.

The cursor now glides faster along a slight arc with Fitts-like
durations, settles briefly before each click, and typing and pauses vary
a little. The randomness is seeded per demo, so re-recording gives the
same motion.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: document the demo harness prerequisites

The demo README now lists the tools the harness needs and what each is
for, and walks through creating the Hugging Face bucket and the optional
traces dataset, with the matching .env settings.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: version linked demo URLs so re-recorded previews refresh

GitHub's image proxy caches a preview by URL, so a demo re-recorded at
the same PR head kept showing the old GIF. The Demo section now links
the files with a content hash in the query string.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: label PRs that feature a video demo

Adds a `video demo` label and a workflow that applies it when a PR
description has a demo written by the UI demo pipeline (the managed
demo:start/demo:end markers, or a preview linking to a demo video),
and removes it when that section is gone.

Assisted-by: Claude (Opus 5.5 Medium)

* ci: label video demos with gh instead of github-script

The label workflow now runs the gh CLI in a shell step, like
pr-draft-label.yml, so it needs no action outside the repository's
allowed list. The video link check only counts videos uploaded for the
PR itself, so a description linking other PRs' demos isn't labeled.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: zoom demos in on details with a two-finger pinch

demo.zoom pinches in on a control or a point, holds, and pinches back out
to 1x, so a detail too small to read in the frame can be shown up close.
It drives Chromium's own pinch gesture, so the page's fixed bars stay put,
and works at desktop and phone sizes alike.

The pinch is evened out where Chromium misses: one anchored near the top
does not zoom at all, so it starts lower, and one near the bottom
overshoots its ratio, so a small pinch from the middle corrects it. The
way out reads the real scale until the page is back at 1x, which the next
gesture needs.

The overlay also draws a fingertip per touch, so a pinch on a touch
device shows as two fingers.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: wait out a cold dev server before a demo starts

A fresh checkout's dev server optimizes its dependencies on the first
load and reloads the page, which can take well past an action's timeout,
so a demo failed before it started. The demo now waits up to 90 seconds
for the app; the recording only starts once it is ready.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: record a video per test in a PR demo, shown in a table

A large PR can split its demo into several shorter videos: each test()
in the ts demo block is recorded as its own video (<sha>-1.mp4, ...),
and the Demo section lays their previews out in a table, each column
headed by its test's title.

GIF previews now keep their longer side at DEMO_PREVIEW_WIDTH and shrink
until they are under 5 MB, since GitHub's image proxy refuses anything
larger and shows a broken image instead.

Importing traces works on a phone viewport too, where Settings sits in
the sidebar under the page until the menu key slides it aside.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: fix demo trace imports on desktop and lay landscape demos out two to a row

Importing traces failed on a desktop whose collapsed sidebar sits under
the page: it now expands the sidebar from the page's menu key on a phone
or from the sidebar's logo on a desktop, whichever makes Settings
reachable. Afterwards it forgets the expanded sidebar, so a demo starts
with it collapsed, as on a first visit.

A PR demo with several videos shows up to four portrait (phone) videos
to a row, or two landscape ones, so each preview stays readable.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: only record other people's PR demos at a reviewed head commit

demo:pr runs the PR's code and demo script on this machine, with the
gh and hf logins, so a PR by someone else now needs --trust <sha>
naming the head commit that was read, and is refused again after new
pushes. Recordings also launch Chromium with its sandbox on, which
Playwright turns off by default.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-03 12:30:49 +01:00
Tarek Dakhran f9448cddc4 model : record layer inputs in LFM3 for spec dec (#31598) 2026-11-03 13:29:10 +02:00
Jasmine-tim 0978a6e902 ggml: avoid signed overflow in ggml_roll bounds check (#30375)
ggml_roll validated each shift with abs(shift), but abs(INT_MIN) is signed
integer overflow UB evaluated before the bounds check runs. Compare with a
signed-safe test instead.
2026-11-03 13:26:29 +02:00
Daniel Bevenius 0ef27309b7 conversion : use original model tensor names in _flush_nvfp4_experts (#31427)
This commit updates the _flush_nvfp4_experts to take a new parameter
which is a dictionary mapping the key to the original model tensor name.

The motivation for this is that I noticed that the tensor names being
looked up were not the original model tensor names, for example:
```console
  model.layers.1.mlp.experts.up_proj.weight
```
But _nvfp4_layer_algo uses the original per-expert names:
```console
(Pdb) p self._nvfp4_layer_algo
{
'backbone.layers.1.mixer.experts.0.up_proj': 'W4A16_NVFP4',
...
```
This means that the lookup will never match for experts and fall back to
_nvfp4_global_algo. If the global algo is W4A16_NVFP4 then this would
go unnoticed, but for a model like
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 which is
MIXED_PRECISION, the expert tensors will not get prec_a4=false so they
would use the native W4A4 path on Blackwell even though the original
model tensors declare them as W4A16_NVFP4.
2026-11-03 11:40:51 +01:00
mmalpe-nvandGaurav Garg 8fc0376721 cuda: potential fix for Q8_1 correction sum (#31247)
* cuda: potential fix for Q8_1 correction sum

* Update ggml/src/ggml-cuda/quantize.cu

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>

* addressed review comment to change the quantize_mmq_q8_1 flow as well

---------

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
2026-11-03 15:54:42 +05:30
Bhargav Krish 740251a920 cuda : gather batched tensor reads into mapped host memory (PR to master) (#31391)
* cuda: gather batched tensor reads into mapped host memory

Assisted-by: Codex

* add comments

Assisted-by: Codex

* simplify the implementation based on checks that exist earlier in the flow

* simplify batched tensor gather indexing

* add conditional dispatch

* address comments - make host registration raii, better test mindful of ci time, assert instead of return false

* cuda : gather small reads and use pinned DMA for large reads

* cuda : use std::vector for host gather descriptors

* cuda : add 8-byte gather copies, generalize tensor test and log registration failures

* common buffer for descriptors

* simplify stream sync + increase to 128k descriptor pinned buf
2026-11-03 15:31:50 +05:30
zyang-dev 71efc559a0 Add nccl support for multi-node tensor parallelism (#28967)
* Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available.

* Drop ggml_backend_comm_allreduce_rank_t and reuse ggml_backend_comm_allreduce_tensor for per-rank communicators

* reuse ggml_backend_cuda_comm_allreduce_nccl for per-rank communicators. Remove the helper function backend_comm_proc_address

* Change ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t to more generic multi-process backend communicator interfaces

* set maxCTAs in ncclConfig_t to 8 for dgx spark in ggml_backend_cuda_comm_init_rank
2026-11-03 14:56:19 +05:30
Nidhi 19e8a86fe8 tools : reject mismatched shards in GGUF split merge (#31579) 2026-11-03 11:11:05 +02:00
Aman GuptaandPratik Narola 259d91709d meta: support GLM-5.3 next sm tensor (#30499)
* meta: support GLM-5.3 next sm tensor

* glm5-next : cont q_absorbed for -sm tensor with non-unified KV

* meta : keep GGML_OP_ARANGE in ggml.h order

* glm5-next : split MLA heads in pairs for -sm tensor

The MLA cache has a single KV head, so the heads on a device are its GQA
ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3
devices the 16 heads of the test model were split 5/5/6 and it aborted.

* meta : keep the split for views inside one slab of the split axis

With a non-unified KV cache, build_attn_mha splits q into streams with a
view. For a permuted q, e.g. the absorbed query of glm5-next, that view
aborted with "view of permuted tensor not implemented". The split is now
kept if every other dim of the source and the view stays inside one slab
of the split axis. The per-device strides are only rescaled above it.

From tetherto/qvac-fabric-llm.cpp#353.

Co-authored-by: Pratik Narola <pratik.narola@tether.io>

* glm5-next : drop the q_absorbed cont

The meta backend can now split the per-stream view of the permuted q, so
the cont is not needed for -sm tensor with a non-unified KV cache. With
the cont, the q of flash attention was not contiguous, and when FA ran on
the CPU (WebGPU) the meta get_tensor aborted.

---------

Co-authored-by: Pratik Narola <pratik.narola@tether.io>
2026-11-03 10:51:10 +02:00
Georgi Gerganov 36ee3caf7a metal : fix cpy kernel for dst non-contiguous in dim 0 (#31581)
kernel_cpy_t_t indexed the dst innermost dim with an implicit element-size
stride instead of args.nb0, so cpy into a permuted view (nb0 != type size)
wrote the wrong addresses. caught by the new test-backend-ops cpy case with
permute_dst = {1, 0, 2, 3} from #31446. addressing is unchanged when dst is
contiguous in dim 0.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-03 10:41:24 +02:00
FamousMonsterrandFamousMonsterr fac3a894fc vulkan: skip maxStorageBufferRange pre-check for non-compute tensors (#31114)
* vulkan: skip maxStorageBufferRange pre-check for non-compute tensors

Pre-allocated leaf tensors and views (GGML_OP_NONE / GGML_OP_VIEW) are never
executed by the backend, but were still rejected by the descriptor-size
pre-check in ggml_backend_vk_device_supports_op when larger than
maxStorageBufferRange on devices without shader_64b_indexing. The scheduler
then aborts with 'pre-allocated tensor (...) in a buffer (Vulkan0) that cannot
run the operation (NONE)' - e.g. the MoE expert cache banks (llama-moe-cache)
of a 256-expert model with --moe-cache-mib on GCN GPUs, where a whole bank
(2.4 GB) exceeds the limit while the consuming ops only access small
per-step views.

Skip the size pre-check for non-compute ops (NONE and VIEW); the buffer type
allocation limit still bounds the actual allocation, and compute ops keep
validating their sources.

Tested on AMD Radeon Pro VII (gfx906, no shader_64b_indexing):
- before: GGML_ABORT 'pre-allocated tensor (moe_cache.0.blk.0.ffn_down_exps.weight) ... cannot run the operation (NONE)' with llama-cli -ngl 99 --cpu-moe --moe-cache-mib 6144 (Cyber-Tiel-Coder-35B-A3B, qwen35moe, 256 experts per layer)
- after: loads and generates at 8.1 t/s (vs 0.6 t/s without the cache)
- regression: -ncmoe 16 path unchanged (19.2 t/s), qwen3moe/qwen35moe without cache unchanged

Fixes #30697

* vulkan: use ggml_op_is_empty for the non-compute size-check skip

---------

Co-authored-by: FamousMonsterr <FamousMonsterr@users.noreply.github.com>
2026-11-03 08:53:04 +01:00
Aman GuptaandPascal b74298c109 delta_net_base: refactor conv state (#30878)
* delta_net: access conv state via views

* fix sm tensor

* delta_net: fill the conv slot index on the host

The rollback slot index of the conv windows is a constant per graph
shape. It is now an I32 input of the recurrent memory filled on the
host, instead of an arange and an argsort computed on the device, and
the meta backend no longer needs a mirrored split for ARANGE.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-03 08:28:35 +01:00
Dongsheng LuoandDongsheng Luo 37b0f9c36a vulkan: fix out-of-bounds NVFP4 UE4M3 scale lookup (#31079)
* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup

On the non-OCP path, ue4m3_fp32_lut has 128 entries and is indexed with the raw scale byte. ggml_validate_row_data accepts every byte and ggml_ue4m3_to_fp32 ignores bit 7, so a byte >= 0x80 reads past the shared array. Vulkan robustness does not cover shared memory.

Size the table, both init loops, and the two host shared-memory budgets to 256 entries. ue4m3_to_fp32_build() already masks the exponent and treats only 0 and 0x7F as special, so entries 128-255 match the CPU. A 256-entry table keeps 0xFF distinct from 0x7F: the CPU decodes 0xFF as a non-zero value and 0x7F as zero.

Each NVFP4 pipeline uses 512 more bytes of shared memory. ggml_fp32_to_ue4m3 never sets bit 7, so models quantized by llama.cpp are unaffected.

Assisted-by: Claude
Assisted-by: Codex

* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup

Assisted-by: Codex

---------

Co-authored-by: Dongsheng Luo <7137760+flyingdoog@users.noreply.github.com>
2026-11-03 07:58:56 +01:00
Georgi Gerganov ca8daad199 llama : drop the redundant cnv from llm_attn_mask (#31558)
The dspark block mask is now derived at input-fill time: llm_attn_mask::block_uniform
fills every token of a sequence with the mask row of its first token, so the
graph-level reshape/repeat in dflash is gone and cnv always aliases raw.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-03 08:11:06 +02:00
Aman Gupta 0f663d2471 dsv4.1: fix llama-context tests with partial offload (#31292) 2026-11-03 08:08:42 +02:00
hesiru 00f113eceb cuda : prevent premature eviction of graphs about to be used (#30662)
* cuda : prevent premature eviction of graphs about to be used

* add last_graph_use_time for entire cuda graph map and testcase for cuda graph

Assisted-by: Codex

* delete the test
2026-11-03 08:06:03 +02:00
shaofeiqiandLi He e6192de0a6 opencl: add initial iq4_xs support (#31339)
* opencl: add iq4_xs AOS mv

Assisted-by: Codex

* opencl: add iq4_xs SOA mv and tiled mm

Assisted-by: Codex

* opencl: add iq4_xs noshuffle gemm/gemv for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: add iq4_xs dp4a for adreno

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Assisted-by: Codex

* opencl: optimize iq4_xs adreno decode and narrow batches

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

Assisted-by: Codex

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-02 21:58:19 -08:00
frogscooper 4402e65401 chat : allow constraint type after gpt-oss final channel (#31556) 2026-11-03 07:55:11 +02:00
Aman Gupta 74101b3cda ci : update the HIP VGPR spill ignore list for the kv_rows FA kernel argument (#31550)
The kv_rows argument added to the FA kernels in #29510 changes their mangled names, so the known spills were no longer ignored.
2026-11-03 07:54:47 +02:00
Leon 713319e284 webgpu: add ARANGE support (#31455) 2026-11-03 07:54:00 +02:00
Aman Gupta ee17f293b6 kv-cache : never return 0 kv_rows for sequences without cells (#31570)
A cache that shares the cells of another one (f.ex. the gemma4 assistant) can build a graph before its sequences have any cells, f.ex. at the server warmup. get_n_kv_rows() then returned 0, which means that kv_rows are not used, so the graph fell back to the dense KQ mask and the scheduler re-planned for it. Later graphs with larger masks then aborted with GGML_SCHED_DEBUG_REALLOC=1.
2026-11-03 07:51:53 +02:00
51fe42c5d1 hexagon: tiled hvx permute (#31446)
* hexagon: add tiled HVX transpose path for permuted CONT/CPY

Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.

Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.

Host:
  - drop extent-1 dims and merge dims that stay adjacent in src into a
    canonical view, stored in the kernel params. There are no minimum
    size limits.
  - the copy is treated as a 2D transpose between dst dim 0 and the
    contiguous src dim, batched over the remaining dims.
  - tiles are sized to 64 KB per buffer and halved until they fit in
    session VTCM. blkk is capped so the halfword vgather offsets fit in
    16 bits.

Kernel, per tile:
  - 2D DMA of blk0 strided src rows (blkk contiguous elements each)
    into VTCM
  - vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
  - 2D DMA of blkk contiguous rows out to dst

Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.

Small dims:
  - packed mode: when dim k follows dim 0 and dim 0 divides the HVX
    lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
    whole dst rows, and each tile goes out as one contiguous DMA.
  - when the contiguous src dim fits in one vector, it is staged exactly
    and the tile is made long along dim 0.

All other copies select the same kernels as before.

Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.

* hexagon: cpy transpose cleanups

- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)

Assisted-by: Claude Opus 5.5

* hexagon: use cpy transpose path for ne[0] == 1 permutes

- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests

Assisted-by: Claude Opus 5.5

* hex-cpy: further improve cpy

Bring in improvements from aizip branch/PR.

Cleanup kparams and replace divs with fastdiv.

Co-authored-by: Yiwei Shao <yiwei@aizip.ai>

* hex-cpy: avoid vtcm bank conflicts

* hex-cpy: trim down the comments

* cpy-test: trim down new tests that cover permute scenarios

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
2026-11-02 21:13:38 -08:00
Aman Gupta 2cbca389f5 dflash: copy hidden features via device copies (#31506) 2026-11-03 09:02:32 +08:00
Aaron Teo 29c720eaf7 docs: update s390x build docs (#31546)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-11-02 20:52:39 +02:00
d02b4c0d66 mtmd: Pocket tts - support for voices without voice cloning (#31429)
* feat: support PocketTTS voices with no voice cloning

The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.

Assisted-by: Claude Fable 5.1

* convert : pocket-tts voices never fail the conversion, log the drift of the state

The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.

Assisted-by: Claude Fable 5.1

* mtmd : list the predefined voices, expose them in GET /props

mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.

Assisted-by: Claude Fable 5.1

* server : an unknown voice is a bad request

A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.

Assisted-by: Claude Fable 5.1

* refactor api

* fix conversion

* small renaming

* fix lint

* tts : a model with predefined voices speaks with the first one by default

Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.

Assisted-by: Claude Fable 5.1

* mtmd : one comment per voice accessor, converter states the rule only

The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.

Assisted-by: Claude Fable 5.1

* clean up redundant code and comments

* mtmd: match the LFM3-Audio voice name case-insensitively

The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-02 18:39:51 +01:00