Commit Graph
11878 Commits
Author SHA1 Message Date
Lumpiasty 9857a45f0c llama: evict recurrent/SSM state on device release
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left
resident when a model's device buffers were released for on-demand VRAM
sharing - llama_memory_recurrent::release_device_buffers() was a no-op
default. Implement it (and restore_device_buffers) with the same
capture-host-shadow / free / reallocate pattern as llama_kv_cache, so
llama_memory_hybrid now evicts both its attention KV and its recurrent
state. The state is read-write, so its shadow is recaptured on every
release.

Assisted-by: Claude
2026-10-24 18:33:19 +02:00
Lumpiasty bf2c1a6992 server: also free the compute-graph scheduler on KV eviction
Under LLAMA_SLEEP_EVICT_KV a cold model still held the scheduler's worst-case
compute buffer (hundreds of MiB, e.g. ~700 MiB for gemma-26B) plus the resident
experts. release_device(evict_kv) now also frees the sched (sched.reset() +
sched_need_reserve), and restore_device() rebuilds it via sched_reserve(), so a
fully-evicted model holds essentially no VRAM (gemma cold: 7147 -> 51 MiB).
Trades a heavier re-warm (weights+KV H2D + sched reserve) for the freed VRAM;
weights-only mode is unchanged (fast switch, keeps KV+compute).

Assisted-by: Claude
2026-10-24 18:33:19 +02:00
Lumpiasty 2feb6aeb27 server: restore weights/KV on wake, not at decode, so KV eviction is safe
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.

Assisted-by: Claude
2026-10-24 18:33:19 +02:00
Lumpiasty 8d80f0673f llama: guard buffer iteration against released device buffers
release_device_weights()/release_device_buffers() leave a null buffer in
ctxs_bufs while the device memory is released for on-demand VRAM sharing.
The memory-breakdown / total_size / clear paths iterated these and called
ggml_backend_buffer_get_size()/get_type()/clear() on the null buffer, tripping
GGML_ASSERT(buffer) and aborting on shutdown of a cold model. Skip null buffers
in llama_model::memory_breakdown and llama_kv_cache::{memory_breakdown,
total_size,clear}.

Assisted-by: Claude
2026-10-24 18:33:19 +02:00
Lumpiasty b9df854de8 server: coordinate model load with the VRAM arbiter to avoid load-time OOM
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.

- add vram_arena_open() (idempotent flock/doorbell setup) and
  vram_acquire_for_load(), called from load_model() before
  common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
  request without a re-warm (drop the init-time go_cold cold-start).

Assisted-by: Claude
2026-10-24 18:33:19 +02:00
Lumpiasty 958257cd60 server: keep the sleep-wake path active when the VRAM arbiter is enabled
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.

Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.

Assisted-by: Claude
2026-10-24 18:33:19 +02:00
Lumpiasty 17165fe2a7 server: optionally evict the KV cache too (Phase 2 of VRAM sharing)
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.

- llama_memory_i: add release_device_buffers()/restore_device_buffers()
  (default no-op). Implemented in llama_kv_cache (D2H shadow of the live
  ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
  llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
  buffers when requested; restore_device() rebuilds them. Public API
  llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
  since the KV shadow adds a D2H/H2D copy of the live cache each cycle.

Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.

Assisted-by: Claude
2026-10-24 18:33:18 +02:00
Lumpiasty 481be8b284 server: on-demand VRAM sharing to time-share one GPU between models
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.

- llama-model: release_device_weights()/restore_device_weights() capture a
  compact host shadow (stable iteration order, view-skipping) and free then
  realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
  restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
  (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
  enforces "resident iff holds token"; an inotify doorbell forces the holder
  to release on contention. The warden thread only touches the task queue, so
  releases run on the loop thread and never race a decode.

Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).

Assisted-by: Claude
2026-10-24 18:33:18 +02:00
Lumpiasty 3df719e1e8 ggml-vulkan: tiled transpose fast-path for concat with transposed source
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).

Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).

Assisted-by: opencode
2026-10-24 18:22:50 +02:00
LumpiastyandClaude Opus 5 ff2a96d2d4 vulkan: record why flash-attn shmem staging stays off for GCN
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.

Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.

Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-24 18:21:42 +02:00
Lumpiasty fb6a8a0761 ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.

Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.

Assisted-by: Claude
2026-10-24 18:21:42 +02:00
LumpiastyandClaude Opus 5.5 c72cb424e1 llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.

Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.

Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.

The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.

Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.

Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-10-24 18:21:42 +02:00
Georgi Gerganov 3f03b17724 metal : remove mul_mm_id amax rescaling (#30924)
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the
per expert-token amax scaling done in the compute graph
(ref: https://github.com/ggml-org/llama.cpp/pull/30861), which keeps the
MoE down-projection activations within f16 range before the op runs.

Drop the two-stage reduction kernels, their scratch buffer and the
GGML_PREC_F32 dispatch added in https://github.com/ggml-org/llama.cpp/pull/26223,
as well as the test cases that drive MUL_MAT_ID src1 out of f16 range.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-24 18:40:19 +03:00
Katara 9a0fde9967 rpc : add bounds check in rdma_recv (#30399) 2026-10-24 18:34:46 +03:00
Aleksander Grygier e168e041ec ui: Models Manager as Route + UI improvements (#30988)
* ui : move models manager to its own route, keep Discover as a dialog

Models now lives at #/models instead of a dialog. Discover opens as a
standalone dialog over it, and leaving the page returns to the chat it
was opened from.

Assisted by: Claude (Opus 5.5 Medium)

* ui : polish the models page layout and filters

Give the models page a page header with the Discover action, calmer
column labels, tinted section and family headers, and an empty state.
Load the model list on a direct visit, slide the detail pane over the
table below xl, and draw the filter toggles as one box matching the
context select, with room for their shadow.

Assisted by: Claude (Opus 5.5 Medium)

* ui : round the models table family headers

Assisted by: Claude (Opus 5.5 Medium)

* ui : fix models page navigation and Discover gating

Record the models page's way back after every navigation, so Back,
a typed URL or a reload no longer return to a stale chat. Escape
closes the detail pane, then leaves the page. Gate every Discover
entry point on one shared check, recheck the API key on /models,
skip the duplicate router fetch on a direct load, and share the
route load between pages.

Assisted by: Claude (Opus 5.5 Medium)

* ui : show model details in a drawer

Open the models manager's detail pane in a drawer, from the right on
desktop and from the bottom on a phone, instead of an inline side
panel and a full-screen cover. Pad the pane on both sides, drop its
close button on a phone, and assert that filtered rows are gone from
the page rather than merely hidden.

Assisted by: Claude (Opus 5.5 Medium)

* ui : make whole models table rows clickable

A click anywhere on a model or quant row opens it, unless it lands on
one of the row's own controls (load toggle, actions menu). The name
button stays the keyboard path to the same action.

Assisted by: Claude (Opus 5.5 Medium)
2026-10-24 18:34:11 +03:00
Aleksander Grygier ea1d7d66f3 fix(ui): align hover card and context popup styling with dropdown menu (#30978)
Use the dropdown menu surface (border, radius, background, shadow,
padding) for the hover card and the context gauge popup, and fix the
hover card open/close animation to use bits-ui data-state attributes.

Assisted-by: Claude (Opus 5.5 Medium)
2026-10-24 18:33:27 +03:00
Georgi Gerganov 71255fefc0 ci : pass per-runner results dir to self-hosted jobs on T4 runners (#31100)
The two T4 runners in the ggml-ci group (ggml-6-x86-vulkan-t4,
ggml-8-x86-nvidia-a10) share the same labels and can both pick up the
same jobs, so extend the per-runner results dir from #31046 to them.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-24 17:17:04 +03:00
Satyam Thakurandorigin-labs-ai 73adf89e38 minimax-m3 : forward quant scales to the graph (fixes #31063) (#31097)
The generic scale pass in llama-model.cpp loads the per-expert
NVFP4 scale2 tensors (ffn_{up,gate,down}_exps_s) and the per-tensor
MXFP8 scales for the dense FFN and the shared expert, but
minimax-m3.cpp never forwarded any of them to the graph.

build_moe_ffn() defaults these to nullptr, so build_lora_mm_id()
skipped its w_s branch entirely and the routed-expert weights came
out global_scale times too large, swamping the residual (reported
G ~= 15123 for M3.1 Flash) and producing token soup.

Forward all scale tensors at the three call sites, matching the
pattern used by flint and the other NVFP4-capable MoE models.

Co-authored-by: origin-labs-ai <origin-labs-ai@users.noreply.github.com>
2026-10-24 16:09:02 +02:00
Alexey Kopytko df9beac715 tests : fix test-llama-context for large n_batch (#31083) 2026-10-24 14:14:58 +03:00
Georgi Gerganov 4d32b29aaa ci : exclude test-speculative-rejection-sampling from macos-latest-arm64 (#31088)
Add test-speculative-rejection-sampling to the ctest -E exclusion list of the macos-latest-arm64 CI job.

Assisted-by: dsh:llama.cpp/ggml-org/Flint-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-10-24 14:14:31 +03:00
Georgi Gerganov 3032209758 llama : deny -sm tensor for arches with moe amax rescaling (#31081)
The amax POOL_1D reduction spans the whole matmul row, so a device cannot
compute it from an axis-0 split. The Meta split-state case reported
MIRRORED, which gives each device a partial max of its own slice. Drop
that case and deny -sm tensor for the arches which build the amax graph,
see [TAG_MOE_AMAX_SCALE].

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
2026-10-24 13:48:32 +03:00
Xuan-Son Nguyenandyomaytk 701fd25b3a ggml : allow MUL_MAT_ID -1 index to skip the computation (#26631)
* ggml : allow MUL_MAT_ID -1 index to skip the computation

* nits

* update vulkan code

* ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec

(cherry picked from commit d8a4b9efb6e2b9abf3a866e847756b7e6bbc6b69)

* fix cuda

* openvino

* exclude CUDA from this PR

* add TODO tag

---------

Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
2026-10-24 12:15:36 +02:00
Daniel Bevenius 3642238cc1 tests : add speculative rejection draft sampling test (#30213)
* tests : add speculative rejection draft sampling test

This commit adds a test for speculative rejection sampling test that
excercises the speculative rejection sampling mechanism in
common/speculative.cpp.

The motivation for this is have a test that verifies that the
speculative draft models emitts tokens that  match the target models
distribution. So when rejection sampling occurs it should still
produce the same probability distribution as if we directly sampled
from the target model.

Running the test:
```console
$ ctest --test-dir build -R "^test-speculative-rejection-sampling$" \
    --n_trials 5000 -V
```
This will generate two models, one target and one draft model.

For subsequent runs use the "fixture exclude any" option to avoid
recreating the models:
```console
$ ctest --test-dir build-cpu-debug -R \
"^test-speculative-rejection-sampling$" \
    -FA "^speculative-models$" --n_trails 5000 -V
```

* remove draft model
2026-10-24 12:47:49 +03:00
Georgi Gerganov 6b9ba4f2e8 ci : fix data race between runners sharing the model cache (#31046)
* ci : fix data race between runners sharing the model cache

Two runners per mac mini share HOME, so concurrent jobs corrupt each
other downloads, converted models and venv under the shared MNT cache.

- gg_wget: download in a temp dir, publish with mv, so readers only
  ever see complete files (-N cannot be combined with -O)
- convert, quantize and wikitext unzip now produce under a per-runner
  scratch dir in OUT, published to MNT via hardlink when missing
- gguf-split and quantize tests get a per-runner TMP_DIR
- python venv moved from MNT to OUT
- metal, vulkan and webgpu mac jobs pass a per-runner OUT path

Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)

* ci : clean up scratch models after tests

Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)

* ci : fix q4_0 publish when a cached model exists

cp -n skips an existing file with exit code 1, so the ln fallback
aborted the qwen3 job under set -e. Publish through a unique temp
file with an atomic mv: readers never see a partial file and a
stale cached q4_0 gets replaced.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)

* ci : cache-first gg_wget to avoid re-downloads

HF sends no Last-Modified, so wget -N re-downloads the full files on
every run. Skip the download when a non-empty cache file exists: the
HF CI artifacts are static. Downloads still go through a temp dir and
an atomic mv, so concurrent runners never read partial files.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)
2026-10-24 09:21:24 +03:00
Ruben Ortlam 8e5a58df87 vulkan: ggml src1 precision support (#30656)
* vulkan: update accumulator precision selection

* handle fp32 overflows through scaling

* handle overflows in copy-to-fp16 activations

* add missing cases

* simplify code

* add tests

* add ggml prec get helpers, use in Vulkan backend

* clean up comments

* revert comment edit

* trim tests

* also handle fp32 src0, fix test scaling src0 values too

* don't scale if disabled

* remove fp16 scaling

* add check to mul_mat

* fix inconsistency

* rename to acc_prec
2026-10-24 06:05:05 +02:00
Arseniy Obolenskiy 1ddffe9f70 HIP: add byte permute fallback when __builtin_amdgcn_perm is unavailable (#30915)
* HIP: add byte permute fallback when __builtin_amdgcn_perm is unavailable

amdgcnspirv rejects target-specific builtins unless they are guarded by `__builtin_amdgcn_is_invocable`, so 66 of 145 HIP files failed to build

Wrap the perm calls in a helper that keeps v_perm_b32 when the target has it and falls back to a portable byte select otherwise

* move perm helper to vendors/hip.h
2026-10-24 12:04:05 +08:00
Xuan-Son Nguyen 7894e8b924 mtmd: move build_inp_temporal_merge to base class (#31065) 2026-10-24 03:38:25 +02:00
Yuri KhrustalevandXuan Son Nguyen edc353842c mtmd : lfm2 resize matches torchvision (#30943)
* mtmd : lfm2 bilinear resize matches torchvision

Lfm2VlImageProcessor and d1-omni vision.py resize uint8 images with
torchvision (antialias=True). That is Pillow's algorithm, except that the
weights are int16 with as many fractional bits as the largest weight allows,
where Pillow uses 22 bits. The Pillow port was 1-2 levels off on about 1% of
the values of the d1-omni and LFM2.5-VL crops.

RESIZE_ALGO_BILINEAR_TORCH rounds the weights of the Pillow port to that
precision and scales them back to 22 bits, which gives the output of
torchvision with the same passes, and "bilinear" maps to it for LFM2 and
d1-omni. Other models keep the Pillow algorithms.

* mtmd : lfm2 bicubic resize matches torchvision

Lfm2VlImageProcessor resizes with torchvision for the bicubic checkpoints too
(d1-3B, LFM2-VL), so "bicubic" maps to RESIZE_ALGO_BICUBIC_TORCH for LFM2: the
Pillow bicubic filter (a = -0.5, as torchvision with antialias) with the int16
weights of torchvision. The Pillow port was 1-2 levels off on about 0.25% of
the values of the d1-3B crops.

* use bitfield, add hparams setter

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-24 02:21:06 +02:00
Hans 9ae73e68f5 mtmd : add Soprano TTS support (#28607)
Add Soprano-80M and Soprano-1.1-80M (https://huggingface.co/ekwek/Soprano-1.1-80M)
as an audio generation model: a Qwen3 backbone plus a Vocos decoder that runs
as a gen-only mmproj.

- convert: detect Soprano checkpoints and export the backbone as qwen3 and the
  decoder.pth Vocos weights as the mmproj
- vocab: add the soprano pre-tokenizer (lowercase, whitespace collapse,
  per-digit split, [UNK] fallback)
- mtmd: add the Vocos graph and ISTFT, allow loading a gen-only mmproj for
  Soprano and include the gen-audio context in mtmd_get_memory_usage
- mtmd-helper: add the Soprano pipeline; the decoder runs over the whole
  utterance, so stream mode sends the audio in one chunk after flush()
- tts: document Soprano usage

Assisted-by: Codex
Assisted-by: Claude Opus 5.5
2026-10-24 01:09:50 +02:00
kurquharandMax Krasnyansky 0ae0e19766 hexagon: add SGN op (#30346)
* hexagon : add SGN support

Assisted-by: OpenCode

* hexagon : optimize SGN with sign bit

Assisted-by: OpenCode

* hex-sgn: add f16

* hex-unary: update SGN to use generic unary_colapse and relax colapse checks to apply to more conditions

* hex-unary: improve dims colapsing and run it in the host-side solver

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-10-23 14:58:44 -07:00
Adrien Gallouët 21ec307146 common : allow concurrent downloads across processes (#29312)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-23 23:52:56 +02:00
WalterCool 9145ddb955 hip: fix bf16 WMMA operand vector type for gfx11/gfx12 (#30885)
The __builtin_amdgcn_wmma_f32_16x16x16_bf16_w32 (and _gfx12) builtins
take their A/B operands as vectors of short (_ExtVector<16, short> /
_ExtVector<8, short>), not __bf16. Declaring the fragments as
ext_vector_type(N) __bf16 fails to compile:

  error: cannot initialize a parameter of type 'short
  __attribute__((ext_vector_type(16)))' with an lvalue of type 'const
  bf16x16_t' (vector of 16 '__bf16' values)

short and __bf16 are both 16-bit and the value is only ever passed to
the builtin via reinterpret_cast over raw bytes, so changing the element
type is bit-identical. The operand type has been short since the builtin
was introduced, so this works across LLVM/ROCm versions without a
version gate. Fixes HIP builds for RDNA3 (gfx1150/1151) and RDNA4
(gfx1200/1201).

Signed-off-by: Pablo Cholaky <waltercool@slash.cl>
2026-10-23 23:31:28 +02:00
thelittlefireman 1a24df1639 ggm-cuda: Enforce DP4A MMQ tile height multiple of warp size (#30837)
Added static assertion to enforce DP4A MMQ tile height requirement.
2026-10-23 21:37:20 +02:00
SreeramandJohannes Gäßler 9ff0d7e90f CUDA: fix build after chunked CUDA graphs (#30804) (#31053)
* CUDA: fix build after chunked CUDA graphs (#30804)

* Apply suggestion from @JohannesGaessler

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Fix newlines in common.cuh

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-10-23 20:47:20 +02:00
Christian Loehnert 06230cb877 allows the usage of 4-10 gpus for stepfun and laguna in tensor parallel (#24554) 2026-10-23 19:56:38 +02:00
Alexey Kopytko 5c31142682 qwen4exp : build the QSA selection without the kq_mask (#30965)
* qwen4exp : build the QSA selection without the kq_mask

* clean

* lang

* qwen4exp : reshape the fused QSA selection to the kq_mask streams

ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while
build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream].
With several streams the shape assert failed at graph reserve.

* test the QSA optimization

* reword

* update build_fused_top_k_mask instead of handwiring

* simplify

* keep il at the end for consistency
2026-10-23 19:49:08 +02:00
Adrien Gallouët da2ee1e7a6 vendor : update LibreSSL to 4.4.0 [no ci] (#31049)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-23 19:12:35 +02:00
Aman Gupta 280141e6d4 CUDA: chunk graphs to amortize graph launch (#30804) 2026-10-24 00:26:16 +08:00
Max Krasnyansky 36e36e63a5 hexagon: add CEIL op (#31006)
* hex-ceil: add CEIL(f32) op

* hex-ceil: add f16 version

* hex-ops: update supported ops

* hex-ops: update docs with supported ops
2026-10-23 07:46:53 -07:00
Aleksander Grygier e1e3b2c409 ui : bump packages to clear npm audit (#31038)
Run npm audit fix, then bump the remaining vulnerable packages within
their current majors: storybook 10.6.2, vitest 4.1.12, vite 7.3.7,
@modelcontextprotocol/sdk 1.32.1, dompurify 3.4.17, mermaid 11.17.2,
@vite-pwa/assets-generator 1.0.4 and @tailwindcss/typography 0.5.20.

Override katex to 0.18.11 and postcss-selector-parser to 7.1.6 since
their dependents still pin vulnerable ranges, bump the sharp override
to 0.35.5, and pin @storybook/svelte so npm does not resolve the 11.0
alpha through the addon-svelte-csf peer range.

Storybook stays on 10.6.2 as 10.6.3 is newer than min-release-age.

Assisted by: Claude (Opus 5.5 Medium)
2026-10-23 16:33:13 +02:00
Kartik Gulia 4231b23dc2 ggml : fix time overflow in ggml_time_us (#30969)
* fix time overflow

* use double
2026-10-23 15:56:59 +03:00
Sigbjørn Skjæret f00c9083b2 ci : add behavior check workflow (#31036)
* remove python from tests

* add behavior and backend ctest labels

* add behavior check workflow

[no ci]
2026-10-23 14:53:44 +02:00
Raul Torres f5027fb213 CANN: missing const qualifiers added for consistency (#30933) 2026-10-23 15:34:20 +08:00
Titaniumtown f64a49819c sycl : pass device into mul_mat_uses_onednn (#31014) 2026-10-23 08:37:53 +02:00
Pranesh GonegandlaandPranesh Gonegandla 1ab09d632e Extend probabilistic sampling for DFlash2 drafts (#30663)
* spec : add probabilistic sampling for DFlash2 drafts

* spec : apply review suggestions

---------

Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
2026-10-23 11:15:36 +05:30
f9599d0cc8 vocab : fix Nemotron-3.5-Embed tokenization (#30974)
* vocab : fix Nemotron-3.5-Embed tokenization

* Update src/llama-vocab.h

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* rename

* Update src/llama-vocab.cpp

Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
2026-10-23 07:31:30 +02:00
Max Krasnyansky 643b0cdacb hexagon: Op Fusion refactoring, optimizations and new fusions like BINARY_NX (back-to-back ADDs, etc) (#30843)
* hex-fusion: support for GELU+MUL, REPEAT+GET_ROWS and BINARY_NX (back-to-back ADD/MUL/...)

* hex-fusion: factor out fusion logic into separate .h and .cpp and update all related code

* hex-fusion: allow for RMS_NORM+MUL fusions for non-contig tensors

* hex-fusion: streamline fusion eligibility checks and cover missing corner cases

* hex-fusion: streamline fusion flow

* hex-opbatch: make n_tensors limit more scalable

* hex-bufs: adjust the trace buff sizing

* hex-glu: cleanup glu-ops interfaces and add kernel_params

* hex-kparams: clean up kparams handling in unary, ssm-conv, and other ops

* hex-kparams: update the rest of the ops PAD,DIAG,... to use kparams

* hex-unary: add support for collapsing dims

* hex-ops: cleanup redundant runtime checks

* hex-ops: consistent use of kernel_type in all kparams

* hex-fusion: remove redundant check in gdn+cpy

* hex-fusion: further streamlining of fusion rules and code

* hex-fusion: cleanup naming

* hex-opfmt: cleanup op formatting

* hex-fusion: bump fusion lookahead to 64

* hex-fusion: fuse SSM_CONV+SILU

* hex-fusion: fix hvx register spills in fused ssm_conv+silu

* hex-fusion: add more GLU fusions

* hex-fusion: remove duplicate checks and use ggml-common helpers

* hex-kparams: refactor kparams into htp-kparams.h/cpp

* hex-fusion: fix binary-nx fusion regression

* hex-kparams: another pass at cleaning up kparams (removing unused, etc)

* hex-kparams: streamling nameing and move _check_ wrappers into .h

* hex-kparams: templetize htp_kparams_check_<op> to reduce code bloat

* hex-ops: de-duplicate op dispatch switches

* hex-kparams: trim down exposed kparams calls

* hex-kparams: simplify type casting of kparams structs

* hex-fence: de-dup fence tensor construction

* hex-opbatch: simplify tensor shape init/checks

* hex-mm: cleanup leftovers of the original QK defines

* hex-fa: fix stack overflow on older npu firmware

* hex-ctx: move dma_caches into ops_ctx
2026-10-22 21:05:44 -07:00
Todd Malsbary 0a9ef00678 sycl: Replace host waits in fattn-mkl with explicit dependencies (#30684)
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
2026-10-22 22:43:12 -04:00
Todd Malsbary 7377bada8c sycl: Add graph record and replay (#28725)
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
2026-10-22 22:39:52 -04:00
pzhu 184f653602 server: Substitute rerank placeholder from the original template only. (#30864) 2026-10-23 00:38:03 +02:00