What and why (#32755 and the 6.5 GiB of shared expert copies), the knob,
F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the
pod validation plan (reference with the loop experts on the GPU, views on,
views off).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
inkling2_loop runs the core layers (6..17 on Inkling-2-Mini) three times, and
post_arch_tensors makes every cache slot share the tensors of its physical
layer. With the experts in host memory (--n-cpu-moe, -ot exps=CPU) the
scheduler makes one device copy per (host tensor, backend) per graph, at the
first split that reads it, and keeps it until the last reader. So:
- the expert copies of all 12 loop layers are alive at once (about 6.5 GiB on
UD-Q4_K_XL), which is most of the ~7 GiB Vulkan0 compute buffer;
- sched_copy_experts runs only for the first reader. When it reads the ids back
(ubatches of 32-63 tokens: MUL_MAT_ID is offloaded from 32 tokens, the F1
skip starts at 4*n_expert ids), it uploads only the experts that pass 0
selected, and passes 1 and 2 read stale VRAM for the others (upstream issue
#32755, "GGGG" output).
Add a post_load_tensors hook to llama_model (called after the weights are
loaded, and on the no_alloc path after the dummy buffers are set). In
inkling2_loop it replaces, for each slot of a later pass, every tensor in a
host buffer with a view of it (name "<name>.visit<pass>", buffer and data of
the original). The scheduler hashes tensors by pointer, so each pass gets its
own copy, freed after that pass, and the copy callback sees the ids of that
pass. Weights in device buffers and CPU extra buffers (CPU_REPACK, not host)
stay shared. The pointer-keyed weight maps (w_scales, prec_policy, hdmd.rot)
get entries for the views. The views live in a model-owned context outside
ctxs_bufs, so the F4 release/restore, tensor counts and the memory breakdown
do not see them; they point only at host buffers, which F4 never releases.
LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors (A/B knob).
Not built or run yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32740 at head 808fd1c4e,
its four commits (1fa93f902, d29f9d489, ff91e4f84, 808fd1c4e) squashed.
Inkling's per-query bias table is applied inside flash attention
instead of a dense [n_kv, n_tokens, n_head] bias tensor, which cuts
Inkling's compute buffers and long-context prefill time on the RX 580.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").
One conflict, ggml-vulkan.cpp use_mask_opt: kept the fork's GCN
`HSK >= 256 || HSV >= 256` clause and its comment (F2) and upstream's
`!use_sparse`, added the PR's `!bias &&`.
Original commit trailer: Assisted-by: Codex
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32735 (head 0ea4ffd00),
cherry-picked unchanged. The Vulkan backend runs SCALE on a
non-contiguous source instead of rejecting it (CPU fallback). Carried
together with #32740 and evaluated as a pair.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").
Original commit trailer: Assisted-By: Opus 5.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
HF_UI_VERSION must use the tag after the '|' in .ui-stamp; bundle path
for agents told not to push; build backup and copy-based rollback;
status, pointers and last-run record for 3bf6b55b3; q35.py build path
note; disk-cache restore test wording; open questions.
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
Upstream #30591 added its own tiled-transpose concat pipelines, selected
for any dim-0 concat whose src1 is transposed. Behind the fork's fast path
(which already takes ne0 >= 32 and honours GGML_VK_CONCAT_FASTPATH) that
selector still caught the decode case (ne0 < 32), bringing back the
decode regression the TILE_DIM gate fixed (FORK.md F3), and
GGML_VK_CONCAT_FASTPATH=0 no longer gave the generic kernel.
Apply the same gate and the knob to upstream's selector.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort
with the production mmproj), the fix, the new mmproj cases (15 PASS), and
that the earlier 35B numbers were measured with --no-mmproj. The 35B run with
production's exact command line (diskcache-window3.sh) is ready, not run, and
a precondition for deploying --cache-disk-path.
Runbook:
- tests must use production's exact model command line, mmproj included;
- the validated build has all three targets;
- the deploy mtime list includes libllama-server-impl.so and
libllama-common.so (llama-server is a stub);
- deploy and rollback steps run as one setsid-wrapped script on the pod,
never as a backgrounded cat && sh -n && setsid chain from kubectl exec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production runs the 35B and gemma with an mmproj, and the first deploy of the
disk prompt cache aborted on that path. Every case had used --no-mmproj. Now
--no-mmproj is in DC_MM, separate from DC_ARGS, and two cases run with the
0.8B's mmproj (--no-mmproj-offload, arbiter env, LLAMA_MMPROJ_ONDEMAND=1),
against their own reference:
- restore-mmproj: turn 1 is saved on SIGTERM and restored after the restart,
prompt_n equal to the reference, output identical;
- mmproj-image: an image request (no restore, answer "red"), then the text
chat (restored, identical), then an image again, then SIGTERM with media in
the slot (no state copied, exit 0, the file count unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deploy of cb48bf224 with --cache-disk-path was rolled back: the first
request to the production 35B aborted in disk_cache_load().
server_tokens::get_tokens() has GGML_ASSERT(!has_mtmd), and has_mtmd is true
whenever an mmproj is loaded, even for a text-only prompt. Production's 35B
and gemma-26B always load one (-hf pulls it). disk_cache_collect() and
server_prompt_disk_cache::save() made the same call, so the shutdown save
would have aborted too. Every test had used --no-mmproj, which hid it.
disk_cache_text() takes the list from get_text_tokens(). A prompt is
text-only when that list is as long as size(), because media positions hold
LLAMA_TOKEN_NULL. So a text-only prompt is saved and restored with an mmproj
loaded, which is the main case (OpenWebUI chats on the 35B and gemma), and a
prompt with media is never saved or restored. When the slot itself holds
media, its reuse estimate is server_tokens::get_common_prefix(). These four
calls were the only get_tokens()/mtmd-unsafe accesses in the feature.
fork-validate --diskcache with the new mmproj cases, beside production:
15 PASS, 0 FAIL, GPU legs included. With the 0.8B's mmproj loaded, a text
chat is restored: prompt_n 58 of 596, output identical to the reference.
An image request is answered "red" with no restore, the text chat after it
is restored, and a SIGTERM with media in the slot copies no state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.
Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.
Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F11 in the per-feature format: design, invariants, the measured costs (CPU
only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at
30 s). Also: the recommended config (not applied), the commit map entries, the
F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache,
the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run
yet). Runbook: --diskcache in phase 1, the pod window script, the deploy
notes (every deploy empties the cache, /root free space, a slower stop), a
post-deploy check, and the cache dir.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs beside production. The model is the small hybrid (Qwen3.5-0.8B), CPU only
(-dev none -ngl 0, no VRAM), with chat requests and greedy decoding. A
reference server that never restarts runs turn 1 and turn 2. The candidate
saves turn 1 on SIGTERM and restarts. Turn 2 must then be restored from disk
with the same prompt_n as the reference (only the suffix processed) and the
same answer. Cases:
- restore plain, with the arbiter env warm, and forced cold by a fake waiter
on want.lock (the save from a released state). Each case records the exit
time and SIGTERM-to-token-free, also once without the cache;
- no restore for another conversation, or with another n_ctx, KV type or
model path (other key);
- a truncated file is removed at startup and a byte-flipped one at load,
never used;
- the size limit evicts the oldest file;
- a write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored: EFBIG) is logged, the
exit stays 0 with no file left, and the next start serves;
- the same restore on the GPU (-ngl 99, arbiter env, warm and cold) when
>= 1.5 GiB of VRAM is free, else SKIP.
start_server gains SRV_WRAP, a command prefix that execs the server.
First run (2026-10-25, beside production): 11 PASS, 0 FAIL; the GPU leg
SKIPped (1219 MiB free). The first run found the CPU_REPACK release crash
(exit 139 in the cold case).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama-swap SIGTERMs a model when the owner switches models (only one large
model fits in RAM), and switching back re-prefills the whole conversation:
about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR,
each slot with a prompt is saved to DIR on shutdown. After the restart, a
request whose prompt starts with a saved prompt is restored from the file,
so only the suffix is processed. Off by default.
Save, in server_context::start_loop() after the loop returns:
- disk_cache_collect() copies the state of each slot with >= 256 text tokens
to host memory (llama_state_seq_get_data_ext, flags NONE) while the device
buffers still exist. A cold server reads its released state through
llama_context_restore_memory_host(), with no device use and no extra copy.
- vram_share_exit() then frees the VRAM and releases the GPU token as before,
so only the D2H copy delays the token release, never the disk IO.
- disk_cache_flush() writes the files. A slot is skipped if its KV does not
cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not
decoded) or if the same token list is already on disk.
File (server_prompt_disk_cache, server-task.cpp):
- A 48-byte header (magic, version, key hash, sizes, payload hash), then a
payload of 8-byte padded sections: tokens, checkpoint table, state, the
checkpoint states. The hash is FNV-1a over the 64-bit words of the payload.
- Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read
is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked
the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file.
- The index is built at startup from the headers. A bad magic, a size that
does not match the header, or an inconsistent table means the file is
removed. A hash mismatch at load also removes it, and nothing is restored.
Other keys and other versions are kept. A .tmp older than 1 h is removed.
- --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the
least recently used *.lpcd files of every model in DIR are removed until the
new file fits; a restore updates the file mtime. Entries the new prompt
extends are dropped after the write and do not count against the limit.
A write also needs its size + 1 GiB free on the file system. A write error
or a full disk is logged; the file is not saved and the exit stays clean.
Key: the build commit, the model path + size + mtime, LoRA and control
vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the
K/V types, the flash-attn type, and the RoPE/YaRN parameters.
Restore, in get_available_slot() after the RAM cache:
- Only while warm (holding the GPU token), with cache_prompt on and no media
tokens. The file must reuse at least 256 more tokens than the slot holds.
- Reusable means one of two things. Either the saved token list is an exact
prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint
at or below the common prefix can be used. A plain-KV model can be cut back
to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the
template rewrites the previous assistant turn (the reasoning is dropped), so
the saved list is almost never a prefix of the next turn. The checkpoints
are exact snapshots, and update_slots() rolls back to one exactly as it does
for the RAM cache. They are restored with id_task -1, so one that fails to
load means a re-prefill, not an abort.
- The slot's current prompt goes to the RAM cache first. A set_data failure
clears the slot and removes the file.
Not supported with speculative decoding (warning, disabled): the draft state
is not saved. Only the slots are saved, not the RAM cache entries.
Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention):
- 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms,
exit 314 ms (125 ms without the cache). After the restart: read 166 ms,
restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full
prefill. The answer is identical to a server that never restarted.
- 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS
+162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms,
turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill.
- The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node
overlay) stalled once: 5.8 s for 64.5 MiB.
llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a process that is still there after 30 s. So the exit
save adds to the model switch time: about 1 s per GB on the PVC.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After llama_context_release_device(ctx, true) the KV cache and the recurrent
state live only in host shadows, and their tensors have no buffer, so
llama_state_seq_get_data cannot read them. The disk prompt cache must save
a cold server's slots at exit, and a shutting-down server must not restore
to the device (it may not hold the GPU token, F7).
llama_context_restore_memory_host() makes the state readable without the
device. restore_device_buffers(to_host = true) wraps each shadow as a
CPU_Mapped buffer (ggml_backend_cpu_buffer_from_ptr) and points the tensors
into it: there is no copy and no allocation. The first version copied into a
CPU buffer instead, which needs the full KV allocation a second time, about
2.7 GB for the 35B at ctx 131072; the node has had an OOM-controller kill
before. The buffer base is rounded down to the 32-byte alignment that
cpu_buffer_from_ptr asserts (a large std::vector is only 16-byte aligned);
those leading bytes are never accessed. With to_host the shadows are kept,
because the buffers point into them.
It sets device_discarded, so restore_device() aborts afterwards. Exit only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A model with weights on the CPU device (-dev none / -ngl 0) segfaulted in
llama_model::release_device_weights() the first time the VRAM arbiter put
it to sleep: CPU_REPACK buffers are not host buffers, so load_tensors marked
them releasable, and capturing the shadow called their NULL get_tensor.
Reproduced with the production build 2fc9f9c35 (Qwen3.5-0.8B, -dev none
-ngl 0, LLAMA_SLEEP_VRAM_ONLY=1 LLAMA_SLEEP_EVICT_KV=1, force a yield with a
waiter on want.lock): exit 139 at the release, before any SIGTERM. The
faulting call (rip 0) returned into release_device_weights.
Only buffers on a non-CPU device are releasable now. These buffers are in RAM
already, so there is nothing to free. Production is not affected: every
arbiter model there has its weights on the GPU, and a releasable CPU_REPACK
buffer would have crashed it. Found by the --diskcache check, which runs
its arbiter cases CPU only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Feedback from the 2026-10-25 deploy, which used only these docs:
- status: production is 2fc9f9c35; F10 deployed and verified on a 35B
GPU load; --moe-exact and median --perf have run on the GPU; last
full fork-validate run (37 PASS) and its timings; the first-candidate
-arm VRAM spike seen twice is an open question.
- runbook: a no-rebase redeploy path; --arbiter beside production only
if 1536 MiB VRAM is free, else in the swapdown window; full --moecache
model path; page-cache eviction before and after phase 2; deploy
steps reordered so the memory check follows stopping llama-swap;
in-place rebuild and system_fingerprint check; mtime check through the
versioned symlinks; rollback pointer numbering and one format; disk
space; nothink alias and a PPM image helper for post-deploy requests;
power1_cap via sysfs; restart and OOM checks; harmless UI warning.
- fork-validate.sh header: note the --arbiter VRAM requirement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With -lm none the GGUF is read() into our own buffers, but the kernel keeps
the file in page cache afterwards, charged to the pod's cgroup: about 20 GB
after a 35B load. On 2026-10-24 the Talos OOM controller killed the pod right
after a load under exactly that pressure. Those pages are pure waste once the
data is copied.
llama_file::drop_cache() wraps posix_fadvise(POSIX_FADV_DONTNEED) (Linux only,
errors logged at debug). load_all_data() drops each copied tensor's range right
after reading it, so the cache never grows past about one tensor, and drops the
whole file at the final cleanup to catch the metadata and partial pages at
tensor edges. Both only when the loader holds no mapping of the files
(mappings.empty(): not mmap, not mmap+mlock, no lazy tensors), so nothing that
is mapped is ever advised away. Split GGUFs are covered (every file in files).
clip.cpp always copies the mmproj via ifstream, so it drops the whole file
after loading it.
Measured on the pod, gemma-4-E2B UD-Q4_K_XL (2.5 GiB) + mmproj BF16 (0.9 GiB),
CPU only, -lm none, cold cache, same binary with the drops disabled as the
control (4 rounds, rotated order):
- file pages resident after load (mincore): 2431-2444 / 940 MiB -> 0 / 0
- peak model-file residency during load: 2.5 GiB -> 1.2-1.5 GiB (largest tensor)
- pod cgroup file after load: +3.37 GiB -> +0; host MemFree after load
0.6 GB -> 2.5-2.8 GB (MemAvailable counts clean cache, so it cannot show this)
- load time: 5.87 s -> 6.82 s median (+0.95 s); a post-load-only drop is
6.56 s (+0.68 s). The cost is synchronous page freeing that reclaim would
otherwise do later; -lm dio (direct IO) costs the same (+0.9 s) and does not
cover metadata or the mmproj.
- greedy output byte-identical; -lm mmap keeps the model file fully resident
and mapped (2498 MiB, no refaults); -lzm on skips the drop (file mapped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 1 now covers the VRAM release/restore check and the handoff latency
A/B against --ref (which must be the production build); phase 2 adds
--moe-exact on the 35B and keeps its scratch dir. Explains median gating and
--perf-report, that a pod restart empties /tmp, that the manual checks are
now automated, and that gdb is not in the pod image (lost in the 2026-10-24
restart). States the commit trailer convention and what llama-swap does with
a child that exits vs one that wedges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README fork section still said "the two code changes below are auto-on"
(the fork has three prefill changes plus the VRAM sharing and server fixes)
and recommended --n-cpu-moe 28 on the 35B, which FORK.md section 4 records as
worse at depth and unable to hold a long context. Replace it with a short
accurate summary that points to FORK.md, the runbook and the scripts. The
Laguna time profile, the iSWA KV note and Laguna's --parallel 1 saving were
only in the README and move to FORK.md sections 4 and 5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- status: no hardcoded fork-commit count (the list includes docs and
scripts; count git log origin/master..HEAD); what is after the production
commit; the commit-trailer convention of this private fork vs upstream's
AGENTS.md rules
- commit map: the commits after the production commit
- F1: validated by the new --moe-exact check and the LLAMA_MOE_IDS_SKIP knob
- F3: every graph that builds concat(state, transpose(x), 0) (bailingmoe3,
clef, kimi-linear, kimi-k3, glm5-next, mamba-base, plamo2, lfm-base,
qwen4exp besides delta-net); only the delta-net path is measured
- F4/F6: the new v-vram-* and a-handoff-latency checks with first results
- F5: encode_mtmd_chunk was split out of process_mtmd_chunk, not renamed
- F6: the vram_cold ordering rule restated from the code (the measured
failure was under the doorbell design; with the inline probe should_sleep
cannot run inside the restore; keep it, and A/B handoff latency before any
change) and why vram_acquire_for_load sets the flag early
- F7: what llama-swap does when a child dies, verified with a scratch
llama-swap: "upstream exited unexpectedly", restart on the next request
- F9: new flags, phases, median gating, 2026-10-24 reference values, the
/tmp and gdb loss on the pod restart, new grep targets and gaps
- section 7: stale comments fixed, what is not yet run on the GPU
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F1 (the routing-ids readback skip in sched_copy_experts) had no automated
check. --moe-exact MODEL reads expert_count / expert_used_count from the GGUF
header, computes the skip threshold ceil(4 * n_expert / n_expert_used) tokens
per ubatch (128 on the 35B), and runs one greedy request with a prompt of
about 2x that in a single ubatch, every expert on the host, so the prefill
takes the skip and each decode step takes the readback. The candidate's 64
tokens must equal the reference's byte for byte; with the LLAMA_MOE_IDS_SKIP
knob in the candidate, also the candidate with the skip forced off (=0), which
proves the skip exact even when the reference carries it, and =log must show
skips, all at >= 4 * n_expert ids. Without the knob skip-engaged SKIPs with
the batch-size argument. It needs the GPU to itself (same guard as --perf,
now shared as gpu_exclusive); FV_TEST_SHARED_GPU=1 bypasses the guard for
script tests against stub binaries only.
Also unset inherited LLAMA_MOE* variables.
Not run on the GPU yet (needs a swapdown window). Dry-run on the pod
against stub servers and the real 35B GGUF header: threshold 128, all-PASS
when outputs match, FAIL on a differing reference, SKIP without the knob,
FAIL for a dense model and while llama-swap runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing checked that a release actually frees VRAM. --arbiter now takes a
baseline of mem_info_vram_used before any scratch server starts, then with A
warm and alone on the scratch arena makes it release through a fake waiter
on want.lock. v-vram-release requires the card-wide delta to drop by at
least 70% of what A added warm; v-vram-restore requires it back (and the
output equal to the reference) after the next request. Production's own
share is re-read at every point from the DRM fdinfo of every process the
script did not start (deduplicated by drm-client-id); if it moved by more
than 64 MiB the check SKIPs instead of blaming the candidate.
Run beside production on the pod (reb1024 vs itself): warm +546 MiB, cold
+4 MiB, restored +546 MiB, production steady at 5053 MiB; 16 PASS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--arbiter only reported the candidate's handoff latency, while the fork's
process rule (two production deploys lost) is to A/B the ordinary handoff,
not only the pathological cases. After the arbiter cases, a-handoff-latency
starts fresh A/B server pairs of each build on the scratch arena and runs
blocks ref, cand, cand, ref of 16 alternating handoffs each. It FAILs if the
candidate median exceeds the reference median by more than max(25%, 200 ms)
and reports both medians, maxima and per-block medians.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--perf averaged the two per-arm llama-bench means of each build, so one
catastrophic sample (CPU contention from the ~60 other pods on the node) moved
the result, against the fork's own rule of medians, never means. llama-bench
now writes -o jsonl (every repetition sample) next to the markdown table, and
the gates use the median of all samples of a build pooled over its two arms
(4 per prefill cell, 6 for decode); the table shows min-max and n.
--perf-report TSV re-aggregates a saved <scratch>/perf.tsv without a GPU.
Checked offline on the pod: the 2026-09 A/B samples in results.txt give
pp2048 reference 250.24 (old mean) vs 247.76 (new median), candidate 280.15
vs 276.93; a synthetic 2026-10-24-like set with one 180 t/s candidate sample
read -9.3% (FAIL) the old way and -0.2% (PASS) with medians.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The llama-bench sweep measured decode as tg32 at the default depth and the
README called it the continuity metric, while tg32 at n_ctx 32 is launch /
barrier bound on this card and is not a valid decode metric (it once read
-36% between builds at parity at tg256). lbsweep.sh now runs prefill
(-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as
two invocations and labels depth cells (tg256@d4096). summarize.py marks
old depth-less tg rows as invalid-metric history. The README examples and
the reference values are updated to the 2026-10-24 production numbers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LLAMA_MOE_IDS_SKIP=0 forces the ids readback in sched_copy_experts (the
upstream path) and =log reports each skip on stderr, so validation can show
that the skip engages at prefill and that output is byte-identical with and
without it in one build. Read once (function-local statics); unset changes
nothing. Not set in production.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments only, plus one log string:
- the arbiter block comment said the KV cache is never evicted; with
LLAMA_SLEEP_EVICT_KV it is (and the scheduler and scratch with it)
- vram_go_cold said vram_ensure_warm() rebuilds the encoder; only
encode_mtmd_chunk() restores it, and only for an encode
- vram_ensure_warm and the mtmd init block named process_mtmd_chunk; the
restore lives in encode_mtmd_chunk
- handle_sleeping_state said waking happens lazily before decode; it
restores eagerly on the wake transition, before update_slots()
- the vram_cold ordering note now says why it still holds with the inline
want-lock probe
- the sleep log line said "keeping KV cache" even when LLAMA_SLEEP_EVICT_KV
evicts it; it now names what actually happens (the prefix
"server entering sleeping state" that fork-validate.sh counts is kept)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FORK.md documents every fork feature (what and why with measured numbers,
where, how it works, invariants, past failures, rebase hazards, validation),
the dead ends measured on this hardware, the production configuration and
the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation,
deploy, rollback and GPU-recovery procedure plus an operations reference.
The README fork section links both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checks the op kernels, the concat fast path, the VRAM arbiter incl. every
shutdown path that has caused a production incident, and prefill/decode
against a reference build. Prints PASS/FAIL per check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GGML_VK_CONCAT_FASTPATH=0 forces the generic path and =log reports each
fast-path dispatch, so validation can prove the path fires at prefill and
not at decode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:
- SIGTERM with a request in flight: terminate() stopped only the task queue.
The HTTP handler kept waiting for results the loop would never send, the main
thread blocked joining it, and the token was released only in the destructor
after that join - i.e. never, until the client went away. Terminate the
result queue too (readers now get "server is shutting down": a 503, or an SSE
error event), and let wait_until_no_sleep give up once stopping.
- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
inside the wake callback, holding mutex_tasks, so terminate() waited for the
current holder to finish and then for a pointless restore of our weights.
Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
stays cold, skips the restore, and the loop exits before update_slots.
Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.
Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A SIGTERM that lands while the start_loop() thread is inside a sleeping
callback deadlocks the server. The callbacks run under
server_queue::mutex_tasks, and the kernel delivers the signal to that same
thread; the handler calls terminate(), which locks mutex_tasks again, and the
thread waits on itself forever.
With the VRAM arbiter that window is wide - a whole device release, or a
blocking wait for the GPU token - and llama-swap sends SIGTERM to evict
models, so it hit production: an evicted model wedged mid-release, never
released token.lock, and every other model queued behind it. SIGTERM could
not stop it either; only SIGKILL did.
Make the handler only write the signal number to a self-pipe, which is
async-signal-safe, and run shutdown_handler on a dedicated thread. terminate()
there simply waits until the loop thread drops mutex_tasks, after which the
loop exits normally. The thread is joined before the objects the handler uses
are destroyed. If pipe() fails the old direct call is kept.
Reproduced 6/6 off production (SIGTERM during release and during the token
wait); with the fix 8/8 clean exits, and handoffs, image requests and greedy
output are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream #28334 removed --mmap/--no-mmap/--mlock and llama-bench's -mmp, so the
harness flags no longer parse: the server would refuse to start and llama-bench
would silently produce no rows. Switch both harnesses to -lm none, the same
load mode, and note --moe-cache-mib as a server-only option worth measuring.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The MoE cache (--moe-cache-mib, #29887) keeps the most recently used host
experts in device banks. The on-demand VRAM path (release_device /
restore_device, used by the server VRAM arbiter and the on-demand mmproj path)
did not know about it, so its banks were never freed on handoff. Restoring them
by reallocation alone would also be wrong: the banks come back zeroed while the
slot maps still mark experts as cached, so the model would silently read zero
weights.
Free the banks on release and empty the cache instead of shadowing it: re-init
every LRU, re-point its slot-map pointers at the host slot maps and clear those
to -1. The banks are reallocated (which also re-inits their views) and cleared
on restore, before the scheduler is rebuilt, and the cache refills from the
host experts as before. Release runs in both modes, weights-only and evict_kv.
device_resident() now covers the cache too, for the early returns in
release/restore, the auto-restore in decode() and
llama_context_weights_resident().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
release_device(evict_kv) drops the scheduler, so a context evicted by the VRAM
arbiter and then destroyed reaches the destructor with a null sched. The upstream
"compute buffer size matches expectation" loop calls
ggml_backend_sched_get_buffer_size() unconditionally, whose GGML_ASSERT(sched)
then aborts: the server exits 134 instead of 0 on every shutdown taken while
cold, which under llama-swap makes an ordinary stop look like a crashed child and
leaves a ggml backtrace in the log each time. It also skipped the rest of the
destructor, so the cold teardown path had never actually run to completion.
Guard at the call site rather than relaxing the assert - the assert is right, and
every other caller reserves the scheduler first. Same shape as the null guards in
synchronize(), memory_breakdown() and the released-buffer iteration.
The loop is a diagnostic size comparison, already reported by sched_reserve() at
load, so skipping it when cold loses nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the
holder's warden turned the inotify event into a queue flag. A ring carried no
notion of "still wanted", so it was lost whenever the holder was not already warm
and listening - during its model load (the watch does not exist yet, and the
kernel does not queue events for a watch that is not there) or inside
restore_device - and any ring that survived past the point its sender had been
satisfied caused a spurious release. Lost rings wedged the waiter permanently,
because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder
had no other reason to release. The wait blocks under mutex_tasks, so the whole
server stopped answering while /health still returned 200.
Express the request as kernel state instead. A waiter holds <arena>/want.lock
shared while it waits and drops it once it owns the token; the sleep decision
probes that lock non-blocking and releases the GPU while anyone is waiting. The
probe needs its own fd - flock treats two open file descriptions of one file
independently, so probing on the waiter's fd would convert our own lock rather
than conflict with it. Nothing can be missed, nothing goes stale, and a waiter
that dies is cleaned up by the kernel.
Probe from should_sleep() on the loop thread rather than from a warden thread.
Routing it through a flag is what made the first attempts fail: start_loop()
holds mutex_tasks from should_sleep() through the callbacks to the wait, so a
warden's request_yield() blocks on that mutex and is admitted only after the flag
has been consumed, latching a release for the next wake. Reading live state where
the decision is made has no edge to latch, and drops the warden's poll latency.
Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but
the loop clears req_stop_sleeping on the way in, so a loop that slept again
before the waiter ran stranded it forever - re-ask on every wake. And a task
queued after the waiter saw us awake could not wake us by itself, so sleep now
also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the
request that woke us is not queued yet, and yielding at once only sends it round
again.
Measured on the RX 580 pod, two servers contending, 60 alternating handoffs:
0 stranded, median 0.309 s against the doorbell's 0.314 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a
loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of
40 layers) build_conv_state emits a transposed concat per recurrent layer, so
the fast path runs 30 times per decoded token.
At one token the transposed source is a single column, so there is no
uncoalesced stride left to fix, but the path still costs two dispatches and an
unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes
the command stream. Measured -17% tg256 on RX 580; the barrier, not the
half-empty transpose dispatch, is nearly all of it.
Require the transposed source to be at least one TILE_DIM wide. That source is
qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the
fast path, decode and small speculative drafts take the generic one.
pp8192 +11% retained, tg256 back to parity, generation byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.
When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.
Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.
Assisted-by: Claude
Add clip_release_device/clip_restore_device (and mtmd_release_device/
mtmd_restore_device wrappers over the vision + audio contexts) that free
the multimodal encoder's device weight buffer to a read-only host shadow
and rebuild it on demand, using the same shadow/free/reallocate pattern
as llama_model weights. No-op for a CPU-backed encoder. This lets the
server drop the ~hundreds-of-MiB vision encoder from VRAM when it is not
encoding an image.
Assisted-by: Claude
On device release with KV eviction, also call ggml_backend_free_scratch()
on each backend so a cold model drops its Vulkan compute preallocations
(reallocated lazily on the next compute via restore).
Also guard llama_context::memory_breakdown() against a freed scheduler:
evict_kv release resets sched to null, so a cold model that is then torn
down (e.g. terminated by the process manager) hit GGML_ASSERT(sched) in
ggml_backend_sched_get_buffer_type. Skip the compute-buffer accounting
when sched is null.
Assisted-by: Claude
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.
Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).
Assisted-by: Claude
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left
resident when a model's device buffers were released for on-demand VRAM
sharing - llama_memory_recurrent::release_device_buffers() was a no-op
default. Implement it (and restore_device_buffers) with the same
capture-host-shadow / free / reallocate pattern as llama_kv_cache, so
llama_memory_hybrid now evicts both its attention KV and its recurrent
state. The state is read-write, so its shadow is recaptured on every
release.
Assisted-by: Claude
Under LLAMA_SLEEP_EVICT_KV a cold model still held the scheduler's worst-case
compute buffer (hundreds of MiB, e.g. ~700 MiB for gemma-26B) plus the resident
experts. release_device(evict_kv) now also frees the sched (sched.reset() +
sched_need_reserve), and restore_device() rebuilds it via sched_reserve(), so a
fully-evicted model holds essentially no VRAM (gemma cold: 7147 -> 51 MiB).
Trades a heavier re-warm (weights+KV H2D + sched reserve) for the freed VRAM;
weights-only mode is unchanged (fast switch, keeps KV+compute).
Assisted-by: Claude
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.
Assisted-by: Claude
release_device_weights()/release_device_buffers() leave a null buffer in
ctxs_bufs while the device memory is released for on-demand VRAM sharing.
The memory-breakdown / total_size / clear paths iterated these and called
ggml_backend_buffer_get_size()/get_type()/clear() on the null buffer, tripping
GGML_ASSERT(buffer) and aborting on shutdown of a cold model. Skip null buffers
in llama_model::memory_breakdown and llama_kv_cache::{memory_breakdown,
total_size,clear}.
Assisted-by: Claude
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.
- add vram_arena_open() (idempotent flock/doorbell setup) and
vram_acquire_for_load(), called from load_model() before
common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
request without a re-warm (drop the init-time go_cold cold-start).
Assisted-by: Claude
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.
Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.
Assisted-by: Claude
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.
- llama_memory_i: add release_device_buffers()/restore_device_buffers()
(default no-op). Implemented in llama_kv_cache (D2H shadow of the live
ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
buffers when requested; restore_device() rebuilds them. Public API
llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
since the KV shadow adds a D2H/H2D copy of the live cache each cycle.
Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.
Assisted-by: Claude
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.
- llama-model: release_device_weights()/restore_device_weights() capture a
compact host shadow (stable iteration order, view-skipping) and free then
realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
(not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
enforces "resident iff holds token"; an inotify doorbell forces the holder
to release on contention. The warden thread only touches the task queue, so
releases run on the loop thread and never race a decode.
Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).
Assisted-by: Claude
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).
Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).
Assisted-by: opencode
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.
Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.
Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.
Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.
Assisted-by: Claude
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.
Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.
Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.
The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.
Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.
Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Used angelX to optimize SYCL/drama 3 coordination, results are as follows
Each output reads only inputs with i*s0 <= idx < i*s0 + K, in the same ascending order as before.
Arc Pro B70, oneAPI 2026.1, test-backend-ops perf, base 9b56ccf00 vs patch, median of 3 same-boot A/B/A rounds, A/A 1.000-1.001:
| input [ne0, ne1] | kernel [K, OC, IC] | s0 | base us | patch us | speedup |
|---|---|---|---|---|---|
| [16, 1536] | [16, 768, 1536] | 8 | 3910.57 | 1348.59 | 2.90x |
| [128, 768] | [16, 384, 768] | 8 | 37735.00 | 1665.26 | 22.66x |
| [1024, 384] | [8, 192, 384] | 4 | 270371.75 | 1471.74 | 183.71x |
| [4096, 192] | [4, 96, 192] | 2 | 534614.63 | 752.74 | 710.23x |
Shapes are the drama 3 codec decoder upsampling stack, added as perf cases. Chained 4-layer stack at 1/4/24 input frames: base 66.3/847.4/28638.5 ms, patch 2.3/6.7/30.7 ms. Kernel numbers, not end-to-end TTS latency.
Correctness: test-backend-ops -o CONV_TRANSPOSE_1D 116/116 on base and patch, and 116/116 rebased on 3f03b1772. Seeded output dump (7.46 MB) byte-identical to base.
Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857). With only those cases excluded: 20145/20173 on base and patch, and every patch-run failure also fails on base.
Co-authored-by: Titaniumtown <titaniumtown@proton.me>
* mtmd: support hibiki realtime duplex
* fix and clean up
* fix long session quality
* fix ci
* tests : add the moshi fusion baseline for Metal
The Metal fusion check found 5 RMS_NORM+MUL fusions in the new moshi
arch and no baseline for them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ui: open the context gauge panel on click as a popover
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the context gauge trigger the ghost button hover
Assisted-by: Claude (Opus 5.5 Medium)
The loop slots copied the physical layers at the end of load_arch_tensors,
before the generic pass attaches the optional .scale tensors, so the later
passes through the core ran NVFP4 experts without their per-expert scales.
Add an optional post_arch_tensors hook that runs after the scale pass and
share the slots there.
Fixes https://github.com/ggml-org/llama.cpp/issues/31680
The matmul_tiled() implementation in tinyBLAS_Q0_PPC requires nc to be a multiple of 8 to perform matrix multiplication correctly.
The previous alignment logic contained a special case for n == 4, which allowed n_aligned to be 4. Consequently, the tiled path could be selected with nc = 4, violating the kernel's tile-width requirement.
This caused incorrect matrix multiplication results, observed as garbled inference output when running Flint Q4_K_M and Gemma 3 27B Instruct.
The fused lightning indexer is always used now, so drop the cparams flags, the auto probe and the non-fused fallbacks.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* Conversion
* Separate model_type
* Add gate_rank to allow building from metadata while testing
* Base model
* Loop graph
* Support the new is_varloop attribute
* Adopt naming scheme from https://github.com/ggml-org/llama.cpp/pull/32588
* Rename to varying
* hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1)
* hip: write the RDNA1 MMVQ values explicitly instead of recursing into the generic table
* hip: list the RDNA1 MMVQ table before RDNA2 in the enum
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ci : run the OpenVINO GPU job only for OpenVINO changes
The self-hosted OpenVINO GPU job ran for most PRs and every master push
on a shared runner, so it queued for hours.
- ci-self-hosted-openvino.yml (GPU): runs only for changes to
ggml/src/ggml-openvino.
- build-openvino.yml: keeps its triggers (OpenVINO changes on PRs, every
source change pushed to master). The Linux job runs on a GitHub-hosted
runner and tests the OpenVINO CPU device only.
* address review comments
* ci : limit the OpenVINO PR CMake trigger to ggml
* ci : drop the OpenVINO Ubuntu job
* ci : trigger OpenVINO PR runs on tests/CMakeLists.txt
* convert: qwen3-asr: put "language ..." output into thinking block
* convert: qwen3-asr: read the chat template from models/templates
The converter embedded its own copy of Qwen3-ASR.jinja. Read the
bundled file instead, as the Kimi-K3 and Kokoro converters do, so
the GGUF and test-chat share a single template.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Llama Chat compiles @ggml-org/llama-voice from ui/lib/voice/src, but the
CMake stage and the Docker UI stages copied only ui/lib/ui, so their npm
build could not resolve the voice imports. Stage ui/lib/voice/src next to
Llama UI and track it as an input of the CMake build.
Assisted-by: Claude (Opus 5.5 Medium)
* mtmd: drama3 voice cloning
The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3:
- conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2
- fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj,
builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2)
- drama3_gen tower: teacher forcing when the caller gives the codes of the next frame
- drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the
reference frames go through the backbone as codec rows and through the tower with their codes
- mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state
Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX
* fix correctness
* clean up
* rm redundant code
* fish_s2: rm unused import
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui: add a voice engine for capture, peaks and transcription
Voice input needs a live waveform, a clip that can be sent and drawn,
and a transcript, whether the model streams text while the user speaks
or not. This adds the engine for it in src/lib/voice, with no imports
of app stores or components, so it can move into a package later.
- VoiceCapture: one microphone start gives live levels (RMS on a dB
curve, fixed history, newest last), PCM16 mono chunks at 16 kHz from
an inline AudioWorklet (ScriptProcessor fallback), and on stop a
16-bit mono WAV clip with its duration and peaks. Tracks, nodes and
the context are released on stop, cancel, a failed start and page hide.
- computePeaks / computePeaksFromAudio: normalised peaks of samples, a
buffer, or a stored base64 or Blob file.
- transcribeClip: POST /v1/audio/transcriptions, with the server's own
error message on failure.
- RealtimeTranscriber: the /v1/realtime?intent=transcription socket,
queueing audio until session.created, a final commit, and a typed
RealtimeUnsupportedError when the model cannot stream or the socket
is refused (a browser websocket cannot send the API key header).
- startVoiceSession and VoiceRecorder: try realtime, fall back to one
HTTP transcription of the clip, and expose levels and live text.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add voice waveform and transcript components to Llama UI
Voice input needs three visuals that the app can feed with its own
recording and transcription logic: the live waveform while recording,
a playable waveform of a recorded clip, and a clip with its transcript
that opens on hover, keyboard focus or a tap.
VoiceWaveformLive draws input levels as many bars as fit the width,
newest on the right, and holds still with reduced motion.
VoiceWaveform plays a clip through its own audio element, fills the
played part in the primary colour, seeks by press, drag and arrow keys
(a slider) and pauses any other clip that was playing.
VoiceTranscriptReveal keeps the transcript in the DOM for screen
readers while it is closed, and shows transcribing and error states.
The waveform math (resampling peaks, bar capacity, seek and time
formatting) is in src/lib/voice with unit tests. The components are
exported from the package root and from @ggml-org/llama-ui/voice, which
Llama Chat resolves to the sources like the other entry points.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add stories and tests for the voice components
Storybook gets UI/Blocks/Voice: the live waveform fed by a fake
microphone meter in a chat form row, the clip waveform in its sizes and
states, and the transcript reveal in each state and on touch. The demo
audio is a WAV synthesised in the story, so no binary file is added.
The play functions check seeking by keys and by press, the reveal on
hover, keyboard focus and a tap, and a clip without audio. Pressing play
needs a real click before a browser plays audio, so a client test covers
play and pause and one clip pausing another.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: store voice messages with their transcript and send it to models that cannot hear
A voice message recorded in the chat form is an audio attachment like an
uploaded file, plus what the app needs to draw and read it. The audio
attachment gets four optional fields: voice (recorded in the app),
transcript, peaks and durationMs. Stored rows, export and import carry
them as they are, so no migration is needed and older messages keep
working.
The transcript is for reading. A model with audio input still gets the
audio alone; a model without it (the user switched models later) gets the
transcript as a labelled text part in place of the audio. When the
model's modalities are not known yet, or there is no transcript, the
audio goes as before.
A transcript often comes back after the message was sent. The new
voiceTranscriptsStore follows each pending transcript by the clip's name,
hands it to a send that happens after it arrived, and otherwise writes it
into the stored attachment of the message it went out with, in the open
chat or, if the user left it, in the database.
Copying a message now includes its voice transcript.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add Voice settings and resolve what the microphone does per model
The user should not have to think about which kind of voice input is in
use, so the app decides it from the conversation's model and two
settings. resolveVoiceMode is a small pure function: with Voice input on
Automatic and a chat model that has audio input, a recording is a voice
message, transcribed by that same model so a router swaps nothing.
Otherwise it is dictation, with the chat model when it can hear, else,
in router mode, with the chosen transcription model. When no model can
transcribe, the microphone is unavailable and says why.
Settings get a Voice section of their own rather than more rows in
General, which is already long: these four belong together and one of
them lists the server's models.
- Voice input: Automatic (default) or Always dictate.
- Transcription model: Same as chat model (default) or, in router mode,
any model with audio input; on a single-model server it only says
whether that model can transcribe. It notes that dictating with
another model loads it beside the chat model.
- Transcribe voice messages: on by default.
- Transcription language: empty for automatic.
The keys live in the settings registry like the others, so defaults,
storage, reset and export need nothing new. The model picker is drawn by
SettingsChatVoiceTab, the way Extensions and Tools draw their sections.
Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: record voice messages and dictation from the chat form
The microphone now does what the conversation's model allows, without
the user choosing a mode: its label says "Send a voice message" or
"Dictate", and when no model can take audio it is disabled and says why
(shown in router mode only when some model there has audio input, so the
reason can be acted on; a single-model server without audio keeps the
send button as before). The microphone errors keep their warning state
and toast, now also for the voice engine's own capture errors.
Recording goes through the voice engine's VoiceRecorder instead of the
MediaRecorder flow. While it records, the actions row becomes a
recording row: cancel, the live waveform, the elapsed time and stop.
Escape or cancel drops the recording. The form keeps its two rows while
recording, so the message stays visible on desktop and the row fits the
phone layouts and the expanded sheet.
- Dictation writes the live transcript into the message at the caret
as it streams. DictationSpan replaces only the dictated stretch, moves
it when the user types before it, keeps what is typed after it, and
hands text the user edits over to them. When nothing streams, the row
shows Transcribing... after stop and the text goes in once it is back.
Cancel takes the dictated text out again. No audio is attached.
- A voice message streams its live text, if any, under the waveform,
not into the message. On stop the clip is attached as a voice file
with its peaks and length, and its transcript is followed by the voice
transcripts store (unless Transcribe voice messages is off).
Realtime transcription is skipped for servers reached through a tunnel
or with an API key, which a browser websocket cannot carry. The first
dictation with a separate router model shows a one-time note that it
loads beside the chat model.
The form takes an optional voiceRecorder, so tests and stories can run
it without a microphone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw audio attachments as playable waveforms with their transcript
A voice message should read as a clip, not as a file chip. Audio
attachments now render with Llama UI's VoiceWaveform inside
VoiceTranscriptReveal, through a direct branch in the core attachment
item (voice is core, so it does not go through the extension renderers):
the removable sm chip in the chat form, and the md clip in a sent
message, on the user bubble's surface.
- The audio plays from an object URL made from the stored base64 (or the
uploaded file's data URL), revoked when the clip leaves the screen.
- A clip stored without peaks, an older voice message or an uploaded
audio file, gets them computed from its audio once per session, in
memory only; uploaded files keep their name under the waveform.
- The transcript opens under the clip on hover, keyboard focus or a tap,
and shows Transcribing... while the voice transcripts store waits for
it, Transcription unavailable when it failed, and nothing when none
was asked for.
- The full-screen preview keeps its player and shows the transcript.
Tests run the chat form with a fake recorder: the microphone per mode,
the recording row, dictation around typed text, the Transcribing...
state, cancel, and a voice message chip whose transcript arrives later
and goes with the send. Stories show the form recording each kind, a
recorded chip, and sent voice messages with the transcript, pending,
and computed peaks.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the old audio recorder and proxy realtime transcription in dev
The voice engine replaced AudioRecorder, so it goes. The dev server now
proxies WebSockets on /v1, so realtime transcription works under vite dev.
The VoiceCapture test waits for the tone to be recorded instead of a fixed
700 ms, which a loaded test run could miss.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: dictate with a model that is already loaded when the chat model cannot hear
The Transcription model setting's Automatic choice now falls back to a
router model with audio input that is already loaded, so dictating with a
text-only chat model works without picking a model and loads nothing.
A picked model still wins, and the empty value is a named constant.
Idea from upstream PR 27567, by NickM-27.
Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: fill the recording waveform, and open voice message transcripts
- The live waveform fills the row from the start: a recording shorter
than the row is stretched over all its bars, a longer one scrolls with
the newest level on the right. No dotted placeholder.
- A voice message's transcript is open under its clip, in the chat form
while it comes in and in the sent message, with the toggle beside it
to fold it.
- Settings > Voice > Collapse transcripts (off by default) keeps them
folded instead, opening on hover, keyboard focus or a tap.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 2a93f2f948ed976e40f693ce5d3acf1168284b05)
* ui: record voice without ducking other audio, from the first word
- Voice input asks for the raw microphone signal: no echo cancellation,
noise suppression or auto gain. Echo cancellation made the system treat
a recording as a call, which lowers or interrupts other playback, and
auto gain faded the first words in.
- One AudioContext serves every recording, with the capture worklet
loaded once, suspended in between and, where the browser allows it,
without an output device, so recording never touches the speakers.
- The chat form prepares it when it gets focus (no microphone, so no
recording indicator), and a mouse starts recording on press.
- A recording starts counting once audio flows: until then the bar says
Starting... instead of running the timer. Development builds log how
long each step of a start took.
- Settings > Voice > Microphone: Automatic (the system default, or the
built-in microphone when the default is a Bluetooth headset, which
would switch to its low-quality call mode), System default, or a
listed microphone.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 76643e7210d21617d734fdfde677e84af7273153)
* ui: show voice text live in the message box, and follow transcripts as spoken
- A model that hears but has no live ASR (e.g. LFM3-Omni) now gives text
while you record: the realtime client commits the audio piece by piece,
at the pauses the levels show, one piece at a time, and the final
commit takes only what is left.
- A voice message's live text shows as the empty message box's
placeholder. Clicking into the box makes it text to edit there, and
what is said next follows it. On stop the text leaves the box and goes
with the clip; when it was edited, the edit is the clip's transcript.
- Llama UI gains VoiceSpokenText and spokenLength(): spoken words in the
full foreground colour, which they keep, the rest muted. A clip's
transcript follows its playback (estimated over the parts with speech,
as there are no word timings), and live text brightens as it arrives.
- The recording bar has a send button that stops and sends in one go.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit ce773017f902f8a5690ab9ecca368cecb6427422)
* ui: let the capture test start with silence
A capture starts with its first audio block, which can come before a
stream's first sound, so the steady tone is checked from where it begins.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit a41ee5337b83fef685a5cbaff176925f1647ce9d)
* ui: show live voice text sooner, and keep silence from becoming "yeah"
- A model that transcribes per commit (LFM3-Omni) gets shorter pieces:
cut at a 0.35 s pause after speech, or at the next quiet moment once a
piece reaches 3 s, so the text follows the speaker within a few
seconds in dictation, the message box and voice mode.
- When the recording stops, the audio after the last piece is committed
only if speech was heard in it. LFM3-Omni transcribes a commit of pure
silence as "Yeah." (or "はい。"), which ended up at the end of the text.
A recording whose only piece is that rest still commits it.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 43e52cbab0fba9d8e84b9491890a406e3c6c90f3)
* ui: transcribe a recording cut at pauses once more as a whole
Live text comes piece by piece, cut at the speaker's short pauses, and a
model reads each piece as a sentence of its own: "I went. To the store."
When the recording ends, one pass over the whole clip gives the final
text, punctuated by meaning, in dictation and in voice mode alike. The
pieces' text stays the live text until then, and is kept when that pass
fails or the recording is longer than two minutes.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 359dd9632e6c88000a1d23c2575bd1e6fcdd9ec1)
* ui: give live text over HTTP for models without a realtime session
A model that hears but cannot hold a realtime session, such as
Qwen3-ASR, was transcribed only after the recording, so dictation
showed no text while speaking. The recording is now cut at the
speaker's pauses as with a realtime session, and each piece sent to
/v1/audio/transcriptions on its own, one at a time, its text joined
live. The audio is kept from the start, so what is said while the
socket answers goes with the first piece. The whole clip is still
transcribed once more at the end, and the pieces' text is kept if
that fails. A server a websocket cannot reach (an API key, a remote
server) gets live text the same way.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say when the browser processes the microphone despite the raw request
The browser keeps one processing setting per microphone: while a call,
another tab or voice mode has it open with echo cancellation, a
dictation that asked for the raw signal is processed too, sounds
distant, and the system may lower other sound. The capture now warns
when it gets processing it did not ask for, and its development log
names the microphone, its rate and what processing is on.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the microphone beside the stop and send buttons
While a reply streams, or once the message has text, the microphone no longer
gives way to the main button: it stays beside it as a quieter, secondary
button, so more can be dictated without clearing the message or waiting for
the reply.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show the voice components' stories with Llama UI
VoiceWaveform and VoiceTranscriptReveal are Llama UI components, so their
stories move to the library's Storybook, under Llama UI/Voice, with the demo
audio they play. Llama Chat's Storybook shows them there too, and its own
voice stories take the demo audio from the library. VoiceWaveformLive stays
with Llama Chat, as its demo records through the chat form's microphone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a noise suppression setting for voice input
Recordings stay raw by default. Settings > Voice can now switch the
browser's noise suppression on; echo cancellation and automatic gain stay
off either way, as they interrupt other sound and fade the first words in.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep live voice text coming when the room is never quiet
Pieces were cut only at a quiet moment, so with music or talk in the room
none was ever cut and no text showed until the recording ended. A piece half
as long again as its longest length is now cut where it is.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add echo cancellation and automatic gain settings for voice input
Settings > Voice now has a switch for each kind of microphone processing the
browser offers. All three stay off by default, so recordings are raw; each
one says what switching it on costs.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: put the microphone after the stop button, filled like it
While a reply streams, the microphone is the last button of the row and
looks like the stop and skip buttons beside it, instead of a bare icon
before them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say when dictated text is being refined, and send it refined
After a recording with live text, the whole clip is transcribed once more
so pauses do not end sentences. The form now says so while that pass is
out, and a send asked for meanwhile waits for it, so the message goes out
with the refined text rather than the pieces.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the send button in place and put the microphone beside it
The send button no longer gives way to the microphone on an empty
message. The microphone stands before it as a quieter, tertiary button,
with the same tooltip, warning and recording states as before.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let a recording be sent while a reply streams, once it has words
While a reply streams, the recording row now shows its send button as
soon as the first transcribed words are in the message, so the
recording no longer has to be stopped first. The microphone also stays
beside the send button once a voice message is attached.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: raise quiet speech for transcription, and say when it is too quiet
A quiet microphone gave the model near silence to transcribe. A new
Voice setting, Boost quiet speech (on by default), raises speech to a
level speech models hear well, up to eight times louder: the audio
streamed for live text as it goes, and the whole recording in one pass
when it stops, so its first words get the same gain as the rest. Only
speech is measured, never the silence around it, loud speech is left
as it is, and a soft limiter keeps raised peaks from clipping. When
speech stays quiet even at the highest gain, the recording row says so.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the voice engine free of app code
The engine in lib/voice imported the microphone messages from the app's
utils and the WAV type from its enums, the only two things keeping it
from moving into a package of its own. The microphone checks and
messages move into the engine (voice/support.ts), next to the capture
that needs them; the uploaded-file helpers stay in the app's utils.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: start voice sessions from one voice service
How a recording reaches the server (routes, API key, whether the
realtime socket can be used) and how the Voice settings become capture
and transcribe options were built inside the chat form's voice hook,
and copied into every other feature that records. VoiceService builds
them in one place: a feature asks it for its session options and adds
only its own callbacks.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tell speech from the room in one place
The pause detector, the speech gain and (in voice mode) the listener for
speech over a reply each followed a noise floor with their own numbers.
A SpeechDetector now does it for all of them, in dB, with one set of
settings in VOICE_SPEECH.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let any text field take dictation
Writing dictated text into a field (at the caret, around the user's
typing, replaced by the whole recording's text when it comes) moves out
of the chat form's hook into a plain createDictation(). The hook keeps
what is the chat form's own: voice messages, toasts and the hint.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move the recording row into Llama UI
VoiceRecordingBar draws the row shown while recording (cancel, waveform
and time, send, and a caption or warning under it) from data alone, so
voice mode or a notes page can show the same row. The chat form passes
its labels and its microphone button as the stop button.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the voice engine's Svelte recorder behind its own entry
VoiceRecorder, the one part of lib/voice written with runes, moves to
lib/voice/svelte, so the rest of the engine is plain TypeScript any app
can use without Svelte.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move voice input into its own package, Llama Voice
The voice engine (capture, the speech signal, transcription over the
realtime socket or HTTP) and dictation into a text field move from the
app to ui/lib/voice as @ggml-org/llama-voice, so another app, such as a
notes app, can use them. The Svelte recorder is its ./svelte entry.
Llama Chat compiles the sources, as it does Llama UI's, and keeps its
server and settings wiring. The package has its own type check, tests
and build, and a CI job for them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show which model does what in voice input
The model picker and the Models manager mark each model that hears:
Dictation on the one that transcribes dictation, Speech to text on
transcription models such as Qwen3-ASR, and Voice on chat models that
take voice messages. A loaded model kept only for dictation sits apart
from the chat models, a model can be made the dictation model from its
row, and Settings > Voice groups the transcription models apart from
the chat models with audio input. A voice store now resolves what the
microphone does, for the chat form and the model lists alike.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: warn of a quiet microphone inside the recording row
The too-quiet line under the waveform made the form grow and jump. The
warning now stays inside the row: the waveform turns the warning colour,
an icon sits by the time, and the advice opens above the row in a
tooltip for a few seconds, then on hover or focus. Screen readers still
hear it when it appears.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the microphone while the pointer is on its button
The first seconds of a recording could come out quiet while the
microphone settled, and the too-quiet warning showed for them. With the
pointer or the focus on the microphone button, the microphone now opens
ahead of the press (only when the page may already use it, so nothing
asks), and the recording takes it at once; it is let go shortly after
the pointer leaves without a press. The warning also waits until the
speech has stayed too quiet for two seconds.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tidy voice input after review
Fix a microphone left open when a recording is cancelled while it starts
or the pointer leaves while it opens, a capture that is still letting go
silencing the next one, and live text stopping after a realtime error.
Share the dictation model choice, the transcript lookup, base64 decoding,
PCM joining and stopping a stream instead of repeating them, and export
only Llama Voice's public API.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin Llama Voice's dev dependencies to Llama UI's versions
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let demos speak into a synthetic microphone
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the microphone for voice input, off the audio files entry
The attach menu's Audio Files entry used a microphone, which now sits
beside the form's own microphone button. The microphone stands for
recording and dictation only; a model's audio modality is headphones, and
the audio files entry is a file with headphones.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: set voice input on the slimmed design tokens
Voice input's components use the type roles, the one corner scale and
shadcn's utility names for the neutral colours, as the rest of Llama UI
and Llama Chat do since the design tokens were slimmed down.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name voice states, platform strings, patterns and tunings
Addresses the review on the voice PR:
- Llama Voice: VoiceCaptureState, VoiceRecorderStatus, RealtimeState and
RealtimeUnsupportedReason are enums (src/enums.ts, exported); DOMException
names, AudioContext states, transcription form fields, socket protocols,
device kinds, worklet messages, error messages, regexes and tuning values
are named constants.
- Llama UI: VoiceTranscriptStatus, VoiceWaveformSize and
VoiceRecordingBarStatus are enums; KeyboardKey, MouseButton and PointerType
join its UI enums; the voice components' sizes, labels, timings, selectors
and patterns move to constants/voice.constants.ts. The voice unit tests
move from src to tests/unit.
- App: ChatFormVoiceState is an enum, the recording bar's status comes from a
map, and the voice code uses the enums above, BooleanString, HEADERS and
new voice constants in place of inline strings, regexes and numbers.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep all of Llama UI's tests under tests
The Model UI formatting spec moves to tests/unit and the button spec to
tests/client, next to the voice specs; the test projects now read only
tests/unit and tests/client.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a transcript's words without the model's own format
Qwen3-ASR names the language before its text (language French<asr_text>...) and
VibeVoice-ASR answers with timed segments as JSON, which the server passes on as they
are. The voice engine now takes both off, over HTTP and over the realtime socket, also
while segments still stream, so the message box and the clip show the words alone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: dictate when a speech-to-text model is picked for the conversation
A speech-to-text model has audio input, so on Automatic the microphone recorded a voice
message for it, which such a model has nothing to answer with. It now dictates, as the
model's badge already says it is for.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say why a live-only model cannot transcribe behind an API key
A browser websocket cannot send the API key, so the realtime socket is left out on a
server that needs one, and a model that only transcribes over it (Voxtral Realtime)
answers the HTTP route with a 501. The clip and the dictation toast now say so instead of
only that transcription is unavailable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pick the dictation model with a microphone on its row
The model that transcribes dictation was marked with a filled Dictation badge. It is now
marked by a red microphone among the row's actions in the model picker, beside the heart,
and every other model that hears shows the same microphone on hover: a click makes it the
transcription model, and a click on the chosen one hands the choice back to Automatic. The
badge keeps saying what kind of hearing model a row is, and the Models manager marks the
dictation model with a small microphone beside its name.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: search message content and Discover models from the search dialog
The search dialog matched chat titles only. It now also scans what was
said in each chat (content, reasoning, tool calls and results, attachment
names and text), newest chat first, stopping at the first match per chat
and after 20 chats, and shows a snippet around the match. Ids, timestamps
and base64 payloads are not searched.
With Discover on, it also lists matching catalog models and, once typing
settles, Hugging Face repos; picking one opens Discover on that repo.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the search dialog one width whatever it lists
The shared dialog sizes itself to its content from md up, so the search
was narrow while empty and widened as results came in. It now takes the
full max-w-2xl there, less a margin on narrower windows.
Assisted-by: Claude (Opus 5.5 Medium)
The unscoped llama-ui on npm is a 0.0.1 placeholder. After the scoped
publish, the workflow renames the same build and publishes it again at
the same version, with a readme note pointing at the scoped package.
Assisted-by: Claude (Opus 5.5 Medium)
llama-tts built the mtmd context from mtmd_context_params_default(), which hardcodes 4 threads, and never copied n_threads over, so -t never reach the mtmd side.
Copy params.cpuparams.n_threads into mtmd_params.n_threads, as mtmd-cli and the server do; with no -t this now follows the resolved default.
Verified: -t 1 now runs single-threaded, -t 4 is unchanged, and the produced WAV is identical.
Assisted-by: Claude
* fix(server-bench): prevent float bin-count TypeError in histogram generation
Symptom: benchmark crashes while generating gen_rate histogram.
Root cause: np.ceil(...) returns numpy.float64, and that value was used where an integer count/shape argument is required.
Fix: cast ceil result to int before +1, ensuring bin count is an integer.
Result: histogram path completes and gen_rate.png is generated reliably.
* Update scripts/server-bench.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ui: load highlight.js on first use
highlight.js and every lowlight language (~900 KB raw) moved out of
Llama Chat's first load into a chunk that loads with the first code block.
- @ggml-org/llama-ui/code: highlightCode stays synchronous. Until
highlight.js has loaded it returns the code escaped as plain text and
starts the load; a $derived or $effect that called it runs again once
it has loaded. New loadHighlighter() and isHighlighterLoaded().
- Markdown: rehype-highlight runs through a lazy rehype step, like
KaTeX's: the first message with a code block waits for it, every
other one renders in the same pass.
First-load JS of the chat page: 3.47 -> 2.56 MB raw, 1.05 -> 0.75 MB
gzip, 0.84 -> 0.60 MB brotli.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load each highlight.js language on first use
highlight.js now loads one language at a time instead of all ~190 at
once with the first code block: every language is its own chunk, and
the core loads with the first one.
- @ggml-org/llama-ui/code: highlightCode looks a language or alias up
in a generated table (scripts/generate-highlight-languages.mjs), loads
it with the languages it embeds (CSS and JavaScript inside HTML, say)
and returns plain text until then; a $derived or $effect that called
it runs again once it has loaded. Auto-detection picks among lowlight's
37 common languages, loaded the first time something auto-detects.
New loadHighlightLanguages(), isHighlightLanguageLoaded(),
resolveHighlightLanguage() and getLoadedHighlightLanguages() replace
loadHighlighter() and isHighlighterLoaded().
- Markdown: rehypeLazyHighlight highlights with lowlight's core and the
languages a message uses, waiting only for one it is the first to use.
rehype-highlight is no longer in the pipeline, since it bundles
lowlight's common languages.
First-load JS of the chat page stays at 2.58 MB raw, 0.76 MB gzip,
0.61 MB brotli: no highlight.js code, only the 14 KB language table.
The first code block now loads its language (1-12 KB for most) and the
20 KB core instead of 900 KB.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the code-block class names and shell language as constants
Assisted-by: Claude (Opus 5.5 Medium)
A printable key pressed on the chat screen while no field, editor,
dialog or menu has focus now moves focus to the chat form, and the
character lands there.
Assisted-by: Claude (Opus 5.5 Medium)
* model: load `ffn_gate_up_exps` scales
The generic scale pass loads per-expert scales for ffn_gate_exps,
ffn_up_exps and ffn_down_exps but not for the merged ffn_gate_up_exps.
Load ~ffn_gate_up_exps_s there~ and expose it as ffn_up_exps_s ~when the layer
has no separate up tensor,~ since build_moe_ffn applies up_exps_s to the
merged gate_up output. Inkling 2 now passes the expert scales to
build_moe_ffn like the other MoE models.
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-model.h
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama : move the indexer key rotation choice into the model classes
The KV cache forced a Hadamard rotation for the lightning indexer keys
based on a list of architectures. The virtual llama_model::indexer_k_rot()
now returns true for the same models. DeepSeek4.1 inherits from DeepSeek4
but was not in the list, so it overrides it back to false.
* llama : rename indexer_k_rot() to lid_k_hdmd()
* ui: drop component tokens that only alias a semantic token
A component token now exists only for what a component decides itself:
a value no semantic token has, or one that only this component changes
between themes. Components that take a semantic token as it is read it
directly, so 88 of the 118 component tokens go, with no change to what
is drawn.
Also drops semantic tokens that repeat another one (status-* is fill-*,
elevation-tooltip is elevation-surface), the unused info lamp tokens and
14 primitives nothing reads, and adds fg-soft for the label of an option
that is not the picked one (tabs, toggle group).
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a short summary on top of DESIGN.md
The rules most changes need, in one screen: components, colour, type,
shape, sizes, surfaces, tokens and no arbitrary values. The sections
below keep the detail.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: derive design tokens by rule and drop the alias layers
Semantic tokens are either a pick (a primitive per theme) or a derivation,
mixed from another semantic token by a fixed rule and declared once: hover
fills, status surfaces and lines, dark shadows (--elevation-strength), the
middle of a matte edge, a lamp's rim and glyph.
Primitives keep colours only. shadcn's colour names stay as Tailwind
utilities and no longer exist as variables; neutral semantic utilities that
duplicated them are gone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: set all text through type roles and move off-scale values onto the scale
Tailwind's text sizes and bracketed sizes become the type roles, with a new
text-caption-sm (10px) for tags and counts. Body and caption roles no longer
set a weight, and line heights are written as Tailwind writes its own, so the
swap from text-sm and text-xs changes no pixel. A unit test keeps old sizes
from coming back.
Arbitrary values with a scale equivalent use it (z-20, max-w-4/5, min-w-32),
and the two easing curves repeated across the app are ease-out-quint and
ease-sheet. DESIGN.md is updated and regenerated.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: bump @ggml-org/llama-ui to 0.5.0
Removed and renamed tokens are breaking for an app that reads them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a display type role for the biggest headline
The greeting on a new chat keeps its 30px from md up, now as text-display
instead of a Tailwind size.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: one corner scale, one set of shadows and three more derived tokens
Six corner roles named values Tailwind's rounded-md to rounded-4xl
already have, and components used the Tailwind names nine times out of
ten. The roles are gone; the whole scale is computed from --radius.
Tailwind's shadow sizes were restated next to the elevation tokens and
did not follow the theme, so they were five times fainter in dark. The
five uses take elevation tokens and a unit test keeps both from coming
back.
A button group's seam is the button's own text colour at 25%, the dark
status text is the dark status fill, and the pressed shadow scales with
--elevation-strength like the inset one.
Assisted-by: Claude (Opus 5.5 Medium)
The model modality icons match the ones the Llama apps use: glasses for
vision, headphones for audio. The attach menu's list of what the model
takes reads the same icons instead of its own microphone and picture.
Assisted-by: Claude (Opus 5.5 Medium)
graph_max_nodes() picked the nodes-per-tensor factor from lists of
architectures. Each model class now gives it through a new virtual,
llama_model::graph_nodes_per_tensor(), which defaults to 8. The
architectures that need 32 or 64 override it in models.h, and DFlash
keeps its hparams-based rule in dflash.cpp.
LoRA adapter nodes are now added for every architecture, not only for
the ones on the default factor.
* ui: move the reasoning effort into the model picker
The chat form's row no longer holds the reasoning button: the model
picker ends with a Reasoning row, in place of Manage models, that shows
the level in use and opens the levels under it. A phone's model drawer
has the same row. A desktop server with a single model, whose selector
opens the manager instead of a picker, keeps the button beside it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a reasoning effort that is not the default on the model selector
Next to the model on the closed selector, a set level shows as its word,
and reasoning that is off as a small crossed-out lightbulb. At the
default, which leaves it to the model, nothing more shows.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the model picker's footer in place while it opens in Safari
WebKit pins a bottom-sticky element against the scaled box of a parent
that zooms in, so the Reasoning row rode up over the model list until
the opening animation ended. The desktop picker now scrolls its search
and list in a box of their own, with the footer under it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep Pele diagrams inert
Pele turns a node with `click ... href` into a live link, where Mermaid's
strict security level left diagrams without links. Unwrap every link in
Pele's output, keeping its children, so the inline block, the live-preview
editor, the preview dialog and the downloaded svg stay inert. The Diagrams
story's flowchart now carries click directives and its play test checks
that no link or href is left.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open diagrams full screen, zoom around what is on screen
The diagram preview dialog fills the window on every device. The preview
places the diagram from its top left corner, so a zoom keeps a screen point
fixed: the wheel and a trackpad pinch zoom around the pointer, the buttons
and +/- keys around the middle of the screen, wherever the diagram has been
panned to. Two-finger pinch, double-click and arrow keys work too. The
diagram opens fitted and keeps fitting until panned or zoomed; Reset view
fits it again.
Adds MermaidPreview and DialogMermaidPreview stories with play tests and a
unit test for the zoom math.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a diagrams trace to the demo fixtures
Assisted-by: Claude (Opus 5.5 Medium)
The MoE cache adds a slot lookup to every MoE layer, and with
--moe-cache-cpu-share a second expert path, but graph_max_nodes() did
not account for it. Models on the default 8*n_tensors budget with few
tensors per layer (cohere2moe: 443 tensors, 49 layers) overflow the
graph context when the cache is combined with a quantized KV cache
(K/V rotation), aborting in ggml_new_tensor with GGML_ASSERT(obj_new).
Fixes#32584
Assisted-by: Claude Opus 5.5
* tests: add CUDA-only MUL_MAT/MUL_MAT_ID cases with results above the fp16 range
IQ1_M weights have no MMQ kernel, so once the batch is bigger than the MMVQ
limit the path is dequantize + cuBLAS. These cases produce results around
1.3e5, past the fp16 max of 65504, so an fp16 result buffer turns them into inf
while the CPU reference stays finite. They are only added for the CUDA backend.
Vulkan deliberately accumulates quantized matmuls in fp16 at default precision
and would fail them by design.
* CUDA: use an fp32 result for quantized src0 in the cuBLAS fallback
On the F16 compute path, quantized weights went through cuBLAS with fp16
accumulation and an fp16 result buffer that was converted to fp32 afterwards.
Anything above 65504 became inf and the next layer turned into NaN (see #30951:
gpt-oss-120b with IQ1_M experts at -ub 512). Set prefer_f32_output for
quantized src0 so the GEMM still uses fp16 operands but accumulates and stores
in fp32, the same mode already used on Volta, RDNA4 and CDNA. Every other
quantized path already accumulates in fp32.
Diagnosis and fix by denizyuret in #30951.
* CUDA: keep the MUL_MAT_ID precision in the per-expert fallback
The per-expert dst_slice kept op = GGML_OP_NONE, so ggml_prec_get_acc returned UNDEFINED and a requested precision was ignored. Set the op to GGML_OP_MUL_MAT. Also revert the fp32 result change and remove the tests.
---------
Co-authored-by: JonSnow1807 <JonSnow1807@users.noreply.github.com>
On Adreno, subgroupAdd() after divergent code returns wrong sums. The
fused ADD+RMS_NORM partial-sum path hits this, so the decode-step norm
is garbage and models emit stray tokens after long prompts.
Clear subgroup_arithmetic for the Qualcomm proprietary driver, which
also turns off add_rms_fusion and the other subgroupAdd paths.
test-backend-ops on Adreno 850 and X2-90: ADD_RMS_NORM 20/25 -> 25/25,
RMS_NORM_MUL_ADD 33/45 -> 45/45. llama 3B Q4_0 on Adreno 850: pp512
and tg128 unchanged.
load_arch_hparams multiplies n_hrm_layers_per_stack, n_hrm_h_cycles and
(n_hrm_l_cycles + 1) into a uint32_t, then asserts the product equals
block_count.
Any factors whose true product is block_count + k*2^32 still pass the
assert; 1073741825 * 2 * 2 = 2^32 + 4 compares equal to block_count = 4.
load_arch_tensors iterates the same three values in int and writes past
the end of the block_count-sized layers vector.
* opencl: support int8 KQ for DK = 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
* opencl: allow query chunk up to 96 for int8 KQ
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
* opencl: extend `build_program_from_source` to allow non-fatal failure
Assisted-by: Codex
* opencl: add int8 KQ for decomposed FA with DK <= 256
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA)
* cuda : SET_ROWS from BF16 into quantized types and F8_E4M3
* cuda : fuse BF16 rope into a BF16 cache
* tests : trim the BF16 rope cases to one per code path
- Allow cls.output.weight to be absent from the mmproj. It is an optional
Qwen3 reranking head that the YuE2 NAR decoder never uses.
- Without this fix, mtmd_init_from_file() fails with
"tensor cls.output.weight is missing from the mmproj", preventing
audio generation from starting.
* ggml-openvino: round the GPU KV length up to a power of two
Review notes (follow-ups, not blocking):
- convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
copies it into the ov::Tensor. Writing the padded rows straight into
padded.data() avoids the second allocation and copy (up to 16 MB per
call at a 16K context).
- The padding is skipped when the mask is not contiguous, and the KV
inputs are then sliced to more rows than the mask has columns. Masks
are always contiguous today; a GGML_ASSERT would make that explicit.
- compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
is_stateful_enabled() is local to ggml-openvino.cpp.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Remove solution C and keep A as default
* ggml-openvino: match the output count before reusing a compiled NPU prefill
The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.
* ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding
* ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix
- compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
ran before the new one and still exported result_norm, so the
result_norm exclusion had no effect. Keep only the new block.
- compute_node_dynamic_dims checked is_activation_input twice in a row.
Remove the second copy.
- process_view_input_new created the dynamic Split on split_dim. With
stateful execution the input drops the leading size-1 axis, so use
split_axis, which already accounts for the rank difference.
* ggml-openvino: hash the output flags into the model-cache fingerprint
The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.
* ggml-openvino: support f16 SCALE and f16/bf16 TRI
test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.
SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.
* ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
layer
The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.
* enable mamba based models on OV CPU-GPU
* revert changes in tests
* docs : add Falcon-H1 to the OpenVINO validated models
Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.
* ggml-openvino: simplify the SSM_SCAN build check and clean up
- Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
always provides the op. Drop the no-op LANGUAGE CXX source property.
- Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
with -Wextra-semi, which fails builds with fatal warnings.
- Add ggml_openvino_is_static() for the repeated NPU or
GGML_OPENVINO_FORCE_STATIC check.
- Share the f16 and f32 KQ mask padding code.
- Drop the redundant inline on in-class static functions.
---------
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
* cuda: add scatter for batched tensor writes
* update test case to mirror get_n test case
* save restore timing logs
* refactor code into batch-copy.cu file
* Adds initial scheduler stress tests
* Adds new sleep op to simplify scheduler testing
* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321
* Adds experimental vulkan GGML_OP_SLEEP
* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.
* Cleanup: Removes CUDA specific comments and nvtx-markers
* Fix: forgot to add sleep specific test case
* Fix oversights identified by CI
* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior
* Fix: Early exit in CPU-only settings (cause for graviton CI fails)
* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds
* Exclude ACCEL devices from new tests due to incompatibility
* Fixes vulkan test failure in llvmpipe
* Adds asserts, improves comments and some checks
* Fixes nitpicks in white-space and variable naming
* Adds comments that certain vk drivers could time OP_SLEEP incorrectly
* Drops limitiation to CPU/GPU in test-sleep
* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)
* Implements feedback for OP_SLEEP to work in place (no copy)
* Apply non-code suggestions from JohannesGaesslers code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail
* Make clang-tidy happy
* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)
* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name
* Drops unnecessary shader_clock.comp test and guards
* Replaces sched_ prefixes with helper_ prefix
* Moves test suite to new RAII types for better memory management
* Adds sleep option to test_chain_all_backends
* Adds sleep option to first-run arm of the y-shaped test
* Fix switchcase order for legibility in stress_test_dag
* Simplfies stress_test_linked_list via double for-loop
* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)
* Fix RPC Proto patch due to GGML_OPS bump
* Adds ggml-cpp.h treatment to test-sleep.cpp
* Adds TODO pointers for subsequent additional tests.
* Apply suggestion to ggml.h
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)
* Apply batched suggestions from gg code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Completes callsite change of gg-suggestion
* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : give Inkling2 its own relative-position attention input
The Inkling2 graph built its flash attention mask from the hybrid iswa
input and passed it to build_attn through a new kq_mask argument. That
argument let a caller pair its own dense mask with the kv_rows of
llm_graph_input_attn_kv_iswa, whose mask follows the per-sequence cell
lists whenever kv_rows is enabled.
Add llm_graph_input_attn_rel_pos instead. It owns dense KQ masks and the
query-to-cell distances, fills both from the same cells, and its
build_attn overload adds the per-head bias to its own mask and never
passes kv_rows. The iswa build_attn is back to its previous signature;
its body moves to build_attn_iswa_impl, which both overloads call.
Inkling2 now folds the per-query sink into the bias with and without
flash attention, so build_attn_mha no longer needs the per-query sink
path.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* inkling2 : make the expert weight scale and final logit softcap optional
Load both keys as optional and skip them when they are zero, as other
models do, instead of rejecting zero at load.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : build kv_rows masks for the hybrid iswa memory
llm_graph_input_mem_hybrid_iswa checked its masks with
can_reuse_attn_mask, which expects kv_rows whenever they are enabled,
but built them dense. With a unified cache and flash attention the
check always failed, and hybrid iswa models (LFM2, LFM3, Inkling2)
never reused a graph. Build the masks with build_attn_inp_attn_mask,
like the plain iswa input.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : make the relative-position input a hybrid iswa input
Address review: llm_graph_input_mem_hybrid_iswa_rel derives from
llm_graph_input_mem_hybrid_iswa and owns the attention and recurrent
inputs, so Inkling2 builds one memory input instead of two. Its
attention input is built with dense masks, can_reuse checks them as
dense, and its build_attn asserts that the mask has no kv_rows (TODO:
a kv_rows variant). build_attn_iswa_impl is now a static helper.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : compose the hybrid iswa rel input, format inkling2
Address review: llm_graph_input_mem_hybrid_iswa_rel owns its attention
and recurrent inputs like the other composite inputs instead of deriving
from llm_graph_input_mem_hybrid_iswa.
Format src/models/inkling2.cpp like the other model sources: grouped
and aligned key and tensor loading, descriptive per-layer names, blank
lines and section comments between the steps, and cb() names for the
intermediate tensors. The graph is unchanged.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
llm_graph_params no longer carries hparams, prec_policy and hdmd, since the model reference already provides them. llm_graph_context sources them from the model and keeps prec_policy as a reference.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal: add a tensor API conv_2d kernel and split long reductions
The conv_2d matmul kernel runs on simdgroup matrices only, and gives
each threadgroup the whole reduction of its output tile. On M5 this
leaves the tensor cores unused, and convolutions with few output
positions and a long reduction (IC * KW * KH in the thousands) run a
handful of threadgroups on an otherwise idle GPU.
Add kernel_conv_2d_mm_tensor for devices with the tensor API: like
kernel_mul_mm, it gathers the patches of 64 output positions into
threadgroup memory as the left operand and reads the kernel rows
from device memory as the right operand. Devices without the tensor
API keep the simdgroup kernel.
Both kernels can now split the reduction until there are enough
threadgroups: each split writes its partial sums to a buffer placed
after the output, as flash attention does, and a small kernel adds
them in a fixed order, so the result stays deterministic. CONV_2D is
marked as an op whose allocation may exceed its output size.
test-backend-ops -o CONV_2D passes, with and without
GGML_METAL_TENSOR_DISABLE.
CONV_2D, F16 kernel, us/run on an M5 Max:
[19,19,256,16] k [4,4,256,4096] 19008 -> 5235
[16,16,128,8] k [3,3,128,512] 333 -> 124
[1536,1536,32,1] k [3,3,32,64] 12086 -> 6512
[33,1,1280,1] k [3,1,1280,1280] s2 659 -> 33
[2054,1,768,1] k [7,1,768,768] 3161 -> 699
* metal: walk the conv_2d patches incrementally, size splits by output
The patch gather of both conv_2d matmul kernels spent most of its
time on address math: 64-bit multiplies, divisions and four bound
checks per value. Walk the reduction index with 32-bit byte offsets
inside the image, updated by additions, with one unsigned compare per
bound; the host checks that an image fits. The tensor kernel no
longer gathers the reduction indices past the end of its split.
The number of splits now targets about 1M partial sums, with at least
3 reduction steps per split, instead of counting the tiles of one
kernel.
CONV_2D, F16 kernel, us/run on an M5 Max:
[19,19,256,16] k [4,4,256,4096] 4998 -> 4512
[1536,1536,32,1] k [3,3,32,64] 6721 -> 5354
[4096,1,1024,1] k [7,1,1024,1024] 2701 -> 2236
[2054,1,768,1] k [7,1,768,768] 1050 -> 871
with GGML_METAL_TENSOR_DISABLE:
[4096,1,1024,1] k [7,1,1024,1024] 9047 -> 8283
* nit: share the CONV_2D simdgroup count between host and kernel
kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per
threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel
constant and res.nsg, so the dispatch and the kernel cannot drift apart.
* metal: fall back to the generic conv_2d kernel for images past 4 GiB
The matmul conv_2d kernels walk the patches with 32-bit byte offsets
inside an image. Instead of asserting on nb13, compute the largest
offset actually read from ne/nb, which also covers strided views of
the input, and route images that do not fit to the generic kernel.
* cont : comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ui: import Lucide icons by path in modules that run in Node
Assisted-by: Claude (Opus 5.5 Medium)
* ui: register highlight.js languages once, from lowlight's modules
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load the pdf.js worker and nerdamer as files of their own
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw Mermaid diagrams with Pele
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin dependency versions and use one Svelte across the UI packages
Assisted-by: Claude (Opus 5.5 Medium)
* ui: build Llama Chat as split chunks
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load KaTeX and pdf.js on first use
Assisted-by: Claude (Opus 5.5 Medium)
* llama : fall back to buffered I/O on EIO
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Preserve bytes already read, resumes at the current file offset
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* vulkan: Implement stream-k MUL_MAT
Stream-k tries to split chunks of 256 K elements evenly across all SMs.
Workgroups that finish a tile write to dst, others write to scratch, and
there's a resolve shader similar to split-k.
The shader code is implemented for scalar+cm1+cm2, but only currently enabled
for cm2. The heuristic for enabling on coopmat2 is pretty simple due to the
large tile size (one workgroup per SM), but would be more complex for the tile
sizes being used for cm1, and it doesn't make sense for me to tune that on
NVIDIA devices.
Assisted-by: Codex
* address review feedback
The MMQ kernels computed the dst tile offset (jt*J*stride_col_dst) in
32-bit int. For a dst with more than 2^31 elements, column tiles that
start past INT_MAX/nrows wrap around and write out of bounds.
This happens with an all-logits output head, e.g. llama-perplexity
-c 32768 -b 32768 -ub 16384 on a 248320-vocab model: the q8_0 output
matmul has a 248320 x 16384 dst (4.07e9 elements) and aborts with an
illegal memory access.
Widen the offsets in mul_mat_q and the stream-k fixup to int64_t, and
add a (disabled by default, ~9 GB) test-backend-ops case.
Assisted-by: Claude Opus 5.5
n_mudd_taps comes from limite.mudd.tap_count, a scalar read separately
from the length-checked tap_indices array. mudd_tap_idx is a fixed
std::array<int32_t, 512> indexed at [i*n_taps] in both the tensor load
and the graph build, so a crafted GGUF with n_layer * n_mudd_taps > 512
reads past the array.
Reject that at hparams load; the one check covers both index sites.
* llama-bench : size the batch to the prompt for models without memory
Assisted-by: Claude Opus 5.5
* server : read up to 16 lfm3-d1 questions in one pass
Assisted-by: Claude Opus 5.5
* cuda: select cuBLAS precision by GPU SKU
Assisted-by: Codex
* cuda: fix remaining GB10 macro reference
* cuda: review comment addressed, store accumulation preference in device info
* Apply batched suggestions from code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* addressed Oliver's comment regarding initialize NVML once and install headers in Windows CI
* move precision selection helper into init
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* model:add Zaya1 8B (Zyphra)
- Added the support for Zaya.
- Fixed the SSM_CONV case, so the model runs on cuda without abort.
Please enter the commit message for your changes. Lines starting
* Made the following changes based on the bot reviewer feedback:
1) Added runtime asserts when loading wrong KV head size, unsupported conv kernel size.
2) Removed the static cast usage.
3) Fixed double load of tokenizer.
4) Updated res_scale_in to be input tensor.
* - Moved the res_scale_in back to layer 0 to fix the cuda bug
* i) Removed the EOS addition.Team agreed that updating Tokenizer is the right move.
ii) Fixed formatting
* Fixed the comment to make it clear
---------
Co-authored-by: varun <varun_subramanian@zyphra.com>
* ui: install @ggml-org/llama-brand from npm
Both packages take llama-brand 0.1.0 from the registry instead of a
codeload tarball, so npm 12 installs them without allow-remote, which
leaves Llama Chat's .npmrc.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin @ggml-org/llama-brand to 0.1.0
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat publish waits for the registry, bin paths without ./
The first publish of @ggml-org/llama-chat took about four minutes to show
up on the registry, so the 60 seconds the check after publishing waited
failed a release that had gone through. It now waits up to ten minutes.
npm normalizes "./bin/cli.js" to "bin/cli.js" on publish and warns that
the bin was "invalid and removed", which reads as if the command was
dropped. Both package.json files now give the paths as npm writes them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the registry check after llama-chat publishes
npm publish already fails when the registry refuses a package, and the
check after it only failed a release that had gone through, while npm
was still processing it. Nothing waits on the registry any more.
Assisted-by: Claude (Opus 5.5 Medium)
The tool permission card took the fields' border (white 30% in dark), which
outweighed everything else on the page; it now has its own token, white 18%
in dark, the focused chat form's edge. The split-button divider had the
keys' own rim colour (white 12% over a white 9% fill) and vanished between
them; it now has its own token, white 20% in dark. Light mode is unchanged.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide Models and MCP behind settings, off by default
Two new settings in General, "Show Models" and "Show MCP", both off by default, like "Enable Discover Models". The WebUI is often the first thing people new to local models see, and most of them don't need a models manager or know what MCP is.
"Show MCP" only hides the way to manage MCP; servers already added keep working. Turning MCP off for real would mean gating every place that reads the server list (health checks, tool groups, prompts, avatars). Anyone who already has servers saved gets MCP shown, so they don't lose the way to manage them.
With Models hidden, every entry point to the manager goes: the sidebar, search, the shortcut and "Manage models" in the selector -- it opens the same full manager. The /models page still opens by its URL, which apps link to. On a single-model server the selector chip used to open the manager; with it hidden, the chip is a plain label.
* ui: keep Models shown while Discover Models is on
Discover opens only from the Models page (its header button and the manager's row actions), so with Models hidden by default, checking Discover showed nothing. Checking Discover now shows Models too and locks its checkbox, the same way it already locks the Hugging Face Hub API one; unchecking Discover leaves Models as it is. Saved configs with Discover on get Models turned on when they load, so people who use Discover don't lose it.
* ui: run Llama Chat with npx @ggml-org/llama-chat
Makes @ggml-org/llama-chat publishable: drops private, adds bin,
files, exports, repository and a prepack build. Its CLI serves the
prebuilt dist on localhost (3000 by default) and forwards every other
request to a llama-server (--server, default http://127.0.0.1:8080),
with the headers llama-server sends for the UI. A thin unscoped
llama-chat package in alias/ forwards to it, so npx llama-chat works.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat revalidates service-worker.js and asks for a build before packing
SvelteKit 3 adds service-worker.js next to sw.js, and llama-server
revalidates it, so the CLI does too. The folder's .npmrc sets
ignore-scripts, so prepack never ran: it goes, and the publishing steps
build dist explicitly and check the packed file list.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat keeps llama-server's DNS rebinding and cross-origin guards
The CLI called llama-server with its own Host and dropped every Origin,
so a rebound host name or another page got past the checks llama-server
makes when --cors-origins is not *. It now drops only its own page's
Origin, passes other Origins and Sec-Fetch-Site through, and, when it
listens on loopback, answers only to localhost and loopback addresses.
`start` on Windows gets its empty title verbatim.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat build writes the app to a folder for any static server
`npx @ggml-org/llama-chat build [--out <dir>] [--force]` copies the
prebuilt app, without source maps, to ./llama-chat or the given folder,
so a plain file server can host it next to the CLI serving it. It
refuses a folder that is not empty unless forced, and the serving flags
and build flags refuse each other's command. The README shows serving
the folder with http-server forwarding the API to llama-server.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: publish Llama Chat to npm from CI when its version is bumped
ui-chat-publish.yml publishes @ggml-org/llama-chat and its llama-chat
alias the way ui-lib-publish.yml publishes @ggml-org/llama-ui: the
version is set by hand, and a commit that reaches master with a version
npm does not have builds dist, checks the packed files and publishes both
with trusted publishing. PRs that change the packages get a dry run.
npm-version-check.mjs takes the package folder, so both workflows share
it. ui/README.md says how releases work, and Chat's AGENTS.md says not
to bump versions unless asked to release.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: call the app Llama Chat in its README
The README's title and a few lines still called the app llama-ui or
Llama UI, the name of the library it is built on. They now say Llama
Chat, and the intro points to Llama UI in ui/lib/ui.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat connects to llama-server's default port, 9931
The CLI's --server default and its examples used 8080, which is not
where llama-server listens. They now use 9931, its default port, so
npx llama-chat reaches a llama-server started without --port.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: a shorter Llama Chat README that matches the code
The README grew to nearly 900 lines, and its tables of components,
hooks, stores and services had drifted from the code. It now keeps what
a reader needs: running it with npx and self-hosting, a short feature
list, development commands, the architecture (the line between Llama UI
and the app, a table of the layers and where they live, a flowchart and
one sequence diagram for sending a message), a few patterns worth
knowing, cross-origin isolation, how the build is embedded in
llama-server, the project tree and publishing. The folders now carry the
detail the tables used to. ASCII replaces the unicode arrows.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat serves only static files, without the API proxy
The CLI forwarded every request it had no file for to llama-server, from
loopback. With --host 0.0.0.0 that let any LAN client reach a llama-server
started on 127.0.0.1 as a local request, tools included, which llama-server
on its own refuses without an API key. The browser now talks to each
llama-server directly, so llama-server's own rules apply. --server is gone,
and anything that is not a file in dist is a 404, or a 405 for other methods.
Assisted-by: Claude (Opus 5.5 Medium)
The deploy failed on master with a 403: HF_TOKEN_UI_STATIC_OUTPUT is
scoped to the UI bucket and cannot write ggml-org/llama-ui-storybook, so
hf upload tried to create the Space. The upload now uses
HF_TOKEN_UI_STORYBOOK when that secret exists, and the job runs on
hf-jobs-cpu-upgrade instead of a GitHub-hosted runner. That image gets
python3-venv from apt for the hf CLI, as the other HF Jobs workflows do.
Assisted-by: Claude (Opus 5.5 Medium)
This commit adds a lambda to check if a specified byte is not a valid
continuation byte in a multibyte UTF8 sequence.
The motivation for this is to improve readability and avoid some code
duplication.
* vocab : extract bpe_rank loading into function
This commit extracts the BPE rank loading logic into a separate function
named `load_bpe_ranks`.
The motivation for this is that the came code is currently used by both
gpt2/hybriddna/whitespace tokenizer models and gemma4 and this avoids
the duplication of code to improve readability and maintainability.
* change signature of load_bpe_ranks
* vulkan: expand bf16 support for unary and binary ops and rms_norm
* cpu: add bf16 support for scale, clamp, leaky_relu, swiglu_oai, swiglu_clamp, tri and diag
* fix sycl fill supports_op condition
* restore CUDA op order
* ui: generate DESIGN.md's token sections from the CSS
Add scripts/design-md.js in ui/apps/chat (npm run design:tokens, and npm
run check:design in CI), which writes the colors, typography and rounded
sections of DESIGN.md's front matter from llama-brand's colors.css and
Llama UI's primitives, semantic tokens and font stacks. The front matter
had drifted: it gave Matte's surfaces as the light base and system-ui for
every text role. Twins are now written wherever a token resolves
differently in dark mode (-dark) or in Matte (-matte).
Describe the floating panel's rim as the CSS draws it, on the border,
and say which parts of DESIGN.md are generated and which are kept by hand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say the Storybooks show accessibility findings without enforcing them
Both Storybooks keep a11y test 'todo': under 'error' 98 of the stories
fail, most on colour contrast of the muted foreground and on nested
interactive controls, which need design changes. Reword the Uses tile
and record the reason next to the setting.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list every Llama Chat group in the Introduction's Sections figure
The figure said it was in sidebar order but left out Forms, Actions,
Badges, Misc, Servers and PWA. Its list moves to sections.ts, and a unit
test checks it against storySort in .storybook/preview.ts, which has to
stay inline there because Storybook parses it as written.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load the package's Llama Mono in both Storybooks
The library's Storybook declared Llama Mono a second time from
tests/stories/brand/fonts, a 72276-byte file that differs from the
76896-byte one the package ships, so its pages showed a Llama Mono that
does not ship. Drop both Storybooks' copies: theme.css already loads the
package's font.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: take Llama Chat's shared Storybook helpers from Llama UI
Llama Chat kept copies of the Editorial, Chapters, StateGallery,
StateGalleryItem, svx.d.ts and docs.css that live with the library's
stories. Its stories now import them as $llama-ui-stories/..., an alias
its Storybook config and tsconfig.json map onto ui/lib/ui/tests/stories,
the way it already compiles the library from source; docs.css and
svx.d.ts are referenced by path. The library's Editorial takes Llama
Chat's newer list numbering. An empty svelte.config.js next to the
stories keeps Llama Chat's svelte-check from loading the library's
vite.config.ts, as src/svelte.config.js does for the sources.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the library's Storybook to Llama Chat's versions
ui/lib/ui used ^10.6.2 with @storybook/addon-mcp at 10.6.2, while
ui/apps/chat pins 10.6.6, so one tree had two Storybooks. Pin the
library's Storybook packages exactly to Llama Chat's: 10.6.6,
addon-svelte-csf 5.1.6 and @chromatic-com/storybook 5.5.1.
Assisted-by: Claude (Opus 5.5 Medium)
* ggml-openvino: fix SWA mask detection
When the SWA cache size matches the full cache size, identify SWA
layers by the number of layers using each mask instead of cache size;
the larger group is SWA, and a tie leaves the layers unclassified.
* ggml-openvino: fix SWA KV cache write indices
Give SWA KV cache index tensors unique input names (a _swa suffix, like
the masks) to prevent them from being merged with the full-attention
cache indices.
* ggml-openvino: classify SWA layers only when the mask shows the window
Two attention masks do not imply SWA: DeepSeek sparse attention uses two
full-attention masks with the same name. When cache extents match,
classify the larger mask group as SWA only once its mask has fewer keys
per row than the other mask. Keep the classification per KV cache buffer
so new sequences reuse the same graph, and include swa_layers when
checking graph reuse.
* ggml-openvino: classify SWA layers from the first graph
Waiting for the window to appear in the mask caused a mid-sequence graph
recompile (~9 s for Gemma-4 at -c 1024 after position 512).
SWA classification already gives each mask group its own mask, KV write
indices, and attention size, which is correct whether or not the mask is
a sliding window. Stateful mask rebuild recovers the window from the
mask, or the causal mask for full attention. Classify the larger of two
mask groups from the first graph and drop the per-KV-buffer bookkeeping.
* ggml-openvino: mark SWA fallback as a model-level heuristic
Add a TODO above the equal-extent fallback to document that the larger
mask group is currently assumed to be the SWA group. Since stateless
graphs only require the two groups to remain distinct, this heuristic
may only be necessary for stateful execution and could potentially be
restricted to that path.
* ggml-openvino: derive attention sizes from the KQ mask shape
Dynamic CPU/GPU graphs previously passed n_seq_active and
attention_size(_swa) as separate inputs. Derive these values directly
from the KQ mask shape using ShapeOf + Gather inside the model. This
allows the GPU plugin to infer the Q reshape and KV slice shapes from
shape inference rather than runtime input values. Q's head size also
becomes static, enabling the fused SDPA kernel.
GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous
separate-input behavior.
The fused SDPA path also requires two supporting changes:
- Clamp the GPU mask to -30000. A key block that is entirely -inf for
a row produces NaNs in the fused kernel.
- Restate the head count and head size after the KV slice. The plugin
loses all slice dimensions when the end index is dynamic, causing
SDPA to fall back to the reference kernel.
Additional graph and quantization improvements:
- Skip the identity reshape of rows in SET_ROWS.
- Share a single Split for one-element view slices in dynamic GPU
graphs (e.g. Gemma-4 per-layer embeddings), replacing one
StridedSlice per layer.
- Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve
the file's native quantization for token_embd/output instead of
requantizing to Q8_0_C.
- Include both environment settings in the graph and model cache keys
and document them.
* ggml-openvino: only reuse Split outputs for exact chunk-aligned views
The dynamic GPU path reused a Split output for any single-element view
by deriving the chunk index from offset / stride. This produced
incorrect results for views that were not chunk-aligned or whose
remaining dimensions did not preserve the source strides, as seen in
test-backend-ops CONT with use_view_slice.
Restrict Split reuse to cases where the view exactly matches a split
chunk, meaning the offset is aligned to the split stride and the
remaining non-unit dimensions preserve the original source strides.
Gemma-4 per-layer embedding slices continue to use the shared Split
path.
* ggml-openvino: initialize ModelExtraInputInfo shape fields
Initialize the newly added `shape_source` and `shape_axis` fields in
the `ModelExtraInputInfo` brace initializer used by
`add_extra_inputs()`.
Without initializers for these fields, GCC emits
`-Wmissing-field-initializers`, causing CI to fail with
`LLAMA_FATAL_WARNINGS`.
* ggml-openvino: remove GGML_OPENVINO_REQUANT_EMBD
The option identifies embedding and output weights by their tensor
names, which are defined by core llama.cpp. Remove it from this PR;
preserving the file quantization of these weights will be addressed in
a separate change that identifies them by their roles in the graph.
* ggml-openvino: keep manual GQA attention 4-D
The manual GQA attention path reshaped Q, K, and V to 5-D so that
query heads sharing a KV head could broadcast against it. The GPU
plugin cannot fuse this graph: it materializes the attention scores
and copies the transposed K and V caches for every layer.
Instead, stack the query heads for each KV head along the sequence
axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask
accordingly. This keeps the graph 4-D from the MatMuls through the
Softmax, allowing the GPU plugin to fuse the attention into its SDPA
kernel and fold the K and V transposes into the fused operation.
Clamp the mask to -30000, matching the SDPA path, because the fused
kernel can return NaN when an entire block is masked with -inf.
Models where the K or V head size differs from the Q head size now
use the SDPA path, since the manual path assumes one head size.
* ggml-openvino: use a 4-D stateless KV cache on GPU
In stateless GPU mode, bind the entire KV cache as a
`[1, ctx, n_heads_kv, head_size]` tensor, with one token per row,
instead of binding the first `n_kv` rows as a
`[1, 1, ctx, n_heads_kv * head_size]` tensor.
Write cache updates with `ScatterUpdate` along the token axis, and
read the cache through `VariadicSplit` and `Transpose`. This allows
the GPU plugin's `StatelessKVFusion` to replace cache reads and writes
with in-place `StatelessKV` nodes, avoiding full-cache copies on every
decode step and repeated memory pool allocations as the bound cache
size grows.
Set `GGML_OPENVINO_DISABLE_KV_4D=1` to retain the previous layout.
* make embedding to have dynamic shape for OV GPU
* ggml-openvino: make the 4-D stateless KV cache opt-in
* add support for Q5_0 data type in OV GPU backend
* resolve CI test error
* ggml-openvino: fix -fa off and optimize grouped-query attention
With -fa off, llama.cpp stores the V cache transposed, with value
elements as rows and tokens along each row. The KV slicing code assumes
one token per row and binds the wrong data when n_kv < kv_size. Detect
transposed caches with GgmlOvDecoder::is_kvcache_token_rows() and skip
slicing for them.
For grouped-query attention, avoid repeating K and V for each query
head. Stack the query heads for each KV head along the row dimension
and let SOFT_MAX apply the scale, mask, and softmax so the GPU plugin
can fuse the operation with the MatMuls.
Read only the current layer's V-cache rows when layers use different
cache sizes. Reject -fa off in stateful mode with a clear error and
document that it requires stateless mode.
* ggml-openvino: fall back to stateless execution when -fa off
With `-fa off`, the graph writes the V cache in transposed form, which is
not supported by the stateful KV cache. Detect this pattern in
`compute_llm_params()` as a `SET_ROWS` into a KV cache through a `[1, n]`
reshape.
When stateful execution is enabled, log a warning and fall back to
stateless execution instead of failing graph conversion.
Update the `-fa off` note in `OPENVINO.md` to document this behavior.
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
* ggml-openvino: preserve flat KV cache layout when -fa is off
With -fa off, the 4-D KV cache layout introduced in #344 breaks MUL_MAT
attention for Phi-3.5 mini and Phi-4 mini, causing oneDNN to fail with
"could not construct a memory descriptor using strides".
Preserve the [1, 1, ctx, n_heads_kv * head_size] layout when the graph
writes a transposed V cache.
* ggml-openvino: extract Q5_0 weights when requantization is off
Q5_0 weights are requantized to Q4_0, but test-backend-ops builds naive
graphs, where requantization is skipped. Q5_0 then reached the normal
weight extraction, which had no Q5_0 case, so every Q5_0 MUL_MAT and
GET_ROWS op test failed with "Unsupported quantized type: q5_0".
Extract Q5_0 as symmetric i8 weights (q - 16) with one f16 scale per 32
weights, the same layout as Q8_0. Models are unchanged: they still
requantize Q5_0 to Q4_0.
* ggml-openvino: Optimize GDN output pre-scale for the NPU static path
The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero.
* Refine the code as review comments
1. Add envs desc into doc
2. Refine the code with removing duplicated env
* ggml-openvino: reuse graph bindings between decode calls
Reduce host overhead outside `infer()` from 0.5–1.1 ms to 0.14–0.38 ms
per token on Panther Lake.
- Reuse KV cache tensors and skip redundant port bindings.
- Use persistent USM host tensors for small GPU inputs and outputs.
- Defer graph-wide consumer lookup until after shape checks.
- Return model and compute params by const reference.
- Preserve tensor extras when memory, shape and type are unchanged.
* ggml-openvino: Add npuw dump envs for profiling
* Add the comments of new npu dump envs into doc
* ggml-openvino : ensure all leaves have unique names, remove input_srcs from graph key
* openvino: slice NPU prefill KV cache by active length
Add GGML_OPENVINO_NPU_KV_SLICE as an opt-in NPU prefill optimization. When enabled, static prefill keeps the graph's attention dimension at the active prompt/KV length instead of expanding it to the full context capacity, and the KV-cache input/output tensors are bound with a shortened context axis over the same backing storage.
This reduces the amount of KV cache and attention-mask data imported by the NPU plugin and lets the prefill graph plan smaller attention buffers for early prompt chunks. For example, with a 4096-token context and a 256-token prefill chunk, the first chunk can expose KV/mask dimensions at 256 instead of 4096, avoiding work over padded cache rows.
The trade-off is graph stability: NPU static shapes now depend on the aligned active attention length for prefill, so a cached prefill model is reused only when the attention sizes match. Decode keeps a fixed full-context graph to avoid recompiling on every generated token, and SWA/ring-cache cases still fall back to full-size binding when the live prefix is not a simple leading slice.
* openvino: use L0 host tensors for NPU buffers
Add GGML_OPENVINO_NPU_L0_HOST_TENSORS as an opt-in NPU buffer allocation mode. When enabled, non-remote OpenVINO backend buffers on NPU are allocated through the NPU default context with create_host_tensor(), and ggml tensor data points directly at that OpenVINO-owned Level Zero host allocation.
The optimization targets the data path rather than the graph math: KV-cache and compute tensors can be passed back to the NPU plugin as OpenVINO tensors backed by importable Level Zero host memory. That avoids an extra ordinary-host-memory-to-plugin-staging copy on each infer boundary, which is especially useful for prefill where large KV/cache buffers dominate input and output traffic.
If Level Zero host allocation fails, the backend logs a warning and falls back to the existing allocation path. The destructor tracks OpenVINO-owned storage so it does not free memory owned by the host tensor. The trade-off is that this is NPU-specific, consumes Level Zero host allocation resources, and improves transfer/import overhead without changing attention compute complexity or tensor shapes.
* openvino: fill NPU prefill masks directly
Add GGML_OPENVINO_NPU_FAST_MASK as an opt-in NPU static prefill input-packing optimization. When enabled, the prefill attention mask is built directly in the OpenVINO input tensor instead of first materializing a padded std::vector and then copying that whole buffer into the tensor.
The new fill_prefill_mask() helper writes each row once: it copies the valid mask prefix, fills padding columns with -inf, handles padded rows, and restores the causal diagonal to zero. This keeps the final mask numerically identical to the existing pad_input() plus set_zero_diagonal() path while removing one intermediate allocation and one chunk_size-by-context_size host copy per prefill chunk.
The optimization targets CPU-side staging overhead, not attention math. It composes with GGML_OPENVINO_NPU_KV_SLICE: KV slice reduces the mask dimensions, while FAST_MASK reduces the work needed to construct whatever mask shape remains. The trade-off is a second mask construction path to maintain, with limited benefit for very small prompts or contexts.
* openvino: make NPU requant layout configurable
Add GGML_OPENVINO_NPU_REQUANT_POLICY as the NPU-side selector for the default weight requantization layout. The existing NPU behavior remains the default: unset or group-128 maps quantized weights to Q4_0_128, while channel-wise selects the experimental Q4_0_C layout. Unknown policy values fail fast during model loading instead of silently compiling a graph with unintended weight packing.
The optimization target is NPU weight bandwidth and compressed-weight code generation. Q4_0_128 keeps weights in a 4-bit symmetric layout with one scale per 128 values, reducing both payload bytes and scale metadata versus smaller groups while preserving a more conservative quantization granularity than channel-wise. Q4_0_C uses one scale over the innermost dimension, which can further shrink metadata and simplify dequant broadcast, but may cost accuracy because the quantization block is much larger.
Include the selected NPU requant policy in the OpenVINO model cache discriminator. The same cgraph compiled with group-128 and channel-wise has different weight constants and dequant shapes, so reusing a cached decoder across policies would be incorrect. This keeps cache reuse stable within a policy while forcing recompilation when the layout policy changes.
Trade-off: policy selection gives a performance/accuracy knob for NPU experiments, but all options still pay the load/compile-time requantization cost. The default remains group-128 for a safer balance of bandwidth, metadata, and accuracy; channel-wise is intentionally documented as experimental.
* openvino: centralize NPU plugin config overrides
Add GGML_OPENVINO_NPU_CONFIG as the recommended catch-all override for OpenVINO NPU plugin and compiler properties. The value is parsed as comma-separated KEY=VALUE pairs and applied after the built-in defaults and named NPU aliases, so advanced users can override any exposed property without adding a new llama.cpp environment variable for every plugin tuning knob.
Keep the common knobs as named aliases for discoverability and compatibility: GGML_OPENVINO_NPU_COMPILER_TYPE maps to NPU_COMPILER_TYPE, GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL maps to NPUW_FUNCALL_FOR_ALL, GGML_OPENVINO_NPUW_UNFOLD_IREQS maps to NPUW_UNFOLD_IREQS, and GGML_OPENVINO_COMPILATION_NUM_THREADS maps to COMPILATION_NUM_THREADS. GGML_OPENVINO_NPU_COMPILE_CONFIG is retained as a legacy alias for NPU_COMPILATION_MODE_PARAMS, while the documentation now recommends expressing it through GGML_OPENVINO_NPU_CONFIG.
This keeps the public tuning surface compact while preserving high-signal aliases for settings with known performance or stability implications. NPU_COMPILER_TYPE can select the driver compiler, which currently generates faster prefill kernels on the tested NPU stack. NPUW_FUNCALL_FOR_ALL and NPUW_UNFOLD_IREQS expose NPUW graph lowering trade-offs around dispatch overhead, memory use, and long-context stability. COMPILATION_NUM_THREADS gives a way to cap compiler worker parallelism to reduce peak host memory during large graph compilation.
Trade-off: a free-form KEY=VALUE string is less type-safe than one env var per option and invalid plugin properties will only be diagnosed by the OpenVINO stack. Applying it last is intentional: it gives one escape hatch for bisecting driver/compiler behavior and for testing new plugin options without rebuilding llama.cpp, while the named aliases continue to document the most important performance knobs.
* openvino: add scatter-elements KV update option
Add GGML_OPENVINO_KV_SCATTER_ELEMENTS as an opt-in lowering for non-stateful single-row KV cache writes. When the flag is enabled and the set_rows indices are the simple decode case, translate_set_rows uses ScatterElementsUpdate with broadcasted rank-4 indices instead of ScatterUpdate.
The motivation is NPU decode performance. The default ScatterUpdate path updates one KV row but the NPU plugin does not perform that update in place; it effectively copies the full destination tensor to write a small slice. As context length grows, that full-tensor movement dominates the per-token KV update cost. ScatterElementsUpdate expresses the same single-row write at element granularity and is measurably cheaper for this shape.
Keep the optimization guarded. Stateful execution keeps its existing concat/update path, and multidimensional indices already use the broader ScatterElementsUpdate lowering. The new flag only changes the common non-stateful decode write, so the default behavior and graph shape remain unchanged unless the user opts in.
Trade-off: ScatterElementsUpdate introduces extra ShapeOf, Reshape, and Broadcast nodes to build indices matching the update tensor. That overhead is worthwhile only when it avoids the NPU plugin's full-destination ScatterUpdate copy, so the path is controlled by an environment variable rather than becoming the unconditional lowering.
* openvino: gather token embeddings from compressed rows
Add NPU token-embedding controls for models whose token_embd.weight would otherwise be widened from Q6_K to FP16. GGML_OPENVINO_TOKEN_EMBD_I8 keeps the embedding table on the normal int8 compressed-weight path, while GGML_OPENVINO_TOKEN_EMBD_I4 requantizes it to the group-128 int4 layout used by the NPU weight path.
The performance target is tied embeddings and large vocabulary tables. When token_embd.weight is also consumed by the lm_head matmul, a normal Gather from the dequantized table gives the dequantization subgraph a second consumer and can force the full table to be materialized. The new gather_compressed_rows helper recognizes the dequantization chain emitted for int4/int8 constants, gathers only the selected packed rows from the leaf constants, and rebuilds the convert/subtract/multiply/reshape chain on those rows.
This reduces memory pressure and avoids dequantizing or materializing a full embedding table when inference only needs the token rows for the current input. The helper is deliberately conservative: it only rewrites static 2D tables above a size threshold, walks only the known compressed-weight node patterns, limits recursion depth, and falls back to the original Gather when the subgraph is not recognized.
Trade-off: int8 and especially int4 token embeddings can change accuracy and may not be faster for small untied embedding tables, because the rewrite adds extra Gather/Reshape nodes and only pays off when it avoids a large full-table dequantization. The default behavior is unchanged; the NPU-specific I8/I4 paths are opt-in through environment variables.
* openvino: stream u4 requantization chunks
Extend the GGML_OPENVINO_REDUCE_COMPILE_MEM streaming requantization path to Q4_0/u4 targets. The previous streaming path intentionally skipped u4 because Q4_0 packs two weights per byte and the zero-point tensor stores two 4-bit values per byte, so the original quantize_q4_0 helper wrote as if every call started at block zero.
Teach quantize_q4_0 to accept a destination block offset and use the absolute block index for weights, scales, and zero-point writes. Streaming requantization emits whole-row chunks in increasing order, and the target block size is required to divide the row width, so no target block straddles a chunk boundary. That keeps packed weight bytes aligned and preserves the even-block assignment before the odd-block zero-point OR for each shared zero-point byte.
With that indexing in place, reduce-compile-memory mode can dequantize a chunk of rows into the scratch buffer and immediately quantize it into the final u4 buffers instead of materializing the entire tensor as temporary F32. This extends the peak-memory reduction already used by Q8/F16 targets to the NPU group-128/channel u4 layouts used for compressed weights.
Trade-off: the u4 path is more sensitive to block alignment than Q8/F16 because of nibble packing, so streaming remains guarded by the same row-divisibility check and only runs when GGML_OPENVINO_REDUCE_COMPILE_MEM is enabled. When the flag is off, the full-materialization path and quantized output layout remain unchanged.
* openvino: fail fast on static recompile after weight release
Extend the GGML_OPENVINO_RELEASE_WEIGHTS safety check to the static graph cache-miss path. Once host weight buffers have been released, any later compile would read invalid/zeroed host pages and bake the wrong constants into a new OpenVINO model.
The dynamic path already rejected this situation. Static execution can still miss the decoder cache when graph shapes or model parameters change, so it needs the same guard before erasing cached infer requests and rebuilding prefill/decode models.
This is a correctness and diagnosability change rather than a throughput optimization. It preserves the memory-saving release-weights mode for stable graph shapes, but turns unsupported recompilation into an explicit abort instead of silently producing a corrupted compiled model.
* openvino: add experimental NPU Q4_0 keep option
Add GGML_OPENVINO_NPU_KEEP_Q4_0 as an opt-in escape hatch for NPU weight handling. When enabled for native Q4_0 tensors, the NPU requant path returns nullopt and keeps the original Q4_0 extraction instead of regrouping the tensor through the selected NPU requant policy.
The motivation is to avoid unnecessary work for weights that are already stored as 4-bit values. The default NPU policy maps quantized weights to layouts such as Q4_0_128, which can reduce scale metadata but requires round-tripping Q4_0 through the requantization path. Keeping Q4_0 native can save that conversion work and preserve the original group-32 packing.
Keep the option experimental and disabled by default. Current NPU driver/plugin stacks can compile the resulting group-32 graph, but inference may hang or fail with ZE_RESULT_ERROR_DEVICE_LOST. The default behavior therefore continues to use the safer NPU requant policy; this flag is mainly useful for debugging driver behavior, measuring the true cost of regrouping, and testing future NPU stacks.
Trade-off: native Q4_0 avoids regrouping overhead and keeps the original quantization blocks, but it gives up the more regular group-128 layout expected by the default NPU compressed-weight path and may be unstable on current hardware/software combinations.
* openvino: unfold NPUW only for decode graph
Compile the static prefill and decode models with separate OpenVINO config maps so the decode graph can opt into NPUW_UNFOLD_IREQS without forcing the same lowering on prefill. The build_static_model helper now receives the config map explicitly, and the decode path gets a copy of the base config with NPUW_UNFOLD_IREQS=YES when the user has not already set it.
The performance target is NPU token generation. Decode executes a long stream of single-token infers, where folded NPUW function-call dispatch overhead becomes visible in every token. Unfolding the NPUW calls into separate infer requests removes that repeated dispatch cost and measured about +20% token-generation throughput on the tested stack, matching the behavior seen in OpenVINO GenAI.
Prefill keeps the original folded config. Prompt processing is a batched workload dominated by larger matmuls, and the folded form is faster there while also using less memory. Splitting the config lets prefill and decode use the lowering that matches their workload shape instead of treating NPUW_UNFOLD_IREQS as a single global compromise.
Trade-off: unfolding can increase memory and request-management overhead, so the automatic default is limited to the NPU decode model. If a user explicitly provides NPUW_UNFOLD_IREQS through the named env alias or GGML_OPENVINO_NPU_CONFIG, that value is respected because the decode override is only applied when the property is absent.
* openvino: tidy NPU tuning helper ordering
Collect the remaining small cleanups from the NPU tuning series. This reorders the NPU KV-slice documentation and environment-variable cache entry next to the other NPU opt-in toggles, adds a short header comment for the centralized NPU helper declarations, and makes the static prefill attention-size assignment in the decoder a little more explicit.
There is no intended behavior change in this commit. The purpose is to make the final source layout match the split commits that introduced the NPU tuning knobs, so future readers can find the related flags and helpers in one place without mixing that mechanical cleanup into the functional performance commits.
* openvino: normalize NPU compile env overrides
Move the default NPU_COMPILER_TYPE=DRIVER setting into the NPU compile_config initializer so the default NPU plugin options are declared in one place.
Route GGML_OPENVINO_NPU_COMPILE_CONFIG and GGML_OPENVINO_NPU_COMPILER_TYPE through the same set_compile_option_from_env helper used by the other named NPU compile option aliases. This keeps the legacy NPU_COMPILE_CONFIG mapping to NPU_COMPILATION_MODE_PARAMS intact while making all optional env-to-compile_config overrides follow the same non-empty-value rule.
GGML_OPENVINO_NPU_CONFIG is still parsed last, so explicit comma-separated KEY=VALUE overrides continue to have the highest precedence over both defaults and named aliases.
* openvino: stream NPU weight requant by default to cut compile peak
Default the streaming requantization path on for NPU so the compile-time F32 dequant transient (~1-2 GB for token_embd) is capped without an env flag. Output is byte-identical to full materialization; GGML_OPENVINO_REDUCE_COMPILE_MEM / MEMORY_OPTIMIZE still override (set to 0 to force off). Kept the g_nonov_weight_cache opt-in to avoid holding token_embd resident (which regressed decode steady memory). pp1024_d0: PrefillPeakPrivWS 5974->4664 MiB (-22%), decode steady and pp/tg throughput unchanged.
* openvino: add self-contained NPU model cache
Add an opt-in embedded-weight cache mode for the static NPU backend. GGML_OPENVINO_SELF_CONTAINED_BLOB=1 exports separate prefill and decode compiled models with ENABLE_WEIGHTLESS=false, uses isolated NPUW weight banks, validates raw GGUF weight manifests, binds imported ports by deterministic frontend order, and serializes the large NPUW compile/export/import operations.
Preserve raw model bytes during cold cache construction by requantizing into separately owned OV tensors instead of rewriting the backend model buffer. This keeps later graph fingerprints and manifests stable when llama-bench creates independent prompt and generation contexts.
Add strict GGML_OPENVINO_SELF_CONTAINED_MMAP=1 warm-import mode. The backend exposes a non-owning buffer_from_host_ptr wrapper over the GGUF mmap, uses raw tensor allocation sizes, avoids allocating/copying the private OpenVINO model-weight buffer, and stages the few mmap-backed runtime constants into small importable NPU host tensors. Invalid or missing cache entries fail clearly instead of trying to requantize into read-only mapped storage.
The normal path and existing weightless MODEL_PTR cache remain unchanged when the new flags are unset. Verified with phi-4-mini: chunk512 pp3072/tg512 cold 1123.10/27.60 t/s and warm mmap 1124.72/27.16 t/s; warm mmap removes roughly 3.9 GiB peak commit in the measured prefill workload. Self-contained blobs consume about 8.34 GiB for the separate pp and tg contexts.
* openvino: add lazy import for self-contained mmap cache
Add GGML_OPENVINO_SELF_CONTAINED_LAZY_IMPORT to make the low-working-set strict mmap path opt in instead of changing the default warm mmap behavior.
When the flag is set with SELF_CONTAINED_BLOB=1 and SELF_CONTAINED_MMAP=1, the static NPU path imports only the current phase blob: prefill imports the prefill blob, decode imports the decode blob later. This avoids the pair-shaped shared compiled-model cache for the lazy path and caches only the current phase infer request.
The regular warm mmap path remains eager and continues to prepare both prefill and decode compiled models. Non-mmap self-contained builds remain serial, and the regular non-self-contained path keeps the existing parallel build behavior.
Also register the new environment variable, document the memory/latency tradeoff, and add profiling-only import timing around model-cache imports.
Validated with ReleaseOV build: cmake --build build/ReleaseOV --target llama-bench llama-simple --config Release. Lazy warm mmap run measured PrefillPeakWS around 5759 MiB versus the eager warm mmap path around 7897 MiB.
* openvino: release GGUF mmap resident pages before blob import
Add GGML_OPENVINO_SELF_CONTAINED_RELEASE_MMAP_PAGES as an explicit opt-in for self-contained compiled-model cache hits.
Before importing a valid self-contained blob, recursively walk the graph to find unique external GGUF mmap-backed weight buffers and drop their resident pages while preserving the mapping address and tensor pointers.
On Linux use page-aligned madvise(MADV_DONTNEED). On Windows, where there is no reliable per-range mapped-file equivalent, call VirtualUnlock for the identified range and trim the current process working set with SetProcessWorkingSetSize(-1, -1). Pages required later can fault back on demand.
Keep the behavior disabled by default and document the physical-memory versus commit-memory tradeoff. Profiling reports the submitted GGUF range and model import duration.
Validated with ReleaseOV llama-bench/llama-simple build and pp3072+tg512 warm mmap lazy import. With page release enabled, PrefillPeakWS dropped from about 5758 MiB to 3053 MiB while throughput remained comparable; PeakCommit stayed about 3331 MiB.
* openvino: add GGML_OPENVINO_NPU_SHARED_BANK to share prefill/decode weights bank
When set, prefill and decode self-contained blobs use one shared NPUW_WEIGHTS_BANK name (ggml-sc-shared-<pfp>-<dfp>) instead of separate per-phase banks. Since both decoders are built from the same model_weights map, NPUW dedups them into a single in-process bank; on warm import the identical (uid,tensor) pairs are content-verified and reused (patched NPUW weights_bank). Cuts resident compiled-weight footprint ~2GB (pp3072 warm: In-Use 7954->5871 MiB, WS 7891->5765) with prefill throughput unchanged.
* openvino: drop SELF_CONTAINED_LAZY_IMPORT and RELEASE_MMAP_PAGES
The shared weights bank (GGML_OPENVINO_NPU_SHARED_BANK) supersedes both: it deduplicates the prefill/decode compiled-weight copy directly, so lazy per-phase import and GGUF-page release no longer reduce steady memory. Remove the two env vars, the self-contained lazy-import branches (defaulting to the eager path that imports both blobs), the release_external_weight_pages helper, and the now-unused ggml_openvino_buffer_is_external / _release_external_pages buffer APIs.
* openvino: demand-page large NPU compute buffer in strict warm-mmap mode
In strict self-contained warm-mmap mode weights come via buffer_from_host_ptr,
so every alloc_buffer is graph scratch/KV. The large prefill compute buffer is
never touched on the host by the on-device fused OV model, yet the NPU L0 host
tensor (create_host_tensor + memset) makes all ~2.4 GiB resident. For buffers
>= 1 GiB in this mode, reserve demand-zero pages (no L0, no memset) so they stay
non-resident; KV/small buffers keep the L0 host tensor for zero-copy import.
pp3072 warm mmap + shared bank: prefill peak WS 5765 -> 3314 MiB (-43%),
In-Use delta ~5940 -> ~3232 MiB; prefill 1256.7 -> 1252.9 t/s and decode
28.27 -> 27.96 t/s (both flat). Commit charge unchanged (demand-paged).
* perf improvement success
* openvino: size static NPU sliding-window masks to the window cache
The static graph built the sliding-window KQ mask at the full context length,
while the windowed layers bind their (smaller) window KV cache. With gemma-4 at
-c 4096 -ub 1024 the window cache is 1536 cells, so SDPA shape inference failed
([1,1,1024,4096] mask vs [1,8,1536,256] K/V) and every prompt errored out.
With GGML_OPENVINO_NPU_KV_SLICE the prefill mask was meant to use
attention_size_swa, but the check matched "swa" in the ggml tensor name, and
llama.cpp names both masks "attn_inp_kq_mask"; the windowed layers got the
full-attention sliced length instead ([1,1,1024,256] vs [1,8,1024,256]).
Identify the sliding-window mask by tensor identity (is_swa_mask) and size it to
ctx_per_seq_swa, both in the graph input shape and in the runtime mask padding.
* ggml-openvino: create the model cache dir under a Windows drive root
make_dirs() mkdir'ed every path prefix, including the drive root "D:", which
fails with EACCES (not EEXIST) on Windows. Any absolute cache path therefore
logged "could not create model cache dir ... caching disabled" and silently
ran without GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. Skip the drive root.
* ggml-openvino: split device-specific passes and NPU config out of shared code
Mechanical refactor, no behavior change (step 1 of the NPU/MoE architecture plan):
* openvino/pipelines/: TranslateSession::apply_transformations() now registers
common passes, then the device's passes, then the execution-mode passes
(stateful KV, static shapes), selected by a per-graph GraphTraits. The GPU MoE
fusions and the GPU eltwise-rank workaround move to pipelines/gpu.cpp; CPU and
NPU pipelines are empty placeholders for device lowerings. Pass order is
unchanged.
* npu/npu_config.*: the NPU (NPUW) compile config built at backend init, the
NPUW strip for cacheless graphs and the per-graph static prefill/decode
variants (HFA prefill, decode NPUW_UNFOLD_IREQS) move out of
ggml-openvino-extra.cpp and utils.cpp.
* npu/npu_quant_policy.*: the NPU requantization rules (requant policy,
token_embd Q6_K, keep-Q4_0) move out of ggml_openvino_get_requant_type().
Verified identical output before/after on OV-CPU, GPU and NPU (gemma-4 E4B).
* ggml-openvino: one KV-slice switch for dynamic and static graphs
KV-cache slicing (bind the KV cache, and size the KQ masks, at the attended
length instead of the full context) existed twice: on by default for dynamic
graphs (GGML_OPENVINO_DISABLE_KV_SLICE to turn it off) and opt-in for static
prefill (GGML_OPENVINO_NPU_KV_SLICE). The static mode is not NPU-specific; it
applies to any static graph, including CPU/GPU with GGML_OPENVINO_FORCE_STATIC.
Describe it as one mechanism with two modes behind GGML_OPENVINO_KV_SLICE
(unset = each mode's default, 0 = off, 1 = on) and two predicates,
ggml_openvino_{dynamic,static}_kv_slice_enabled(), used by the KV binding,
the static graph shapes, the mask padding and the cache keys. The two old
variables remain as deprecated aliases.
* ggml-openvino: initialize all ModelExtraInputInfo fields for the HFA inputs
dev_backend_openvino added shape_source/shape_axis to ModelExtraInputInfo; the HFA past-KV and mask
initializers still listed four fields, which GCC rejects under -Werror=missing-field-initializers.
* ggml-openvino: make new NPU defaults opt-in and fix static model conversion
- Convert the static prefill and decode models one after the other, both
before either is compiled. The decoders share the weight nodes;
converting on two threads crashed intermittently inside libopenvino,
and converting decode after the prefill compile gave wrong output on
Bonsai-8B.
- Compile prefill and decode in parallel only with
GGML_OPENVINO_NPU_PARALLEL_COMPILE=1, since it raises the host memory
peak.
- Unfold NPUW function calls on the decode graph only with
GGML_OPENVINO_NPU_DECODE_UNFOLD=1. The unfold default gave wrong output
on gemma-4-E4B and slowed decode on several small models.
- Stream NPU weight requantization only with
GGML_OPENVINO_REDUCE_COMPILE_MEM or GGML_OPENVINO_MEMORY_OPTIMIZE, as on
the other devices.
- Fix a -Werror=missing-field-initializers build error in utils.cpp.
- Declare ggml_openvino_get_npu_requant_type() only in
npu/npu_quant_policy.h.
- Docs: document the new switches, deprecate stateful execution, show
stateless-only results in the validated models table, and update the
validation system.
* ggml-openvino: honour the -1 skip id in MUL_MAT_ID
Upstream made an id of -1 in MUL_MAT_ID skip the slot and zero its row of
the result. The translator passed -1 straight to GatherMatmul or Gather,
so 169 of the 743 MUL_MAT_ID op tests on CPU (all skip=1) failed.
A skipped slot now looks up a valid expert, and the result is multiplied
by a 0/1 mask of the kept slots. For the packed MXFP4 path (Gather and
MatMul), clamping the id to expert 0 is enough. GatherMatmul computes a
wrong row when a token names the same expert twice, so the general path
gives each skipped slot an expert that no other slot of the token uses.
TopK indices are never -1, and model graphs take the ids from the
router's TopK. Those graphs skip the handling and stay unchanged, so the
GPU MoE fusion still matches them.
* ggml-openvino: Gather rows before dequantizing a quantized embedding table
1. Optimize gather to fetch row before dequant
2. add NPU token_embd quantization option
* ggml-openvino: select the NPU token_embd type with GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT
GET_ROWS had two rewrites that gather the rows of a quantized table before
dequantizing them: gather_compressed_rows, which runs on every device, and
gather_before_dequant, enabled on NPU by GGML_OPENVINO_GATHER_BEFORE_DEQUANT.
Both match the int4 and int8 chains built by make_int4_weights() and
make_int8_weights(). Keep gather_compressed_rows and drop the second rewrite
and its switch.
The NPU type of a Q6_K token_embd.weight was set by two booleans,
GGML_OPENVINO_TOKEN_EMBD_I8 and GGML_OPENVINO_TOKEN_EMBD_I4, and by
GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT. Keep only the string variable, with the
values f16 (default), q8_0_c, q4_0_128 and q4_0_c, so the int8 and group-128
int4 settings stay available. An unknown value aborts, as for
GGML_OPENVINO_NPU_REQUANT_POLICY, instead of silently using f16. The variable
replaces the two booleans in the NPU compiled-model cache key.
* align PAD pads to input rank in stateful mode
* Gate the changes under is_stateful check to address review comment
* ggml-openvino: lift Q tensors for non-RoPE layers to rank 4 in stateful mode
* ggml-openvino: unify KV slicing under GGML_OPENVINO_KV_SLICE, on by default
Replace ggml_openvino_dynamic_kv_slice_enabled() and ggml_openvino_static_kv_slice_enabled() with a single ggml_openvino_kv_slice_enabled(). KV slicing binds the KV cache, and sizes the KQ masks, at the attended length instead of the full context.
GGML_OPENVINO_KV_SLICE now defaults to 1 on CPU, GPU and NPU; set it to 0 to disable. Dynamic graphs (CPU/GPU) bind a view over the cache without recompiling. Static graphs (NPU or GGML_OPENVINO_FORCE_STATIC) compile prefill at the aligned attended length and recompile when it changes; decode keeps its full-context graph.
Remove the deprecated aliases GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_NPU_KV_SLICE from the env var list, the model-cache settings, docs/backend/OPENVINO.md and the backend README.
Behavior change: static KV slicing is now on by default on NPU, so prefill compiles one graph per distinct n_kv (multiples of 256).
* ggml-openvino: keep stateful execution with -fa off
Keep the V cache token-major in stateful mode instead of falling back to
stateless execution. SET_ROWS appends to the state, and PERMUTE reads it
back in the layout attention needs, so MUL_MAT does not transpose it.
Dropping a reshape in grouped attention lets OpenVINO fuse it into SDPA.
For padded V caches, slice after the transpose so the cache is read in
place, without a copy that is recompiled for every KV length.
This removes the stateless fallback for -fa off.
* ggml-openvino: Add the stateful check to make the gate safer
* ggml-openvino: support gemma-2 soft-capped attention
Gemma-2 caps its attention scores as softcap * tanh(scores / softcap)
before the softmax. FLASH_ATTN_EXT now applies the cap on the manual
attention path, as SDPA has no soft-capping, and the decoder looks
through the SCALE, TANH, SCALE chain of -fa off when it matches
attention.
One helper finds the K cache of an attention for both the
sliding-window pre-pass and the main pass. The pre-pass copy did not
see through the chain, so the sliding-window layers used the
full-attention sizes once the context exceeded n_swa + n_ubatch.
Soft-capped attention keeps the flat KV cache layout: with the 4-D
layout the GPU plugin fails in its oneDNN path.
* ggml-openvino: enable NPU Level Zero host tensors by default
Back NPU buffers with Level Zero host memory unless GGML_OPENVINO_NPU_L0_HOST_TENSORS=0.
The plugin imports these buffers without a host-to-device copy each infer; on phi-4-mini
Q4_0 (pp3072+tg512) decode goes from 7.2 to 10.6-10.8 t/s. Allocation failures still fall
back to plain host memory.
* ggml-openvino: enable NPU fast prefill mask by default
GGML_OPENVINO_NPU_FAST_MASK now defaults to 1; set it to 0 to fall back to pad_input() + set_zero_diagonal().
The fast path builds each static prefill mask chunk directly in the OpenVINO input tensor (row memcpy of the valid prefix, -inf padding, zero diagonal), avoiding a temporary chunk_size x context_size vector and a full-tensor copy per chunk. The result is bit-identical to the fallback path.
* ggml-openvino: add opt-in exact zero points for asymmetric weights
Asymmetric quantized weights (Q4_K, Q4_1, Q5_1, and non-requantized
Q5_K) represent each group as `w = d * q - m`. The backend currently
converts these to u4/u8 weights with an f16 scale and an integer zero
point `round(m / d)`, which can shift all weights in a group by up to
half a quantization step.
Add `GGML_OPENVINO_EXACT_ZERO_POINT=1` to preserve the zero point in f16
instead, using the same path already used for MoE expert weights.
This is disabled by default because, with current OpenVINO releases, an
f16 zero point moves many GPU fully-connected layers off oneDNN,
resulting in slower prompt processing.
* ggml-openvino: rename npu/ to device/ and openvino/pipelines/ to openvino/pass_registry/
npu/ holds device-specific backend behaviour (NPU compile config and
requant policy), and pipelines/ only registers passes per device; the new
names describe what the folders contain. No functional change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs: document NPU self-contained model cache and shared weights bank
Add GGML_OPENVINO_SELF_CONTAINED_BLOB, GGML_OPENVINO_SELF_CONTAINED_MMAP and GGML_OPENVINO_NPU_SHARED_BANK to the OpenVINO environment variable table, and a note describing the export/import workflow and the requirement to keep NPU settings identical between runs.
* ggml-openvino: remove experimental GGML_OPENVINO_NPU_KEEP_Q4_0
Keeping native Q4_0 (group-32) on NPU compiles but hangs the device at inference (ZE_RESULT_ERROR_DEVICE_LOST), so the option was never usable. NPU Q4_0 weights now always go through GGML_OPENVINO_NPU_REQUANT_POLICY. Drop the env var, the requant bypass, its self-contained blob fingerprint term and the docs entry.
* ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares
All changes of https://github.com/ravi9/llama.cpp/pull/358 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.
Commits of the PR:
- Update fix to just maintain fp32 precision instead of modifying the graph
- Limit the fp32 precision spreading in the graph by fencing it with Convert. Prevents perf degrade.
- Update OPENVINO.md
- Update flash_attn_ext.cpp
- Update utils.cpp
* enable flash_attn with sink input
* ggml-openvino: fix FLASH_ATTN_EXT input detection and sink scale
translate_flash_attn_ext read get_input_names()[3] and [4] on every
FLASH_ATTN_EXT node, also on nodes with only 3 or 4 inputs. The
out-of-bounds read crashed test-thread-safety and
test-speculative-rejection-sampling. The inputs are q, k, v, then the
optional mask and sinks, and supports_op rejects sinks without a mask,
so the input count tells which inputs are present.
With sinks, the scale was replaced with 1.0. llama.cpp passes the
attention scale in op_params[0] and adds the sinks separately, so keep
op_params[0]. The "scale" input branch is removed: ggml's src[4] is
only ever the sinks.
Only SDPA applies the sinks. The manual GQA path and the NPU HFA
prefill path drop them, so attention with sinks now always takes the
SDPA path. SDPA has no logit soft-capping, so supports_op rejects sinks
together with soft-capping.
* ggml-openvino: accept flash attention on layers without RoPE
On layers without RoPE (SmolLM3: every 4th layer), Q reaches
FLASH_ATTN_EXT through PERMUTE -> VIEW -> RESHAPE. The support check
rejected this pattern, causing these layers to fall back to CPU and the
split graph to fail with -fa on/auto.
* ggml-openvino: keep only weight-divisor DIVs on CPU
The GPU DIV guard for per-channel scale vectors (`MPT`'s `act_scales`) currently checks only the divisor's shape. During single-token Qwen2-MoE decode, the shared-expert gate `silu(x) / x` has a `[1,1,1,1]` divisor that matches this shape, causing its DIVs to be kept on the CPU.
As a result, every decode step incurred CPU execution and Qwen1.5-MoE decoded at only ~2 tok/s.
Apply the guard only when the divisor is a model weight, so runtime tensors such as the Qwen2-MoE gate divisor can remain on the GPU.
* ggml-openvino: Qwen MTP and multi-seq optimization with new OV ops
All changes of https://github.com/ravi9/llama.cpp/pull/370 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.
Commits of the PR:
- openvino : support Qwen MTP decoding
- openvino: Optimize TopK
- openvino: remove single-slot recurrent cache shortcuts
- openvino: fuse generic recurrent caches and remove shape waits
- openvino: include dense Q8 weights in all INT4 requantization
- openvino: use two GDN outputs for snapshots
- docs: explain native OpenVINO recurrent cache contracts
- docs: clarify Qwen3.5 purpose of native OpenVINO cache ops
- llama: skip unallocated hybrid graph inputs
- quantize: preserve Qwen hybrid layer metadata when pruning
- openvino: emit fused recurrent computation and rotate device caches
- openvino: limit KV slicing to attention caches and share compiled blobs
- tools: add natural-text server benchmark and OpenVINO profile summary
- docs: record fused-cache branch split and UD validation
- server-bench: preserve native token counts and save request timings
- openvino: document MTP regression and host USM alias recovery
- openvino: record large-context allocation diagnosis and default-batch validation
- server-bench: count buffered native tokens and reject stream errors
- openvino: document UD split and unified server benchmarks
- openvino: record Vulkan long-context baseline and GPU-fit controls
- openvino: finalize UD benchmark report and MTP comparison
- openvino: use stable in-place recurrent caches and optional separate convolution
- openvino: accept matching empty axes in DIV repeat translation
- openvino: remove the obsolete cache rotation environment setting
- openvino: validate imported KV results in their four-dimensional layout
- docs: record the bounded in-place KV and convolution benchmark rerun
- docs: complete default-batch OV unified-KV 32K benchmarks
- openvino: remove the separate recurrent convolution path
- openvino: handle zero-logits batches and changing sampler sequences
- docs: compare default and large Qwen3.6 benchmark batches
- docs: use the historical Vulkan default-batch reference consistently
- server-bench: use an integer histogram bin count
- benchmarks: summarize current OV unified MMLU timings
- docs: record cache-only unified KV MMLU results
- openvino: preserve the TopK index port in GET_ROWS
- openvino: reuse the MTP sampling model across active slots
- openvino: expose selection flags and offsets in graph dumps
- docs: consolidate Qwen OpenVINO defaults and performance evidence
- docs: retain continuous-request MTP benchmark results
- docs: compare stable MTP with the non-MTP serving baseline
- openvino: isolate GDN snapshots and skip naive cache fusion
- ggml: document Qwen3.6 GenAI benchmarks
- ggml-openvino: keep PR changes inside the backend folder
* ggml-openvino: drop the extra semicolons after OPENVINO_OP
OPENVINO_OP expands to member function definitions that end in a closing
brace, so a semicolon after it is an empty declaration inside the class.
GCC 15.2 reports it with -Wextra-semi, which ggml enables for GCC 8.1 and
newer, so a build with LLAMA_FATAL_WARNINGS=ON (the default in ci/run.sh)
fails in gated_delta_net.hpp. The CI compilers do not report it.
recurrent_cache_update.hpp has the same pattern twice. It is compiled only
when the OpenVINO package provides the recurrent cache-update ops.
* ggml-openvino: make q8_0_c the default NPU token_embd requant type
Default of `GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT` changed from `f16` to `q8_0_c`.
Applies only to NPU and only to a Q6_K `token_embd.weight`; other devices and
quantization types are unchanged.
* ggml-openvino: do not treat the recurrent state gather as out_ids
The output_len scan now runs for every model and takes the first GET_ROWS
whose index is a graph input. On Qwen3.5 that is the recurrent state gather
of build_rs, which indexes the inp->s_copy leaf directly, so output_len
became 1. In static (NPU) mode the prompt model then returned one logits
row per chunk, and llama-perplexity read stale rows (PPL about 240000 on
Qwen3.5-2B). The same predicate also gave the s_copy input the out_ids
shape and data in static mode.
is_output_idx now skips GET_ROWS that read a recurrent state cache. The old
predicate stays as is_get_rows_input_idx for the dynamic-dim bookkeeping,
so the s_copy leaf keeps its dynamic length there.
* ggml-openvino: prefer the size-preserving dim when a RESHAPE repeats a stride
The RESHAPE dynamic-dim rule matches the source's dynamic stride against
the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4
E2B has one KV head, so [head_size, n_tokens] -> [head_size, 1, n_tokens]
has the same stride on dims 1 and 2. The first match picked the size-1 head
dim, so the K/V views lost their dynamic token dim and became static
reshapes with the first graph's token count. Every later graph with another
token count failed.
Prefer the stride match that keeps the dynamic size, and fall back to the
first stride match for reshapes that change it.
* ggml-openvino: correct the RESHAPE dynamic-dim comment for size-1 dims
The comment above the RESHAPE stride match in compute_node_dynamic_dims()
still said the match is unambiguous because result strides are strictly
monotone. A size-1 dim has the same stride as the dim after it
([head_dim, T] -> [head_dim, 1, T] with one KV head), so it is not.
Explain the size-1 case in the block comment, drop the duplicate comment
inside the loop, and name the source dim's size once.
* examples: add llama-chat-replay
Replays JSONL prompts like OpenVINO GenAI chat.py (-pf, -i chat history, greedy, -o, -pc metrics). The first prompt is prefilled plus one decode step during load (Warmup time) and the memory is cleared, so compile/import is counted in Load time as in GenAI.
* ggml-openvino: make the NPU lazy host threshold configurable, default 2 GiB
With a large context the KV cache (~1 GiB at ctx 8K) crossed the 1 GiB threshold, lost its Level Zero host tensor in strict warm-mmap mode and made decode copy the KV every token. Default to 2 GiB so only the larger prefill compute buffer stays demand-zero, and allow overriding it with GGML_OPENVINO_LAZY_HOST_THRESHOLD_MB.
* ggml-openvino: fix NPU HFA prefill with ordinal-suffixed KV input names
KV cache inputs are now named with a graph-ordinal suffix (cache_k_l0#l12), but flash_attn_ext looks the HFA past input up as ggml_hfa_past_cache_k_l<layer>, so GGML_OPENVINO_NPU_HFA_PREFILL failed with 'ggml_hfa_past_cache_k_l0 not found in tensor map'. Name the past inputs after the bare cache name and resolve the source cache by that name when binding.
* ggml-openvino: keep the NPU prefill model on cache hits after decode
With KV slicing, decode calls overwrite the shared decoder's attention size with the full context, so the next prefill compared that against its sliced size, missed the cache and rebuilt the decoder and both infer requests (plus a cold first chunk). Remember the attention sizes the prefill model was built with and compare against those.
* ggml-openvino: bind NPU HFA past KV zero-copy in the cache layout
The HFA prefill past K/V inputs were [1, heads, past, head_size] copies of the cache, relaid out per chunk on the host and allocated per decoder (256 MiB on phi-4-mini at 3k). Declare them in the ggml cache layout [1, past, heads, head_size], bind the first past rows of the cache directly and transpose in the graph; NPUW HFA reads the tiles from the BSHD past (needs the matching NPUW HFA support). HFA input ABI v4.
* ggml-openvino: size the NPU decode KV window by GGML_OPENVINO_NPU_DECODE_BUCKET
Static decode attended the whole context, so every token read and wrote the full KV cache. With a bucket base set, decode attends the smallest base*2^k rows covering the next position (clamped to ctx): the KV inputs and masks are sliced to the window, the compiled pair's decode graph is built at the window the prefill reaches, and other windows are built or imported from the self-contained cache on first use with the pair's settings and bank. With self-contained blobs, keep the token_embd weight node for the later window models so they reuse the pair's bank entry instead of embedding a second vocab copy. The static build/convert lambdas become functions shared with the window path.
* ggml-openvino: allocate the NPU KV cache on demand with GGML_OPENVINO_NPU_KV_ON_DEMAND
The NPU KV buffer was committed for the whole context up front. With the option set, the KV buffer is swapped for demand-zero memory at its first KV tensor and every KV tensor gets its own storage in a registry, grown in base*2^k rows (base = decode bucket or 1024) as the static graphs reach further. Host accessors, KV inputs and outputs, sliced KV views and the HFA past bind that storage; when it moves, idle requests bound to the old storage are recreated so it is released.
With decode buckets and KV slicing the storage also takes the layout GenAI uses on NPU: K is [heads][rows][head_size] and V is [heads][head_size][rows], each bound whole to prefill and decode. SET_ROWS scatters new K rows on axis 2 and V rows on axis 3, the attention permute only cuts the window, the HFA past is the storage itself, and attention multiplies the probabilities by V^T with transpose_b instead of SDPA. A chunk-sized scatter on the innermost axis is slow on NPU, so the HFA prefill returns each chunk's V^T rows and the host copies them into the cache. Host reads, writes and fills go through a layout-aware copy, storage growth re-packs each head, and the layout is part of the model cache fingerprint.
* ggml-openvino: NPU weightless blob cache with one shared weights file
GGML_OPENVINO_NPU_WEIGHTLESS_BANK=1 (with GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR) exports weightless prefill/decode NPUW blobs plus one <prefill>-<decode>.weights.bin. Weight Constants are laid out once by data pointer and their WeightlessCacheAttribute is retagged with the real offsets. Warm import uses ov::weights_path, and NPUW dedups both models into one bank through the LazyTensor hashes, so the OpenVINO weights_bank duplicate-uid patches are not needed. Decode KV-window models resolve the shared weights through the pair's file, or get their own file when the pair was imported. Also drop the old blob and manifest before publishing a re-export: std::rename does not replace an existing file on Windows.
Based on anikulk/llama.cpp 4ab270aa9.
Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com>
* ggml-openvino: prebuild all NPU decode KV windows with GGML_OPENVINO_NPU_DECODE_PREBUILD
Decode windows were built lazily when the sequence first grew into them, which stalled generation for a compile without a cache and made a strict-mmap import fail for windows the export run never reached. GGML_OPENVINO_NPU_DECODE_PREBUILD replaces GGML_OPENVINO_NPU_DECODE_BUCKET: 0 (default) keeps decode over the whole context; 1 sizes decode by 1024*2^k KV windows and the call that builds or imports the prefill/decode pair also builds or imports the decode graphs of every other window up to the context, from the pair's graph and compute params. Requests are still created on first use. Window building is split out of the request lookup so both paths share it, and window ports are checked against the pair's decode ports.
* Revert "examples: add llama-chat-replay"
This reverts commit 4eded2c52c2df9fa1251673a523f8c7cded20d7c.
* ggml-openvino: initialize the new compiled-graph and build-args fields
ov_compiled_graph and static_build_args gained members, but three brace
initializations still listed only the old ones. GCC reports
-Wmissing-field-initializers there, and CI builds with warnings as errors.
Set the fields by name, and pass {} for weights_path, which the window
build sets afterwards.
* ggml-openvino: fix gemma-4 when only the output rows reach the KV-shared layers
With one sequence, gemma-4 now drops the rows that are not output before
its first KV-shared layer. Those layers take their positions through
GET_ROWS(inp_pos, out_ids), and the backend mishandled three things:
- The positions are reshaped from [n] to [1, n]. That matched RESHAPE
case 1, which marks the wrong dim dynamic. Use case 3, which keeps
the source dim as dim 1.
- ROPE shared one sin/cos table per frequency setting, so the KV-shared
layers reused the table built from the full inp_pos. Add the position
input to the cache key.
- compute_llm_params took input_len from the last attention op. That is
now a KV-shared layer with only the output rows, so the static (NPU)
path padded prefill to the wrong length. Take the largest query length.
Stateful execution still fails on the per-layer embedding view and is
not covered here.
* ggml-openvino: keep the inp_pos layout for gemma-4's gathered positions
A batch can have no output rows, for example a prompt chunk that
llama-cli processes before its last tokens. gemma-4 then reshapes an
empty position gather for the ROPEs of its KV-shared layers. The size
checks of the RESHAPE cases cannot place a dim of size 0, so the
positions got the wrong layout and the ROPE multiply failed on CPU and
GPU. A RESHAPE that feeds the ROPE positions now always takes the
inp_pos layout [1, 1, 1, n].
* ggml-openvino: enable stateless GPU execution for MLA models
The per-head MLA weights (`attn_k_b` / `attn_v_b`) are 3D quantized tensors. Their `MUL_MAT` operations were falling back to the CPU, splitting the graph at every layer and causing GPU execution to fail.
Accept these tensors in the GPU support check and address the following issues exposed by enabling GPU execution:
* Quantization: Use exact zero points only for MoE expert weights. U4 weights with F16 zero points fail in batched GPU MatMul.
* Dynamic VIEW dimensions: Select the correct dimension when a size-1 dimension shares its stride (e.g. `k_pe`), and match KV cache views by stride.
* KV cache V view: Translate the V view of the K cache as a reshape while preserving the dynamic cell axis.
* Flash Attention: Accept the MLA Q (`CONCAT`) pattern and V views derived from another view.
Limitations: The GPU plugin fails on batched compressed weights with static shapes. Keep these unsupported in op tests and `GGML_OPENVINO_FORCE_STATIC`.
* openvino : use writable recurrent cache inputs
Assisted-by: Codex
* ggml-openvino: fix decode crash for Qwen3.5 4B
When the dynamic size is 1, the code now keeps the innermost match. When the size is greater than 1, it still uses the MLA rule, so the MLA k_pe fix still works for multi-token graphs.
* ggml-openvino: keep the GDN state in the OpenVINO layout
Adds an opt-in switch, GGML_OPENVINO_GDN_NATIVE_STATE. When it's on, the GATED_DELTA_NET recurrent state stays in the layout the OpenVINO GatedDeltaNet op uses, [B, H_v, key_dim, value_dim], instead of the ggml layout [B, H_v, value_dim, key_dim]. That removes the two state transposes the translator adds on every call.
* ggml-openvino: support Ling-mini-2.0 group-routed MoE
Ling-mini-2.0 selects experts by group, but the group-selection graph
used ops that were not supported by the backend. This left 370 nodes on
the CPU and reduced decode to ~1.5 tok/s.
* Translate the batched 4D `GET_ROWS` gather used for group scores.
* Support `SET_ROWS` for compute tensors, not just KV caches.
* Preserve the dynamic token dimension in `FILL`.
* Skip the `DIV` repeat when dimensions already match.
* Use the tracked token dimension for `RESHAPE` / `GET_ROWS` op cases,
so a graph compiled during one-row warmup also works for ubatches
with more output rows.
* ggml-openvino: disable output staging
The backend stages small outputs (up to 4 MB) in a USM host buffer
and binds larger outputs directly to the ggml buffer. When the number
of outputs changes between calls to the same graph, as in
llama-perplexity, an output port can switch from USM host memory to
plain host memory.
The OpenVINO GPU plugin mishandles this switch for some dynamic
outputs, causing out-of-bounds writes on subsequent calls. This
results in `CL_OUT_OF_RESOURCES`. For example,
`llama-perplexity -c 2048 -ub 512` fails on the fourth ubatch with
Gemma-2, Gemma-4, Phi-3.5, Phi-4-mini, and Qwen3.5.
Disable output staging until the plugin issue is fixed, so each output
is always bound the same way. Input staging remains enabled.
* ggml-openvino: match attention masks by identity
Gemma-4 prunes non-output rows before its KV-shared layers. These
layers select the corresponding mask rows with GET_ROWS(mask, out_ids).
Mask parameters were named by their consumers, but GET_ROWS is not
recognized as a mask consumer. As a result, the sliding-window layers
selected their rows from the full-attention mask instead of the
sliding-window mask. This either caused the mask add to fail
broadcasting beyond the sliding-window cache (d1024 and above with
-ub 512) or allowed attention beyond the sliding window at shorter
contexts.
Track graph-input attention masks by identity and assign them the
same parameter name for every consumer. The SWA pre-pass now keeps
each layer's own attention mask, as the KV-shared layers read the
same cache with mask rows derived from it.
* ggml-openvino: preserve input mask names and window sizes
KV-shared layers appear at the end of the graph, and their derived
masks are not graph inputs. Processing these masks overwrote the
names and sliding-window sizes recorded from the original input
masks with empty values.
Preserve the names and window sizes of graph-input masks so the
attention dimensions continue to be derived from the original
mask shapes.
* ggml-openvino: accept empty tensors in stride checks
In Gemma-4 batches without output rows, the attention tensors of
KV-shared layers are empty. Their strides beyond the empty
dimension are zero, causing has_strides_on_element_grid() to reject
them.
This sent SOFT_MAX, the KQV MUL_MAT, and CONT to the CPU with
-fa off. It also caused the per-layer input crossing the graph
split to be bound with the wrong layout.
Accept empty tensors in the stride check. Their contents are never
read, so the remaining strides do not need to satisfy the element
grid constraints.
* ggml-openvino: classify transposed V caches using v_trans
With -fa off, the V cache is transposed. When the attended length
is the whole cache, as with a prompt that fits in one ubatch, its V
view is contiguous and was mistaken for the K cache layout.
The K layout path does not keep only the rows of the layer's own V
size. Gemma-4 allocates 512 V rows in every layer, the size of its
widest layer, and its sliding-window layers use the first 256. These
layers then attended with all 512 rows, and prompts fitting in a
single ubatch failed with a broadcast error.
Use v_trans to identify the V cache layout instead of relying on
contiguity.
* ggml-openvino: use per-layer Split in stateful execution
Stateful execution was excluded from the per-layer Split path
used for Gemma-4's per-layer embedding slices. The generic
re-slicing path therefore cut the token axis instead of the layer
axis.
Allow stateful execution to use the per-layer Split and adjust
its axis by the rank dropped during stateful graph construction.
* ggml-openvino: centralize stateful KQ mask lookup
Stateful execution rebuilds the KQ masks from positions. The
existing replacement logic applied to every mask input of
FLASH_ATTN_EXT and SOFT_MAX, including masks derived through
GET_ROWS.
For Gemma-4's KV-shared layers, this replaced the row-selected mask
with the full rebuilt mask, losing the output-row selection. As a
derived mask's name does not name the sliding window, these layers
also got the full-attention mask instead of the sliding-window one.
Add get_kq_mask_input() to replace only the actual mask parameters.
Use it for FLASH_ATTN_EXT, SOFT_MAX, and GET_ROWS, so the derived
masks select their rows from the rebuilt masks, and the lookup is no
longer duplicated.
* ggml-openvino: Removed duplicated if clause.
* fix flash_attn with sink conversion to OV SDPA
* ggml-openvino: drop the unused sink-without-mask path in FLASH_ATTN_EXT
supports_op already rejects FLASH_ATTN_EXT with sinks but no mask, so the
translator never sees a 4-input node whose fourth input is the sink; the
name check could only reach the assert after it. Restore the input-count
detection, fetch the sink where it is used, and format the sink reshape.
* ggml-openvino: fix dimensions bug in stateful recurrent-state remainder writeback
* ggml-openvino: move scalar scales before weight MatMuls on GPU and NPU
GPU and NPU run the graph in f16. Granite multiplies the attention and FFN
outputs by residual_scale right after their output projections. For some
tokens the ffn_down output is far above the f16 maximum (65504), so it
becomes inf and every later layer turns NaN: granite-4.0-350m and
granite-4.0-1b answer with repeated tokens on GPU, and llama-perplexity
reports NaN.
Add MoveScaleBeforeMatMul. It rewrites Multiply(MatMul(x, W), s), with a
scalar constant |s| < 1 and a constant W, as MatMul(Multiply(x, s), W). The
result is the same up to rounding and the MatMul output stays in range.
Register it for GPU and NPU; CPU runs in f32.
* ggml-openvino: translate convolution rollback snapshots
Assisted-by: Codex
* ggml-openvino: run GDN rollback snapshots in OpenVINO on CPU
Speculative decoding that rolls back recurrent state (MTP, EAGLE3, DFlash)
builds GATED_DELTA_NET with K = n_rs_seq + 1 state snapshots and copies them
into the recurrent cache. supports_op accepted the snapshots only on GPU, so
on CPU the GDN ran in ggml while the copy, whose destination is the OpenVINO
cache buffer, ran in OpenVINO. The copy read the packed ggml GDN output
wrongly and stored wrong states without an error: test-llama-context
rollback failed and MTP accepted no drafts.
translate_gdn_snapshots builds plain GatedDeltaNet ops on token prefixes and
does not depend on the device. Accept the snapshots on every device with
dynamic shapes and keep rejecting them only for static execution (NPU and
GGML_OPENVINO_FORCE_STATIC). Update the two doc sentences that said the
snapshots need the GPU.
* ggml-openvino: abort when GDN snapshots would run in ggml
The snapshot copy writes into the recurrent cache in the OpenVINO buffer, so
it always runs in OpenVINO. When OpenVINO rejects the GATED_DELTA_NET that
produces the snapshots (per-key gating as in Ling-3.0, a permuted v, static
execution), the GDN runs in ggml and the OpenVINO side reads its packed
output wrongly. Depending on the shapes, this silently corrupts the cache or
fails inference with a reshape error. Abort when the graph is scheduled,
with the reason.
* ggml-openvino: share one weights file and NPUW bank across all NPU prefill/decode pairs
With GGML_OPENVINO_NPU_WEIGHTLESS_BANK, every prefill/decode pair (one per KV window, e.g. llama-server's startup probe and a long prompt) wrote its own weights file and used its own NPUW bank, so import kept the weights resident once per pair (+2.1 GiB for phi-4-mini). Pairs now use a model-level <wfp>-<wfp>.weights.bin and bank; the weights file records a layout/content hash and is reused when identical, with a per-pair file and bank as fallback. Old caches are invalidated.
* ggml-openvino: export and preload NPU prefill blobs for every KV window
With head-major KV the prefill also runs at a KV window picked by prompt length, so an import-only run failed for prompts whose window was never exported. After a pair is exported, also export the prefill model of every other 1024*2^k window (weights resolved through the pair's shared weights file), and on import preload them into a process-wide cache of imported blobs that pair and decode-window builds reuse. Cache the source fingerprint of weights outside OpenVINO buffers, which made each model fingerprint cost ~9 ms. A window switch now costs ~60 ms instead of ~250 ms.
Rename GGML_OPENVINO_NPU_DECODE_PREBUILD to GGML_OPENVINO_NPU_BLOB_PREBUILD since it now covers prefill as well as decode windows.
* ggml-openvino: drop invalid prefill blobs and clear blob caches on backend free
The prefill preload imported each KV window's cached blob without a try, so
one blob that failed to import failed the graph compute, and the blob stayed
in the cache for every later run. Remove the blob and its manifest and log a
warning, as the pair import does; the window is built when a prompt first
needs it.
The imported-blob map and the fingerprints of weights outside OpenVINO
buffers lived until the process exited. A process that loads models one
after another kept every compiled model resident, and a later model whose
weight landed at a freed weight's address reused that weight's fingerprint.
Clear both with the shared compiled-model cache when the last backend is
freed.
Also apply clang-format to the changed lines.
* ggml-openvino: enable NPU KV windows by default
With GGML_OPENVINO_NPU_BLOB_PREBUILD unset, every NPU decode step ran a
static graph that attends the whole context, so decode was slow for any
sequence much shorter than -c. On a Lunar Lake NPU, Llama-3.2-1B decodes at
0.79 t/s with -c 4096 and 0.20 t/s with -c 16384, against 11.1 and 12.0 t/s
with KV windows. Each window adds one decode graph to compile at load: with
-c 4096, load goes from 11.1 s to 14.6 s for Llama-3.2-1B and from 61.2 s to
81.6 s for Llama-3.1-8B, whose decode goes from 0.20 to 0.82 t/s.
Default GGML_OPENVINO_NPU_BLOB_PREBUILD to 1; 0 keeps the full-context
decode graph. Update the env table and the Boolean-flag note, which said an
unset flag is always disabled.
* ggml-openvino: add POOL_1D and allow F32 EXP on GPU
* skip empty tensors
* ggml-openvino: detect the gated LFM2 conv input as a recurrent sequence
A reshape whose output reaches SSM_CONV through concat(conv_state,
transpose(x)) gets op_case 6, which keeps the active sequence count apart
from the token count, as the conv state does. The LFM2 and LFM3 short
convolutions gate x first, so the transposed tensor is
mul(view(x), view(x)) and the check missed it. The reshape got op_case 5
instead, and:
- in stateful mode, which drops the leading dim for op_case 5, TRANSPOSE
got a rank-3 input with a 4-axis order and graph conversion failed;
- with more than one sequence, the concat with the conv state failed.
Accept the gated input too. The output for a single stateless sequence
does not change.
* ggml-openvino: always stream weight requantization
Requantizing Q6_K/Q5_K weights to 8-bit (or F16 for some tensors) previously converted the entire tensor to f32 first. For large output or embedding matrices, this required 2–2.5 GB of temporary memory.
The chunked conversion path already processes 256 rows at a time and produces identical output. It was previously enabled by `GGML_OPENVINO_REDUCE_COMPILE_MEM` or `GGML_OPENVINO_MEMORY_OPTIMIZE`; use it by default whenever the target block size divides a row. Q4_1_64, which has no chunked quantizer, still converts the whole tensor.
The variable retains its other effects.
* ggml-openvino: allocate host buffers as untouched OS memory
Host buffers (weights, KV cache and compute) were allocated with `ggml_aligned_malloc` and explicitly zeroed with `memset`. This made the entire weight buffer resident before any tensors were loaded.
Allocate host buffers using `mmap(MAP_ANONYMOUS)` on Linux and `VirtualAlloc` on Windows instead. Both provide zero-initialized memory without touching the pages, allowing the OS to bring them into memory as tensors are written.
This preserves the zero-initialization required by ggml-org/llama.cpp#20857 while avoiding unnecessary memory residency during model loading.
Remove the cache-only allocation branches, the corresponding `ggml_aligned_free` path from the destructor and the `data_owned_by_ov` member it used, since they are no longer needed.
* ggml-openvino: release GGUF file pages once a weight is converted
Once `set_tensor` converts a weight into the backend buffer, its source pages in the mapped GGUF file are no longer needed for that buffer. Release these clean pages using `madvise(MADV_DONTNEED)` on Linux and `VirtualUnlock` on Windows. The mapping itself remains intact, and pages can be loaded again from the page cache or file if needed.
Only release whole pages within tensor ranges belonging to read-only file mappings. Skip buffers that wrap the mapping itself.
Keep locked pages resident. On Linux, `MADV_DONTNEED` fails on mlocked memory. On Windows, check page lock status with `QueryWorkingSetEx` and skip ranges containing locked pages, since `VirtualUnlock` would otherwise unlock them.
* ggml-openvino: key weight constants by name for the NPU model cache
The NPU compiled-model cache (GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR without the
weightless bank) imports a blob with the freshly converted model as ov::hint::model,
and NPUW finds each constant by its WeightlessCacheAttribute bin_offset. The keys came
from a per-model counter, and a constant that already had a key kept it. Weight nodes
are shared between models: between the NPU prefill and decode models, and across the
process through the tensor extras. So a weight kept the key from whichever model
converted it first.
The decode model for a KV window is built both as the decode half of the
prefill/decode pair and as a standalone window model, under one fingerprint, and the
two builds got different keys. Importing the blob of the other build failed with
"Couldn't find Constant in cache!", and the decode model was recompiled and exported
again on every run. With GGML_OPENVINO_NPU_BLOB_PREBUILD on by default, this happens
whenever a prompt reaches a larger KV window.
Key weight constants by a hash of the weight name and their position in the weight's
subgraph, above the range of the counter. Other constants are created per model and
keep the counter.
* ggml-openvino: drop CACHE_MODE=OPTIMIZE_SIZE from the NPU plugin cache
With GGML_OPENVINO_CACHE_DIR set, the NPU compile config added
CACHE_MODE=OPTIMIZE_SIZE next to NPUW_CACHE_DIR. With it, the NPU plugin
compiles the NPUW submodels with weights separation, which fails for them:
with NPUW folding, "No WeightlessCacheAttribute has been found in any of the
model's Constant nodes"; without folding, a driver compile error. So every
NPU run with GGML_OPENVINO_CACHE_DIR failed on its first compile.
Keep the default cache mode. NPUW_CACHE_DIR still caches the compiled
submodels.
* ggml-openvino: load each self-contained NPU blob once per process
A self-contained blob (GGML_OPENVINO_SELF_CONTAINED_BLOB) embeds its NPUW
weights bank, and NPUW cannot load a blob into a bank that already holds
those weights: Bank::read_and_add_tensor asserts. Two cases did this on
every run that imported the cache:
- Pairs that differ only in the prefill model share the decode model and
its bank. A long prompt builds such pairs, and each one imported the
decode blob again. The import failed, so the decode model was compiled
and exported again for every pair, until the run ran out of memory.
- The decode model of each KV window uses the bank of the pair's decode
model, so the import of a window blob failed and stopped the run.
Keep each self-contained model in the process once it is imported or
compiled, and reuse it for every pair with the same fingerprint. Turn the
KV windows off for self-contained blobs; the weightless bank keeps them.
* ggml-openvino: update the Docker image GPU drivers to 26.40.40196.8
The OpenVINO image pins the Intel GPU drivers at compute-runtime
26.35.39758.10. With that driver, gemma-4-E4B hangs on the Lunar Lake
GPU. With 26.40.40196.8 it runs.
Take the GPU driver pins from the compute-runtime 26.40.40196.8 release:
IGC v2.42.6, compute-runtime 26.40.40196.8 and gmmlib 22.10.2.
* ggml-openvino: fix long NPU prompts failing with KV slicing
With GGML_OPENVINO_KV_SLICE, on by default, a static prefill is compiled
at the attended KV length that llama reports for the ubatch. That length
grows with every ubatch of a long prompt, and each new length compiled
another prefill/decode pair, with GGML_OPENVINO_NPU_BLOB_PREBUILD (on by
default) also its decode KV windows. The shared compiled cache keeps
every pair until the backend is freed.
Each loaded prefill model keeps NPU memory for its inferences. On a Lunar
Lake NPU, a prefill model loaded next to two or three others cannot get
it: DRM_IOCTL_IVPU_BO_CREATE fails with "Buffer size is too big", and
pfnAppendGraphExecute returns ZE_RESULT_ERROR_UNINITIALIZED. With -c 4096
and the default -ub 512, a 2200-token prompt failed on smollm2-1.7B,
Phi-4-mini, Ministral-3b and gemma-3-4b.
- Round the attended length of a sliced static prefill up to a power of
two, clamped to the context. A growing sequence now compiles a prefill
model when its attended length doubles, not at every ubatch. The added
rows are masked, so the output does not change.
- When only the prefill attention size changes, the new pair takes over
the decode models (base and KV windows) of the entry's current pair,
because they do not depend on that size, and the current pair is
dropped before the new prefill model compiles. A pair that another
backend may use stays cached. Self-contained pairs are exported as one
blob pair, so they keep their own decode models.
- The taken-over KV window that covers the next decode position becomes
the base decode model, as in a pair built at that length, so the first
decode step does not have to create an infer request for it.
* ggml-openvino: pre-scale the GDN q input by default on the NPU
The NPU GatedDeltaNet kernel writes its output in FP16 and flushes
values below 6.1e-5 to zero, and the RMS norm after it amplifies the
loss. GGML_OPENVINO_GDN_Q_SCALE lifts the output above that floor, but
it was off by default. On a Panther Lake NPU with the compiler bundled
in OpenVINO, Qwen3.5-2B scores a perplexity of 14.77 without it and
13.00 with it (12.93 for the same weights on CPU), and its greedy chat
no longer loops while thinking.
- Enable the pre-scale by default with the factor 64.
GGML_OPENVINO_GDN_Q_SCALE=0 turns it off.
- Scale the eps of the norm only when the GDN that feeds it is
pre-scaled. A norm that reads the GDN output directly got the scaled
eps on every device, while q is scaled only on the NPU. With the
variable set, Qwen3.5 gave wrong output on CPU and GPU.
- Add the factor to the model cache fingerprint of NPU graphs with a
GATED_DELTA_NET, so a cached model is not reused across settings.
- docs: update the GGML_OPENVINO_GDN_Q_SCALE row and note.
* openvino-ggml: Decompose the GDN for LNL
1. Automatically decompose LNL GDN prefill into chunked MatMuls; retain the native op for decode.
2. Isolate lowering selection, fix positive-gate handling, and add overflow guards.
3. Update cache identity and add chunk-boundary regression cases.
* ggml-openvino: fix GDN decomposition in decode and on long prompts
Two problems remained with the GATED_DELTA_NET decomposition on a Lunar
Lake NPU.
The decomposition covered only multi-token (prefill) graphs, and decode
kept the GatedDeltaNet op. Under Linux the decode kernel loses the
device as well: after a successful prefill, the first generated token
failed with ZE_RESULT_ERROR_DEVICE_LOST in llama-completion, llama-bench
and llama-server.
The chunk masks were new constants in every converted model. The NPUW
weights bank is shared by all compiled models (NPUW_WEIGHTS_BANK=shared)
and identifies a constant by its data pointer, also after the model
that owned it is gone. When a longer sequence needs a second prefill
model, its masks can get the addresses of freed masks of the first one
and are then read as those. The Qwen3.5-2B perplexity over 512 tokens
changed between 12.99 and 56.94 from run to run, and a 1900-token
prompt gave garbage.
- Lower a single-token GATED_DELTA_NET to one recurrence step when the
decomposition is on: the chunk formulas for a chunk of one token,
where P = 1 and the decay is 1.
- Keep one instance of each chunk mask for the whole process.
- test-backend-ops: add a single-token case with two sequences, repeated
heads and a positive gate.
- docs: GGML_OPENVINO_GDN_DECOMPOSE covers single-token graphs.
* openvino-ggml: Enhance the safe env parsing
* ggml-openvino: keep the GDN q pre-scaling on when its variable is unset
ggml_openvino_gdn_prescale_enabled() returned false when
GGML_OPENVINO_GDN_Q_SCALE is unset, which switched the default-on
pre-scaling off. Qwen3.5-2B on a Panther Lake NPU went from a perplexity
of 13.00 back to 14.77.
- Treat an unset variable as enabled. Zero, negative and non-numeric
values still turn the pre-scaling off.
- Comments: ggml_openvino_getenv_int() no longer goes through std::atoi.
- docs: a boolean flag with a non-integer value is no longer always
disabled; a flag whose default is 1 stays enabled.
* ggml-openvino: use the PLUGIN compiler for GDN graphs on the NPU
The NPU driver compiler, the default, gives wrong output for graphs
with GATED_DELTA_NET (Qwen3.5). Qwen3.5-2B on a Panther Lake NPU answers
with unrelated text and has a perplexity of 16.77, against 13.00 with
the compiler bundled with OpenVINO. Other models stay on the driver
compiler, which builds faster prefill kernels.
- Compile static graphs with GATED_DELTA_NET with
NPU_COMPILER_TYPE=PLUGIN, unless GGML_OPENVINO_NPU_COMPILER_TYPE or
NPU_COMPILER_TYPE in GGML_OPENVINO_NPU_CONFIG sets the compiler type.
- Add the choice to the model cache key of these graphs, so blobs from
the driver compiler are not imported.
- docs: describe it in the GGML_OPENVINO_NPU_COMPILER_TYPE row.
* ggml-openvino: fix gemma-4 E2B on the Lunar Lake NPU at small contexts
The NPU 4000 compiler computes ScaledDotProductAttention wrongly when K
has one KV head and is square, that is when the attended rows equal the
head size. gemma-4 E2B has one KV head, 512-wide full-attention heads
and 256-wide sliding-window heads, so a context of 512 tokens or less,
and a prefill that attends 256 or 512 rows, gave wrong output.
For that shape on NPU 4000, build the attention from MatMul, Softmax and
MatMul, as the transposed-V path already does. Other shapes and devices
keep the op.
* ggml-openvino: support multiple GPUs with -sm layer
* ggml-openvino: check GGML_OPENVINO_* variables and reorganize the docs
The backend ignored a misspelled GGML_OPENVINO_* name and a value that
was not valid, and the docs listed every variable in one table.
Code:
- One table holds the known variables and their types. A value that is
not valid stops the backend with a message that names the variable and
the accepted values. An unknown GGML_OPENVINO_* name gives a warning.
- With log verbosity 4, one line shows the variables in effect.
- GGML_OPENVINO_GPU_CONFIG and GGML_OPENVINO_CPU_CONFIG pass plugin
properties to the GPU and the CPU, as GGML_OPENVINO_NPU_CONFIG does
for the NPU. If the plugin rejects a property, the error names the
variable.
- GGML_OPENVINO_STATEFUL_EXECUTION gives a deprecation warning.
- The key of the compiled model cache covers the two new variables,
GGML_OPENVINO_KV_SCATTER_ELEMENTS and
GGML_OPENVINO_DISABLE_ELTWISE_RANK_ALIGN.
- GGML_OPENVINO_ENABLE_CACHE, which no code read, is no longer
registered.
Docs:
- docs/backend/OPENVINO.md: Quick Start, collapsible blocks, runtime
configuration in groups (basic, plugin properties, advanced tuning,
compiled model cache, deprecated), updated validated models and known
limitations, corrected stale entries.
- dev_doc/OpenVINO-ENV-VARS.md: the variables for development and the
NPU properties that the backend sets.
* docs : add LFM2.5 and Ling-mini-2.0 to the OpenVINO validated models
* ggml-openvino: rebuild converted weights in get_tensor
set_tensor converts a quantized weight in place into the extracted or
requantized layout, so get_tensor returned bytes that are not in the ggml
format. A reader such as the CPU reference in test-backend-ops then
computed on garbage and produced NaN (MUL_MAT cases with weights=1).
get_tensor now rebuilds the ggml bytes for such a weight: it evaluates the
weight node on the host and quantizes the values back to the tensor type.
The result is close to the original data, not bit-exact. Other reads are
unchanged, and the inference path does not call get_tensor on weights.
* ggml-openvino: drop the GDN test cases from test-backend-ops
Restore tests/test-backend-ops.cpp to the upstream version.
* docs : improve the OpenVINO backend docs
* docs : restore the OpenVINO intro and the development note
* ggml-openvino: fix slow performance of MoE models with requantize and workaround to use fused MOE Compressed
* ggml-openvino: document the fused GPU MoE op limits
Converting K-quant MoE experts to Q4_1_64 on GPU lets the plugin use its
fused MoE op, which runs in f16 only. granitemoe models give wrong output
with it, and an f32 inference precision hint fails to compile. Document
GGML_OPENVINO_MOE_OP=0 as the way to turn the op off, and that
llama-perplexity with MoE models needs -b equal to -c.
Also split the long TODO comment over two lines.
* ggml-openvino : FILL for in-place clear of recurrent cache, sticky NaN fix
* restore this part
* ggml-openvino: bump the model-cache version for the recurrent-cache reset change
Recurrent-cache resets now clear slots with Select instead of Multiply,
so a NaN in a cleared slot no longer survives. Compiled models cached
before this change keep the old graph, so bump the fingerprint version
to rebuild them. Also note how SCALE and FILL store their parameters.
* ggml-openvino: read rope_factors_short/long as weight constants
create_weight_nodes() registers the ROPE frequency factors as
"rope_freqs.weight", but set_input_output() and compute_model_inputs()
kept the tensor's own name. For models whose factors are named
rope_factors_short or rope_factors_long (Phi-3.5, Phi-4, MiniCPM with
LongRoPE) the factors became a Parameter fed from the host weight buffer.
That buffer is empty in cache-only mode and after the host weights are
released, so these models gave garbage output with
GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY, GGML_OPENVINO_RELEASE_WEIGHTS
and GGML_OPENVINO_MEMORY_OPTIMIZE.
Use the same name in all three places, so ROPE always reads the weight
Constant. The graph changes, so bump the model-cache version to 7.
* ggml-openvino: import cached models from a memory-mapped tensor
A stream import reads the whole compiled blob into a heap buffer while
the plugin builds its own copy of the weights. On CPU this held the blob
twice at the peak. Import from ov::read_tensor_data() instead, which maps
the file. This lowers the CPU cache-hit and cache-only peak memory; GPU
memory is unchanged.
* ggml-openvino: do not write CPU model-cache blobs for MoE models
The CPU plugin exports its MoE op (GatherMatmulCompressed) but cannot
import it again with OpenVINO 2026.4 and 2026.5 ("unsupported opset").
Every later run then failed the import, compiled again and rewrote the
blob. Skip the blob for such models on CPU and print a warning.
In cache-only mode no blob exists for them, so the generic advice to run
once without cache-only cannot help. Print an error that says cache-only
does not work for MoE models on CPU instead.
* ggml-openvino: fix GGML_OPENVINO_DEBUG_INPUT on GPU
GPU inputs can be device (USM) tensors, whose data() throws, so every
GPU compute failed with GGML_OPENVINO_DEBUG_INPUT=1. Print a host copy
of remote tensors instead.
* docs : add OpenVINO memory optimization tips and MiniCPM-V-4.5
Add a "Tips for Memory Optimization" section to OpenVINO-ENV-VARS.md
with examples for the spill directory, the compiled-model cache, host
weight release on GPU, CPU MoE compile memory and the NPU weightless
cache, and link it from OPENVINO.md. Add MiniCPM-V-4.5 to the validated
models.
* ggml-openvino: run MoE models with F16 and BF16 weights on GPU
On GPU, supports_op rejected MUL_MAT_ID with non-quantized expert weights
and GET_ROWS with BF16 data, so the scheduler ran those ops on the CPU.
For F16 MoE models the CPU MUL_MAT_ID then read the strided top-k view as
expert ids and crashed. For BF16 models the token embedding came from the
CPU and reached the OpenVINO graph as a static-shaped input, which failed
once the token count changed. Both ops translate correctly on GPU, so
remove the two rejections.
* ggml-openvino: fix BERT embedding models on CPU and GPU
BERT models (bge-m3, nomic-embed-text-v1.5) failed on every device:
- A VIEW of a weight with a GGML_OP_NONE source was marked as having a
dynamic token dimension, a rule meant for KV cache views. The token type
row view of BERT was then never sliced, and the whole two-row table was
added to the activations. Skip the rule for model weights.
- A row range view of a 2D weight now becomes a Slice of the weight rows,
as GET_ROWS already does.
- Attention without a KV cache pads K and V up to the mask width, which
depends on the token count. PAD baked the amount from the first graph,
so later token counts gave mismatched shapes. Compute the end padding
from the mask width at run time.
The NPU still fails for these models.
* docs : add bge-m3 and nomic-embed-text-v1.5 to the OpenVINO validated models
---------
Co-authored-by: Sabnis, Omkar <omkar.sabnis@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yu, Zijun <zijun.yu@intel.com>
Co-authored-by: Proshunin, Nikolay <nikolay.proshunin@intel.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
* ui: Llama UI's own Storybook in ui/lib/ui
The library gets its Storybook (`npm run storybook`, `npm run
build-storybook`) with a story for everything it holds, under Llama UI:
the foundations, the 29 base components (as actions, forms, display,
overlays and navigation), the app components (ActionIcon, SearchInput,
SyntaxHighlightedCode), the Model UI and the native app mockups. The
stories import the package by its published names, aliased onto src/lib,
run as tests under `npm test` (the storybook project in vite.config.ts),
and are type-checked with the rest of the package. `npm run dev` starts
Storybook next to the SvelteKit check page; the toolbar switches the
color mode and the surface theme, and pages that are mostly prose are
.svx, through mdsvex.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: a Storybook for Llama Chat
Llama Chat's Storybook gets an introduction, Agents › Design guidelines
(DESIGN.md), Llama UI's stories as the library has them, and Llama Chat:
its own foundations (safe areas, the splash screens), then one section per
src/lib/components/app/<area> (Chat with the chat screen, form, messages,
attachments and tabs; Content, Models, MCP, Navigation, Settings, Dialogs,
Forms, Actions, Badges, Misc, Servers and PWA), each accessibility story
next to its component's. It replaces the flat pages it had (chat message,
chat form, markdown content, the PWA refresh alert and their fixtures,
and the ChatForm single-row exploration).
The stories run on the new tokens and surfaces with the color mode and
surface theme in the toolbar (by mode-watcher's key, the ColorMode and
SurfaceTheme enums and applySurfaceTheme), and pages that are mostly
prose are .svx in an editorial frame (AGENTS.md says how to write them
and where a story goes). The story tests run Llama Chat's stories and
leave the library's to the library. The brand and communications pages
are in ggml-org/llama-brand; the logo files come from @ggml-org/llama-brand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let the ChatScreen story store its reply before the next one starts
The ContinueConversation story ended while its reply was still being stored,
and the next story reset the stores under it. It now waits for the reply to
finish.
Assisted-by: Claude (Fable 5.2)
* ui: publish the Storybook to its Hugging Face Space on every UI change
A new UI Storybook workflow builds ui/apps/chat's Storybook whenever
ui/apps/chat or ui/lib/ui changes on master (or on demand) and uploads it
to the static Space ggml-org/Llama-UI-storybook, with a Space card naming
the commit it was built from. Files the new build no longer has are
removed; the Space's .gitattributes stays. It runs only in
ggml-org/llama.cpp, takes the UI's Hugging Face token
(HF_TOKEN_UI_STATIC_OUTPUT), and the Space can be changed with the
HF_SPACE_UI_STORYBOOK variable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: run the library's tests, its stories among them, in CI
The Library checks job type-checked and packed ui/lib/ui but never ran its
tests, so the library's stories, which now hold every library component's,
went untested in CI. It installs Chromium and runs `npm test`.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw a floating panel's rim on its border, so its edge is one line
Menus, popovers, dialogs and the other matte-float panels carry a 1px border that the
surface made transparent, and drew their inner highlight as an inset shadow. An inset
shadow sits inside the border, so the 1px of fill between it and the outer ring showed
as a second edge. The border now draws the highlight, right inside the ring.
Assisted-by: Claude (Opus 5.5 Medium)
The implicit GEMM conv2d kernel recovers the channel and the kernel
position of each reduction index with three integer divisions by
runtime divisors, once per loaded row of every tile. Outside the 3x3
and 1x1 specializations this integer division bounds the kernel:
removing it halves the time on long reductions.
Divide with the precomputed magic numbers of fastdiv instead. The 3x3
and 1x1 specializations keep their compile time divisors.
test-backend-ops -o CONV_2D: 2006/2006 on CUDA0.
CONV_2D, F16 kernel, us/run on an RTX PRO 6000:
[2054,1,768] k [7,1,768,768] 221 -> 138
[16390,1,384] k [7,1,384,384] 433 -> 266
[65542,1,192] k [7,1,192,192] 420 -> 271
[8240,1,256] k [11,1,256,256] 165 -> 108
[19,19,256,16] k [4,4,256,4096] 1672 -> 1076
* server: fix lora and recurrent
* tests: run the hybrid SWA checkpoint test with one token per ubatch
With and without the cache, the prompt was not split into the same
ubatches, and the generated tokens are decoded one at a time while the
uncached run processes them in a batch. On CUDA this moves the top
logprobs of the tiny model by up to 0.03 and swaps near ties, so the
test failed while the cache behaved as expected.
With one token per ubatch both runs go through the same kernels and
the output is compared exactly.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ui: publish @ggml-org/llama-ui from CI when its version moves
ui/lib/ui reaches npm only when someone remembers to publish it by hand, which let 0.3.0
sit a day behind master. The version in its package.json becomes the release: a commit
landing on master with a version npm does not have gets built, linted and published with a
provenance attestation, through OIDC rather than a stored token, as release-publish.yml
already does for GitHub releases. A library change carrying no bump publishes nothing and
says so, on the PR and on the run.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: bump @ggml-org/llama-ui to 0.4.0
The library on npm, published before the design system PR merged, holds no entry points for
the Model UI, the app components or the shared constants and enums, and no dependency on
@ggml-org/llama-brand, whose colours the primitives now build on. Consumers import the new
subpaths and get ERR_PACKAGE_PATH_NOT_EXPORTED.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: name no environment for the Llama UI publish job
Creating a repository environment needs admin rights the UI maintainers do not hold, so a job
that names one cannot be set up by them. The job works without it: npm's trusted publisher
matches on the repository and the workflow file, and its environment field stays blank to
match. Nothing else in the job depends on an environment.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: address review on the Llama UI publish job
- the advisory version check leaves ui.yml for the publish workflow, on a pull_request trigger
whose path filter keeps it off PRs that touch only Llama Chat: a chat-only PR was told to
bump a library it did not change
- publish and verify run on master only. The trusted publisher matches on the repository and
the workflow file, not the branch, so a dispatch from a PR branch - anything with write
access can start one - could publish code no one reviewed. With no environment, this was the
only branch protection in place.
- the registry read after a publish retries six times rather than sleeping once, so a release
the registry is slow to serve is not reported as a failure
- a build metadata suffix (1.2.3+build) is not a prerelease, and versions order by semver's
prerelease rules, so 0.4.0-beta.1 behind a released 0.4.0 is refused instead of shipped to next
- the summary step passes its values through env like the others, actions: read is gone, and the
concurrency comment claims only what the setting does
- npm older than 11.5.1 is refused up front, as it cannot publish with an OIDC token
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: note the two trusted-publishing traps in the Llama UI README
A publisher names a workflow file that must already be in .github/workflows, so it is added
after this workflow lands rather than before, and one created now publishes only through
staged publishing until npm publish is permitted for it. Provenance comes free with either,
and only on a GitHub-hosted runner.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* Update .github/workflows/ui-lib-publish.yml
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* jinja: impl scope-chain key lookup
* add test for call_statement lambda
* chat: add tts_dict for kokoro G2P template
* more testing
* value_float_t range check
* no json-in-meta, add llama_model_meta_arr_str/_n
* mtmd: add kokoro tts support
* detailed guide about G2P and text norm port
* refine G2P
* clean up
* migrate batch api
* nits
The fast CONV_2D path (implicit GEMM with MMA, or cuBLAS for
pointwise convolutions) required an F16 kernel. F32 kernels fell
back to the naive direct convolution, which is much slower than
im2col + mul_mat on large channel counts.
Convert F32 kernels to F16 in a pool buffer, as is already done for
the input, and take the same path with F32 accumulation. The Metal
implicit GEMM kernel already stages both operands as F16.
* Added support for Qwen3-TTS ICL/reference mode
Assisted-By: Claude Opus 5.5
* fix correctness
* attach transcription to bitmap
* nits
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* sampling : fix crash when both ignore_eos and grammar are on
* server : rework, throw 400 when grammar and ignore_eos are on
* sampling : move the check into the sampler (review suggestion)
* tests : remove redundant tests
As discussed in #31955, the live preview is now opt-in. With it on, the editor renders blocks as widgets, and moving the cursor down from the first line of a prompt with a few blocks (a heading, a list, a code block) jumps to the last line; with it off, the cursor moves one line at a time. Users with a saved config keep their current value, since the full config is stored in localStorage.
The message input also gets spellcheck, autocorrect and autocapitalize back. CodeMirror turns them off by default, so they were off with the preview on or off; the former textarea and contenteditable inputs had the browser's defaults.
* llama : keep the k-pool layout prefix on sequence edits
Assisted-by: Claude Code
* llama : address review of the k-pool layout update
* Append and edit share one branch, using stale > sq.pos_min.
* LLAMA_KPOOL_VERIFY=1 checks every update against a full rebuild. The scan moved into kpool_scan() so the check can reuse it.
* The test turns the check on for its rollback context, skips -sm tensor, and drops kimi-k31.
* Two comments that were no longer true are fixed.
Assisted-by: Claude Code
* llama : check the k-pool layout update in debug builds instead of LLAMA_KPOOL_VERIFY
* llama : remove the k-pool layout verification
* llama : inline the k-pool scan again
* remove dangling new line
* do not search twice
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
* ui: bring @ggml-org/llama-ui into ui/lib/ui
The design system package (tokens, fonts, surfaces, components and Model UI), so far in the temporary ggml-org/llama-ui
repository at edaf518, moves to its home in ui/lib/ui, built with @sveltejs/package and published as @ggml-org/llama-ui
from here. ui/lib holds the packages, one folder each. Its references to tools/ui now name ui/apps/chat, its
package.json points at this repository, and the UI workflow type-checks and packs it. ui/README.md describes the
layout.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: Llama UI design system: tokens, surfaces and brand
The design system from allozaur/docs/storybook-overhaul: three tiers of
tokens, matte glass surfaces, Llama Sans and Llama Mono, and the base
components on the new tokens (Tabs and GlassKey among them). It also
restyles the stack's features with it, replacing the plain stand-ins they
used until now. DESIGN.md is generated from the tokens. The Storybook that
documents it is the next PR.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: have Llama Chat use @ggml-org/llama-ui for its base components, tokens and fonts
Llama Chat kept its own copy of the design system: 29 base components, the
three tiers of tokens, the surfaces, the theme and the fonts, next to the 12
components ui/lib/ui held. The copy goes, and the library gets what it lacked.
ui/lib/ui
- the 17 base components the app had and the library did not (alert,
alert-dialog, button-group, checkbox, collapsible, context-menu, dialog,
drawer, dropdown-menu, empty, hover-card, radio-group, scroll-area,
skeleton, table, tabs, textarea), with vaul-svelte for the drawer
- the inset-group and selected-row tokens and surfaces
- sources import each other by relative path instead of $lib, which in an
app points at the app's own src/lib
- theme.css no longer imports tw-animate-css: it is a peer dependency that
the consumer imports before the theme, as it does Tailwind
- 0.3.0
ui/apps/chat
- imports the base components as @ggml-org/llama-ui/ui/<name> and cn/tv as
@ggml-org/llama-ui/utils; app.css imports the library's theme.css
- compiles the library from source, with no install or build of it:
scripts/vite-plugin-llama-ui.ts resolves its entry points and what it
imports, and svelte.config.js gives TypeScript the same paths
- src/lib/components/ui, styles/tokens, styles/surfaces.css and assets/fonts
are removed; components.json moves to the library
Builds
- CMake stages ui/lib/ui next to the app, keeping the layout of ui/, and
rebuilds when a library source changes
- the Dockerfiles copy the library's sources; the Nix derivation takes ui/
as its source
The built CSS has the same rules in the same order, apart from one
zero-specificity rule and the @font-face rules moving, plus ten utilities
and two tokens that only the library's own components use.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: AGENTS.md places Llama UI in ui/lib/ui
Llama Chat's AGENTS.md says where each part lives: Llama UI is
ui/lib/ui, and Chat imports its tokens, fonts and base components
from there.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: Llama Sans and Llama Mono from llama-brand's latest, and shell commands marked in bash code
The fonts in ui/lib/ui come from llama-brand's latest, with optical sizes
from 10 to 36. Bash code blocks mark shell commands and flags.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open submenus outside the menu they come from
A submenu was rendered inside its parent menu, so the menu's scroll box
clipped it: the glass menus' backdrop filter, and the filter and transform
of the open animation, make the menu the containing block even for the
fixed-position submenu. Dropdown and context submenus now render in a
portal, like the menus themselves.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw submenus like the menu they open from
Submenus had smaller corners than their menus, and context menus had none
of the dropdown's glass. Context-menu parts now match the dropdown's, and
both submenus take the menu's corners and height cap, with their first row
level with the row that opens them, 4px past the menu's edge. Sub-triggers
take the items' radius and pointer.
The submenus' height cap and zoom origin named the dropdown's CSS variables,
which bits-ui does not set on submenus; they now use its --bits-menu-* ones.
The chat group submenu takes the actions menu's z-[999999] layer, as it no
longer renders inside that menu.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a late current-node write off a closed or switched conversation
updateCurrentNode checked the active conversation, awaited the database write, then wrote currNode to whatever was active after it. When the conversation closed during the write it threw, and the reply ended in a streaming error dialog. When another conversation opened, that one got the wrong node.
Assisted-by: Claude (Fable 5.2)
* ui: have Llama Chat use the library's Model UI and app components
Llama Chat still kept its own copies of what the library holds above the
base components. It now takes them from @ggml-org/llama-ui too, and its
copies go.
ui/lib/ui
- ModelId gains what Chat's had: the model's own draft kind, draft
sidecars with their quant (ModelDraftSidecars, new), aliases, hideName
and a tooltip, in the same badge order
- entry points for them, like ui/<name>: @ggml-org/llama-ui/app/<group>
(actions, content, forms, misc) and @ggml-org/llama-ui/models
ui/apps/chat
- ActionIcon, SearchInput, SyntaxHighlightedCode, Logo,
ModelCapabilityIcons, ModelDownloadProgressBar and ModelDraftSidecars
come from the library; Chat's copies and the two constants only they
used are removed
- ModelId and ModelOrgAvatar stay as thin wrappers: ModelId parses the raw
id and applies the user's display settings, ModelOrgAvatar hides the
avatar while Hub metadata is off
- the svelte/* TypeScript path falls back to Svelte's types/index.d.ts:
svelte/reactivity has no file of its own, so a library source importing
it got its types from ui/lib/ui's own install when there was one
Assisted-by: Claude (Opus 5.5 Medium)
* ui: Llama UI takes the brand's colours and logo from llama-brand
@ggml-org/llama-brand is now a dependency of the library, by the same GitHub
tarball Llama Chat already installs, which now names it @ggml-org/llama-brand
too.
- primitives.css drops the 25 tokens it held copies of (the neutral
greys, sand, the skies and cinder) and builds on llama-brand's
css/colors.css instead; the skies' gradients point at llama-brand's.
Apps import that file before theme.css, as they do Tailwind and
tw-animate-css: the library's stylesheets import no package, so an app
that compiles them from source resolves it from its own node_modules
- Logo draws llama-brand's icon, in currentColor, and assets/logo.svg goes
- Llama Chat's TypeScript paths resolve llama-brand from its own
node_modules, like the library's other dependencies
The fonts stay as the library's WOFF2 copies until llama-brand ships
WOFF2 builds of them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: have Llama Chat take the library's constants, enums and helpers
Llama Chat redefined most of what the library exports next to its
components. Where the two were the same, Chat now re-exports the library's
from the files that held them, so its own imports stay as they are.
- the library gets entry points for them: @ggml-org/llama-ui/constants,
@ggml-org/llama-ui/enums and @ggml-org/llama-ui/code, and
@ggml-org/llama-ui/models carries the model data with the Model UI
- from the library now: the shared CSS classes and code-block constants,
the byte units, the memory-fit constants and minMemoryTierGb, the bit
depth labels, the capability and modality icons, labels and order,
ColorMode, TooltipSide, BooleanString, the model enums, ModelModalities,
formatParameters, highlightCode and trimCodePadding
- the library's modality order follows Chat's: vision, video, audio
- Chat keeps its own where they differ: UI_DATA_ATTRS and
ModelDownloadProgress are supersets of the library's (the latter extends
it now), and formatFileSize counts in 1024s
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the chat form's add key and model selector a tighter shadow
In light, the two keys inside the chat form, white on its white glass, took
the control elevation, whose soft spread blended into the form. They take a
new elevation-key instead: a 0.5px hairline all round and a 1px drop. In
dark they keep the control elevation. Chosen from the options in
Scrapbook › UI › Composer keys.
- semantic tokens: --elevation-key (light: the hairline and the drop; dark:
the control elevation)
- component token: --chat-form-key-shadow, for the add key and every model
selector trigger
- DESIGN.md names it next to the other elevations
Assisted-by: Claude (Opus 5.5 Medium)
Browsers already have tabs, so a second tab bar inside the page adds a layer most people don't need. The setting stays, so anyone who wants tabs can turn them on. Users with a saved config keep their current value, since the full config is stored in localStorage.
* ui: replace deprecated packages and fix npm audit findings
Swap the mdast stub for @types/mdast, move the katex override to 0.19.0,
and override glob, source-map and html-encoding-sniffer under
workbox-build and http-server so installs no longer pull deprecated
packages. Apply the non-breaking npm audit fixes to the lockfile.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: migrate Llama Chat to SvelteKit 3 and its built-in service worker
Move to SvelteKit 3, Vite 8, TypeScript 6 and Storybook 10.6. The
SvelteKit options move from svelte.config.js into sveltekit() in
vite.config.ts, $lib imports become #lib subpath imports from
package.json, and $app/environment becomes $app/env. Modules named
server.* or living under a server/ folder are server-only in SvelteKit 3,
so the llama-server components, store and enums move to llama-server
names.
@vite-pwa/sveltekit and Workbox give way to SvelteKit's own service
worker in src/service-worker, registered by usePwa(), which precaches the
app files and leaves API requests to the network. A small plugin emits
manifest.webmanifest. static/sw.js removes the worker of older builds, and
llama-server serves a self-removing service-worker.js as well as sw.js
when the built-in UI is off or replaced.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: use SvelteKit 3's update detection and Svelte 5's current patterns
A page the service worker doesn't control now learns of a new build
from SvelteKit itself: `updated` from $app/state checks
_app/version.json on navigation, on focus and every hour. It replaces
the version Llama Chat kept in localStorage and compared by hand.
Reload on the update prompt pulls in the new worker and activates it
before reloading, so one click lands on the new build, whether or not
a worker controlled the page.
Following Svelte's current guidance:
- usePwa keeps its state in $state rather than a store, and no effect
checks `browser`
- nearViewport, autoResizeTextarea and the custom CSS injection are
attachments ({@attach}) rather than actions (use:)
- class lists use arrays rather than class: directives
- the chat contexts come from createContext, typed, in place of
setContext/getContext with string keys
- ActionIconCopyToClipboard takes $props() rather than `export let`
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move to ESLint 10
ESLint 9 is out of support. Every plugin the app uses supports 10:
typescript-eslint, eslint-plugin-svelte, perfectionist,
simple-import-sort, eslint-plugin-storybook and eslint-config-prettier.
@eslint/js and @eslint/compat follow.
ESLint 10's recommended rules ask for two things the code now does:
errors rethrown from a catch keep the caught error as their `cause`,
and no variable starts with a value that every path overwrites.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: override minimatch to clear npm audit's last finding
ESLint 10.14, @eslint/config-array 0.23.5 and typescript-eslint 8.72
depend on minimatch 10, which GHSA-g3hv-75wg-m249 (denial of service
via globparts amplification) affects below 11.0.1. Their fixed releases
move to minimatch 11 but are all younger than the repo's 7-day
min-release-age, so an override pins minimatch 11.0.5, the newest the
rule allows. The three only call new Minimatch(), match() and
minimatch(), which minimatch 11 keeps. npm audit now finds nothing.
Assisted-by: Claude (Opus 5.5 Medium)
This commit removes a call of the CUDA_SET_SHARED_MEMORY_LIMIT macro as
the fallback decision was moved out of launch_mul_mat_q.
launch_mul_mat_q now has the fallback template parameter which was added
in Commit 6eddde06a4 ("CUDA: refactor MMQ
kernel configuration (#24127)").
This code was then refactored in Commit
c49edf0963 ("CUDA: fuse FFN gate/up
matmuls and GLU in MMQ with interleaved gate/up rows (#29948)") which
added the has_fusion template parameter but it kept both macro calls.
launch_mul_mat_q only launches one type of kernel now (this is now
decided earlier in the templated call chain) and each specialization
will set the limit itself.
The motivation for this is to simplify the code and improve readabilty.
* fix the SYCL fa with mkl error of a mask broadcast over heads or batches; add argument cache-ram to start-svr.sh
* add argument cache-ram to win-start-svr.bat
* fix format issue, remove --cache-ram in script: move to another PR
* Only run webgpu CI on my fork
* Add webgpu only workflow
* resolve merge conflicts
* save state
* save kernel
* add kernel back in
* fix merge issues with kernels
* updates
* update tests
* change invocation count and workgroup to be u32
* clang format
* clang format
* update docs
* format
* remove extra file
Replace the per-element modulo of i0 with fast_div_modulo in the ravel and
unravel bin-bcast kernels, drop the now-unused ne10/ne20 kernel arguments,
and reject divisors that do not fit in 32 bits when the magic values are
computed, as the CUDA implementation does.
Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
* mtmd: support YuE2-3B
* improve speed
* add CFG
* fix streaming
* add cfg control from api
* check ctx size
* fix n_predict
* add flow_steps config
* nits
* fix batch api
* nits
* adapt
* mtmd: yue2 samples from its own saved logits
The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.
* mtmd: let the yue2 snake chain fuse
The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* opencl: release the WMM flash-attention mask fallback image
The fallback image is created on every call of the bin flash-attention
kernel over a sub-buffer of the K cache and was never released, so each
call kept a K-cache sub-buffer alive. Every call with more than one query
row takes this path, including each speculative-decoding verify step, so
the objects pile up and the server slows with every request.
* opencl: keep q8_0 K/V at head size 512 off the WMM flash-attention kernel
With q8_0 K/V at DK=512 the bin flash-attention kernel gives wrong results
whenever V holds different values from K (test-backend-ops FLASH_ATTN_EXT
hsk=512 q8_0, max error 2.04 on X2-90); f16 K/V at DK=512 is correct. The
backend's own flash-attention kernels already decline q8_0 at DK=512, so the
shape is now handled the same way with or without the kernel library.
* opencl: decline WMM flash-attention prefill beyond the device's 3D image limits
The WMM path repacks Q/K/V into 3D images; V is n_kv wide, so past the device's
maximum 3D image width (about 16k tokens of context on X2-90) clCreateImage
failed and the call asserted. Such shapes now take the regular flash-attention
path.
* ui: move tools/ui to ui/apps/chat
The chat app moves to its own root folder, ui/apps/chat, as the first step of splitting the UI into one library
(ui/lib) and the apps built on it (ui/apps). The code is unchanged; every path that pointed at tools/ui now points at
ui/apps/chat: CMake adds it from tools/, and the Dockerfiles, the Nix package, the workflows, the labeler, CODEOWNERS,
.editorconfig, the git hooks, the dev and demo scripts and the docs follow it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name Llama Chat's package @ggml-org/llama-chat
It was still called llama-ui, the name of the library that now sits beside it
in ui/lib (@ggml-org/llama-ui). The package stays private; the app's display name
and the llama-ui.zip build artifact are unchanged.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: call the built-in web UI Llama Chat in docs, help text and comments
The app in tools/ui is Llama Chat; Llama UI is the component library it is
built from. Prose that said "Web UI", "web UI" or "WebUI" for the app now
names it: the READMEs, SECURITY.md, the --ui help text, the CMake option
description, one server log line, code comments and the Storybook intro.
Names that others depend on are unchanged: the deprecated --webui flag
aliases, the LLAMA_*_WEBUI CMake and environment names, the webui_settings
field, the storage migration keys and the Nix attributes.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name tools/ui as a former path of Llama Chat in the code-review skill
Changes under the old tools/ui path, as well as tools/server/webui, still
get the Llama Chat checklist.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name Llama Chat's path in the security policy's exclusions
Llama Chat no longer lives under tools/server/, so the exclusion listed
under it now names ui/apps/chat. What is covered stays the same.
Assisted-by: Claude (Opus 5.5 Medium)
The flash attention shaders indexed the mask without its head
dimension, so masks with ne[2] > 1 gave wrong results. The mask offset
and the vec block states now include the head, broadcast as
head % mask->ne[2] like the other backends. The params buffer grows to
256 bytes to fit the two new mask fields.
The shared memory estimate dropped kv_shmem when only one of K and V
is read directly, while the shaders keep it unless both are, which made
the K f16 / V q8_0 cases exceed the workgroup storage limit. The vec kv
tile now takes both flags: the memory estimate drops kv_shmem only when
both are direct, and the tile still divides the KV padding when either
is. get_alloc_size and supports_op use the same rule as the pipelines,
and supports_op accounts for the register tile fallback.
* ci : use vX.Y.Z tags as docker tags
When the Docker workflow is manually dispatched from a vX.Y.Z tag, use that tag verbatim for the image tags instead of appending the build number and commit hash.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ci : require a tag for manual docker runs
Manual dispatches of the Docker workflow must now use a tag ref, so the image tags are derived from the selected tag. The check can be skipped with the new skip_ref_check input. Scheduled nightly builds are unchanged.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* vulkan: make the radix top-k tie selection deterministic
The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.
Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.
* optimize shader for AMD, Intel and Nvidia
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.
Fixes#30576
Assisted-by: GitHub Copilot
Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.
Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.
Also trim the output_config.effort test to a single level.
* ggml-cpu: use VNNI for Q2_0 dot products
Keep the upstream AVX2 unpacking, accumulation, and fallback. Use VNNI for the integer dot product on supported x86 CPUs.
Assisted-by: Codex
* ggml-cpu: use VBMI for Q2_0 unpacking
Use multishift to unpack Q2_0 weights on CPUs with AVX-512 VBMI and VL. Retain the existing shift-and-mask unpacking on other CPUs.
Assisted-by: Codex
* Update ggml/src/ggml-cpu/arch/x86/quants.c
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* fixed support for DFlash2 for Ripple 65B
* spec: Add NoPE handling to DFlash
(cherry picked from commit 89ea8e3e77e3359d17902a2db136022d5d419d88)
* spec: Fix NoPE not being picked up by DFlash
(cherry picked from commit 43393b95fda2ce7e7d243e2650bf7b1cadf1c732)
---------
Co-authored-by: James Pickering <james_pic@hotmail.com>
* Fail tests on missing or unsupported type
* Fix output
* Remove whitespace
* Fail if no tests ran regardless of backend support
* Revert unnecessary cosmetic change
* lfm3 : put the normed tensor first in residual adds to enable norm fusion
Assisted-by: Claude Opus 5.5
* tests : update Metal fusion baseline for lfm3
Assisted-by: Claude Opus 5.5
* convert : add Inkling2 text GGUF conversion
Assisted-by: Codex
* llama : implement Inkling2 text inference
Assisted-by: Codex
* tests : cover Inkling2 text support
Assisted-by: Codex
* conversion : dequantize packed Inkling2 experts in row chunks
Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.
Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.
Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.
Assisted-by: Codex
* convert : add Inkling2 vision mmproj support
Assisted-by: Codex
* mtmd : implement Inkling2 still-image encoder
Assisted-by: Codex
* inkling2 : scope image loading
Assisted-by: Codex
* inkling2 : enable flash attention with folded bias and sinks
Assisted-by: Codex
* cuda : route per-head flash attention masks through vec
Assisted-by: Codex
* mtmd : support Inkling2 temporal frame means
Assisted-by: Codex
* style : restore shared image comment
Assisted-by: Codex
* tests : enlarge Inkling2 context fixture
Assisted-by: Codex
* inkling2 : load per-layer expert feed-forward widths
Assisted-by: Codex
* tests : add Inkling2 Metal fusion baselines
Assisted-by: Codex
* inkling2 : group and align registry entries
Assisted-by: Fable
* convert : regenerate the Inkling2 tokenizer hash with the update script
Assisted-by: Fable
* conversion : revert the print_registered_models change
Assisted-by: Fable
* conversion : sort the Inkling2 model map entries
Assisted-by: Fable
* convert : map Inkling2 vision checkpoint names in the tensor table
Assisted-by: Fable
* convert : map Inkling2 text checkpoint names in the tensor table
Assisted-by: Fable
* tests : document the Inkling2 packed expert conversion test
Assisted-by: Fable
* tests : cover Inkling2 checkpoint and transformers tensor names
Assisted-by: Fable
* llama : place Inkling2 fields and list entries by file convention
Assisted-by: Fable
* gguf-py : align Inkling2 tensor mapping comments
Assisted-by: Fable
* mtmd : place Inkling2 entries by file convention
Assisted-by: Fable
* tests : place Inkling2 cases after the existing ones
Assisted-by: Fable
* cuda : index per-head flash attention masks in the tile and mma kernels
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : project the relative bias before the gather, drop arch branches
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* convert : require Inkling2 hparams, add inkling2 pre-tokenizer
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* mtmd : cap Inkling2 image size, drop video frame averaging
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : fix CI build, Meta device assert and Windows conversion test
Assisted-by: Claude
* tests : use the default sliding window for the Inkling2 test model
Assisted-by: Claude
* simplify mtmd part
* give _v arch for vision
* remove alpha channel handling, it should be a follow up
* explicit rule about audio + vision models
* add audio support
* rm audio
* rm gguf mtmd rope base
* address review comment
* nits
* do not normalize embd from mtmd
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* common : reject out-of-range grammar_triggers type instead of aborting (#30835)
* tests : remove the invalid grammar trigger type regression test (#30835)
* vulkan: Explicitly include SPIRV-Headers
Distros that don't put the headers in a system include path need
to specify it to the compiler. So we use the appropriate CMake
target to include it.
Assisted-by: Codex
* vulkan: Add explicit headers location to workaround OSX CI runner conf
* vulkan: optimize Flash Attention cm1 for RDNA3 and RDNA4
* reduce shmem use through buffer reuse
* clean up and fixes
* limit q_in_regs to prefill
* vulkan: add flipped FA path, each subgroup owns 16 Q rows
* cleanup
* move flip shader into own file
* replace kvsep with direct load mode
* fix nvidia hang, tune AMD
* only store pv in sfsh if necessary
* fix skip condition
* nvidia tuning
* reduce occupancy on RDNA4 for large attention head sizes
* fix sparse FA fp16 underflow bug
Template soft_max_f32_parallelize_cols_single_row to keep column
elements in registers when they fit, reducing memory traffic from
5 global memory passes to 2 passes.
Assisted-by: Gemini 3.8 Flash
* vulkan: optimize matrix multiplication for narrow N
Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.
* Attempt to handle wave64
* add n=16 tile for cm1
* apply RDNA split_k heuristic to narrow_n generally
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* chat: allow tool calls inside reasoning blocks for Ministral-3 and Qwen3
Fixes#29248Fixes#20837
* chat: address review feedback for Ministral-3 reasoning end
- Remove <function= from Qwen3 (intentionally not used outside outdated coder models)
- Ministral-3: use choice between [/THINK] and peek([TOOL_CALLS]) instead of optional
* chat: add [TOOL_CALLS] to Ministral-3 thinking_end_tags
* chat: simplify Ministral-3 reasoning end choice
* chat: restore peek lookahead for Ministral-3 tool call start
* chat: keep [TOOL_CALLS] only in parser, not in thinking_end_tags
* cont : revert tool call tag removal and update tests
---------
Co-authored-by: Alen Požun <kukirin-ui@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
The thought channel is normally closed with <channel|>, but the model
may open it and go straight to a tool call without closing it. Stop the
reasoning at <channel|> OR <|tool_call>, so an unclosed thought does
not swallow the following tool call.
Fixes: #32231
Assisted-by: Qwen 3.8 27b
The qwen2vl graph passed NULL for the ffn_up/gate/down scales, so the
per-tensor NVFP4 scales loaded by the generic pass were never applied
and the model produced garbage (PPL ~1e6 on CUDA, NaN on CPU).
Pass the scales to build_ffn like qwen2 does.
Fixes#32101
* mtmd : allow custom video decoder
* also adapt to wllama requirements
* clean up
* add test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* dflash : move the DSpark markov head and backbone graphs to dspark.cpp
Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and
graph_mistral_large4 move verbatim. build_inp_target_features and
build_dspark_markov_head become static members of llama_model_dflash so
both files can use them.
* dflash : detect the head and the backbone once, split the loaders per file
load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2,
DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4).
The head, backbone and base layer loaders become member helpers in
dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph
dispatches on the backbone.
The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through
a helper in dflash2.cpp, right after their head tensors instead of in
the base layer loop; the other tensors keep their creation order. The
DFlash2 loader drops its copy of the conv group size check, which master
never reached.
* dflash : share the injection input, token embeddings and lm head lookups
build_inject_inp (target features + fc + encoder norm), get_tok_embd and
get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral
Large 4 graphs. build_inp_target_features is file-static again.
* dflash : split the DFlash decoder into a graph per head
graph_base creates the shared inputs and provides build_inject,
build_inp_tokens, build_layers and build_logits. graph_dflash
(dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and
graph_lilicorr (lilicorr.cpp) assemble them and add their head; the
selector, scorer and markov head builders move to the file of their
graph, and the DFlash2 local conv run by build_layers to dflash2.cpp.
build_arch_graph dispatches on the backbone, then on the head.
The op order and the tensor names are unchanged.
* speculative : split the DFlash driver into a class per head
common_speculative_impl_draft_dflash stays the DFlash implementation and
becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its
protected constructor only reads the draft metadata; init() then sets up
the contexts with the options of the draft (draft length, backend
sampling, device features, masked h_nextn). Each derived class overrides
draft() with the path of its head from the former draft(). DFlash2 also
overrides begin() to reseed its generator per request, and LiliCorr
overrides process() to store the context features of each injected
chunk, which its noise block carries as its state.
The factory keeps the current routing: draft-dspark picks DSpark,
draft-dflash picks DFlash2 or LiliCorr from the draft metadata and
DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs
the DSpark reader instead of a mix of both. sample_from_anchor is logged
by DSpark only.
* speculative : add the draft-dflash2 and draft-lilicorr types
draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts,
which draft-dflash keeps running too: it picks them from the draft
metadata. draft-dflash2 and draft-lilicorr must match the draft model,
draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a
DFlash draft is rejected, and a run uses at most one DFlash family type;
these errors leave speculative decoding uninitialized.
The type detected from the draft GGUF is now the concrete one, and
draft-dflash2 and draft-lilicorr download the dflash sidecar.
* docs : describe the DFlash2 and LiliCorr drafts
Add the DFlash2 and LiliCorr sections and the new types, and explain
that draft-dflash also runs DFlash2 and LiliCorr drafts.
* speculative : drop the unused DFlash is_mrope
Nothing reads it since the migration to common_batch (#29385), which
picks the number of M-RoPE position rows from the rope type of the draft.
* doc: update spec dec
* address review comments
* requirements : support Python 3.15
torch 2.11.0, numpy 2.4, matplotlib 3.10, pandas 2.3, pillow 11.3 and
cffi 1.16 have no Python 3.15 wheels, so the CI runner on 3.15 cannot
install the requirements.
Python 3.15 gets the first releases that ship 3.15 wheels through
version markers, and every version below 3.15 stays as is.
pyproject.toml now allows 3.15 and picks torch 2.15.0 there.
* requirements : apply review suggestions
numpy 2.5.4 from Python 3.14, matplotlib 3.11.2 and pandas 3.0.6 from
3.11, cffi 2.1.1 and pillow 12.3.0 for every version. cffi 1.16.0 has
no 3.14 wheel and fails to build there, so this also fixes the
tokenizer test requirements on 3.14.
* hex-bufs: combine queue and fence buffers
* hex-bufs: use n_threads for queue sizing
* hex-mmap: optimize buffer stacking at the host to avoid iova fragmentation
* hex-mmap: add simply retry for spurios mmap failures
* hex-mmap: use eager mode for extended mmap retries
* vulkan: add TOP_K_MASK
Assisted-by: Claude Opus 5.5
* vulkan: fix the TOP_K_MASK barrier and use the common offset helper
Assisted-by: Claude Opus 5.5
* vulkan: declare ggml_vk_top_k_mask in the common header
Assisted-by: Claude Opus 5.5
* spec : skip p_min for sampled MTP and DFlash2 drafts
Under rejection sampling the acceptance at a position is 1 - TV(p, q),
which the draft's top probability does not tell, so the p_min cutoff
dropped draft tokens the target would have accepted. Skip it when
MTP or DFlash2 samples the draft. draft-simple keeps it: each extra
draft token costs a full draft model pass.
* spec : replace the sampled-draft p_min skip with a warning
p_min behaves as on master again for every drafter and sampling mode.
Instead, log a warning when --spec-draft-p-min > 0 is used with
--spec-draft-sampling probabilistic, suggesting p_min 0: under
rejection sampling the draft's top probability does not tell how much
of the draft the target accepts. The cut is not skipped because with
MTP at long drafts it can still pay off.
* spec : warn about p_min only for drafters that use it
The n-gram drafters ignore p_min, and DSpark uses it as the cutoff of its confidence head, so the warning is skipped when only these are configured.
---------
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* grammar : limit the number of alternatives
* grammar : introduce an AST for grammar parsing
* grammar : revert AST refactor and alternatives limit
This reverts commits:
- 84f3b57c4d6c482e7187a30735d16b02de3784fa
- 393a707ebb7e6f798abdb8af34cb1bc102f4aaa6
* grammar: optimize big grammar matching by using left-factorization
* grammar : preserve character class boundaries during left factoring
* grammar : prevent infinite expansion during left factoring
* grammar : use ASCII characters in comments
* grammar : test empty suffixes after left factoring
* grammar : deduplicate left factoring logic
* grammar : add comments
* grammar : use number parameters and clarify RSS output
* grammar : clarify character class test names
* grammar : remove performance benchmark
* grammar : share immutable rules
---------
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
The kernel launches one block per row and 256-wide tile, so the
{2, 16777217} test case asks for 16777217 blocks of 256 threads. HIP
refuses a grid dimension above 2^32 threads and the ROCm jobs fail
with "invalid configuration argument".
The host now launches the grid in chunks of at most UINT32_MAX / 256
blocks and passes the chunk offset to the kernel. Real sizes fit in a
single chunk and keep their timings.
This commit removes the llama version test-cmake and instead uses the
version that is supplied by CMAKE_PREFIX_PATH.
The motivation for this is to avoid issues when bumping major versions.
The D=256 vec flash attention kernel with F8_E4M3 K and V uses 257
VGPRs (255 + 2 spilled) on gfx908, one over the limit of the check.
The spill is a perf issue only, like the Q4_0 D=256 vec kernel
already in the ignore list.
Both variants of the kernel join the ignore list.
[no ci]
* server: read a small temp file in the read_file tool test
test_tools_builtin_read_file read its own file, which grows with every
added test. On Windows the CRLF checkout puts it at 16664 bytes, over
the 16384 byte read limit, and the tool refuses it.
The test now writes a small file of its own and checks the exact
content read back.
* server: use a temp cwd in the x-tool-cwd tool test
test_tools_builtin_cwd_header read test_tools_builtin.py through the
x-tool-cwd header, which hits the same 16384 byte read limit on the
Windows CRLF checkout, and wrote its shell marker into the source tree.
The test now uses a temp directory as cwd, reads a small file from it
and checks the shell marker lands there.
The vision tests download their images from huggingface.co, which
resolves to a private address on the runners inside the HF network.
The media URL check refuses it with "host is not a public address"
and the server-cuda job fails.
The vision fixture enables media_allow_private_urls. The private
address check keeps its own coverage in
test_vision_image_url_private_address.
The same-origin tests over the LAN address block forever on the Metal
runners: the request to the host's own LAN address never returns and
has no timeout, so the job runs until the 6h limit.
get_lan_ip() now probes /health on that address with a short timeout
and returns None when the host cannot reach it, which skips the test.
The test requests themselves use the default request timeout so a
hang fails the test instead of stalling the job.
* fix(hexagon): make host build work with clang-cl on Windows ARM64
Two CMake fixes for building ggml-hexagon natively on Windows-on-ARM64
with x64-hosted clang-cl (validated on Snapdragon X Elite, discussion in
qualcomm/GenieX#1390):
- Some clang drivers (e.g. x64-hosted toolchains used on ARM64) don't
predefine the arch macro, which makes winnt.h fail with "No Target
Architecture". Define _M_ARM64/_ARM64_ explicitly for the host TUs.
- The Hexagon SDK's incs/stddef include (AEEStdDef.h and friends) poisons
the MSVC header chain when it leaks onto dependent TUs' include paths
via htp_iface's PUBLIC includes. Keep it PRIVATE to the interface stub
and system-level for the backend's own TUs (which include <AEEStdErr.h>
directly).
Originally validated 2026-08-26 as reported in the issue; applies
cleanly to 15759e0.
* hex-cmake: trim down excessive comments
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* llama : remove deprecated `llama_set_warmup`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : remove remaining deprecated API
Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* fit-params : remove stale `llama-ext.h` include
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : document `llama_model_dflash_selector_top_k`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : promote the NextN / speculative API from `llama-ext.h`
Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : polish the NextN API promotion
Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : prune deprecated API
* cont : bump RPC version
* cont : minor
* ggml : remove outdated documentation from ggml.h
The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:
- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`
The block also ended with 5 empty `TODO` sections.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)
* rpc : bump RPC protocol major version
The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.
The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: losewayy <1134285612@qq.com>
* CUDA: few-row MMQ for 2-24 src1 columns
New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.
* nit fixes
* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat
* few-row quantize: DS4 sum from the quantized values, as in #31247
* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build
* few-row MMQ: dispatch order MMVQ, few-row, MMQ
With --sleep-idle-seconds, a handler passes wait_until_no_sleep()
while the server is awake, then keeps using the server state, for
example to parse the body and tokenize. If the idle timer expires in
that window, the main loop enters sleep and the callbacks destroy the
model: the handler reads freed vocab memory, and a task posted after
that stays in the queue with nobody awake to process it.
Count the requests that passed wait_until_no_sleep() and are not done
yet, and do not enter sleep while the count is positive. The response
generator releases its count in its destructor, so every return path
of a handler gives it back.
Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
HunyuanVL resamples its position embedding on the host into a float
buffer sized for F32, then reads ggml_nbytes() of the tensor into it.
The graph never uses the tensor, so any type loads, and an F64 tensor
overflows the buffer at the first image request.
Require the position embedding to be present and F32 when loading the
HunyuanVL tensors.
The MiniCPM-V resampler ends with build_mm(mm_model_proj), whose output
width is ne[1], while n_mmproj_embd() returns ne[0]. Both are equal in
real models, the projection is square. A non square projection passes
the text model n_embd check and only aborts at the encode, on the
output buffer size check.
Return ne[1], so the width matches the graph output.
* mtmd: require the MLP projector and patch embedding tensors
The loader marks the patch embedding and the LLaVA projector tensors
as optional, while the graphs and n_mmproj_embd() dereference them
unconditionally. A GGUF without them crashes on a null tensor during
the warmup.
Load the patch embedding as required for the vision encoders that
start with it, which is all of them except GEMMA3NV, DEEPSEEKOCR and
DEEPSEEKOCR2. Load mm.0 and mm.2 as required for MLP, and mm.0, mm.1,
mm.3 and mm.4 for the Yi-type MLP_NORM variant, selected by mm.3.
The graph no longer tests mm.2 and the yasa2 assert on the patch
embedding is covered by the loader.
* mtmd: keep the Yi-type llava TODO
The Yi-type variant is still detected by the presence of mm.3, the
TODO on that detection stays.
The Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Ling3-VL graphs build their
input with a temporal merge that assumes a 2x2 patch grid, and the
Qwen2-VL merger reshapes by 4. Any other spatial_merge_size from the
GGUF reaches a GGML_ASSERT at the first image request and aborts the
server.
Reject a spatial_merge_size other than 2 when loading the hparams.
The SpacemiT compute case was commented out, so the op stayed on the scalar CPU path. Type dispatch stays with the other ops, and use_ref keeps the reference path on the generic CPU kernel.
Assisted-by: Codex
Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
* vulkan : Intel customized FA shader for prefill usage
* vulkan : adding check of max_bias/logit_softcap for Intel FA kernel path
* vulkan : add m16n16k16/SIMD32 support for Xe FA prefill shader
* vulkan : Intel FA shader and host refinement, with accuracy issues fixes
* vulkan : Limit the SLM usage to 48 KiB for Intel prefill FA shader
* vulkan : Update Intel FA shaders to handle cases with large head dimension K, unaligned Q head stride and multi head mask cases
* vulkan : Intel FA shader and host refinement to align with existing design, extreme cases handling, shape check relaxing.
* vulkan : Intel FA shader and host update to support q as tensor view, unaligned batch stride q. Add env variable for enforcing XE FA path for other vendor GPUs with 16x16x16 CM1 and SIMD32 support
* vulkan : Inte FA shader and host update to fix the potential SLM over 48 KiB issue, enhance the buffer access robust and add asymmetric qk/pv ration support.
* vulkan : Inte FA shader and host update to add SLM 48 KiB size check, remove unnecessary extensions and finetune the unroll/dont_unroll loops
* vulkan : Intel FA shader and host update to add SLM size check function, use real device max SLM size in shader specialization constants.
* vulkan : Intel FA shader and host update to consolidate host helper function, remove redundant buffer sync, bypass unused pipeline and buffer, fix subgroup shmem conflict and add negative case handling.
* vulkan : Intel FA host update to correct get_xe_fa_tuning_params() function log message, and add workgroupMemoryExplicitLayout/workgroupMemoryExplicitLayout16BitAccess prerequisite feature check.
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back
* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ
* ggml-cuda: Tests for the fused scale
* ggml-cuda: reuse existing graph to test fusion in mmq epiloque
* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency
* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions
* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
* ggml-backend : prefetch only inputs computed by earlier splits
The copy callback of a split runs after the prefetch and can write tensors that a later split reads, e.g. the MoE cache uploads missed experts into its bank. A prefetch of such a tensor reads it before the write.
* llama : skip the MoE cache when its device cannot run the cached mul_mat_id
Otherwise the scheduler runs the matmul on another backend and copies the whole bank every ubatch.
The 32x256 pack already keeps a separate uint8 zero point. Complete the M1 path and use the same HP kernel as Q4_0.
Assisted-by: Codex
Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
* ui: add search_chats tool
* add from_timestamp / to_timestamp
* ui: search_chats review fixes
- keep search_chats disabled on import, copy tool policy on fork
- stream messages without attachment payloads, one transaction for time ranges
- keep the output within budget, cut whole messages from the older end
- reject impossible date-times, quote titles and attachment names
- typed readOnly tables, move localIsoString to formatters as formatLocalIso
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
npm 11.20 refuses git dependencies by default (EALLOWGIT), so npm ci
failed on fresh installs. Pin llama-brand to the same commit through
an HTTPS tarball, which also adds an integrity hash to the lockfile.
Assisted-by: Claude (Opus 5.5 Medium)
Each graph result reserves metadata for max_nodes, but many graphs use
only a fraction of that capacity. The vector resize initializes the
whole reserve and makes otherwise unused pages resident. Allocate the
reserve with new uint8_t[] instead.
* fix aot for >4g garbage output issue; add --aot in build script
* correct the aot flag
* support no-xmx build, update the code for no-xmx building, rename nossl to no-ssl
ggml_sycl_set_main_device() runs at the start of every graph compute. Its
early-return check called dpct::get_current_device_id(), which locks the
device manager mutex and looks up the thread-to-device map. Use the
thread-local cached id that the rest of the backend already uses.
Assisted-by: Claude Code
Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
* CUDA: skip empty MoE tiles in MMQ stream-k fixup
mul_mat_q_stream_k_fixup loads ids_dst[col_low + j] for all j < J before
checking j_sup. mul_mat_q returns early on col_low == col_high, the fixup
kernel did not. mm_ids_helper emits one extra tile per expert that may be
empty; for trailing experts col_low == ncols_dst, so the load reads one
element past the J_best-1 padding added in #29953. The kernel returns at
j >= j_sup before storing anything, so results are unaffected.
Return early when j_sup <= 0, like the main kernel. The return is
block-uniform (tsj is loaded once per block) and precedes __syncthreads().
Add test_mul_mat_id_unused_tail to test-backend-ops: only the first
n_mats/2 experts are selected, so the trailing experts have zero rows and
their tiles are empty; m=33, k=512 keeps tile efficiency low enough that
stream-k splits along K and launches the fixup kernel.
* CUDA: drop the extra MUL_MAT_ID test case and the explanatory comment (review)
test_router_restricted_cors_same_origin_reaches_child still named
ggml-org/tinygemma3-GGUF, which the router no longer finds since the test
models moved to ggml-org-dev, so the Server test job fails on master.
Assisted-by: Claude (Opus 5.5 Medium)
* opencl: add decomposed FA
Assisted-by: Claude Opus 5.5
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: build FA MQ kernel on first FA call
* so allows warmup to early build neccessary kernels
Assisted-by: Claude Opus 5.5
* opencl: handle inf for fully masked row
* opencl: refine condition for using fa decomp
* opencl: support dp4a for PV mm
Assisted-by: Codex
* opencl: add direct PV for FA decomp to avoid V transpose for small batch
Assisted-by: Codex
* opencl: enable direct pv by default
* opencl: expand X2E decomposed FA to smaller heads and batches
- Lower the default head dimension threshold from 256 to 128 on X2E.
- Allow DK64 with P8, no attention sinks, GQA >= 2 and n_kv >= max(1024, 2*n_q).
- Allow P8 with V transpose for 2-31 queries when direct PV is unavailable, DK >= 128 and n_kv >= 1024.
Assisted-by: Codex
* opencl: adjust format
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* ui : keep downloads in Discover sidebar during search, sort base-org repos first
- Show the sticky Downloading section regardless of an active search query
- Order search results so repos published by their base model's org lead
Assisted-by: Claude (Opus 5.5 Medium)
* server : throttle download progress per file
Files of a download run in parallel threads, but progress updates were
throttled with one shared timer. A small draft sidecar reports far more
often than the main weights, so it took nearly every slot and the main
file's updates, including the first one carrying its total size, were
dropped until the draft finished. The UI then showed the draft's progress
first and restarted from 0% for the model.
Throttle each file on its own and always pass a file's first update.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list a downloading repo once and match base orgs case-insensitively
A search hit that is downloading already shows under Downloading, so the
results leave it out. Base model orgs compare without case, as the hub
treats them.
Assisted-by: Claude (Opus 5.5 Medium)
* server : tell clients a download stopped on request
When a client paused or cancelled a download, every other client following
/models/sse got a bare download_failed and could not tell it from an error,
so the web UI showed "Download failed" for a pause made in the Llama app.
A stop asked for through unload or delete now fails with {"stopped": true}.
Deleting a paused download, which has no instance left, now sends
model_remove as deleting a running one does, so other clients drop its row.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : follow downloads other clients start, pause or cancel
A download another client started (e.g. the Llama app) already showed up
from the feed; it now also picks up the sidecars the list reports for it.
A stop another client asked for reads as a pause instead of a failure, and
a removal clears the download row even when the list does not carry it yet.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : take the logo and favicons from llama-brand
The web UI's logo mark and favicons now come from llama-brand, added as a
dev dependency from GitHub and pinned to a commit, so the UI draws the
same icon as the README cover instead of its own copy of an older one.
Logo.svelte inlines the brand's icon and fills it with currentColor, and
the PWA asset generator starts from the brand's web favicons, which carry
their own margin and optical centring, so the UI's colouring and padding
script, its test and its logo.svg go.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : follow llama-brand's master instead of a pinned commit
Assisted-by: Claude (Opus 5.5 Medium)
* server: fix --cors-origins localhost handling
* server: validate Host on loopback connections against DNS rebinding
A page whose own name resolves to 127.0.0.1 reaches the server with
that name in Host and a matching Origin, so the same-origin shortcut
let it through. On a loopback connection, Host must now name loopback
or the host of an allowed --cors-origins entry. The LAN tests reach
the server through a real interface instead of a forged Host.
* fix python type check
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Allocate gf_res_reserve and both gf_res_prev entries once in the llama_context constructor. Add llm_graph_result::set_max_nodes() and reset the existing instances in sched_reserve() instead of recreating them. get_gf_res_prev() now only selects the arena for the current batch.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
- remove build_delta_net_chunking and build_delta_net_autoregressive
- always use the fused gated delta net
- make graph_max_nodes independent of the ubatch size
- drop the fused GDN probes and fused node tags
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
ggml-sycl.dll built with oneAPI 2026.2 imports mkl_intel_thread.3.dll and libimalloc.dll, which are not part of the release package. LoadLibrary fails and the SYCL backend is silently skipped, llama.cpp falls back to CPU.
Adding both DLLs fixes the load: measured on Arc B570 with LFM2.5-8B-A1B Q4_K_M, pp4096 goes from 404 t/s (CPU fallback) to 2232 t/s (SYCL) with the extracted b12244 package.
Assisted-by: GLM (z-ai/glm-5.3-flash)
Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com>
* CUDA: re-instantiate graphs when the cluster dims of a kernel change
* CUDA: keep the cluster dims with the graph instance, check them only on Hopper+
Assisted-by: Claude Opus 5.5
* CUDA: use CUDA_CHECK and a helper struct for the graph cluster dims
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CUDA: add a TODO for the graph cluster dims test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call
Assisted-by: Claude Opus 5
* ROCm/HIP: corrected by only setting the workspace on ROCm 7.0 or newer
* cpu: accumulate flash attention V in F32 in the one-chunk kernel
The one-chunk kernel accumulated an F16 V cache in an F16 running sum
that is only normalized at the end, which overflows past 65504 on long
contexts: inf, then NaN on every following token.
Every V type now accumulates in F32, like the tiled kernel. F16 rows go
through ggml_vec_mad_f16_f32, which converts and multiplies in the same
register (F16C/AVX512, NEON, RVV) with no round trip through a temporary
F32 row, so decode keeps master speed on x86 and loses about 4% at 4k
context on ARM, where master used native F16 FMA.
* cpu: widen the F16 V rows like ggml_vec_dot_f16_f32
ggml_vec_mad_f16_f32 required F16C and FMA, so AVX builds without FMA
fell back to a scalar loop and ran 3x slower than master. It now
follows the guards of ggml_vec_dot_f16_f32: AVX512, then AVX with F16C
through GGML_F32x8_FMA, then NEON and RVV, then the scalar loop.
* additional FA test cases
Assisted-by: Codex
* vulkan: allow dequantized KV optimization with CM2
Enable the KV dequantization optimization with coopmat2, guarded by a heuristic
based on query tokens*GQA ratio.
While doing this we saw very large speedups for certain head sizes, and it was
because those head sizes were incorrectly treated as unaligned. So also fix
those to be treated as aligned, which should help for when the dequant
optimization doesn't apply.
* gqa uses nbq2 for the stride, so we need to check that value for alignment
* tests : use VARS_TO_STR18 for flash attention
Assisted-by: Codex
CodeMirror takes text input through EditContext on Chrome for Android
only, and there the chat editor never brings up the virtual keyboard.
The editor now always takes its input through contenteditable, as on
desktop and iOS.
A press in the input box outside the text also focuses the editor, in
the collapsed form too, so the whole box is a tap target again as the
textarea was.
MMF tiles 32 rows per block, so a BF16 matrix with few rows (e.g. the
qwen4exp hyper-connection projections, 320 rows with K = 10240) runs only
a handful of blocks and leaves most SMs idle at batch sizes 2..8. MMVF
runs a block per row and wins there when K is long enough.
MMVF's cost grows with rows x batch while MMF's stays roughly flat in that
range, so use MMVF only while rows * ne11 * 256 <= nsm * min(K, max_k) and
K >= 1024. The budget scales with the SM count, which keeps GPUs with few
SMs (e.g. the A16) on MMF. Blackwell allows up to 8 columns and
max_k = 6144; older GPUs stop at 5 columns and max_k = 4096.
Add eval cases at 128x2560, 512x2560 and 320x10240 for n = 2, 3, 4, 5 and 8.
* CUDA : add sm70 FlashAttention config case for the 256x256 head
* CUDA : tune sm70 FlashAttention for 256x256 ncols=32
Assisted-by: Codex
---------
Co-authored-by: Jirka Svítil <mistrjirka@users.noreply.github.com>
* cuda : add MMA FlashAttention support for head sizes 40 and 72
Head sizes 40 and 72 (used e.g. in vision towers of Gemma 3/4 and Qwen3-VL) were excluded from every tensor-core dispatch branch and fell back to the tile kernel.
The fp16 MMA fragment covers 16 halfs in the head direction, so the in-kernel geometry is padded to the next multiple of 16 (40 -> 48, 72 -> 80). Register arrays, shared memory tile strides and MMA loop bounds use the padded size, while global memory loads and stores the real one.
Enabled on NVIDIA Turing and newer while Volta and AMD keep the tile kernel. The RDNA/CDNA config rows and the mma_tile_sizes specialisations are needed because the explicit instantiations are compiled for every target of a translation unit.
* tests : add the head size = 40 encoder perf shape
* cuda : address review comments on the head size 40 and 72 kernels
* Initial CUDA WIP version
* Use fp8 intrinsics
* Do W8A16 for FP8 in GEMV case
(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB
| model | size | params | backend | ngl | n_ubatch | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 | 2781.53 ± 8.48 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 | 37.56 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 @ d32000 | 1616.31 ± 2.61 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 @ d32000 | 34.11 ± 0.03 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 | 2542.00 ± 0.51 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 | 37.36 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 @ d32000 | 1762.86 ± 1.33 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 @ d32000 | 34.09 ± 0.02 |
* Optimize GEMV kernel for FP8
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2
Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099
* Remove trailing whitespace
* Add supported for strided copies to/from FP8 in CUDA
* Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh
* Fix odd-rows for convert/getrows, possible with FP8
* Fix FP8 fallback in cuBlas
* Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8
* Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte
* Add todo about accumulator/precision honoring
This is beyond the scope of the stacked PR
* Avoid races in fp8_fallback
* Allow Hopper and other BW SKUs for FP8
* Use intrinsic over explicit manual conversions
* Resolve TODO
As per spec, we don't have to preserve signs
* Add commend on cublaslt workspaces
* Use convert_unary_cont_cuda over block dequant in convert.cu
* Use get_rows_cuda_float for fp8 kv cache
* Make FP8 copies work on all IHV's
* More cleanup
* Simplify by falling back to shared cuBLAS helper
* Fuse weight_scale multiplication for FP8 also
* More code-cleanup
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4
Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix
* Use F32 activations in fallback mmvf path
* Explicitly link against cublaslt
* Move FP8 conversion intrinsics into ggml_cuda_cast
Cleaner this way
* WIP src1/prec for fp8 in MMVF
* Remove comment
* Keep only f8 x f8 KV cache for now
* Just do FP8 x FP32 x FP32 in MMVF as its fast enough
* Update ggml/src/ggml-cuda/ggml-cuda.cu
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
* Add batched FP8 GEMMS in cublas as suggested by code review
https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644
* Extract fp8x2 -> float2 load into helper
* Searapte copy checks to/from fp8
* Compiler massaging to avoid repeated address offsets
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Modify `ggml_sycl_op_clamp` to use the unary helper function
`ggml_sycl_detail::ggml_sycl_op_unary` to handle non-contigous or
strided views. The fix uses a pattern that matches other stride-safe
unary ops, and is a simpler execution path.
Deleted dead template `unary_op_clamp_kernel`, the flat `clamp` kernel
launcher from element_wise.cpp, and deleted (now) unused
`SYCL_CLAMP_BLOCK_SIZE` from presets.hpp.
The hidden desktop strip relied on inherited `visibility: hidden` to keep
its content from catching clicks. Since the sidebar's sections fold
(#31936), an open section's CollapsibleRegion sets `visibility: visible`
again, and its rows' context-menu triggers carry an inline
`pointer-events: auto`, so they caught clicks meant for the page.
Make the hidden strip's content inert, which no descendant can undo, and
add a client test that hit-tests the hidden sidebar's area.
Fixes#31971
Assisted-by: Claude (Opus 5.5 Medium)
* media : remove the legacy llama0 and llama1 brand assets
The brand assets live in ggml-org/llama.brand; none of these is referenced
in the repository.
Assisted by: Claude (Opus 5.5 Medium)
* readme : use the new llama.cpp cover from llama.brand
Assisted by: Claude (Opus 5.5 Medium)
* ui: show audio in Discover for models whose mmproj has an audio encoder
Discover took audio only from the pipeline tag, so omni models tagged
any-to-any, such as Gemma 4 and LFM 3 Omni, showed vision but never audio,
though their mmproj carries an audio encoder.
Read the encoders from the mmproj's GGUF metadata instead: a ranged read of
its first 64 KB, once per repo, for clip.has_vision_encoder and
clip.has_audio_encoder. The Discover list and details now show audio when
either the pipeline tag or the mmproj says so.
Assisted by: Claude (Opus 5.5 Medium)
* ui: take Discover modalities from the mmproj's encoders in one place
Review fixes for the mmproj header read:
- drop vision for audio-only mmprojs: the converter writes
clip.has_vision_encoder only when true, so a header read in full without
it has no vision encoder; a header cut short leaves vision as it was
- accept only a 206 to the ranged read, so a server that ignores Range
never streams the whole mmproj
- encode the mmproj path in the resolve URL
- derive the list and details modalities with one HuggingFaceService
helper, so both views agree
- point the key comment at tools/mtmd/mtmd-impl.h, where the keys live now
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a gated repo's mmproj from being read again on every mount
A gated, private or missing mmproj answers 401, 403 or 404 to every read,
so caching nothing made each mount of its Discover row request it again and
log an error. Cache it as unknown instead: no audio, and vision from the
mmproj being there, as before the header was read.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move Models, MCP and Settings to a sidebar footer
The sidebar's actions split in two: New chat, Search and Projects stay at
its top, and Models, MCP and Settings move to a footer at its bottom, under
a thin rule, in the expanded sidebar and its collapsed strip alike. On a
phone the footer keeps clear of the home indicator, and Models stays
desktop-only as before. The search dialog still lists every action.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : group the sidebar footer in an inset surface
The design system gains inset-group-surface, for rows grouped inside a glass
panel: white on the light glass and a faint tint of the text colour on the
dark one, with the sidebar's own ambient edge, no inner highlight, a faint
wide shade under it and the card radius. It is matte-glass pointed at its
own component tokens, like the sidebar and chat form surfaces, and reads as
part of the panel: no tight contact shadow or solid border like a button,
no inset shade like a field.
The sidebar footer uses it in place of the rule above Models, MCP and
Settings. DESIGN.md and the Surfaces page document it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : give the sidebar's logo button the add button's key look
Whenever the sidebar shows, open or peeking, its logo button takes the same
key as the chat form's add button (matte-key, white in light mode, a light
tint in dark), in place of its own muted fill. While the sidebar is hidden
the logo has no fill and no shadow, and hover and focus never change the
fill. It keeps its round shape and its own size.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : put search in a round key at the top of the phone sidebar
On a phone the sidebar's Search row moves to a round key at the right end
of its top row, across from the logo, with the same key look as the logo
and the chat form's add button. It opens search as the row did. Desktop
keeps the Search row.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : size the phone sidebar for a thumb and list its footer with the actions
On a phone the sidebar's actions and rows are 48px tall at 17px, flush one under
the other with 16px corners, its section headings 15px and grey, and its groups
28px apart. MCP and Settings join the top actions there: the footer is a desktop
pattern.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : mark the selected sidebar row with a tinted inset surface
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the phone sidebar's round keys white on the grey page
Assisted-by: Claude (Opus 5.5 Medium)
* ui : open Settings and MCP as pages on a phone, dialogs elsewhere
On a phone the sidebar opens them at #/settings and #/mcp-servers, with the menu key and a
title like any phone page. Pinned under the status bar, Settings' section tabs fill the
screen above them in the page's colour.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : open a phone sidebar's pages behind it, settled before they slide back
A tap in the phone sidebar now opens its page while the current one is still aside and slides
back once it has rendered, instead of swapping pages mid-slide. The slid page's fixed bars are
pinned again as the page under them changes, the lock ends with the slide itself, and a
phone's new chat shows its form and greeting at once, without the move and entry they have
on a desktop.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style the sidebar footer group and the selected row with master's classes until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add GlassKey with master's styles for the phone sidebar's round keys
Assisted-by: Claude (Opus 5.5 Medium)
* ui: center the top edge of a new chat's form on the screen
The form rested 42px below the middle up to 2xl and 6px above it past
that; one drop, from the resting card's height, now puts its top edge on
the middle at every width from md up.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: round the sidebar's action rows concentric with the footer group
The sidebar stacks four surfaces: the page, the sidebar, the footer group
inset in it and a row in the group. app.css now names the group's radius
and padding, and derives the row radius from them (16px less 4px), so the
footer's rows run parallel to its corners; the actions at the top of the
sidebar share that radius. The footer moves to its own component, the
README documents the pattern, and a client test holds the rule on the
rendered footer.
A selected action row was bg-accent, the same grey as a hovered row in
light and darker than the footer's fill in dark, so it read as hovered or
not at all; it is now raised and edged, while a hover only tints.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: import demo chats through the Settings page on a phone
A phone opens Settings as a page now, not a dialog, so the demo harness
waited for a dialog that never came and every phone demo with traces
failed; it now imports through whichever of the two opens.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: count the sidebar's rows from their items in the surface test
The test expected three footer rows and two top actions, which the
stacked PRs change as they add Projects, Library and Servers; it now
expects as many rows as the sidebar has items.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a line on Meta+Enter off macOS too
The chat form takes Meta+Enter as Ctrl+Enter and leaves the newline to
the editor, but the editor bound only Mod-Enter, which is Ctrl off
macOS, so on Linux Meta+Enter neither sent nor added a line. This failed
the client test on CI's Linux runners, on master too.
Assisted-by: Claude (Opus 5.5 Medium)
This commit removes the unused ids and type_src0 parameters from
quantize_row_q8_1_cuda.
The motivation for this is that these parameters were required in the
past because this function was used in combination with the function
pointer quantize_cuda_t, which has been now been removed, and this
function no longer needs to adher to the function pointer signature.
Refs: https://github.com/ggml-org/llama.cpp/pull/31884.
* Add FP8 KV-cache support
* Add tests for scaled, FP8 KV-cache to test-backend-ops
* Fix get_*_scale mapping and dequnatize FP8 also before rope shift
* Normalizes cache-scale names independently of projection layout.
While fused KQV is currently impractival for non-shared scales, this is
an improvement for future use regardless
* Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs
* Update comment wording
* Move E2E FP8 KV cache tests to test-llama-archs
They belong there as they are not just a test for ggml's backend ops
* Increase support surface, remove seed
* make scaled kv generic
* Fix CI failings for test-kv-cache
https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392
* CI: execute kv-cache-scale tests
* exclude fp8-kv tests from OpenVINO
* Fix linting errors of test_kv_cache_scales
* Disable kv-ffload=false + multi-GPU tests for FP8
See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413
* Adjust test label for test-kv-cache.cpp
* Fix conflict-resolution mistakes in test-kv-cache.cpp
https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392
* Move KV tests to test-fp8.cpp
https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4410518158
* Remove tmp file for test-kv-cache-scales.py on Windows as well
* Fix fp8 kv cache test by population kv_source_impl
* Exclude models that consume KV cache directly
* Remove obsolete comment
* Allow BF16 scales as well for KV cache
* Rename *_cache_scale to *_cache_s
https://github.com/ggml-org/llama.cpp/pull/28392#discussion_r4517194999
* Move `is_type_scaled()` to `llama-kv-cache.cpp`
* ui: let a menu action open a submenu
Group and project pickers open as flyouts from a chat's menus. Master's menu
list has no submenus, so bring them back next to its tooltips, with a close
callback for picks made inside them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : store groups and let conversations join them
Groups live in their own IndexedDB table (schema version 2) with a name.
groupsStore creates, renames and deletes them; deleting a group keeps its
chats, out of any group. A conversation records the group it is sorted
into, and a group picked before the first message is kept pending and
applied when the conversation is created.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : show each group as a sidebar section
Every group is a sidebar section next to Pinned and Recent, folding,
showing its first six chats and dragged into place like them. A chat
dropped on a group joins it and leaves Pinned; dropped on Recent, it
leaves its group. Recent is called Ungrouped once a group exists, and its
menus create a group in a small naming dialog; so does a right-click
anywhere in the list that no row or heading takes, down to the empty space
under the last section. A group's heading starts a new chat in it, and its
menus rename or delete it; deleting keeps its chats, back in Ungrouped.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move chats into groups from their menus
A chat's menus, in the sidebar and on right-click, move it into a group
or take it out of its group. Desktop menus show a searchable group flyout
whose New group entry names a group in the group dialog and moves the chat
into it; a phone gets a picker dialog instead, which creates a group from
what is typed.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the group picker from a phone's hold menu
The hold menu is anchored to the row, so it showed the desktop flyout,
which ran off the left edge of the screen and whose New group closed
without naming one. On a phone, Move to group now opens the picker
dialog, as intended.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open a new chat in a tab of its own
With conversation tabs on, New chat from the sidebar, a group's heading
or the shortcut replaced the open chat's tab. It now adds a tab, as the
tab bar's "+" does, and the open chat keeps its own.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list a phone's groups under the picker's search
On a phone the picker fills the screen, and its grid spread the rows
over the height, so the list sat in the middle. It is now a column: the
list starts under the search and scrolls in the rest of the screen.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fix gaps in chat groups found in review
- A phone shows a group heading's New chat and menu, and the Ungrouped
heading's menu, which were reachable only by hover or right-click.
- A pinned chat moved into a group from its menu leaves Pinned, as a
drag onto the group already did, so the move shows.
- Only the sidebar's New chat, a group's New chat and the new-chat
shortcut open a new tab; stepping past the last chat, search and the
models drawer show the new-chat screen in the current tab again.
- An export carries each chat's group, and an import sorts the chat
into the group of that id or name, or creates it; a group the file
does not carry is dropped.
- The group picked with a group's New chat survives leaving the
new-chat tab and coming back.
- A phone's menus skip any submenu in the shared menu list, not only
the group one.
- Deleting a group forgets its section's place and fold.
- Groups load newest first by creation, so a rename no longer moves one
after a reload.
- groupsStore looks groups up in a map and answers which live group a
chat is in, for the three places that checked it by hand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a forked chat in its source chat's group
A fork copied the working directory but not the group, so it landed in
Ungrouped.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move a chat with its whole fork family between groups
Moving a chat into a group, or out of one, takes its parents and its
children along, so a fork tree never splits across sidebar sections. A
pinned member that joins a group leaves Pinned with the rest. Group
sections render the tree with its depth, as Ungrouped does.
* ui: show the group name prompt as a small centered card on phones
The new and rename group prompt opened full screen on a phone, like a
large dialog. It is now built on the alert dialog, as the chat rename
prompt is, so a phone shows it as a card in the middle of the screen.
Its Cancel is a plain button, so Enter in the name field saves.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: space a phone's group sections like Ungrouped
On a phone a group or Pinned section put its first chat right under the
heading, where Ungrouped leaves a row gap. They now leave the same gap,
and the sections sit a little further apart than the rows inside them.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Pascal <admin@serveurperso.com>
This commit removes the unused quantize_cuda_t from quantize.cuh.
The motivation for this is that the usage of this typedef as a function
parameter was removed in Commit 74976e1aef
("CUDA: remove -sm row, refactor cuBLAS (#24216)") and there is no longer
any usages of it.
The sampling chains are read on the host after the graph compute, and the
graph addresses are reused across builds with different selected rows: the
token-generation reserve graph has one output row per sequence, while the
actual block draft has n_max rows per sequence. The reserve allocates the
unselected chains without keeping them, so their buffers alias, and the
actual graph then reuses those addresses by node index.
set_outputs only marked the tensors stored for the selected rows, so the
unselected per-(sequence, row) chains were freed and reused. Collect every
chain tensor in t_sampling_all and mark all of them as outputs there.
Reproduces with -np 2 (non-unified KV) and 2 concurrent requests for
draft-dspark, draft-dflash and draft-eagle3; with different prompts one
sequence's acceptance collapses (eagle3 0.15 vs 0.40). After the fix:
dspark 0.71/0.78, dflash 0.56/0.44, eagle3 0.35/0.42.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ui: expand the chat form into a large editor for longer messages
The chat form expands into a large editor once a message has markdown
beyond plain paragraphs or reaches four lines, and a Maximize button in its
top-right corner toggles it by hand. The expanded form starts around half the
screen height and grows with the text up to the space below the tabs bar,
animating its height by mirroring the editor's content height.
On a new chat the expanded form keeps the collapsed form's top edge and the
greeting only fades out, waiting for a collapse to settle before it fades
back in. In a conversation the messages follow the form as it grows, so the
latest one stays in view above it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: make the expanded chat form a full-screen sheet on phones
In the expanded editor Enter adds a line and Cmd/Ctrl+Enter sends.
On a phone the expanded form is a full-screen sheet with no corners or
border. The page around it and Safari's bars take its colour, and the
sidebar's logo button and the working-directory chip step aside while it is
open. It moves by relative offsets and a clip on the main thread, so its
edges, text, actions and chip stay in step in every engine, and its content
keeps the collapsed card's places and widths, so nothing reflows when it
settles. Expanding does not focus the editor on a phone.
On a new chat on desktop the form is lifted by its sticky bottom offset
instead of a translate, and the expanded size is built on the collapsed
card's real height, so its top edge holds still through any sequence of
quick toggles.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the phone menu toggle and the sheet's focus edge while expanded
On a phone the menu toggle rides on the page, above the chat form: it now
fades out while the expanded form covers the screen, and comes back when
it collapses. The expanded sheet also drops the glass rim and inner ring,
so focusing the editor no longer outlines the whole screen.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the expanded phone sheet clear of the status bar
The expanded chat form fills the screen on a phone, under the status bar
an installed app draws over the page. Its text now starts below the bar
(pt-safe) and its collapse button sits below it too (top-safe-2.5).
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the phone's context gauge while the chat form is expanded
The gauge pinned to the phone's top right sat over the expanded sheet's
collapse key. Like the menu toggle across from it, it now fades out while
the expanded form covers the screen and comes back when it collapses.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: lift the expanded phone sheet's actions above the iOS keyboard
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move the chat form with a short, quick drift, without sliding after it lands
Its position transition came back on before the browser had laid out the new position,
so it slid the form there after the fade; the drift is now 10px each way and quicker.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style the expanded chat form with master's surfaces until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tighten the expanding chat form after review
- Expand on markdown only in finished lines: a lone `-`, `#` or `1.` on the
line being typed no longer opens the large editor.
- Show "Cmd/Ctrl+Enter to send" by the send button while expanded.
- Move the expansion state, the phone sheet and the held collapsed height
into useChatFormExpansion, and the sheet's geometry into pure functions.
- Drop the working-directory chip's phone animation, which never ran since
the phone chip moved to the + menu.
- Clearing or sending mid-move cancels the sheet's animation.
- A click only continues at the end of the text when it lands below it.
- Reuse ActionIcon for the toggle and INPUT_CLASSES for the card, derive the
centered form's offset from its drop, name the shared easing and fade.
- The greeting owns its fade behind the expanded form.
- MarkdownEditor's view is reactive, so following its content starts once it
exists.
Assisted-by: Claude (Fable 5.2)
* ui: keep the expanded chat form to the message being written
- The form expands on its own only for what is written or pasted, and
another conversation starts collapsed: a restored draft no longer opens
the large editor, or the phone sheet, on arriving at a chat.
- In the expanded editor the pickers open by the line being written, above
it or, short of room, below, instead of above the form's top edge.
- The sidebar's swipe is off while the phone sheet covers the screen.
- The send hint shows only beside the send button, where the row has room.
- The inner area keeps master's focus shadow, except on the phone sheet.
- The card and the form's position honour reduced motion.
- The phone greeting's place accounts for the wrapper it is counted from.
- The sheet's animation tolerates the form going away mid-move.
Assisted-by: Claude (Fable 5.2)
* ui: turn the typed pickers off on phones, and scroll long messages
- On a phone, typing `/` or `@` opens no picker for now: each mobile
browser moves the page its own way as the keyboard opens, so a list by
the form can not stay clear of the text being written. A note marks it
for later.
- The editor scrolls inside the form again: MarkdownEditor's own box sat
between the input and the editor without taking its height, so a message
longer than the form ran under the actions instead of scrolling.
Assisted-by: Claude (Fable 5.2)
* ui: settle the expanding chat form's review findings
- A phone expands the form only by hand: the sheet switching in mid-typing
moves the text under the keyboard.
- The typed pickers are off wherever a finger is the main pointer, not
below a viewport width, so a narrow desktop window keeps them and an
iPad loses them. The prompt picker opened from the + menu floats by the
sheet's actions on a phone.
- A message put back after a failed send expands the form again.
- Only the left button moves the caret from a click below the text.
- The sheet's fill is one token, used by the sheet, the page and Safari's bars.
- The collapsed-height hold is its own module and no longer finds the
editor's elements by class name.
- Shorter comments.
Assisted-by: Claude (Sonnet 5.5)
* ui: keep a one-line message on the chat form's actions row
On a desktop the main chat form is a single pill while its message is one
line: the add button, the text, then the model selector and send. A second
line, or a line that wraps beside the actions, brings back the two rows,
and they stay until the text is shorter than where it wrapped, so the
wider row can not flip it straight back. The editor stays mounted: the
actions join the text's row by CSS alone.
The single row's corners are `--chat-form-single-row-radius` in app.css, a
pill by default; the "Scrapbook / ChatForm single row" story shows the real
form with other radii.
Assisted-by: Claude (Opus 5.5)
* ui: give the one-line chat form 1.25rem corners
Assisted-by: Claude (Opus 5.5)
* ui: pick the working directory from the + menu on desktop too
The chip under the chat form is gone: the + menu offers the working
directory everywhere but a phone's chat, whose header shows it. The menu
item names the directory once one is set, and the picker gains a row to
clear it, which the chip's X used to do.
Assisted-by: Claude (Opus 5.5)
* ui: put the single-row chat form behind a Display setting
"Single-row chat form for short messages", on by default.
Assisted-by: Claude (Opus 5.5)
* ui: send the chat form on Enter, and add a line with Shift or Ctrl/Cmd
Enter sends wherever the caret is, the expanded editor and code blocks
included; Shift+Enter and Ctrl/Cmd+Enter add a line, continuing lists and
quotes. With "Send message on Enter" off the two swap as before. The
expanded editor's send hint and the code-block rule are gone with it.
Assisted-by: Claude (Opus 5.5)
* ui: start the chat form on the row layout it will settle on
Assisted-by: Claude (Opus 5.5)
* ui: show a set working directory under the chat form again
On desktop the chip under the form is back once a directory is set, and
the + menu offers the directory only until then.
Assisted-by: Claude (Opus 5.5)
* ui: add a line on Enter with an on-screen keyboard
A phone's Return has no Shift, so on a touch screen Enter adds a line and
the send button sends, as "Send message on Enter" turned off does; on a
desktop Enter still sends.
Assisted-by: Claude (Opus 5.5)
* cuda : Lowering VMM pool reservation size to 8 GB on GPUs with combined memory due to CUDA out of memory errors
Assisted-by: Gemini Argon
* cuda : Halve VMM pool reservation size and retry on failure
Assisted-by: Gemini Argon
* cuda : only shrink VMM pool on CUDA_ERROR_OUT_OF_MEMORY
Assisted-by: Gemini Argon
* cuda : log warning when VMM pool reserves less than max size
Assisted-by: Gemini Argon
ggml_cuda_graph, ggml_cuda_concurrent_event and ggml_cuda_stream_context
are only used by ggml-cuda.cu, but keeping them in common.cuh meant any
change to them rebuilt every CUDA translation unit. Move them to graph.cuh
and leave forward declarations in common.cuh. The context constructor,
cuda_graph() and any_cuda_graph_*() are now defined in ggml-cuda.cu, and
the stream context is held through a unique_ptr.
Also define ggml_cuda_kernel_can_use_pdl once in ggml-cuda.cu instead of
as a static function in the header, so there is one PDL cache instead of
one per translation unit, and fold the GGML_CUDA_PDL env check into it.
* ui: use a live-preview markdown editor for the chat form input
Replace the contenteditable rich input with a CodeMirror editor that shows
the markdown source of the block being edited and renders every other block
through the MarkdownContent pipeline and stylesheet, so they match chat
messages. Plain text types exactly as before, the value stays the exact
markdown source, and mention links show as atomic badges.
The editor lives in a shared MarkdownEditor component, and the highlight.js
theme loader is shared between MarkdownContent and SyntaxHighlightedCode.
The old rich input, its tokenizer, source history and their tests go away.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the production build more memory, note what mdsvex is for
The single-bundle build now includes CodeMirror and runs past Node's default
heap limit on CI runners. Raise it in the build script, so local and CI builds
behave the same on every platform.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the live preview's blocks while only the cursor moves
Moving the cursor re-walked the syntax tree and rebuilt every block's
decoration. Keep the blocks with the decorations and rebuild only when the
text changes or the cursor enters another block.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: point the chat form story at the editor and drop the engraved placeholder until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the markdown editor for screen readers
CodeMirror's content is a textbox with no name of its own, so the chat form
input was announced as an unnamed text box, and could not be found by its name
as the old input could. It takes an aria-label, its placeholder by default,
which makes the chat form's "Type a message..." again.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: format the chat form input's classes
Assisted-by: Claude (Opus 5.5 Medium)
* ui: share the markdown styles and diagram drawing between messages and the editor
The editor and MarkdownContent each @imported markdown-content.css into their own
<style>, so the bundle carried two scoped copies of it. Both now render into a
MarkdownSurface, the one element that carries the stylesheet; the few rules for
MarkdownContent's own template elements are global under .markdown-content.
Drawing mermaid and svg blocks and the actions of their header buttons move to
markdown-diagrams.ts, so the editor's rendered blocks can draw diagrams the same
way messages do.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: harden the live preview editor after review
- onChange skips only the editor's own syncs from `value`, marked by an
annotation, instead of reporting only edits that carry a user event
- Enter and the mention badges ask the syntax tree whether they are in code, so
~~~ fences and indented code count, and a closing fence ends a block
- a picked mention is one undoable edit, through replaceText
- rendered blocks draw mermaid and svg diagrams with their header buttons, a $$
math block with blank lines renders as one block, and a link reference
definition stays raw instead of rendering to nothing
- rendered blocks render again when the mention path settings change
- a drag in a rendered block selects its text and copy takes it, across blocks;
a click still opens the block
- the block margins are the ones a chat message renders (each heading is a first
child there), checked by a test; the blank lines after a raw block are its gap
- the tree comes from syntaxTree, without forcing a full parse on every edit
- placeholder, label, extensions, mention widgets and render key follow prop
changes after mount, through compartments
- the editor uses the markdown language alone, which drops the html, css and
javascript parsers markdown() pulls in for html tag completion (-70 KB gzip)
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the CodeMirror packages and set the build heap in .npmrc
The new dependencies are pinned like the rest of package.json. The build heap
moves from a node call on vite's bin path to npm's node-options, so the build
script is plain `vite build` again and the setting reaches it on every platform.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: update the live preview block by block and estimate block heights
An edit used to build every block's decorations and widgets again, which the
editor then compared one by one. The previous decorations, and the mention badges
of raw blocks, now move through the edit, and only the blocks whose range, text,
raw or rendered state, or gap changed get new ones; the texts of the blocks an
edit left alone are reused instead of sliced again. A test drives random edits,
cursor moves and focus changes and checks each result against a full build.
Rendered blocks report the height they were last measured at, or a guess from
their text, so CodeMirror's height map for blocks off screen in a long document
stays close and scrolling stays steady.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: skip the editor's redundant syncs, reconfigures and theme reloads
- syncing `value` compares it with the text the editor last reported or synced, so
a keystroke coming back through onChange no longer serializes the document again
- the prop effects reconfigure the editor only when a prop changed, not once more
right after it is created (which rebuilt the whole preview)
- loadHighlightTheme leaves the theme's style element alone while the mode stays,
so a message or an editor mounting no longer restyles the whole page
Tests check that typing in one block leaves the other rendered blocks without a
single DOM mutation, and that mounting reconfigures nothing.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: simplify the live preview to a plain rebuild in three small modules
Rebuilding the decorations of every block on an edit measures the same as
patching only the changed ones (0.09 ms against 0.11 ms per keystroke at
200 blocks, 0.9 ms against 1.1 ms at 2000), and equal widgets keep their
DOM either way, so the incremental update and its test go.
- markdown-blocks.ts collects the document's blocks, markdown-block-widget.ts
renders one, markdown-live-preview.ts decides which ones show raw text
- one bounded cache holds a block's HTML and its measured height
- the render key is a plain option: a new one reconfigures the preview,
which keeps the editor's focus
- MarkdownEditor follows its props with one helper instead of five effects
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep setext headings raw in the live preview editor
`test` over `---` is a setext heading, so the preview rendered it as an h2
and the underline vanished once the cursor left the block. The editor now
shows a setext heading's text as typed at all times, like a link
reference definition.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a setting to turn the live markdown preview off
"Live markdown preview in the message input" (General) is on by default.
Off, the input shows the plain markdown text, with file mention badges
still in place; switching it takes effect without a reload.
Assisted-by: Claude (Opus 5.5 Medium)
normalize_quotes_to_json copies raw control characters verbatim while
converting pythonic tool arguments to JSON, so calls like
python(code='a<LF>b') produce invalid JSON in the arguments returned to
clients; replaying that history then fails hard. Escape C0 control
characters inside quoted strings via escape_json_string_inner.
* sycl: Add device-side MoE scheduler
Assisted-by: Codex
* Add width-heuristic override bit and Q2_K and Q3_K types
Assisted-by: Codex
* tests: cover skipped MUL_MAT_ID slots on the device MoE schedule
Assisted-by: Claude Opus 5.5
* sycl: keep graphs on for lazily reordered MUL_MAT and reject BN=64 combos on the device MoE schedule
The graph check disabled SYCL graphs until every MUL_MAT weight with a
reorder extra was reordered, which never happens for some (dense MXFP4,
the fused GLU plain path), so graphs stayed off and the check reran on
every compute. Upstream already reorders those during recording with
the async memory extension, so only MUL_MAT_ID weights are reordered
ahead of recording now.
ggml_sycl_grouped_dequant_gemm_device_supported() accepted any XMX
combo, but the device schedule only runs with BN == GGML_SYCL_FG_BN, so
a 32x64 combo (GGML_SYCL_XMX_GATHER_SHAPES=4 or 8) made the graph check
pass and the compute fall back to the host sync path while recording.
Assisted-by: Claude Opus 5.5
* sycl: keep the ggml_sycl_mul_mat_id_needs_sync name for the graph check
Assisted-by: Claude Opus 5.5
* fix: stop erroneously reordering host-offloaded tensors
With -ncmoe, prompt processing offloads the CPU-resident expert matmuls
to the GPU: the scheduler uploads the experts into a copy in the compute
buffer before every run. init_tensor gave that copy a reorder record, so
the MUL_MAT_ID gather path reordered every offloaded expert tensor on
every ubatch, and the next upload overwrote the result.
Qwen3-30B-A3B Q4_K_M, 1x B60, llama-perplexity -c 4096 -ub 2048
-ncmoe 48, 8 chunks, on master ac8e1694:
master 116 s, PPL 7.739
master XMX_GATHER_TYPES=0 67 s, PPL 7.738
with this change 67 s, PPL 7.729
Refs ggml-org/llama.cpp#31262
Assisted-by: Claude Opus 5.5
* Remove SYCL-specific test comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sycl : name the MUL_MAT_ID path predicates after what they decide
ggml_sycl_mul_mat_id_mmvq_ok -> ggml_sycl_mul_mat_id_should_use_mmvq
ggml_sycl_mul_mat_id_mmvq_shape_ok -> ggml_sycl_mul_mat_id_mmvq_eligible
ggml_sycl_mul_mat_id_grouped_shape_ok -> ggml_sycl_mul_mat_id_device_sched_eligible
node_is_device_local -> graph_node_is_in_local_device
ggml_sycl_gg_rows -> ggml_sycl_grouped_gemm_rows
ggml_sycl_gg_tile -> ggml_sycl_grouped_gemm_tile (from #29245, renamed to match)
"eligible" is a necessary condition; the type and width checks follow
elsewhere. "should_use" means the path runs. No functional change.
Assisted-by: Claude Opus 5.5
* sycl : reorder Q3_K MoE experts and load them as vectors in the grouped GEMM
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: cwriter <cwriter@localhost>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vulkan: tiled transpose for dim-0 concat with a transposed source
The delta-net conv-state path does ggml_transpose() straight into a dim-0
ggml_concat(), so the generic concat kernel reads src1 with the transposed
stride, one element per thread, and neighbouring lanes land on different cache
lines. Stage a 32x32 tile through shared memory instead, which keeps both the
load and the store coalesced. 33 columns pads away the bank conflicts.
Gated narrowly: dim 0, 2D, unquantized 4-byte elements, contiguous src0/dst
rows, and src1 actually transposed. Everything else falls through to the
existing kernel.
Qwen3.6-35B-A3B on gfx1151/RADV, 30 gated-delta-net layers: the CONCAT op goes
11877 us -> 957 us, and pp2048 at ub2048 goes 1224 -> 1628 t/s. Generated text
is unchanged.
The approach follows the tiled transposed concat Gaetan Puleo wrote for the HIP
backend in his Strix Halo fork.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* enable by default, add smaller shader sizes
* clean up comments
* improve shader
* Add non-contiguous concat tests
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
* handle x or y axis overflow
---------
Co-authored-by: Nathan Wilson <nath.flagman151@passmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
* ggml-rpc: fail gracefully on unreachable endpoints and failed handshakes
rpc_dispatcher::start() aborted the whole process (GGML_ABORT -> SIGABRT) when
the endpoint could not be parsed, the transport failed to initialise, the socket
could not connect, or the HELLO handshake failed. A typo in --rpc or a
temporarily down node killed the host application, and the reason went to a log
stream the host may never see.
Make the connect-time failure reportable instead of fatal: start() returns bool
(LOG_ERROR + false; the dispatcher thread only starts after a successful
handshake), negotiate_hello() returns false on a half-open socket,
get_dispatcher() returns nullptr and does not cache failed attempts, and every
consumer treats nullptr as "endpoint did not answer" with the conservative
behaviour of its domain (allocation -> nullptr, alloc-size -> lower bound,
device memory -> 0/0, events -> no-op, device count -> 0, and add_server()
already maps 0 devices to nullptr).
Mid-stream failures keep the existing RPC_STATUS_ASSERT by design (fail fast
against silent corruption) and are intentionally out of scope.
Field motivation: on Android (llama.rn + phone RPC cluster) an unreachable
worker crashed the app; the crash buffer showed
abort <- ggml_abort <- rpc_dispatcher::start <- ggml_backend_rpc_add_server.
* tests : add a regression test for unreachable rpc endpoints
Calls ggml_backend_rpc_add_server with a socket that nothing listens on and with a
malformed endpoint, and expects nullptr from both. Without the fix in this branch
the first call aborts the process, so the test cannot even fail cleanly.
* ggml-rpc remove low-value comments
* tests : drop the internal device-count check after the rpc registry refactor
---------
Co-authored-by: figkcoci-debug <figkcoci-debug@users.noreply.github.com>
With head sizes above 128 and KV >= 1024, scalar flash attention uses
16 rows per workgroup. On the Qualcomm Adreno driver this variant
spills registers to scratch memory (264 bytes per invocation on the
X1-85, none at 8 rows) and runs about 40x slower. The largest
test-backend-ops case then becomes a single dispatch of several
seconds, past the Windows GPU timeout, and the test never completes on
the X1 Elite.
Use at most 8 rows per workgroup on Qualcomm when hsk or hsv > 128.
Smaller heads keep 16 rows, which is faster for them on the X2 Elite.
Adreno X1-85, FLASH_ATTN_EXT hsk=256 kv=4096: nb=64 711 ms -> 17 ms,
nb=512 no result -> 125 ms; test-backend-ops completes. Adreno X2-90:
Gemma 4 E4B pp512 at depth 4096 with flash attention 81-90 -> 117 t/s.
On a device without mmap support, --lazy-mode auto resolved to off even
with --load-mode mmap, so llama_prefetch_rows never ran and the rows of
the large tensors faulted in one page at a time. Resolve it to off only
when mmap is off; the default iGPU load path is unchanged.
Fixes#31943
* server: add oai-compat /v1/audio/speech endpoint
* add e2e pytest
* server: share the s16 rounding, type speech form fields
The WAV writer of the audio helper truncates while pcm_f32_to_s16
rounds, so wav and pcm differ by one LSB. Both now round.
pcm_f32_to_s16 takes a pointer and a count and converts the TTS
bytes without an intermediate copy.
/tts and /v1/audio/speech parse their body with parse_tts_body, so
multipart fields such as speed get their type before the speech
conversion checks them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* mtmd: support gemma 4 input audio stream
* use 480ms step
* server: enable realtime embeddings only for audio generation
The realtime slot forced embeddings on, which marks every row of a
batch as an output. Only a model that generates audio reserves
n_batch outputs, so a chat model with an audio stream (Gemma 4)
aborted on its first batch of audio rows. The hidden state is only
read by the audio generation pipeline.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
A cold dev server reloads the page once it has optimized its
dependencies, and that reload can land right after the harness opened
the phone sidebar, closing it again before Settings is clicked. Repeat
the pass over the menu keys up to three times while Settings is not
reachable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: reveal the mobile sidebar by sliding the page aside
On a phone the sidebar now stays put under the page, and the page slides
right to uncover it, as a rounded card with the card border. The menu
button rides on the page; a swipe from the left edge opens the sidebar,
and a swipe back or a tap on the page closes it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: polish the mobile sidebar reveal and add a hold menu to rows
Holding a conversation row on a phone opens its action menu anchored to
the row, in place of the hidden three-dot trigger. The page no longer
widens the document when it slides aside, so iOS stops panning it on the
first open.
The page and sidebar backgrounds start in the page color and blend into
the sidebar tint as the sidebar opens, following a drag frame by frame,
and a theme-color tag follows them for the browser bars. The sidebar's
contents fade in on a log curve and scale up slightly, and sit at the
same inset as the menu toggle, which now uses a two-line icon in a
filled circle.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the context gauge on phones and lock the slid-aside page
On a phone the context gauge leaves the chat form for the top right of
the chat screen, in a round button across from the menu toggle. Both
take the scroll-down button's frosted look.
While the sidebar is out, and until the page is fully back in place,
the page neither scrolls nor takes taps; a tap or drag on it still
closes the sidebar.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the mobile sidebar with a swipe from anywhere
A rightward swipe from anywhere on the page now opens the phone sidebar
too, beside the edge swipe that competes with the browser's own back and
forward gesture. It starts after a short sideways move, skips text
fields and content that scrolls sideways, and keeps the page from
scrolling under it. Letting go opens or closes the sidebar past 30% of
the way or on a flick.
The top buttons and the chat form now hold their place on screen while
the page is slid aside, instead of jumping to the far ends of a
scrolled conversation.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the model org name on mobile unless a setting keeps it
A phone's model selector is narrow, so it now drops the organization
from model names by default. A new Display setting, "Show organization
name on mobile", brings it back; it applies only while the organization
name is shown at all.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: close the mobile sidebar before following a tap in it
On a phone, picking a conversation, New chat, Search, MCP or Settings
in the open sidebar now slides it shut first and acts 180ms later, so
the page or dialog appears once the sidebar is out of the way. With the
sidebar already closed, or on a desktop, the action runs at once.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: make the app feel native on phones and run it under the status bar
Applies the mobile-native checklist: no tap highlight or landscape text
inflation, instant taps on buttons and links, no label selection or
callout on a held control, hover styles only where a pointer can hover,
and 16px textareas on phones so focusing them does not zoom the page.
The page now runs under the notch (viewport-fit=cover), so the installed
iOS app tints its status bar from the page and the sidebar. The menu
toggle, the context gauge, the sidebar and the top of the chat and
models pages pad themselves clear of it, and the locked, slid page
reaches down by the status bar's height, which iOS leaves out of it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a viewport height readout for debugging on phones
With VITE_DEBUG set, a dev build shows the window, viewport unit and
safe-area heights, and the elements that reach past the window's bottom
and make the page scroll. It is loaded on demand, so production builds
leave it out.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fill the screen from launch in the installed iOS app
iOS 26 gives an installed app with viewport-fit=cover and a translucent
status bar a viewport short of that bar (WebKit bug 301108), clipping
the bottom of the page and lifting the chat form, but only while the
page is shorter than the screen. In the installed app the page now
reaches the screen plus the status bar, capped at the screen, so it
gets the full screen from the start without ever scrolling.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: raise the phone's corner controls in the installed app
The installed app's status bar already sets the menu toggle, the context
gauge and the open sidebar's logo apart from the screen's edge, so they
sit just below it rather than a full 1rem lower. They share one value,
which stays at 1rem below the inset in a browser tab.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add safe-area utilities for what is pinned to a screen edge
The page runs under the status bar and over the home indicator, which an
installed iPhone app always shows. New pt-safe, pb-safe, top-safe and
bottom-safe utilities, each with a spacing step (pt-safe-12 is 3rem plus
the top inset), keep pinned content clear of them instead of hand-written
calc()s. A --safe-area-bottom joins --safe-area-top, and the chat screen and
Models page use pt-safe-12.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open search in a full-height drawer on phones
A phone used to leave for the /search page. It now opens the same search
as the desktop dialog, in a bottom drawer that fills the screen below the
status bar. Keyboard shortcuts, the Enter hint and the keyboard highlight
stay on desktop, and so does the models manager, as in the phone sidebar.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show every dialog full screen on phones, clear of the status bar
A phone now shows every dialog full screen, whatever size it asks for on a
desktop, instead of each full-screen dialog repeating its own classes. Its
top and bottom borders take the status bar's and the home indicator's
height, transparent, so an installed app's dialogs keep their content clear
of both with no change to their padding. The attachment preview, which has
its own full-screen content, pads itself and lowers its close button the
same way.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fade the page out under the status bar of the installed app on phones
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show search in the full-screen dialog on phones instead of a drawer
Assisted-by: Claude (Opus 5.5 Medium)
* ui: start a phone page's title a row below the corner controls
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the slid page's bottom bars against the screen in the installed iOS app
Assisted-by: Claude (Opus 5.5 Medium)
* ui: cast a soft shadow from the slid page onto the phone sidebar
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the scroll-down key at the real bottom on iOS and show it only well above
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fade the status-bar fade out as the page slides aside for the sidebar
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw the slid page's shadow above its rounded frame, so it follows the corners
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a phone chat's title and working directory at the top, and pick it in a drawer
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give iOS Safari a solid top and fade the bottom of the screen on phones
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let iOS Safari find the top strip, which it skips when it takes no taps
Assisted-by: Claude (Opus 5.5 Medium)
* ui: lengthen the top fade and lift the chat form a little in phone browsers
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give iOS Safari's bottom toolbar the page's colour, not the chat form's
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw the phone's context gauge key like the menu key across from it
Assisted-by: Claude (Opus 5.5 Medium)
* ui: confirm opening the phone sidebar, its hold menu and its rows with a haptic tap
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the screen-edge fades on the page as it slides aside for the sidebar
Fading them out with the slide made the page flicker; the rounded frame over the
slid page already clips them at its corners.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the working directory chip optional for hosts with their own trigger
The phone layout opens the picker from the chat's header, so ChatForm hides the
chip there. Master's version of the chip has no such switch; bring it back.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style layer 5's additions with master's tokens until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the design system's stylesheets and fonts from app.css until it lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep master's font stacks until the design system brings its fonts
Assisted-by: Claude (Opus 5.5 Medium)
* ui: restore master's colour palette in app.css until the design system lands
Resolving the safe-area commit took the design system's app.css, which drops
the palette for its token files and aliases the shadcn names to semantic
tokens. With the tokens gone those aliases point at themselves, so every
colour went invalid: transparent surfaces and currentColor borders.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: center alert dialogs on phones instead of sliding them up as drawers
An alert is a short question, not a drawer: on a phone it is now a centered
card, 90% of the screen wide and as tall as its text, scrolling only past the
screen's height. It zooms in as on desktop, and its footer drops the margin
the sheet needed.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: space the working directory drawer for a phone
The drawer reused the popover's compact picker: Browse sat against the search
field, the divider ran past the rows and each row had its own inset. In the
drawer the picker now takes the header's inset, gives Browse room, lines its
rows up with the search results and sizes its text for a phone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tap the haptic engine when a message sends, and once per hold
A phone now confirms a sent message by feel, at the moment the send goes
through (not on every press of the button), and before any await, so it still
runs inside the tap or keypress that iOS needs to play its tick. The hold that
opens a row's menu now taps the engine from the long-press hook itself, once:
the hook vibrated and the row tapped as well, which buzzed twice on Android.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the gauge dial sizes, the chat form's bottom offset and the sidebar fade
Review follow-ups: the context gauge dial reads its grid, radius, stroke and
track from one table per size instead of ternaries; the chat form's distance
from the screen's bottom per platform and the sidebar content's fade duration
are named constants. The viewport debug readout moves out of src, next to the
tests, as it is a debug tool the app only loads in a dev build.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say why the microphone can't record instead of failing silently
A llama-server opened over plain HTTP on the local network or over Tailscale
is not a secure context, where browsers hide the microphone, so a tap on the
mic did nothing but log to the console. The button now shows a crossed-out mic
in the warning tone with the reason in its tooltip, and a tap says it in a
toast, as a phone shows no tooltip. A start that fails names the browser's
reason (access denied, no microphone, in use by another app) the same way,
and the next tap tries again. The recorder passes the browser's error on
instead of replacing it with a generic one.
Assisted-by: Claude (Opus 5.5 Medium)
* model : allow loading external PLE files
Assisted-by: Codex
* convert : add separate PLE exports
Assisted-by: Codex
* tests : fix PLE export test lint
Assisted-by: Codex
* model : make --ple an alias of --shard, allow multiple shard locations
Reuse the multi-shard loading path for external PLE files instead of a
separate option.
* --ple is an alias of --shard: both take a comma separated list and may
be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
common_model_load() uses llama_model_load_from_splits for two or more
paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
memory measurement load every shard
Assisted-by: GPT-6-Astra
* rm -isnm
* make sure to dedup paths
* rm test_convert_ple
* address comments
* allow ple shard to override tensor
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* Windows unbuffered io
* Fix read len calculation
* Add test and model load timing
* chunked fallback read + removing filepointer semantic divergence between linux and windows
* tests : remove unused mmap test arguments
Assisted-by: Codex
* remove timing, set file pointer after chunk read, test refinement
* Update tests/test-llama-mmap.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Update src/llama-mmap.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* test refinement
Assisted-by: Codex
* remove duplicate constexpr
Assisted-by: Codex
* adding label behavior to tests
Assisted-by: Codex
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* server : functionality test to make sure every time when we add new spec dec method the existing ones are not broken.
* server: use compact models for speculative decoding CI
* ci: avoid GHA cache for speculative models
* ci: run speculative tests on DGX Spark
* server: address speculative test review
* server: use consolidated speculative models
* server: refine speculative test calibration
* docs: record speculative Spark validation
* ci : run the server-speculative steps with bash
* server: recalibrate speculative test floors on GB10
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server : accept top-level schema in response_format with type json_schema
The OpenAI-compatible chat completions endpoint only applied the schema
when it was nested inside response_format.json_schema.schema. When the
schema was provided directly as response_format.schema (as documented in
the server README, and used by some clients), it was silently ignored
and generation ran unconstrained.
Also mention json_schema in the error message for invalid response_format
types, document the OpenAI wrapper form in the README, and add a test
case for the top-level schema form.
Fixes#10732
* server : let the json_schema wrapper take precedence by presence
Pick the schema source based on whether response_format has a
json_schema key, instead of falling back when the parsed wrapper schema
is empty. An empty schema {} is valid (any JSON value), so the old check
let a top-level schema override an explicit {} in the wrapper. Use
json_value() for the top-level schema too, like the rest of the
function, and add tests for the precedence.
---------
Co-authored-by: linsen <251731047+linsen458-spec@users.noreply.github.com>
* ggml: implement the backward pass for TRI
GGML_OP_TRI had no case in the backward switch, so a trainable graph containing it aborted with "unsupported ggml op for backward pass: TRI". The op is linear, so its gradient is masked the same way. The incoming gradient is not necessarily contiguous (the backward of a transpose hands over a view) and ggml_tri requires contiguity, so it is made contiguous first.
test_tri gains a post_transpose variant (out = transpose(tri(a))) that aborts as above without this change.
* ggml : simplify the TRI backward pass
Pass the gradient directly to ggml_tri and drop the transpose test. The existing TRI cases cover the backward pass for contiguous F32 gradients.
The dummy-model generator fills every tensor from a plain normal pool.
`blk.N.ffn.act.scales` is a divisor in the mpt graph, so with seed
1224489258 it draws |v| ~ 1e-6, and `ffn_act = gelu(...) / scales` reaches
2.7e5. Metal stages both operands of `kernel_mul_mm_f32_f32` as half, so
those activations overflow to inf, the norm downstream turns the inf into
nan and the logits come out nan, while the CPU reference stays in f32 and
stays finite. Both NMSE values then become nan and the check fails.
Give `ffn.act` the same treatment as `rope_factors` and keep its magnitude
away from zero.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* ui : fold the sidebar's conversation sections
Pinned and Recent conversations fold under their heading. The whole heading
row is the toggle; its chevron follows the title, shows on hover and stays
while the section is folded. Folded sections are remembered across reloads.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : show a sidebar section's first six chats
Pinned and Recent list their first six chats and end with a Show more
row that opens the whole section; Show less, sticking to the bottom of
the sidebar while the section scrolls under it, closes it back. The
expanded sections last for the session.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : rearrange the sidebar's sections and chats by dragging
Pinned and Recent can be dragged by their title. The real section follows
the pointer, tinted like the sidebar's resize handle and edged with a
faint border, the other sliding aside as it passes; let go, it glides from
there into its new slot. The order is remembered across reloads, and a
section never moved keeps its default place. All sections now scroll
together.
A chat dragged onto Pinned is pinned, onto Recent unpinned. A lifted copy
of the row follows the pointer while the row slides out of its section.
The section under the pointer takes the same tint and border; a section
ignores its own chats. Let go, the row slides in where its section sorts
it, the rows below making room, and the copy glides into it; anywhere
else, it slides back into its own section.
While dragging, the rows and headings passed over take no hover. Dragging
needs a mouse or a pen, so a finger still scrolls the sidebar; the sidebar
scrolls near its edges, and Escape cancels.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : set the sidebar's sections apart from their rows
A section heading now reads above its chats: larger, semibold and in the
full foreground colour. Chat titles drop to a regular weight in a slightly
softened colour, full again when the chat is open or hovered, and "Show
more" drops to a regular weight in the muted colour. Chats inside a
section sit closer together, sections further apart, and the sidebar's
actions get more room above and below.
Assisted-by: Claude (Opus 5.5 Medium)
* fix(rocm,ci): Update ci workflow to use devlab ci runners for gfx1151
* ci : move gpu-rocm job to Strix Halo devlab runners
Assisted-by: Claude
* ci : ROCm 10 wheels with per-GPU device packages for gpu-rocm; add gfx1201
The runners' system ROCm 7.2.1 cannot compile the current HIP code for
gfx1151 ("Invalid dpp_ctrl value" in mmf-instance-ncols_*.cu). Go back
to the TheRock ROCm 10 wheels with the device-<gfx> extra, so rocBLAS
has the target's kernels, and keep ci/run.sh's results and models under
$RUNNER_TEMP instead of the runners' RAM-backed home. Add a gfx1201
(Radeon AI PRO R9700) row with HIP graphs disabled.
Assisted-by: Claude
---------
Co-authored-by: David Hekimian <dhekimian@users.noreply.github.com>
* metal: make ggml_tri select instead of scaling by the comparison
The kernel multiplied by the 0/1 comparison, which turns a +-inf outside the triangle into NaN, while the CPU kernel writes a plain zero there. Select, like the CPU does.
Adds test_tri_special, which puts +-inf into the cells that have to be discarded, for all four triangle types. It fails against the previous kernel and passes with this change.
* Just trim down the repetitive comments.
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3
Assisted-by: Claude Opus 5.5
* feat: add temp parameter in llama-tts
* temp: fix default hardcoded values
* fix: remove too narrow test
* fix: addressing PR comments
The rotation angle of a position p is rounded in f32 to about
p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach
it through different rounding sequences (cumulative product versus
pow). With positions up to 8192 this puts the NMSE floor between
backends near 8e-8, so the default 1e-7 threshold fails at random on
Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for
n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
* ggml : fix for the radix TOP_K thread timing bug
* make the key an ordered 64-bit pair
* a rewrite without shift-skip logic
* fix casts
* warp-contiguous layout
* reduce the scope removing vulkan+metal changes
* turn it off for Metal/ROCm, note about cub::DeviceTopK
* fix the hip build
* skip Vulkan as well
* popc lambda from fattn
* do block_cols math in int64_t
* make it allowlist bc SYCL looks non deterministic too
* ggml : make Metal radix Top-K tied selections deterministic
The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.
Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.
This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : enable deterministic top-k check on Metal, add radix-path tie cases
Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.
Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : use coalesced accesses in the Metal radix Top-K compaction
The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.
Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : remove the extra top-k tie and perf cases
The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal: scan the radix bins and warp totals with a simdgroup
The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.
* tests: enable the deterministic top-k check on ROCm
HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.
* cuda: remove the redundant radix top-k counter reset
Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.
Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).
Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.
Assisted-by: Claude Sonnet 5
Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
* ggml : switch to F32 src1 in the CPU backend
Change the default computation for GGML_TYPE_F16 to use F32 for src1
data in matrix multiplications: vec_dot_type(F16) is now F32 and the dot
product goes through the new ggml_vec_dot_f16_f32 (x = F16 src0, y = F32
src1), so src1 is no longer quantized to F16 and worst-case inputs that
only fit in the F32 FP range are handled correctly.
- conv_2d/conv_3d: store im2col patches (the inner mul_mat src1) as F32
and pass separate src0/src1 types to the inner mul_mat; the previous
F16 patches made the inner mul_mat widen src1 into the conv work
buffer, corrupting the patches and crashing on large inputs
- conv_transpose_1d/2d (F16 kernel): keep src1 in F32 in the work buffer
and use ggml_vec_dot_f16_f32; update work buffer planning accordingly
- supports_op: restore MUL_MAT support for F16 src1, which is now
widened to F32
- ggml_vec_dot_f16_f32: add GGML_SIMD support by converting chunks of x
to F32 and reusing the F32 SIMD dot
- mul_mat/mul_mat_id: add TODOs for a more efficient F16 x F16 path that
can be used if ggml_prec_get_src(dst, 1) allows converting src1 to F16
MUL_MAT f16 x f32 perf on M5 Ultra (GFLOPS):
- m=2048, n=512, k=16 : 120.9 -> 101.7
- m=2048, n=512, k=64 : 150.5 -> 344.2
- m=4096, k=14336 GEMV: 63 -> 343-415
- m=4096, n=512, k=14336: 65.6 -> 770.2
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : add some F16 x F32 vectorized paths
* cont : enable the fused F16 x F32 dot on AVX without FMA
The 256-bit fused path of ggml_vec_dot_f16_f32 required AVX2 and FMA,
so CPUs with AVX and F16C only fell back to the chunked conversion and
F16 decode was 4x slower than before. Enable it on __AVX__ && __F16C__
and accumulate with GGML_F32x8_FMA, which does mul + add without FMA.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls
Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.
* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends
Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.
* tests: cover GGML_PREC_BF16 in the flash attention tests
The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.
* llama: request GGML_PREC_BF16 for the KQ multiplication
* cont : add TODO in clip.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* opencl: add kernel_gemv_noshuffle_q8_0_f32_mc
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for q5_1, q4_k
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: adreno kernel k floor 256 for x2
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for more quants
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: rename function
* opencl: disable bin FA for small verify batch
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* hexagon: use absolute skel path
* hexagon: support non-ascii dir and long path in skel-abs path
* hex-libdir: add support for FASTRPC_MANAGE_DSP_LIBRARY_PATH and GGML_HEXAGON_LIBHTP_DIR on all platforms
* hex-libdir: keep extra fwd slash in abs path on Android and Linux
* hex-libpath: always start with absolute path then fallback to session ctl then env
* hex-libdir: add CDSP1,2,3 to the fallback, needed for the IQ10
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* hex-nhvx: apply nhvx updates to the latest master
* hex-nhvx: set N_HVX and N_THREADS based on the arch to avoid overallocation
* hex-build: add support for enabling HTP_DEBUG via build.py
* hex-trace: move tracing state/helpers into new htp-trace.h header
* hex-dcache: cleanup dcache #defines
* hex-hnvx: minimize the diff by reflowing the macros
* hex-ci: fix python type/linter warnings
* hex-nhvx: consitent logging
* mtmd: support MageVL-Cowork model
* mtmd: keep magevl rope_freqs at model level and check its shape
The rope factors are shared by every layer, so they live in clip_model
instead of a pointer copied into each clip_layer. The loader now checks
that v.rope_freqs is F32 with d_head elements and that d_head splits
into the 4:6:6 pairs the graph expects.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* moe-cache : compute part of the misses on the CPU
--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.
- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
selected experts, so the CPU part does not wait for the device, and its
result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
simulator of the cache
* scripts : add moe-cache-tune.py to pick the MoE cache arguments
The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.
docs/moe-cache.md describes the MoE cache arguments and the script.
* add am17an as codeowner for MoE cache
* moe-cache : upload all the misses until the cache is full
With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.
* add backend sync for async transfer queue
* moe-cache : let the scheduler copy the results of the CPU part
* Fix MSVC compilation
* remove sync change
* llama-graph : extract the CPU share condition into can_cpu_share
* llama-graph : make llm_graph_params::moe_cache const
The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* scripts : replay the ClassLRU policy in moe-cache-tune.py
* moe-cache : pick CPU misses by CRF instead of random (#31853)
* moe-cache : remove the backend logic of the CPU part from the graph
The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.
ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.
* llama-graph : return the MoE cache slots and CPU ids in a struct
The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.
---------
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
The kernel_group_norm_f32 kernel and its host dispatch ignored the
batch dimension (ne03) of the source tensor: only ngrp threadgroups
were dispatched and each normalized the flat range
[tgpig*gs, (tgpig+1)*gs) with gs = ne00*ne01*ceil(ne02/ngrp). For
tensors with ne03 > 1, all but the first ne00*ne01*ne02 elements were
left unwritten (exposed by the new GROUP_NORM test with
ne=[3,1,1,65537], num_groups=1).
Dispatch one threadgroup per (group, i03) pair (ngrp in x, ne03 in z,
same work distribution as the Vulkan backend) and compute per-batch
statistics, matching the CPU reference. Also clamp the group element
count for the last group when ne02 is not divisible by ngrp and use
int64_t indexing.
The kernel now only requires ggml_is_contiguous_rows(op->src[0])
instead of full contiguity: it addresses src0 and dst with their own
strides (nb03 and the dst strides are added to the kernel args - dst
can be a strided view of src0 with ggml_group_norm_inplace) and falls
back from the flat fast path to a strided path that decodes the linear
element index when rows or channels are padded. The flat/strided
selection is done on the host in
ggml_metal_library_get_pipeline_group_norm and compiled into one of
two specialized pipeline variants via a FC_group_norm_flat function
constant, so the dead path is eliminated at pipeline creation.
GROUP_NORM support is also gated on F32 src and dst in supports_op -
the kernel only implements f32 and an f16 input would silently be read
as f32.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* server: visit parent slots before child slots
A parent slot wakes its waiting child slots once it finishes the shared
prompt prefix, while slots are visited in id order. When a child task
lands on a lower slot id than its parent, the child is visited before
the parent wakes it and waits an extra batch to process its suffix.
Non-child slots are now visited before active child slots, so every
child woken by its parent joins the same batch.
* server: check systemone test events through test tags
The systemone tests matched free-form debug messages, including the
formatted f_keep value of the prompt cache. The server now emits
__TEST_TAG_WAIT_PARENT_SLOT__ when a child slot waits for its parent,
__TEST_TAG_PROMPT_CACHE_RESTORE__ on every prompt cache restore and
__TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when the whole cached prompt is
a prefix of the new one, compared on token counts.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* Only run webgpu CI on my fork
* Add webgpu only workflow
* Add WebGPU group_norm kernel
* clean up
* resolve merge conflicts
* parameterize data type; clean up cmake changes
* add nullptr check
* clean up webkit experiments
* more cleanup
* more cleanup
* clang format
* Remove unused feature check for WebGPU
* clean up supports_op condition and update documentation
* add perf tests for group_norm
* clean up doc with script and undo change in test-backend-ops
* Do 2d dispatch of workgroups
* Do 2d dispatch of workgroups
* Unstage unrelated changes
* undo docs churn
* add new test case
* undo small change
* undo small change
* ggml-backend-meta : minimum number of compute views per buffer
Assisted-by: Claude Opus 5.5
* tests : revert the deepseek41 test model to 6 layers
Assisted-by: Claude Opus 5.5
* hexagon: fuse post-norm residual add into RMS_NORM_MUL
* hex-fusion: refactor RMS_NORM fusion and add dedicated RMS_NORM_MUL_ADD opcode
Improves performance by keeping things in HVX registers.
Also simplifies overall fusion logic and logging.
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* model: add Kimi-K3.1-Pro text model
K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.
Indexer:
- the selection is an additive mask over the whole cache, not a gather of
the top-k rows, so it reuses the existing sparse flash attention path;
off that path the mask stays correct but still costs a kq over the cache
- the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
because dsa_config declares no eps of its own
Graph and cache:
- new arch with four indexer tensors per DSA layer, and a hybrid memory
that keeps a K-only MLA cache for those layers, recurrent state for the
KDA layers, and the indexer key cache
- the graph node budget shares the K3 entry
Conversion:
- KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
- dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
plus the indexer LayerNorm eps; the layer list is 1-indexed like
linear_attn_config's, and the converter asserts that it covers exactly
the full-attention layers
- the routed experts stay MXFP4 and are repacked as in K3, not dequantized
- the vision tower and mm_projector are skipped: text only for now
- the arch key follows the checkpoint's own model_type ("kimi_k31") and the
loader registers KimiK31ForConditionalGeneration; the release name carries
a -Pro suffix that nothing in the config or the tensor names uses
Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.
Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.
* converter: avoid clutter in the k3 base
* narrow comment to match model
* update metal fusion expectations
* restore
* sync with PR 31733 refactoring
* hexagon: Generalization of CONV_2D implementation for CONV_3D
- New op HTP_OP_CONV_3D is wired through htp-ops.h, main.c (shares op_conv_2d) and the op remap in htp-kparams.cpp.
- conv_3d runs on the existing HMX tiled (implicit GEMM) conv2d path. KD and KH are folded into one virtual kernel row (KD*KH), and output rows are counted as N*OD*OH.
- htp_kparams_solve_conv3d checks the conv_3d layout and calls the tiled solver htp_kparams_conv2d_tiled, which takes an is3d flag.
- The tiled kernel takes the sizes and strides that differ between conv_2d and conv_3d from a struct htp_conv_geom. Only the input stage unfolds depth.
- 256-entry DMA queue can no longer overflow
Assisted-by: Claude Code
* hexagon: modify checking to eliminate divide-by-zero hazards
* hexagon: conv2d tiled: compute the conv_3d unfold divisors on the host
The KH and OH fastdiv values used to unfold conv_3d kh into (kd, kh) and
oh into (od, oh) were built per op on the device, each with a 64-bit
divide. Pass them in the kernel params instead; they fit in the space the
window variant already reserves, so the struct size does not change.
Assisted-by: Claude Code
* hexagon : preserve fold scheduling in conv2d for small size
hexagon : support arbitrary positive conv2d strides
Support positive horizontal strides for WHCN and CWHN inputs. Preserve
load/shuffle paths for strides 1/2 and use gather for larger strides.
hexagon : improve conv2d tile size selection
Prefer the largest feasible output-channel block and derive spatial
chunk capacity from VTCM usage. Compare capacity and thread-aligned
candidates using per-thread work and chunk overhead.
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* hex-conv: fix compiler warning
---------
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* tests : qualify isnan and signbit with std::
The unqualified calls rely on the C math.h names also being present in the
global namespace, which is not the case for the NetBSD 11.99.9/x86_64 gcc
14.4.0 build reported in the issue. Prefix the calls with std:: to select
the standard C++ overloads declared by <cmath>.
ref: https://github.com/ggml-org/ggml/issues/1681
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : qualify C math calls with std::
Follow-up to the isnan / signbit fix: make the remaining math calls in
test-quantize-fns.cpp consistent with the std:: isfinite / max / numeric_limits
already used in the same file.
The float variants are overloads in C++, not distinct names, so the mapping is
fabsf -> std::fabs, copysignf -> std::copysign, nextafterf -> std::nextafter -
not std::fabsf & friends, which do not exist in libstdc++ and would break the
gcc build. NAN and INFINITY are macros and are left unqualified.
Only behavioural difference: array_rmse() fed a double accumulator to sqrtf(),
now std::sqrt() keeps the computation in double. The result is more accurate
and differs from the previous value by at most 1.4e-7 relative, three orders of
magnitude below the tightest tolerance in this test.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The self-hosted CPU runners are containers with 8 CPUs, but
std::thread::hardware_concurrency reports the cores of the host. The
table mode of test-llama-context sizes its pool from that count, so it
runs 4 models x 8 threads on an 8-CPU job and hangs.
Add TEST_ARG_HARDWARE_CONCURRENCY and TEST_ARG_PARALLEL_WORKERS as the
defaults for the auto-selected counts of test-llama-context and of
test-backend-ops, and set them to 8 and 4 for cpu-x64-high-perf.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* moe-cache : LRFU/MQ-inspired hybrid cache
* make the clock advance by N tokens: use the real token count (ne[1] is always 1, ids have n_expert_used entries per token)
* sel is a RESHAPE node, the orig input is in src[0], so sel->src[0]->ne[1] is the count
* c for the class index, last_use for last use time
* explain the roles of these
* compute the key once
* use the key directly instead of re-reading
* show how the score is computed from the age
* remove the branch and clarify exp2 and log2 relationship
* reword the header comment and elaborate on unused slots and sorting
* not a copy, not just skip
* std::array maybe
* Revert "std::array maybe"
This reverts commit 7649b1f172db4941a854ad115bec0c7e062b65b9.
* half life it is
* add key and iteration helpers
* extract decayed_crf
* bump classes to 8, document the formula, display the cache size in token worth count
* assert the computed number of classes matches the half life
* opencl: use the multi-column q6_K vocab head GEMV for 2..4 tokens on X2E
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q6K_MC3 to GGML_OPENCL_Q6_K_MC3
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q40/Q41/Q4K_MC3 to GGML_OPENCL_Q4_0/Q4_1/Q4_K_MC3
GGML_OPENCL_Q4K_MC3_DQ and GGML_OPENCL_Q4K_MC3_LDS become GGML_OPENCL_Q4_K_MC3_DQ
and GGML_OPENCL_Q4_K_MC3_LDS.
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: remove unnecessary debug probe
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: clean up
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hex-repack: move repacking/tiling into graph-optimize
* hex-tiling: simplify tiling rules and remove obsolete helpers
* hex-ops: use a hash for de-duplicating tensor descriptors
* hex-bufs: improve tensor hash
* hex-ops: accept empty (zero-elements) ops
* hex-bufs: enable hostbuf by default and reject tensors that do not fit into max_vmem
* hex-bufs: improve error message and update readme
* hex-bufs: make sure to sync flush opbatch in all sync set_tensor paths
* feat : added ricursiveai ripple support
* fix : mandatiry SWA, scale through helper, comments back
* fix : metal
* model : make ricursive RoPE/NoPE explicit and validate the layer pattern
Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.
Assisted-by: Claude Opus 5.5
* model : make ricursive swiglu_clamp per-layer
Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.
Assisted-by: Claude Opus 5.5
* model : use the ricursive per-tensor scales and RoPE parameters
Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.
Assisted-by: Claude Opus 5.5
* convert : drop the ricursive chat_template.jinja override
The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.
Assisted-by: Claude Opus 5.5
* fix : remove NVFP4 checkpoint quantization block
* match #31733 refactor
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* RDNA3/4 Vulkan Large-Tile Split-K Optimization
* Add single-operator performance test cases
* Update the policy to use driver-reported LDS usage
* Add comments
* Use the simpler split_k rule from review
Assisted-by: GitHub Copilot
* tests : drop the split_k test cases
* vulkan : simplify the RDNA3/4 split_k condition
* metal : support scale-bias routing norm in the TOPK_MOE fusion
The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sqrt(softplus) MoE routing with a selection bias
Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sigmoid MoE routing with a selection bias
The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.
- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
tail; the selection-bias ADD is required, same as the sqrt(softplus)
chains
- generalize the gating function constant from a sqrt(softplus) bool
to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
deepseek2/32, kimi, minimax, ...) now fuse their routing, and
muse-spark prefill drops one MOE_REDUCE fire because the reordered
graph allocates the routing weights inside the reduce output range,
which the dst/input overlap check correctly declines
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : remove low-signal comments
* ocd : valign ggml_metal_fusions
* CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a)
* Addressing reviewers comments, mma.cuh and physical warp size
* CUDA/HIP: single mma.cuh matrix-core lightning indexer ::CDNA::
* Support RDNA3
* CUDA/HIP: stream heads through the mma lightning indexer kernel
Process 16 heads at a time with double-buffered Q staging instead of keeping
all heads resident in shared memory, and drop the GGML_INDEXER_NO_MFMA env.
* enable for CUDA
* CUDA: Remove wmma lightning-indexer, also run head 32 through mma kernel
* CUDA: Nv mma dosent transpose accumulator
---------
Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* vulkan: add F16 input to the FWHT
The Vulkan FWHT accepts F32 input only. This adds an F16 source variant of the
shader, for both the subgroup and the shared-memory paths, so the kernel reads
an F16 source directly instead of requiring a converted copy.
The F16 shader needs shader-float16, so the pipelines are only created when the
device reports it. A null pipeline makes ggml_vk_can_use_fwht fall back, which
is the same path an unsupported width already takes.
supports_op and the dispatch both already route through ggml_vk_can_use_fwht,
so there is a single place asking whether an op qualifies; nothing to
consolidate here the way the CUDA/Metal ports needed.
test-backend-ops on an M5 Pro (MoltenVK, fp16: 1): MUL_MAT 1128/1128,
including all 24 Hadamard cases (18 existing F32, 6 new F16).
Assisted-by: Codex
* vulkan: FWHT kernels for block widths above 512
The Vulkan FWHT covers block widths 64 to 512; wider blocks fall back to the
dense matmul. Extend the width list to 8192 and pick the shader per width: the
subgroup shader keeps n/subgroup_size values per invocation, so widths above
512 (or a subgroup narrow enough to exceed 64 values per invocation) use the
shared-memory shader with one row per workgroup and a 256-wide block. The rows
per workgroup become a specialization constant so the dispatch matches.
Pipelines for a width are only created when the device's shared-memory and
workgroup-invocation limits allow it; otherwise that width keeps the fallback.
This is the Vulkan counterpart to #29100 (CUDA). Stacked on #29101; the last
commit is the one to review.
Assisted-by: Codex
* checkpoints: reduce overhead
Assisted-by: Codex
* rebase
Adapt the slot save/restore checkpoint appendix from #26004 to the
shared_ptr checkpoint list and the common_state_data buffers.
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: cwriter <cwriter@localhost>
* ggml : support `truncate=false` for YaRN
YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.
Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.
* support CUDA backend
* add tests
* move RoPE helpers to internal header
* format
* conversion : resolve quantized_layers from source tensor names
This commit changes how the per layer NVFP4 quant_algo are resolved from
hf_quant_config.json quantized_layers.
The keys in quantized_layers are module paths in the source checkpoint,
and these can be either be modules like 'lm_head' or parent modules
like 'layers.0.ffn.experts'. Currently these keys are passed through
filter_tensors, but they are not tensor names. So a filter_tensors that
keeps an explicit list of tensor names, like Nemotron's with --mtp
would drop them. That means there was no `_nvfp4_layer_algo entry`, and
`_tag_prec_a4` fell back to `_nvfp4_global_algo`.
This commit records the mapping from source tensor name to filtered
tensor name in index_tensors. Then in prepare_tensors each source
tensor is resolved to its closest quantized_layers parent, and
the algo is stored under the filtered tensor name, which is what
_tag_prec_a4 later looks up.
Converting Nemotron 3.5 Lightning NVFP4 with --mtp now produces:
```console
general.tensor_extra.name = ['output.weight']
general.tensor_extra.prec_a4 = [False]
```
whereas on master no general.tensor_extra fields are written. For the
non-MTP conversion, the general.tensor_extra fields are identical to
master.
Also manually verified (using pdb) that this works for
DeepSeek-V4.1-Flash-1017-NVFP4.
Refs: https://github.com/ggml-org/llama.cpp/pull/31602#issuecomment-6603337403
* conversion : shorten comment
* conversion : also remove '.bias' suffixes
Prevent malformed realtime session updates from throwing an uncaught
exception and terminating the server
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* fit : account for skipped NextN layers when setting n_gpu_layers
Without MTP, fit counts n_layer + 1 layers while the model loader counts
n_layer_all + 1 for n_gpu_layers, including the NextN layers which hold no
weights. A fit result that offloads all layers then leaves layer 0 on the CPU
("offloaded 65/66 layers" for Qwen3.8-27B on two GPUs).
Add the skipped NextN layers to n_gpu_layers and to the tensor split of the
device that holds the output layer so the loader assigns the layers that
fit measured.
Fixes#31531
* fit : pass hp_ngl_skip by reference like the other hparams
* fit : add hp_ngl_skip to common_get_device_memory_data
* graph: store the conv state of every rollback arch in one set_rows
The rollback slots of the causal conv history were written with one
copy per slot in qwen4exp, glm5-next, kimi-k3, bailingmoe3, lfm and
mamba2, while delta_net already wrote them with a single set_rows.
build_conv_state_store now does this for all of them: one copy without
rollback, otherwise one set_rows of the overlapping windows into the
min(n, n_rs_seq + 1) slots a rollback can read.
The slot order {n_rs_seq, ..., 0} is computed once per graph and each
conv layer takes a view of its tail. Its size does not depend on the
ubatch, so the reserved graph covers every ubatch shape, and the graph
input that held it is gone.
A model that is not causal never rolls back, so the context clamps
n_rs_seq to 0 for it, like for an arch without rollback support.
* nits: compute the conv slot order in the graph and keep the n_head name of bailingmoe3
* graph: build the conv slot order in each conv layer
The slot order {K-1, ..., 0} of the conv state store was one tensor
shared by every conv layer of the graph, built on the device of the
first one, so each layer on another device copied the view of its
tail. Each conv layer now builds its own order next to its set_rows,
and llm_graph_input_rs no longer holds it.
* tests: fix CI flakes of TOPK_MOE caused by near ties
With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.
Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.
* tests: build the TOPK_MOE logits by inverting the gating function
Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.
The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
The first command buffer takes the first 10% of the graph nodes and the
rest is split evenly between the extra command buffers. A fusion cannot
span two command buffers, so a boundary that falls inside a fusable
group runs that group unfused, and the output then depends on the node
count of the graph.
Move each boundary to the end of the fusable group it falls into, so
the encoders fuse the same nodes as with a single command buffer.
This commit adds nvfp4 scales to qwen2's build_ffn call which are
currently passed as NULL.
The motivation for this is that a model converted to NVFP4 will
currently crash because the scales are loaded but never applied in
the FFN:
```console
llama-completion: llama.cpp/src/llama-sampler.cpp:1218:
void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*):
Assertion `found' failed.
```
The attention already applies its scales but ffn is missing them.
* opencl: use the small-N f32 GEMV by default on X2E
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: use the q5_K multi-column verify GEMV by default on X2E
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* cuda: fuse NVFP4 gate/up and GLU in MMQ
* cuda: honor the requested src1 precision in fused gate/up MMQ
* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels
* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming
* cuda: check gate_scale before loading the fused NVFP4 gate scale
* llama : fix on-device state copy sizes for quantized KV caches
The on-device sequence state copies (LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) computed
their element count as size / ggml_element_size(). For a quantized type such as
q8_0 or q4_0 that is the number of blocks, not elements, so only part of each
range was copied: the restored state was wrong, or the copy crashed.
Compute the count as size / ggml_type_size() * ggml_blck_size() for the save,
the restore and the copy between differently chunked ranges, and assert that
each range covers whole blocks.
* llama : drop the one-line comments
* llama : drop the redundant cast in the state copy size
* server : realtime: send an error event instead of crashing on exceptions
* server : route websocket handlers through ex_wrapper, remove the realtime test
* server : realtime: keep the previous tools when the chat template rejects new ones
* server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapper
* server : realtime: clarify the tool schema check comment
* opencl: use more workgroups for q4_K gate up + GLU
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: handle FA for GQA 2 and 6 with head size 256
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: handle FA for GQA 16 with head size of 128, 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: handle FA for GQA 2 with head size 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* ui : share directory resolution and the cwd row
Move the lookup of a browser-picked folder to ToolsService.resolveDirectoryPath, let the working directory picker run without its chip, and split the 'Set working directory to' row into its own component, so other views can pick and show a folder the same way.
Assisted by: Claude (Opus 5.5 Medium)
* ui : open an item's actions on right-click
Sidebar conversations and models manager rows open the same actions as
their "more actions" menu on right-click. Both menus render one shared
list of actions, so they cannot drift apart. Phones keep the browser's
long-press; the dropdown's drawer already holds the actions there.
An action can render a flyout submenu in place of its row, through the
kit of the menu it is in, and close the menu after a pick.
Adds the shadcn-svelte context-menu component, mirroring the
dropdown-menu one.
Assisted by: Claude (Opus 5.5 Medium)
* ui : close conversation tabs from a right-click menu
A tab's right-click menu closes it, the other tabs or the tabs to its
right, and stops a response the tab is generating. Closing tabs that
include the active one keeps the tab the menu was opened on.
Assisted by: Claude (Opus 5.5 Medium)
* ui : open a message's actions on right-click
User and assistant messages open the same actions as their action icons
on right-click. Selected text, links, media, code and fields keep the
browser's own menu, so copying them works as before. The fork dialog
opens from the actions context, like the delete one, so both the icon
and the menu can open it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : scope right-click menu shortcuts and selection to their item
Show the conversation shortcuts only on the active row, as they act on
the active conversation, and keep the browser menu only for a selection
inside the message. Name the close-tab shortcut, and drop the unused
chip toggle, menu submenus and shared cwd row.
Assisted-by: Claude (Opus 5.5 Medium)
* fix(ui): show extracted text in PDF attachment preview
Assisted-by: Claude (Opus 5.5 Medium)
* feat(ui): show page thumbnails for PDF attachments sent as images
Assisted-by: Claude (Opus 5.5 Medium)
* ui: stop retrying PDF preview conversion after it fails
A failed PDF-to-image conversion or text extraction reset its loading
flag, which re-ran the effect and retried forever. Remember the failed
item for each view and only retry when the item changes, so the error
state stays on screen.
Add a PdfConversionError story that feeds a broken PDF.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the text of PDFs sent as images and tidy the PDF preview
Store the extracted text of a PDF sent as images in its content, so the
Text view reads it instead of parsing the PDF again; only messages stored
with the old placeholder extract it. The model still receives the images
only. Guard retries on the error state, as the item remounts per id, skip
an empty base64 payload, and share the status markup between views.
Assisted-by: Claude (Opus 5.5 Medium)
* gguf-py : validate every tokenizer merge pair, not only the first
The transformers 4.45 pair format was detected by checking merges[0],
then every element was converted unchecked: a non-pair element raised
a bare TypeError, and pairs with the wrong arity or non-string parts
were silently mangled into invalid merges.
Validate each element (2-string list) up front and raise the existing
"Unknown tokenizer merges format" error for anything else.
Assisted-by: Devin
* gguf-py : remove test_vocab.py per review
The RISC-V vector intrinsic specification requires __riscv_v_intrinsic
to be defined when the compiler supports the API, even when the vector
extension is not selected. Clang therefore defines it for rv64gc,
causing ggml to enter RVV branches where vector types are rejected
because the required ISA extensions are absent.
Use __riscv_v for optional RVV implementations; this macro represents
the selected target ISA. Retain __riscv_v_intrinsic checks in SpaceMIT
sources, where they validate mandatory compiler API support rather than
select optional vector code.
Closes: https://github.com/ggml-org/ggml/issues/1535
* ggml: bound views by the bytes their strides reach
A view was checked against its source with the size of a contiguous
tensor of the same shape, computed before its strides were set. Views
whose rows overlap failed that check although they stay inside their
source, and views with wider strides passed it while reaching past
the end. The view functions now pass their strides to the tensor
constructor, which checks the bytes they actually reach, like
ggml_nbytes. The conv windows of delta_net are created with all their
rows at once instead of being widened after the check.
* ggml: assert that a strided view starts from contiguous elements
The view functions set nb[0] to the type size, so the source must have
contiguous elements along dim 0 for the view to read them.
The gguf writer staged each tensor into a temporary vector before
writing it. For host-buffer tensors the data pointer is directly
writable, so skip the staging copy and write it straight to the file.
This avoids the per-tensor allocation, zero-init and memcpy.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* common : fix -hf tag matching inside longer quant names
":F16" picked a BF16 file when the repo had both, since "BF16"
contains "F16". The same applied to ":Q4_XS" vs "IQ4_XS". The tag
now has to start at a token boundary in the path.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* common : accept underscore as -hf tag boundary
Some repos separate the quant with an underscore, e.g.
stepfun-ai/Step-3.5-Flash-GGUF-Q4_K_S ships
step3p5_flash_Q4_K_S-00001-of-00012.gguf, which :Q4_K_S no longer
matched. Add _ to the token boundary and pin the tag and default
lookup cases.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* hexagon: initial support for LIGHTNING_INDEXER op
* hex-lght-idxr: hmx for prefill
* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute
* hex-lght-idxr: pipelined hmx path for decode and prefill
* hex-lid: add support for K (f32)
* hex-lid: add support for K (f32)
* hex-lid: add support for q4_0
* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills
* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel
* hex-lit: update dma pipeline to use canonical dbl buffering in all paths
* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths
* hex-lit: add support for dims collapse
* hex-lit: make kparams constistent with other ops
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* Optimizing Gemma4 graph for prompt processing
TODO:
- Currently complete microbatch needs to contain tokens which does not require the logits; need to optimize this further
- There are a few more operations that are performed after the kv cache is calculated, need to safely optimize them.
* Not filtering the rows when they are unmasked
* Cropping out the attention mask too.
* gemma-4: improved pruning, mask cropping and PLE scaling
Prune before the first KV-shared layer with the single-stream, causal, non-embedding safety gates.
Use cropped masks only during shared-layer attention.
Preserve requested layer-input embeddings in the KV prefix.
Remove the PLE contiguous materialization and fold equivalent PLE scaling into RMSNorm parameters/weights.
* spec-mtp: Enable masked h_nextn extraction only when the draft context shares the target KV cache
This lets Gemma 4 prune unused prompt rows before its shared layers so that we keep the full perf P.P gains with MTP.
Avoid extracting target hidden states for output-free prompt chunks in that shared-MTP path.
When a batch has a selected final token, extract only that row unless the whole verification batch is selected.
Keep ordinary MTP and classic speculative decoding behavior unchanged: they still use unmasked h_nextn because the draft
context must catch up on all target rows.
speculative-simple needs an update to select the final prompt row only for shared-KV MTP,
so the optimization works there without changing other speculative modes.
And it looks like the server already selects its final prompt row for first-token sampling.
* spec-mtp: make sure to not drop first draft round
* gemma4: enable pruning based on static cparams.kv_unified/n_seq_max=1
* gemma4-mtp: simplify prunning logic and avoid graph branches/re-allocs
* gemma4: update pruning gates to use only n_seq_max==1, kv_unified can now be used for multi-seq
* spec-simple: revert spec-simple changes for now
* gemma4: remove causal_attn from pruning gate (last source of reallocs)
* cont : adapt to llm_attn_mask changes
* cont : add TODO
---------
Co-authored-by: Meet Amin <meetamin378@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* hexagon: faster TOP_K for the backend sampler (skip tiles, then block scan)
Skip the bitonic sort for tiles whose max cannot reach the running k-th
best value, then replace the tile sort with top_k_scan(): reject 256-float
blocks with one HVX max-reduce and insert only surviving elements into a
sorted k-entry list. Used for k in 2..64 and rows wider than 1024, in both
the single-row and multi-row paths.
Ties keep the first-seen index. NaN is treated as -INF.
* hex-top-k: eliminate scalar reads from vtcm
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Playwright's bundled Chromium lags behind Chrome security releases
(1.56.1 ships 141, and even 1.64.1 ships 156.0.8078.4, before the
156.0.8078.73 fixes), and Playwright launches it with --no-sandbox by
default. Outside CI, use the installed, auto-updated Chrome with the
sandbox enabled. CI keeps the bundled Chromium, so the self-hosted
runner's preinstalled 1.56.1 browsers still work.
Assisted-by: Claude (Opus 5.5 Medium)
* hexagon: add support for DSV4_HC_PRE
* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization
* hex-dsv4-hc-pre: double-buffered DMA for prefill
* hex-dsv4-hc-pre: refactor DMA block setup
* hex-dsv4-hc-pre: precompute kernel params
* hex-dsv4-hc-pre: support for n_hc != 4
* hexagon: add support for DSV4_HC_POST
* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block
* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks
* hex-run: add support for test-llama-archs tool
* hex-dsv4: refactoring and optimizations
Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.
Tested with test-llama-archs -a qwen4ext and test-backend-ops
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* speculative : save and restore MTP hidden state
Assisted-by: Codex
* tests : remove MTP state test
Assisted-by: Codex
* speculative : simplify MTP checkpoint state
Save only the boundary hidden row; verification rows are rebuilt on the supported restore paths.
Assisted-by: GPT-6
---------
Co-authored-by: zouyongxing <zouyongxing@bytedance.com>
* mtmd : add Nemotron 3.5 Transcribe support
Assisted-by: Claude
* convert : fix vocab of draft models for Nemotron-H targets
Assisted-by: Claude
* convert : reject non-BF16 Transcribe checkpoints
Assisted-by: Claude
* mtmd : make speaker FIFO predictions local
Assisted-by: Claude
* mtmd : normalize the mel in encode_windows
Assisted-by: Claude
* convert : map the generated speaker kernel
Assisted-by: Claude
* mtmd : validate Nemotron Transcribe mmproj shapes
Assisted-by: Claude
build_conv_state_at makes a contiguous copy of the conv input tail
before writing it into the recurrent state, so every GDN layer issues
two device copies per generated token where one is enough. On
Qwen3.8-Flash-Next (36 GDN layers) that is 72 copies per token, now 36.
Speculative decoding adds one more per rollback slot. ggml_cpy reads
the strided view directly, as the shared build_conv_state already does,
so the extra copy is dropped.
* ui: add Playwright harness for recording demo videos
Scripted walkthroughs under tools/ui/demo are recorded with Playwright and
exported as H.264 mp4 for pull requests (demo/pr) and docs (demo/docs).
Fixture MCP servers and seeded settings keep recordings deterministic, a
drawn cursor and captions make the flow readable, and DEMO_UI_DIR points
the recording at another checkout such as a PR worktree.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: record demo scripts from PR descriptions and a configurable storage
Demo scripts no longer live in the repo. `npm run demo:pr -- <number>`
records the `ts demo` block from a PR description against the PR head,
uploads the mp4, a GIF preview and the script to DEMO_STORAGE, and links
the preview in a `## Demo` section of the PR description.
`npm run demo:docs` records the docs scripts kept in DEMO_STORAGE.
DEMO_STORAGE (hf:// buckets or file://) and the public URL are read
from tools/ui/.env.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: import generated conversations into demo recordings
Demos can now start with existing chats: `test.use({ traces })` imports
llama.app JSONL traces through Settings > Import/Export before the video
starts. Built-in traces are generated deterministically, other names are
downloaded from DEMO_TRACES_DATASET, and `demo:traces` pushes the built-in
ones there. Adds `demo.scroll` for wheel scrolling.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep demo captions clear of the chat form and isolate concurrent runs
Captions move to the top of the frame, since at the bottom they covered
the chat form a demo is often about. Each recording run gets its own
Playwright output folder, so a run starting elsewhere no longer clears
another run's video before it is saved.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: record demos at 2x and 60 fps with human-like timing
Recording goes through Chromium's screencast instead of Playwright's
video, which encoded VP8 at about 1 Mbit/s at 1x. Frames are captured at
the device pixel ratio (DEMO_SCALE, default 2) and encoded into a 60 fps
H.264 mp4, starting once the page has settled.
The cursor now glides faster along a slight arc with Fitts-like
durations, settles briefly before each click, and typing and pauses vary
a little. The randomness is seeded per demo, so re-recording gives the
same motion.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: document the demo harness prerequisites
The demo README now lists the tools the harness needs and what each is
for, and walks through creating the Hugging Face bucket and the optional
traces dataset, with the matching .env settings.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: version linked demo URLs so re-recorded previews refresh
GitHub's image proxy caches a preview by URL, so a demo re-recorded at
the same PR head kept showing the old GIF. The Demo section now links
the files with a content hash in the query string.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: label PRs that feature a video demo
Adds a `video demo` label and a workflow that applies it when a PR
description has a demo written by the UI demo pipeline (the managed
demo:start/demo:end markers, or a preview linking to a demo video),
and removes it when that section is gone.
Assisted-by: Claude (Opus 5.5 Medium)
* ci: label video demos with gh instead of github-script
The label workflow now runs the gh CLI in a shell step, like
pr-draft-label.yml, so it needs no action outside the repository's
allowed list. The video link check only counts videos uploaded for the
PR itself, so a description linking other PRs' demos isn't labeled.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: zoom demos in on details with a two-finger pinch
demo.zoom pinches in on a control or a point, holds, and pinches back out
to 1x, so a detail too small to read in the frame can be shown up close.
It drives Chromium's own pinch gesture, so the page's fixed bars stay put,
and works at desktop and phone sizes alike.
The pinch is evened out where Chromium misses: one anchored near the top
does not zoom at all, so it starts lower, and one near the bottom
overshoots its ratio, so a small pinch from the middle corrects it. The
way out reads the real scale until the page is back at 1x, which the next
gesture needs.
The overlay also draws a fingertip per touch, so a pinch on a touch
device shows as two fingers.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: wait out a cold dev server before a demo starts
A fresh checkout's dev server optimizes its dependencies on the first
load and reloads the page, which can take well past an action's timeout,
so a demo failed before it started. The demo now waits up to 90 seconds
for the app; the recording only starts once it is ready.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: record a video per test in a PR demo, shown in a table
A large PR can split its demo into several shorter videos: each test()
in the ts demo block is recorded as its own video (<sha>-1.mp4, ...),
and the Demo section lays their previews out in a table, each column
headed by its test's title.
GIF previews now keep their longer side at DEMO_PREVIEW_WIDTH and shrink
until they are under 5 MB, since GitHub's image proxy refuses anything
larger and shows a broken image instead.
Importing traces works on a phone viewport too, where Settings sits in
the sidebar under the page until the menu key slides it aside.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fix demo trace imports on desktop and lay landscape demos out two to a row
Importing traces failed on a desktop whose collapsed sidebar sits under
the page: it now expands the sidebar from the page's menu key on a phone
or from the sidebar's logo on a desktop, whichever makes Settings
reachable. Afterwards it forgets the expanded sidebar, so a demo starts
with it collapsed, as on a first visit.
A PR demo with several videos shows up to four portrait (phone) videos
to a row, or two landscape ones, so each preview stays readable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: only record other people's PR demos at a reviewed head commit
demo:pr runs the PR's code and demo script on this machine, with the
gh and hf logins, so a PR by someone else now needs --trust <sha>
naming the head commit that was read, and is refused again after new
pushes. Recordings also launch Chromium with its sandbox on, which
Playwright turns off by default.
Assisted-by: Claude (Opus 5.5 Medium)
ggml_roll validated each shift with abs(shift), but abs(INT_MIN) is signed
integer overflow UB evaluated before the bounds check runs. Compare with a
signed-safe test instead.
This commit updates the _flush_nvfp4_experts to take a new parameter
which is a dictionary mapping the key to the original model tensor name.
The motivation for this is that I noticed that the tensor names being
looked up were not the original model tensor names, for example:
```console
model.layers.1.mlp.experts.up_proj.weight
```
But _nvfp4_layer_algo uses the original per-expert names:
```console
(Pdb) p self._nvfp4_layer_algo
{
'backbone.layers.1.mixer.experts.0.up_proj': 'W4A16_NVFP4',
...
```
This means that the lookup will never match for experts and fall back to
_nvfp4_global_algo. If the global algo is W4A16_NVFP4 then this would
go unnoticed, but for a model like
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 which is
MIXED_PRECISION, the expert tensors will not get prec_a4=false so they
would use the native W4A4 path on Blackwell even though the original
model tensors declare them as W4A16_NVFP4.
* cuda: gather batched tensor reads into mapped host memory
Assisted-by: Codex
* add comments
Assisted-by: Codex
* simplify the implementation based on checks that exist earlier in the flow
* simplify batched tensor gather indexing
* add conditional dispatch
* address comments - make host registration raii, better test mindful of ci time, assert instead of return false
* cuda : gather small reads and use pinned DMA for large reads
* cuda : use std::vector for host gather descriptors
* cuda : add 8-byte gather copies, generalize tensor test and log registration failures
* common buffer for descriptors
* simplify stream sync + increase to 128k descriptor pinned buf
* Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available.
* Drop ggml_backend_comm_allreduce_rank_t and reuse ggml_backend_comm_allreduce_tensor for per-rank communicators
* reuse ggml_backend_cuda_comm_allreduce_nccl for per-rank communicators. Remove the helper function backend_comm_proc_address
* Change ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t to more generic multi-process backend communicator interfaces
* set maxCTAs in ncclConfig_t to 8 for dgx spark in ggml_backend_cuda_comm_init_rank
* meta: support GLM-5.3 next sm tensor
* glm5-next : cont q_absorbed for -sm tensor with non-unified KV
* meta : keep GGML_OP_ARANGE in ggml.h order
* glm5-next : split MLA heads in pairs for -sm tensor
The MLA cache has a single KV head, so the heads on a device are its GQA
ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3
devices the 16 heads of the test model were split 5/5/6 and it aborted.
* meta : keep the split for views inside one slab of the split axis
With a non-unified KV cache, build_attn_mha splits q into streams with a
view. For a permuted q, e.g. the absorbed query of glm5-next, that view
aborted with "view of permuted tensor not implemented". The split is now
kept if every other dim of the source and the view stays inside one slab
of the split axis. The per-device strides are only rescaled above it.
From tetherto/qvac-fabric-llm.cpp#353.
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
* glm5-next : drop the q_absorbed cont
The meta backend can now split the per-stream view of the permuted q, so
the cont is not needed for -sm tensor with a non-unified KV cache. With
the cont, the q of flash attention was not contiguous, and when FA ran on
the CPU (WebGPU) the meta get_tensor aborted.
---------
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
kernel_cpy_t_t indexed the dst innermost dim with an implicit element-size
stride instead of args.nb0, so cpy into a permuted view (nb0 != type size)
wrote the wrong addresses. caught by the new test-backend-ops cpy case with
permute_dst = {1, 0, 2, 3} from #31446. addressing is unchanged when dst is
contiguous in dim 0.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* vulkan: skip maxStorageBufferRange pre-check for non-compute tensors
Pre-allocated leaf tensors and views (GGML_OP_NONE / GGML_OP_VIEW) are never
executed by the backend, but were still rejected by the descriptor-size
pre-check in ggml_backend_vk_device_supports_op when larger than
maxStorageBufferRange on devices without shader_64b_indexing. The scheduler
then aborts with 'pre-allocated tensor (...) in a buffer (Vulkan0) that cannot
run the operation (NONE)' - e.g. the MoE expert cache banks (llama-moe-cache)
of a 256-expert model with --moe-cache-mib on GCN GPUs, where a whole bank
(2.4 GB) exceeds the limit while the consuming ops only access small
per-step views.
Skip the size pre-check for non-compute ops (NONE and VIEW); the buffer type
allocation limit still bounds the actual allocation, and compute ops keep
validating their sources.
Tested on AMD Radeon Pro VII (gfx906, no shader_64b_indexing):
- before: GGML_ABORT 'pre-allocated tensor (moe_cache.0.blk.0.ffn_down_exps.weight) ... cannot run the operation (NONE)' with llama-cli -ngl 99 --cpu-moe --moe-cache-mib 6144 (Cyber-Tiel-Coder-35B-A3B, qwen35moe, 256 experts per layer)
- after: loads and generates at 8.1 t/s (vs 0.6 t/s without the cache)
- regression: -ncmoe 16 path unchanged (19.2 t/s), qwen3moe/qwen35moe without cache unchanged
Fixes#30697
* vulkan: use ggml_op_is_empty for the non-compute size-check skip
---------
Co-authored-by: FamousMonsterr <FamousMonsterr@users.noreply.github.com>
* delta_net: access conv state via views
* fix sm tensor
* delta_net: fill the conv slot index on the host
The rollback slot index of the conv windows is a constant per graph
shape. It is now an I32 input of the recurrent memory filled on the
host, instead of an arange and an argsort computed on the device, and
the meta backend no longer needs a mirrored split for ARANGE.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup
On the non-OCP path, ue4m3_fp32_lut has 128 entries and is indexed with the raw scale byte. ggml_validate_row_data accepts every byte and ggml_ue4m3_to_fp32 ignores bit 7, so a byte >= 0x80 reads past the shared array. Vulkan robustness does not cover shared memory.
Size the table, both init loops, and the two host shared-memory budgets to 256 entries. ue4m3_to_fp32_build() already masks the exponent and treats only 0 and 0x7F as special, so entries 128-255 match the CPU. A 256-entry table keeps 0xFF distinct from 0x7F: the CPU decodes 0xFF as a non-zero value and 0x7F as zero.
Each NVFP4 pipeline uses 512 more bytes of shared memory. ggml_fp32_to_ue4m3 never sets bit 7, so models quantized by llama.cpp are unaffected.
Assisted-by: Claude
Assisted-by: Codex
* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup
Assisted-by: Codex
---------
Co-authored-by: Dongsheng Luo <7137760+flyingdoog@users.noreply.github.com>
The dspark block mask is now derived at input-fill time: llm_attn_mask::block_uniform
fills every token of a sequence with the mask row of its first token, so the
graph-level reshape/repeat in dflash is gone and cnv always aliases raw.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cuda : prevent premature eviction of graphs about to be used
* add last_graph_use_time for entire cuda graph map and testcase for cuda graph
Assisted-by: Codex
* delete the test
A cache that shares the cells of another one (f.ex. the gemma4 assistant) can build a graph before its sequences have any cells, f.ex. at the server warmup. get_n_kv_rows() then returned 0, which means that kv_rows are not used, so the graph fell back to the dense KQ mask and the scheduler re-planned for it. Later graphs with larger masks then aborted with GGML_SCHED_DEBUG_REALLOC=1.
* hexagon: add tiled HVX transpose path for permuted CONT/CPY
Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.
Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.
Host:
- drop extent-1 dims and merge dims that stay adjacent in src into a
canonical view, stored in the kernel params. There are no minimum
size limits.
- the copy is treated as a 2D transpose between dst dim 0 and the
contiguous src dim, batched over the remaining dims.
- tiles are sized to 64 KB per buffer and halved until they fit in
session VTCM. blkk is capped so the halfword vgather offsets fit in
16 bits.
Kernel, per tile:
- 2D DMA of blk0 strided src rows (blkk contiguous elements each)
into VTCM
- vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
- 2D DMA of blkk contiguous rows out to dst
Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.
Small dims:
- packed mode: when dim k follows dim 0 and dim 0 divides the HVX
lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
whole dst rows, and each tile goes out as one contiguous DMA.
- when the contiguous src dim fits in one vector, it is staged exactly
and the tile is made long along dim 0.
All other copies select the same kernels as before.
Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.
* hexagon: cpy transpose cleanups
- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)
Assisted-by: Claude Opus 5.5
* hexagon: use cpy transpose path for ne[0] == 1 permutes
- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests
Assisted-by: Claude Opus 5.5
* hex-cpy: further improve cpy
Bring in improvements from aizip branch/PR.
Cleanup kparams and replace divs with fastdiv.
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* hex-cpy: avoid vtcm bank conflicts
* hex-cpy: trim down the comments
* cpy-test: trim down new tests that cover permute scenarios
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* feat: support PocketTTS voices with no voice cloning
The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.
Assisted-by: Claude Fable 5.1
* convert : pocket-tts voices never fail the conversion, log the drift of the state
The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.
Assisted-by: Claude Fable 5.1
* mtmd : list the predefined voices, expose them in GET /props
mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.
Assisted-by: Claude Fable 5.1
* server : an unknown voice is a bad request
A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.
Assisted-by: Claude Fable 5.1
* refactor api
* fix conversion
* small renaming
* fix lint
* tts : a model with predefined voices speaks with the first one by default
Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.
Assisted-by: Claude Fable 5.1
* mtmd : one comment per voice accessor, converter states the rule only
The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.
Assisted-by: Claude Fable 5.1
* clean up redundant code and comments
* mtmd: match the LFM3-Audio voice name case-insensitively
The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* llama : use llm_attn_mask and kv_rows for unified attention
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont : optimize `get_seq_cells`
* llama : build the kv_rows lists from per-sequence cell bitmaps
Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.
Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).
* cont : make get_kv_rows_p_min private
* cont : minor metal
* llama : use kv_rows for every ubatch of a unified cache
A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.
Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.
* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The note under 'Run the inference' still said that the CANN backend only
supports FP16/Q4_0/Q8_0 models, while the DataType table in the same file,
and ggml-cann.cpp (BF16 is enabled for MUL_MAT, GET_ROWS, CPY and CONT for
every non-310P device), say otherwise. Spell out the per-device support and
mark BF16 as unsupported on 310P instead of leaving the cell empty.
GGML_CANN_NO_PINNED is read by ggml-cann.cpp when allocating host buffers
and when advertising the host_buffer device capability, but it was not
documented, so the escape hatch for a failing pinned host allocation was
not discoverable.
Signed-off-by: li-lizhe <147392333@qq.com>
* vulkan: add tensor-parallel all-reduce communication
* vulkan: require shared semaphores for tensor communication
Assisted-by: Codex
* vulkan: reduce directly from shared host buffers
Assisted-by: Codex
* vulkan: fix ring all-reduce races with 3+ devices
Two hazards in the tensor-parallel ring all-reduce made -sm tensor output
non-deterministic with three GPUs:
- With n = 3, step t's host-slot region starts at t * cels * 2 bytes, which
is not a multiple of 32, so consecutive steps' transfer copies share a
32-byte sector of the imported host memory. On NVIDIA the later copy's
leading bytes (up to 10 F16 values at the start of every chunk) end up
stale in host memory, even when the copies are ordered by a semaphore.
Give every step its own 256-byte-aligned region. n = 2 and n = 4 already
had aligned regions for the usual sizes, so their layout is unchanged.
- A single-shot all-reduce leaves every device's host slot being read by all
peers, but the next ring round only ordered its first transfer after nextd,
and only from round 2 on. With n >= 3 a device could overwrite its slot
while prevd was still summing it. Wait for all peers on the first ring
round after a single-shot all-reduce.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vulkan: fold the tensor-parallel all-reduce into the compute submissions
With -sm tensor the meta backend drives every device from one thread and calls
graph_compute once per subgraph (two per layer), so decode was bound by the CPU
cost of submitting, not by the GPUs: ~1500 vkQueueSubmit per token for a 64-layer
model on two GPUs, with the GPUs idle ~40% of the time.
- Don't split subgraphs by flops while a communicator is active. The heuristic
was tuned for whole graphs and turned each small subgraph into ~5 submits.
- Fold the small (single-shot) all-reduce into the compute submissions: the
last command buffer of a subgraph stays open, the all-reduce appends the
gather copy to it and submits it, then opens the next command buffer with the
peer waits and the reduce, which the next subgraph is recorded into. Each
subgraph now costs one submit per device. The reserved progress value is
attached by whichever path submits the open context.
- Alternate between two host slots, placed outside the ring's regions, so the
gather no longer needs a WAR wait on the peers' previous reduce.
- On synchronize, submit the deferred work of every device in the group first,
so the tail of the graph runs on all devices in parallel.
- Wait for the transfer queue as well as the compute queue before freeing or
reallocating communication buffers, and don't leave a stale cap behind when a
reallocation fails.
qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 32.2 -> 49.7 t/s, 3 GPUs 20.9 ->
40.0, 4 GPUs 15.0 -> 27.6; pp512 on 2 GPUs 2138 -> 2320. Greedy output is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* - Two-phase all-reduce over shared host memory: write own slot + signal, wait peers + add from their slots
- One exported timeline per device, no backend hooks; comm work recorded in the backend compute ctx
- F16 wire above 1 MiB, F32 below; small reductions ride with the next graph submit
- Double-buffered, grow-only host slots; zero-size splits zeroed
- Remove ring, transfer-queue leg, device staging, shrink, Windows shims
- comm_init returns nullptr when unsupported; add GGML_VK_COMM_DISABLE
- No flops-based submit splitting for graphs below 1 GFLOP
[rebased onto the ring all-reduce fix and the folded all-reduce, which this design replaces; their backend hooks are removed here]
* add minimal submit split size
* replace opaque_fd with sync_fd for spec compliance
* add precision check
* add semaphore pool
* rename env var, add logging if requirements not met
* reduce comments
* code improvements
* vulkan: use posix_memalign for the tensor-parallel host buffers
std::aligned_alloc is unavailable on Android bionic below API 28, which
upstream switched away from in #30828; do the same for the communicator's
host slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vulkan: synchronize the tensor-parallel all-reduce through the host
Exchanging SYNC_FD semaphores costs a kernel wake-up on every all-reduce:
on NVIDIA the waiting GPU starts ~35 us after its peer has finished, which
is most of the idle time in a decode step. OPAQUE_FD semaphores avoid that
but must not be imported into another physical device.
Use only device-local timeline semaphores instead. Each device signals
done[i] once its partial sum is in host memory (after a barrier that makes
it available to the host), and a bridge thread signals go[i] from the host
once all peers of i are done. The waiting side makes the host memory
visible with a host-write barrier, since its submit predates the data.
The thread spins for up to 200 us on each wait (GGML_VK_ALLREDUCE_SPIN_US)
and then blocks, so it is idle when there is no work. This also takes the
per-round fd export, dup and import calls off the main thread and drops the
VK_KHR_external_semaphore_fd requirement and the binary semaphore pool.
Graphs below 1 GFLOP, such as tensor-parallel decode slices, are no longer
split into several submits.
qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 37.2 -> 45.4 t/s, 3 GPUs
24.1 -> 35.5, 4 GPUs 16.5 -> 27.1; pp512 unchanged. Greedy output is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* use sync_fd on mesa drivers
* refactor for readability
* move sync_fd semaphore handling to thread, trigger mesa submit threads
* move allreduce functions to new cpp file
* clean comments
* smaller fixes, code improvements
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* Update ggml-sycl.cpp
Increases -sm tensor prompt eval t/s by +19% with zero PPL change.
* Update ggml-sycl.cpp
Replace first pair of wait() calls with ext_oneapi_submit_barrier(). The wait() calls was discovered to cause MTP degradation.
* Update ggml-sycl.cpp
Adding comm_exchange function pointer to switch between 2 and 4 buffer ggml_sycl_comm_exchange versions @ initialization time (versus hot path) controlled by SYCL_ENABLE_DEFAULT_CONTEXTS=1 setting.
* Update ggml-sycl.cpp
adding back 2nd half of 4-buffer free.
* Update ggml-sycl.cpp
Removing extra whitespace.
* Update ggml-sycl.cpp
Excluding Windows until it is validated on Windows.
The router only removes models it took from the Hugging Face cache, but
the models manager offered "Delete from disk" for every model, so a model
from --models-preset or --models-dir failed with a "not removable" toast.
Read the source and can_remove fields of /models and show the delete
action disabled for those models, with a tooltip (a hint line in the
mobile drawer) saying how to remove them instead.
Assisted-by: Claude (Opus 5.5 Medium)
Use the popover background, softer border, rounded-xl corners and
shadow-md like dropdowns, hover cards and the settings dialog, and
match the dialog title line height.
Assisted-by: Claude (Opus 5.5 Medium)
A message queued while a conversation was busy was re-sent through
sendMessage, which targets the active conversation, so switching chats
before the turn ended delivered it into the wrong one. Re-send only when
the owning conversation is active; otherwise keep it queued there and
flush it when that conversation is opened again.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show interrupted state for tool calls that never returned
The chat-level streaming flag was applied to every assistant message, so a
tool call left without a result in an earlier message kept spinning while a
newer reply streamed. Scope streaming to the last assistant message and mark
finished calls without a result as interrupted.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: mark a shell command stopped before any output as interrupted
exec_shell_command creates its result message before running, so stopping
it mid-run leaves an empty result instead of none.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: don't reload a loading model on select, prefer it in pre-selection
Assisted-by: Claude (Opus 5.5 Medium)
* ui: skip hidden models in automatic model selection
Assisted-by: Claude (Opus 5.5 Medium)
* ui: don't count a model being unloaded as loading
Assisted-by: Claude (Opus 5.5 Medium)
* webgpu: keep scratch alloc sizes monotonic in the batch length
The flash attention vec path and the MMVQ path append a scratch area
after dst, but get_alloc_size only accounted for it when the current
batch takes that path. The size was therefore not monotonic in the
batch length: a graph reserved at n_ubatch (tiled path, no scratch)
or at one token did not cover batches of 2 to 19 query rows, and
GGML_SCHED_NO_REALLOC builds aborted on an unexpected reallocation.
Pass the row count to both path predicates and size the scratch with
the row count clamped to the longest batch the short path serves.
* nit: align the mmvq pipeline key with clang-format 22
* webgpu: address review @yomaytk, name the mat-vec row threshold
* hexagon : fix DMA transfers with strides > 24 bits
On v75+ the 2D DMA descriptor stores row_size and strides in 24 bits.
dma_queue_push sent oversized strides to fallback_2d, which only splits
nrows, so strides of 16 MB or more were truncated - wrong addresses.
Fix will fall back to one transfer per row in that case.
Also added two CONT tests with source strides over 16 MB.
* hex-dma: refactor the fallback paths to remove duplication
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Limit the shared prefix to the first required output position of the parent and each child task.
This keeps decision output embeddings available in the final batch when questions share option tokens.
Fixes#30482
Signed-off-by: aiqubits <aiqubits@hotmail.com>
* chat : accept the think block in the qwen3-coder response format with reasoning_format none
Assisted-by: Claude Opus 5.5
* chat : define both qwen3-coder think block variants up front
This commit extracts all apple jobs except for macos-latest-arm64 into a
new daily workflow.
The motivation for this is to reducd the load on the macos runners.
The duplicated output looks up token_embd in its buffer context, so
token_embd must exist first. With the old order a load where input and
output share a buffer type (CPU-only, -ngl 0) held the table twice:
gemma-4-E2B-it Q4_K_M, no mmap, CPU model buffer 3163.73 -> 2947.73 MiB.
* memory: refuse rollbacks past the snapshots of the last ubatch
The recurrent memory and the DSV4 cache accepted any rollback up to
n_rs_seq, whether or not the last ubatch wrote that snapshot. Each now
records the deepest rollback its last ubatch guarantees and seq_rm
refuses anything past it:
- recurrent: n - 1 for a ubatch of n <= n_rs_seq + 1 tokens, 0 for a
longer one, which only guarantees the final state
- DSV4: min(n, n_rs_seq), the snapshots are rebuilt from the state
before the ubatch
The dead rollback block of the rollback test becomes the assertion it
asked for, and rollback_bound checks the exact bound of each memory
after ubatches of 1, 2, n_rs_seq + 1 and n_rs_seq + 2 tokens.
* tests: address review from ggerganov
Detect architectures with llama_arch_from_string in comp_padding and
arch_reserves_single_seq, as in rollback_bound.
gfx117X is more like RDNA4, with 128b WMMA. But they're not identical. Adjust the defines accordingly to have a new RDNA4_M and use it where it makes sense.
Assisted-by: Codex
Assisted-by: GitHub Copilot
Co-authored-by: Mario Limonciello <mario.limonciello@amd.com>
* WIP OCP FP8 E4M3 support
* Fix python type-checker
* Materialize input_scales also for FP8 when converting
* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4
1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
weights are reshaped
* Allow remaining FP8 tensors to be dequantized during conversion
* Allow tensor-strategy for FP8-only compressed-tensor ckpts
* Fix and test gguf-py behavior for FP8_E4M3
* Disallow QKV-fusion for FP8 temporarily
* Fix conversion of fused qkvz & FP8 for Qwen3-Next
* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws
* Emit warnings when not fusing FP8 during conversion
* More FP8 conversion fixes
* Add supported for strided copies to/from FP8 in CPU
* remove unneeded scratch buffer from FP8 cpu cpy/dup path
* add explanatory comments and TODOs
* Add dedicated encoder and decoder tests for FP8
* Whitelist ggml_op_get_rows in metal instead of blacklisting
required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list
https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512
* Fix more conversion bugs
* Defer weight materialization during conversion
* Blacklist fp8 in metal backend instead of white-list
* Adjust indents
* Fix lint + execute tests
* Add fast CPU kernel for F8_E4M3
* Conversion fixes
* Fix remaining gaps
* Enable tied weights with shared scales
* Make mandatory-F32 rules apply to FP8 during conversion
* Fix python type checks
https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512
* Fix kaleidai unit-test failures
https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150
* Relax numeric precision constraints for pre-check
* Leave fast-FP8 CPU on RISC-V for follow-up
We already exclude arm from the fast-path
* Allow output scale tensors to be located in any gguf shard
* Move test_conversion_fp8.py test execution to behavior-check.yaml
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it
* Install all requirements also on Windows for behavior-check
Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512
* Unmap lazy tensors before clean-up in Windows
Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512
* LF line endings
https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512
* Extract FP8 tests from llama-arch to its own test-fixture
See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221
* Update gguf-py/gguf/scripts/gguf_convert_endian.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ui: land at the bottom of a conversation every time it opens
Opening a long conversation often stopped short of its end:
- the press grace counted from page load (pressedAt = 0), so a chat that
landed within 400 ms of a refresh did not follow the rows realizing
after the jump
- the click that opened a chat (sidebar row, tab) started the same grace,
so navigating by click had the same problem
- any scroll event away from the bottom stopped following, including the
ones the browser fires itself while the chat settles
- the chat screen outlives its routes, so going back to the last landed
conversation through the new-chat screen skipped the landing entirely
scrollToBottom() now clears the grace, only an upward scroll away from the
bottom stops following, and every navigation lands anew. Navigations into
a chat also disable SvelteKit's scroll reset, which moved the previous
conversation to its top for a few frames.
Where overflow-anchor is supported, a 1px pin after the messages is the
only scroll anchor while following, so the browser keeps the bottom in
place itself; WebKit keeps the resize observer.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let nested agentic content scrollers chain to the chat
Reasoning, shell output and code blocks used overscroll-behavior:
contain, so the wheel stopped dead at their edges instead of
scrolling the conversation.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: stop following the stream as soon as the reader scrolls up
A trackpad or smooth wheel scrolls in steps smaller than the at-bottom
threshold, so each step up was re-stuck and snapped back on the next
growth while a response streamed. Any upward move that is not content
shrinking now releases at once.
The attach effect also read the hook's own state, so every flip
re-attached and re-measured, which undid such a release.
Assisted-by: Claude (Opus 5.5 Medium)
ChatFormActions gated the mic and stop buttons on a `canSubmit` prop that
ChatForm never passed, so it was always false and typing text did not swap
them for the submit button. Use `canSend` instead and also hide the
skip-reasoning button while there is content to send.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: reveal the collapsed sidebar from its logo and toggle it there
The collapsed sidebar now shows only the logo; hovering it reveals the
icon strip until the pointer leaves. The expanded sidebar drops its
top-right collapse button and the logo collapses it on hover instead.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fade the peeked sidebar in at full width and keep the toggle slide
Hovering the logo of the collapsed sidebar now fades in the fully open
sidebar, floating over the page at its full width, instead of resizing
it. Opening or closing with the toggle still resizes and slides: the
sidebar docks to strip width while it slides, then floats out of sight,
and it stays hidden under the pointer after a toggle close. The unused
icon-only strip is removed from the sidebar actions.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: stack the sidebar above the chat tabs on desktop
The peeked sidebar floats over the page, but the chat tab bar shared its
z-index and came later in the page, so the tabs were drawn on top of it.
Assisted-by: Claude (Opus 5.5 Medium)
MCP server groups in the settings Tools tab now get a group checkbox like the
built-in, server and browser groups, toggling the server-scoped tools key.
Indeterminate checkboxes now use the checked state colors.
Assisted-by: Claude (Opus 5.5 Medium)
Ask the browser to keep IndexedDB and localStorage data under storage
pressure, so conversations and settings are less likely to be evicted.
Assisted-by: Claude (Opus 5.5 Medium)
* gguf-py : add Limite arch
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* convert : add Limite support
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* llama : add Limite support
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
---------
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* metal : faster MXFP4 mat-vec kernel
Decode the fp4 values with the fp16 bit trick instead of a threadgroup lookup table, and load the packed nibbles as aligned 32-bit words. The kernel no longer uses threadgroup memory.
* metal : launch contiguous unary and binary ops with full threadgroups
The contiguous unary path and the row-broadcast binary path used one thread per threadgroup. Use up to 256 threads per threadgroup and bound-check the tail.
* metal : split GLU rows across threadgroups
A GLU row ran on one threadgroup, so single-row decode used a single threadgroup for the whole op. Split each row over threadgroups of up to 256 threads.
Requested in the #31125 review. The cases duplicate coverage that is
already in the file and add CI time. Test-only, nothing changes in the
kernel, the docs or the CMake option.
Assisted-by: Claude
* hex-upscale: basic upscale ops (not fully vectorized)
* hex-upscale: optimize nearest 2x, 4x and bilinear 4x
* hex-upscale: add support for non-contig rows and dims collapsing
* hex-upscale: use vgather instead of scalar staging
* hex-upscale: move all kernel select logic to the host, review comments and cleanups
* tests: add permuted UPSCALE test to cover qwen3-vl image->text scenarios
* sycl : add DPAS flash attention kernel for quantized KV cache
Decode and small batches (1 to 31 query tokens) now run flash attention
straight on q4_0 and q8_0 KV blocks, instead of scalar dequant or
converting the whole cache to f16 per call. Q.K uses s8 x s4 and s8 x s8
DPAS on XMX, V.P stays f16. dpas.s4.s8 is inline vISA since joint_matrix
has no s4.
Covers head sizes 64 to 512, GQA up to 32 and attention sinks. Prompts
stay on the existing kernels. GGML_SYCL_FA_DPAS=0 turns it off.
B580, Llama 3.1 8B Q4_K_M, q4_0 KV, tg128: 31.3 to 54.2 t/s at 16K
context. FLASH_ATTN_EXT 4582/4584, same 2 failures as master.
Assisted-by: Claude
* sycl : 192 head dropped, Wildcat Lake added to docs
Assisted-by: Claude
* sycl : stricter checks from review
Assisted-by: Claude
* [SYCL] : rename XMX macro, add GGML_SYCL_SUPPORT_XMX option, document DPAS
Assisted-by: Claude
* sycl : widen reordered q4_0 dequantization stores
The reordered-layout Q4_0 conversion kernel processed one qs byte per
work-item with two scalar stores, strided 16 elements apart. Give it
the same treatment the Q8_0 kernel got in #30775: a flat
sycl::range<1> launch with four qs bytes (8 outputs) per work-item
inside one 32-element block and two vec<dst_t,4> stores.
Same float math and rounding as the scalar kernel; output is
bit-identical. The existing k % QK4_0 == 0 assert guarantees whole
32-element blocks, so the k/8 launch covers every row exactly. The
now-unreferenced scalar reorder kernel is removed.
Arc Pro B70, primed prefill (prefill after decode on a
reorder-installed tensor): +6.4% to +7.3% pp512/pp2048 on Q4_0
models (tested up to 27B dense); fresh-layout prefill and decode
unchanged; PPL within run noise.
* sycl : vectorize reordered q4_K dequantization stores
The reordered-layout Q4_K conversion kernel already loads its
quantized inputs through sycl::vec but writes the dequantized
results with scalar stores. Build the results in a vec<dst_t,n>
register and emit two vector stores instead. Same per-element
math; output is bit-identical.
Arc Pro B70, primed prefill (prefill after decode on a
reorder-installed tensor): +14.7% to +15.3% pp512/pp2048 on
Q4_K-dominant models (tested up to 27B dense); fresh-layout
prefill, decode and PPL unchanged.
* sycl : remap reordered q5_K dequantization to vec4 loads/stores
The reordered-layout Q5_K conversion kernel gave each thread only
four outputs fed by scalar ql/qh byte loads plus a local-memory
scale broadcast, so the stores were never its bottleneck: widening
the stores alone measured within noise on mixed-quant models and
+0.5% on a q5_K-dominant model. Re-decompose to 32 threads per
block, each consuming four consecutive ql and qh bytes and emitting
two aligned vec<dst_t,4> stores; a 4-element run shares one scale
pair, so the per-element math and the output are identical.
Arc Pro B70, primed prefill on a q5_K-dominant 27B model: +3.6%
pp512/pp2048 over the scalar kernel; fresh-layout prefill, decode
and PPL unchanged. The GET_ROWS reorder path moves to the same
32-thread decomposition.
* sycl : remap reordered q6_K dequantization to vec4 stores
The reordered-layout Q6_K conversion kernel wrote four scalars per
thread, strided 32 elements apart, each pair of neighbors sharing
neither instruction stream nor store transaction. Re-decompose the
same 64-thread work-group so each thread emits one aligned
vec<dst_t,4>: a 4-element run shares one 16-element scale group,
one ql nibble source and one qh bit pair, so the per-element math
and the output are identical to the scalar kernel.
Arc Pro B70, primed prefill: +28.7% pp512/pp2048 on a Q6_K 27B
model; fresh-layout prefill, decode and PPL unchanged.
* sycl : vectorize reordered q3_K dequantization stores
Each thread of the reordered-layout Q3_K conversion kernel already
computes four consecutive outputs; build them in a vec<dst_t,4>
register and emit one vector store instead of four scalars. Same
per-element math; output is bit-identical.
Arc Pro B70, primed prefill: +16.8% pp512/pp2048 on a Q3_K-dominant
27B model; fresh-layout prefill, decode and PPL unchanged.
* sycl : vectorize reordered iq3_xxs/iq3_s dequantization stores
Both reordered-layout IQ3 conversion kernels write eight contiguous
outputs per thread as scalar stores (four from each grid lookup).
Build two vec<dst_t,4> registers and emit two vector stores. Same
per-element math; output is bit-identical.
Arc Pro B70, primed prefill: +20.3% to +21.1% pp512/pp2048 on
IQ3_XXS- and IQ3_S-dominant 27B models; fresh-layout prefill,
decode and PPL unchanged.
* sycl : remap reordered q2_K dequantization to vec4 stores
Re-decompose the reordered-layout Q2_K conversion kernel the same
way as Q6_K: one aligned vec<dst_t,4> per thread instead of four
scalars strided 32 apart. A 4-element run shares one scale byte and
one 2-bit lane, so the per-element math and the output are
identical to the scalar kernel.
Arc Pro B70, primed prefill: +13.1% pp512/pp2048 on a Q2_K-dominant
27B model; fresh-layout prefill, decode and PPL unchanged.
* sycl : widen reordered mxfp4 dequantization stores
Give the reordered-layout MXFP4 conversion kernel the same flat
sycl::range<1> launch as Q4_0: four qs bytes (8 outputs) per
work-item inside one 32-element block and two vec<dst_t,4> stores.
Same LUT lookups and per-block scale; output is bit-identical. The
k % QK_MXFP4 == 0 assert guarantees whole blocks, so the k/8 launch
covers every row exactly. The old scalar kernel is removed.
The reordered layout for MXFP4 is installed on the MoE expert path
(reorder_qw_mxfp4_moe); dense MXFP4 tensors keep the normal layout.
Arc Pro B70, MXFP4 MoE 20B model: +57% pp2048 / +61% pp512 prefill
(primed and fresh alike), decode unchanged, PPL within the model's
run-to-run noise.
SCALE now accepts a src0 view with arbitrary row strides. translate_scale
used the raw input, and on the dynamic path a VIEW translates to its whole
source tensor, so a strided view input scaled the entire source. The model
output then had the source's size, and copying it back overran the result
buffer, which crashes test-backend-ops on the SCALE view cases.
Slice the input with process_view_input_new, as the other translators do.
Reject an in-place SCALE on a non-contiguous view: its result is copied
back densely and would land in the wrong elements.
* hexagon: IM2COL over multiple threads with output-col blocks
Assisted-by: Claude Opus 5.5
* hex-im2col: fix padding handling
* hex-im2col: loop unroll common cases
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
test_cancel_request enables context shift so the generation only ends
when the request is cancelled, then polls /slots with a deadline.
Without context shift the generation fills the context and frees the
slot on its own on fast machines, so the test passes even with
cancellation disabled.
test_server_sleep uses the existing wait_for_sleep helper instead of a
fixed sleep.
* common: bound control vector layer indices by the model layer count
A direction tensor layer index is parsed from the tensor name with no
upper bound, and the buffer size and write offset are computed in int.
A large index overflows both, so the accumulation writes before the
start of the buffer.
Pass the model layer count to the loader, reject indices above it like
the existing zero index check, and compute the sizes in size_t.
* mtmd: fill M-RoPE positions by whole merge tiles
The position loops of the Qwen-VL, GLM, PaddleOCR and Qwen2.5-VL
encoders hardcode a 2x2 merge inside a stride of n_merge, so a patch
grid that is not a multiple of the merge writes past the positions
buffer. Iterate over whole merge tiles with n_merge inner steps, which
bounds the writes by the patch count and gives the same positions on
aligned grids.
PaddleOCR also requires a merge aligned image at graph build, as the
Qwen-VL family already does.
* opencl: run the tiled q4_K lm_head through a tiled GEMM at batch > 1
* opencl: split tiled dp4a q4_k gemm into its own file and test for dp4a
Assisted-by: Codex
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hexagon: add conv2d op
* hex-conv2d: add double-buffered staging
* hex-conv2d: full 64-wide activation block transpose straight from staged buffer
* hex-conv2d: implicit gemm
* hex-conv2d: remove scalar path and kernel
* hex-conv2d: unify conv2d kernel params for all paths
* hex-conv2d: scale HMX result before f16 storage and then restore during unpacking to prevent overflow
* hex-conv2d: fix f16 overflow in HMX paths
* hex-conv2d: refactor to rebase to new op fusion machinery
* hex-conv2d: restructure DMA pipelines and trace scopes
* hex-conv2d: removed the HVX_WIDTH and HVX_TILED paths
* hex-conv2d: remove in-loop vector spills
* hex-conv2d: refactor kparams, vtcm-layout and dma-pipeline to match the other ops
* hex-conv2d: add basic support for CONV_2D_DW
Co-Authored-By: Pranav Dhinakar <pdhinaka@qti.qualcomm.com>
* hex-conv_2d_dw: use DMA in all paths
* hex-conv_2d_dw: support for chunking larger inputs
* hex-conv_2d_dw: remove scalar/hvx access for w tensor (replaced with sync dma copy)
* hex-conv-2d: fix warnings
* hex-conv2d-dw: vectorize all paths
* hex-conv-2d-dw: tree-based reductions and unrolling to optimize
* hex-conv-2d-dw: uninline compute loops from dma loops
* hex-conv-2d-dw: move all kernel selection into the host solver and simplify/specialize kernel dispatch
* hex-softmax: update obsolete gates to enable SOFT_MAX for mmproj
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Pranav Dhinakar <pdhinaka@qti.qualcomm.com>
* mxfp4 sve 256 impl completed
* working with pretty good perf improvement
* review comments addressed
* review comments for SVE 256 check
* trailing white spaces removed
---------
Co-authored-by: Anubhav Srivastava <anubhav.srivastava@fujitsu.com>
* llama : fix DSV4 compressor padding for simple batches
Count actual tokens per sequence when padding compressor writes. Add a graph reuse regression test for odd-sized DeepSeek V4 and V4.1 batches.
Assisted-by: Codex
* tests : cover multiple sequences in DSV4 padding regression
Exercise one and two sequences at different positions while keeping a fixed ubatch shape.
Assisted-by: Codex
* llama : remove simple batch splitting for DSV4
Use the existing sequence splitting to keep n_seq_tokens consistent with
the compressor padding calculation. Extend the graph reuse regression to
6-token batches.
Assisted-by: Codex
* codeowners : refresh teams and maintainer PRs link
Update the CODEOWNERS team legend with the current `ggml-org` team
memberships, and extend the maintainer PRs link in the README to cover
all current members of the `maintainers` team.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : list remaining backend sub-teams
Add the `ggml-*` sub-teams that were missing from the team legend at the
top of the file: base, cpu, et, hip, openvino and riscv. The legend now
covers every sub-team of the `ggml` and `llama` maintainer groups.
No ownership rules change - only the comment block is extended.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : use ggml-hip team for HIP backend
Route `/ggml/src/ggml-hip/` to `ggml-org/ggml-hip` instead of a single
collaborator, so review requests reach the whole team.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : add llama-examples team
List the newly created `llama-examples` team in the team legend at the
top of the file.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : use llama-examples team for examples
Collapse the per-directory rules that only listed a single collaborator
into a single `/examples/` rule owned by `ggml-org/llama-examples`.
Directories owned by other collaborators keep dedicated rules, since the
last matching pattern takes precedence in CODEOWNERS.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* cont : update zdnn
* codeowners : use llama-base team for src
Route `/src/` to `ggml-org/llama-base` and list the new team in the
legend at the top of the file. Rules for individual `src` files are kept,
since the last matching pattern takes precedence; `/src/models/` drops
its individual owner, which is already covered by `llama-models`.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* mtmd : add LFM3-VL and LFM3-Omni video input
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* mtmd : reserve the soundtrack command to avoid a GCC 11 warning
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui: build the reasoning menu from the template effort levels
When the chat template reports effort levels, the menu offers Default,
Off and those levels, and a pick is sent as reasoning_effort. A
template that reports none keeps the token budget menu.
Picking a model other than the one that produced the last answer resets
the conversation effort to Default, and an effort the current menu
lacks resolves to Default.
* ui: wait for a cold router model before building the request
A router serves the props of a loaded model only, so a request sent to
a cold model was built without its template effort levels and fell
back to a token budget. The request now waits for the model to load,
joining a load already in flight, and a stop during the wait sends
nothing.
Status waiters hold a list per model so a load and the requests that
wait on it each get theirs, and a props request in flight is shared
with a second caller instead of answering null.
* fix: clamp the A prefetch to end_k in the Vulkan coopmat1 int8 matmul, like the B prefetch
* fix: zero the A scale for k steps past end_k in the Vulkan coopmat1 int8 matmul instead of relying on the clamped block
* chat : strip trailing whitespace from parsed reasoning and content
Assisted-by: Claude Fable 5.1
* chat : stop qwen3-coder reasoning and content before template separators
Assisted-by: Claude
* tests : remove the Qwen parser case from the chat session tests
Assisted-by: Claude
* common : handle partial prefix until match at end of generation
---------
Co-authored-by: Zeshawn Zahid <zzshawnn@gmail.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
* ui : show save and reset settings buttons only when relevant
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move reset settings button next to reload app, pad settings caption
Assisted-by: Claude (Opus 5.5 Medium)
* ui: support OGG audio attachments by converting them to WAV
Fixes#24164
Assisted-by: Claude (Opus 5.5 Medium)
* ui: convert all non WAV/MP3 audio uploads to WAV
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep new chat draft when adding a system prompt
Fixes#24338
Assisted-by: Claude (Opus 5.5 Medium)
* ui: remove unused initialMessage from chat form
Assisted-by: Claude (Opus 5.5 Medium)
* ui : replace the sidebar search mode with a search dialog
Search opens a spotlight dialog (Cmd+K or the Search entry under New chat in the sidebar) with keyboard navigation over recent chats and sidebar actions. Phones keep the search page.
Sidebar actions show their shortcuts, with new ones for Settings (Cmd+,) and Models (Shift+Cmd+M).
Assisted-by: Claude (Opus 5.5 Medium)
* ui : ignore search dialog keys while an IME is composing
Enter confirms an IME candidate and the arrow keys move through the candidates, so the search field leaves them to the IME instead of opening the highlighted result or moving the highlight.
Assisted-by: Claude (Opus 5.5 Medium)
* server : add realtime conversation and transcription on /v1/realtime
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* examples : default the realtime client to the server port 9931
Assisted-by: Claude Opus 5.5
* server : sample realtime responses right after their own decode
Slots respond in turns on one context: sampling at the start of the next step read the logits of whichever slot decoded last, so concurrent sessions mixed their outputs.
Assisted-by: Claude Opus 5.5
* move more stuff to server-realtime, making it self-contained
* examples : fix the type check of the realtime client
Assisted-by: Claude Opus 5.5
* examples : use the model card prompts in the realtime client
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B
Assisted-by: Claude Opus 5.5
* convert : keep the speech task templates out of the d1 GGUFs
Assisted-by: Claude Opus 5.5
* small refactoring: no_batching
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: expose the reasoning effort levels of the chat template
The capability probe renders a user turn for each candidate level
(minimal, low, medium, high, xhigh, max) and for an unknown level. A
level is honored when it renders without raising and differently from
the unknown level, which shows the template fallback. Levels sharing a
render are aliases and the highest one is kept.
The caps serialize to JSON, and chat_template_caps in /props reports
the list as reasoning_effort_levels, from lowest to highest.
* jinja: keep the caps free of the json dependency
The caps keep reporting their flags as a map, and the effort levels
stay a plain member. common_chat_templates_get_caps() builds the JSON
the server reports from both, so jinja depends on json only for the
probe inputs.
In --models mode the thread count per model is computed and printed,
but never set in the params, so every context runs on the common
default, the number of physical cores. With several models in
parallel this oversubscribes the CPU, and the busy-wait barriers of
the CPU backend make it collapse.
Set the count in cpuparams and cpuparams_batch.
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* convert : read the LFM3-Audio config.json as released
A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).
Assisted-by: Claude Opus 5.5
* mtmd : disable flash attention for the LFM3-Audio depthformer
Assisted-by: Claude Opus 5.5
* tts : add a voice option for LFM3-Audio and LFM3-Omni
The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.
Assisted-by: Claude Opus 5.5
* clean up template handling
* convert : detect LFM3-Audio and LFM3-Omni by model_type
Assisted-by: Claude Opus 5.5
* move banned tokens --> logits bias --> mtmd_helper
* leave mtmd-cli unchanged
* no more template_asr_system
* left over
* llama : keep the LFM3-Audio encoder convs and codec unquantized
Assisted-by: Claude Opus 5.5
* common, mtmd : shorten comments
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* common : support deferred tools and tool_reference content parts
Assisted-by: Claude Opus 5.5
* common : credit the deferred tools co-author
Co-authored-by: Aldehir Rojas <765227+aldehir@users.noreply.github.com>
Assisted-by: Claude Opus 5.5
* tests : fix the type check of the deferred tools test
Assisted-by: Claude Opus 5.5
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* tests : set the SWA rope base of the LFM3 test model
Assisted-by: Claude Opus 5.5
* llama : enable recurrent state rollback for LFM3
Assisted-by: Claude Opus 5.5
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>
* gguf-py : align the lfm3vl mapping comments
Assisted-by: Claude Opus 5.5
* llama : fix CRLF line endings from the review suggestions
Assisted-by: Claude Opus 5.5
* tests : add the LFM3 Metal fusion baseline
Assisted-by: Claude Opus 5.5
* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* llama : build the k-pool sparse selection independently of the kq_mask shape
qwen4exp (QSA) and glm5-next pick cache cells by index and no longer read the kq_mask values, but they still shaped the selection like the dense kq_mask. Build the selection as [n_kv, n_tokens] over the cache cells instead, and split it into the streams of the K view right before the attention.
This lets the dense mask change its layout (e.g. per-sequence row lists) without touching the sparse layers.
* improve comment
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : drop the unused build_fused_top_k_mask args, review nits
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* tests : speed up tensor data generation and model saving
Refactor save_models to create only the model, not a llama_context,
which is not needed for saving. Replace the slow std::normal_distribution
draw with a splitmix64 + box-muller normal generator.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : use a shared pool of normal draws for tensor data
Generate a pool of normal draws once with std::mt19937 and
std::normal_distribution, then have every tensor copy a cyclic slice of
it instead of running a per-tensor box-muller. The name-keyed slice
selection keeps the draw deterministic.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : log the size of saved models
Report the produced file size in bytes and MB after each save.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : simplify the model save log
Put the model name, target and size on one line per saved model.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* vulkan: allow dim 3 broadcast in MUL_MAT
supports_op rejected every MUL_MAT whose src0 and src1 differ in ne3,
although mul_mm, mul_mmq and mul_mat_vec already broadcast src0 over
dim 3 through broadcast3. Such a matmul then ran on the CPU with a
copy of the weight, and the split layout changed with the number of
streams in the ubatch.
Keep the strict check for MUL_MAT_ID only, and route the broadcast
cases away from mul_mat_vec_nc, the one path that requires equal ne3.
* tests : harden MUL_MAT with permuted dim 3 broadcast cases
Harden the MUL_MAT coverage with a weight broadcast over dim 3 against
a permuted per-stream activation, the shape of an MLA output
projection with a non unified KV cache, for a single token, several
heads and several tokens per stream.
The expand button wrapping the system message card defaulted to
inline-block. WebKit takes an inline-block's baseline from its last
line even when that line is clipped by the card's max-height, so the
row grew to the height of the whole prompt and left a large empty gap
below the card. Make the button a block so the row matches the card.
Assisted-by: Claude (Opus 5.5 Medium)
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold
estimate with to-be-loaded model size included
use no_alloc to get memory requirements for model load
only set model memory_mb if not previously calculated
use memory margin instead of total size limit, apply to each device separately
add server memory debug logging
move llama_context_device_memory function to llama-ext.h
fix model count exceeded check
improve memory_per_device map naming
improve variable naming, fix style
also strip models memory margin from child processes
cont : clean-up
replace device memory map with buft memory map. Use llama_get_memory_breakdown
extract duplicated check into helper function
move model memory estimation to subprocess
precompute name->buft map, map GPU host types to CPU buft
cleanup unused variable
remove duplicated init calls
* adapt to upstream changes
* wip
* common_memory_reports internface
* SERVER_ROLE_MEASURE
* server-models-sched
* fix build
* server : fix memory accounting of measure children and claim spin
A measure child runs no_alloc, so the free memory it reports does not
include its estimate: subtracting self there erased the memory held by
other processes, down to zero with --fit on, and the next load could
overshoot the margin. Subtract self only for instance reports.
A queued load only checked --models-max before claiming, so when the
memory margin blocked it behind a busy model, load() threw and the
waiter retried in a tight loop. Refuse the claim while the limits are
exceeded and no idle model can be unloaded, so the waiter sleeps on
the condition variable until one goes idle.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* model : add MiniCPM-OCR-Jev support
The model is a Qwen3.5-VL trunk with a value head read at the last token: it
scores an OCR text against an image rendered from it, it does not generate.
Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone.
The class scores are the pooled output of the sequence, read with last pooling,
so the conversion turns score.weight into cls.output and sets
embedding_length_out. The checkpoint has no LM head, so text generation is
rejected instead of using the output weights duplicated from the token
embeddings.
The task is fixed, so its prompt is entirely in the systemone template, the
same way h2o-lightning keeps its default there. The processor of the checkpoint
allows up to 16384 merged tokens per image, more than the Qwen-VL default of
mtmd, so the mmproj keeps the image budget of the preprocessor config. The
checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as
other models do.
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* fix code
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* use gguf meta classifier_output_labels
* remove redudant code changes
---------
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* metal : add GGML_OP_TOP_K_MASK
Port the fused top-k mask from the CUDA backend: a grid-stride kernel fills the output with -inf, then one
threadgroup per row copies the mask values of the kept cells over it. Function constants drop the score,
blocks and extra paths that a given graph does not use.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* metal : clean up top-k mask kernel formatting
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : relax scale to non-contiguous views
SCALE requires src0 to be 1D-contiguous. Relax this to dim-0
contiguity with arbitrary row strides (ggml_is_contiguous_rows),
same as the unary ops in the Metal backend.
- ggml: relax the assert in ggml_scale_impl
- cpu: address src0 rows through their byte strides (f32, bf16)
- cuda: stride-aware kernel, fastdiv row decomposition
- metal: support bf16 scale (guarded unary instantiations)
- sycl: keep the gate at contiguous, the kernel is flat f32
Add view variants to test_scale. Verified on CPU, Metal and CUDA.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* cuda : skip the softcap fusion on non-contiguous views
The fused softcap kernel reads src and writes dst as dense arrays,
so a view reaching SCALE -> TANH -> SCALE now that SCALE accepts
non-contiguous rows would be read wrongly. Keep such graphs on the
strided kernels until the fusion becomes stride aware.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* tests : fix multi-seq split replay against trained models
The rollback snapshot is captured by the graph that decoded the rolled-back
tail, while the reference context decodes the same prefix with different
batch shapes. The two states differ in the last bits and trained models
amplify the difference with every replayed token. Compare only the first
replayed token of each seq, then restore the rollback state into the
reference context before the seq-1-only tail.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : require bit-exact seq-1 isolation in split replay
The seq state save is not always bit-lossless - SWA-masked cells are not
saved - so reload the restored state back into the rollback context. That
way both contexts enter the seq-1-only tail with byte-identical memory,
including the cell layout, and the tail logits must match bit-exact. Also
assert that the restored state round-trips bit-identical.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : drop the n_swa partial-state check from ps
The check keys off llama_model_n_swa(), which returns 0 for dsv4 even
though its raw KV cache has a 128-cell sliding window. The partial state
grows with the tokens there by design, so the flatness check is wrong.
Keep the partial <= full invariant.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Every caller passes n_pad = 1 and the only two users clamp it to 256.
Drop the parameter, the members and the trivially true kv_size assert.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The org-less group renders without a header, so when it followed a named
org its models looked like they belonged to that org. Put it ahead of the
named org groups.
Fixes#25227
Assisted-by: Claude (Opus 5.5 Medium)
On desktop, the reasoning effort toggle now gets a muted background on
hover and while its menu is open, matches the model selector's 24px
height, uses a 13px icon and sits 5px from the selector.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add conversation compaction
Add a compaction action that asks the model to summarize the
conversation so far and stores the summary in a new compaction node
appended to the branch. Later requests slice the branch at the last
compaction node, keeping the system prefix and continuing from the
summary instead of the full history, which frees up context while the
complete history stays visible and navigable in the UI.
The slice is applied once at the request funnel, covering the plain
send, the agentic flow and the KV cache pre-encode, plus the continue
path that bypasses it. The summary request is a one-shot outside the
SSE replay machinery, streamed live into the node with a full rollback
on abort, error or empty result.
The action is exposed as a /compact slash command and a Compact button
in the context gauge popup, with an overridable summarization prompt
in the settings.
* ui: pre-fill KV cache and refresh context gauge after compaction
Once the summary is stored, pre-encode the compacted branch and stamp
the returned prompt timings on the compaction node: re-encoding the
reduced prefix is part of the compaction itself, so the server KV
cache and the context gauge reflect the summary right away.
The summary stream is registered under a prefixed identity
(compact::convId), so the visibility kick and byte-offset resume
protect it through tab switches while the reload probe never
reattaches it as a chat turn. The stamped node becomes the shared
context truth for the gauge and the processing state restore, and the
stale live state that kept the gauge full is cleared.
* chore: format
* ui: allow a dedicated model for conversation compaction
An opt-in setting reveals a model selector whose options come from the
live router model list. The chosen model writes the summary and is
loaded on demand when it is not resident, while the default path keeps
resolving to the conversation's own model.
* ui: align the compaction model field with project conventions
* ui: fix the model option helpers import path
The compaction model selector pulls filterModelOptions and
groupModelOptions from the models component folder, where they no
longer live, so the build fails to resolve the module. Point the
import at the navigation folder, which now owns them.
Missed during the rebase: the helpers moved in a commit that does
not touch any compaction file, so nothing conflicted.
* ui: import the model option helpers from models/utils
* ui: follow the moved models selector in the compaction model field
The models manager moved ModelsSelectorList under ModelsSelector and
the model option helpers to /utils, where groupModelOptions takes
the options alone. Drop the stale copy the rebase kept and read the
helpers from their new home.
* ui: drop the dedicated compaction model
A summary written by another model loses the prefix the conversation
model cached, so the whole history is prefilled again, and the model
then resumes from text it did not write. The conversation model
always writes its own summary.
* ui: keep the last steps of a compacted conversation
A compaction now folds the steps the model reads but the last ones,
a step being a user message or an assistant message with its tool
results, and stores how many it kept on the compaction node. The
model then reads the system prefix, the summary, the kept steps and
what follows.
The summary request is the request the chat sends, with its options,
tools and message shaping, cut before the preserved steps: the server
reuses the prefix it cached and only the instruction is new. The
number of preserved steps is a setting, 10 by default.
* ui: run every compaction summary through one path
Three callers needed the same thing - the automatic pre-encode pass, the
action in the context gauge, and re-running a summary from the block - and
each of them had grown its own copy of the request, the streaming and the
storing.
There is one path now: a summarization owns a node, streams into it, and
reports itself through a ledger the rest of the app reads (`isCompacting`,
which node belongs to which conversation). Re-running a summary is the same
call on an existing node, so the two entry points cannot drift apart again.
* ui: give the compaction row its own summary UI
The summary was a line of text in a tool-call-shaped collapsible, which is
how a compaction reads as a side note rather than as the thing the rest of
the conversation now stands on.
The row is its own block: a divider that names the boundary, and the summary
inside it with the model that wrote it, its statistics, and the same actions
an assistant turn has - copy, edit, fork, regenerate and delete all apply to
the summary, and it collapses to the divider once a summary is in, because
the shrunk context is the headline and the prose is the detail.
The divider is a variant of the shared collapsible block, which also needed
the toggle and the title split into snippets so a caller can lay them out
without copying the markup. The action icons no longer take a role: they
render what a message offers, and a compaction offers the same as any other
stored message.
* ui: report compaction progress on the row
The row owns the run, so the row says how it is going: the prompt progress
in the header while the context is being read, the phase once the summary
starts streaming, and the token count once it is in. Before the first
summary token there is nothing to collapse, so the header stands on its own
instead of offering an empty block.
* ui: keep the summarized turns quiet while a compaction runs
The turns being summarized used to keep their own "Processing..." indicator
and live reading stats while the summary streamed, so the conversation said
the same thing twice in two places. The row reports the run, so the turns it
replaces stay silent until it is done.
* ui: hand the compact action to the context gauge
Compacting is a decision about the context, so the action belongs with the
numbers that show the context filling up rather than inside the form: it sits
at the bottom of both the hover card and the drawer, under the breakdown it
acts on, quiet enough to stay a footnote and disabled - with the row's own
spinner - while a summary is already running.
* ui: follow the bottom, otherwise leave the view alone
The screen scrolled itself with a controller of observers, intervals,
direction heuristics and landing timers, and still managed to move the view
while the reader was reading: any mutation that grew the branch was treated
as a reason to keep following.
The page is the scroller, so it follows when the reader sits at the bottom,
and stays where it is otherwise. A resize observer plus native scroll
anchoring does the rest, and `pointerdown` buys the reader a moment of grace
to be caught mid-drag.
* ui: expand collapsible bodies with a height transition
Expanding a block snapped it open: the body was mounted the moment `open`
flipped, so a tall block moved the conversation around it in one frame.
The body now goes through the same region the models table uses. The height
interpolates and the visibility follows it, so a body is unmounted once the
collapse has finished rather than the moment it starts - a closed block
still costs nothing while tokens stream past it.
* ui: open a compaction summary at the top of the screen
Toggling a summary left the reader wherever they were, so a conversation that
had just compacted showed the divider at the bottom of the screen and read
down from there.
A toggle now scrolls the page once and lands the divider a breath under the
tabs, both ways: the summary is read from its beginning, and closing it returns
to the same boundary instead of wherever the page happens to come to rest. The
scroll waits for the body to reach its height, because before that the page is
too short to scroll the divider up there and the call would clamp.
That wait is the shared collapse duration rather than a transition event: an
engine without `interpolate-size` animates the region's grid row instead of its
height, so a listener on `height` would never fire there and the page would not
move at all.
* ui: spin the compaction icon while the summary runs
The divider reported the phase in its title while the icon still showed a
finished block, so a run in progress and a summary that had landed looked the
same in the one place the eye reads first. The icon spins while the summary is
running and folds back once it is in.
* ui: let the compaction summary speak without the reasoning trimmings
The summary is the thing the conversation now stands on, not a note beside
it, and the conversation model always writes it, so the body drops the
reasoning block's rule and indent (`flushBody` keeps them for the blocks
that borrow them) and the model selector under the summary is gone - there
is nothing left to swap.
* ui: drop the model override from the compaction path
The summary row no longer offers a model selector, so nothing hands
regenerateCompaction a model anymore. Remove the parameter from the
whole path, with the on-demand load of another model it carried: the
summary always comes from the model the request funnel resolves.
* ui: let the compaction summary think as the chat does
The summary request forced thinking off, overriding the reasoning
effort the user chose. The summary is the text the conversation
resumes from, the task where reasoning helps the most, so the request
now carries the very options of the chat.
* ui: pre-encode the compacted branch as the next turn reads it
The post-compaction pre-encode sent the branch without tools, while
the tools render ahead of the messages, so it warmed a prefix the next
turn never reads and that turn prefilled the whole compacted context
again. The pre-encode now sends the tools, message shaping and
reasoning handling of the chat, so the next turn only prefills its new
message.
* ui: pre-encode the branch the next turn actually reads
The per-turn pre-encode sent the messages of the request plus one
assistant message holding the streamed text, without tools. With
tools on, it replaced the slot the next turn extends with a prefix
nobody reads again, and after an agentic turn its history did not
match the stored calls and results. Without agentic, it never ran.
Both pre-encodes now read the stored branch up to its leaf, cut at the
last compaction, and send it with the tools, message shaping and
reasoning handling of the chat. The plain completion path triggers it
too.
* ui: refuse a compaction summary the output limit cut
The stream handler read every chunk but dropped the finish reason, so
a summary cut by max_tokens folded the conversation into half a text.
The handlers now hand the finish reason to onComplete, and a summary
that ends on length rolls the compaction back with an error.
The summary was framed through a replacement string, so $& or 32560 in
it changed the text the model read; it is framed as written now.
* ui: wait for the collapse transition in the tool call tests
The collapsible body now unmounts once its collapse animation ends rather
than as soon as the block closes, so the "folds back" assertions ran before
the body was gone. Wait out COLLAPSIBLE_TRANSITION_MS first.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* ui: open model details in a global drawer without leaving the page
The model info actions in the selector used to navigate to the models page
to open its details pane, and a phone opened a separate dialog instead. The
drawer now lives outside the models table, driven by uiStore, so the info
icon opens it over whatever page is showing.
The models dropdown and the phone picker stay open under the drawer's
backdrop, and keep focus while the drawer opens and closes, so the trigger's
tooltip no longer pops over the drawer.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let the model details drawer reopen after a drag to dismiss
Dragging the drawer down wrote vaul's own open state, which the one-way open prop then kept closed, so the info action could not open it again until a refresh. Bind open both ways through the store instead.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: close the models selector from the details drawer chat actions
Assisted-by: Claude (Opus 5.5 Medium)
Extract the keyboard-shortcut delete confirmation into DialogConversationDelete
and reuse it in the sidebar and mobile search page instead of the native
window.confirm prompt.
Assisted-by: Claude (Opus 5.5 Medium)
* hexagon: add GGML_HEXAGON_VTCM_MB and GGML_HEXAGON_NTHREADS caps
Change-Id: Idffe1613c8db6f96aa842d43d1b2efaed45462a8
* hexagon: rename VTCM cap to GGML_HEXAGON_VTCM_SIZE, drop NTHREADS cap
Change-Id: If4c5b9ce9fd6b725c3561421c2295d40599d012d
* hexagon: make GGML_HEXAGON_VTCM_SIZE an override, accept K/M/G suffixes
Change-Id: Ib8eea84c22f24e6d986b37f215e504b046478c2c
The hyper-connection and DSA graph builders wrote their extra outputs
(post / comb gates, carried top-k selections, inject gates) through
ggml_tensor** out-params. Return them in a small struct next to the
primary output instead.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Every dst row comes from src0 or src1, or for dim 0 from the row of
src0 followed by the row of src1, so a single kernel covers every
type: a dense source row is one memcpy, quantized types included,
and a strided source is copied element by element with the element
size known at compile time.
Rows are split across threads instead of i2 alone, which is often 1,
and are copied in tiles of 32: for a transposed source the elements
are walked in the outer loop, so the rows of a tile read neighboring
addresses instead of jumping a full row stride per element.
Add transposed b cases to test_concat, the layout of the delta-net
conv input, and a perf case for it.
* sycl: add set_tensor_2d for faster tensor-split loading
Pack host rows into chunks of up to 64 MiB when destination rows are
contiguous, avoiding a separate upload call for each row. Keep the
existing per-row copy path for non-contiguous destinations.
On two Arc Pro B70s with gpt-oss-120b UD-Q4_K_XL, total llama-perplexity
runtime drops from 263-268 s to 114-115 s. A full-load readback check
verified 26,784,256 rows with no mismatches.
* sycl: batch 2D tensor uploads through shared staging slots
Implement set_tensor_2d to speed up tensor-split model loading by
packing rows into the existing pinned staging slots and uploading
each batch with a single device copy.
Share staging initialization with set_tensor and wait only when
reusing a slot, allowing CPU packing and device copies to overlap.
Fall back to per-row uploads for noncontiguous device rows,
oversized rows, or unavailable staging.
---------
Co-authored-by: Jon Staples <250234834+Captain-Tripps@users.noreply.github.com>
* tests: pin model resolution on real repo listings
Extend test-model-resolution with 63 repo listings captured verbatim
from the HF API, chosen to cover every file naming pattern found across
1237 GGUF repos of the major publishers and pruned to whole quant groups.
Each repo pins its resolved plans and its cache listing.
The expectations follow what each file really is per its GGUF header,
not what the current resolver picks, so a mismatch is a resolution bug.
Mismatches are collected and reported together before the test fails.
* tests: leave the listing unchecked on paths the filesystem refuses
Some real split names (DevQuasar's Model.Q2_K.gguf-00001-of-00012.gguf
under a quant folder) exceed MAX_PATH once materialized in the temporary
cache on Windows, so the listing of such a repo is skipped with a note
while its plans stay checked.
* tests: pin real listings to master on GGUF convention names only
Replace the real repo table with 171 HF listings whose file names follow
the GGUF naming convention (lowercase sidecar prefix, - delimited
components, quant last), one per layout pattern found across 1174 GGUF
repos and pruned to whole quant groups. Repos outside the convention are
dropped. Each repo pins the plans and the cache listing master gives
today, so the test is a pure harness: any change in resolution shows up
here, and mismatches are reported together.
test-llama-archs included src/llama-arch.h and src/llama-model-saver.h,
which prevented building on Windows with shared libraries since the
used symbols are not exported from the DLL.
- declare llama_arch_* wrappers around the internal llm_ arch query
functions in the llama-ext.h staging header, together with
llama_arch_kv_name() and llama_arch_supports_save()
- slim llama_model_saver down to the private implementation of
llama_model_save_to_file(): the fixture-writer half used by the test
moves into tests/test-llama-archs.cpp as a local kv_writer
- replace the roundtrip saver usage with the public
llama_model_save_to_file() writing to a temp path (also removes the
tmpfile() fd leak and its Windows workaround)
- move test-llama-archs, test-generate-models, test-llama-context and
test-fusion out of the NOT WIN32 OR NOT BUILD_SHARED_LIBS guard
Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
* Metal : stop overflow from happening in large matrix output offsets
Assisted-by: Codex
* metal : widen batch offsets and MUL_MAT_ID token indices
Assisted-by: Codex
---------
Co-authored-by: feni6 <270037094+feni6@users.noreply.github.com>
* ui : land opened conversations at their bottom before the first paint
Pin the page to its bottom with a ResizeObserver until it stops resizing or the reader interacts, and mount the last rows eagerly so the landing frame shows real content.
Assisted by: Claude (Opus 5.5 Medium)
* ui : make conversation tabs browser-style
A plain navigation shows the chat in the current tab instead of adding one, and the new-chat tab turns into the conversation it starts. New tabs come only from explicit actions: the + button, Cmd/Ctrl-click or middle-click on a sidebar chat, and its Open in new tab menu item, which open in the background. Closing a tab moves to its neighbour among the tabs the view shows.
Assisted by: Claude (Opus 5.5 Medium)
* ui : share the tab surface and tab bar fade styles
Move the tab item surface (shape, resting and active looks) to TAB_SURFACE_* class constants and the fade under the tab bar to a global chat-top-fade class, so other elements can match the tabs.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fit the chat column below the tab bar
The tab bar reported 1.25rem of its height while it takes 3.3125rem (pt-3.25 + h-10), so a short conversation still left a few pixels to scroll. Use the bar's full height and drop the extra 1rem from the column's minimum height, so it fills the viewport exactly; the centered new-chat form moves up by the same 1rem to keep its place.
Assisted by: Claude (Opus 5.5 Medium)
* ui : stop the page from bouncing past its ends
Elastic overscroll on the root scroller dragged the sticky and fixed bars (tab bar, chat form) along at the top and bottom of a chat, and they jumped back on release. Set overscroll-behavior: none on html and body.
Assisted by: Claude (Opus 5.5 Medium)
* ui : stop scrolled dialogs from bouncing past their ends
Dialogs scroll on their own (the settings panel, the search results), so the page rule did not reach them and their sticky navigation still jumped on overscroll. Turn off overscroll on dialog panels and on the scroll containers inside them. Only scroll containers: Firefox counts an overflow-hidden box such as truncated text as one, and `none` there kept the wheel from scrolling the list around it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fit a first exchange opened by the working directory row
The last answer's minimum height leaves room for the last question only, so the synthetic row that a new chat starts with (Set working directory to ...) made a short first exchange overflow by that row. When everything above the last question is synthetic, count those rows as part of it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : keep following a streaming response until the reader scrolls up
Auto-scroll stopped whenever the page moved up, including moves the
reader did not make: scroll anchoring and blocks re-rendering shorter
while a response streams. Only the reader's wheel, touch, scroll keys
or scrollbar drag now count as scrolling up.
The send scroll also looked up the new user bubble by position, which
picked an older message when the answer row had not rendered yet.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fade the chat form between its centered and bottom positions
The form no longer slides across the column between the new-chat center
and the bottom. It fades out drifting toward where it goes and fades
back in where it lands, quicker when opening a chat than when going back
to a new one. Its wrapper keeps the same height in both positions, so
the messages never shift while it moves.
The new-chat greeting fades in as the form settles, also on a refresh or
a first visit. Reduced motion keeps only the fades.
Assisted by: Claude (Opus 5.5 Medium)
* ui : let the expanded sidebar be resized by dragging its edge
Bring back the desktop sidebar resize handle that was dropped with the
shadcn sidebar primitive. The width is clamped between 18rem and 32rem,
persists across reloads, and resets on a double-click of the handle. The
tab bar follows the same width so a wider sidebar never covers it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : rubber-band the sidebar edge past its bounds and drag it shut
Past the max width the edge resists harder the further it is pulled, and
the same happens below the min width until the pull passes 9rem, where it
snaps to the pointer and letting go hides the sidebar. A release outside
the bounds springs the width back to the nearest one.
Assisted by: Claude (Opus 5.5 Medium)
* ui : show the sidebar edge strip faintly on hover and solid while dragging
The strip is clipped to the sidebar's rounded corners through its own
overlay, so the sidebar itself is never clipped.
Assisted by: Claude (Opus 5.5 Medium)
* ui : remember whether the desktop sidebar is open across reloads
A phone opens the sidebar as an overlay, so it always starts closed there
and its toggling never overwrites the desktop preference.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fade the sidebar labels and conversation list with its width
The fade follows the live width, so it tracks both the toggle's width
transition and a drag below the minimum width.
Assisted by: Claude (Opus 5.5 Medium)
* ui : persist the sidebar width on release and clamp a restored one
A drag now updates only the live width and writes it to localStorage once
when released. A width restored from localStorage is clamped to the
current bounds, so a value saved under older bounds does not stick.
Assisted-by: Claude (Opus 5.5 Medium)
* kv-cache : use ggml_backend_tensor_copy to refill the DSV4 K mirrors
* llama : move the Engram pad id to llama_hparams and the token map to llama_model
The saver no longer needs the deepseek41 model class.
* deepseek41 : drop the F32 src1 request on the HC mixes matmul
* ggml : add softplus TODOs in the backends that still use log(1 + exp(x))
* convert : make the DSV4/DSV4.1 modify_tensors generators, drop the redundant vision filter
* deepseek41 : clean up comments
* deepseek41 : drop the F32 src1 requests on the router and indexer matmuls
* convert : use format_tensor_name directly in DeepseekV4Model
* s/comp_kv_src/comp_il_src/g
* cont : fix TODOs for log1p
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cuda : use std::string instead of VLA for archName
Modernize the code (old C style) and fix the following warnings:
```
ggml/src/ggml-cuda/ggml-cuda.cu:180:19: warning: variable length arrays in C++ are a Clang extension [-Wvla-cxx-extension]
180 | char archName[archLen + 1];
| ^~~~~~~~~~~
ggml/src/ggml-cuda/ggml-cuda.cu:180:19: note: read of non-const variable 'archLen' is not allowed in a constant expression
ggml/src/ggml-cuda/ggml-cuda.cu:179:9: note: declared here
179 | int archLen = strlen(devName);
| ^
```
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Add a comment about archName
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
* common, grammar: add recursive depth limit, make it a requirement
* catch more recursive functions
* common : tidy up peg parser changes
* grammar : reject candidates in a loop, without a depth limit
The depth limit in llama_grammar_reject_candidates_for_stack counted
the code points of a vocab token, so any token of 256 chars or more
was always rejected under a grammar, such as the 512-space token of
GLM-4.7-Flash or the long space tokens of Laguna-S-2.1.
The rejection now walks the code points with its own stack of levels
instead of recursing: there is no limit and no stack risk, whatever
the length of a token from the GGUF, and the masks match master.
* grammar : track rejected candidates as indices into the input
At a given code point a candidate only differs from its input by the
position of its code point pointer, so the levels now carry indices
into the input candidates instead of copies. The first level needs no
special case and the rejects map back to the input directly.
Grammar sampling gets faster than master on GLM-4.7, Laguna, Qwen3.6
and gemma-4 with the json, list and spaces grammars, same masks.
* nit : shorten the comment of llama_grammar_reject_step
* common : add depth_guard for schema walks
* grammar : move GRAMMAR_RECURSIVE_LIMIT into the .cpp
* common : reduce recursion limit for peg parser
* improve
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
Co-authored-by: Pascal <admin@serveurperso.com>
* llama-quant : fix quantizing 4D tensors
nrows_total was ne[1]*ne[2], so for a 4D tensor only the first ne[2]
slices were quantized and the rest of the rows were dropped. Use
ggml_nrows(tensor) to count all rows, including ne[3].
Fixes#29916
* llama-quant : detect expert tensors by name and fix imatrix size for 4D tensors
The MXFP4_MOE path used ne[2] > 1 to detect MoE expert tensors. This is
wrong for non-expert tensors with 3 or more dims, e.g. MLA projections
(BailingMoE3) and 4D conv kernels. Use the tensor name instead: all
tensors used by GGML_OP_MUL_MAT_ID are named *_exps or *_chexps. This
also removes the BailingMoE3 special case.
The imatrix holds one vector of ne[0] values per 2D slice, so the
expected size is ne[0]*ne[2]*ne[3], not ne[0]*ne[2]. This matches how
the imatrix is indexed per row for 4D tensors in llama_tensor_quantize_impl.
Assisted-by: Claude Opus 5.5
* cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows
For prompt processing, run the gate and up matmuls of a dense FFN as one
mul_mat_q launch and apply the GLU in the write-back.
- The loaders take the gate weights as a second pointer and fill the weight
tile with up and gate rows interleaved in chunks of 8, so each thread holds
an up value and the matching gate value (rows r and r + 8 of an MMA tile).
- The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[]
per thread. The fused write-back computes the GLU from registers and writes
half the rows.
- The fused path uses the regular tiling (no stream-k) with the column tiles
innermost so that neighbouring blocks share weight tiles.
- Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.
* cuda: assert the fused MMQ gate argument and trim redundant comments
* cuda: move the fused MMQ hint check to should_fuse and add TODOs
* cuda: pass prec_src1 to the fused MMQ write-back
* cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back
* cuda: require has_fusion explicitly, drop the fusion-only src1 precision check
* cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile
* cuda: inline the MMQ tiling-efficiency check
* tests: one fused gate/up MMQ case per type and GLU op
* cuda: always launch the fused MMQ kernel with one block per tile
* tests: shorten the fused gate/up MMQ test comment
* cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs
* cuda: run the fused MMQ kernel on the plain tiling path
* cuda: put the column tiles innermost in the MMQ plain tiling path
* vulkan: Add a larger tile size for conv3d (coopmat2 only)
This adds a larger tile size, in practice it is only for coopmat2 where
we don't need shared memory to store the output (would use more than 64KB).
Add some additional representative backend tests. At least, this is what codex
came up with from looking at sd.cpp Wan VAE.
* only apply large tile size for newer driver versions
WM of the small tiles was 32, or 8 for 8-wide subgroups, whatever the
tile's BLOCK_SIZE and WARP. With 16-wide subgroups (Mali) the second
subgroup of a workgroup wrote into the next tile's columns. With 8-wide
subgroups some _id tiles left rows uncomputed. Derive WM so there is one
warp tile per subgroup.
Tested on my Pixel 9 Android phone. Before the change there were 20
MUL_MAT and 6 MUL_MAT_ID failures. After the change they all pass.
Fixes#28637
Assisted-by: Claude Opus 5.5
* hex-tools: support for --compare option in the profile.py
* hex-unary: improve collapse logic to also do splits if nrows is small
* hex-colapse: handle all colapsing at the host and make it generic
* hex-colapse: remove leftover unary helper
* chat : reject JSON nested more than 128 levels before walking it
The schema builder, the jinja engine and the decision code walk request
JSON with recursion. A request nested deep enough overflows the stack of
the HTTP thread and llama-server dies (SIGBUS on macOS, SIGSEGV on Linux):
for a tool schema from a few hundred levels on macOS and about 10000 on
Linux.
Check the depth first and answer 400 past 128 levels: the chat inputs in
common_chat_templates_apply (tools, tool-call arguments,
chat_template_kwargs, json_schema), the json_schema of /completion, and the
whole /v1/systemone body. A gemma4 tool result that is too deep stays text.
* fix the root, not the call side
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add DeepSeek-V4.1-Flash (deepseek41)
* deepseek41 : remove MXFP8, convert the Engram table to BF16
* deepseek41 : compute the compressed-KV read layers once in the cache
* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le
* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather
* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer
* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle
* deepseek41 : keep il last in build_hc_mixes
* deepseek41 : sort the top-k rows before the sparse gather
* tests : add deepseek41 to the Metal fusion baseline
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* hexagon: KV-head team split for HMX flash attention in multicore
* hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag
0 = off (token-split only); 1 = head-split, plus team-split when
n_kv_heads < N_cores so stranded cores still participate (e.g. gemma-4
with 2 KV heads on 4 cores). Where heads divide evenly across cores,
plain head-split is used.
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the
per expert-token amax scaling done in the compute graph
(ref: https://github.com/ggml-org/llama.cpp/pull/30861), which keeps the
MoE down-projection activations within f16 range before the op runs.
Drop the two-stage reduction kernels, their scratch buffer and the
GGML_PREC_F32 dispatch added in https://github.com/ggml-org/llama.cpp/pull/26223,
as well as the test cases that drive MUL_MAT_ID src1 out of f16 range.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ui : move models manager to its own route, keep Discover as a dialog
Models now lives at #/models instead of a dialog. Discover opens as a
standalone dialog over it, and leaving the page returns to the chat it
was opened from.
Assisted by: Claude (Opus 5.5 Medium)
* ui : polish the models page layout and filters
Give the models page a page header with the Discover action, calmer
column labels, tinted section and family headers, and an empty state.
Load the model list on a direct visit, slide the detail pane over the
table below xl, and draw the filter toggles as one box matching the
context select, with room for their shadow.
Assisted by: Claude (Opus 5.5 Medium)
* ui : round the models table family headers
Assisted by: Claude (Opus 5.5 Medium)
* ui : fix models page navigation and Discover gating
Record the models page's way back after every navigation, so Back,
a typed URL or a reload no longer return to a stale chat. Escape
closes the detail pane, then leaves the page. Gate every Discover
entry point on one shared check, recheck the API key on /models,
skip the duplicate router fetch on a direct load, and share the
route load between pages.
Assisted by: Claude (Opus 5.5 Medium)
* ui : show model details in a drawer
Open the models manager's detail pane in a drawer, from the right on
desktop and from the bottom on a phone, instead of an inline side
panel and a full-screen cover. Pad the pane on both sides, drop its
close button on a phone, and assert that filtered rows are gone from
the page rather than merely hidden.
Assisted by: Claude (Opus 5.5 Medium)
* ui : make whole models table rows clickable
A click anywhere on a model or quant row opens it, unless it lands on
one of the row's own controls (load toggle, actions menu). The name
button stays the keyboard path to the same action.
Assisted by: Claude (Opus 5.5 Medium)
Use the dropdown menu surface (border, radius, background, shadow,
padding) for the hover card and the context gauge popup, and fix the
hover card open/close animation to use bits-ui data-state attributes.
Assisted-by: Claude (Opus 5.5 Medium)
The two T4 runners in the ggml-ci group (ggml-6-x86-vulkan-t4,
ggml-8-x86-nvidia-a10) share the same labels and can both pick up the
same jobs, so extend the per-runner results dir from #31046 to them.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The generic scale pass in llama-model.cpp loads the per-expert
NVFP4 scale2 tensors (ffn_{up,gate,down}_exps_s) and the per-tensor
MXFP8 scales for the dense FFN and the shared expert, but
minimax-m3.cpp never forwarded any of them to the graph.
build_moe_ffn() defaults these to nullptr, so build_lora_mm_id()
skipped its w_s branch entirely and the routed-expert weights came
out global_scale times too large, swamping the residual (reported
G ~= 15123 for M3.1 Flash) and producing token soup.
Forward all scale tensors at the three call sites, matching the
pattern used by flint and the other NVFP4-capable MoE models.
Co-authored-by: origin-labs-ai <origin-labs-ai@users.noreply.github.com>
Add test-speculative-rejection-sampling to the ctest -E exclusion list of the macos-latest-arm64 CI job.
Assisted-by: dsh:llama.cpp/ggml-org/Flint-GGUF (RTX 5090 32GB, DDR5 128GB)
The amax POOL_1D reduction spans the whole matmul row, so a device cannot
compute it from an axis-0 split. The Meta split-state case reported
MIRRORED, which gives each device a partial max of its own slice. Drop
that case and deny -sm tensor for the arches which build the amax graph,
see [TAG_MOE_AMAX_SCALE].
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : allow MUL_MAT_ID -1 index to skip the computation
* nits
* update vulkan code
* ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec
(cherry picked from commit d8a4b9efb6e2b9abf3a866e847756b7e6bbc6b69)
* fix cuda
* openvino
* exclude CUDA from this PR
* add TODO tag
---------
Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
* tests : add speculative rejection draft sampling test
This commit adds a test for speculative rejection sampling test that
excercises the speculative rejection sampling mechanism in
common/speculative.cpp.
The motivation for this is have a test that verifies that the
speculative draft models emitts tokens that match the target models
distribution. So when rejection sampling occurs it should still
produce the same probability distribution as if we directly sampled
from the target model.
Running the test:
```console
$ ctest --test-dir build -R "^test-speculative-rejection-sampling$" \
--n_trials 5000 -V
```
This will generate two models, one target and one draft model.
For subsequent runs use the "fixture exclude any" option to avoid
recreating the models:
```console
$ ctest --test-dir build-cpu-debug -R \
"^test-speculative-rejection-sampling$" \
-FA "^speculative-models$" --n_trails 5000 -V
```
* remove draft model
* ci : fix data race between runners sharing the model cache
Two runners per mac mini share HOME, so concurrent jobs corrupt each
other downloads, converted models and venv under the shared MNT cache.
- gg_wget: download in a temp dir, publish with mv, so readers only
ever see complete files (-N cannot be combined with -O)
- convert, quantize and wikitext unzip now produce under a per-runner
scratch dir in OUT, published to MNT via hardlink when missing
- gguf-split and quantize tests get a per-runner TMP_DIR
- python venv moved from MNT to OUT
- metal, vulkan and webgpu mac jobs pass a per-runner OUT path
Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)
* ci : clean up scratch models after tests
Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)
* ci : fix q4_0 publish when a cached model exists
cp -n skips an existing file with exit code 1, so the ln fallback
aborted the qwen3 job under set -e. Publish through a unique temp
file with an atomic mv: readers never see a partial file and a
stale cached q4_0 gets replaced.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)
* ci : cache-first gg_wget to avoid re-downloads
HF sends no Last-Modified, so wget -N re-downloads the full files on
every run. Skip the download when a non-empty cache file exists: the
HF CI artifacts are static. Downloads still go through a temp dir and
an atomic mv, so concurrent runners never read partial files.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)
* HIP: add byte permute fallback when __builtin_amdgcn_perm is unavailable
amdgcnspirv rejects target-specific builtins unless they are guarded by `__builtin_amdgcn_is_invocable`, so 66 of 145 HIP files failed to build
Wrap the perm calls in a helper that keeps v_perm_b32 when the target has it and falls back to a portable byte select otherwise
* move perm helper to vendors/hip.h
* mtmd : lfm2 bilinear resize matches torchvision
Lfm2VlImageProcessor and d1-omni vision.py resize uint8 images with
torchvision (antialias=True). That is Pillow's algorithm, except that the
weights are int16 with as many fractional bits as the largest weight allows,
where Pillow uses 22 bits. The Pillow port was 1-2 levels off on about 1% of
the values of the d1-omni and LFM2.5-VL crops.
RESIZE_ALGO_BILINEAR_TORCH rounds the weights of the Pillow port to that
precision and scales them back to 22 bits, which gives the output of
torchvision with the same passes, and "bilinear" maps to it for LFM2 and
d1-omni. Other models keep the Pillow algorithms.
* mtmd : lfm2 bicubic resize matches torchvision
Lfm2VlImageProcessor resizes with torchvision for the bicubic checkpoints too
(d1-3B, LFM2-VL), so "bicubic" maps to RESIZE_ALGO_BICUBIC_TORCH for LFM2: the
Pillow bicubic filter (a = -0.5, as torchvision with antialias) with the int16
weights of torchvision. The Pillow port was 1-2 levels off on about 0.25% of
the values of the d1-3B crops.
* use bitfield, add hparams setter
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Add Soprano-80M and Soprano-1.1-80M (https://huggingface.co/ekwek/Soprano-1.1-80M)
as an audio generation model: a Qwen3 backbone plus a Vocos decoder that runs
as a gen-only mmproj.
- convert: detect Soprano checkpoints and export the backbone as qwen3 and the
decoder.pth Vocos weights as the mmproj
- vocab: add the soprano pre-tokenizer (lowercase, whitespace collapse,
per-digit split, [UNK] fallback)
- mtmd: add the Vocos graph and ISTFT, allow loading a gen-only mmproj for
Soprano and include the gen-audio context in mtmd_get_memory_usage
- mtmd-helper: add the Soprano pipeline; the decoder runs over the whole
utterance, so stream mode sends the audio in one chunk after flush()
- tts: document Soprano usage
Assisted-by: Codex
Assisted-by: Claude Opus 5.5
* hexagon : add SGN support
Assisted-by: OpenCode
* hexagon : optimize SGN with sign bit
Assisted-by: OpenCode
* hex-sgn: add f16
* hex-unary: update SGN to use generic unary_colapse and relax colapse checks to apply to more conditions
* hex-unary: improve dims colapsing and run it in the host-side solver
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The __builtin_amdgcn_wmma_f32_16x16x16_bf16_w32 (and _gfx12) builtins
take their A/B operands as vectors of short (_ExtVector<16, short> /
_ExtVector<8, short>), not __bf16. Declaring the fragments as
ext_vector_type(N) __bf16 fails to compile:
error: cannot initialize a parameter of type 'short
__attribute__((ext_vector_type(16)))' with an lvalue of type 'const
bf16x16_t' (vector of 16 '__bf16' values)
short and __bf16 are both 16-bit and the value is only ever passed to
the builtin via reinterpret_cast over raw bytes, so changing the element
type is bit-identical. The operand type has been short since the builtin
was introduced, so this works across LLVM/ROCm versions without a
version gate. Fixes HIP builds for RDNA3 (gfx1150/1151) and RDNA4
(gfx1200/1201).
Signed-off-by: Pablo Cholaky <waltercool@slash.cl>
* CUDA: fix build after chunked CUDA graphs (#30804)
* Apply suggestion from @JohannesGaessler
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Fix newlines in common.cuh
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* qwen4exp : build the QSA selection without the kq_mask
* clean
* lang
* qwen4exp : reshape the fused QSA selection to the kq_mask streams
ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while
build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream].
With several streams the shape assert failed at graph reserve.
* test the QSA optimization
* reword
* update build_fused_top_k_mask instead of handwiring
* simplify
* keep il at the end for consistency
Run npm audit fix, then bump the remaining vulnerable packages within
their current majors: storybook 10.6.2, vitest 4.1.12, vite 7.3.7,
@modelcontextprotocol/sdk 1.32.1, dompurify 3.4.17, mermaid 11.17.2,
@vite-pwa/assets-generator 1.0.4 and @tailwindcss/typography 0.5.20.
Override katex to 0.18.11 and postcss-selector-parser to 7.1.6 since
their dependents still pin vulnerable ranges, bump the sharp override
to 0.35.5, and pin @storybook/svelte so npm does not resolve the 11.0
alpha through the addon-svelte-csf peer range.
Storybook stays on 10.6.2 as 10.6.3 is newer than min-release-age.
Assisted by: Claude (Opus 5.5 Medium)
* hex-fusion: support for GELU+MUL, REPEAT+GET_ROWS and BINARY_NX (back-to-back ADD/MUL/...)
* hex-fusion: factor out fusion logic into separate .h and .cpp and update all related code
* hex-fusion: allow for RMS_NORM+MUL fusions for non-contig tensors
* hex-fusion: streamline fusion eligibility checks and cover missing corner cases
* hex-fusion: streamline fusion flow
* hex-opbatch: make n_tensors limit more scalable
* hex-bufs: adjust the trace buff sizing
* hex-glu: cleanup glu-ops interfaces and add kernel_params
* hex-kparams: clean up kparams handling in unary, ssm-conv, and other ops
* hex-kparams: update the rest of the ops PAD,DIAG,... to use kparams
* hex-unary: add support for collapsing dims
* hex-ops: cleanup redundant runtime checks
* hex-ops: consistent use of kernel_type in all kparams
* hex-fusion: remove redundant check in gdn+cpy
* hex-fusion: further streamlining of fusion rules and code
* hex-fusion: cleanup naming
* hex-opfmt: cleanup op formatting
* hex-fusion: bump fusion lookahead to 64
* hex-fusion: fuse SSM_CONV+SILU
* hex-fusion: fix hvx register spills in fused ssm_conv+silu
* hex-fusion: add more GLU fusions
* hex-fusion: remove duplicate checks and use ggml-common helpers
* hex-kparams: refactor kparams into htp-kparams.h/cpp
* hex-fusion: fix binary-nx fusion regression
* hex-kparams: another pass at cleaning up kparams (removing unused, etc)
* hex-kparams: streamling nameing and move _check_ wrappers into .h
* hex-kparams: templetize htp_kparams_check_<op> to reduce code bloat
* hex-ops: de-duplicate op dispatch switches
* hex-kparams: trim down exposed kparams calls
* hex-kparams: simplify type casting of kparams structs
* hex-fence: de-dup fence tensor construction
* hex-opbatch: simplify tensor shape init/checks
* hex-mm: cleanup leftovers of the original QK defines
* hex-fa: fix stack overflow on older npu firmware
* hex-ctx: move dma_caches into ops_ctx
* server: realtime API: send complete after committed audio drains
* server: realtime API: complete a commit at its end row
Completion is triggered when the stream runs dry, which is a timing
proxy: audio appended while the tail drains leaks into the committed
item, a final commit before the drain merges both items, and back to
back commits collapse into one.
mtmd_audio_stream_flush() records the row count at the end of the
flushed audio (mtmd_audio_stream_get_n_rows_flushed), next to the rows
produced so far (mtmd_audio_stream_get_n_rows). The slot keeps a FIFO
of these end rows and completes each commit once every row up to it is
decoded.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* opencl: add Adreno depthwise conv2d
Assisted-by: OpenCode
* opencl: use local program for depthwise conv2d
Assisted-by: OpenCode
* opencl: use singed type for args
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
qwen4exp published the hc-wide residual [n_embd, hc, n_tokens] as
t_layer_inp, but the drafter reads layer inputs as [n_embd, n_tokens]
rows: llama_context sizes each embd_layer_inp buffer as n_embd * n_batch
and the DFlash encoder input is n_extract * n_embd wide. Evaluating a
DFlash draft over a qwen4exp target therefore aborts on the first request
in llama_context::extract_layer_inputs.
Collapse the streams with a plain mean before exposing them, as deepseek4
already does for its hyper-connection residual, and build the mean only
for the layers a drafter actually requested.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* Multi-column on Intel XMX matrix engines refactored
Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).
* Fixed regression bug
real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4
* Removed GGML_TYPE_Q8_0 activation
* rewritten xmx_mul_mat
64-byte alignment and 16-byte pitch check on the activations
* Implemented fused gate/up kernel
One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.
* Removed the temporary XMX_NOGLU environment switch
* Q2_K min-sum optimization
Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group
* Added dispatch check and GRF sweep
tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.
* Renamed variables
* Added 2 environment variables
GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.
* Update esimd.hpp
1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.
2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.
The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
* nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type
* gguf-py : write Metaspace prepend_scheme as GGUF metadata
* vocab : parse metaspace prepend_scheme into enum, rename key
* vocab : use metaspace prepend_scheme in detokenize, add TODOs for other tokenizers
* vocab : use constant for metaspace escaped space
---------
Co-authored-by: Martin Cimmino <martin.cimmino@domyn.com>
* server : support typed content input for rerank endpoint
Assisted-by: Opencode/Qwen3.8 27B
* server : address rerank multimodal review findings
- reject multimodal rerank input when the model has no rerank chat template (images were silently dropped)
- translate handle_media download/base64 failures to 400 instead of 500
- reject negative top_n with 400 instead of letting resize throw
- fix comment typos, redundant comments, indentation and trailing whitespace
- README: document that TEI input accepts the same shape as Jina; drop stray blank line
Assisted-by: opencode
* Trimmed README
---------
Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
* sycl: fix CONV_2D wrong results for non-contiguous input
The kernel indexes the `input` as a contiguous [W, H, C, N] tensor, but
supports_op claimed the op for any layout. A non-contiguous (e.g. `CWHN`
permuted) input was read with the wrong element offsets, silently
producing wrong results.
Gate the CONV_2D op on `input` and `kernel` contiguity, matching the
CUDA backend contract, and assert input contiguity in the op.
`test-backend-ops -b SYCL0 -o CONV_2D`: goes from 16 failures to 0,
none.
```text
2006/2006 tests passed
Backend SYCL0: OK
Backend 2/2: CPU
Skipping
2/2 backends passed
OK
```
Assisted-by: pi (unsloth/Qwen3.8-27B-GGUF:Q4_K_M via llama.cpp)
* sycl : CONV_2D cwhn=1 support
Make the SYCL backend compute `GGML_OP_CONV_2D` correctly for `cwhn=1` (CWHN
channel-order) inputs and kernels, so the previously failing test-
backend-ops CONV_2D cases pass on SYCL instead of being skipped or
miscomputed.
- Implement CWHN layout support in `ggml/src/ggml-sycl/conv2d.cpp`
(`ggml_sycl_op_conv2d`).
- Update SYCL `supports_op` for `GGML_OP_CONV_2D` in
`ggml/src/ggml-sycl/ggml-sycl.cpp` to accept both layouts.
Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF via llama.cpp)
* sycl docs : update ops for SYCL
* feat(ui): add Show tool call in progress display setting
Add the showToolCallInProgress key and its checkbox to Settings > Display,
right next to its "always show" counterpart. Upstream dropped a setting of
this exact name in 43b5e63589, folding it into alwaysShowToolCallContent
because the old name promised expansion that outlives the call; this brings
back the narrower, in-progress-only meaning.
The entry is registry-driven, so SETTING_CONFIG_DEFAULT, the dialog field and
the ui_settings admin baseline pick it up without extra wiring. It ships
enabled, mirroring Show thought in progress, and the always-show help text is
reworded so the two toggles read as a pair rather than contradicting each
other.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)
* feat(ui): expand tool call blocks while they are in progress
Drive the default expansion of TOOL_CALL_STREAMING and TOOL_CALL_PENDING from
showToolCallInProgress and leave TOOL_CALL, the terminal state, to
alwaysShowToolCallContent. A finished call now folds back on its own once its
result lands, which is exactly how the reasoning block behaves when the
thought completes.
Manual toggling keeps precedence over both defaults in either direction:
isExpanded() only consults the settings while expandedStates holds no entry
for the section.
Client tests pin the truth table for the two toggles, including the automatic
fold-back and both manual overrides.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)
* fix(ui): keep running tool calls expanded
A call the agentic loop reports as running now outranks its section type.
exec_shell_command persists an empty result before reading any output, so a
live call is already TOOL_CALL and folded back, which hid the streaming
output the setting is meant to show.
Move the expansion default to isSectionExpandedByDefault so the running-id
rule is covered by a truth table, and test the pending -> finished transition
instead of only the finished state: a block that only ever got mounted
finished cannot prove it folds back on its own.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)
* feat(ui): stream the exec command from the first character
The exec parser read its args strictly, so while the model was still
typing the call the block title stayed empty and the body claimed
"Running...". Parse the args partially (the read_file / edit_file /
write_file pattern), keep a terminal cursor while the args JSON is
incomplete, and stand the conversation cwd in for the x-tool-cwd that
only arrives with the result message, so the prompt is there from the
first character. The body stays empty while the args stream, shows
"Executing command..." once the call is executing, and the wrapper
drops its padding for empty phases so the title sits flush with the
frame.
Assisted by: Delta (zai-org/GLM-5.3)
* fix(ui): stream file tool bodies even before a path arrives
write_file and edit_file refused to render until their args carried a
path, so a call that streams its body first (or a same-named tool with
other arg names) showed an empty block. Meta now carries optional
fileName/filePath, path alias reading moves to shared helpers, and the
blocks fall back to the raw args JSON when no known content key is
present.
Assisted by: Delta (zai-org/GLM-5.3)
* fix(ui): keep an empty write_file body instead of the raw args
A call carrying {"content":""} fell through to the raw-args fallback, so
a finished write of an empty file showed the args JSON. The parser now
leaves content undefined when the key is missing and the block tests
!== undefined, so an empty write renders as an empty code block with its
byte count.
Assisted by: DeepSeek-V4.1-Flash (deepseek-ai)
Two quants of one repo pulled each other's draft: the badge read every
--model-draft arg of the repo, so each row showed both drafts.
Assisted-by: DeepSeek-V4.1-Flash-1017
* opencl: add F32-to-I32 ARGMAX
Assisted-by: OpenCode
* opencl: init to 0 to be consistent with cpu
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* Add Flint model (FlintForCausalLM). Assisted by Codex
* cleanup
* one more
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : add SenseNova U1 understanding support
Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.
- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
tensors and concatenating temporal and spatial Q/K normalization
weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.
Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.
* conversion : treat one-expert SenseNova U1 configs as dense
* mtmd : add SenseNova U1 vision support
Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.
The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.
* mtmd : fix SenseNova U1 vision token layout
* model : clean up SenseNova U1 RoPE metadata
Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.
Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].
* mtmd : simplify SenseNova U1 projector integration
Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.
Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.
* mtmd : fix SenseNova U1 image positions and attention
* fix spacing
* mtmd : restore generic test input chunks
* mtmd : restore SenseNova U1 integration after upstream merge
* Update conversion/sensenova_u1.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : remove SenseNova U1 BF16 activation casts
Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.
Assisted-by: Codex
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama: create new fused op for QSA scoring
* enroll glm5.3-next too
* tests : initialize the remaining tensors in test_kpool_mask
The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check.
* ggml: generalize the k-pool mask op into ggml_top_k_mask
The k-pool mask op only covered the pooled QSA selection of qwen4exp
and glm5-next, while the DSA models and the DeepSeek V4 CSA build the
same top-k mask by hand with fill, set_rows and add.
ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask
cells of the live picks and the extra cells, and sets the rest to
-inf. Every row tensor is 4D in the layout of top_k, and the mask is
broadcast to those rows, so a selection can also be made per head.
score, blocks and extra are optional: without score every pick is
live, without blocks a pick is a cell, and extra holds the cells a
row keeps without a pick.
llm_graph_context::build_fused_top_k_mask views the row tensors in
the layout of kq_mask and registers the fused node. The k-pool
models, the DSA attention of build_attn and the DeepSeek V4 CSA mask
use it when the probe enables the op, and keep their current path on
backends without it.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ggml-opencl : add IM2COL_3D support
Assisted-by: OpenCode
* opencl: use same type for kernel and host
* opencl: fix for older adreno
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* server : forward llama.cpp sampling params on /v1/messages
* server : pass the request through on /v1/messages instead of copying listed keys
* NIT: Change comment from multiline to single line
* server : accept llama.cpp params under extra_params on /v1/messages
* server : remove test case for trivial functionality
* server : show extra_params in the /v1/messages README example
* ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table
* ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path
* ggml-cpu : add the VXE gelu_quick path
* tests : add test-gelu-quick for the f32 CPU kernels
* tests : drop inf/nan inputs from test-gelu-quick
* llama : scale moe down projection per expert token
Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. Undo the scale on the output with the same exact power-of-two factor. Handle POOL_1D in the Meta split-state logic so a split along axis 0 becomes a replicated per-device partial result, enabling tensor parallelism.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : simplify comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : refactor amax src1 scaling result
Return the rescaled activation and the optional scale tensor through a small struct instead of an output pointer.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : add TODO + clean-up
* mtmd: add audio out stream api
* add /tts endpoint
* add docs
* wire up mtmd_helper_model_can_chat
* upload speaker_ref via form-data
* expose seed to api
* fix header for pcm
* wip
* fix penalty_last_n
* fix server_output_limits
* fix pocket-tts stream
* wip
* add server_schema for tts
* add server routing
* server: coerce seed and stream from multipart form fields
Form fields arrive as strings, so -F seed=42 and -F stream=true were
rejected by the schema. seed joins the integer fields, parsed as a long
long so a full uint32 seed fits, and stream is read as a boolean when
it comes as a string.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server: add realtime transcription API
* drop transcription_sessions
* proxy ws
* add docs
* server: let the API key middleware guard the websocket
httplib runs the pre-routing handler before the websocket upgrade, so
the API key middleware answers 401 at handshake like any other route.
This covers the router proxy, whose children run keyless, and accepts
the same key forms as HTTP (Bearer, raw key, X-Api-Key).
* server: keep realtime slots out of the prompt loop
A realtime slot waiting for audio is GENERATING with nothing to decode,
so it fell through the prompt loop and took slot_batched. A request on
a higher slot then failed can_batch_with on the task type and starved
for the whole session. Realtime slots have no prompt: the loop sets
them up once and always leaves them.
* server: start the realtime session once its slot is live
The audio stream was created in the prompt loop, which skips a slot
that cannot batch with the one generating, and session.created went out
before the task was posted: early appends failed and the reader quit,
leaving the socket open and the slot held until the HTTP timeout.
The slot now sets up its stream at launch and signals it with the begin
result, session.created waits for it, and a reader that stops closes
the socket so the session ends with it.
* server: name the committed item in input_audio_buffer.committed
Per the Realtime spec, item_id is the item made from the committed audio
and previous_item_id the one committed before it, null for the first.
committed now comes before the transcript that completes that item, and
both go out with the item switch under one lock, so a delta cannot land
between them.
* server: use the GA transcription session shape
The session object follows the GA Realtime schema: type transcription,
with the input format, transcription model, turn detection and noise
reduction under audio.input. The rate stays the model sample rate, which
is what the client must send for now.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama : move LFM2 shortconv and FFN blocks into a shared LFM base
* Update src/models/lfm-base.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* return logprobs in spec path
* server: add regression test for logprobs with speculative decoding
Asserts accepted draft tokens report the target model's logprobs,
comparing a draft-enabled run against a no-draft baseline (#24271).
Fails on master (top_logprobs empty, logprob 0.0), passes with the
fix from #27196.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KARBWpPcijcBXzJEvttuwk
---------
Co-authored-by: Jeonghyun Lee <sig.jhl@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* mtmd : lfm2 tiles stretch the image to the grid
Lfm2VlImageProcessor.crop_image_to_patches resizes the image straight to
(512 * rows, 512 * cols) before it cuts the tiles. llama.cpp kept the aspect
ratio and added black bars for LFM2 (only d1-omni stretched), so the tiles of
d1-3B and every LFM2-VL / LFM2.5-VL model differed from transformers unless
the image had the aspect ratio of the grid. With PAD_NONE the tiles are the
same as in transformers.
* mtmd : lfm2 layout with Python rounding in double precision
Lfm2VlImageProcessor.smart_resize rounds with Python's round (half to even)
and computes beta and the tile grid aspect ratio in double. llama.cpp used
std::round and float, so 31926 of the 16.7M sizes in [1,4096]^2 got another
thumbnail size or grid than transformers (and d1-omni vision.py), e.g.
400x600 -> 416x608 instead of 384x608 and 720x1600 -> 2x5 tiles instead of
2x4. The lfm2 smart resize and grid choice now follow the Python code and
match it on every size of that range.
* fix
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add H2O-Lightning-4B decision model
Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b
(Qwen3.5-4B with a merged decision LoRA).
None of the existing types match this model. It answers with " A"
rather than "A", uses up to 255 labels, and was trained on Python
json.dumps text, so floats like 12.0 must render as 12.0.
The converter writes the decision metadata and the systemone template
from the model's serve_config.json and refuses a config that doesn't
match. Tests skip unless the tiny test model is available locally.
Assisted-by: Claude Opus 5.5
* rm redundant code
* put labels to gguf
---------
Co-authored-by: Jonathan C. McKinney <jon.mckinney@h2o.ai>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation
* tests: satisfy router disconnect type check
* server : remove timing-sensitive router cancellation test
Assisted-by: Codex
* server: include output_index in Responses API SSE events
* server: reuse Anthropic streaming code for fix output_index
* server: remove unnecessary generated_tool_call_ids and oai_resp_tool_calls_started
* mtmd : read LFM2 tiling params from GGUF metadata
Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from
processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp
with the previous constants as defaults. do_image_splitting=false maps to
min_tiles=max_tiles=1 (no tiling).
Rebased version of #25524, original implementation by Michael DM (@michael-dm).
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* fix test
---------
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* metal : optimize argmax and pool_1d
Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : add pool_1d perf cases
Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : minor fix
* model : drop the redundant precision overrides in the Kolibri-1 router
Follow-up to the review in #30673: the router matmul works with the
default precision, so both ggml_prec_set_* calls can go away.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* cont : fix comment
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* opencl: fix OOB read in `mul_mv_mxfp4_f32_flat`
* opencl: remove large-N guard that forces CPU weight placement
* this condition claims that large batch produces incorrect results on
adreno and cannot be reproduced
* model loader uses N512 to probe op support, resulting in cpu fallback
* thus this condition does not seem necessary
* opencl: cap num of subgroups by workgroup size for q4_0 mc3
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add small batch variant for q4_k dp4a GEMM
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a narrow variant of mul_mm_f16_f32_l4_lm
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a one subgroup variant for gemv_noshuffle_q1_0_f32 for big M
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
---------
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
The __ARM_NEON branch defines both GGML_CPU_COMPUTE_* macros but selects
only FP16_TO_FP32, so FP32_TO_FP16 falls through to the portable
software conversion while neon_compute_fp32_to_fp16, a single fcvt,
stays defined and referenced by nothing. __POWER9_VECTOR__ and
__riscv_zfhmin select both.
Add an #error pair that refuses a branch defining a COMPUTE macro
without selecting it. That is how this stayed unnoticed: the fallback
silently supplies a correct software implementation, so the only
symptom is a hardware instruction nothing ever reaches. The
FP16_TO_FP32 half carries a !defined(__F16C__) carve-out, because PR
30343 keeps that direction on the lookup table on purpose.
Replace the comment above the fallback: it singled out NEON and POWER9
and pointed at a selection that now happens in the branch itself.
* spec : save the dynamic ngram cache file
* save the cache at the common_speculative_impl_ngram_cache destruction
* remove save_static/save_dynamic flags as well as the relevant TODO
* if the dynamic cache file does not exist, start with an empty cache
that is written on exit. if it exists but cannot be read, abort
instead of overwriting it.
* spec : fix draft_one not updating the ngram cache history correctly
* common_ngram_cache_update received only the new tokens with none of
the tokens before them causing ngrams ending at the new token to not
being formed, leading the context cache to only ever hold the prompt
* fix: pass the full history to common_ngram_cache_update with nnew set
to the number of new tokens the way llama-lookup does and also reset
cache_size to 0 on each new request
---------
Co-authored-by: Petros Sideris <petros.sideris@nokia.com>
* server : pass id_slot through the Anthropic /v1/messages conversion
server_chat_convert_anthropic_to_oai copies a small whitelist of common
params from the Anthropic request into the internal OAI-style body and
left id_slot out. Every other completion endpoint honours id_slot; a
proxy that pins a session to the slot holding its KV cache needs the
same on /v1/messages, or it silently falls back to automatic slot
selection and, on hybrid/recurrent models, pays a full re-prefill.
Adds a test asserting a request with id_slot set is served by that
slot, observed via GET /slots.
* server : shorten comments
* rm test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* tests : make MoE router selection independent of the backend noise
The router scores are in the range [0, 1), so a random draw can leave the
last expert (or expert group) still selected and the first one dropped tied
within the noise between the backends. They then route a token to different
experts and the logits differ by 4e-3 instead of the usual 5e-7.
Beam is the only arch here with a real top-k (8 experts, 2 used) and a group
top-k (4 groups, 2 used), so it is the only one that hits this: 2 of 20 seeds
failed on an M2 Ultra, the other MoE archs select 2 of 2 and cannot tie.
Fill the selection bias with a ramp of step 2 to give every expert a score
range of its own, which turns the top-k into a function of the expert index
alone. The bias affects only the selection, the weights still come from the
unbiased probs, so the MoE output keeps its magnitude. Beam now matches the
CPU reference within 4e-7 for all of 160 seeds.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : tell the CPU and the GPU of an Apple SoC apart in the device table
Both devices report the same description ("Apple M2 Ultra" here), so the
device column of the test table had two indistinguishable rows and a failing
row did not say whether the Metal device or the host CPU produced it.
Add the device name, which is also what -b takes, unless the description
already is the name.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* hex-build: bump toolchain, enable v85
* hex-build: bump toolchain to v0.8 in the CI and build scripts
* hex-build: add a little python script for swiv needed on some iot devices
* hex-build: fix flake8 warnings
* hex-arch: update version cap to v85
* hex-inspect: flexible TOOLS_ROOT
* hex-build: update to snake_case naming
* hex-build: update leftover old toolchain refs
* hex-target: update build and run scripts to support ubuntu with adb transport
SearchInput already renders its own clear button.
Hide the browser's native search cancel button to avoid showing both controls.
Fixes#28627
Signed-off-by: aiqubits <aiqubits@hotmail.com>
* init
* wip
* wire it up
* make it clear that it's experimental
* add props.connect_code
* nits
* fix windows build
* server: do not advertise a connect code without the tunnel
--connect-code on its own had no effect other than being served in
/props, so the Web UI showed a remote access code that no tunnel was
serving. Clear it when --connect is off, and say so once at startup.
The teardown comment claimed that terminate() closes the child stdout.
It does not: the log thread leaves fgets() only once every writer of
that pipe is gone, so a descendant of llama-connect would block the
join forever. Describe the real contract instead.
Also restore the --reasoning-preserve default in the generated docs.
* Update tools/connect/CMakeLists.txt
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
* server: dial llama-connect on the bound port and a reachable host
Pass ctx_http.port so --port 0 and the GCP override work, pick the
dial host from --host (wildcards map to loopback, IPv6 is bracketed)
and refuse --connect when only unix sockets are bound.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
In the attachment preview, SyntaxHighlightedCode's wrapper (overflow-auto) was not the element that ended up owning horizontal overflow: the inner code box kept its content clipped at the wrapper's width, so a single line longer than the preview pane was cut off with no scrollbar (scrollWidth == clientWidth while the code's own content measured tens of thousands of px).
Give the pre/code boxes their intrinsic width (width: max-content, min-width: 100%) so long-line overflow propagates to the scrolling wrapper. Verified in the running WebUI: with a 6000-char line, wrapper scrollWidth goes from 990 (clipped, unscrollable) to 50449 with a working horizontal scrollbar; short content still fills the container exactly as before (min-width: 100%).
Fixes#30796
The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a
single dispatch, reading experts and weights while writing dst. The graph
allocator computes tensor liveness on the unfused graph, where the inputs are
fully consumed by earlier nodes of the fused group, so it can legally recycle
the input memory as dst. In the fused kernel this becomes a
threadgroup-vs-threadgroup race that produces rare non-deterministic wrong
results (observed with test-llama-archs -a kolibri on the meta backend).
Reject the fusion when dst overlaps either input, same guard as the
mul_mat_add fusion. The unfused MUL + ADD path is always correct.
Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
* cuda: cap Stream-K blocks at the number of k-blocks
Stream-K normally launches one CUDA block per SM. For small matrices,
this can exceed the total number of output-tile/k-block pairs, leaving
blocks with no work and adding launch and fixup overhead. Cap the grid
at that count to remove these excess blocks without changing the useful
rounded K ranges or numerical results.
Assisted-by: Codex
* cuda: cap Stream-K blocks at the number of k iterations
Cap the grid at the number of K_vram iterations across all output tiles, rounding up partial iterations. This removes extra CUDA blocks left by the k-block cap.
Partial iterations can change FP32 summation grouping. MUL_MAT and MUL_MAT_ID backend tests pass on RTX 5090.
* Update ggml/src/ggml-cuda/mmq.cuh
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ui : keep markdown tables inside the chat column on phones
The phone rule made the table wrapper full-bleed with -1rem margins, so the
first and the last column sat under the screen edge. Keep the wrapper inside the
chat column and cap it at its width, so a wide table scrolls in place instead.
Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* test : show a table-only model answer in the chat message story
Adds a story with an answer that is mostly a table, so the table layout can be
checked at a phone width.
Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* ui : indent markdown lists by 2rem
1.5rem put the marker too close to the item text.
Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* The problem here in the repo is gguf-split can silently produce corrupted tensor output when the input GGUF file is truncated. The fix is to add tensor-data bounds checks and short-read checks in gguf-split.cpp and export-lora.cpp, and a regression test that checks truncated input is rejected and valid tensor data is preserved.#
* Bug : GGUF-split can silently produce corrupted tensor output when the input GGUF file is truncated. Fix : add short-read checks in gguf-split.cpp (in copy_file_to_file in split_strategy) and export-lora.cpp (in read_tensor_data of file_input) so that truncated input is rejected.
* tests: add beam fusion rows to MTL baseline
Regenerate tests/fusion/MTL.csv with test-fusion --record. The newly
added beam architecture has no baseline rows, so test-fusion fails with
2 failing rows (MUL+ADD x6, RMS_NORM+MUL x25). All other arch counts are
unchanged.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* skills: add update-fusion-table skill
Add a skill for regenerating a per-device test-fusion baseline. It keeps
the workflow from tests/fusion/README.md and the parts that are easy to
get wrong: diff the recorded file before applying it, the distinct
meanings of a FAIL row, and never recording a partial run.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
.pi/gg/SYSTEM.md repeated the full agent ruleset. Move it to .pi/gg/APPEND_SYSTEM.md and keep only the llama.cpp-specific rules, so it can be appended to a main system prompt with --append-system-prompt.
Drop the .pi/SYSTEM.md symlink setup from wc2wt.sh and pr2wt.sh. Worktrees no longer need it now that the folder holds an addendum instead of a full prompt.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* CUDA: fix KQ write-after-read race in flash_attn_ext_vec
Assisted-by: ChatGPT
* CUDA: apply review suggestions for KQ sync comments
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* hex-fa: support for masked block skipping to reduce DMA fetching
* hex-fa: finetuning the hmx->hvx crossover thresholds (boosting gemma-4)
* hex-fa: update hmx gate to account for mdev.count
Port of the Vulkan HC_POST_GATE fusion (#29520). hc_combine builds the
hc_post weights as 2*sigmoid(inject/hc); dsv4_hc_post_f32 now takes a
gated template flag and computes scale_out*sigmoid(scale_in*p) from the
raw weights, so the chain runs as one kernel instead of four.
* ggml: support non-contiguous SUM (CPU, CUDA)
* tests: add non-contiguous SUM cases
* tests: SUM over non-contiguous views
* vulkan: support non-contiguous SUM
* webgpu: support non-contiguous SUM
* hex-hadamard: add support for hadamard ops
* hex-mm: reduce pkt counts in the q8_1 quantizer and vec_dot_q1
* hex-q1/q2: add support for flat and tiled GET_ROWS for Q1_0 and Q2_0
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* common : resolve <quant>-<sidecar> download tags and list cached sidecars
A Q4_0-mtp style tag now resolves the sidecar file when no model file matches it, so a solo draft or mmproj download actually pulls the file. Cached sidecar files list as their own entries so the state survives a restart, and removing such a tag deletes only the sidecar.
Assisted-by: pi:zai-org/GLM-5.3
* ui : optional sanitized raw HTML in markdown
Add an allowHtml prop to MarkdownContent: raw HTML found in the markdown is
rendered after DOMPurify sanitization instead of being escaped as literal
text. Default stays escaped.
Assisted-by: pi:GLM-5.3-Flash
* ui : add the discover models components
The Discover surface: the curated-catalog list with its search and
skeletons, the details pane with its readme, metadata, Hub stats and
download options, and the standalone download progress bar that replaces
the plain one. The quant download button sizes its select with a new xs
trigger, so the select gains that size and the muted box look. The store
stays incomplete when a repo fails, so the next mount retries it.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : put the discover view in the models manager
The manager dialog gains its second view: the Discover Models call to
action fades the table out, slides the title with its compass mark, and
fades the discover surface in; the arrow button returns to the table.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
# Conflicts:
# tools/ui/src/lib/components/ui/button/button.svelte
* ui : add the Enable Discover Models setting
The flag gates browsing and downloading HuggingFace GGUF models, so it is
declared on the discover layer; the repo-tree lookup in the manager layer
above reads it from here.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : resume model downloads from the pause point
A stopped download keeps its partial file instead of deleting it, so
re-posting the tag resumes from disk. The first progress record now
carries the resume offset, the paused snapshot survives into the
resumed entry, and a pause settles as soon as it is requested.
Assisted-by: pi
* ui : lead the discover list with downloads
The discover sidebar lists running downloads above the suggested models,
and steps aside while a query is active. A paused download stays in the
manager table, where it is resumed or dropped.
Assisted-by: pi
* ui : make resuming a paused download work
Re-posting the tag of a paused download hit the router's "already
exists" check, because a paused download parks its entry as DOWNLOADED
until the next reload: a download entry that is not running is now
replaced and resumed. The UI keeps the paused row when the request still
fails, instead of dropping it and looking finished.
Assisted-by: pi
* ui : remember unfinished model downloads
A partial file is named after its content hash, so nothing on disk says
which model it belongs to once the router restarts. The router now keeps
the tags of the downloads it started, clears a tag when the download
finishes or is deleted, and lists what is left as still downloading
without starting anything. The UI adopts those tags as paused on every
list fetch, so a refresh or a fresh open shows an incomplete download and
can resume it from the files already on disk.
Assisted-by: pi
* ui : show a paused download's state on its chip
A paused quant chip reads as paused at rest, the way a running one shows
its spinner, and offers the resume action on hover. The selector's
download row gets a View in Discover action beside its cancel.
Assisted-by: pi
* ui : collapse the download options by default
The copyable download command starts collapsed so the option rows
lead; expanding it stays one click away.
Assisted-by: pi
* ui : keep decision models out of the hub lists
Decision models answer with calibrated probabilities instead of text,
so llama-ui cannot run them: the hub data layer drops them from the
catalog and every search, keyed by the decision-model tag or a
text-classification pipeline.
Assisted-by: pi
* ui : read download tags as quant and sidecar
A `<repo>:<quant>-<sidecar>` tag now parses into its quant and sidecar
instead of one opaque quant token, and ModelsService.draftSidecarsByRepo
collects the router listing's sidecar-only entries per repo, replacing
the inline copy in the models store.
Assisted-by: pi
* ui : cluster draft downloads under their main model
Tracked downloads group by repo: the main weights lead and the draft
sidecars ride inside their item as compact sub-lines, on the selector,
the discover list and the manager table. The main model's controls
pause, resume and cancel the whole cluster, and the manager's download
rows select into the configuration pane.
Assisted-by: pi
* ui : act on tracked downloads from the table rows
The table passes its clusters' draft sub-lines to the model rows. A
row's actions cascade pause and resume, an in-flight row offers Cancel
download while a paused one keeps Delete from disk, and the confirm
dialog's copy follows the state.
Assisted-by: pi
* ui : give downloads a configuration pane of their own
Selecting a tracked download opens the pane in a download mode: live
progress with pause, resume and cancel, and See in Discover, instead of
metadata and load actions that a model without files cannot offer.
Assisted-by: pi
* ui : make the discover details self-fetching and responsive
ModelsDiscoverDetailsContainer owns the details, tree and readme
fetches so hosts outside the discover view can render the pane, and the
details layout adapts to narrow panes with container queries instead of
viewport breakpoints.
Assisted-by: pi
* ui : download a matching draft with the main model
A main quant download now pulls the quant-matched draft of the most
preferred kind the repo ships (DFlash before MTP), fire-and-forget, so
speculative decoding is ready when the main lands. Manual draft chips
stay available; the options row also collapses to one column in narrow
panes.
Assisted-by: pi
* ui : keep discover off by default and couple it with the hub setting
Discover Models starts unchecked. Enabling it turns the Hugging Face
Hub setting on and disabling it turns it back off, so the checkbox
never lies; with discover off the hub setting stays available on its
own for the display data. Stored and reset state are corrected the same
way on load.
Assisted-by: pi
* ui : open the manage models dialog on the manager view
The dialog now opens on the manager instead of returning to the last
view, discover entries still land on the discover view while the
setting allows it, and the discover surfaces follow the setting's
state.
Assisted-by: pi
* ui : give the discover search room for its focus ring
The sticky search bar insets from the panel edges so the focus ring
stays inside the scroll container, and rows scroll behind an opaque bar
instead of through it.
Assisted-by: pi
* common: settle sidecar names on the listing and drop unfinished partials
An infix or trailing sidecar token is ambiguous: draft heads use it
(Model-MTP-BF16.gguf), so do full models with embedded heads
(Model-Q8D-MTP.gguf). Such a name is a sidecar only next to token-free
main weights of the same family, while mmproj or imatrix anywhere in the
name, or a leading token, decides alone (mmproj_f16.gguf).
A quant tag always carries its bit count, so a sidecar named after no
quant (mtp-Model-it.gguf) gets the bare tag: the listing, the plan and
remove all use it. An imatrix tag resolves into a plan slot of its own,
downloaded but never wired into the params.
remove resolves the tag the way a download does and deletes the partial
files of every planned blob, so cancelling a running download or
dropping one left unfinished by an earlier run leaves nothing on disk.
The router passes its token and releases its lock around the call, and
records the unfinished downloads with an atomic write.
llama download succeeds on a sidecar-only plan and prints its paths.
* ui : move dialog views and draft states into enums and constants
The models dialog's views become the ModelsDialogView enum, the draft
subline's states join ModelRowDownloadState, the view fade and the
copy-feedback reset move to ui constants, and the dialog's compass
marks come from DISCOVER_ICON.
Assisted-by: pi
* ui : drop the unused discover surface
The dead openDiscoverModels twin, the commented-out command divider,
the details pane's unused onClose and its header's close button, and a
redundant type cast leave; the command's copy feedback joins the shared
reset constant.
Assisted-by: pi
* ui : trim the download option comments and highlight the chat template
The comment verbosity the options panel and its chips carried shrinks
to the non-obvious why, and the chat template dialog renders through
SyntaxHighlightedCode as jinja instead of a plain pre block.
Assisted-by: pi
* common: only a .gguf file is a sidecar, and mmproj/imatrix as whole words
A non-GGUF file named like a sidecar (quantization-mmproj-Q8_0.json) is
picked as the mmproj, and a sidecar word inside a longer word of a model
name (Phi-2-OmniMatrix.i1-Q4_K_M.gguf) drops the model.
* ui: dedupe draft sidecar badges per quant
The draft sidecar badges deduped by kind and repo without the quant, so
with a draft cached at two quants only one badge survived, and a delete
of the other main quant found no badge to remove its sidecar with.
Also restores the getDownloadEntries doc lost when the clusters method
landed above it.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: adopt the incomplete download status and drop the paused-ids registry
The store inferred an unfinished download from a downloading entry it
held no local state for, which misread a download running in another
tab as paused and offered a resume the server rejects. The listing
carries the "incomplete" status now, so the adoption keys off it
directly.
The paused ids no longer need to survive in localStorage: the server
remembers them across restarts, and the progress snapshot was stale
after a reload anyway. The stale localStorage key is removed.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: quant-match the pause and cancel cascades
Pause, resume and cancel cascaded to every draft sidecar of the repo
while delete only took the quant-matched ones, so canceling a Q4_0
download also killed a Q8_0 draft that belongs to another quant's
main. Every cascade acts on the quant-matched drafts now, matching
what a delete takes with it.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: collapse the duplicated draft subline markup
The downloaded and in-flight branches repeated the identical connector,
label and badge markup; one block with a conditional tail keeps the two
states in sync.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: scrub positioned styles out of sanitized model card html
The style attribute stays allowed for the KaTeX output, which let a
third party card draw a fixed overlay over the app. The sanitize hook
drops position:fixed and position:absolute values and leaves the KaTeX
layout styles alone.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: run the discover mount fetch once and reuse the shared repoOf helper
fetch() toggles its loading flag synchronously inside the mount effect,
which re-ran the fetch on every toggle and retried failed catalog repos
endlessly; the effect now untracks the call. The local repoOf copy and
the duplicate services import give way to the shared helpers.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: narrow the spec-type map to the draft sidecars
SPEC_TYPE carried empty entries for the aux sidecars it is never
indexed with; the map and the command component now type the badge as
a draft sidecar through a draftBadgeOf helper instead of a cast.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: name the decision-model tags and markdown cache keys
The decision-model hub tag and pipeline tag move into the huggingface
constants next to the other pipeline tags, and the markdown transform
cache prefixes get names instead of bare one-letter tokens.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: read the discover setting through one truthiness check
Three places read the setting with !== false and one with Boolean();
all four read it the same way now.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: compare the models dialog view against its enum
The template compared the view state against the 'manage' literal in
three spots while the script used the enum.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* nit: Remove `+` before draft sidecar badge
* server: list unfinished downloads as incomplete
A download that never finished was reported as "downloading", the same
status a live download carries, so a client could not tell a running
download from a dead one. The tags remembered in unfinished-downloads
now list under a dedicated "incomplete" status, without an instance.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* server: download a model with its sidecars as -hf does
Resolution goes back to master. POST /models takes the sidecar kinds to
pull with the model (mmproj by default) and the download child gets them
as -hf flags, so the server picks the files. /models lists real models
only, each with the sidecars it has on disk, and incomplete downloads
keep their sidecars across restarts. A stopped download keeps its
partial files for the next attempt, and DELETE purges them through the
same plan the download follows.
* ui: download sidecars as parameters of their model
The download options keep their chips, but a chip no longer posts a
sidecar tag: the store maps it to its model and the sidecar kinds to
pull, the quant chip takes its companion draft in the same download,
and draft badges come from the sidecars each cached model lists.
The server adds the sidecars a cached model lacks when it is posted
again, and DELETE removes a single sidecar of a cached model.
* ui : mirror the server's sidecar lookup in draft badges
The server resolves a sidecar by the case-sensitive `<kind>-` token in the same
directory as the weights. The UI matched the token loosely, so a badge could
offer a draft the server never pulls.
Assisted-by: DeepSeek-V4.1-Flash
* ui : post the model's sidecar kinds with its download
The store keeps one download per model: the chip's kinds ride along with the
model's own sidecars, and a download row no longer expands into one row per
sidecar. Draft badges carry no quant of their own yet, so a sidecar of a cached
model is never labelled with the model's quant.
Assisted-by: DeepSeek-V4.1-Flash
* ui : join a quant's draft into its download chip
One chip per quant, its draft badged inside it and its size covering both
files. The draft is the file the server would pick: the same directory, the
exact quant, else the closest bit depth, so a 4-bit model can pair with a
16-bit head.
Assisted-by: DeepSeek-V4.1-Flash
* ui : drop the draft download rows from the manager and selector
A download is one row per model again: the drafts ride with it, and the rows
read the sidecars on disk off the listing instead of pairing sidecar entries
with their model. Pause, resume and cancel act on the model's own download.
Assisted-by: DeepSeek-V4.1-Flash
* ui : keep the discover download list one row per model
The list leads with one row per downloading model, its sidecars as sub-lines.
A row badges only the draft kinds the server can resolve, so a badge never
outlives the file its lookup would pick.
Assisted-by: DeepSeek-V4.1-Flash
* ui : keep the discover chips readable in dark mode
The metadata chips, the chat template button and the View on Hugging Face link
carried a faint border and no surface over the dialog background, so they were
near invisible. They take the tertiary button surface now, and their icons size
with size-* so the variant's base rule leaves them alone.
Assisted-by: DeepSeek-V4.1-Flash
* ui : carry the draft through the download options
One chip per main quant, its draft badged on it and the chip's downloaded state
covering both files. The drafts sit behind an opt-in checkbox with an inline
type pick, and each row opens its own command: it pins the row's quant and only
names the draft type, so -hfd is gone.
Assisted-by: DeepSeek-V4.1-Flash
* ui : line the details skeleton up with the loaded pane
Same root padding and container context as the loaded view, so the container
queries resolve alike. The download options card pads its blocks instead of
itself, the readme gets its card, and a description line stands in for the
catalog paragraph.
Assisted-by: DeepSeek-V4.1-Flash
* common : drop the partials of a removed sidecar
Removing a sidecar alone left the .downloadInProgress files of the sidecars a
download had planned in blobs/. The removal purges them through the same plan
the download follows, and the test covers a finished draft beside an unfinished
one, whose partial goes while the model keeps its files.
Assisted-by: DeepSeek-V4.1-Flash
* server : name why a download is refused
A model that is already downloading, or that is running, answered "already
exists". It now says which: "is already downloading", or "is running, unload it
first", both before the resumable check. The router test covers the two.
Assisted-by: DeepSeek-V4.1-Flash
* ui : post a download once per click
Two quick clicks on the same chip posted twice, and the second answer was
"already exists". The store tracks the model as pending, returns early while it
is pending or in flight, and reads it as in flight before the server answers, so
the chip shows the download at once. The new client test pins the flows: one
POST per double click, the chip reading downloaded once the list reports the
model, a resume keeping its sidecars, a cancel dropping the whole download, and
a reload restoring the paused state.
Assisted-by: DeepSeek-V4.1-Flash
* ui : keep a plain download row the height of the others
The sub-lines wrapper of a download row was rendered even with nothing in it, so
its padding made the row taller than a plain one and the overlaid progress bar
sat in that empty strip. It renders only with sub-lines now, in the selector and
in the discover list, and the selector row takes the phone padding its
neighbours have.
Assisted-by: DeepSeek-V4.1-Flash
* ui : badge a repo's draft once
The args of a loaded model and the listing of a cached one report the same draft
in different words, one with the file's quant and one with the kind alone, so
the merge kept both and a repo row showed the same kind twice. The badges merge
per kind now, the args one winning as it knows the quant, and the test pins both
sources of a real repo.
Assisted-by: DeepSeek-V4.1-Flash
* server : report the sidecars of a preset or a download on its way
A model the preset also names is registered as a preset, so its drafts were
never reported and no row could badge them. A download on its way reported none
either, though the record of the kinds it pulls is right there: those win over
the plan now, the plan resolves through the preset's hf-repo, and a models dir
stays out of it.
Assisted-by: DeepSeek-V4.1-Flash
* ui : badge a download's drafts and open its model from the row
A draft still coming down badges the model id now, the way a landed one does,
while an on-disk draft keeps its checked sub-line. The row itself opens the
model, the status slot beside it pauses or resumes, and the manager resolves a
focus on a download row so its panel opens.
Assisted-by: DeepSeek-V4.1-Flash
* common : report a removed sidecar that only had a partial
Removing a sidecar alone dropped the result of the partial purge, so a sidecar
that only ever landed as a .downloadInProgress blob answered false and the UI
had no reason to refetch. The purge result is folded into the return now, and
the test covers a partial-only removal.
Assisted-by: DeepSeek-V4.1-Flash
* ui : ignore a pause while the download POST is still on its way
The chip reads as downloading before the server answers, so a quick second click
unloaded a model that had not started and left the store neither in flight nor
paused. A pause returns early while the POST is pending, before it touches the
feed or the paused marker. The client test holds the POST open and pins it.
Assisted-by: DeepSeek-V4.1-Flash
* ui : offer a missing draft as its own chip
A model already on disk without its draft had no way to add it: the quant chip
carried the draft, so it read as a download of both. The quant chip keeps to its
own state now, its joined draft shows only while it rides along or is on disk,
and a downloaded quant missing its draft gets a chip of its own - badge, size,
download - that posts the model with that kind, so the server keeps the files
already there and pulls the draft.
Assisted-by: DeepSeek-V4.1-Flash
* server : list a paused sidecar download by the files on disk
A model on disk whose sidecar download stops keeps its unfinished record,
and the listing reported the kinds that record plans as if they were on
disk: the UI checked a draft that never landed and offered no way to add it.
The planned kinds now stand in only while the download runs.
* chore: Clean up comments
* server : list sidecars only for a model that resolves from a HF repo
A preset on a local file has no HF repo, yet the listing resolved its name
as one: the plan threw on the invalid repo format and GET /models failed
for every model. The sidecars now come from the repo a preset names or a
cached model is listed under, and a local file lists without them.
* common : fix download cancel regression
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* common : preserve model partials when removing sidecars
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* server : one sidecar table and an unfinished downloads class
Address review from @ggerganov.
The sidecar kinds live in one table in common/download, mapping each name to
its download option and plan file; the server validates, builds its options
and lists cached sidecars through it instead of its own lists.
The unfinished downloads move into server_unfinished_downloads, a record with
its own mutex and a struct per entry, and server_models exposes them through
unfinished_downloads_get and unfinished_downloads_remove.
* common: one sidecar table, and removals that keep shared files
The resolution, the cached listing and the removal follow a single
table of sidecar kinds, so a kind is declared once with its file
keyword, whether it is a draft, its option and its plan file. The
server reads the same table for the kinds it accepts, the default
sidecars and the child flags.
Removing a tag keeps the files another cached tag still loads, such as
an mmproj named after the removed quant. A sidecar removed on its own
goes for every tag that resolves it.
An unfinished download records the exact paths of its partials when
it starts. A removal deletes those without asking the hub, and keeps a
partial another recorded download also writes. The record is taken
once the child is spawned, and the files go under the models lock, so
a download of the same tag cannot start while they are removed.
* server: drop stale partials and refuse a tag that resolves nothing
Recording a download again deletes the partials of the earlier record
that the new plan no longer writes, such as those of an older hub
revision or of a sidecar the new request leaves out.
A posted tag that resolves no file of the repo is refused before the
download starts. Removing a model follows a single path, whether the
reload erased its entry during the wait or not.
* server: fix the CI race between the router download tests
The pytest workers run in parallel on one shared cache, so a download
test could find the file another test had just fetched and see no
progress, or start from the unfinished downloads record another test
wrote. Each download test now runs on an empty cache of its own.
* server: refuse a DELETE of an unknown name and decide downloads in load()
A DELETE of a name that is neither a model nor a recorded download
went to common_download_remove, and without a tag it wiped the whole
repo cache, including the files of a loaded quant or of a download on
its way. Such a name is now not found, as it was before.
A POST that downloads over an existing entry checked its state outside
the lock, so a load of the same model in between got its instance
killed. The checks now run in load(), under the lock, right before the
spawn. The fake load time of the tests applies to model loads only,
since a download worker loads none.
* server: add incomplete_sidecars to the /models API
A sidecar added to a cached model and then paused kept its record, but
the listing showed only the complete model, so a fresh UI lost the
pause. The model entry now carries incomplete_sidecars, the kinds its
unfinished download has not written yet; a POST with the kind resumes
one and a DELETE with it drops its partial.
* common : drop a kept partial when a resume gets 416
Now that partials survive between attempts, an old or complete
partial makes every retry fail with the same 416.
Delete it so the next attempt can start over from zero.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* common: drop a kept partial when a resume gets 200
A server that ignores the range answers 200 to a resume, and the kept
partial then fails every attempt the same way a 416 does. Drop it as
well, so the next attempt starts over without a range.
* server: refuse a download that shares a partial with one on its way
Two downloads of quants that share a file, the mmproj or an mtp picked
by the closest bits, appended to the same partial and renamed a file
with interleaved bytes. A download whose plan writes a partial that a
running download also writes is now refused until that one is done.
* common : check the saved etag before resuming a partial download
Before resuming, compare the etag saved with the partial to the one
the server now serves. If it is missing or different, start over,
so a resumed download never mixes bytes from two versions of the file.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
* tests: give the random model generator a per-tensor filler table
Muse Spark makes test-llama-context fail on state_load, cp_h and cp_d on about
six percent of the random seeds, with an NMSE of 1.0e-05 to 1.4e-05 against a
threshold of 1.0e-05. The sliding window is what exposes it: a restored state
keeps the window alone, so the attention of an SWA layer reduces over a
different cell layout and the continuation is not bit exact. Every SWA model
carries that, at 1e-08. Muse Spark carried it at 1e-05.
The generator is what turns 1e-08 into 1e-05. A norm preserving residual gate
holds alpha^2 + beta^2 == 1 and the conversion resolves both weights from one
raw weight, while the generator draws every tensor on its own. The pair then
scaled the stream and the branch independently, the stream lost an arbitrary
factor at each layer, and the model became an amplifier. The weights the test
ran on were not a Muse Spark.
Per-tensor rules lived inside the write path, where a flag picked the values of
a tensor inside each of the two branches that store them, so a new rule meant a
new flag threaded through both. The file asks to be rid of that in a TODO of its
own. Give each rule a filler, pick one from a table keyed on a substring of the
tensor name, and keep a single write path behind it. Mamba moves into the table
and Muse Spark joins it: the gate filler draws the raw weight once per block and
resolves alpha and beta the way the conversion does.
Over a thousand seeds on a CPU build the worst case drops from 1.36e-05 to
2.48e-06 and the median from 4.10e-06 to 5.48e-07, with no seed above the
threshold. The other 131 models come out identical to the bit, since no other
architecture carries these tensors. ctest is green on a CPU build and on a CUDA
build.
* tests: draw every tensor filler from fill_normal
The fillers repeated the generator and the distribution each time. The
normal draw now takes its key and the other fillers transform its
values, so the generator lives in one place. The generated models are
identical to the bit.
* fuse grouped experts
* add tests
* hip fix
* add to graph_optimize
* apply suggestion from am17an
* hip fix
* fix hip fix
* fix fix fix
* gate on hip version instead
* model : add Kolibri-1 (Aleph-Alpha) support
Kolibri-1 is a bilingual German/English MoE reasoning model with 78B
parameters and 3.46B active per token: 50 layers, 384 routed experts plus
one shared expert.
Attention interleaves four sliding layers at a 513 window with one full
layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE.
GQA runs 48 query heads against 4 KV heads at head_dim 128 with a
QK-RMSNorm, and every block carries four RMSNorms, so the attention and the
MLP output are normalized before they rejoin the residual stream. Routing
picks the experts from logits + expert_bias and weights them with the
sigmoid of the unbiased logits, without renorm.
The converter reuses AfmoeModel for the per-expert checkpoint layout and
the expert selection bias, and writes the shared expert and the per layer
sliding window pattern. The released FP8 checkpoint needs no special code,
it goes through the generic weight_scale_inv path. Three tensor names are
added to the mapping (post_attn_norm, post_ffn_norm,
moe.router.expert_bias) and the 128k vocabulary registers the qwen2
pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA
and the sliding window pattern.
Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash)
* Update src/models/kolibri1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Add fusion records for kolibri1
* Update src/models/kolibri1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* feat(convert): Attempt to come up with 256 single-token, semantic option markers
Each switch model needs to identify a collection of single-token option
markers that can be mapped to their respective logits for the output
probability distribution. The simple upper/lower/numbers set only yields 66
unique markers, but the model can support up to 255. The challenge is that
which combinations of characters result in single-token markers is model-
specific, so we do a discovery exercise at conversion time by looking
through the specific model's vocab for single-token sequences of 1+ plain
letter strings (eg `A`, `AA`, `BB`, etc).
Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* docs: Note granite-switch with 256 option markers
Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Remove extra newline
Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(convert): Use the computed labels in the template so c++ side is source of truth
Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Use LEV case for GRANITE_SWITCH w/ computed labels
Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* docs: Consolidate g-s into 255 bucket
Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
softplus_f32 and tile_softplus_f32 ran a scalar logf(1.0f + expf(x)) loop.
Replace both with a shared hvx_softplus_f32_aa() helper built from the
backend's existing hvx_vec_exp_f32 / hvx_vec_log_f32 primitives, so no new
numerical approximation is introduced.
hvx_vec_exp_f32 clamps its input to (-88, 88) before exponentiating, so a
naive log(1 + exp(x)) composition would saturate to ~88.72 for large x
instead of returning x. The Q6_V_vmux_QVV select keeps the scalar
reference's x > 20 guard (ggml_compute_softplus_f32 in ggml-impl.h),
which is what makes the vector path match the CPU result.
Assisted-by: Claude
* feat(convert): Add decision.chat to support decision models that can also chat
Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(convert): Add conversion for Granite Switch as decision model
The template follows from the usage sample:
https://huggingface.co/ibm-granite/granite-switch-4.1-8b#guardian-and-decision-side-by-side
Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(common): Add support for chat+decision models in common
Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(common): Add granite switch decision model enum
Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(server): Plumb through decision_chat bool to support decision+chat models
Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(server): Add Granite Switch decision support
Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* docs: Update server README for granite switch decisions / chat+decisions
Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* Revert "feat(convert): Add decision.chat to support decision models that can also chat"
This reverts commit 952ef567647e53ac9beda7a1f79d6927240d068d.
* Revert "feat(common): Add support for chat+decision models in common"
This reverts commit 93ed782e3a8b2fd5968732c6525c7ad29ad4bea2.
* Revert "feat(server): Plumb through decision_chat bool to support decision+chat models"
This reverts commit 97e9af60ddaccbd82f1eb0a488adff43b16ee095.
* fix: If Granite Switch supports decisions, it always supports text too
Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* docs: Remove reference to decision.chat in server README
Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client
common_http_client() was ignoring standard proxy env vars, causing
downloads via -hf and similar flags to fail silently in environments
behind an HTTP proxy (corporate networks, air-gapped infra, etc.).
Checks HTTPS_PROXY/https_proxy for https:// URLs and HTTP_PROXY/http_proxy
for http:// URLs, in that order. If a proxy URL is set and parseable,
passes it to httplib's set_proxy() and set_proxy_basic_auth() as
appropriate. Malformed proxy URLs are silently ignored so existing
callers aren't affected.
Fixes#21694
* common: drop misleading comment about generic proxy fallback
The comment referenced a 'generic one' that doesn't exist in the code — both
HTTP_PROXY and HTTPS_PROXY are scheme-specific. Reviewer noted the mismatch.
Addresses review feedback from @ServeurpersoCom.
* common : warn when the proxy URL from the environment is malformed
* common : honor NO_PROXY when selecting the http proxy
* ci : retrigger checks (test-thread-safety flake)
* common : bypass the proxy for loopback destinations
llama cli talks to its spawned server on 127.0.0.1 through common_http_client, so with HTTP_PROXY set and no NO_PROXY the health polling went to the corporate proxy, which rejects loopback targets, and the wait loop never exits ("Loading model..." hangs forever).
A remote proxy can never reach the client machine's own loopback, so these destinations are now always contacted directly, before NO_PROXY is consulted: localhost, *.localhost, 127.0.0.0/8, ::1 and 0.0.0.0. Similar to Go's net/http. Proxy handling for downloads and remote --server-base endpoints is unchanged.
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
* docs: move proxy tip from README to server docs
per review: the top level README stays lean; the proxy howto now lives with the server docs under Usage
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
---------
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
Co-authored-by: texasich <texasich@users.noreply.github.com>
Co-authored-by: texasich <texasich@gmail.com>
Co-authored-by: marecabo <23156476+marecabo@users.noreply.github.com>
Use the instance's is_mistral_format flag when DeepseekV2Model reloads
configuration, allowing Mistral MoE checkpoints to read params.json.
Fixes#29899
Signed-off-by: aiqubits <aiqubits@hotmail.com>
* llama-eval : add sandboxed execution for code suites
Adds a lightweight replacement for the Docker containers usually used to run
generated code, so a benchmark can be executed without a daemon, an image or
root.
A cached venv is provisioned per suite and reused; candidate programs run in a
throwaway working directory as their own process group, under the strongest
isolation the host offers:
bwrap read-only root, tmpfs /tmp and $HOME, no network
unshare no network, via --map-current-user
rlimit cpu/memory/file-size/process caps only
Candidate code must not run as uid 0, which is why "unshare -r" is a
last-resort tier that warns: root bypasses file permission bits, so a test
asserting that a write to a chmod 0444 file fails would instead see it succeed.
Two details worth noting for the bwrap tier. Only paths hidden by the $HOME
tmpfs are bound back, because bwrap cannot mount onto a symlink and
version-manager alias directories usually are one. And the venv's base
interpreter tree is resolved by walking the symlink chain, since a
version-managed interpreter is commonly reached through an unversioned alias
that startup resolves through.
Suites can declare provisioning steps, which run once at build time while the
network is still up, so test time never needs it.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : add HumanEval and ClassEval code suites
Adds two execution-graded suites alongside the existing text-graded ones. The
model's reply is parsed for a python code block, combined with the task's own
unit tests, and run in the sandbox.
--dataset humaneval 164 function-level problems, scored pass@k
--dataset classeval 100 class-level tasks, scored at class level (feeding
pass@k) and method level, matching upstream
pass@k reuses the existing chunk mechanism: --n_cases at a multiple of the
dataset size gives that many samples per problem, aggregated with the unbiased
estimator from the Codex paper.
Program construction and result interpretation are dataset-owned hooks, because
the two suites differ: one HumanEval problem is a function judged by a single
assert, whereas one ClassEval task is a class judged by many independent test
methods, and reducing that to pass/fail would throw away partial credit. Suites
also declare their own package set, provisioning steps, interpreter and timeout.
The package set is derived from the dataset's own imports rather than a
requirements file. HumanEval turns out to need no third-party packages at all;
ClassEval needs thirteen, two of which must be declared because no import
mentions them -- lxml is selected by string in ClassEval_44, and scipy is a
gensim requirement.
Notes on ClassEval specifically:
* gensim publishes no wheel past cp313, so 3.13 is preferred; on 3.14+ the two
pure-Python APIs it uses are provided by a small shim instead.
* The three nltk corpora it needs are pre-seeded at venv build time. These are
the modern names -- the dataset asks for the since-renamed punkt and
averaged_perceptron_tagger.
* It defaults to a patched fork, ilintar/ClassEval, because three upstream
tests cannot be passed by anything: a hard-coded 2024 date checked against
datetime.now(), an assertion on the dataset author's own hostname, and an
exact float comparison. Reference solutions go from 96/100 to 100/100 classes
and 2182/2196 to 2196/2196 test methods. No skeleton is touched, so what the
model is asked to write is unchanged; --dataset-source selects the original.
--exec-seed (default 0) seeds the RNG before each test, so suites whose tests
draw random inputs score reproducibly. ClassEval_58 fails its own reference
solution about 30% of the time without it.
The simulator gains both suites, so the harness can be exercised without a
model.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : fix silent isolation downgrade, generalise the sandbox runner
The isolation probe ran the interpreter without binding it back, so whenever
this tool is itself started from a venv under $HOME -- the usual case, since it
needs `datasets` -- bwrap's tmpfs hid the interpreter, the probe failed, and the
sandbox quietly fell back to a tier that does not hide $HOME at all. Nothing in
the output looked wrong when that happened, and the existing suites have been
running that way. The probe now uses the same interpreter and the same binds the
suite will actually get.
Two additions the agentic suite needs, neither of which changes the existing
suites:
* run_argv() runs an arbitrary command in a caller-owned directory that survives
the call. run() is now a thin wrapper over it, so HumanEval and ClassEval take
exactly the same path as before. A multi-step agent needs its edits to persist
across many invocations, and a suite for a language other than Python needs to
drive a linter and a test runner rather than a single source string.
* RLIMIT_AS is now per-sandbox, defaulting to what it was. It caps reserved
address space rather than resident memory, which is the wrong instrument for a
runtime that reserves a large region and commits little of it: node's
TypeScript type-stripping goes through WebAssembly, reserves gigabytes per
instance, and aborts at 4 GiB with an out-of-memory error that reads like a
fault in the code under test.
HumanEval stays at 164/164 and ClassEval at 100/100.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : add SACB, a multi-turn agentic coding suite
`--dataset agentic` is a different shape from the other code suites. Rather than
answering a single prompt, the model is given a small but real repository, a
narrow tool set and a bug report, drives its own multi-turn conversation, and is
graded by running a test suite it was never allowed to see.
The corpus is SACB (Simple Agent Coding Benchmark), 60 tasks over three
hand-written repositories in Python and TypeScript, hosted at ilintar/SACB.
Grading needs no container: a cached venv gives ruff and mypy for Python, and
node's own test runner plus tsc for TypeScript, so the whole dependency set is
four packages.
Three constraints are deliberate:
* No shell, and no way to run the tests. The only feedback channel is `lint`. A
syntax or type error can be driven out mechanically; a logic error has to be
reasoned about. A harness that lets an agent iterate against the test suite
measures something closer to search than to understanding.
* Two edit tools, one line-addressed and one content-addressed. Which one a
model reaches for, and whether it keeps line numbers straight after its own
earlier edit, is itself a signal.
* The tests never touch the working tree. They are held outside it and copied
into a throwaway copy only once the agent has stopped, so they can be neither
read nor edited. Every tool path is resolved and confinement-checked against
the repository root, so a symlink planted inside the tree cannot reach out of
it.
A task counts as resolved when every test that was failing passes and nothing
that already passed broke. Both sets are derived by running the tests against
the defective tree and against the reference fix, never written by hand, so a
task whose defect no test exercises cannot enter the corpus. Many tasks carry
several independent defects, so the fraction of target tests passed is reported
alongside, separating "fixed two of the three faults" from "changed nothing".
An episode that arrives with no tool call and no `finish` is nudged rather than
ended. Models routinely narrate their analysis mid-task, and treating that as
completion ends the episode with no edits and scores a reasoning failure that
never happened; about one episode in four exits that way.
Nothing here is installed unless this suite is selected, and then only for the
language being run: agentic_eval is imported lazily and each toolchain is keyed
per language.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : make the agentic hooks typed rather than discovered
`ty` flagged nine unresolved attributes, of which the seven in main() predate
the agentic suite: `EvalState.dataset` is Optional and was dereferenced without
narrowing.
* `loaded_dataset` turns the Optional into a value in one place, raising rather
than handing back None, so callers do not each have to narrow it.
* `is_agentic`, `prepare()` and `run()` are declared on BaseDataset. They were
reached through getattr with a default at three call sites, which no checker
can see through; `run()` now raises a clear error naming the suite instead of
failing as a missing attribute.
* The sandbox setup is one branch on `is_agentic` rather than two tests of
`grader_type`, which also drops a duplicated condition.
No behaviour change: HumanEval stays at 164/164, and `ty check` on
examples/llama-eval is clean.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* feat: API key
* fix reasoning and response length for agentic runs, add option to save entire transcripts
* llama-eval : fix ty unresolved-attribute on loaded_dataset.ae
Assisted-by: AI coding agent
* llama-eval : trim comments and remove non-ASCII from README
Assisted-by: AI coding agent
* gguf-py : add numpy Q1_0 and Q2_0 de/quantization support
* cont: added q1_0 / q2_0 to the test list removing the small test
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
---------
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
* CUDA: HIP: rms_norm: load float4 at a time and store x in registers
* Add template paramter comments
* use ggml_cuda_is_aligned for src
* nits
* also use ggml_cuda_is_aligned for src in fused variants
* fix nits
* tests : update the Metal fusion baseline
Re-record the MTL fusion baseline with the current build: 276 rows over 131
dummy models.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : fail test-fusion when a baseline row stops matching
The report was built from the counters the run produced, and the baseline was
only ever used as a lookup table, so a fusion that stopped matching produced no
row and was never compared. qwen4exp and mellum lost their fusions this way
while the Fusion / metal job stayed green.
Report a baseline row that the run does not reproduce as a regression. Scope
the pass to the arches that were actually run so --model keeps working.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : never write inline comments unless the user asks
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : keep the mellum dummy model softmax-gated
mellum reads expert_gating_func from the GGUF since #30578, while the fixture
writes sigmoid for every MoE arch, so the Mellum dummy model stopped building
the softmax MoE routing fusion. Write softmax for mellum, matching the arch
default and the GGUFs that conversion/mellum.py produces.
Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : use emoji in the AI stats disclosure
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : list the disclosure model after the hardware
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : use monochrome symbols in the AI stats disclosure
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : add muse-spark to the Metal fusion baseline
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round
Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors
* hex-get-rows: vectorize raw k-quant ukernels
* hex-binary: refactor dma-loop thread functions (no need for macros, etc)
* hex-quant: move/cleanup flat and tiled dequant ukernels
* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights
* hex-get-rows: remove redundant cast
* hex-repack: proper filtering for BF16 that were leaking through during the tests
* hex-repack: streamling tiling rules (broken MM for now)
* hex-repack: fix mmid after tiling updates
* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars
* hex-mm: fix valid_rows calc in various kernels
* hex-mm: remove unused hvx_mm_nx_2d_flat
* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16
* hex-mm: another pass at hvx-tiled f16 matmul
* hex-get-rows: optimize small src1
* hex-get-rows: one more pass at recovering get-rows perf regression
* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py
* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul
* hex-mm: fold tiled_f16 checks into the general flow
* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op
* hex-quant: proper tail handling in new dequant ukernels
* hex-get-rows: remove unused code and cleanup support checks
* hex-mm: cleanup ne1 padding/tiling checks
* hex-mm: cleanup fusion checks for tiled tensors
* hex-repack: add a little can_repack helper to reduce duplication
* hex-repack: never repack host buffers
* hex-repack: factor out repack functions into separate .h and .cpp
* hex-tensor: cleanup tensor ifaces for tiled/padded sizes
* hex-repack: update k-quant tiling comments and cleanup #defines
* hex-dcache: factor out l2/dcache dirty tracking and flushing
* hex-tensor: update docs and remove #defs duplication
* hex-tensor: remove unused/incorrect get_data interface
* hex-get-rows: cleanup vtcm layout build
* tests: disable huge MM tests
* hex-dcache: add missing header
The scheduler uploads each view of a host input on its own, so the per-layer reshape of new_pool_idxs (and of sel_mask in glm5-next) was copied once per layer.
* enable OPENSSL=ON as default in build script, update it in CI and dockerfile
* change to LLAMA_BUILD_BORINGSSL on windows
* refactor
* rm Download UI build
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model: add Muse Spark Open (text, vision and audio)
Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.
Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.
The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Muse Spark: drop the redundant forward expand
* Muse Spark: drop the redundant forward expand and tighten the comments
* Update src/llama-quant.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-graph.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* pi : expand AI disclosure with hardware and stats
Prefer PI_HARDWARE for the hardware field, spell out the disclosure block
template (model link, harness, hardware, stats) and source [STATS] from
~/.pi/stats.sh. Record the hardware in the Assisted-by commit tag.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : drop square brackets from placeholders
Brackets inside a markdown link text break the link on GitHub. Describe the
disclosure bullets in words instead of embedding a bracketed template.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Small helper around the GitHub Actions API: lists the workflow runs that are not
finished yet (in_progress, queued, pending, waiting, requested), grouped by status.
Supports selecting runs by status, by branch (server-side, to reduce API traffic)
or by a regex on the branch name, and can cancel the selected runs via --kill.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
The first evaluated token attends to the positions above
pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the
whole window, but the reuse threshold asked for two more cells.
A saved SWA state keeps only the cells inside its window, so a
state read back from a slot file or from the RAM prompt cache was
always rejected and the slot rolled back to a checkpoint.
The SWA threshold now matches the window, while a state without SWA
keeps the exact bound. The test harness gains override_kv, used to
give the SWA test model a window smaller than the test context.
[no release]
The slot file held the target context and the context checkpoints,
but never the draft context, so a restored slot drafted with an
empty or stale draft KV until the cache refilled.
The file now ends with the live draft state, the full draft context
and the speculative state, in the same appendix style as the
checkpoints. A draft state that does not fit the draft context is
dropped and the draft starts empty, while older files and older
servers keep working.
[no release]
Slot selection takes free slots by id or by last use only, so child
tasks often land on a slot holding a parent state while empty slots
are idle, and overwrite it without saving it to the prompt cache.
Free slots are now ordered empty first, then least recently used, for
both the LRU fallback and the child slots, and a child task saves the
state it overwrites. With 4 grouped decision requests in flight and
12 states in rotation, prompt cache reads go from 5 to 36.
[no release]
The registry hard-coded 10 for AGENTIC_MAX_TURNS while DEFAULT_AGENTIC_CONFIG
said 100. The registry seeds SETTING_CONFIG_DEFAULT, so the loop's
`|| DEFAULT_AGENTIC_CONFIG.maxTurns` fallback never fires and users got 10
turns. Reference the shared constant, as the adjacent MCP timeout does.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* Add Support to maion-coder arch
* iAdd Support to support Draft models
* Address comments
* Remove maion context shift function
---------
Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
tojson wrote nan, inf and -inf for these floats, as values and as dict
keys; json.dumps writes NaN, Infinity and -Infinity. The string and safe
filters of a float keep nan and inf, as Python's str() does, so they no
longer use the tojson function. float_to_json is renamed float_repr, as
it now gives str(); float_json gives the json.dumps text.
Also say in a comment why a last digit 9 is not bumped at a power of 2.
* vulkan : fix concurrent first device init
* tests : load models in parallel in test-thread-safety
Loading them serially never exercised concurrent first backend init, so the crash reported in #29764 was invisible to this test.
* vulkan : create fence before retaining device in buffer context
* replace oneAPI package by DLE+oneDNN package
* support to install oneDNN in windows, in release and docker
* for internal test
* remove test code
* Update intel.Dockerfile
* llama : split llama_prefetch() into merge and prefetch steps
Merge the ranges first, then call the system functions.
This fixes -Wunreachable-code-return on Windows.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Use const_cast
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* vulkan : copy strided f16 KV once so reads spread across memory channels
Assisted-by: Claude (Opus 4.8 / Opus 5)
* tests : cover the f16 KV copy to scratch
Assisted-by: Claude (Opus 4.8 / Opus 5)
* vulkan : gate the f16 KV scratch copy to AMD
Assisted-by: Claude (Opus 4.8 / Opus 5)
* vulkan : reuse the generic copy shader for the f16 KV scratch fill
The f16 clause needs only a strided copy, not a dequant, so drop the
dedicated shader and dispatch the existing copy pipeline instead. The
copy is described in source memory order so the strided access lands on
the writes, which stream, instead of the reads, which would concentrate
on a few memory channels.
Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5)
* vulkan : extend the strided KV scratch copy to bf16
Assisted-by: Claude Opus 5
* vulkan : clean up comments
Assisted-by: Claude Opus 5
* consolidate copy shaders
* disable on RDNA4
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* cpu: add F16 SWIGLU_OAI reference implementation and F16 activation op test coverage
Adds ggml_compute_forward_swiglu_oai_f16() to the CPU backend, mirroring
the existing F32 path, and extends test-backend-ops with F16 test cases
for SWIGLU_OAI so downstream backends(e.g. Hexagon) can be CPU-diffed
for correctness.
* ggml-openvino: fix f16 SWIGLU_OAI type mismatch in alpha/one constants
translate_glu_swiglu_oai creates alpha_const and one as f32 constants
unconditionally, so f16 inputs hit a type mismatch in Swish/Add on GPU
("Swish inputs must have the same type but they are: f16 and f32").
translate_glu_swiglu_clamp already upcasts to f32 and converts back at
the end; apply the same pattern here.
---------
Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
* llama: honor PARTIAL_ONLY in the plain KV cache
A non-SWA llama_kv_cache rolls back with seq_rm, so it holds no
partial state, yet state_write ignored the flag and serialized every
cell. A draft context made of a plain KV cache (MTP head, dense -md
draft) therefore copied its whole KV into each server checkpoint,
growing it by the draft KV size per token.
The cache now writes and reads nothing for PARTIAL_ONLY unless it is
SWA, and the iSWA wrapper no longer gates its base cache since it
filters itself.
* Update src/llama-kv-cache.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama: delegate PARTIAL_ONLY to the caches of every memory wrapper
The hybrid, hybrid-idx and dsa-iswa wrappers skipped their attention
caches on PARTIAL_ONLY, which also dropped an SWA window that is
partial state. Each cache now applies the flag itself, so the
wrappers only delegate. The indexer is marked stale only when a
restore rewrites cells.
test-llama-context gains a case checking that the partial state is
not larger than the full state and, without SWA, does not grow with
the tokens.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
An embedding-only graph has no logits, and build_pooling() returns
early when embeddings are off, so since the models no longer expand
t_embd the graph ends up empty and its inputs are never allocated.
Expand t_embd in that case.
* server: save only restorable slot states to the prompt cache
The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.
Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.
* server: test that decision slots skip the prompt cache
Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.
* server: restore any reusing task from the prompt cache
Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.
Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
* spec : add lilicorr support
* convert : use convention for hparam assignment
* add todo for hparam use in llama-batch.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* typo fix
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The clamp kernel indexed source and destination as one flat contiguous array, so
a strided view as input (the new CLAMP v=1 case) read the wrong elements. Add a
strided kernel_clamp_nc, as the other unary ops have, and take it whenever the
source is not contiguous.
* sycl: persistent IQ3_S/IQ3_XXS weight reorder for SYCL decode and prefill
Add persistent reordered weight layouts for IQ3_S and IQ3_XXS. Weights
are reordered once at load time; the reordered MMVQ dot products then
walk dense super-blocks with dp4a and SWAR sign expansion instead of
gathering bits from the interleaved standard layout. Reorder-aware
dequantization feeds the batched prefill GEMMs so prompt processing
sees the same benefit.
Also harden the oneDNN build check: a missing DNNL package or a
CPU-only DNNL build now fails at configure time instead of silently
producing a degraded backend.
Measured on Intel Arc Pro B70 with Qwen3.8-27B-GSQ-RCO-IQ3_S (MTP),
-ngl 999 -b 4096 -ub 1024 -fa on:
tg256 @ 0 ctx 12.92 -> 23.54 tok/s (+82%)
tg256 @ 9,216 12.33 -> 21.69 tok/s (+76%)
tg256 @ 64,000 9.76 -> 14.91 tok/s (+53%)
pp8192 833.9 -> 833.6 tok/s (flat)
Qwen3.5-9B Q4_K_M / Q3_K_M are unchanged within noise (largest delta
-0.5% pp, tg flat), so standard quants are not affected.
The MMVQ GLU fusion widening and rms_norm+scale fusion are
intentionally not part of this change; master already carries them
via #28931.
* removing wait as per review request
* sycl : fix fusion.cpp line ending
* ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (#30342)
#30270 routed both scalar conversions through F16C. For FP16->FP32, Clang 20.1.8
lowers the intrinsic to a vpinsrw merging into the live accumulator register of
the quantized dot products, which serialises the loop (19 vs 4 cycles/iteration
estimated, 45-65% slower Q8_0/Q4_0 MUL_MAT_ID kernels measured). Keep F16C for
FP32->FP16 as intended and the table for FP16->FP32.
* ggml-cpu : comment out the F16C FP16->FP32 define instead of removing it (review)
* ggml-cpu : size set_rows conversion buffer for all threads
SET_ROWS is planned with n_tasks = 1, but every thread runs it and uses its own row of the F16 -> F32 conversion buffer, so threads > 0 wrote past the work buffer when the graph had another multi-threaded op.
* move set_rows to n_task = n_threads group
* cuda, hip: GDN chunked kernel
Ported-from: https://github.com/tetherto/qvac-fabric-llm.cpp/pull/257
Source-commit: 53f0027d8070715fe9ad2131f3201bb0001d4eed
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
* fix older builds
* review fixes
* cuda: enable GDN MMA on tested Ampere and Ada GPUs
Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback.
Assisted-by: Codex
* cuda : use refactored swizzle API in GDN MMA
* ggml : GDN only guarantees the final state when n_tokens > K
* cuda : add short GDN MMA phase comments
---------
Co-authored-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
* skill : add tools/ui review checklist to code-review skill
Adds a WebUI (tools/ui) section distilled from a year of review
feedback on tools/ui PRs: constants/enums placement, layering
(stores/services/utils/hooks), DRY and dead code, Svelte 5 idioms,
comment hygiene, streaming perf, frontend security, and PR hygiene
(static build, demos, scope discipline). Scoped to tools/ui only.
* skill : expand tools/ui constants/enums ruleset
Replaces the short constants bullet list with a decision ruleset
distilled from review outcomes: when a literal becomes a string
enum vs an enum-keyed map vs a declarative config vs a grouped
as-const object vs a scalar constant, naming and suffix
conventions (_REGEX, _MS, _PX, semantic separators), placement
and composition rules, and what legitimately stays inline.
* skill : drop stale static-build-commit requirement for tools/ui
Static build output is no longer stored in the repo - the UI is
built by CI and bundled at release time. Keep the requirement that
npm run build succeeds and the UI build job is green.
* skill : address copilot review of tools/ui checklist
- scope the video-demo requirement to user-facing interactive/visual
changes instead of every tools/ui PR
- narrow the constants trigger to reused/domain-significant literals,
keeping the inline boundary as the escape hatch
- carve out independent literals for persisted/wire-contract test
assertions so they don't become tautological via shared constants
- allow simple presentation-local $derived in components; only shared
business logic, orchestration and parsing leave the component
- match the existing barrel documentation style instead of requiring
per-export JSDoc everywhere
- keep non-obvious why/rationale comments; move only examples and
edge-case enumerations into tests
- point at the actual interpolate-size/grid-template-rows transition
pattern in CollapsibleRegion.svelte instead of calc-size()
* skill : use spaces for nested list indentation
EditorConfig CI requires space indentation for all files; the
scalar-constants sub-bullets used tabs. Matches the 2-space nested
list convention used in docs/.
* ggml: fix conv_1d result layout for batched input
The GEMM behind conv_1d folds the batch into its rows, so the product
sits in memory as [OL, N, OC] while the reshape claims [OL, OC, N].
The shape is right and the contents are transposed, so nothing fires:
no assert, no abort, the graph just runs and the channels come out
interleaved with the batches. With N == 1 the two readings describe
the same buffer, which is why the three vocoders in the tree never
hit it, they all squeeze to 2d right after the call.
Reshape to the real layout and permute OC and N back. The N == 1 path
keeps its direct reshape, so the current callers build the same graph
and pay for no extra copy.
tests/test-conv-1d.cpp compares one batched convolution against N
single batch convolutions built from the same weight and the same
input slices. Five of the seven geometries fail before this change and
all seven pass after. N == 1 and OC == 1 pass either way, since
swapping two dimensions when one of them is 1 changes nothing.
The batched path matters to downstream audio projects where several
synthesis lanes share a single graph. Nothing in mtmd batches a
vocoder today, each one streams one flow and carries its left context
between calls, but the op is correct now if one of them grows a lane
dimension.
* ggml: build the same conv_1d graph for every batch size
Address review from ggerganov: drop the N == 1 branch of ggml_conv_1d.
The GEMM result is always reshaped to [OC, N, OL] and permuted back,
as ggml_conv_2d does, so every batch size builds the same graph. For
N == 1 this adds a cont on an already ordered tensor.
* common : add --spec-draft-n-ubatch
Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).
The default 0 keeps the current behavior of following --n-ubatch.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
[no ci]
Added that the oneapi toolkit includes the sycl dependencies. This makes it clear for first time users, that the toolkit includes all necessary dependencies, so that the user is guided to install the toolkit if they do not already have the dependencies separately
When every device reports zero free memory the split points are
normalized by a zero sum, the NaN split sends layers to a device index
past the end and loading aborts with vector::_M_range_check.
- add rpc_log_format_size() and print byte counts in KiB/MiB/GiB across the
RPC client, server and transport traces
- add rpc_log_format_tensor() and include tensor name, op, type and shape in
the set/get/memset/copy/init tensor logs
- fix rpc_cmd_name() returning UNKNOWN for SET_TENSOR_2D, GET_TENSOR_2D,
COMM_INIT, COMM_ALLREDUCE and COMM_FREE
- move the log helpers to ggml-rpc/log.{h,cpp} under a common rpc_log_ prefix
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
tojson printed floats through a C++ stream (6 significant digits, no
".0"), so 48.858844 became 48.8588 and 1.0 became 1 in tool definitions,
tool-call arguments and other template output. Write the shortest digits
that read back as the same value, as json.dumps does. The string and safe
filters of a float use the same code.
These graphs only produce embeddings, and build_pooling() already
expands its output for every pooling type, so the expand at the end of
the model graph is redundant.
The RDNA3.5 config table selects a 64-row / 128-thread tile up to J=32 and a
128-row / 256-thread tile above that. On gfx115x the smaller tile leaves enough
LDS free for a second workgroup per CU, while the larger one does not.
Extend the 64-row tile to the (type, J) pairs where it measured faster on
gfx1151, and leave every other entry untouched. J=96 is deliberately excluded
because it did not benefit.
test-backend-ops perf on gfx1151, 100 MUL_MAT shapes over the affected types
(N in 48..512, M both padded and unpadded), 3 rounds: median +2.1%, best
+26.0%, no shape outside run-to-run noise. 5 MUL_MAT_ID shapes: median +8.0%.
Assisted-by: Claude Opus 5
* llama : build the k-pool selection in the causal mask layout
- the glm5-next indexer folded the causal mask in through a [n_kv, n_tokens]
view: one reshape of the mask ahead of the add, and one of the result ahead
of every layer reading the selection, including the shared-indexer layers
- build the selection directly in the mask layout, as qwen4exp already does,
and drop the now-degenerate reshape in its attention as well
- assert the selection width matches the input it is taken from
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
* unicode : add opt-in NFC normalization for Unigram tokenizers
Note: used help from Codex, Sol 6.1 Medium
* Fixing initial bot comments + linter error (using existing out instead of print)
* removed test_nfc_tokenizer
The d1-omni and kev prompts escape "<|name|>" in the request text with
std::regex_replace. libstdc++ recurses for each char of the name, so a
state of "<|" and 30000 letters overflows the stack of the HTTP thread
and llama-server dies with SIGSEGV.
server_decision_escape_special_tokens() writes the same text with a loop.
test-server-component compares it with the regex on every string of up to
6 chars of "<|>a-".
* hexagon: add configurable HTP DCVS/HMX power mode
* hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup)
* hexagon: add --hex-power-mode flag to run.py
Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following
the existing --hex-nhvx pattern. The env var must be set before the
process touches the Hexagon backend registry (see comment in
ggml_hexagon_init), so this is exposed as a runner flag rather than a
common/arg.cpp CLI option.
* hexagon: keep HTP awake for DCVS power profiles
* Apply suggestion from @max-krasnyansky
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The no-cache attention pads K, V and the mask to 256, but T5 also adds
a relative position bias of [n_tokens, n_tokens], so the add with KQ
fails unless n_tokens divides the padded width. Pad the bias with
zeros up to the KV dimension of the mask, like K and V.
* llama : do not map foreign ranges into a backend buffer
A buffer created from a host pointer pins everything that it wraps. When
several contexts share the same file (lazy tensors, MoE tensors kept on the
CPU with -cmoe), the single buffer that spans the first and the last tensor
of a context also wraps the ranges owned by the other contexts, pinning data
that the backend never reads.
For models with per-layer token embeddings this pins the whole lazy tensor
into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the
point of reading it lazily.
Instead, compute the file ranges of each context and subtract the ranges
owned by the other contexts, then create one buffer per range. Weights that
are mapped in more than one context (e.g. token_embd.weight) keep being part
of the range of every context that uses them.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : enable prefetch on macOS
The prefetch hint of the tensors marked as prefetchable is a no-op on macOS,
so their pages are faulted in one by one while the graph runs. madvise
(MADV_WILLNEED) behaves as expected here, so add the platform to the guard.
With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read
from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to
735 / 749 / 788 t/s for pp512 / pp2048 / pp4096.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : use a plain buffer list per context
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* graph: skip the LM head for pooled embeddings
A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.
The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.
* graph: drop the redundant t_embd expand
Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.
* graph: add build_logits() for the LM head output
Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.
* graph: call build_logits() before build_sampling()
Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.
* nit
* tests : consolidate the context state tests into test-llama-context
test-save-load-state, test-recurrent-state-rollback and
test-state-restore-fragmented check the same object from three places:
the KV cache and the recurrent state of a llama_context, saved, loaded,
copied, rolled back. Each had its own main, its own loop over the
models, its own copy of the token and the comparison helpers, and a run
of all three loaded the same models several times.
They are one binary now, and the case is the unit of the test:
tests/llama-context/test-case.h the interface a case implements
tests/llama-context/cases-kv.cpp the cases of the KV cache
tests/llama-context/cases-rs.cpp the cases of the recurrent state
tests/llama-context/test-util.cpp tokens, generation, comparison
tests/llama-context/test-runner.cpp the models of a case, on threads
tests/llama-context/test-log.cpp the log file of a model
A run is a table: a row per model, a column per case, a cell holds the
verdict of that pair. The models load on several threads (-j, --threads,
min(cores, 4) by default), the table is printed when it is complete and
the log of a model that failed is printed after it, in the order of the
rows. A case returns a verdict and the reason for it, and the runner is
the only one who reports - the test does not use tests/testing.h, a run
over 130 models is a matrix of verdicts and not a tree of subtests.
Staging no longer travels through the environment: llama_model_set_attn_rot
replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one
thread used to change for the models the other threads were loading, and
llama_memory_get_buffers is added for the cases that copy the buffers of
a memory. common_log_set_stdo keeps the console quiet while the table is
being built.
ggml_aligned_malloc allocates with posix_memalign when the thread
sanitizer is on. On macOS it allocates with the Mach VM, which the
sanitizer does not intercept, so it never sees a page range change hands
and reports the first write into recycled memory as a race - 30 of them
in this test, and every one of them false.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : run single-threaded by default
* cont : try to build the tests on all platforms
* cont : trim down some body fat
* cont : fix
* cont : fix the fix
* ggml-cuda: gate CUDA graph debug logs behind GGML_CUDA_GRAPH_DEBUG
* ggml-cuda: cache env var lookups in static const instead of re-reading per call
* ggml-cuda: read env vars exactly once via ggml_cuda_env_flag/ggml_cuda_env_int
* ggml-cuda: consolidate env flag reads into single static accessors
* ggml-cuda : drop env helper comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : remove NDEBUG gates for CUDA graph logs
* vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount
* vulkan : fix narrowing in chunked matmul dispatch
* vulkan : zero fill the A side of mul_mmq for k steps past the end
The A side skipped the shared memory write for the k steps past end_k, while the B side was already told the bound and zero filled. The A registers are read for all BK_STEP steps, so the stale scale went into the dot product and produced NaN at k = 32.
Pass the bound to block_a_to_shmem and write zeros when it is false, for every DATA_A type, mirroring the B side.
Also assert that the B and D subbuffer sizes are not VK_WHOLE_SIZE before they are adjusted per chunk, and add two batched cases of the chunked shape.
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* vulkan : remove redundant matmul tile adjustment
---------
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* tests: cover weighted scalar broadcasts
Assisted-by: OpenCode
* ggml : support extended scalar DMA binary ops
Assisted-by: OpenCode
* hex-binary: remove all non-dma paths
* hex-binary: add support for non-contig strides and cleanup dma preable a bit
* hex-binary: cleanup the gating and simplify dma retry
* hex-geglu-erf: add a gate for F32 only, we dont support F16 for this op
* tests: cover weighted repeat broadcast
Assisted-by: OpenCode
* hex-binary: update REPEAT and COMPLEX kernels to use dma as well (missed them last time)
* hex-add-id: add dma support for src2 (idx)
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The no-cache path builds a [n_tokens, n_tokens] mask and passes K and V
unpadded, so FLASH_ATTN_EXT sees K->ne[1] == n_tokens. The CUDA kernels
for head size 512 require a multiple of FATTN_KQ_STRIDE, so the op falls
back to the CPU for most lengths, and the -fa auto probe (1 token)
disables flash attention for such models.
Pad the mask like llama_kv_cache::get_n_kv and pad K and V with zeros up
to the KV dimension of the mask. clef builds its own causal mask and
pads it the same way.
* hexagon: move the converting CPY in blocks of rows per DMA descriptor so short rows are not DMA-latency bound
* hex-cpy: cleanup cpy cursor handling
* hex-cpy-concat: introduce threaded dma/hvx gather-row helper
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
* hex-cpy: remove leftover rejections for extended buffers
* hex-cpy: relax the gate for fast-path
* hex-cpy: remove scalar divs and vtcm access and use explicit gather_sync
* hex-cpy: add proper support for mdev work splitting to fastpath
* hex-cpy: optimize dense gather-row tasks
* hex-cpy: unroll gather-rows loops
* hex-cpy: add fastpath for f16
* hex-cpy-concat: support for colapsing 3/4D dims
* hex-cpy: double-buffer support
* hex-cpy: factor out gather-rows into more of an op by itself
* hex-cpy-concat: dedup kernel-params
* hex-cpy-concat: step-down nr to fit instead of hard fail
* hex-gather: replace div with a mask
* hex-gather-rows: cleanup dma pipeline and remove the stalls
* hex-gather: specialize compute funcs
* hex-gather: reduce scalar reg.spills and func calls
* hex-gather: do not recompute rows-per-thread on the npu
* hex-build: add support for building/installing just hexagon
* hex-cpy-concat: remove redundant barrier in concat and move cpy barrier under fence
* hex-gather: annotations for dma-out
* hex-mdev: make all key vtcm_acquire sync state atomic (fixes random hangs in mdev setups)
---------
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
* vulkan: pack FMA for f16 accumulation, no dot product op
* vulkan: dot_product packs two rows per f16 FMA
* vulkan: rename dot_product to dot_product_v2
Give llama_build() and llama_build_and_test() a LINK keyword (default:
llama-common) so call sites can declare what a test actually needs:
- ggml-only tests now link just ggml: test-opt, test-metal-graph-optimize,
test-backend-ops, test-tiled-mulmat, test-barrier, test-quantize-fns,
test-quantize-perf, test-rope, test-col2im-1d, test-gguf, test-alloc
- llama-only tests drop llama-common: test-unicode, test-sampling,
test-grammar-parser, test-llama-grammar, test-batch-alloc,
test-gbnf-validator, test-quant-type-selection
- both mtmd tests link mtmd only, dropping the LLAMA_TEST_NAME /
target_link_libraries workarounds
Both helpers now set cxx_std_17 explicitly, since the tests no longer
inherit it from llama-common's PUBLIC interface.
tests/testing.h no longer includes common.h; the only symbol it used from
there (string_join) is inlined.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* metal : fix mul_mat_id limits and kernel name
* metal : fix mul_mat_id with duplicate ids
kernel_mul_mm_id_map0 assumed at most one match per token and a fixed
capacity of ne21 entries per expert, which produced wrong or dropped
output rows and could overflow into the slice of the next expert when
the ids contain duplicates.
build a compact per-expert list of the (token, slot) entries instead:
- pass 1 counts the entries of every expert, thread 0 computes the
offsets of the expert slices (tpe now holds [base, count])
- pass 2 writes the entries with one unconditional store per token
(conditional stores of divergent lanes are very slow), the dummy
value of the tokens without a match lands on a padding slot of the
slice that is never read - duplicates are handled by a rare branch
- kernel_mul_mm_id reads the base/count of its expert and loops over
the entry tiles that do not fit into the grid
The ids buffer is sized as ne20*ne21 + ne02 (entries + padding).
883/883 test-backend-ops MUL_MAT_ID tests pass, llama-bench is within
run-to-run variance of the previous behavior.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* metal : make mul_mat_id map0 ne02 a function constant
use FOR_UNROLL for the id loops of kernel_mul_mm_id_map0 and turn ne02
into the function constant FC_mul_mm_id_map0_ne02 (FC_MUL_MM + 7), so
that the threadgroup offsets of cnt/bas/sids and the prefix-sum bound
fold at compile time.
the pipeline name now encodes ne02 (same convention as the ssm_conv
nc/silu constants) and the map0 pipeline is compiled with the constant
value set. ne02 is no longer part of ggml_metal_kargs_mul_mm_id_map0 -
the dispatch grid still comes from the tensor ne[2] on the host.
no measurable perf change: 883/883 test-backend-ops MUL_MAT_ID tests
pass, op-level and llama-bench results are within run variance.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : clean-up + TODO
* metal : dispatch mul_mat_id entry tiles from the prepass
follow the approach of the vulkan implementation (#29998): instead of
launching a grid over experts x ne21 entry tiles and letting most of
the threadgroups early exit, kernel_mul_mm_id_map0 now also emits a
row map - the total tile count plus one (expert, first entry, valid
rows) descriptor per 32-entry tile that has work.
kernel_mul_mm_id is launched with a static upper bound on the number
of tiles (full tiles over all entries plus at most one partial tile
per expert), reads its descriptor and processes a single tile - the
tail early exits. This removes the round loop that was needed to cover
duplicate ids past the grid (register pressure: th_max 768 -> back to
the plain mul_mm level) and duplicates are now handled implicitly,
since they only produce more descriptors within the bound.
The grid shrinks by ~8-16x on typical shapes (e.g. 128 experts,
8 used, 512 tokens: 128 tiles instead of 2048 entry tiles, of which
94% only launched to exit) and ne21 is no longer needed by the mm
kernel kargs.
883/883 test-backend-ops MUL_MAT_ID tests pass. llama-bench
(Qwen3.6-35B-A3B Q8_0) is now faster than master on all pp sizes
(pp2048/4096/8192 +2%, pp512 +0.5%), tg32 unchanged.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : add smem assert
* examples : fix draft sampling and rejection handling
This commit fixes two issues in the speculative example which are
described in detail in the linked issue.
Resolves: https://github.com/ggml-org/llama.cpp/issues/29975
* examples : add kv-unified requirement for parallel > 1
Add an optional ggml_backend_sched_realloc_callback that user code registers with
ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a
graph are not sufficient and have to be reserved again. The callback reports whether the
backend assignments and the graph size changed since the previous scheduling.
llama_context registers the callback when creating the scheduler (wrapped into the new
llama_context::sched_new helper) and treats the reallocations that follow a change of the
backend assignments as expected, counting the rest by graph size - unchanged or changed
from the previously scheduled graph. Both counters are reported on one line in the perf
stats.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* ui: fix text attachment preview overflow
Set the max height to 80dvh and the max width to full on screens
below medium and 80vw from medium onward.
* style: run formatter
* hexagon: add IM2COL_3D support
Assisted-by: OpenCode
* hexagon: add DMA paths for IM2COL_3D
Assisted-by: OpenCode
* hexagon: avoid scalar fp16 conversions in IM2COL
Assisted-by: OpenCode
* remove in-loop spills in pre-v81 devices
Assisted-by: OpenCode
* hex-im2col: re-write to use canonical dma dbl-buff pipeline, remove divs and very inefficient fp conversions
* hex-im2col: move kernel_params and vtcm layout precompute into the host
* hex-im2col: remove explicit tracking of number of dma-pushes
* hex-im2col: remove incorrect uint64_t casts
* hex-im2col: no need for macros for dma loops and pass compute func ptr via local context
* hex-im2col: use cursor pattern, tracing and proper n_threads handling
* hex-im2col: support non-contigue views
* hex-im2col: minor reformatting
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* opencl: fallback when Q4_K weight images exceed device limits
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: avoid incomplete subgroup in rms_norm
* opencl: refine conditions for dense bin kernels and fix bcast for q4_0
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
---------
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs
GGML_RPC_DEBUG is now parsed as a number: 0/unset disables debug logs,
1-3 emit increasingly detailed output (events, per-command trace,
transport detail). Non-numeric values fall back to 1.
The duplicated env/macro blocks in ggml-rpc.cpp and transport.cpp are
replaced by a shared log.h, which becomes the single choke point for
all logging of the RPC backend: LOG_ERROR/LOG_WARN/LOG_INFO for
unconditional severity logs and LOG_DBG/LOG_DBG2/LOG_DBG3 for the
verbosity-gated ones. The transport files no longer need ggml-impl.h,
and the server banner now also goes through the ggml logger (stderr).
Missing logs are added on both the client (handshake, buffer ops,
tensor transfers, graph computes, cache decisions) and the server
(per-command dispatch, graph nodes), including the negotiated
transport via the new socket_t::transport_name().
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* rpc : align log levels with the documented verbosity semantics
The first pass introduced GGML_RPC_DEBUG levels, but many call sites did not
follow the documented semantics and some logs bypassed the macros entirely:
- Route the remaining raw GGML_LOG_* call sites through the local LOG_* macros
so that log.h stays the single choke point for RPC logging
- Emit per-tensor transfers and per-command handler traces at level 2, and
one-shot lifecycle events (backend and buffer type creation, buffer
allocation, graph compute, graph cache eviction) at level 1
- Drop the duplicated enqueue trace in rpc_dispatcher::send, work() already
logs every dispatched command together with its round-trip timing
- Promote degraded-operation events to LOG_WARN: remote allocation failure,
RDMA falling back to TCP, unexpected peer disconnect
- Log graph cache eviction on both sides, since client and server must stay
in sync for the incremental graph update to be valid
- Distinguish Unknown command (opcode out of range) from Unhandled command
(valid opcode with no handler)
- Fix format specifiers (%zu for size_t, %u for device ids, 0x for hex
pointers) and a missing newline in an error log
- Document that llama.cpp applications drop ggml debug records unless the
verbosity threshold is raised with -lv 5
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* vulkan: compute every row of an expert in mul_mm_id when ids repeat
* vulkan: handle mul_mat_id duplicates in the prepass rather than looping
Extend the "hoist row ids" optimization to always be enabled and to emit a
compact list of tile descriptions that need to run, and to emit multiple tiles
when needed. Then we can launch a tighter upper bound on the number of
workgroups and the tail can trivially early exit.
---------
Co-authored-by: Anjielon <[EMAIL_REDACTED]>
A request asking for a busy id_slot still ran the prompt cache update
on that slot before being deferred. When the RAM cache held a better
match, it was loaded into the slot while another request was still
generating there, and that generation continued on the wrong context.
The busy slot is now returned as is and the request waits for it.
* common : read a GGUF's trained context from its metadata
common_get_gguf_n_ctx_train opens only the file's metadata (no_alloc,
like common_get_decision_type) and reads <arch>.context_length, so a
caller can learn the trained context without loading the model. It
accepts both u32 and u64 values and returns 0 when the file is missing,
unreadable, invalid, or reports no context length.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* server : report the trained context in the models listing
update_caps already resolves the model file offline to read its
modalities, so it now reads the trained context from the same GGUF
metadata, and GET /models reports it as context_length when it is
known. A router listing then carries the context without any Hub
request, which lets the UI sort and filter by it offline.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : take the trained context from the models listing
The router now reports context_length per model, so the option mapping
fills contextLength from it and the manager reads it before the Hub
record. The Context column, the context sort and the context filter
then work with the Hugging Face Hub API turned off. A browser suite
guards the sort and the search, the Hub-cache driven context filter and
the re-sort when details arrive after the sort was clicked.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : mark favorite models with a heart
A favorited model shows a rose heart in the selector even before its
row is hovered, and the crossed heart takes its place on hover, so
unfavoriting stays one hover away. The manager table marks its
favorited rows with the same heart after the badges and capabilities.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* fix: UI text nit
* fix: UI nits
* fix: Favorite models grouping in models table
* feat: Remove sorting from Status column in Models Table
* server: read the GGUF metadata once per model
Read the decision type and the trained context in a single GGUF open,
accept only a UINT32 context length like the model loader, and reset
n_ctx_train with the other caps so a failed refresh drops it.
* fix: Post-review fixes
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* CUDA: fuse copy of updated state snapshots into recurrent cache with ssm_scan
* CUDA: remove redundant cuda copies with K==1 (non spec-dec) scenario as well
* opencl: skip kernel_cpy_f32_f32_pack on A6X to avoid shader compiler crash
* The A6x compiler backend found in iot device with a623 (E031.50.31.01)
cannot handle kernels with a large number of arguments. Skip this
kernel for A6x to avoid compiler crash
* opencl: A6X constant-fold workaround for get_local_size in GEMV kernels
* opencl: add Adreno 623 to A6X GPU detection list
* model : use exact GELU for ModernBERT encoders
Assisted-by: Codex
* model : keep tanh GELU aliases on ggml_geglu
Assisted-by: Claude Opus 5.5
* model : map gelu_python to ggml_geglu_erf
Assisted-by: Claude Opus 5.5
The reserve builds n_outputs_max_per_seq sampling chains per sampler,
while a decode built one per output row, so the graph changed its
topology after the reserve and GGML_SCHED_NO_REALLOC builds aborted on
the next same sized graph. Every sampler now builds
n_outputs_max_per_seq chains, the ones without a row of the ubatch on
the padding row and not selected, and graph_max_nodes counts them.
ggml_acc_impl narrowed a size_t offset to int32_t without checking that it
fits, so a large offset could truncate to a negative int32_t. The forward
then sign-extended it to a huge size_t and the bounds assertion wrapped,
allowing an OOB write below the dst buffer. Check the offset before the
narrowing, matching the existing check in ggml_set_impl.
* meta : handle views of tensors allocated on the host
A view shares the memory of its view_src, so ggml-alloc never allocates a view in
the buffer of the split it lands in - the scheduler copies the source into the
split and the ops that use the view read that copy. The view node itself is a noop
and does not need a split of its own, but the meta backend asserted when one was
left inside a meta split:
- ggml_backend_meta_get_split_state() dereferenced tensor->buffer->context
- the graph rebuild mapped every node with ggml_backend_meta_buffer_simple_tensor()
Accept such nodes when they are views of host tensors, which also generalizes the
previous s_copy_main workaround. This fixes the assert hit by KV cache views when
using --split-mode tensor with partial offload.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* archs : re-enable sm tensor for K2 Horizon
* cont : add TODO and reference
The mixed batch path of PR-29622 writes the token rows with set_rows
into a dup of the embeddings. WebGPU did not support DUP, so the dup
ran on the CPU while the set_rows writing into it was scheduled on
WebGPU, which then bound a CPU buffer and crashed. DUP is the same copy
as CPY and CONT and now goes through the same path.
* ui : add the drawer, sheet and grouped list primitives
Add the drawer and sheet overlay components, the toggle and toggle
group, the shared searchable input, the collapsible sections and the
grouped list, and the near viewport helper they position with.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : add the shared model components and data layer
Add the model row pieces shared by the selector and the manager, the
models store with its download status feed, the huggingface and
migration services, and the model utils, enums and constants they
read through.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : rework the models selector around its providers
Group the selector components under their own folder, rework the list
around the provider grouping, add the mobile trigger and the download
item, and derive the selection state in one hook.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : add the models manager
Add the manager table with its repo, quant and status rows, the
filters and the toolbar, the row actions in a drawer, the downloads
section, the manage models dialog and the stories and tests.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : rework the chat form actions and the mobile experience
Move the add actions into a drawer and a sheet, float the model
actions in a bar on a phone, open the context panel in a drawer, and
derive the attachment, reasoning and tools menus in hooks.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : move the mcp servers to the sidebar rail and tidy the dialogs
Move the mcp servers dialog to the sidebar rail with its menu
entries, even out the dialogs on a phone, and group the mcp and
settings stores behind their own modules.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : polish the shell
The secondary button gets its own look back and the chat add button its
own light surface. Pressable elements get a pointer cursor again, the
font rendering smooths on the app shell, and the agent skills stay out
of prettier's way.
The root layout props probe rework that came with this polish reads the
providers api url and stays with the providers change instead.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : wrap the model id classes
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : load the model the new chat CTA picks
Start a new chat selected the model but left it unloaded, so the chat
opened against a server with nothing in memory. Both CTAs share a load
helper now.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* refactor: Post-review fixes
* hexagon: fix IM2COL patch-embed DMA ring overflow
The exact-tiling (stride == kernel, no pad/dilation) IM2COL DMA kernel
issues IC*KH DDR->VTCM descriptors per output row without checking the
return value of dma_queue_push(), and then pops IC*KH times. The per-thread
DMA ring holds 256 entries and a push into a full ring returns false
and drops the transfer, so for IC*KH > 255 the remaining rows of the
VTCM staging buffer were never written and stale data (often NaN/inf)
leaked into the output.
Solution is to retire the oldest descriptor when the ring is full, just as the blocked
kernel in the same file already does, and wait with dma_queue_flush().
For testing, added exact-tiling test cases with IC*KH > 256 (2D 1x1, 2D 2x2 patch
embed and 1D, F16 and F32 dst), which fail on HTP without this fix.
* Apply suggestion from @max-krasnyansky
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* CUDA: radix top-k for large row counts
Replaces CUB's per-row DeviceTopKKernel with a grid-over-rows radix select,
gated on GGML_CUDA_TOPK_RADIX_MIN_ROWS. On qwen4exp at 34,816 tokens this cuts
top-k from 1,671,253 launches / 5,761.8 ms to 2,329 / 941.8 ms.
* CUDA: select the TOP_K implementation by shape
Replace the nrows/ncols special case with the decision boundary from #28547
(as implemented in #29278): bitonic for short rows, radix select for several
long rows, and DeviceTopK or CUB argsort for a single long row. The
thresholds stay overridable at build time.
Two refinements on top of that boundary:
- bitonic stays in use for rows up to a padded 1024 while the rows fit in one
wave of blocks (nrows <= number of SMs); radix select pays a fixed cost of
about a dozen launches that only amortizes over more rows
- with DeviceTopK available, it handles up to two rows
Radix select now processes rows in chunks so its scratch memory stays bounded,
and the bitonic path keeps its chunking. HIP and MUSA keep their previous
thresholds.
Add perf cases around the bitonic/radix crossover to test-backend-ops.
* CUDA: make top-k comments less verbose
* CUDA: remove the TOP_K width limit from supports_op
* CUDA: use DeviceTopK for single-row TOP_K if available
* CUDA: avoid ncols overflow in the TOP_K bitonic check
* CUDA: share the row chunking helper between argsort and top-k
* CUDA: do the TOP_K radix blocks_per_row math in int64_t
* CUDA: rename GGML_CUDA_TOP_K_NROWS_THRESHOLD_DEVICETOPK to GGML_CUDA_TOP_K_NROWS_THRESHOLD
* CUDA: share one sort helper between the bitonic and CUB TOP_K paths
* CUDA: update the TOP_K TODO, threshold and chunking comments
* tests: add TOP_K cases that span several row chunks
* CUDA: use int64_t col in the TOP_K radix loops, fix threshold comment
* CUDA: limit TOP_K and ARGSORT support to ne[0] <= INT_MAX
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* CUDA: fix CCCL version guard breaking on major version rollover
The guard compared the major and minor components independently:
CCCL_MAJOR_VERSION >= 3 && CCCL_MINOR_VERSION >= 1
Minor resets to 0 whenever a new major series is cut, so on CCCL 4.x
this evaluates as 4 >= 3 && 0 >= 1, i.e. false. STRIDED_ITERATOR_AVAILABLE
stops being defined and argsort silently falls back to the
init_offsets path. Nothing warns and the build still succeeds, so the
regression is a quiet performance loss rather than a compile error.
CCCL already exposes the version as a single packed integer in
MMMmmmpp form, which is what its own version header uses:
CCCL_VERSION = MAJOR * 1000000 + MINOR * 1000 + PATCH
so 3.4.3 is 3004003 and ">= 3.1" is a plain ">= 3001000". One
comparison, with no component arithmetic left to get wrong.
Checked against a hand-written "version >= 3.1" reference over 2.9.9,
3.0.0, 3.1.0, 3.1.99, 3.2.0, 3.4.3, 3.9.9, 3.99.99, 4.0.0, 4.2.7 and
5.0.0: no divergences. The old guard disagreed at 4.0.0 and 5.0.0.
Verified on RTX 4070 (sm_89), CUDA 13.4, CCCL 3.4.3:
- cmake --build build --config Release: exit 0
- test-backend-ops test -o ARGSORT -b CUDA0: 98/98 passed, CUDA0 OK
Note that a passing regression test does not on its own prove the guard
is still taken, since the fallback path passes too. Preprocessing the
real translation unit confirms the strided-iterator branch is the one
compiled in: counting_iterator is present, init_offsets is not.
Signed-off-by: Heitor <heitorgm@outlook.com>
* Update ggml/src/ggml-cuda/argsort.cu
* Apply suggestion from @ORippler
---------
Signed-off-by: Heitor <heitorgm@outlook.com>
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* server : preserve context checkpoints across slot save/restore
Append the checkpoints after the packed server_tokens payload added in #26640
and count them in n_written / n_read, so a restored slot can still roll back to
a checkpoint instead of re-processing the whole prompt.
* server : drop draft checkpoint data that does not match the draft context
Restoring a slot saved with a different draft KV cache type aborted in
load_dft(). Test-load one draft checkpoint on restore and drop the draft
data if it does not fit, instead of crashing. Adds a regression test.
Co-authored-by: Igor Okulist <okigan@gmail.com>
* server : harden the checkpoint appendix of slot save files
Bound each blob size by the bytes left in the file before allocating, open the
file with UTF-8 paths on Windows like the llama state payload, fall back to full
prompt re-processing when a checkpoint restored from a slot file fails to load,
and replace the 1024 count cap by keeping the last n_ctx_checkpoints while reading.
* server : report an incomplete checkpoint appendix as a failed slot save
Return an error to the client when the appendix cannot be written, like a
failed payload write, and make the oversized-blob test declare a size that
cannot be allocated, so an unbounded allocation fails the test.
* server : reject an empty target state in the checkpoint appendix
A saved checkpoint always holds a target state, an empty blob would roll back
without restoring anything. Also log with the slot id, and load the draft test
model from the HF cache instead of a second download.
* common : return bool from checkpoint load_tgt / load_dft
A checkpoint restored from a slot file falls back to full prompt re-processing
when it fails to load, a checkpoint created in memory still aborts.
---------
Co-authored-by: Igor Okulist <okigan@gmail.com>
The bucket search in topk_nary_search.comp started from the range
[0, 0xFF800000), which ends just below the ordered-uint mapping of +inf,
so +inf and NaN were never counted. A workgroup block with fewer than k
countable values left the ballot empty and the shader read uninitialized
shared state (hang/device lost on NVIDIA, wrong indices on AMD), and a few
+inf in a block were selected without being counted, dropping real top
values.
Map NaN to -inf on input, start from [0, 0xFFFFFFFF) so every value is
counted, and clamp the top bucket's end (2^32) instead of wrapping to 0.
The k = 1 path compared float bits as signed integers, which orders
negative values backwards; compare floats instead.
Add test_top_k_inf to test-backend-ops: negative values, fewer than k
+inf and many -inf, for k = 1, 10, 40.
Assisted-by: Claude Opus 5.5
* cuda : support arbitrary striding for unary ops on f16, f32, and bf16
* Remove added newline
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* sycl: fuse the delta-net alpha gate (add + unary + mul)
* tests: cover the fused add + unary + mul chain
* sycl: give the fused alpha gate a flat path and pin the node skip
* model : support classifier_activation for rerankers
Assisted-by: Claude Opus 5.5
* model : map classifier gelu to gelu_erf and accept tanh
Assisted-by: Claude Opus 5.5
* model : default act_cls to tanh, ModernBERT falls back to gelu_erf
Assisted-by: Claude Opus 5.5
* ggml-cuda: assign two GDN state columns per warp
* ggml-cuda: use 4 GDN state columns per warp at S_v=128
* ggml-cuda: default cols_per_warp=4
* ggml-cuda: address GDN review nits
* hex-bufs: add support for alloc_buffer_n
* hex-bufs: add support for splitting large tensors into separate buffers
* hex-bufs: update GGML_HEXAGON_MBUF to accept three values dyn,static,total
* hex-bufs: bump dyn. default to 512MB since 128MB causes perf regressions with big MOEs
* hex-run: add --no-embd-offload option to simplify command lines on devices that need it
* Update scripts/snapdragon/run.py
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
---------
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
* model : add LiquidAI/d1-omni-600M decision model
Assisted-by: Claude Opus 5.5
* mtmd : keep conformer GLU sigmoid on CUDA
Assisted-by: Claude Opus 5.5
* server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal
Assisted-by: Claude Opus 5.5
* common : rename decision type d1omni to lfm2-d1-omni, server : make images an alias of files
Assisted-by: Claude Opus 5.5
* bugfix: infinite recursion caused by a tool named 'call'(#29967)
* chat : use index for schema and argument rules
* tests : remove tests
* tests : add expect_rules to peg test parser
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
* model : add LiquidAI/d1-3b decision model
mtmd : read LFM2 image resize algo from GGUF
Assisted-by: Claude Opus 5.5
* common : rename decision type d1 to lfm2-d1
Assisted-by: Claude Opus 5.5
The CUDA FWHT covers widths 64 to 512. It runs one row per warp and keeps N/32
values per lane, so wider blocks need more registers per lane than that layout
allows.
fwht_cuda_block runs one row per thread block with 256 threads, so each thread
keeps N/256 values. Stages below the warp width still shuffle, those up to the
block width go through shared memory, and the rest stay in registers. Same
butterfly and sign convention as the warp kernel.
Widths 64 to 512 keep the warp kernel. 1024 through 8192 use the new one, for
both F32 and F16 sources. ggml_cuda_op_mul_mat_use_fwht (the shared
supports_op/dispatch predicate added in #29096) does not check width, so it
needed no change here: any width it admits that ggml_cuda_op_fwht can't serve
already falls through correctly to the cuBLAS path.
Rebased onto current master with #29096's F16 commit underneath it, since this
depends on the same F16 template infrastructure; that commit applied cleanly,
the only conflict was in test-backend-ops.cpp where an unrelated intervening
commit's own test additions landed near this block.
test-backend-ops on an A10 (lambdalabs): MUL_MAT 1297/1297, including all
FWHT/Hadamard cases (18 existing, 4 new F32 wide, 4 new F16 wide, 2 new
many-rows, 1 too-big boundary moved to 16384).
* llama: share the nextn tensor flags between models
Follow-up of the TODO in glm5-next: move the trunk-only and MTP-only
detection that each model copied into a nextn_flags helper of
llama_model_base. It probes the first trunk layer and the first NextN
layer, and adds TENSOR_SKIP when MTP is not loaded. qwen4exp probes
hc_attn_norm since it has no attn_norm.
deepseek4, nemotron-h, qwen35, qwen35moe, qwen3next and qwen4exp now
also accept a trunk-only file, like the other models.
* llama: avoid capturing structured bindings in the nextn flags
Lambdas that capture structured bindings need C++20, and GCC 15
rejects them under -Werror, so the models read the trunk and MTP
flags into plain variables.
The generic few-row MMA kernel works for any type with a 16-weight
dequantizer, so it now also takes BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K,
TQ2_0 and the IQ types. Each type starts at the row count where it beats
the current kernels on an M3 Ultra: 5 rows for TQ2_0, 4 for BF16, 3
for MXFP4, Q2_0, Q2_K and IQ4_NL, and 2 for the others.
test-backend-ops perf -o MUL_MAT, m=4096, k=14336, M3 Ultra, time of this
change over master (mean of two interleaved runs each): 0.23 to 0.98 from
the threshold to 8 rows, 0.24 to 0.33 at 9 to 16 rows, and 0.99 to 1.01 at
1 and 512 rows.
* metal : fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT
ggml_metal_op_mul_mat_mma picks the residual of a fused MUL_MAT+ADD as
"the ADD operand whose op is not MUL_MAT". When both operands of the ADD
are mat-mul outputs (x = W1 @ u + W2 @ v), that test is true for both, so
the residual resolves to the fused mat-mul's own, never-written output and
the kernel adds whatever that buffer holds.
The fusion check (ggml_metal_mul_mat_add_operand) already selects the
operand by identity; make the encoder do the same.
Clef decision models hit this in their head (proj_option_context @ ctx +
proj_option_lexical @ lex, 9 option rows): on Metal, /v1/systemone
probabilities collapse toward uniform (billing 0.28 where the CPU backend
gives 0.977, Cloudflare_clef-flash Q8_0), deterministic per memory layout,
correct with GGML_METAL_FUSION_DISABLE=1. Not a quantization issue: the
same file is right on CPU.
Add a MUL_MAT_ADD mode to test-backend-ops where the residual is a second
mat-mul; on Metal it fails 27 of 28 cases before this change (the one pass
is f16 n=2, under the MMA row threshold, so nothing fuses).
* Update tests/test-backend-ops.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : add GLM5-Next NextN (MTP) graph
Build the GLM5-Next multi-token-prediction head as graph_mtp: the NextN block
embeds enorm(tok)+hnorm(h) through eh_proj, runs one plain DSA layer and the
shared lm_head, reusing the trunk's builders through the no_build tag ctor.
llama_memory_recurrent also tolerates a partial seq_rm when the context holds
no recurrent layers, which is what the MTP draft context needs.
Assisted-by: Claude
* llama : glm5-next: skip dead compute in headless NextN forwards
A NextN forward with no output rows (the MTP catch-up and the draft-context
prefill) persists only through its cache writes, so the headless graph keeps
the MLA latent, indexer key|gate and pooled-key writes and drops the query
path, the indexer selection, the attention body, the FFN and the LM head. The
4-token catch-up falls from 6.9 ms to 0.33 ms of kernels; the greedy output
hashes and the draft acceptance are unchanged.
Assisted-by: Claude
* llama : glm5-next: fix NextN extraction contracts and shared-tail rollback
Three fixes from the architectural review. The headless graph prune now also
requires that no unmasked nextn extraction is live, because that mode reads
n_tokens hidden rows regardless of the logits flags. Masked extraction
publishes the hidden rows gathered by the output ids, so a batch whose output
flags are not a prefix exports the right rows. A partial recurrent rollback
whose tail cell is shared with another sequence is now rejected instead of
silently moving that sequence's tail.
Assisted-by: Claude
* llama : glm5-next: tidy comments in the MTP changes
Assisted-by: Claude
* llama : glm5-next: crop the MTP graph to the output rows instead of pruning it
Replace the headless NextN prune with the crop pattern the other MTP
graphs use: gather the attention output and the block input at the
output ids before the position-wise FFN and the shared head. A NextN
forward with no output rows (the MTP catch-up and the draft-context
prefill) then runs the FFN and the head over zero rows. The 4-token
catch-up falls from 6.9 ms to 2.9 ms of kernels; greedy output hashes
are unchanged.
Assisted-by: Claude
* glm5-next: use the nextn crop helpers in the MTP graph
Replace the local crop condition and the masked select of t_h_nextn
with crop_before_nextn and crop_after_nextn, so the MTP graph narrows
its rows the same way as the main graph and the other models.
Describe the shared cell and empty filter branches of the recurrent
partial rollback.
* glm5-next: load MTP-only and trunk-only GGUF files
Make the trunk tensors optional when the file only holds the NextN
layer, and the NextN tensors optional when the file only holds the
trunk, so the split MTP GGUF loads as a draft model.
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sampling : use greedy selection for eligible temperature-zero chains
Assisted-by: OpenAI Codex
* Apply suggestion from @ggerganov
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sampling: simplify zero-temperature greedy eligibility
Allow the same greedy selection on CPU and grammar/reasoning-budget paths.
Keep distribution sampling for dynamic temperature and requested probabilities.
Cover the common sampler selection and probability behavior in the existing sampler tests.
Assisted-by: OpenAI Codex
* sampling : use greedy selection after final top-k with k=1
Assisted-by: OpenAI Codex
* cont : clean-up
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* convert : Fix token configuration for PLaMo-3
The reasoning and tool calling tags in PLaMo-3 consist of three tokens
each. For example, for reasoning:
* reasoning start: `<|plamo:begin_`, `think`, `:plamo|>`
* reasoning end: `<|plamo:end_`, `think`, `:plamo|>`
Registering `<|plamo:begin_`, `<|plamo:end_`, and `:plamo|>` as
`USER_DEFINED` so that they are parsed correctly.
Also PLaMo-3 models use <|plamo:tag|> as EOT, while PLaMo-2 models
use <|plamo:op|>.
Take EOT token as a parameter and look it up so that PLaMo-2 and
PLaMo-3 can use their appropriate ones.
* use NORMAL instead of USER_DEFINED
* fix mixed different model GPUs issue
* Update ggml/src/ggml-sycl/ggml-sycl.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sycl: accelerate GLM MLA prefill with MKL flash attention
GLM-4.7 Flash uses an MLA shape with 576-wide Q/K heads, a 512-wide
V head, GQA 20, and F16 KV. The SYCL dispatcher rejects this shape
because the normal MKL flash-attention gate requires matching K/V
widths and caps the head dimension at 512, so prompt processing falls
back to the substantially slower TILE kernel.
Admit only the validated 576/576/512, GQA-20 F16 shape to the existing
MKL pipeline. Keep all other mismatched K/V shapes on their current
fallback paths.
Handle GLM's V cache as a narrower strided view of K rows. Select the
strided F16 descriptor when row stride is padded, and alias K/V
dequantization buffers only when their logical widths match. Restrict
the stride exception to a real V view sharing K's row stride.
Add the exact 576/512, GQA-20 prompt-path backend test.
On an Intel Arc Pro B70 at master e613ef2, pp8192 improves from
432.80 to 1292.29 tok/s (2.99x, +198.6%). tg256 remains unchanged
within noise at 45.67 versus 45.65 tok/s. The exact MLA test passes
and debug output confirms MKL dispatch.
* sycl: store MKL flash attention scores in F16
Keep the QK GEMM output in F16 instead of F32. The online softmax still
converts each score to F32 for its max, exponent, and sum, so the
per-element math is unchanged apart from score rounding, and the F32
matrix was being written only to be consumed as F16 probabilities.
The F32 score matrix is the largest flash-attention intermediate on this
path; storing it as F16 halves its size and traffic. This builds on the
coalesced softmax loads from 1aa2954bd, which read each score row
cooperatively, so the smaller dtype pays off.
Measured on an Intel Arc Pro B70 with the dispatch from the previous
commit, -ngl 999 -b 4096 -ub 1024 -ctk f16 -ctv f16 -fa on:
pp8192 1583.9 -> 1657.1 tok/s (+4.6%)
pp64000 610.0 -> 684.5 tok/s (+12.2%)
pp131072 ~354 -> 402.1 tok/s (+13.5%)
tg256 at 8k context is unchanged (32.64), and the FLASH_ATTN_EXT suite
shows no new failures. The exact GLM MLA backend cases pass against CPU.
Adjust the ~354 baseline figure if you prefer citing only measured pairs (the 131k dispatch-only point came from the equivalent maintained build). Optionally add Assisted-by: <tool name> per the contribution guidelines since AI contributed to the change.
* Revert "sycl: store MKL flash attention scores in F16"
This reverts commit 265f9748164dc88e02678425b75f520ccd528b32.
* hex-cpy: replace more paths with dma and simplify l2flush
* hex-cpy: use DMA in all sametype paths
* hex-cpy: rewrite the rest of the copy paths (diff type) to use dma
* hex-concat: use dma for multi-dev path which also removes the need for l2-line alignment
* hex-concat: proper support for mdev splitting
* hex-concat: cleanup ctx and kern params usage
* hex-cpy: cleanup contex and remove left-over non-dma checks
* hex-cpy: clean dma_cpy naming
* hex-cpy: proper kernel params and kernel selection
* hex-build: resolve left-over rebase conflicts
* hex-cpy: update dev guide to clarify 128 byte alignment requirement
* hex-concat: make sure we go through mdev barrier
* hex-concat: make sure to flush dma-queue
* hex-cpy/concat: cleanup kparams and vtcm layout handling
* hex-cpy: remove dead check for contig (routed to diff kernel) and update comments
* hex-dev: update developer guide based on latest changes
* hex-cpy: safe skip of noop copies
* hex-dup: route DUP to CPY
The XIELU CUDA kernel template is already generic over the element
type; only the F32/F16 type assertion and the else-if dispatch were
missing. Add the nv_bfloat16 branch to the launcher, and drop the
temporary supports_op gate in ggml-cuda.cu that rejected BF16+XIELU.
test-backend-ops gains two BF16 cases ([10,5,4,3] and [512,16,1,1]).
docs/ops/CUDA.csv and docs/ops.md are regenerated; the F32 xIELU row
flips from no to yes as well, i.e. the previous record was stale.
Tested:
- Mac CPU: xIELU F32/F16/BF16, 6/6
- Mac Metal: existing F32/F16, 4/4; BF16 still unsupported
- RTX 4090 CUDA: xIELU F32/F16/BF16, 6/6
- RTX 4090 CUDA BF16-only: 2/2
- git diff --check passes
* vocab : implement PLaMo-3 tokenizer pre-segmentation
The PLaMo-3 tokenizer inserts hard boundaries before running the Unigram
DP, around <|plamo:...|>-looking text, and around runs of at least 4
identical characters or 2 spaces. Without them llama.cpp tokenizes code
indentation and repeated punctuation differently from the reference.
Reproduce the two re.sub() passes in llm_tokenizer_plamo2 by encoding
each segment independently.
* add vocab type "plamo3"
* Update src/llama-vocab.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* misc change
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model: K2 Horizon gguf conversion code
* model: loading hparams and tensors in k2-horizon.cpp
* model: K2 Horizon compute graph
* model: K2 Horizon compute graph adjustment and registering tokenizers
* model: K2 Horizon chat template and accomodate safetensors naming
* unicode : add the K2-Horizon pre-tokenizer splitter
The K2-Horizon regex had no arm in unicode_regex_split_custom and fell through to the
general std::regex fallback, which fails two ways.
On MSVC std::regex rejects \p{...}, so no K2-Horizon GGUF loads on Windows at all:
llama-quantize, llama-imatrix and llama-perplexity all abort with
regex_error(error_escape) before a token is produced.
Where the fallback does compile it is still wrong. unicode_regex_split collapses each
codepoint to a single byte naming its Unicode category before matching, and U+200C/U+200D
are category Control, which has no entry in k_ucat_cpt, so both become the 0xD0 fallback
byte. The literal and alternatives in K2's regex can then never match and
every ZWNJ or ZWJ ends a letter run.
The splitter is the existing llama3 one with a single rule widened, since K2's regex
differs from llama3's only in that a letter run also takes marks, ZWNJ and ZWJ.
tests/test-unicode.cpp gains a case for this: it fails before the change with
[Amy] [ZWNJ khaham] and passes after with the run intact.
* tests: expand K2 Horizon unicode splitter coverage
* unicode: handle K2 Horizon case folding and empty input
Assisted-by: Codex
* jinja : support sequence indices in selectattr and rejectattr
Assisted-by: Codex
* model : add K2 Horizon dense and MoVA support
Includes the K2 Horizon implementation from ifm-ai/llama.cpp with converter, tensor-parallel and model save/reload fixes.
Assisted-by: Codex
* chat : support K2 Horizon reasoning and tool calls
Assisted-by: Codex
* conversion: remove obsolete K2 Aurora alias
Assisted-by: Codex
* k2-horizon: enforce response schemas and load YaRN betas
Constrain final JSON after reasoning, accept flexible JSON tool envelopes,
enforce XML dialects, and handle repeated or alternate thinking markers.
Load YaRN beta metadata instead of retaining the default values.
Add schema, streaming, continuation, and model reload regressions. Validate
CUDA and CPU builds and 0.9B, 4B, and MoVA conversation/tool round trips.
Assisted-by: Codex
* renaming template fixture
* adressing cisc follows ups
* desloppify the parser / adress aldehir comments
* clean test-chat
* remove fallback : model trained mostly on high anyway
* fix k2 attn_v_exp tn splitting and metal fusion baseline
* k2-horizon : forward expand views before sums
* k2-horizon: copy embds before group norm to fix TP
* disable tesnor parallelism
---------
Co-authored-by: Ryandito Diandaru <ryandito.diandaru@mbzuai.ac.ae>
Co-authored-by: WestWaters <mario.papaleo2013@gmail.com>
Co-authored-by: Natani L. Mayday <71436458+TaskPuppyNatani@users.noreply.github.com>
Co-authored-by: West <100190545+WestWaters@users.noreply.github.com>
Co-authored-by: aaryamonvikram <aaryamonvikram@gmail.com>
Co-authored-by: aaryamonvikram <96529820+aaryamonvikram@users.noreply.github.com>
The gather path attended over the selected latents with a plain
matmul and softmax. It only ran with n_ubatch <= 16, and the flash
attention backends now skip the masked rows through n_kv_max, so
the scatter path covers every case.
Drop the gather flag, the gathered attention branch and
gather_mla_rows. set_input_kpool always maps padding to the n_kv
sentinel, and the slot mask becomes sel_mask since only the
scatter reads it.
* ggml: fix CLAMP on non-contiguous views (CPU, CUDA)
CUDA clamped ggml_nelements values flat and ignored the view strides.
CPU addressed row j as j*nb01 and ignored nb02/nb03. Both now follow the
strides of dims 1..3; CUDA supports_op requires contiguous rows, like
Metal. test_clamp gains a non-contiguous view case.
* cuda: clamp kernel uses fastdiv for the view strides
A 1.0 loader (e.g. Android 8.1) has no vkEnumerateInstanceVersion, so
backend init called a null pointer. Treat it like any loader under 1.2.
Fixes#29871.
Assisted-by: Claude Opus 5.5
* mimo2 : always emit h_nextn
the other nextn-capable models set it unconditionally
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* models : consolidate nextn row cropping into shared helpers
- replace the duplicated crop conditions and the per-model flags (narrow_early,
crop_before_ffn, crop_last_layer, emit_h_nextn) with two helpers on llm_graph_context:
crop_before_nextn() / crop_after_nextn()
- models that only tested embeddings_nextn_masked now share the same condition, so they
crop the last layer before the nextn capture whenever extraction is off
- t_h_nextn is now set unconditionally in mimo2, qwen4exp and deepseek4 (as in the other
nextn-capable models); host-side reads stay gated by cparams.embeddings_nextn
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
The speculative MTP init enables NextN extraction on the target and draft
contexts after both were created and their schedulers reserved. With
unmasked extraction the trunk graph keeps every token through the last
layer instead of cropping to the output rows, so the first decode
reallocates to that batch's shape and the next, wider batch trips
GGML_SCHED_DEBUG_REALLOC. Invalidate the reserve when the flags change so
the next compute re-reserves with the new graph shape.
Assisted-by: Claude
* ggml: refactor selective expert copying to user code
* tests: enroll two models into selective expert copy test
* tests: use deepseek2 as test model
* improve comment in ggml-backend.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont: fix whitespace
* cont : better comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml-openvino: skip unselected graph branches and support DUP
Upstream #29622 adds a mixed token/embd branch to every input
embedding graph through ggml_build_forward_select(). Its nodes are
not flagged for compute, but the backend translated them anyway,
and the DUP in that branch was unsupported, so the scheduler split
the graph and passed the embeddings across the split with a fixed
token count. The first single-token decode then failed
(test-thread-safety on CPU and GPU).
Build the OV model from the compute nodes only, and translate a
same-type contiguous DUP like CONT so the graph stays on one backend.
* ggml-openvino: make inp_scale_rows token dim dynamic
#29622 also moves the per-token embedding scale (gemma3, gemma3n,
gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim
and pad it per chunk on the static (NPU) path.
* ggml-openvino: skip GPU MUL_MAT op tests with unbound Q4_1/Q4_K weights
Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU
plugin fails to compile that form for some row counts with "clFinish,
error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on
the MUL_MAT cases added in #29869 (e.g. m=1000, n=2, k=1024). Model
weights use a u4 zero point and are not affected.
Report these cases as unsupported on GPU until the plugin is fixed.
Op tests check support before allocating, so the check matches unbound
weights only; model loading probes with a dummy buffer and keeps its
weights on the GPU.
* ggml-openvino: create FILL in the output type
translate_fill always built an f32 constant, so an f16 FILL produced
f32 data and the copy back overran the f16 output buffer. Use the
output type for the constant.
* ggml-openvino: reject CONCAT with a quantized type
Quantized inputs are dequantized when translated, so the backend cannot
write a quantized CONCAT output. Report it as unsupported, as for CPY
to a quantized type.
* ggml-openvino: handle the single recurrent state gather of build_rs
#29856 changed build_rs to gather all recurrent states with one GET_ROWS
on the s_copy leaf and take the ubatch and extra states as views of it.
The stateful path matched only the previous form, a GET_ROWS per view of
s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU
("is_axis_valid(axis, r)" in a Concat).
For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the
active-state gather, keep the rank-4 layout of reshapes that read a view
of it, and map the copy of the empty extra-state view to the single-slot
remainder writeback. Do not warn about the dynamic dim of empty views.
* openvino: align eltwise operand ranks to work around a GPU-plugin defect
* openvino: match the MoE fusion on the rank-3 stateful graph
* ggml-openvino: do not unsqueeze an RMS norm output in AlignEltwiseOperandRanks
The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract
whose operand ranks differ. In gemma-3 the lower-rank operand of the
post-attention residual add is the norm output, and unsqueezing it makes
the GPU plugin compute the layer wrongly: gemma-3 returns empty answers
on GPU with stateful execution.
Skip the rewrite when the lower-rank operand is an RMS norm output.
* docs : update OpenVINO validated models
---------
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
* llama: re-pool each shared k-pool rep once
With shared cells every pool is re-pooled, and since the pooled keys
are always scattered, the pools a seq_cp shares between sequences
wrote the same rep row from several scatter entries, a data race on
the CPU backend. Mark each rep once: the sharing sequences read the
same row through pool_cells.
* llama: assert whole-sequence seq_cp in the hybrid idx memory
The recurrent state is always copied whole whatever the range, and a
k-pool cell shared by a partial copy could carry two pool groupings
with a single pooled row. Every caller copies whole sequences, so
reject partial ranges instead of supporting them.
* llama: drop the k-pool cache_safe mode
With whole-sequence seq_cp, sequences sharing cells share their pools
too, so the pooled row of a shared rep is valid for all of them. Mark
each rep once in every ubatch instead of re-pooling everything while
cells are shared, which removes the sharing scan and the stale-all
workarounds in seq_rm, state_read and state_drop. seq_cp now only
stales the destination.
* hexagon: head-parallel flash_attn partitioning for row-split multicore
In row-split mode each core computes its output row shard of every
MUL_MAT, but flash_attn was previously partitioning by Q tokens
(flat qrow split) instead of by heads. This forced every core to
read the full KV cache (all n_kv_heads), negating the memory
bandwidth benefit of multicore on flash_attn.
Change both HMX and HVX flash_attn kernels to partition by KV heads
when n_kv_heads is divisible by n_cores: core i processes heads
[i*n_kv_heads/N, (i+1)*n_kv_heads/N) exclusively, reading only its
head shard of the KV cache. Falls back to the original token-block
split when n_kv_heads % n_cores != 0 (e.g. Gemma-4 with 2 KV heads
on 4 cores).
Controlled by GGML_HEXAGON_FA_HEAD_SPLIT (default 1 = on).
The flag is packed into bit 1 of the existing is_dst_fp32 kparams
byte to stay within the 128-byte kernel_params blob limit.
Measured gains at 4c row-split (PP t/s, ubatch=1024):
Qwen3-0.6B: 6977 -> 11026 (+58%)
llama-3.2-3B: 3717 -> 5522 (+49%)
Qwen3.5-4B: 2739 -> 2855 (+4%)
Gemma-4 MoE: no change (MoE FFN dominates, fallback path)
TG is unchanged (flash_attn is a small fraction of decode time
relative to the matmul+barrier cost per layer).
* hex-fa: cleanup kern_params and head-split selection
* hex-fa: add -fa-head-split option to run.py
* hex-mdev: update matmul solver to account for reduced work in row-split scenarios
* hex-mmid: better work splitting by expers in multi-dev scenarios
* hex-fa: update HMX gating based on the model/n-hvx/ctx-len sweep
* hex-fa: precompute softcap/scale on the host
* hexagon: flatten matmul into 2d to use HMX in multi-sequence
* hex-mm: cleanup kparams and use collapse to 3/4D -> 2D mapping
* hex-mm: fix typo in collapse fallback
* hex-mm: another pass at consistent naming for act tensors
* hex-mm: add support for colapsing dims in fused matmuls
* hex-build: fix WoS build errors
* hex-mm: make sure to enforce dst stride in can_collapse
* hex-fa: add a onliner commit for head-split check
* hex-fa: remove unused local head_split var
* hex-fa: tighten up can_split checks
* hex-mm: update unfused paths to use act instead src1
* hex-mm: make sure to check all dsts for splitting
* hexagon: fix the second weight chunk address in the batched HMX matmul prologue
* hexagon: F16 activation and ragged N in the HMX matmul
* hex-mm: tighten the ragged/split checks in mdev cases
* hex-mm: enable MM fusion for F16 activations
* hex-mm: pass tiled sizes to the solver in fused paths
* hex-mmid: remove scalar divs from expert mapping loops
* hex-mmid: proper cacheline safety enforcement for mdev splits
* hex-mm: improve solver for mdev split scanarios and tail handling
* hex-mm: remove redundant checks
* hex-mm: fix fused HMX MUL_MAT_NX drops the final partial tile for quantized weights
* hex-mm: better handling of ragged shapes (removes scalar memset of vtcm)
---------
Co-authored-by: ebateni <ebateni@qti.qualcomm.com>
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* cuda: stage the lightning indexer queries in head passes for MUSA
MUSA archs 21 and 22 cap static shared memory at 28 KB, and the tile
kernel staged the queries of all four heads next to the key tile for
33 KB. The queries are now staged in passes of
LIGHTNING_INDEXER_TILE_HEADS_PER_PASS heads: two on MUSA for 25 KB,
four elsewhere where the single pass folds to the previous kernel.
* cuda: use the vector lightning indexer kernel on MUSA
Address review from am17an: the tile kernel stays off MUSA, whose archs
21 and 22 cap static shared memory at 28 KB, below the 33 KB the tile
needs, so MUSA keeps the vector kernel it ran before. This replaces the
head passes, CUDA and ROCm run the merged kernel unchanged.
* server: support vision input for Clef
* move input_attn_causal to private
* extend old server_batch::embd
* server_batch::token::pos to multi dim
* nits
* fix abort
* fix img tokens cap
* fix yield_to_queue mutate data
* server: reject partial media truncation
* server: keep only the keep_first fix
Drop the mtmd test helper change, which no longer builds since
clip_image_f32_batch stores its entries by value, and drop the
vision test: no test fixture reaches a cut between two adjacent
media chunks with a reused cache (tinygemma3 uses SWA and wraps
images in text tokens, tinyopenjev and small-test are recurrent),
so the test passed or failed independently of the fix.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* vulkan: sparse flash attention for quantized K/V
Assisted-by: Claude
* vulkan: single-scan sparse FA index compaction
The compaction ran one workgroup per mask row and walked the row in
BLOCK_SIZE chunks, with a workgroup scan per chunk. For decode that is
one workgroup doing KV/1024 barrier-bound iterations, so at 128k cells
it cost more than the sparse attention it feeds.
Split the row into contiguous segments instead: one per subgroup with
ballot counting over coalesced loads, or one per thread without
subgroups. A single scan over the segment counts then gives each
segment its output offset. The index list stays ascending.
* llama : fix unexpected graph reallocation in the k-pool models
Both k-pool models built a graph shape that depends on state the
full-context reserve cannot know:
- qwen4exp branched on inp->cache_safe, which turns false as soon as
llama_memory_seq_cp shares cells (e.g. batched-bench -pps): the QSA
layers swapped scatter+gather for fill+concat and dropped the
new_pool_rep leaf, so the decode graph had 12 fewer nodes than the
reserved one
- glm5-next branched on gather = n_tokens <= 16 && n_kv > n_sel, so the
TG decode built the gather shape (7564 nodes) while the last reserve,
the PP one, had the dense shape (7762 nodes)
Either mismatch forces a decode-time re-reserve that drops the
worst-case sizing and bakes in the current state, so the next state
growth (n_pool, n_kv, n_new) needs more room at an unchanged graph size
and aborts under GGML_SCHED_DEBUG_REALLOC=1. Reproduce with, e.g.:
GGML_SCHED_DEBUG_REALLOC=1 ./bin/llama-batched-bench \
-hf ggml-org/GLM-5.3-Flash-GGUF:Q2_K -npp 2500 -ntg 32 -npl 1,2 \
-c 32768 -pps -kvu
Always scatter+gather the pooled keys, and pick gather from context
constants only: n_ubatch bounds every ubatch, top_k + kpool - 1 bounds
n_sel. Every graph of a context then shares one shape, which the
reserve covers, and the dense path measured faster than the gather path
at 2.5k and 16k context.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* llama : drop the unused k-pool cache_safe graph API
The k-pool graphs no longer branch on cache_safe, so nothing reads
get_kpool_cache_safe() or the conditional new_pool_rep any more: both
models always pass the scatter target, which set_input_kpool now
requires instead of merely preferring.
Also drop the cache_safe copy in kpool_build_sizes(), a sizes-only
helper. The layout and state flag itself stays, it still decides which
pools a layout with shared cells must re-pool.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* tests : add a shared-seq graph reserve regression test
Decode a prompt into seq 0, share its cells with seq 1 via
llama_memory_seq_cp (what llama-batched-bench does for -pps), then keep
decoding both sequences. For the k-pool models sharing clears
cache_safe, which changes the graph topology while the pools keep
growing, so a scheduler that re-reserves with the current state
instead of the worst-case one aborts under GGML_SCHED_DEBUG_REALLOC=1.
The test registration sets that flag, and the test aborts on both
k-pool models before 2220411ec1.
kimi-linear and minimax-01 are skipped: they reserve the final pp graph
with n_seqs = 1 (see [TAG_RESERVE_DIAG_DECAY] in llama-context.cpp), so
every multi-seq graph has a different layout and re-reserves by design.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : add TODOs
* cont : fix comment
* cuda: match the moe weighted reduction on empty ubatches
ggml_cuda_match_moe_weighted_reduction rejected tensors with zero
rows. A ubatch without outputs shrinks the last layer to zero rows
through inp_out_ids, so graph_optimize dropped its alloc dep there and
the scheduler graph lost one node compared to the reserved one. The
scheduler then re-reserved at the size of that ubatch, and the next
ubatch with the same node count but larger tensors aborted under
GGML_SCHED_DEBUG_REALLOC=1.
The compute loop already skips empty nodes before trying any fusion,
so the guard only made the alloc deps depend on the row count.
* tests: build the rollback test only where internal symbols link
The shared-seq case calls llm_arch_from_string, which libllama does
not export through LLAMA_API, so linking test-recurrent-state-rollback
fails on Windows with shared libraries. Its build now sits in the
NOT WIN32 OR NOT BUILD_SHARED_LIBS block, next to test-llama-archs and
the test registration it already lives under.
* tests: skip archs by name in the shared-seq reserve test
The skip of kimi-linear and minimax-01 went through llm_arch_from_string,
which libllama does not export through LLAMA_API, so the test could not
link on Windows with shared libraries. It now compares the
general.architecture string directly, and the test builds on every
platform again.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* kv-cache: save exact KV rotation metadata, reject restoring mismatched rotation
* tests : move the state rotation test to test-save-load-state
the test is now part of the save/load test matrix and runs against
every model under test, like the rest of the suite
it probes the KV cache type combinations supported by the model and
treats models that do not use attention rotation as passing vacuously
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : skip unsupported KV caches
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60
On the SpacemiT X60, IME matrix acceleration only covered Q4_0/Q4_1/Q4_K.
Q8_0 had no IME1 kernel, and since the SpacemiT build sets
GGML_CPU_REPACK=OFF there was no repack path compiled in either, so Q8_0
had no accelerated path at all and ran roughly ten times slower than
Q4_0 for prefill on the same board.
- add make_block_q8_0x16 and the Q8_0 repack entry: interleave the
weights into the 16-column layout the IME1 vmadot sequence expects
- add ime1::gemm_kernel_i8i8, an int8 x int8 IME1 kernel with a
single-row and a 4-row A path; the 4-row path loads each B panel once
and reuses it across 4 rows of A
- add quantize_a_4row_i8 for the 4-row activation quantization
- wire both into forward_mul_mat and the repack factory for Q8_0
- docs: mark Q8_0 as supported on X60
Correctness was checked against a quant-exact integer reference for
K = 32 up to 4096, with a max relative error of about 1e-6, and by
checking that generation stays coherent across several prompts.
Tested on Milk-V Jupiter (SpacemiT X60), Bianbu 2.1.1, gcc 14.2, with
Qwen2.5-0.5B-Instruct Q8_0. llama-bench -t 4 under taskset -c 0-3, 5
repetitions on an idle board: pp128 goes from 10.70 to 93.87 t/s. Q4_0
is unchanged at 106.40 -> 107.51 t/s, as expected since this does not
touch that path.
* ggml-cpu : move q8_0_16x32 decl to IME1 section
* ggml-cpu : align q8_0 IME1 kernel assignments
* cuda: tile the lightning indexer over keys and tokens for 4 heads
With too few heads for a wmma tile, a block scores 64 keys against 8
tokens: the keys are staged once in half precision, the queries one
head at a time, and each thread owns one key for two tokens, so no dot
product needs a cross thread reduction. Batches smaller than a token
tile keep the vector kernel. test-backend-ops measures 4 heads.
* cuda: multiply the lightning indexer tile in float
Address review from am17an: the half2 products overflow once a single
q * k exceeds the f16 range. The queries stay in float in shared memory
and each half2 of keys is widened once for both tokens, so every
product and sum is computed in float.
* cuda: widen each lightning indexer key once for all heads
The tile kernel stages the queries and weights of every head at once,
so each key element is widened from half once and feeds all heads,
with a single barrier. F16 keys are copied into the tile without a
float round trip. Keeping the keys in float in shared memory measures
slower, the occupancy drops.
* cuda: stop the lightning indexer tile from spilling registers on ROCm
Each thread of the tile kernel now scores two keys for a single token,
so a warp shares its token and the query reads are broadcasts: six
shared reads per element pair instead of nine for the same products.
The inner loop is unrolled by 8, which keeps gfx908 at 63 VGPRs with no
spill where the fully unrolled loop needed over a thousand, and makes
the kernel 36x faster on an R9700 and slightly faster on CUDA.
* metal : few-row MMA mat-mul and batched copies for speculative decoding
Speculative decoding verifies a few draft tokens per step. Without the tensor API, Metal ran these mat-muls with the mat-vec kernels, whose time grows with every src1 row, so DFlash2 decoding on an M3 Ultra was slower than serial decoding.
- add mat-mul kernels for 2..16 src1 rows on 8x8 simdgroup matrices: each weight is dequantized once for all rows, and the simdgroups of a threadgroup split K. Q4_0, Q8_0 and Q5_K have their own kernels, F32, F16, Q4_1, Q5_0, Q5_1, Q4_K and Q6_K use a generic path over the 16-weight dequantizers, and Q4_0 at 2 rows uses a 2-row variant of the mat-vec kernel
- use them only on MTLGPUFamilyApple7+ without the tensor API, from the row count at which they beat the mat-vec kernels on an M3 Ultra (F32: 6, F16, Q4_K, Q5_0, Q5_1: 3, other types: 2)
- fusion table: MUL_MAT + ADD adds a same-shape residual in the MMA store, and up to 16 adjacent same-layout f32 copies between the same two tensors run as one dispatch
- the fusion checks and ggml_graph_optimize take the device props, so the reorder packs MUL_MAT + ADD only on devices that can fuse it, at every src1 row count
- views do not count toward GGML_METAL_FUSION_MAX when the reorder packs a group, so 16 recurrent state snapshot copies with views between them stay one group
- the encoder checks the inner nodes of a fused group for concurrency, tracks written views by their extent, and does not count the destination of a CPY as a read
- CONCAT splits long rows across threadgroups when there are few rows
- tests: few-row MUL_MAT, MUL_MAT_ADD, CPY_BATCH and CONCAT cases in test-backend-ops (with a prepare_graph hook for the copy order), test-metal-graph-optimize, test-metal-cpy-batch-alias
* metal : remove the CPY_BATCH fusion and the memory range changes
Remove the batched copy fusion with its kernel and tests, and revert the
memory range changes, as suggested in review. The memory ranges, the
graph reorder and the CPY encoder are again the same as on master.
* cont : clean-up
* cont : drop has_tensor gate
* cont : clean-up operand/residual logic
* cont : drop Q4_0 ne11=2 special-case
* cont : add kernels/mul_mv_mma.metal
* cont : consolidate mma pipeline selection logic
* cont : decouple fusion logic from device props
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* log, server: make router child lines carry their own colors
The logger writes the color reset after the trailing newline, so the
reset opens the next line. On the shared pipe of a router child it lands
in front of the next state command, which the router then misses, and
the line break that works around it shows up as an empty log line on
every progress update.
The reset now goes before the trailing newlines, so every line is self
contained and the command goes back to its plain framing. The router
passes its effective color setting to its children, whose output ends
up in its terminal, and leaves that option out when comparing presets
on reload.
* log: enable ANSI colors on the Windows console
A Windows console renders ANSI sequences only in virtual terminal mode,
which nothing turns on for the logger, so llama-server prints raw escape
codes on the Windows 10 console while llama-cli, whose console code
enables it, shows colors. The logger now enables virtual terminal mode
on stdout and stderr when it turns colors on, and keeps colors off when
a console cannot render them. Pipes and files take the sequences as is.
* server: separate the router child commands from its logs
The child sent its state commands on the same pipe as its logs, so the
router had to pick them out of the log stream by a line prefix, and any
unterminated write in front of a command made the router miss it. This
resolves the TODO at the spawn that called for splitting stdout and
stderr.
The child now keeps stdout for the commands and points everything else
written to stdout at stderr, before anything is written. The router
reads both pipes, handles the commands from stdout and forwards stderr
as the log, and warns about any other line on the command pipe.
* server: address review from ngxson
The single server_child is now created first in the entry point and its
constructor keeps stdout for the commands, so the stream is a member of
the instance instead of a static, and init() is gone. The instance is
passed down to the server, while the CLI entry point creates its own.
* Update tools/server/server.cpp
---------
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
* llama: support both embd + raw tokens in batch
* add to test-llama-archs
* also check case llm_arch_supports_mixed_batch = false
* constant graph topology
* have dedicated input for mixed case
* rm set_tensor_backend
* is_embd --> type
* consolidate m-rope pos handling into one place
* nits
* ggml-cpu: vectorize BF16 K tails in tinyBLAS
* tests: Skip tinyBLAS when use_ref is enabled so CPU tests compare against the vec_dot path.
* ggml-cpu: vectorize tinyBLAS F16/F32 tails
A TOOL_ID node that arrives after TOOL_CLOSE wrote through `current_tool`,
which still pointed into the just-destroyed `pending_tool_call` optional
(use-after-free, then a second free of the id buffer). Clear the pointer on
reset.
This is part of the fs::path modernization series.
That was also the opportunity to remove fs_list().
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* chat : honor json_schema in Ling 3.0 parser
Ling 3.0 only built a grammar for tool calls and did not handle inputs.json_schema, so response_format requests were left unconstrained.
Add an eager response-format grammar path with precedence over tools, following the existing parser patterns. Require </think> before JSON when thinking is enabled and do not allow trailing prose after the JSON response.
Fixes#29652.
Assisted-by: Claude Opus 5.5
* chat : require Ling 3.0 think block for response formats
build_rs gathered the extra states (n_rs - n_seqs rows) with their own
get_rows. The worst-case reserve has n_rs == n_seqs, so that node was
sized at zero rows, and any ubatch whose cells are not contiguous forced
a graph reallocation at an unchanged node count, which aborts under
GGML_SCHED_NO_REALLOC.
A single get_rows now gathers the n_rs states: the ubatch states and the
extra states are views of it, and its size only depends on n_rs, which
the reserve already sets to the maximum. A custom getter (mamba ssm_scan)
gathers from the second state, so a single sequence ubatch copies no
state. The views are built once per graph in the input to keep the host
overhead of the graph unchanged.
* ggml-openvino : Qwen3.5 MoE perf (#312)
Squash of ravi9/llama.cpp#312:
- ggml-openvino: add detailed inference profiling (Yu, Zijun)
- ggml-openvino: use remote output tensors by default (Yu, Zijun)
- ggml-openvino: optimize single-sequence recurrent state (Yu, Zijun)
- opt1: remove recurrent reset for single sequence, opt2: direct gdn outputs (break parallel sequence) (Yu, Zijun)
- fix parallel sequences (Yu, Zijun)
- ggml-openvino: simplify graph cache key (ynimmaga)
- enable stateful for qwen35 single sequence (Yu, Zijun)
- Fix after rebasing (Yu, Zijun)
- Add k-requant option q4_asym64 (Yu, Zijun)
- Fix qwen35 llama-bench -p 0 (Yu, Zijun)
- Simplify RESHAPE translation (Yu, Zijun)
- openvino: fuse MoE routing (Yu, Zijun)
- openvino: fuse GDN qk normalization (Yu, Zijun)
- openvino: enable GPU MoE fusion by default (Yu, Zijun)
- ggml-openvino: add cache_only mode to import cached compiled model on disk directly (Yu, Zijun)
- openvino : report the device allocation limit to ggml (Łukasz Ślusarczyk)
- Fix windows build (Yu, Zijun)
Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com>
Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
* ggml-openvino: Update doc of compiled model cache
* openvino: implement PRD-compliant device enumeration and memory reporting
* openvino: fix multi-device listing issues from review
- Only the device selected by GGML_OPENVINO_DEVICE reports as GPU; the
other OpenVINO devices report as IGPU so llama.cpp does not offload to
them. Initializing a non-selected device logs a warning.
- Name devices OPENVINO<i> again and show the OpenVINO id in the
description. Raw "CPU" names shadowed the ggml CPU backend.
- Support GPU.N: create the OpenCL queue on OpenVINO's own context for
the selected device, and replace "GPU"/"NPU" string comparisons with
ggml_openvino_is_gpu()/ggml_openvino_is_npu().
- An unavailable GGML_OPENVINO_DEVICE is now an error that lists the
available devices, instead of silently falling back to CPU.
- Memory: cap iGPU/NPU free memory at system available memory, fall back
to system memory instead of 0/0 when the plugin lacks memory
properties, and ignore host USM allocations in GPU usage.
- Initialize the device config once under a lock, even if OpenCL setup
fails.
- Fix supports_op return type for non-selected devices (build error).
* openvino : take USM entry points from the selected device platform
clGetExtensionFunctionAddressForPlatform was called on the first platform
returned by clGetPlatformIDs. The address it returns is only valid for the
platform it was queried on, and the first platform is not always the one that
holds the device OpenVINO selected.
On a host whose first platform comes from another vendor the lookup returns
null, and then every read, write and memset on a GPU buffer fails with
"clEnqueueMemcpyINTEL not available".
Look both entry points up in init(), on the platform of the device OpenVINO
picked, and keep them in the device config next to the command queue.
Assisted-by: Claude Opus 5
* openvino: fuse MoE experts for models with a fused gate_up weight
FuseMoeCompressed only matches models whose gate and up projections are
separate GatherMatmul ops. gemma-4 packs both into one expert weight and
splits the result after the GEMM, so its MoE block stayed unfused and ran
the expert GEMMs as per-token GEMVs.
Add FuseMoeCompressedFusedGateUp, which matches that shape
(one GatherMatmul -> Slice/Slice -> Gelu(ERF) -> Multiply) and folds it into
the same MOECompressed op, using GEMM3_SWIGLU with GEGLU_ERF. The fused
weight, scale and zero point are split into gate/up halves by copying raw
bytes, since a graph Slice would be rewritten to StridedSlice and constant
folded, whose reference evaluator crashes on sub-byte types.
gemma-4 also applies a per-expert output scale to the down projection before
the router weights. MOECompressed takes only one per-expert weight, so that
scale is folded into the routing weights, which is exact.
The op reads the zero point straight off a weight port and needs an integer
Constant there, so the matcher requires one and leaves natively quantized
experts (exact f16 zp) to the unfused path.
gemma-4-26B-A4B on Arc B390, GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all,
llama-bench -p 512 -n 128 -r 2, against a GGML_OPENVINO_MOE_OP=0 baseline:
pp512 66.16 -> 1608.73 t/s, tg128 25.94 -> 26.46 t/s. Perplexity over 12
chunks is unchanged (1451.3 +/- 177.9 unfused vs 1427.6 +/- 175.1 fused).
No effect without that requant option, on models with separate gate/up
weights, or on CPU. test-backend-ops -b OPENVINO0 is unchanged by this
commit: two MUL_MAT_ID m_v cases fail, the same two on the unmodified base.
* openvino: fix rank-3 axis handling so MoE works under stateful execution
Stateful execution drops the leading size-1 batch dim, so OV tensors are rank
3 while GgmlOvDecoder::get_shape/get_stride still report GGML_MAX_DIMS=4
reversed entries. Several MoE ops derive OV axis indices straight from that
metadata, so they picked the wrong axis. A MoE model with
GGML_OPENVINO_STATEFUL_EXECUTION=1 aborts while building the graph:
Check 'is_axis_valid(axis, r)' failed at src/core/src/validation_util.cpp:336
While validating node 'opset11::TopK ... _ffn_moe_probs ...'
Axis 3 out of the tensor rank range [-3, 2].
Fix idiom throughout: take the axis from the real OV rank, or shift a
metadata-derived axis down by metadata_rank - actual_rank.
argsort.cpp the router top-k axis is 2 on rank 3, not 3. This is the
abort quoted above.
add.cpp the MoE expert-sum bypass collapses the 8-ADD chain into one
ReduceSum on hardcoded axis 2, which on rank 3 reduces n_embd
instead of the expert axis. Now rank-2, with the following
Unsqueeze at rank-3.
get_rows.cpp squeezing a hardcoded {0,1} also strips the batch dim
whenever it is 1, which is every decode step. Squeeze down to
the trailing two dims instead.
mul_mat_id.cpp pick the reshape dims by actual rank, and skip the trailing
Unsqueeze that re-adds the batch dim.
view.cpp the expert-plane slice had the Slice axis, dst_ov_axis, the
ShapeOf+Gather index and the Reshape target all rank-4.
utils.cpp process_view_input_new's "translate_view already resolved
this VIEW, skip re-slicing" shortcut required equal ranks. 4
vs 3 never matched, so every resolved expert plane got
re-sliced. Now compares the common trailing dims. Same axis
shift for the Slice in the view-chain walker.
Stateless is unchanged by construction: every edit is gated on the actual
rank, so axis_shift == 0 reproduces the previous code exactly. Checked on
OV-CPU by diffing greedy output against the unmodified base for dense
gemma-4-E2B, granite-1b-a400m and gemma-4-26B-A4B; all identical.
granite-1b-a400m on OV-CPU aborts with the error above before this change;
after it, it generates and is byte-identical to stateless. Dense gemma-4-E2B
is identical stateless vs stateful both before and after. test-backend-ops
-b OPENVINO0 is unchanged: two pre-existing MUL_MAT_ID m_v cases fail, the
same two on the unmodified base.
gemma-4-26B-A4B is a poor correctness vehicle here. On OV it already drifts
into degenerate repetition a few tokens in, in stateless as much as stateful,
and the two modes diverge somewhere inside that degenerate region instead of
matching token for token. Each mode is self-reproducible across runs.
Known limitation: FuseMoeCompressedFusedGateUp does not match the rank-3
graph, so a MoE model run with GGML_OPENVINO_STATEFUL_EXECUTION=1 loses the
prefill fusion while gaining decode. gemma-4-26B-A4B on Arc B390,
GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all, llama-bench -p 512 -n 128 -r 2:
unfused (GGML_OPENVINO_MOE_OP=0) pp512 66.16 tg128 25.94
fused, stateless (default) pp512 1608.73 tg128 26.46
fused, stateful pp512 66.18 tg128 29.91
Stateful is opt-in and off by default, and MoE did not run there at all
before this, so nothing that previously worked regresses. Making the pass
match rank 3 is the follow-up.
* OpenVINO Backend: Upgrade graph cache to use node_idx, src_idx, node type
* ggml-openvino : enable more comprehensive conv fusion
* enable conv ops
* Reject kernel size 0 and support IM2COL_3D
* openvino : abort when the GPU remote context cannot be created
init() logged the error and returned, which left the device name a GPU but
remote_context empty. The remote buffer and tensor paths assert only on the
device being a GPU and then dereference that empty optional.
Those paths have no host fallback, and a device that OpenVINO listed should
have a working OpenCL context, so stop instead of continuing. An OpenCL stack
that is broken as a whole is still caught earlier by the device availability
check, which falls back to CPU.
Assisted-by: Claude Opus 5
* openvino : fix build warnings
The single-argument form of the OpenVINO RTTI macros is the intended one, but
their selector macro leaves __VA_ARGS__ empty, which -Wpedantic reports on
every pass and op header. Turn that warning off for this backend only, the
way ggml-cuda and ggml-sycl already do for their own third-party warnings.
Also drop a break and a dead assignment around a GGML_ABORT, which is noreturn.
Assisted-by: Claude Opus 5
* OpenVINO Backend: Support common MTMD ops
* ggml-openvino: give a reshaping view its own ov::Tensor
* ggml-openvino : compute HARDSIGMOID and EXPM1 in f32
HARDSIGMOID used a 1/6 constant in the input type, which is not exact
in bf16, and EXPM1 lost precision for small inputs in f16. Both now
compute in f32 and convert back, except on NPU where the f32 path
gives wrong results.
Fixes the HARDSIGMOID/EXPM1 test-backend-ops failures on GPU.
* ggml-openvino : update device selection and --list-devices
Show the selecting GGML_OPENVINO_DEVICE value and active device in
--list-devices, startup logs, and backend tests.
Clarify OpenVINO selection uses GGML_OPENVINO_DEVICE, not -dev.
* openvino : remove unreachable OpenCL queue checks
A remote buffer exists only on a GPU device, and init() aborts there if the
queue cannot be created, so the queue is never null at these call sites.
Assisted-by: Claude Opus 5
* openvino : update OpenVINO to 2026.4.1 and GPU drivers to 26.35.39758.10
* docs : update OpenVINO validated models and GPU driver version
* ggml-openvino : skip empty views when giving a reshaping view its own tensor
A zero-size view can sit at the end of a GPU USM buffer (Qwen3.5 recurrent cache). Wrapping it as a remote tensor throws "shared USM buffer has smaller size (0)".
Assisted-by: Claude
* ggml-openvino : rebind the cached decoder when llama passes a different graph
llama keeps separate graphs for batches with and without outputs. llama-server splits the prompt into chunks for context checkpoints, so a cached decoder could be reused with a graph built in other memory and bind the previous chunk's input tensors. SWA and recurrent models then lost most of the prompt in llama-cli and llama-server.
Assisted-by: Claude
* docs : update OpenVINO validated models
Smoke test on Lunar Lake (32 GB) with the two fixes above. Re-add the Qwen3.5 and gemma models.
Assisted-by: Claude
---------
Co-authored-by: Yu, Zijun <zijun.yu@intel.com>
Co-authored-by: ynimmaga <ynimmaga@users.noreply.github.com>
Co-authored-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
Co-authored-by: haarika-madaka <haarika.madaka@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
* qwen4exp : halve the indexer score memory
The indexer scored all heads in one product and rectified a copy of it,
so two [n_pool, n_idx_h, n_tokens] f32 tensors were live at once, the
largest buffers of the graph at long context. Each head now gets its
own product, rectified and summed in place into one [n_pool, n_tokens]
score.
* qwen4exp: let the allocator reuse the indexer score buffers
Address review from CISC: use plain ggml_add and ggml_relu in the
indexer head loop. The graph allocator already runs them in place when
their source has no other consumer, so the _inplace variants are not
needed. The compute buffer and the speed are unchanged.
* cuda: support 4 heads in the lightning indexer
Dispatch 4 heads to the vector kernel, too few for a wmma tile, and
accept them in supports_op. test-backend-ops covers 4 heads.
* metal: take the lightning indexer head count as a function constant
The kernel reads the head count from a function constant and zero fills
the last head tile, so any head count runs and 64 heads is unchanged.
* qwen4exp: compute the indexer score with the lightning indexer
Address review from am17an: the unweighted sum of the rectified head
scores scaled by 1/sqrt(head_dim) is the lightning indexer with every
head weight set to that scale, so the indexer calls
ggml_lightning_indexer on the pooled keys with an f16 pool mask. The
keys are read once for all heads and no per head score is
materialized.
* vulkan: tile the lightning indexer over keys and tokens
A workgroup scores 64 keys against 8 tokens: the keys are staged once
in shared memory, the queries one head at a time, and each invocation
owns one key for two tokens, so no dot product needs a cross invocation
reduction. The subgroup variant and the flat dispatch are gone, the grid
is keys x tokens x streams.
* vectorize vulkan loads and use fp16 dot product
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* Make the drafter probabilistic and the target verify by rejection sampling
* Drop stale spec_draft_q before drafting
* Fallback to argmax sampling for grammar-constrained requests and adding flag for enabling probabilistic draft sampling. Default flag value is greedy.
* Support grammar-constrained requests in rejection sampling
* Fix - renormalize distribution after masking
* copy rng on sampler copy and re-accept drafted tokens on replay
* Fix draft sampler sharing the target's rng stream
* Simplify the rejection sampler's inputs and move replay to the server
* Truncate the draft candidates along with the draft
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* ggml-quants : avoid invalid rounding in qkx3 scale search
The imatrix scale search can produce an infinite, NaN, or otherwise out-of-range value when the fitted minimum collapses to the maximum or makes the range extremely small. That value is then passed to nearest_int and can trip its assertion in Debug builds.
Clamp the quantization level to [0, nmax] before rounding so valid in-range values behave the same as before while invalid scale-search results no longer reach nearest_int.
Add regression coverage for degenerate imatrix groups across q2_K, q4_K, q5_K, q4_1, and q5_1.
Fixes#29804.
Assisted-by: Claude Opus 5.5
* tests: print degenerate imatrix quant types
* ggml-cpu : fix soft_max_back wrong output when dst aliases src1
GGML_OP_SOFT_MAX_BACK is listed in ggml_op_can_inplace, so the graph
allocator may assign dst to alias either src0 (dy) or src1 (y).
The result was built in several steps:
ggml_vec_cpy_f32 (nc, dx, dy);
ggml_vec_acc1_f32 (nc, dx, -dot_y_dy);
ggml_vec_mul_f32 (nc, dx, dx, y);
ggml_vec_scale_f32(nc, dx, scale);
When dst aliases src1, the first step overwrites y and the third step
then reads the overwritten values, so the output is silently wrong.
Aliasing dst with src0 is unaffected. The CUDA kernel completes its
reduction before writing and is already safe.
Replace the sequence with a single fused loop that reads both sources
before writing, which is correct under either aliasing.
Add a regression test that marks dy as a graph output so the allocator
is forced to alias dst with y, asserts that the alias actually
happened, and compares against values computed on the host.
* cont : remove comment
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* metal : add tensor API flash attention kernel for F16 KV
* cont : add tensor FA kernels for DK=DV=512 and DK=576, DV=512
* cont : support attention sinks, ALiBi and logit softcap in the tensor FA kernel
* cont : add tensor FA kernel for DK=192, DV=128
* init conversion
* convert: ok
* model loaded
* add server code
* improve conversion script
* support shared prompt prefix
* add docs, imorove UX a bit
* add vision support
* add openjev tiny model for testing
* add dev docs
* support lev & kev
* clean up
* fix lev noul
* fix py lint
* nits docs
* clarify about not supporting date_facts
* Adding wide-load mmvq for Q8_0 and esimd dmmv for q8_0
Assisted-by: Codex
* remove guard for q8_0
* remove docs
* Simplify by committing to clean code without fallback
* Add feature flag as requested
Assisted-by: Claude Opus 5
---------
Co-authored-by: cwriter <cwriter@localhost>
* ggml : add `alloc_buffer_n` to buffer type interface
Add alloc_buffer_n method to ggml_backend_buffer_type_i
interface, with a public API ggml_backend_buft_alloc_buffer_n.
- Default implementation in ggml-backend.cpp handles multi-buffer
splitting and tensor allocation via ggml_tallocr
- Meta buffer type provides custom implementation that creates
per-device sub-contexts and delegates to simple buffer types
- ggml_backend_alloc_ctx_tensors_from_buft now collects tensors
into a list and delegates to the new API
- Remove temporary ggml_backend_meta_alloc_ctx_tensors_from_buft
- Add NULL alloc_buffer_n to all existing buffer type
interfaces (cpu, metal, openvino, hexagon, webgpu, zdnn, virtgpu, repack)
Assisted-by: llama.cpp:local pi
* cont : fix `cur_buf_size` init after flushing a buffer
* ggml : add TODO tag for shared buffer split logic
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* tests : add alloc_buffer_n coverage
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* cont : fix compile warnings
* tests : add descriptions for alloc_buffer_n tests
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ggml : address review comments on alloc_buffer_n
- restore GGML_LOG_ERROR on buffer alloc / tensor init failure in the
default impl (name the failing tensor)
- check the malloc result and drop the _impl indirection in
ggml_backend_alloc_ctx_tensors_from_buft
- remove comments that restate the code
- fix the TAG_ALLOC_SHARED_BUFFER_SPLIT typo
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ggml : add get_alloc_size_n to buffer type interface
- Add ggml_backend_buft_get_alloc_size_n public API
- Add optional get_alloc_size_n callback to ggml_backend_buffer_type_i
- Share tensor->buffer planning between alloc_buffer_n default and get_alloc_size_n default
- Replace unchecked realloc with std::vector in alloc_buffer_n default
- Make ggml_backend_alloc_ctx_tensors_from_buft_size use the new API
- Add test-alloc coverage for get_alloc_size_n
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* cont : report malloc failure
In tool.uv.sources, torch was unconditionally pinned to the custom
pytorch CPU index, which lacks macOS Darwin wheels and causes uv sync
to fail on macOS. Add the sys_platform == 'linux' marker to match the
existing Poetry dependencies configuration.
Assisted-by: Antigravity
Resolves: https://github.com/ggml-org/llama.cpp/issues/29176
* hexagon: add q2_k and q3_k quant type support
* hex-qk: consistent allocation of src1_row_size
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* tests : simplify function signature
* llama : clamp kpool re-pool bound to existing pools
The n_tokens/kpool + n_seqs_unq bound on n_new_g overshoots when a batch
fills the whole cache: n_ctx tokens complete exactly n_ctx/kpool pools, so
the +1 pads new_pool_idxs/new_pool_rep one entry past n_pool_real. Graph
reserve only covers n_pool_real entries, so the first full-context decode
builds bigger tensors than reserved and ggml-alloc demands a graph
reallocation (abort under GGML_SCHED_DEBUG_REALLOC=1).
Clamp the bound to n_pool_real: a ubatch can never mark more pools than
the cache holds, and reserve's n_pool_max already covers that.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : cap to n_pool_max
* hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA
* hex-cpy: various fixes on top of the concat optimizations
Removed CONCAT_DMA_MIN_ROW logic, it was broken with 64-bit DMA.
While it's kinda silly to use DMA for tiny stuff if that tensor gets mapped to an extended buffer the only way to read it is DMA.
Added missing dma_queue_flush() calls.
Added additional guards for conditions we don't support.
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* convert : write Gemma embedding scale for DFlash drafts
A DFlash draft shares the target's token embeddings. Gemma scales them by sqrt(hidden_size) in the forward pass, and the draft config does not state that scale, so the converted draft read unscaled embeddings.
Take the scale from the target config when the draft config has none.
Assisted-by: Claude
* convert : check with get_model_architecture for gemma models
* cuda : route sm70 to the Turing MMVQ nwarps table
Volta (sm_70) has no MMVQ parameter table of its own and falls through
to GENERIC, which launches K-quant batch-1 decode (ncols_dst == 1) at
nwarps=4. sm_70 shares TURING's tuning: the K-quant vec_dot prefers
nwarps=2 there. Route sm_70 to the existing MMVQ_PARAMETERS_TURING
table in both the device and the host table selector.
Measured on one Tesla V100 32GB PCIe (PG500-216, driver 580.178.04,
CUDA 12.0.140) with Qwen3.8-27B Q4_K_M, tg128, interleaved A/B in 6
ABBA blocks with paired per-block deltas: +1.091 t/s = +3.17 %
(t = +49.0, all six per-block deltas positive); perplexity
bit-identical (6.3697 +/- 0.04066 both builds, wiki.test.raw). The
patched build's K-quant mul_mat_vec_q kernels launch at nwarps=2
(cubin EIATTR_MAX_THREADS) while Q4_0/Q8_0 stay at nwarps=4, and the
same measurement on the September master base gave +3.84 % (t = 85).
The tuning originates from the V100-focused fork anyei/llamacpp-v100
(MIT), commit b912d1b1e, which carries a dedicated
MMVQ_PARAMETERS_VOLTA table; a cubin-level comparison confirmed that
routing sm_70 to the existing TURING table is equivalent for the
K-quant batch-1 path this change affects, so this is the minimal
2-line form. https://github.com/anyei/llamacpp-v100/commit/b912d1b1e
Original-patch-by: anyei <angelyoelroblesmercedes@gmail.com>
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: tkittich <tkittich@gmail.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* metal : release temporary private transfer buffers
Assisted-by: OpenAI Codex
* metal : fix order and formatting
---------
Co-authored-by: Niklas Wenzel <dev@nikwen.de>
* CUDA: Handle compute type for NVFP4 on cublass path
Signed-off-by: ynankani <ynankani@nvidia.com>
* Use BF16 compute type for quantized models if HW allows
Signed-off-by: ynankani <ynankani@nvidia.com>
* Set acc prec to bf16 for nvfp4 as it needs atleast bf16 range
Signed-off-by: ynankani <ynankani@nvidia.com>
* Update ggml/src/ggml-cuda/ggml-cuda.cu
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* preserve op_params for per-expert matmul
Signed-off-by: ynankani <ynankani@nvidia.com>
---------
Signed-off-by: ynankani <ynankani@nvidia.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Pinning to >= 3.4.3 is required to enable DeviceTopK, which was affected by
a race condition https://github.com/NVIDIA/cccl/pull/10627.
We will relax this for future CTK versions which will bundle CCCL >
3.4.X (CTK 13.5 will bundle CCCL 3.5.0 for example)
* BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS
* AOCL-Blas : Add an AOCL-BLAS Quick Start and drop the fixed version path
* AOCL-BLAS doc : Note on ZenDNN
LLM-jp-4.1 uses the GPT-OSS format, but its tokenizer decodes a space
after every special token and parallel tool calls are separated by
<|end|>. The GPT-OSS handler rejects this output, so add a dedicated
handler, selected by the chat_format=llm-jp-harmony-v1 declaration in
the chat template.
Assisted-by: Claude Fable 5.1
* vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3
The original tokenizer configs for PLaMo-2 and PLaMo-3 have
`add_bos_token: true` and `add_eos_token: false` , but
_set_vocab_plamo() did not write the BOS/EOS metadata. The
PLAMO2 tokenizer path also ignored add_bos/add_eos during
tokenization.
Write the settings from tokenizer_config.json and honor them in
the PLAMO2 tokenization path. GGUFs without these keys keep the
previous behavior.
* Update conversion/base.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
The committed docs/ops/CPU.csv is out of sync with the current
test-backend-ops suite: 11 ops with CPU support (COL2IM_1D,
MUL_MAT_HADAMARD, SWIGLU_CLAMP, MUL_MAT_W4A4/W4A8, MUL_MAT_ID_W4A4/W4A8,
DSV4_HC_COMB/PRE/POST, LIGHTNING_INDEXER) are missing entirely, and
many other ops have fewer test cases than the suite generates now.
docs/ops.md (which CI requires to match the CSVs) therefore
understates CPU support.
Regenerated with:
test-backend-ops support -b CPU --output csv > docs/ops/CPU.csv
scripts/create_ops_docs.py
Note: ADD1 now reads unsupported on CPU because ggml_add1 is
GGML_DEPRECATED and the suite no longer generates test cases for it;
the CPU implementation itself is still present.
Assisted-by: Xing
#27941 disabled -sm tensor for qwen4exp because test-llama-archs asserted on the
Meta device once the fixture carried a PLE layer:
GGML_ASSERT(ggml_backend_buffer_is_meta(tensor->buffer)) at ggml-backend-meta.cpp:476.
With host-resident embeddings the PLE gather is a CPU node and hc_init (the REPEAT
that fans the embedding out to the hc streams) was first reached through layer 0's
PLE path, after that gather. ggml_backend_sched_split_graph pass 2 expands a device
assignment upwards only until it meets a CPU node, so the REPEAT stayed on the CPU
and the later reshape of hc_init inside the meta split viewed a host-resident node.
Expanding hc_init right after it is built puts the REPEAT directly before the first
device node, where pass 2 assigns it; the embedding reshape stays in the CPU split
and is copied in as a split input, as in deepseek4.
dequantize_block_iq4_nl writes QK_K values per block, but a row can be shorter than that (an IQ4_NL row is only guaranteed to be a multiple of QK4_NL). Threads whose 32-value sub-block starts at or past k currently read and write past the end of the row. Skip those sub-blocks; for rows that are a multiple of QK_K the check never fires.
* hex-allreduce: add support for safe scatter mode
* hex-allreduce: pare down excessive comments
* hex-allreduce: re-write to remove register spills
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* llama: preserve original batch order for layer inputs
Assisted-by: Codex
* tests: cover layer-input order across KV layouts
Assisted-by: Codex
* tests: exercise layer-input ordering on CUDA devices
Assisted-by: Codex
* llama: make layer input reordering compatible with tensor split
Copy each microbatch tensor from offset zero and restore original row order after synchronization. Extend the layer-input regression to cover tensor split and repeated reads and decodes.
Assisted-by: Codex
* llama: restore token order for unmasked NextN embeddings
Use the original-token mapping for unmasked NextN rows, including when
layer-input capture is disabled. Keep masked NextN rows on the logits
output mapping and preserve offset-zero tensor copies.
Extend the existing regression to cover NextN alone, combined layer
capture, and masked outputs with repeated decodes and getters.
Validation: all 256 CPU/CUDA/tensor configurations pass. Qwen3.8-27B
Q4_K_M MTP completes MT-Bench at concurrency 16 before and after.
Assisted-by: Codex
* ggml: fix WebGPU reservation and OpenVINO hidden-state capture
Reserve WebGPU vector attention scratch across batch sizes and refresh reservations when NextN capture settings change. Preserve requested OpenVINO outputs, dynamic shapes, sequence counts, and current graph bindings.
Extend existing WebGPU regression coverage and enable strict allocation checks.
Assisted-by: Codex
* llama: defer regression test and backend fixes to follow-ups
Keep this PR focused on restoring token order for layer inputs and unmasked NextN embeddings. Remove the added regression test, OpenVINO and WebGPU changes, and the separate NextN reservation change.
Assisted-by: Codex
* llama: keep n_embd declaration in its original position
Assisted-by: Codex
* llama : pass token count to layer input extraction
Assisted-by: Codex
* llama : name original batch indices batch_idxs
Assisted-by: Codex
* llama : name extracted embedding indices embd_batch_idxs
Assisted-by: Codex
* llama : tag target embedding reordering
Assisted-by: Codex
* llama : tag extraction and name the index capture flag
Assisted-by: Codex
After the device decode, flip causal_attn off, decode n_ubatch/2 then
n_ubatch tokens. Both have the same node count, so a shape that depends
on the flag makes the second reallocate at an unchanged graph size,
which aborts under GGML_SCHED_NO_REALLOC. Skipped for the encode archs.
* convert: fix LoRA conversion crash for Qwen3.5 V-head reorder
_reorder_v_heads does reshape+permute+reshape to reorder V heads from
grouped to tiled order. LoraTorchTensor.reshape() cannot split its
row dimension (A matrix), so converting Qwen3.5 LoRA adapters that
target out_proj crashes with NotImplementedError.
Fix: detect LoRA tensors and apply the equivalent index permutation
directly — column reorder (dim=last) permutes A's columns, row
reorder (dim=0) permutes B's rows. This is mathematically identical:
(B @ A)[:, perm] == B @ A[:, perm]
(B @ A)[perm, :] == B[perm, :] @ A
Verified: both paths produce exactly zero diff against the full-tensor
reorder on random (rank=32, 4096×4096) matrices.
Fixes#21125
Signed-off-by: Radu Swigler <radu@swigler.com>
* convert: add ty: ignore for hasattr-guarded LoRA call
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix comment
* nowrap
---------
Signed-off-by: Radu Swigler <radu@swigler.com>
Co-authored-by: Radu Swigler <radu@swigler.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
The sparse indexer mask is built with a set_rows scatter. Padded pools,
absent sequences and missing tail cells all pointed to the same n_kv
sentinel row, and invisible pools picked by top_k to fill the selection
overlap the tail cells of the token, so several CPU threads wrote the
same element (ThreadSanitizer data race in the sanitize CI).
Allocate the slot mask for both selection paths and route every dead
slot to its own dump row n_kv + slot. Live slots address disjoint cells,
so the scatter indices of a token are unique.
* ggml: fix integer overflow guard for zero-element tensors
* ggml: validate number of elements in tensor to prevent integer overflow
* ggml: fix error print
* cli: exit on stdin EOF and drop the console wide Ctrl+C broadcast
On Windows the simple input reader sends CTRL_C_EVENT to every process
attached to the console when stdin reaches EOF, killing unrelated
processes such as a supervising agent. The CLI only stopped on EOF
because of that self inflicted SIGINT; on POSIX, and with the advanced
reader, it spins forever printing prompts.
Drop the broadcast so both platforms just return an empty read, and
treat an empty read as EOF in the chat loop and the model selection,
since a submitted line always ends with a newline.
* cli: keep the newline of a trailing "/" and stop mtmd-cli on EOF
A lone "/" came back as an empty read and was taken for EOF, and
mtmd-cli only stopped on EOF through the removed broadcast.
The fixture recycles its two blocks over 8 cache slots, so the fp16
error builds up past the 1e-4 NMSE bound on the Vulkan T4 and WebGPU
jobs of Models Backend. Two l-cycles keep every branch of the cycle
loop and halve the error.
* llama: llama_prefetch_rows
* llama: support row prefetch on Windows
Apply the Windows port contributed by @praneshgo unchanged.
Source: https://github.com/ggml-org/llama.cpp/pull/29599#issuecomment-5887721014
* avoid exposing llama-mmap in model code, route via llama-impl
* add windows check, only prefetch in lazy mode
* cont : clean-up
* cont : fix build
* cont : clarify padding token for gemma4
---------
Co-authored-by: Pranesh Gonegandla <pranesh.iitp@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml : add BF16 unary, GLU, binary and scale ops (CPU, CUDA)
* ggml-cpu : use per-op _bf16 functions for BF16 unary and GLU ops
Assisted-by: Claude Opus 5.5
* CUDA: use ggml_cuda_cast in binbcast and unary kernels to fix the HIP bf16 build
* ggml-openvino : reject BF16 SCALE and mixed-type BF16 ADD/MUL/SUB
* cpu: accept BF16 in src1 of mul_mat
ggml_conv_1d_dw builds its im2col in F32 when the kernel is BF16, then
calls ggml_mul_mat(im2col, kernel), which puts F32 in src0 and BF16 in
src1. The CPU backend refused that combination, so it was reported as
unsupported on every backend and never compared against anything.
Widen BF16 into the F32 work buffer, next to the existing packing of F32
into vec_dot_type. This is the arithmetic the Metal mat vec kernel
already uses, both operands promoted to float and accumulated in float,
so the two agree exactly rather than approximately.
Cover it with a conv_1d_dw test over F32, F16 and BF16 kernels, plus
three mul_mat cases with BF16 in src1.
* vulkan: reject BF16 in src1 of mul_mat unless src0 is BF16
supports_op only checked the src1 type for non contiguous tensors, so
a contiguous BF16 src1 was accepted and the pipeline lookup asserted.
The only BF16 src1 path is the BF16 x BF16 multiply, every other src0
type now reports the op as unsupported and the scheduler keeps it on
the CPU.
The BF16 kernel case of the conv_1d_dw test needs the f32 x bf16
mat vec variants of the Metal backend, which land separately.
This commit adds an optional --add-bos token command line option to the
run-org-model.py script.
The motivation for this is that there are models, for example Gemma4,
that explicitely set the add_bos value to true in llama-vocab.cpp even
if the original model does not set this value to True.
It would be nice to be able to force the models to agree on the bos
token so that logit verification can proceed.
Refs: https://github.com/ggml-org/llama.cpp/pull/21500
* ui : shared model display primitives
Extract ModelCapabilityIcons (canonical Tools/Reasoning/Vision/Video/Audio
order) out of ModelId and reuse it there, add the shared DialogConfirmDownload
for destructive download actions, the discover org avatar with dark-mode
inversion and the thin download progress bar, and rework ModelId badges to take
thinking/tool-use support directly.
Assisted-by: pi:GLM-5.3-Flash
* ui : remember hub avatars that failed to load
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : render shared model row hints as native titles
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : fix badge guard for draft sidecars, keep parameter precision
hasBadges now counts draft sidecar badges, so a sidecar-only model still
renders. Billions keep one decimal for hub counts and stay bare for whole
values. Avatar failures track the org instead of the instance, and the
download progress bar no longer pulses while determinate.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : model download pipeline
Track HuggingFace downloads end to end: the server download/cancel endpoints,
a status manager fed by the /models/sse download progress events, and a
models-discover store holding the catalog and detail state for the discover
view. Downloaded and in-flight entries are excluded from the loadable model
list.
Assisted-by: pi:GLM-5.3-Flash
* ui : route sidecar tag lookup through the sidecars util, validate the paused list
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : model memory-fit estimation
Replace the raw runtime-memory estimate with the app's compatibility check:
the smallest real Mac memory tier that fits a model file, budgeted as
RAM x 0.75 minus fixed overhead with headroom on the file size. The constants
move to lib; the unused runtime-memory estimate is dropped. browser-info's
OS detection is exported for reuse.
Assisted-by: pi:GLM-5.3-Flash
* ui : cover the memory-fit and tool-use heuristics in tests
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : Hugging Face Hub data layer
Add HuggingFaceService and its constants/enums/types: GGUF repo search, file
tree and model detail fetching, quant/sidecar filename analysis, shard-set
collapsing and the llama.app catalog feed, plus an orgOf() helper on the model
name utils.
Assisted-by: pi:GLM-5.3-Flash
* ui : strip provider tilde prefix from hub avatar urls
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : trim redundant comments in the HF data layer service
Per review: drop JSDoc that restates the method name and inline comments
that restate the code; keep only comments carrying non-obvious context.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : harden the HF data layer error typing, cover the helpers in tests
Carries the HTTP status on retryable fetch errors instead of matching the
message text. Marks expand-dependent catalog fields optional and documents
the data/models index pairing. Adds table tests for the pure helpers.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : model id grammar for sidecars, quants and capability parsing
Extend the shared model id parser with sidecar tokens (draft variants and
auxiliary imatrix/mmproj files), weight-file and custom-quant regexes, and add
the tools capability to ModelCapabilities; the selector option row picks it up
from the model's declared capabilities.
Assisted-by: pi:GLM-5.3-Flash
* ui : escape sidecar tokens in the regex alternation
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : type-safe API types, fetch helpers and download-ready models store plumbing
Assisted-by: pi:GLM-5.3-Flash
* ui : document the model list index pairing, fix an em-dash
Assisted-by: pi:zai-org/GLM-5.3-Flash
* Update tools/ui/src/lib/components/app/chat/index.ts
Co-authored-by: Pascal <admin@serveurperso.com>
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* openvino: serve GET_ROWS on a weight view from the base Constant
Resolve view_src when collecting weight Constants so a view over a
quantized weight no longer becomes a dynamic typed Parameter, and fold
the row offset of the view into the gather indices instead of slicing
the dequantization subgraph.
* openvino: lift the quantized GET_ROWS view rejection
The supports_op rejection of a quantized src0 view with a nonzero
offset keeps the vs0 GET_ROWS cases of #28253 away from OpenVINO.
The weight view now resolves to the base Constant with the row offset
folded into the gather indices, so the rejection goes away.
#27773 adds the glm5-next arch without its rows in the Metal fusion
baseline, so test-fusion --check fails on it. The rows come from
test-fusion --record on an M5 Max, and --check passes 270/270.
The MUSA vendor header never defined __CUDA_ARCH__, so every architecture
test in the shared ggml-cuda sources evaluated to 0. Kernel bodies gated on
the architecture therefore compiled to nothing, for example the q8_0 -> f16
dequantization kernel in convert.cu, whose NO_DEVICE_CODE fallback expands to
an empty body in host code.
Report the newest architecture like the HIP backend does and exclude the
NVIDIA-only features explicitly, as they are not usable on MUSA. Define it
for device passes only: CUB uses defined(__CUDA_ARCH__) to detect device
compilation, which is also how nvcc behaves.
Drop the now-redundant defined(__CUDA_ARCH__) checks in the architecture
comparisons: __CUDA_ARCH__ is undefined in host passes for CUDA and MUSA, and
HIP defines it for every pass, so both forms select the same branch.
* Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx
* Add initial MTP support
* Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements.
* Review driven changes, remove env vars, protect tensors
* Strip MTP for initial PR
* Clean up after mtp strip
* Clean up after mtp strip
* Update speculative.cpp
* Update llama-context.h
* Clean up after mtp strip
* Fix tokenizer ignore merges
* Improve quantization protection selection
* Refactor mhc helpers, graph base
* Lint Fixes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Skip glm5-next in model saver, fix CRLF
* Skip glm5-next in sweep
* Remove T4 fallback
* Review cleanup
* Review suggestions
* Defer separate MTP gguf handling to MTP PR, drop filter
* Repad n_head_kv
* kpool init apply
* Order by descending score
* Drop guard
* read kpool from hparams, clarify kpool cache flags, remove kpool_build_state(nullptr)
* Add glm5-next support to model saver and add arch test fixture
* Review cleanup
* Kpool pooled caching clarify
* Add multi stream support
* Finish Rebase
* Sparse FA fir DSA prefill
* Const
* Update llama-model.cpp to fix rebase error
* gguf-py : merge tensor map entries for HC tensors
* model : use build_gdn_l2_norm in GLM5_NEXT implementation
* chore : remove trailing whitespace
* model : use new OP precision setting API in GLM5_NEXT implementation
* mtmd : use ggml_swiglu_clamp in GLM5V and apply the image token limit
The two clamps around swiglu_split are what ggml_swiglu_clamp already does,
so the clamp bounds collapse back to one value. GLM5V also never called
set_limit_image_tokens(), so --image-max-tokens had no effect.
Assisted-by: Claude Opus 5
(cherry picked from commit 46d18e12d422be4cc04a70e4a9a9e0168bb3d5b7)
* llama : keep the GLM5-Next k-pool layout across ubatches
The layout was rebuilt from a full cell scan on every ubatch. Pools are fixed
by the positions relative to the sequence's first one, so the layout now lives
on the memory and a ubatch only appends to it.
A sequence edit no longer stales every pooled key either, only the ones at or
after the edited position, which makes a tail seq_rm free. The pooling subgraph
is built unconditionally so the graph shape no longer changes every kpool
tokens, and the pool axis is folded into rows before soft_max, which otherwise
exceeds the CUDA gridDim.y limit past n_kv 262144.
Assisted-by: Claude Opus 5
(cherry picked from commit 5d1c40b93e17fddbf73b785efe43e0d02ccb3977)
* model : write the GLM5-Next recurrent rollback checkpoints
The conv state and the delta net state were only written to the live row, so a
rollback restored whatever the checkpoint rows happened to hold. Take the same
route as kimi-k3: build_recurrent_attn for the state, and write all K_rs conv
groups. That also drops a state view that assumed contiguous rows.
Enroll the arch in test-recurrent-state-rollback, which catches this under its
garbage-filled cache pass.
Assisted-by: Claude Opus 5
(cherry picked from commit 5ace37e86d5d448e83ef5dde5632c748185b18cd)
* llama: fix PR #27773 test-save-load-state restore failure
Clear the attention and indexer cache data after a failed hybrid state restore so restored NaNs cannot affect a later sequence.
Assisted-by: Codex
* llama: fix PR #27773 gpu-rocm graph reallocation
Reserve the full GLM5-Next pool capacity and dirty pool count. The gpu-rocm Test step aborts when n_new grows while the graph node count stays fixed; CUDA, Vulkan, Metal, and WebGPU checks report the same error.
Assisted-by: Codex
* llama : fix GLM5-Next k-pool layout staleness after edits and shared teardown
Two defects in the cross-ubatch k-pool layout added by the k-pool commit:
1. Wrong results. An edited sequence only rebuilt its pool layout when its cell
count changed, so if the first ubatch after an edit added back exactly as many
cells as were removed, the stale position-to-cell list survived. With a unified
cache and more than one sequence, where another sequence takes the freed cells,
the reused layout points at the wrong cells (CPU: large logit drift, CUDA: NaN).
Rebuild whenever the sequence is stale, not only on a size mismatch.
2. Slowdown. "shared" mode was assumed to end only with an edit that forces a
rebuild, but sharing also ends when the other sequence is removed. The survivor
kept shared = true, pinning cache_safe off and re-pooling every pool on every
ubatch (server trigger: n>1 completions with -kvu, via the seq_cp in
copy_state_to). In seq_rm, if the layout has shared cells, stale every sequence
so one rebuild re-derives sharing and cache_safe returns to 1.
Assisted-by: Claude Opus 5
* llama : fix build_attn_mha stream stride for non-contiguous q
build_attn_mha split the batch into streams with a stream stride of
q->nb[3]/n_stream. That only equals one stream's span, (ne[2]/n_stream)*nb[2],
when q is contiguous. GLM5-Next is nope-only, so it does not concat a rope part
and passes the permuted q_absorbed straight in, where nb[3] != ne[2]*nb[2]; the
stride was then n_head times too large and every stream s >= 1 read another
head's queries. Split-KV (-np N without --kv-unified) multi-stream prefill was
wrong for every stream past the first. Unified KV and decode were unaffected
(n_stream == 1, and decode takes the gather path). Other MLA models concat rope
so q is contiguous and the computed value is unchanged for them.
Compute the stride from the token dimension, which is identical for a
contiguous q.
Assisted-by: Claude Opus 5
* llama : re-derive GLM5-Next k-pool sharing on state_read/state_drop
The shared-cell teardown added to seq_rm (stale every sequence when the layout
has shared cells, so a survivor does not keep shared = true and pin cache_safe
off) was missing from the other paths that can free shared cells: state_read
and state_drop staled only the one sequence. Apply the same re-derivation there
and correct the comment that claimed sharing ends only via an edit or seq_rm.
Assisted-by: Claude Opus 5
* quant : drop duplicate GLM5-Next hc_ filter
The hc_ name filter was listed twice in the GLM5_NEXT protection block.
Assisted-by: Claude Opus 5
* glm5-next: scope K-pool cache access to indexed operations
* glm5-next: keep K-pool access in hybrid index memory
* glm5-next: keep mHC graph builders model-local
* glm5-next: mark only touched pools per ubatch
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
* hexagon: add F16 support for activation ops (SILU/GELU/GELU_QUICK/GEGLU/SWIGLU)
Widens ggml_hexagon_supported_activations() to accept F16 (src0/dst/src1
must agree on type), and adds F16 per-thread worker functions in
act-ops.c mirroring the existing F32 workers, backed by new HVX f16
kernels (hvx_sigmoid_f16_aa, hvx_tanh_f16_aa, hvx_mul_mul_f16_aa,
hvx_min_scalar_f16 family).
SILU, GELU, GELU_QUICK, GEGLU, and SWIGLU are verified correct on-device
(QRD8850) via test-backend-ops CPU-diffed correctness tests. SWIGLU_OAI's
F16 path is code-complete and builds clean on host + all 4 DSP arch
variants (v73/v75/v79/v81), but has no F16 test-case coverage in
test-backend-ops and is therefore unverified on-device in this change.
* hex-ops: align macros
* hex-ops: minor formatting
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
GGML_PAD(nbytes, alignment) wraps to 0 when nbytes is within
(alignment - 1) of SIZE_MAX, which silently bypassed the size
overflow guard in gguf_init_from_reader. Reject the tensor before
padding when nbytes + (alignment - 1) would overflow.
Adds a test-gguf handcrafted case (F32, ne = [4, 2^30-1, 2^30+1, 1])
whose ggml_nbytes = 2^64 - 16 lands in the wrap window. Fails on
master, passes with the guard.
* model : support classifier_pooling for ModernBERT rerankers
Assisted-by: Claude Opus 5.5
* model : read classifier pooling type in load_hparams
Write classifier.pooling_type from _try_set_pooling_type whenever the
config has classifier_pooling, and read it in
llama_model_base::load_hparams. ModernBERT falls back to mean when it
is unspecified.
Assisted-by: Claude Opus 5.5
* conversion : only accept cls and mean for classifier_pooling
Assisted-by: Claude Opus 5.5
* model : rename classifier_pooling_type to pooling_type_cls
Assisted-by: Claude Opus 5.5
* hex-concat: reduce pkts in gather/transpose hot loop
gather directly into dst buffer, use special instruction for gather sync
* hex-concat: use fastdiv
replace calls to sw divide with fastpath
* hex-concat: optimize DMA-HVX pipeline and add transpose helpers
Without CUB (HIP, MUSA) argsort ran the bitonic kernel with one thread
per padded column, so any row above 1024 entries launched an invalid
block configuration. Each thread now owns several columns, every stage
of the network runs all owned columns before the barrier, and the block
is capped at 1024 threads. Shared memory becomes the only bound, which
supports_op checks against the device instead of a fixed 1024.
Rows up to 1024 run the same work as before. Bit-exact with the CUB
path on rows of 2048.
It turns out Intel doesn't particularly like loading F32s one at a
time and we already have the _2aliagned load logic in mul_mat_vec,
so here we use it.
While we do already check all the requirements to load elements 4
at a time across [B]F16 and F32, it turns out [B]F16 loading 4 at a
time is sometimes slower on very specific shapes on Intel BMG.
Loading 4 at a time is a bit faster on F32, but its not material
and I assume might be slower on other platforms.
Note that we also need to validate `a_offset` is 2-aligned in
`mul_mat_vec.comp`, which was missing in the original 2-way-load
patch.
Some selected speedups from `test-backend-ops perf` on a B60.
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1704 runs - 767.17 us/run - 117.44 MFLOP/run - 153.08 GFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 2556 runs - 529.81 us/run - 117.44 MFLOP/run - 221.66 GFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1704 runs - 727.13 us/run - 234.88 MFLOP/run - 323.03 GFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 2130 runs - 528.84 us/run - 234.88 MFLOP/run - 444.15 GFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1704 runs - 702.19 us/run - 352.32 MFLOP/run - 501.74 GFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1988 runs - 532.14 us/run - 352.32 MFLOP/run - 662.08 GFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1278 runs - 919.50 us/run - 469.76 MFLOP/run - 510.89 GFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1917 runs - 543.69 us/run - 469.76 MFLOP/run - 864.03 GFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1197 runs - 892.12 us/run - 587.20 MFLOP/run - 658.21 GFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1881 runs - 575.17 us/run - 587.20 MFLOP/run - 1.02 TFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1498 runs - 716.40 us/run - 939.52 MFLOP/run - 1.31 TFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 1819 runs - 576.36 us/run - 939.52 MFLOP/run - 1.63 TFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 134 runs - 7467.09 us/run - 60.13 GFLOP/run - 8.05 TFLOPS
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0): 134 runs - 7478.12 us/run - 60.13 GFLOP/run - 8.04 TFLOPS
mut_mul_id selected its matmul tile with total token count.
For MoE dispatch grid the true N per workgroup is per-expert rows.
At pp128 on Sarvam 30B that is 6, not 128, so the picker took the l-tile for ~6 live rows.
Most workers in each group had nothing to do.
This wasted time. The slow part was 55% of the whole job.
* fix c++ odr by properly using GGML_COMMON_DECL_CPP
* using actual field rather than macro
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: XZiar <xziar@xziar.xziar>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vocab : keep </s> NORMAL in PLaMo-2 and PLaMo-3
The PLaMo-2 and PLaMo-3 vocabularies mark </s> as NORMAL. Current
EOG token heuristic matched it by text and added its attribute
to CONTROL.
Skip this heuristic for the PLAMO2 vocab type so </s> stays NORMAL
and is not treated as EOG.
* use <|plamo:eos|> for detection
With --path or --no-ui, /sw.js returned 404, and a 404 does not remove a service worker, so browsers kept showing the cached built-in UI. Serve a worker that unregisters itself, clears its caches and reloads open tabs. A sw.js in the --path folder is still served first.
Assisted-by: Claude Opus 5.5
- Check for buffered write errors when closing downloaded files.
- Use UTF-8 paths when writing ETag files on Windows.
- Write in binary mode on Windows.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
graph_inputs was populated while splitting the graph, so it only
contained the inputs that are used as srcs of some node. With pipeline
parallelism (n_copies > 1), each graph input contributes n_copies leafs
to graph_copy, so switching between batches that consume different
inputs (e.g. token batches that do not use the embeddings input vs
image batches that do) changed the graph composition. This shifted the
input copies in graph_copy, making the backend ids comparison report
spurious changes and forcing the scheduler to re-reserve. The
re-reserve could then record smaller input sizes (e.g. out_ids with
n_outputs = 0) and abort later on a graph with an unchanged size via
GGML_SCHED_DEBUG_REALLOC.
Collect the inputs after the split instead, from all input leafs of the
graph, so that the graph composition depends only on which inputs
exist, not on which inputs are used.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* musa: build the docker image and CI container from the MUSA SDK images
Use registry.mthreads.com/mcconline/musa_sdk:5.2.0-{devel,runtime}-ubuntu22.04-s5000
instead of registry.mthreads.com/mcconline/inference/pytorch:2.9.1.post1-py3.10-musa5.2.0-mp31-devel-ubuntu22.04-amd64
for the MUSA docker image and the MUSA CI container, and let the runtime stage use the
runtime image instead of reusing the devel one, which drops the MUSA toolchain from the
published images.
* musa: install the MUSA headers and loader path the SDK images omit
musa_sdk:5.2.0-*-s5000 does not ship the cub and thrust headers that the MUSA
backend builds against, and its runtime image does not register
/usr/local/musa/lib with the dynamic loader.
Install both header packages in the build stage and in the MUSA CI container,
and write the loader path in the runtime stage.
* musa: install libmthreads-compute for the MUSA runtime library
The MUSA SDK images do not install libmthreads-compute, which provides
libmusa.so.1 in /usr/lib/x86_64-linux-gnu, so linking anything against the
MUSA backend fails.
* musa: install libmthreads-compute in the runtime stages
The MUSA runtime image does not install libmthreads-compute, so the published
images would have no libmusa.so.1 at run time.
---------
Co-authored-by: yeahdongcn <yeahdongcn@users.noreply.github.com>
* ggml : speed up model loading
A crafted model could hang the server for a very long time, try with:
llama-cli -hf angt/test-gguf-1Mkv -hff model.gguf
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Avoid empty keys
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Fix
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ci : update the oneAPI toolkit to 2026.1
oneDNN is removed from Intel Deep Learning Essentials in 2026.0, so
staying on the deep-learning-essentials path would silently lose oneDNN
support when the toolkit version is updated. Switch both the Ubuntu and
Windows CI jobs to the new unified Intel oneAPI Toolkit installer,
which still includes oneDNN (until 2027.0) and keeps the component IDs
unchanged for the Windows install script.
Measured with the same code (b10899) built with oneAPI 2026.1 vs the
2025.3-based release build on Arc B570: prompt processing 1331 vs 434
t/s (3.1x), token generation 50.1 vs 45.3-48.0 t/s.
Assisted-by: GLM (z-ai/glm-5.3-flash)
* docs : update the SYCL backend build requirements for oneAPI 2026.1
With the 2026.0 release the Base toolkit and the HPC toolkit are
combined into the oneAPI Toolkit, and oneDNN is removed from the Deep
Learning Essentials package. Update the install instructions, the
verified release table and the news section accordingly.
Assisted-by: GLM (z-ai/glm-5.3-flash)
* ci : update the release workflow for oneAPI 2026.1 and Level Zero SDK 1.33.1
Align the release package build with the CI build update:
- oneAPI toolkit 2025.3.3 -> 2026.1 (the unified oneAPI Toolkit)
- Level Zero SDK 1.28.2 -> 1.33.1, and the Debian package names
(level-zero/level-zero-devel -> libze1/libze-dev)
- The Windows DLL copy list for the 2026.1 runtime: sycl9.dll and the
.6/.3 MKL library versions
Assisted-by: GLM (z-ai/glm-5.3-flash)
* ci : remove the removed .spv fallback files from the Windows DLL copy list
oneAPI 2026.1 no longer ships libsycl-fallback-bfloat16.spv and
libsycl-native-bfloat16.spv (the OpenCL fallback mechanism changed), so
the copy step failed with exit 1.
Assisted-by: GLM (z-ai/glm-5.3-flash)
* devops : update the oneAPI toolkit image in the Intel Dockerfile
Assisted-by: GLM (z-ai/glm-5.3-flash)
---------
Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com>
* server : support multimodal input for /v1/embeddings (Qwen3-VL-Embedding)
Accept the OpenAI-style wrapped content array format for multimodal
embedding requests. Each {"content": [...]} object is one input that
produces one embedding; text parts are concatenated and image_url parts
are decoded via handle_media then spliced with process_mtmd_prompt.
The legacy formats (plain string, token arrays, mixed arrays, and the
{prompt_string, multimodal_data} object) continue to work unchanged via
tokenize_input_prompts. Bare content arrays (the unwrapped shape) are
rejected with a migration message.
Also disables KV prefix reuse for stateless embedding/rerank tasks so
that repeated inputs do not incorrectly share cached KV across requests.
Assisted-by: Opencode Qwen3.8 27B
* clean up comments and docs
* refactor
* add tests
* support video and audio inp
---------
Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* models: pad on the left with ggml_pad_ext
The Parakeet, LFM2-Audio, Granite Speech and Gemma 4 audio encoders
build a left padding as a right pad followed by a roll, and DFlash2
concatenates a zero filled block in front of the previous tokens.
ggml_pad_ext does both in one node now that every backend supports a
left padding. The Gemma 4 audio embeddings are bit identical.
* models: skip the DFlash2 taps that only read padding
A tap at or past block_size shifts every row out of the block, so its
term is zero. The loop runs min(kernel_size, block_size) taps.
* adapt common
* add common_batch
* wip
* wip: spec
* cont
* common_speculative_process
* server_batch to use common_batch
* rm some stale calls
Assisted-by: Claude Fable 5.1
* migrate mtmd
* handle imrope, handle return val of add()/add_embd()
* add spec zeros vector
* add warning on zero fill path
* tests : use llama_context_ptr in test-recurrent-state-rollback
Replace raw llama_context pointers with llama_context_ptr and drop the
manual llama_free calls and cleanup lambda.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* tests : run test-recurrent-state-rollback over all dummy models
Add a --models DIR mode that mirrors test-save-load-state: iterate every
dummy model, report PASS/FAIL/SKIP in a table and fail only when a model
fails. Register a single ctest entry with ARGS --models instead of the four
per-model registrations.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* cont : fix typo
* metal : allow fusing 0-element nodes to keep graph packing shape-independent
The fusion packing in ggml_metal_fusion_max excluded 0-element tensors and
the topk_moe/moe_reduce checks rejected n_tokens == 0, so graphs decoding
batches with no outputs packed differently from the worst-case reserved
graph. The Metal optimizer then reordered the nodes differently and
ggml_gallocr_needs_realloc failed on the layout mismatch, forcing an
unexpected graph re-reserve (caught by GGML_SCHED_DEBUG_REALLOC).
Treat empty tensors like their non-empty counterparts: match them in the
pattern sequence and only reject genuinely malformed shapes. Fused kernels
dispatch zero threadgroups for empty graphs, which is a legal no-op.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* tests : run test_multi_seq_split_replay as a separate test
test_multi_seq_split_replay was invoked at the end of test_rollback,
so its result was folded into the rollback status and it only ran when
the rollback part passed.
Give it its own test_status return, run both tests independently over
both cache fills via a shared run_tests helper, and report them as
separate rollback / split replay columns in the --models table with
per-test summaries. The exit code fails when either test fails.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* tests : loosen the split replay nmse bound to 1e-4
test-generate-models seeds its weights from std::random_device, and some
generated lfm2 models drift up to ~1.7e-5 nmse on the split replay due to
rounding noise, tripping the previous 1e-5 bound. Raise the bound to 1e-4
so the random generations stop flaking.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* tests : reuse run_tests_for_model in single-model mode
The single-model path duplicated the model init and the non-recurrent
check from run_tests_for_model; route it through the shared helper
instead. Model load failures now return FAIL rather than SKIP so that
--model with a broken file still exits non-zero, and the helper loads
with model_only like the --models loop does since the tests create
their own contexts.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86
* add AVX2 support for masked loading and storing in simd_gemm_ukernel_tail
* ggml-cpu: fix FA softcap handling for padded KV tiles
* metal: support left and circular padding in GGML_OP_PAD
Align Metal with CPU, CUDA and Vulkan: shift the source coordinates by
the left paddings, wrap them around with the same wrap_around when
circular, and read the source through nb00, which also fixes a right
padding of a permuted source. A test case covers it.
Drop the f32_4 kernel: its selection is disabled as slower, and it
fails two pad cases once enabled.
* metal: use a function constant for the circular pad variant
Address review from ggerganov: replace the bool template with FC_PAD,
as FC_upscale_aa does, so the pad kernel is compiled once and
specialized per pipeline.
* context : do not re-reserve the scheduler when toggling causal_attn
`llama_context::set_causal_attn()` marks the scheduler to do a full re-reserve on every change of the flag. For vision inputs, this flag is flipped twice around each non-causal image chunk for Gemma models, resulting in two expensive `sched_reserve()` passes per image. This is especially slow for multi-image or video inputs.
The cost of a re-reserve scales with context and ubatch configurations, so larger settings pay more per image (see table below).
The re-reserve is unnecessary in this case because `causal_attn` only changes the values written to KQ mask, not tensor shapes or any other buffer sizes.
Note: `causal_attn` is a graph reuse key (`llm_graph_params` via `cparams`), so a new graph is built regardless of `sched_need_reserve`, so this doesn't change the graph rebuilding behaviour.
llama-server with gemma-4-26B-A4B Q4_0 + BF16 mmproj, 130-token images,
cache_prompt=false, prompt_ms median of 3 (before -> after):
| images | config | H200 before -> after | RTX 4090 before -> after |
|-|-|-|-|
| 1 | `-c 8192 -ub 512` | 134 -> 105 ms (1.27×) | 201 -> 119 ms (1.69×) |
| 24 | `-c 8192 -ub 512` | 2278 -> 1562 ms (1.46×) | 3559 -> 1748 ms (2.04×) |
| 24 | `-c 32768 -ub 2048` | 5379 -> 1584 ms (3.40×) | 13377 -> 1759 ms (7.61×) |
Generated output remains identical before and after.
* qwen4exp : make the indexer bias shape independent of causal_attn
The block/cell bias path was selected on cparams.causal_attn, so the
causal and non-causal graphs differed in tensor shapes and ops. With the
re-reserve removed (previous commit), a runtime flip resulted in
reallocating the compute buffers, which would fail under
GGML_SCHED_NO_REALLOC.
This commit selects the block path from the mask shape only, independent
of causal_attn. causal_attn is instead passed to set_input_qsa.
causal_attn is fixed per graph as it's part of the reuse key. Causal
values are unchanged. Non-causal values now follow the reference rule,
where every visible block competes on score and only unpooled cells are
always selected.
* context : state the causal_attn shape rule in the comment
* cont : add TODOs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vulkan: read the batch stride of an in place src0 from nb[2]
A dim01 contiguous tensor can still be a view whose batches are
strided by more than ne[1] rows, the first rows of a KV cache for
example. Both the mat-vec and the matrix paths read such a tensor in
place but passed ne00*ne01 as the batch stride, so every head past
the first read the wrong rows. The same applies to src1. The stride
now comes from nb[2] whenever the tensor is used in place; the value
is unchanged for a contiguous tensor.
test-backend-ops gets an m_v parameter on test_mul_mat, the number of
rows of a in memory, and two cases at the shapes of a decoder self
attention over a cache.
* vulkan: size the in place A and B ranges by their strided extent
The matrix path bound src0 and src1 to the shader with a range of
elements times type size, which ends before the batches of a strided
view. Pipelines with bounded access read zero past that range, so the
same view that the mat-vec path already handles gave wrong results
on Intel and on NVIDIA without coopmat2. The range now comes from
ggml_nbytes when the tensor is read in place.
* vulkan: address review from jeffbolznv
Bind the in place A and B of the matrix path with ggml_vk_subbuffer,
which spans to the end of the buffer, so a strided view is in range
without computing its extent.
mul_mat_id reads the batch stride of an in place src0 and src1 with
the same helper as mul_mat. test_mul_mat_id gets an m_v parameter,
the number of rows of as in memory, and a case whose experts are
strided by more rows than it uses.
* vulkan: read the batch stride of an in place src0 in mul_mat_vec_id
The single token path of mul_mat_id passed ne00*ne01 as the batch
stride of A, so a strided expert view read the wrong rows. The stride
now comes from ggml_vk_batch_stride like the other three paths, and
src1 follows the same rule.
test_mul_mat_id gets a single token case over the strided view.
* vulkan: address review from jeffbolznv
The batch stride of an in place tensor is taken from nb[2] as
nb[2] / type_size * block_size, which holds when nb[2] is padded and
not a multiple of nb[1]. A test_mul_mat case with a padded batch stride
covers it.
* vulkan: keep the A and B ranges exact in mul_mm
The quantized A loads of mul_mm carry no row bound and rely on the
descriptor range to read zeros past the last row of a partial tile.
Binding A and B up to the end of the buffer let those tiles read the
leftovers of a previous node and hung the NVFP4 mul_mm on NVIDIA
without coopmat2. The range is the strided extent of a tensor read in
place and the staged size otherwise.
* server : allow splitting RANK pooling for causal LLM rerankers
Rerank models fall into two categories: bidirectional cross-encoders
(BERT, etc.) that require all tokens in a single physical batch, and
causal LLMs repurposed as rerankers (Qwen3, Qwen3-VL) that can use
chunked prefill like any other decoder.
Previously the server rejected all RANK-pooling inputs larger than
n_ubatch, and the graph builder hardcoded QWEN3/QWEN3VL arch checks to
determine last-token pooling. This broke long-document and multimodal
reranking for causal models.
Fix: expose llama_get_causal_attn(ctx) so the server can check the
effective runtime attention type (reflecting any --attention override
or set_causal_attn call). Also expose llama_model_is_causal(model)
for querying the static architectural property from GGUF metadata.
can_split() now permits chunked prefill for RANK pooling when the
context is causal. The graph builder's inline arch check is replaced
with the same cparams.causal_attn predicate, removing the duplication.
Assisted-by: Opencode/Qwen3.8-27B
* remove unused llama_model_is_causal, fix whitespace
Assisted-by: opencode
---------
Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
- register --rpc unconditionally and call llama_supports_rpc() only from its handler
- print server "initialization ..." log after args are parsed
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
Recent PLaMo-3 models use YaRN, while some earlier PLaMo-3 models do not.
The recent PLaMo-3 store their YaRN settings as flat config keys
(rope_scaling_factor, initial_context_length) and build the dict at runtime
in Plamo3Config.rope_parameters. The current converter misses these settings
and writes plain RoPE metadata to GGUF. Mirror the runtime settings into
rope_parameters so the corresponding rope.scaling.* is written to GGUF.
The SYCL FWHT covers 64 to 512 via the standard butterfly network, plus
384/640/768/1280 via the Kronecker/Paley construction added separately in
Hadamard hint can produce (1024, 2048, 4096, 8192); those still fall through
to the default case and run as a dense GEMM against the materialized
rotation tensor, correct but O(n^2) instead of O(n log n).
fwht_kernel_wide runs one row per work-group instead of per sub-group, so
each work-item keeps N/NT values rather than N/WARP_SIZE. Butterflies below
the sub-group width still shuffle; those up to the work-group width go
through work-group local memory; the rest stay in registers. Same butterfly
and sign convention as the existing narrow kernel.
ggml's SYCL backend registration (dpct::dev_mgr) unconditionally requires a
GPU-labeled platform to exist and throws before any op-level test can run,
so test-backend-ops could not be exercised on this box (a GPU-less pod) even
via the CPU device. Verified instead with a standalone harness: the same
kernel body run through a real SYCL CPU device (Intel oneAPI DPC++ 2026.1,
OpenCL CPU backend), checked against an independent recursive-doubling
Hadamard reference, cross-validated by first running the existing unmodified
narrow kernel through the identical harness and confirming it passes (rules
out a reference-convention bug before trusting a pass on the new code).
Random-input results for all four widths, single- and multi-row:
N=1024 NT=256 rows=1 max_abs_err=1.7e-07 max_rel_err=4.9e-04 PASS
N=2048 NT=256 rows=1 max_abs_err=1.9e-07 max_rel_err=2.0e-04 PASS
N=4096 NT=256 rows=1 max_abs_err=2.0e-07 max_rel_err=1.4e-04 PASS
N=8192 NT=256 rows=1 max_abs_err=2.5e-07 max_rel_err=3.8e-03 PASS
N=1024 NT=256 rows=7 max_abs_err=2.4e-07 max_rel_err=1.0e-03 PASS
N=2048 NT=256 rows=5 max_abs_err=3.0e-07 max_rel_err=9.4e-04 PASS
N=4096 NT=256 rows=3 max_abs_err=2.7e-07 max_rel_err=1.7e-03 PASS
N=8192 NT=256 rows=2 max_abs_err=2.5e-07 max_rel_err=1.9e-03 PASS
This covers the kernel algorithm itself; it does not exercise the ggml
dispatch/supports_op integration end to end, which needs a real GPU (or a
SYCL GPU plugin) to get past backend registration. test-backend-ops build
is verified: fwht.cpp recompiles with zero warnings as part of ggml-sycl.
* hex-topk: trying to improve/cleanup the pipeline
* hex-sampling: add STEP op
* hex-sampler: add SUM op
* hex-sampler: update CPY to support sampling cases
* hex-binary: add support for chunking to handle large logits
* hex-argmax: super basic version of ARGMAX
* hex-binary: support for scalars in extended buffers
* hex-binary: fix wrong indexing for dim 1 broadcasts across dim 2 slices
* hex-argsort: fix missing header
* hex-sampler: cleanup dma usage in the sampler related ops, and binary
* hex-build: disable autovectorizer, it is better to use explicit hints for critical loops
* hex-binary: fix perf regression due to is_1d fallback
* hex-ops: update supported ops
* cuda: add F16 input to the FWHT
The CUDA FWHT accepts F32 input only. This makes the source type a template
parameter, so the kernel reads an F16 source directly instead of requiring a
converted copy. The F32 path is unchanged.
supports_op accepts an F16 src1 against an F32 src0 for the Hadamard hint.
Every other F16 src1 against a non-F16 src0 is still refused.
ggml_cuda_op_mul_mat_use_fwht is the single predicate both supports_op and
the dispatch call now share, checking contiguity and same-shape(src1, dst)
in addition to the type/hint conditions above. Without a shared predicate,
supports_op could admit an op that ggml_cuda_op_fwht then rejects only after
the unconditional same-shape assert has already fired; that gap predates
this change (it applies to the existing F32 path too) but this PR is what
touches supports_op, so it closes it here.
test-backend-ops on an A10 (lambdalabs): MUL_MAT 1297/1297, including all
24 Hadamard cases (18 existing F32, 6 new F16).
* cuda: use ggml_cuda_cast in the FWHT load, drop the comment
* llama : add discard for deferred state writes
* llama : add tensor zeroing helper for backends without tensor memset
* llama : clear K/V data after failed sequence restore
* llama : clear recurrent state data after failed sequence restore
* llama : simplify discard and restore cleanup
* llama : report error when abnormal cell count is found in state_read_meta
* llama : clear attention state on hybrid restore failure
* tests : cover failed state restore cleanup
* llama : clear MLA state on dsa restore failure
* tests : update test for rebased test suite
* llama : clarify comment in llama_memory_recurrent::state_read
* Added tiled mul_mat.
For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8,
one routine per quent. Then microkernel computes 16x16 tiles before writing out
256x256 float reults to main memory.
Tests/benches in tests/test-tiled-mulmat.cpp. 3-6x speed improvement
for large matmul, break even at 4096x64 * 64x4096, 80% performance (net
loss) for GEMV. Error rates trivial (order of 1-e04 max, 1-e05 rmse).
* Fixes for ARM/windows builds
* more windows fixes, ggml-cpu.h isn't visible in MSVC for some reason
* unified iqp + tiled on the Q5_K, IQ4_XS set for benchmarking, updated benchmark
* Fixed accidental removal of llama_build_and_test(test-backend-ops.cpp)
* First integration of iqp code
Co-authored-by Bartowski <3266127+bartowski1182@users.noreply.github.com>
* Cleaning up declaration of iq unpacking helpers to align with the bit unpackers
* Removed iqp path
* Fix cross-platform warnings
* Disabling benchmarks unless explicitly enabled
* Fix backend_init for DLL-based builds, add self and bartowski to CODEOWNERS for tiled
* Put benchmarks behind a flag
* kernel fix for AVX2, iq quants
* Fix for asan, leaking memory in test-tiled-mulmat and avoid stack use after return
* guarding env flags with std::call_once
* Simplified repacking for VNNI to a single call per macrotile
* No threadlocals anymore, aligned wdata access
* Doing aligned reads since we ensure alignment with padding in wdata
* Eliminated per-thread gather of Q8_K rows in mul_mat_id, we now gather/repack in a single pass. Repack method now takes pointer array to support both dense/normal and mmid paths. Interface with ggml-cpu.c simplified as a result
* Unified/simplified dispatch and support checks. Put details on wdata needed inside the kernel.h body, simplified interactions with ggml-cpu.c.
* Cleanup includes and whitespace, update src1_repack to return false if we don't need a special repack, so the common case is handled by driver
* Better detection of win32 and additional whitespace fixes
* Gating fuzz tests behind a parameter and some extra prints to try and fix slow CI hosts
* Optimized AVX2 kernel
* Changed interleave format and added ability to interleave in-place after dequant
* Repacks now happen in-place, 16x64 microtiles are independent of each other
* Only repack rows in groups of 16 as they're needed. Save work in low n_rows cases and optimize L1 usage in other cases
* Use long panels for memory-bound regime (M <= 16), reintroduce IQP path for benchmarks
* Fix unused warnings and cleanup. Improved IQ dequantization speed.
* Removed separate process benchmarks
* Revert "Removed separate process benchmarks"
This reverts commit 0688cf43d5e37ec7a52828c5576df563514fe760.
* AVX2 optimizations and guards for tests on windows
* Removed temp perf harness
* Remove perf-mulmat from build
* Removed IQP path, simplified tests to not use sub processes
* Cleaning up alignment of wdata
* Whitespace fixes and aligning L2 workspace to clean 512kb boundaries
* Update ggml/src/ggml-cpu/tiled/tiled-kernel.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Cleanup merge-duplicated declaration of test-backend-ops target
* Undo accidental line deletion in ggml.c
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The original function was broken on Windows for some unicode paths
Paths without a trailing separator now create the last directory too,
matching the function name. All current callers already include a
trailing separator, so this change does not affect them.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
which resulted in different greedy transcripts for 4.5% of English and 6.5% of Japanese
test utterances. In Japanese, some differences changed entire words.
This change:
* uses `log(x + 2^-24)` instead of clamping to the log floor
* uses a symmetric Hann window, equivalent to `torch.hann_window(periodic=False)`
* adds the normalization epsilon to the standard deviation instead of inside the square root
Only the `lfm2a` preprocessor opts into these behaviors. Other audio preprocessors are unchanged.
Tested on top of 84e76d8 using `llama-server` with CUDA and `temperature=0`, compared against
http://github.com/Liquid4All/liquid-audio fp32.
Test set:
* 200 LibriSpeech `test-clean` utterances (EN)
* 200 Common Voice `ja` test utterances (JP)
* identical 16 kHz audio passed to both implementations
| Greedy transcript identical to `liquid-audio` | Without fix | With fix |
| --------------------------------------------- | ----------: | ----------: |
| EN F16 | 191/200 | 200/200 |
| JP F32 | 187/200 | 200/200 |
| JP F16 | 187/200 | 199/200 |
The remaining JP F16 difference is a comma and matches the reference implementation's own bf16
output.
Mel relative L2 error versus `liquid-audio`:
* EN: 3.2% -> ~2e-6 median
* JP: 3.9% -> ~2e-6 median
* opencl: add A8 Q5_K non-MoE non dp4a + dp4a binary kernel
* opencl: fix s transpose - s only transposed for bin kernels
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* vulkan : fix build issue of legacy glslc version by adding GGML_VULKAN_COOPMAT_GLSLC_SUPPORT macro check for Intel FA shader compiling
* vulkan : add preprocess condition to filter out unsupported FA 2 phases kernels before creation.
* vulkan : move lock_guard for Intel FA shader pointer creation under CM1 compiling preprocessor
* metal: FWHT kernels for block widths above 512
The Metal FWHT covers widths 64 to 512, one row per simdgroup with N/32 values
per lane. Wider blocks need more registers per lane than that layout allows.
kernel_fwht_tg runs one row per threadgroup with 256 threads, so each thread
keeps N/256 values. Butterflies below the simdgroup width still shuffle, those
up to the threadgroup width go through threadgroup memory, and the rest stay in
registers. Same butterfly and sign convention as the simdgroup kernel.
Widths 64 to 512 keep the simdgroup kernel. 1024 through 8192 use the new one,
for both F32 and F16 sources.
The wide kernels allocate float[N] of threadgroup memory, 32 KB at 8192, so the
size check takes the device limit and reports those widths as unsupported where
they would not fit. Without that a device with less threadgroup memory would
accept the op and then abort on a nil pipeline.
test-backend-ops on M5 Pro: MUL_MAT_HADAMARD 26/26, MUL_MAT 1265/1265.
* cont : add TODOs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* rpc : include nb in the get_alloc_size cache key and floor the result at ggml_nbytes
* cont : remove redundant comment
* cont : add TODO
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* musa: use 16-byte copies for MUSA like sm_70+
ggml_cuda_get_max_cpy_bytes() derives the copy width from __CUDA_ARCH__. mcc
never defines it, so MUSA fell into the generic branch and returned 8 bytes
instead of the 16 bytes that every sm_70+ target gets. The value sizes the
per-thread copy unit of the FlashAttention K/V staging code (fattn-common,
fattn-vec, fattn-tile, fattn-mma-f16 shared-memory loads) and of mmq-vec-dot,
so every MUSA FlashAttention kernel moved half as many bytes per instruction.
On an MTT S5000 (mp_31, MUSA SDK 5.2.0) with Qwen3.8-27B-UD-Q4_K_M, -ngl 999,
-p 512 -n 64, -fa on: 751.15 -> 794.73 t/s prefill and 15.59 -> 15.69 t/s
decode. -fa off is unchanged (1050.05 -> 1052.86 t/s prefill), FLASH_ATTN_EXT
is unchanged (3984 ok / 0 fail / 1323 unsupported) and perplexity is
unchanged.
* musa: enable the CUB paths on MUSA
GGML_CUDA_USE_CUB and USE_CUB are selected by "CUDART_VERSION >= 11070", which
the MUSA SDK never satisfies: CUDART_VERSION is not defined anywhere under
/usr/local/musa/include, so the condition is always false and every CUB-based
path stayed compiled out on MUSA even though the SDK ships CUB and the kernels
build for mp_31. Select them from GGML_USE_MUSA as well. The device-wide
algorithms are usable too: cub::DeviceSegmentedSort compiles and produces
correct results on mp_31.
This lifts the ne[0] <= 1024 limit that ggml_backend_cuda_device_supports_op
applied to ARGSORT and TOP_K on MUSA. On an MTT S5000 (S5000, mcc 5.2.0):
ARGSORT 48 ok / 52 not supported -> 100 ok / 0 (CUDA parity), TOP_K 0 ok /
354 not supported -> 527 ok / 0. The other 20 per-op suites are unchanged, the
Qwen3-0.6B f16 (14.4679) and Qwen3.8-27B iq4_nl (5.1724) perplexities are
unchanged, and the 0.6B graph keeps the same nodes and splits (18 CPU + 18
MUSA0, SET_ROWS 1008) as before.
* musa: take the upstream code path where the toolkit supports it
Several guards were written for an older MUSA toolkit. Verified against MUSA SDK
5.2.0 and on an MTT S5000 (mp_31):
- device init: query cudaDevAttrCooperativeLaunch instead of hardcoding false.
The device reports cooperativeLaunch=1 and musaLaunchCooperativeKernel works
(verified with a kernel whose result was checked).
- device init: keep prop.warpSize instead of overriding it with 32. The device
reports 32 anyway, so this only removes the divergence.
- CUDA_SET_SHARED_MEMORY_LIMIT and the FA shared-memory raise: musaFuncSetAttribute
returns success and sharedMemPerBlockOptin is 192 KiB, so the kernels can use
more than the default 48 KiB.
- vendors/musa.h: add the cudaDeviceGetAttribute and cudaDevAttrCooperativeLaunch
mappings the device-init change needs.
Measured on one S5000 with Qwen3.8-27B Q4_K_M (-ngl 999, -r 3): pp512 968.27 ->
957.09 t/s, tg64 10.09 -> 10.23 t/s, FLASH_ATTN_EXT sweep identical (3975/3982
both), perplexity identical (80.2841 +/- 7.26772 both).
* musa: drop compile-time guards that MUSA's runtime gates already cover
mcc never defines __CUDA_ARCH__, so the arch-gated fallbacks in this group
were already taken on MUSA and the GGML_USE_MUSA guards on top of them only
kept the upstream text from being compiled:
- wkv.cu: the "#pragma unroll" suppression has no effect on the generated
code that is not already covered by the surrounding guards
- common.cuh: the MUSA-only __builtin_unreachable() in no_device_code() is
not needed to silence the compiler
- ssm-scan.cu: the SSD (Mamba-2 prefill) block and its dispatch are gated at
runtime by GGML_CUDA_CC_IS_NVIDIA(cc) and turing_mma_available(cc), which
are both false for PH1 (cc 0x100310), so compiling them changes nothing
- common.cuh: warp_reduce_max(half2) is guarded the same way as
warp_reduce_sum(half2) (FP16_AVAILABLE); the MUSA-only guard left the
function with no return statement. It has no caller today.
MTT S5000 (mp_31, MUSA SDK 5.2.0), MUSA_ARCHITECTURES=31: build rc=0. Against
an unmodified build of the same tree on the same card, FLASH_ATTN_EXT
(3984 ok / 0 fail / 1323 unsupported), SSM_SCAN (15/0), RWKV_WKV6 (6/0),
GATED_DELTA_NET (38/0) and MUL_MAT (1299/0/385 unsupported) are identical, and
perplexity with -fa on is bit-identical (5.1639 +/- 0.36673, 4 chunks).
* musa: do not use MMQ on PH1
test-backend-ops on an MTT S5000 (mp_31, MUSA SDK 5.2.0) fails 260 cases and every
one of them goes through the MMQ path:
- MUL_MAT with a batched src1 (any bs/nr != [1,1]): 109 cases across all
quantized types, e.g. 12 of 13 cases at n=16, while the plain [1,1] layout
passes
- every quantized MUL_MAT_ID: 147 cases, while the f16/f32 variants of the same
shapes pass
- MUL_MAT with more than ~512 tokens: 4 cases (n=509..4096); the small-n cases pass
The cuBLAS/dequant path is correct for all of them and the MMVQ path used for
small batches is unaffected, so quantized matmuls now take that path on PH1
instead of returning wrong values. 27B perplexity with default flags goes from
nan to finite, and the full suite reports 0 failures out of 22237 cases.
The MMQ defect itself (fastdiv, __umulhi, uint3 kernel parameters and
__CUDA_ARCH__-based MMA availability were all checked and are correct on this
part) is not addressed here.
* musa: keep the block barrier of the fused TOPK_MOE kernel reachable
topk_moe_cuda returns early for the rows past the end of the graph, but one block
covers TOPK_MOE_ROWS_PER_BLOCK (8) rows, so the last block is only partially filled
whenever n_rows is not a multiple of 8. On MUSA a warp that has already returned
blocks the block wide __syncthreads() below, which makes the kernel hang and the
launch time out. CUDA tolerates the exited warps, which is why the CUDA numbers
never showed it.
For MUSA, clamp the row index of those warps to the last row so that every warp of
the block reaches the barrier; they recompute the last row and write the same
values. The CUDA code path is unchanged.
On an MTT S5000 (mp_31) the fused TOPK_MOE cases change from a launch timeout with
no completed case to 418 ok / 0 not supported / 0 failed, i.e. the CUDA result, and
the other 101 per op suites are unchanged (0 failed, no count changes).
* musa: enable GATED_DELTA_NET
The op was turned off for every MUSA target because mcc could not build the kernel
at the time. The current toolkit builds it: with mp_31 and MUSA SDK 5.2.0 the file
compiles with zero errors and all 36 test-backend-ops GATED_DELTA_NET cases pass
against the CPU reference. 27B perplexity is unchanged.
While the op is refused, the scheduler has no choice but to run it on the CPU: 48
GATED_DELTA_NET nodes per forward pass. On an MTT S5000 (Qwen3.8-27B Q4_K_M, -ngl
999, one container, -r 3):
pp512 (FA off) 964.51 -> 2119.26 t/s
tg64 (FA off) 10.15 -> 15.50 t/s
* musa: name the stream capture query API for the graph aware kernels
argsort.cu and mean.cu call cudaStreamCaptureStatus, cudaStreamIsCapturing and
cudaStreamCaptureStatusNone inside their USE_CUDA_GRAPH blocks, but the MUSA
compatibility headers do not alias those names, so building with the experimental
GGML_MUSA_GRAPHS option fails with 7 errors in those two files. Map the three
names to their musa* counterparts, under the same guard that enables the graph
code, so the default build is untouched.
The option stays off by default: on an MTT S5000 the captured path measured
slower (pp512 693 vs 772 t/s, tg128 15.20 vs 15.39 t/s over two sessions) and the
borderline MUL_MAT cases are not reproducible between runs.
* musa: build the CI and docs for PH1 (MTT S5000)
The MUSA CI job and the documented default still targeted the first generation
(MTT S80, MUSA_ARCHITECTURES=21) while the current MUSA SDK targets PH1
(MTT S5000, 31). Move the job, ci/run.sh's default and the build docs to 31,
and run the job in the PH1 MUSA SDK devel image:
registry.mthreads.com/mcconline/inference/pytorch:2.9.1.post1-py3.10-musa5.2.0-mp31-devel-ubuntu22.04-amd64
That image needs two things the previous one did not: python3-venv for the
ccache-buckets step, which builds a virtual environment for the Hugging Face
CLI, and no time prefix on the build command, because container jobs run their
steps with sh and the image ships no time binary.
- #28068 builds the GDN q/k l2norm as ggml_scale(ggml_rms_norm(x, eps/n), 1/sqrt(n)). This adds 2 SCALE nodes per GDN layer, 96 extra kernel launches per ubatch on Qwen3.8-27B (48 GDN layers).
- The extra kernels take no measurable GPU time, but each launch has a host/driver cost. It is small with plain batch processing and about 10x larger with draft-mtp speculative decoding.
- rms_norm_f32 gets a do_scale flag, the same pattern as do_multiply/do_add, so the fused path shares the kernel, the reduction and the launcher. It computes scale * (rsqrt(mean + eps) * x), which matches the unfused rms_norm + scale bit for bit, so #28068 numerics are kept.
- Fusion only fires when SCALE has no bias and the rms_norm output has a single consumer (ggml_can_fuse).
- Metal (#28948) and SYCL (#28931) already fuse the same pattern.
Measured on 2x GTX 1080 Ti (sm_61, PCIe 3.0 x16 + x4), i7-13700KF, Windows 11, driver 582.66, CUDA 12.9.
Qwen3.8-27B-UD-Q4_K_XL, -ngl 99 -ts 53,47 -ot token_embd=CPU, master fee39dd92.
llama-bench -ub 128,512 -p 512,2048 -n 128 -r 5, tok/s:
build pp512@128 pp2048@128 pp2048@512 tg128
master 367.7 419.1 385.4 12.90
master + fix 372.6 420.6 388.6 12.98
+1.3% +0.4% +0.8% +0.6%
llama-server cold prefill, -c 56000 -ub 128 -b 2048, draft-mtp n-max 3 p-min 0.5, mean of 2 rounds x 3 reps:
build pp 8000 pp 20000
master 356.5 322.0
master + fix 371.4 (+4.2%) 337.4 (+4.8%)
- Launches per ubatch go from 1032.9 + 841.7 back to 978.9 + 799.7 (CUDA0 + CUDA1), the b10828 count. The GPU op sum is unchanged.
- test-backend-ops RMS_NORM_SCALE, NORM_SCALE, RMS_NORM_MUL_ADD, RMS_NORM_MUL_ROPE, RMS_NORM, RMS_NORM_BACK, NORM, L2_NORM and SCALE all pass on both GPUs.
- Perplexity is identical to the unfused build: 3.2030 +/- 0.0559 at -c 2048, 16 chunks.
- Draft acceptance counts per request match the unfused build.
Assisted-by: Claude Opus 5.5
- return early when the graph has no nodes
- drop the redundant reset of capture_compute: the decrement at the top
of the function already transitions the counter from 0 to -1, so a
capture happens exactly once
- hint at METAL_CAPTURE_ENABLED=1 in the capture error message
- pass capture_compute == 0 (not the raw counter) as use_capture to
ggml_metal_op_init, so GPU debug-group markers are only emitted on the
captured compute
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : cache sparse FA indices in shared memory
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : simplify shared memory size calculation
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* pi : update general
* metal : unroll sparse index load
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* hexagon: fix accuracy issue in Q8_0 N=1 MUL_MAT
* hex-quant: fix register spills
* hex-mm: use dma for all dyn.quant paths
Co-authored-by: Aparna M P <aparmp@qti.qualcomm.com>
* hex-mm: remove obsolete run_quant_task
* hex-mm: update tracing to properly wrap the events
* hex-mm: use act for activation data in all paths
* hex-mm: use act_ instead of src1_ to avoid confusion in fused kernels
* hex-mm: remove/reroute the rest of the non-DMA act (aka src1) logic
* hex-dma64: yet another pass at cleaning up the dma_addr_t casts
* Update ggml/src/ggml-hexagon/htp/matmul-ops.h
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update ggml/src/ggml-hexagon/htp/matmul-ops.c
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update ggml/src/ggml-hexagon/htp/matmul-ops.c
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update ggml/src/ggml-hexagon/htp/matmul-ops.c
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Aparna M P <aparmp@qti.qualcomm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* vulkan: add int8 coopmat quantized matmul shader
* apply scales inline
* use scalar sums
* probe and directly access coopmat values instead of going through shmem
* add q8_0 support
* add BK_STEP to shader, default to 2
* use larger workgroups
* double buffering
* preload scales
* coopmat load first, then wmma
* use float for scales
* add faster RDNA int->float conversion
* workgroup scheduling for cache proximity
* clean up
* use wave32
* restructure for vgpr use
* skip computation for inactive tiles
* only force subgroup size 32 on AMD RDNA
* use BK_STEP 4
* fix compilation
* move quant-specific prefetch function out of main file
* add q4_1, q5_0, q5_1 support
* restructure mmq cm1 functions
* enable mul_mat_id support
* fix segfault
* fix mul_mat_id bug
* support iq4_nl and mxfp4
* remove elem row/col fast path, invalid for RDNA4
* use shmem arrays for LUTs
* use 4-byte loads where possible
* add q3_k, q4_k, q5_k, q6_k and nvfp4 support
* fix l warptile
* improve performance
* improve performance
* improvements
* dedup b scales
* merge shmem arrays
* undo uint8_t, gate to RDNA3/4
* add RDNA4 architecture, use for hardcoded coopmat elem thread access, set BK_STEP back to 4
* improve offset application
* clean up
* fix iq4_nl and nvfp4 performance
* rdna4 tuning
* use BK_STEP 2 on MUL_MAT_ID
* adapt to upstream changes
* fix shmem support function, clean up comments
* fix warptile logic
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
* vulkan: add IQ4_XS support to the coopmat1 integer matmul shader (#28440)
Adds IQ4_XS to mul_mmq_cm1: dedicated block_a_load/block_a_to_shmem that
expand both nibbles of each packed32 word through cm1_kvalues, LOAD_VEC_A 8
and an IQ4_XS-sized a_panel_bytes estimate for the L2-friendly scheduling.
Assisted-by: OpenAI Codex
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* avoid compiling f16 acc shader variants
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* model : fold Ling 3.0 VL into the BailingMoeV3 architecture
Assisted-by: Scout
* model : keep shared NORM rope list intact when gating bailingmoe3 on mrope sections
---------
Co-authored-by: aetherbird <aetherbird@users.noreply.github.com>
* test-save-load-state : print a per-model results table in --models mode
in --models mode the output was very heavy: every model printed its
token dumps, per-test headers and PASS lines. instead, silence all
logging except the table itself (common_log_set_verbosity_thold(0)
leaves only LOG / LOG_LEVEL_OUTPUT) and print one row per model with
one column per test, colored PASS/FAIL/SKIP cells, row by row.
- run_save_load_tests_for_model returns a test_suite with a dynamic
std::vector<test_status> and continues past failures: tests 3-5 are
SKIPped when the baseline (test 1) fails, model init failure skips all
- per-test token dumps, test headers and PASS lines are demoted to
LOGV(LOG_LEVEL_INFO, ...) so they still show in single-model mode
- the table header/rows derive their columns from test_names; the
model name is printed and flushed before the suite runs so the model
currently in flight is always visible
- single-model output and exit codes are unchanged
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* test-save-load-state : print example usage on -h
add a print_usage callback passed to common_params_parse, so -h/--help
also shows example commands for the tool-specific --models option and
the -lv verbosity level
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* test-save-load-state : remove comments
ref: https://github.com/ggml-org/llama.cpp/pull/29316
Assisted-by: pi:llama.cpp/Qwen3.8-27B
make-release-desc.sh now emits "Changelog since [vX.Y.Z](<repo>/releases/tag/vX.Y.Z)"
instead of a plain version string, so the release notes link back to the previous release.
The repo URL is derived from the origin remote (SSH or HTTPS); if it cannot be
resolved (local run without origin), the title falls back to plain text.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Resolve the target arch with get_model_architecture so vision targets
(e.g. Lfm2VlForConditionalGeneration) map to their text model for the vocab.
Fix double rope reorder for LFM2/LFM2.5 DSpark drafters
* tests: add backend option to test-llama-archs
* Update tests/test-llama-archs.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* remove extra space
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Restore get_cache_directory() as fs::path as string() can be lossy on Windows
Partially reverts #29125
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
ggml_conv_1d_dw builds its im2col as f32 when the kernel is bf16, then
multiplies the two, so a depthwise convolution over bf16 weights asks
for kernel_mul_mv_f32_bf16, which was never instantiated. The base, the
_4 and the _short families are filled in next to their bf16 neighbours,
inside the same runtime guard, so a device without bf16 support is
unaffected.
* CUDA: enable sparse-fa for dsv4 prefill (again)
* CUDA: unroll the query loop of the sparse mask scan
The query loop of flash_attn_mask_to_sparse_indices has a runtime trip
count, which keeps the unrolled scan over the values of a lane from
issuing its loads together. Template the kernel on ncols1 so the loop
is bounded at compile time: batch one decodes compile to straight line
code and the scan drops from 46 to 17 us at 49k columns on sparse
decode shapes.
* CUDA: pick the out of bounds check of the sparse mask scan in host code
The query loop of the ncols1 == 8 scan keeps a runtime bound and an
early exit, so it does not unroll past its first iteration. Template the
kernel on whether the last group of queries is partial, decided on the
host from n_queries, and hoist the column bound out of the loop: the
loop becomes straight line code and the batched sparse op at 49k
context drops from 586 to 244 us.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* jinja : parse unary +/- before variables
Lexer already emits unary_operator for -n / +n, and runtime executes
unary -. Parse them at multiplicative precedence so slices like
items[:-n] and GigaChat indent[:-indent_factor] work.
* jinja : keep filters/tests outside unary operands
Unary +/- must bind only the primary/postfix operand so -n|abs is
(-n)|abs, not -(n|abs). Add unary + and filter/test regression coverage.
Signed-off-by: sinksilk <785976238@qq.com>
---------
Signed-off-by: sinksilk <785976238@qq.com>
The OpenAI chat completions API specifies content part type "video_url"
with a {"url": ...} object, and clients typically send data: URIs
(e.g. data:video/mp4;base64,...). The llama-server only accepted the
non-standard "input_video" type and rejected data: URIs for video
(accept_base64_uri=false), so any OpenAI-conformant client failed with
"unsupported content[].type" or "Invalid uri format".
- accept "video_url" as an alias of "input_video"
- read the media object from whichever key was used
- allow data: URIs for video (data:video/*), as already done for images
This commit adds a new recipe/target to the Makefile which allows the
logits verification to be run on pre-existing model outputs.
The motivation for this is that for large models it can take a long time
to run them models, and especially for the original model which seldom
changes this is very time consuming. With this change we can run the
original model one which will store the tokens and logits, and then
manually run the converted model and the run use this recipe to verify
them against the orignal model.
* dspark: add Gemma 4 draft support
Add GGUF conversion and runtime support for full-attention and SWA Gemma 4
DSpark drafts, including tied output weights and boolean backbone metadata.
Assisted-by: Codex
* dflash: infer Gemma draft features from metadata
* vulkan: optimize IQ4_XS matmul kernels
Assisted-by: OpenAI Codex
* vulkan: address IQ4_XS review nits
- drop the dead LOAD_VEC_A != 8 branch in the IQ4_XS shmem load; iq4_xs is
in lut_load_vec_a()'s "8" list, so that path is never generated
- disable MMVQ for IQ4_XS on Intel (27.3% tg regression on A770)
- remove a stray empty line in types.glsl
Assisted-By: Claude Opus 5 <noreply@anthropic.com>
Since #28732 our internal symbols are exported. A duplicate copy dlopened and
dlclosed by ggml_backend_load_all() then interposes them, so its destructors
destroy the live vk_instance and later device queries hit the GGML_ASSERT on
vk_instance.device_indices. Hidden visibility exports only GGML_BACKEND_API,
as before #28732.
Fixes#29138
Assisted-by: henk:claude-fable-5
Avoid returning references through lambdas that hold a local cast pointer, which triggers -Werror=dangling-reference in some CI compilers. Reuse the precomputed select_expr pointer directly.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* server: route every model load through the queue
A model loaded by the fast path has no queue entry, so tick() evicts
it at its LOADED transition before its own request is proxied. Every
load now joins the queue, whose entry protects the model until its
waiters leave.
* server: do not admit requests into a stopping model
A request for a model that is being stopped still sees it LOADED and
is proxied into the dying child. Such a request now joins the queue
and is served by the next instance. The stopping mark is cleared
under the same lock that sets UNLOADED, so no request can see a
model that is neither stopping nor unloaded while its child is gone.
* vulkan : Intel FA kernel optimization for split k path
* vulkan : Host code update for Intel split k FA kernel path selection, fix A770 Linux op test failures
* vulkan : use symmetric coopMatMulAdd() in flash_attn_decode_phase_1 shader to resolve test op failre on A770 Linux with 26.2.3 mesa driver
* vulkan : fix editorconfig issue in flash_attn_decode_phase_2.comp
---------
Co-authored-by: Liu, Russell <russell.liu@intel.com>
* metal : gate mul_mm_id src1 rescale behind ggml_prec
Assisted-by: Claude Fable 5.1
* ggml-webgpu: reject MUL_MAT_ID when src1 precision is F32
* cuda/vulkan: reject MUL_MAT_ID in supports_op when src1 prec is F32
fix `supports_op` to return false for failing backends when the specified src1 precision is f32
Assisted-by: Claude Fable 5.1
---------
Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
* model : add DFlash layer-input taps for HunyuanVL
DFlash speculative decoding needs the target graph to expose the residual
stream entering each layer (res->t_layer_inp[il]) - the draft model reads
those tensors to build its cross-context. Qwen3 and the other DFlash-capable
targets register them, but the Hunyuan graphs do not, so serving a DFlash
draft against a HunyuanOCR target aborts during the first graph build:
GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null")
Register the tensor at the top of the layer loop, mirroring qwen3. The
layer input is the residual stream entering layer il, i.e. the output of
layer il-1, which is what the draft's target_layers metadata refers to
(the converter writes target_layer_ids+1). hunyuan-dense.cpp reuses this
graph, so it is covered as well; hunyuan-moe has a separate graph and is
untouched.
The vector is only read when a speculative implementation enables those
layer ids, so there is no behaviour change without a draft model.
Tested with tencent/HunyuanOCR 1.5 and its DFlash draft: image requests now
run, draft acceptance is ~0.5 and the OCR output is byte-identical to the
non-speculative run.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft conversion against HunYuan targets
Converting a DFlash draft with a HunYuan target failed in two ways.
1. DFlashModel.set_vocab() reuses the target class' vocab handling by
calling it unbound with the draft instance, but HunYuanModel.set_vocab()
called self._fix_special_tokens(), a method that only exists on
HunYuanModel, so the conversion always aborted with
AttributeError: 'DFlashModel' object has no attribute '_fix_special_tokens'
Make the vocab helpers module-level functions taking the model
explicitly, so they do not depend on the instance being a HunYuanModel.
They have no other callers, so the two id lookups are folded into
_fix_special_tokens().
2. The delegated call runs with self.dir_model pointed at the target but
keeps the draft's self.hparams, so config lookups inside the target's
vocab code (the pad_token_id < 0 guard, eod_token_id) read the draft's
config instead of the target's. That aborts on targets with
pad_token_id = -1 (e.g. the HunyuanOCR v1.0 checkpoint) and otherwise
writes special token ids that disagree with the target.
Add _vocab_hparams(): it returns the target's config (with text_config
merged to the root, as TextModel does) when the model is a draft
converted with --target-model-dir, and the model's own hparams
otherwise, so a normal conversion is unaffected.
Tested: converting tencent/HunyuanOCR/dflash succeeds with both the 1.5 and
the v1.0 target; converting the base model without --target-model-dir
produces a byte-identical GGUF to before.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft vocab against HunYuan targets
Switch hparams to the target config for the duration of the borrowed
set_vocab(), matching the existing dir_model swap, instead of teaching
HunYuanModel::set_vocab about draft models.
* convert : fix HunYuan special token ids for DFlash drafts
* convert : use load_hparams for HunYuan special token ids
* convert: add MiMo-V2.6 support
Hoist the K3 mxfp4 conversion repack into base.py so it can be reused
Remove decoder from mmproj convert
* Update conversion/mimo.py
* fix: use autoparser
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
The snapdragon CI builds packages only to feed the QDC device tests, so
Hexagon NPU binaries never reached the releases page. Build both targets
in release.yml and attach them as release assets.
This commit tweaks the Toaster element to include a close button.
These toasts often cover other UI elements like the model selector, and
this change avoids having to wait for them to disappear on their own
(e.g. after a load failure).
* hex-gdn: start putting together HMX support for GDN
* hex-gdn: working hmx but not-pipelined and slow for now
* hex-gdn: re-write vtcm layout handling and prep for pipelining
* hex-gdn: starting to pipeline hmx and dmas
* hex-gdn: add hvx threading for most pipeline stages
* hex-gdb: add detailed trace events
* hex-gdn: vectorize expfs and use aligned hvx reads/writes
* hex-gnd: vectorize the rest of expf
* hex-gdn: optimize tail processing (pad partial chunks)
* hex-gdb: avoid float up/down casts in hot loops
* hex-fa: remove float up/down casts from inner loops
* hex-gdn: do exp() in f16 to improve HVX utilization
* hex-gdn: optimize tiler
* hex-hmx: bump hmx-queue to 128 and dispatch all GDN gemms at once
* hex-gdn: further pipeline improvements
* hex-gdn: optimize gdn prep stage
* hex-gdn: yet more tweaks to optimize GND_SOLVE task and pipeline
* hex-gdn: improve accuracy and optmize gdn-prep further
* hex-gdn: fix rebase conflict
* hex-bufs: revert max_bufsize enforcement, it is enough to just enforce max_vmem
* hex-scripts: improved inspect script to avoid false alarms in reg spill detector
* hex-fa: improve inline softmax with in-reg VKQ32 accum
* hex-fa: minor improvement for dma pipeline in hvx kernel
* hex-fa: reduce ddr reads by 20-30% during token gen
* hex-gdn: proper alignment for hvx vtcm spads
* llama-context : report graph inputs and input tensors during sched reserve
- fix the tg (token generation) graph bs label to use n_seqs instead of a hardcoded 1
- report the number of graph inputs from llm_graph_result::inputs for both the pp and tg graphs
- report the number of input tensors (nodes and their src tensors flagged with GGML_TENSOR_FLAG_INPUT)
- log a warning when an input tensor has an op other than GGML_OP_NONE
- log a trace line for each input tensor and the nodes (name and op) that use it
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* cont : count input tensors before reserving the sched
* wip
* llama-graph : name the unnamed graph input tensors
- name the kv-cache idxs input tensors (attn_inp_k_idxs, attn_inp_v_idxs)
- name the recurrent state copy idxs input tensor (rs_s_copy)
- report the input tensor shape in the sched_reserve trace
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* llama-context : rename "graph inputs" to "graph input objects"
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* llama-context : report the sched reserve graph stats on a single line
- print nodes, splits, input objects and input tensors in one line
- when the pp and tg graphs differ, print each value as 'pp / tg'
and annotate the line with the batch sizes used for each graph
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : pad logs
The 5-argument load_ldmatrix added in 1884824fd only defines tile<16,8>, so the Volta tile<8,4> does not match. See https://github.com/ggml-org/llama.cpp/issues/29222 for details. Building on 1884824fd, generalize the tile shape of the 5-argument load_ldmatrix from <16,8> to <I,J>, so the non-swizzle branch forwards to the 3-argument loader for any shape. Local compilation and testing passed.
Assisted-by: DeepSeek V4.1 Flash (OpenCode)
convert_unary handles the contiguous case through the general strided kernel,
one element per thread: each lane reads 4 bytes and writes 2. Converting the
activations for a bf16 matrix multiplication that way moves 126 MB in 1021 us
on gfx1151, about 65% of what the memory system can do.
Give the contiguous path its own kernel that takes four elements per thread
through a vector type, so a warp loads 512 bytes at a time instead of 128. It
is used only when the element count is a multiple of four and both pointers
carry the alignment the vector type needs, and falls back to the strided
kernel otherwise.
Model level, Qwen3.8-Next-Flash IQ3_XXS on gfx1151, llama-bench -ub 2048 -r 6,
mean of the last 3 reps, ABBA counterbalanced:
pp2048 688.0 680.0 -> 694.3 691.1 +1.26%
tg128 24.8 24.8 -> 24.8 24.8 +0.14%
Every conversion in a prefill takes the new kernel (kernel trace: 1146
convert_unary_cont_vec4, no convert_unary). Output is bit identical; MUL_MAT,
MUL_MAT_ID, CPY, CONT, GET_ROWS and SET_ROWS pass.
Assisted-by: Claude Opus 5
* tests/test-backend-ops : allow regex entries in the -o filter
so far -o only accepted a comma separated list of exact op names or
full test case strings. entries that are not plain op names are now
treated as regexes matched against the op name (e.g. "MUL_MAT.*"),
while plain names keep their exact-matching behavior so that
"-o ADD" does not match ADD_EX etc.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : don't print the FA vec slice log when not needed
* tests/test-backend-ops : reformat the help text
use the same style as the other tools, with separate sections for
modes, options, and examples
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Allow configuring --temp, --top-p, --min-p, --repeat-penalty,
--presence-penalty and --frequency-penalty via LLAMA_ARG_* so
llama-server can be fully controlled from an EnvironmentFile
(e.g. systemd on Debian).
Use `llama-gen-docs` to regenerate the readme files.
In router mode, authentication belongs to the router. unset_reserved_args()
already unset LLAMA_API_KEY, but did not unset LLAMA_ARG_API_KEY_FILE.
When --api-key-file was passed, children re-validated against file keys only,
causing clients using --api-key to 401 on chat completions (#28820).
In addition, router internal calls without auth headers (such as
POST /v1/streams/lookup and DELETE /v1/stream) were silently rejected with 401.
Unset LLAMA_ARG_API_KEY_FILE in unset_reserved_args() so no API keys reach
child instances. This keeps keys out of child argv, ensures all keys the router
accepts work end-to-end, and prevents router internal stream calls from 401ing.
Fixes#28820
* Fixed json enum handling
Added common_json_value handling for enum values.
Added tests/test-json.cpp to cover testing of some aspects of common_json.
* Removed tests as requested.
* Applied recommended style and simplification
Simplified by delegating enum constructor to the constructor of the underlying type
Matched style of surrounding templating code
* Implemented ARM NEON DP q1 4x4 repack
* Hoisted out scaling by b_d in gemm
* Added 4x8 NEON I8MM repack kernels
* Cleanup for q1 arm repack
* Added missing aliases for arch fallback
* Corrected unused var statements
* Extended table guard condition to account for i8mm w/o dp build
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Moved new declarations and references to groups' top
* Moved declarations for uniformity
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* hex-dma64: enable support extended buffer mappings and 64bit dma
hex-dma64: expand binary ops to support more DMA scenarios
hex-dma64: add binary-ops.h
hex-dma64: add --hex-dma64 to run.py and fix minor issues
hex-dma64: update SSM_CONV to use dma with proper support for 64bit
hex-ops: remove obsolete gate for % 128 in binary ops
hex-l2: dont check weight tensors against dirty ranges
hex-dma64: most binary ops now support dma
hex-dma: use dma_addr_t instead of plain uint64_t to avoid overhead on older targets
hex-dma: update all dma users to use dma_data (instead of pointers)
hex-dma64: simplify lazy buffer mapping and clonning
hex-fusion: factor out try_fuse_common that checks for dma64 buffers
hex-bufs: minor cleanup for mmaping logic
hex-bufs: simplify buffer clonning
hex-ssm-conv: tighten gating checks and check vtcm size in kparams
hex-binary: fix incorred mod/wrap in scalar ops
hex-binary: make sure to call precompute kparams in support checks
hex-dma64: update addr handling in mm,concat,binary
hex-dma64: fixing up leftover of dma_addr_t conversion
hex-binary: redo the kernel selection again and fix regressions in MOEs
hex-binary: specialize per-type/per-op
hex-binary: vtcm-layout and per-src dma-queue
hex-dma64: update dma_push to transparently handle 64bit/extended
* hex-cpy: fix improper rebase with the fixes for cont. tensors
* hex-dma-cpy: update CPY to use safe dma rows/size limits
* hex-mmap: bump number of mmaps to 64 to allow avoid eviction in larger models
* hex-dma: add support for the secondary ring as a fallback for too-large transactions
* hex-rope: fix freq_factors access with 64bit dma
* hex-dma: audit all ops for proper use/gards for 64bit addresses
* hex-dma64: uninline glu-compute funcs to avoid register pressure due to 64bit addr math
* hex-dma64: refactor binary ops to separate dma loops
* hex-devel: add inspect script to help with dbg and analysis
* hex-dma: refactor dma-pipelines in unary-ops
* hex-dma: rewrite softmax to use dma
* hex-dma: rewrite GDN dma loops and improve HVX register usage
* hex-gdn: fuse GDN+CPY
* hex-mm: factor out HVX solver
* hex-mm: remove hvx-flat kernels, the chunked version now handles vtcm limits much better
* hex-buffs: reject huge buffer allocations that we cannot memory map
* hex-inspect: add logic to look for float promo calls
* hex-mm: reduce HVX register spills in HVX prompt kernels
* hex-bufs: do not double count buffers from tensors in the same op
* hex-roll: fix merge conflict
* hex-dma: reroute all matmul ddr kernels to new chunked dma/vtcm kernels
* hex-dev: update developer docs to include inspection for register spils and float promos
* hex-ops: forgot to add new headers
* hex-softmax: fix gpt-oss dims
* hex-dma64: cleanup dma_addr_t casts
* hex-dma64: add support for dma/vtcm for flash-atten with sinks
* hex-mm-add: fix MUL_MAT+ADD fusion with bias.weights in extended bufs
* hex-add-id: add support for dma for src1 (exp. table)
* hex-dma: imrpove v73 fallback paths
* hex-bufs: do not drop extended mappings during va defrag
* hex-scripts: fix flake8 warnings
* hex-docs: fix editor-config warnings
* hex-inspect: fix warnings from ty
the dsv4_hc_pre kernels hardcoded hc = 4 via a constexpr used with
simd_shuffle, so the op was rejected by supports_op for any other hc
and fell back to CPU. Kimi-K3 uses dsv4_hc_pre with hc equal to the
number of banked checkpoints in the cross-layer residual stack, which
grows with the layer index.
pass n_hc as a function constant (FC_DSV4_HC) with per-n_hc pipeline
variants, and loop over it in both pre kernels with direct loads
add test-backend-ops cases for hc = 1, 2, 3, 5, 8 and 65, gated and
not gated
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ui : let the chat column shrink below its content width
The chat column is a flex item, so its automatic minimum size kept it as wide as the widest row inside it. Message rows cap at max-w-3xl plus padding, so a narrower window pushed a page-level horizontal scrollbar.
Set min-w-0 on the column so the inner scroll containers take over.
Assisted-by: pi:deepseek-ai/DeepSeek-V4.1-Flash
* ui : wrap markdown tables in a scroll container
Markdown tables render as a bare <table>, which keeps its content-driven minimum width and can stretch the chat column past the window. The table-wrapper CSS already existed, but nothing produced the wrapper.
Add a rehype plugin that wraps each table in div.table-wrapper, following the existing enhance-* plugins.
Assisted-by: pi:deepseek-ai/DeepSeek-V4.1-Flash
* ui : scroll long inline content inside markdown blocks
Long unbreakable content (inline code, paths, hashes) widened the message row and spilled over the neighbour elements. Give each markdown block a horizontal scroll container, and the content root one as well, since the trailing block renders with display: contents and has no box of its own.
Assisted-by: pi:deepseek-ai/DeepSeek-V4.1-Flash
* ui : use exact transition properties for markdown images
transition: all repainted every property and 300ms felt sluggish. Name transform and box-shadow at 200ms ease-out, and gate the hover scale behind (hover: hover) and (pointer: fine) so touch taps do not trigger it.
Assisted-by: pi:deepseek-ai/DeepSeek-V4.1-Flash
* ui : fit wide image attachments to the message width
Attachment thumbnails used a fixed height with w-auto, so a wide image kept its aspect-driven width and, being flex-shrink-0 in a right-aligned bubble, overflowed to the left of the message row.
Cap the thumbnail with max-height and max-width instead of a fixed height so it scales down proportionally, and let it shrink outside the single-row carousel.
Assisted-by: pi:deepseek-ai/DeepSeek-V4.1-Flash
* ui : keep long tool call titles inside the message row
A tool title could not shrink below its content, so a long path escaped the message row. Let the title span shrink and scroll, and for the file tools put the value on its own line only when it does not fit, with the value as the only scroll container.
Assisted-by: pi:deepseek-ai/DeepSeek-V4.1-Flash
* ui : render get info as a collapsible block with a table
get_info rendered its own always-open row with the values trailing the label. Use the shared ToolCallBlock chrome so it collapses like the other tools, and list os and cwd as table rows with the key as a row header.
The error and pending states now show inside the body, including the plain-string errors the server tools path produces.
Assisted-by: pi:deepseek-ai/DeepSeek-V4.1-Flash
* test : pin the server mode in the add menu a11y story
The story asserts the add menu's first enabled item is the reasoning submenu, which is mounted only outside router mode. The vitest dev server proxies /props to whichever server is running, so the assertion depended on the machine's server mode and failed whenever a router was up.
Pin the mode in the story, including props.role so a re-detection cannot flip it back.
Assisted-by: pi:deepseek-ai/DeepSeek-V4.1-Flash
* ui: wrap long markdown tokens instead of scrolling every block
Making each markdown block and the content root a horizontal scroll
container turns any hover transform into a scrollbar: the blockquote
translate and the image zoom overflow their block and flash a scrollbar
under it. Each block also becomes a block formatting context, so the
paragraph margins stop collapsing across blocks and the spacing doubles.
Drop both overflow-x rules and let long unbreakable tokens wrap with
overflow-wrap: break-word on the content root. break-word leaves the
min-content width untouched, so wide tables and code blocks keep
scrolling inside their own containers.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* metal: add F16 input to the FWHT
The Metal FWHT kernel accepts F32 input only. This change makes the source
type a template parameter, so the kernel reads an F16 source directly instead
of requiring a converted copy. The F32 instantiations are unchanged.
The pipeline name now carries the source type, and supports_op accepts an F16
src1 for the Hadamard hint at the four sizes the kernels cover. Every other
F16 src1 path still goes through ggml_metal_supports_mul_mat_op.
These are the test cases mentioned in #27779.
test-backend-ops on M5 Pro: MUL_MAT_HADAMARD 16/16, MUL_MAT 1265/1265.
* metal: ask the same FWHT question in supports_op and the dispatch
supports_op admitted an F16 src1 on the type, the hint and the width alone, but the
dispatch also requires src1 and dst to be contiguous and the same shape. A Hadamard
hinted MUL_MAT that passed the first and failed the second reached the generic path,
which has no F32 src0 by F16 src1 kernel, and aborted on a nil pipeline:
kernel not found in any metal library: base = 'kernel_mul_mv_f32_f16_4'
ggml_metal_encoder_set_pipeline: nil Metal pipeline
ggml_metal_use_fwht now holds the whole condition and both callers use it, so they
cannot drift apart again. The added test case has src1 and dst of different shapes,
which aborted before this change and is declined by the Metal backend after it.
* metal: branchless butterfly select in the FWHT simdgroup kernel
Review suggestion. Replaces the ternary in the shuffle stages with
val2 - val + 2*((lane & i) == 0)*val, which is the same value without the
select.
Measured on M5 Pro, interleaved A/B, five rounds, first discarded, on a
Hadamard matmul with block 512 and 65536 rows so the kernel rather than the
launch dominates: 1324.6 us before, 1285.0 us after, a 3.0% gain, and faster
in every round. At the shapes already in the perf suite the op runs 1.6 to
3.9 us against a 1.6 us launch floor, so the difference is not visible there.
FOR_UNROLL on the same loops was also measured and made no difference, the
delta changing sign between rounds, so it is not included.
* metal: move the FWHT dispatch predicates to ggml-metal-common
Review feedback. ggml_metal_use_fwht and ggml_metal_fwht_supported_size were
static inline in ggml-metal-device.h. They now follow the
ggml_metal_op_mul_mat_use_mm pattern: declared in ggml-metal-common.h and
implemented in ggml-metal-common.cpp, which is already the home for helpers
shared between supports_op and the op dispatch. The predicate is named
ggml_metal_op_mul_mat_use_fwht to sit alongside the _use_mm pair it parallels.
This also fixes the macos-latest-arm64 build. The header needed ggml-impl.h
for ggml_get_op_params_i32, but ggml-metal-device.h is reached from
tools/tuning through ggml-metal-tuning.h, and that target does not have
ggml/src on its include path. ggml-metal-common.cpp already includes
ggml-impl.h, so the accessor is used normally there and the header goes back
to needing nothing extra.
* metal: keep the FWHT size check internal and group the dispatch helpers
Applies the patch from the review. ggml_metal_fwht_supported_size becomes
static in ggml-metal-common.cpp since nothing outside it needs the size list,
which also drops stdint.h from the header again, and
ggml_metal_op_mul_mat_use_fwht joins the existing _use_mm declarations under
their shared comment instead of carrying its own block.
* tests: drop the mismatched-shape Hadamard case
I added a case with m != k to cover an abort, but the hint is a promise that
src0 is a Hadamard matrix, so src0 is square and dst has the same shape as
src1. Every other case in the suite holds to that. The case was not a valid
op, and on CPU it compared the FWHT against a real matmul of a non-square
src0, which cannot agree.
The supports_op and dispatch conditions still come from one predicate, which
is what keeps them from disagreeing on contiguity.
* chat: add dedicated Ling 3.0 (Bailing V3) parser
Ling 3.0 Flash templates pre-open the think block in the generation
prompt, so the model never emits an opening <think>, and a tool call can
arrive before any </think>. The generated autoparser terminated reasoning
only at the close tag, which classified such tool calls entirely as
reasoning_content: clients received content="" with no tool_calls and
agent loops died as reasoning-only turns.
Adds a specialized parser that terminates reasoning at the think close
tag or at a <tool_call> start, mirroring the hand-written Qwen3-Coder and
Kimi K3 parsers and the reference vLLM/SGLang Ling3 parser (which treats
<tool_call> as an implicit reasoning terminator). Detection is gated on
the <role>...</role> section markers, unique to this family among the
tagged-argument templates.
Adds the Ling 3.0 Flash chat template and tests covering the
unclosed-think tool call (full parse and streaming), healthy closed-think
paths, trailing prose, parallel calls, marker-like strings in argument
values, string-union and non-string argument types, and
reasoning_format=none.
Assisted-by: Kimi Code
* tests : move Ling 3.0 test
---------
Co-authored-by: aetherbird <aetherbird@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
* server-models : show source per model in log
- Show [source] tag (preset/models_dir/cache) per model instead of cryptic * marker
- Show HF hub cache path in the 'Loaded cached model presets' log
- Add hf_cache::get_cache_dir() public accessor
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : pad log
* metal : add top-k MoE fusion
Adds a Metal fusion for SOFT_MAX + ARGSORT + GET_ROWS with optional
routing-weight normalization and scale, matching the top-k MoE fusion
available in the CUDA and Vulkan backends. The fused kernel writes the
selected expert ids and routing weights directly, eliding the separate
softmax, argsort, get-rows, sum-rows, clamp, div and scale kernels.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : add MoE weighted reduction fusion
Fuses MUL(experts, weights) plus the expert VIEW/ADD chain into one kernel
that computes the weighted sum directly. The graph_optimize hook keeps the
expert and weight buffers alive until the fused output so the allocator cannot
reuse them while the kernel is still reading them.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* tests : expose MoE weighted reduction in fusion baseline
Use 2 experts per token in the generated MoE test models so the Metal
MoE weighted reduction fusion (MUL + ADD) is exercised by test-fusion.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : fuse RMS_NORM + SCALE
Adds NORM/RMS_NORM + SCALE fusion to the Metal backend by reusing the
norm+mul kernel with a scalar scale flag. Adds test coverage for both
NORM+SCALE and RMS_NORM+SCALE and regenerates the fusion baseline.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use function constant for RMS_NORM + SCALE
Replaces the runtime use_scale karg with a Metal function constant. The
norm+mul kernel is compiled with FC_norm_use_scale=false for MUL fusion and
FC_norm_use_scale=true for SCALE fusion, so the fused kernel has no runtime
branch.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use function constant for top-k MoE with_norm
Replaces the runtime with_norm karg with a Metal function constant. The
top-k MoE kernel is compiled separately for the normalized and non-normalized
routing variants, removing the runtime branch.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : rename moe_weighted_reduction suffix to moe_reduce
Shortens the MoE weighted-reduction fusion identifiers, kernel, pipeline,
matcher, args struct, and test op name from moe_weighted_reduction to
moe_reduce.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : add MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion
Adds dense mat-vec activation fusion for sigmoid/silu and bias+softplus.
The mat-vec kernels apply the activation/bias epilogue via function
constants, avoiding the separate unary/add passes.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : revert MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion
The mat-vec activation fusion regressed decode throughput on Qwen3.6-35B-A3B
by ~8% (tg32 81.5 vs 88.5 t/s). The regression is caused by loss of
concurrency: the standalone unary kernels previously overlapped with other
mat-vec work, while fusing the activation into the mat-vec kernel serializes
it on the critical path.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : add SSM_CONV + UNARY (silu) fusion
The SSM_CONV kernels apply silu directly via a function constant, eliding
the separate unary pass. Regenerates the fusion baseline.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : address fusion review comments
- Fix declaration/table alignment
- Rename top-k MoE kargs fields to val_clamp / val_scale
- Move moe-reduce alloc-deps handling into a general fusion helper
- Remove the public moe-reduce matcher API
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : fix unused parameter in top-k MoE fusion check
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : guard SSM_CONV fusion lookup behind use_fusion
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : track all fused outputs in graph reorder
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : keep top-k MoE logits alive until fused output
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : refactor alloc deps to pattern-driven approach
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : check fused kernel destination in concurrency tracking
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* meta : forward graph_optimize to underlying backends
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use vector for fusion table
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* meta : keep graph_optimize unimplemented
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* parallel : fix non-deterministic prompt selection
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* parallel : support dummy models and add global logits run hash
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : sync cross-device copies with destination completion event
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : avoid const_cast in fusion alloc deps
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : skip fusions with aliased sources
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : hide fusion pattern definition
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use vector fusion op sequences
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : drop redundant struct keywords
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : add alloc deps comment separator
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : generalize fusion output memory ranges
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : rename fusion out_offsets to outs
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : avoid dst vector in memory range check
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : optimize fusion matching and multi-output handling
- use pointer arithmetic for fusion info count lookup
- avoid heap allocations in top-k MoE and MoE reduce pattern matchers
- use fusion outs for multi-output subgraph checks
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* Revert "parallel : support dummy models and add global logits run hash"
This reverts commit 57c7caf941c1b43c270fd5009c9f175063522e96.
* fusion : update MTL.csv
* metal : unroll constant loops in top-k MoE kernel
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use function constants for top-k MoE n_expert and top_k
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : rename fusion kargs to scale and clamp
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use function constants for moe_reduce and ssm_conv
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* fusion : update MTL.csv
Add support for the new DSV4 HC op variants used by qwen4exp:
- hc_pre with per-element sigmoid gate (gated variant)
- hc_post with identity mixing (comb == nullptr)
Assisted-by: pi:llama.cpp/Qwen3.8-27B
argsort_f32_i32_cuda_cub called the one-shot DeviceRadixSort::SortPairs
API with d_keys_in == d_keys_out (temp_keys, temp_keys). CUB's internal
double-buffer ping-pong requires distinct key buffers: with aliased
buffers the sort partially overwrites its own input mid-pass and emits a
corrupted permutation, surfacing as intermittent garbage indices (e.g.
backend top_k over a 248k-column vocab on Maxwell/CUDA 12.5/CCCL 2.x,
which then triggered out-of-bounds gathers in downstream get_rows).
Use a distinct keys-out buffer for all six call sites (plain and
segmented, ascending and descending, size-query and execute).
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Oliver Simons <osimons@nvidia.com>
Allow HMX flash-attention to run with head_dim not a multiple of 64
(e.g. SigLIP head_dim=72), by operating on DK/DV rounded up to 64 with
zero-filled tail lanes.
* ggml-cpu: add F16 input to the FWHT
The CPU FWHT accepts F32 input only. This change makes the source type a
template parameter. The CPU path now accepts F16 input and F32 input.
The CPU MUL_MAT reference now converts an F16 src1 to F32. It does this when
the caller sets the Hadamard hint.
No backend has an F16 FWHT kernel yet. The test cases come with the backend
changes that add one.
* ggml-cpu: assert the F16 FWHT input path, and use the bulk converter
Address review feedback.
The F16 branch writes plain floats into wdata, which is only correct when
vec_dot_type is F32. That invariant held because supports_op only accepts an
F16 src1 for the Hadamard hint with F32 src0 and dst, but nothing enforced it.
Assert it next to the existing src1 type check so widening supports_op cannot
silently break the write.
Replace the hand-rolled conversion loop with ggml_cpu_fp16_to_fp32.
* llama: read the SWA pattern as a period or a per-layer array
Add llama_model_base::load_swa_pattern(), which reads
sliding_window_pattern either as one flag per layer or as a period
expanded by set_swa_pattern(), and use it in every loader that reads
the key as a period.
These loaders silently ignored an array and applied their default
period, although the converters of olmo2, gemma3n and exaone4 write
arrays. The published GGUFs match the defaults, so their outputs do
not change. The loaders that already accepted both forms lose their
duplicated scalar-then-array block, and use their declared default
period when the key is absent.
* model-saver: write the SWA pattern and the MLA SWA geometry
Write sliding_window_pattern as one flag per layer, nextn layers
included, for every model using SWA. The array is never collapsed to
a scalar, since the loaders read a scalar as a period.
Also write the MLA key/value lengths and KV LoRA rank of the SWA
layers, required by dots3note.
This enables the saver for plamo3, gemma3, cohere2, cohere2moe,
olmo2, exaone-moe, afmoe, mimo2, spark2_5, muse-glimmer, mellum,
laguna, granite_swa, dots3note and maple, all passing the bit-exact
roundtrip of test-llama-archs.
* vulkan: add IQ3_S MMQ matmul kernels
* Make block_a_to_shmem do 2-byte loads (110 bytes is divisible by 2)
* Align the check, IQ3_S is also using K tile size
* vulkan: raise the hoisted row-id limit for mul_mat_id to 512 experts
The expert-count shader (count_experts.comp) sizes its shared arrays
with BLOCK_SIZE, which is 256. Because of that, row-id hoisting is
switched off for any model with more than 256 experts, and every
mul_mat_id workgroup has to rescan the whole ids tensor on its own.
Qwen3.8-Flash-Next has 512 experts and was quietly running on that
slow path.
This change sizes the arrays with a separate MAX_EXPERTS constant (512),
clears them in a loop instead of one entry per thread, and raises the
matching limit on the host side.
On Strix Halo at batch 2048 the expert matmuls drop from 12.5 to 9.5 ms
(iq3_s) and from 14.0 to 7.5 ms (iq4_nl) per op, and prompt processing
gets about 19 % faster at 8k tokens. test-backend-ops MUL_MAT_ID passes
(891/891) with new 512-expert test cases.
Assisted-by: Claude Fable 5.1
* vulkan: raise the hoisted row-id limit for mul_mat_id to 1024 experts
Follow-up to review feedback: 1024 matches LLAMA_MAX_EXPERTS instead of
stopping at 512. The three shared arrays in count_experts.comp grow to
3 * 1024 * 4 = 12 KiB, which fits the 16 KiB that Vulkan guarantees for
maxComputeSharedMemorySize.
Adds mul_mat_id test cases at 1024 experts alongside the existing 512
ones. test-backend-ops MUL_MAT_ID passes on Vulkan (RADV, Strix Halo,
Radeon 8060S): 889/889.
* Update to openvino-2026.4
* Update OV docs
* ggml-openvino : fix clangd and MSVC warnings
* fix int to ptr cast, more internal linkage enforcement, and avoiding duplicate switch case
---------
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
* ui: fix accidentally removed reasoning menu in single model mode on desktop
* ui: formatting task run to fix storybook test
* ui: mount the add menu reasoning submenu outside router mode only
The models selector already owns the reasoning submenu in router mode,
so the add menu only mounts it in single model mode. The first enabled
item of the add menu is now the reasoning submenu, the accessibility
story expects it.
---------
Co-authored-by: Ben Babik <work@benjaminbabik.com>
Co-authored-by: Pascal <admin@serveurperso.com>
* ci : add API/ABI check to make-release workflow [no ci]
This commit adds an API/ABI compatibility check to the make-release
workflow.
The motivation for this to allow us to detect any potential breaking
changes in API/ABI compatibility between releases and fail the the
release if there are any.
The workflow can be triggered manually as before and this check can be
skipped if needed as it does take some time which might be useful when
doing a dry-run and not specifically interested in the API/ABI check.
By default this will check the current release against the latest
release, but this can also be configured in the workflow, or in the
script run on the command line, to check a different tag.
* add check for minor version bumps [no ci]
This commit also changes the build type to be RelWithDebInfo so that the
reported information is more useful.
* gguf : align the data section relative to the GGUF start, not the file
gguf_init_from_file_ptr reads a GGUF from the current file position, but padded
the data section from file offset 0, so a GGUF embedded at an offset that is not
a multiple of the alignment loaded without error and returned wrong tensor data.
Also adds llama_adapter_lora_init_from_file_ptr, and disables mmap with a warning
when an embedded data section is not aligned, instead of asserting in ggml.
Assisted-by: Claude Opus 5
* llama : load lora from path through the FILE* variant
The test now checks that mmap is disabled only for an unaligned offset.
Assisted-by: Claude Fable 5.1
* Update ggml/src/gguf.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Update include/llama.h
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* llama : error on unaligned mmap of an embedded GGUF, drop test-load-file-ptr
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Both im2col.comp and im2col_3d.comp declare D_ptr without an explicit
buffer_reference_align, so glslang emits writes through it as Aligned
16. The shaders advance the pointer by D_SIZE, a per-variant define
set to 4 for float and 2 for float16_t, so most write addresses are
not 16-byte aligned. This triggers
VUID-RuntimeSpirv-PhysicalStorageBuffer64-06315 under GPU-AV.
Declaring buffer_reference_align = D_SIZE matches the alignment to the
actual write stride and takes validation hits from 20 to 0 for both
IM2COL and IM2COL_3D.
Fixes#28960
* model: calculate split states for attn_qkv from n_head * n_embd_head_k
required for gemma4 with --fuse-qkv, where n_embd is 5376 but Q is 8192.
* model: handle fused full attention layers for qwen35/qwen35moe
* model: add TODO: [TAG_SPLIT_QGATE_QWEN]
llama probes weight placement with a rope where all params are 0, so rejecting
n_dims == 0 or freq_base == 0 puts rope_freqs on the CPU. That splits the decode
graph at every full-attention layer (gemma-4-E2B: 5 splits instead of 2).
Assisted-by: Claude Opus 5
* model : add support for HrmTextForCausalLM (DFM Mimir 1B)
HRM-Text runs two transformer stacks (low, high) in an alternating cycle over the same token stream. The low-cycle state z_l starts from a learned [n_embd] tensor and is broadcast over positions.
- conversion: new writer for the fused gqkv projection (order gate,q,k,v) remapped to llama.cpp q/k/v plus a separate sigmoid gate tensor
- loader: block_count = lps * h_cycles * (l_cycles + 1) cache slots aliasing 2*lps physical blocks via struct copies
- graph: looped build with sigmoid-gated attention, SwiGLU FFN and parameterless RMS norms; learned embedding_scale applied in build_inp_embd
- saver: pointer-deduplicated layer loop (looped archs alias tensors)
- tests: hrm_text fixture (lps 1, h 2, l 3) in test-llama-archs
Limitations:
causal attention only - the upstream prefix-LM mode is not implemented (the prefix_lm GGUF key round-trips unused).
The KV cache holds one entry per pass: 128 layers for Mimir 1B, i.e. 4x a same-width 32-layer model - about 3072 MiB at ctx 4096 in F16 (halves with q8_0 KV + FA).
Every token runs all 128 block passes, so decode cost is roughly 4x a dense model of equal width (2.65 t/s BF16, 8-thread desktop CPU).
Verified against the HF reference: identical argmax at 334/334 positions across 20 prompts (BF16 GGUF vs FP32 golden).
q8_0 requant: 95.8% top-1, all remaining misses inside the HF top-5 (accumulated error over 128 sequential blocks).
AI usage disclosure: YES
Used GLM-5.3 for the majority of code AI-generated under my direction, all gates verified locally.
All in all I could say that I have written less than 20% of the code and most of the heavy lifting has been done by the model. As such, this should be considered experimental.
* Update conversion/hrm_text.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-arch.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : add gguf_writer methods for hrm_text metadata
replace raw add_uint32/add_bool calls with dedicated GGUFWriter methods, following the add_embedding_scale pattern
Assisted-by: GLM-5.3
* convert : map regular hrm_text tensors via tensor_mapping
delegate unfused checkpoints to the base tensor mapping; training-style attn. names are renamed to self_attn. so the patterns match
Assisted-by: GLM-5.3
* model : format hrm-text build_* calls as in other models
one argument group per line, matching sibling model files
Assisted-by: GLM-5.3
* llama : move hrm z_l_init table entries out of the nemotron group
place the name and tensor-info entries with the other global input tensors
Assisted-by: GLM-5.3
* convert : slim down hrm_text comments
Assisted-by: GLM-5.3
* convert : build hrm_text block tensor names from the {bid} template
The tensor map holds concrete per-block names, so format the template
with the computed layer index before handing it to super().
* llama : name hrm metadata keys in their own hrm. namespace
The four keys are arch-independent, unlike the arch-substituted
Keys.LLM entries, so group them under Keys.HRM (like Keys.Split) and
rename the llm_kv entries to LLM_KV_HRM_*. Only our own GGUFs carry
the old hrm_text.* keys; they are regenerated.
* Update src/llama-model-saver.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama : keep hrm metadata keys arch-substituted
Per review: the GGUF keys stay "{arch}.h_cycles" style, so the Python
members drop the LLM_KV_HRM_ prefix and keep arch templates; C++ keeps
the LLM_KV_HRM_* enums. GGUF output is unchanged - existing files and
HF uploads stay valid.
* Update gguf-py/gguf/constants.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-arch.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-arch.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : rename hrm writer methods to add_hrm_*
Generic names like add_h_cycles/add_prefix_lm are too broad on the
shared GGUFWriter; prefix them with hrm_ like the metadata keys.
* model : fix meta-split lookup for archs with aliased cache slots
Cache tensors of archs that alias physical blocks across looped slots
(hrm_text, nanbeige with num_loops > 1) can reference block indices
without weight tensor names. Take the output projection from the layer
array instead of asserting; all other lookups are unchanged.
* model : replicate hrm_text tensors on meta devices instead of splitting
The aliased cache slots rotate split states differently from their
physical weights, so the meta-split execution invariants (set_rows
requires the cache state to match the token indices) cannot hold for
any device count. Replicate all hrm_text tensors on every meta device
instead; single-device and non-meta paths are unchanged.
Assisted-by: Claude Sonnet
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
The `sizeof(int16_t)` branch in `permute_transpose_impl` calls
`rvv_transposed_s32_mn_to_nm` instead of `rvv_transposed_s16_mn_to_nm`.
This is a copy-paste bug from the `sizeof(int32_t)` branch above it.
The s32 function uses 32-bit segment load/stores (`vssseg8e32.v`) on 16-bit
data, reading 2x bytes per element and producing completely wrong
transposition results -- 14 out of 16 positions are corrupted for a 4x4
int16 matrix.
The correct function `rvv_transposed_s16_mn_to_nm` already exists (line 390)
and is used elsewhere in flash attention (line 1488).
The server caches the most recent compute graph per device so that
GRAPH_RECOMPUTE can re-execute it without resending tensor data. The
cached graph nodes hold direct pointers to backend buffers that were
live at graph_compute() time. If any of those buffers is later
released via FREE_BUFFER, the next GRAPH_RECOMPUTE re-executes the
cached graph through the dangling pointers (use-after-free).
The bug is reachable by an unauthenticated remote client. The
dangling pointers point into chunks an attacker can reshape via
subsequent ALLOC_BUFFER/SET_TENSOR commands, and the resulting
read/write through the cached graph is sufficient to leak libc
addresses and hijack the buffer iface vtable used by BUFFER_CLEAR,
yielding remote code execution.
Discard all cached graphs in free_buffer(). The existing null-check
in graph_recompute() then rejects the request and the client falls
back to GRAPH_COMPUTE on the next call.
No protocol or API change.
It's found the MoE ncols_opt tile heuristic needs to be broadened
to include the RDNA3.5 architecture.
The code change is implemented in ggml/src/ggml-cuda/mmq.cu
and just change the GGML_CUDA_CC_IS_RDNA3_0 to
GGML_CUDA_CC_IS_RDNA3 in the condition.
The dense dispatch logic remains unchanged.
The Test machine configuration we used is
AMD Radeon 8060S, gfx1151 (RDNA3.5), 20 CU, wave32
+ AMD Ryzen AI MAX+ 388, 8C/16T, 23.79 GB RAM
we complete the Correctness verification and performance evaluation as follows:
test-backend-ops test -b ROCm0 -o MUL_MAT -p type_a=<q4_K|q5_K|q4_0|q5_0>
test-backend-ops test -b ROCm0 -o MUL_MAT_ID -p type_a=<q4_K|q5_K|q4_0|q5_0>
all pass: MUL_MAT 64/64, 29/29, 48/48, 14/14;
MUL_MAT_ID 84/84, 3/3, 74/74, 3/3
Performance result on target machine:
LFM2.5-8B-A1B-UD-Q4_K_M (Q4_K MoE) +16.198% [+12.704, +19.799] 8/8
Qwen1.5-MoE-A2.7B-Q2_K (Q2_K MoE) +6.189% [ +5.245, +7.141] 8/8
pooled (16 pairs) +11.081% [ +7.972, +14.279] 16/16
Token generation (tg128) is unchanged on the Q4_K MoE model and +2.188%
[+0.905, +3.488] on the Q2_K one.
Use BN/2 as the default for BNover2 and as the disabled fallback for BNover4, and remove the enable gate from the MUL_MAT_ID BN/2 branch. The BN/4 branch remains gated by enable_smaller_matrices, while the p.N path is unchanged.
* test-backend-ops: reproduce MUL_MAT_ID NaN for activations beyond f16
The Metal mul_mm_id path narrows src1 to `half` for the simdgroup MMA
(`S1 = half` in every instantiation; ggml-metal.metal:10582 and :10595,
mirrored at :10643/:10654 in the tensor-ops path). f16 saturates at
65504, so a model whose activations exceed that produces inf, and
`simdgroup_multiply_accumulate` then turns the whole 8x8 accumulator
tile into NaN. The mul_mv_id path used below `ne21_mm_id_min` (32)
carries the same values in f32 and is correct, as is every CPU path.
This was untestable before: `init_mul_mat_id_tensors` initializes
uniform [-1, 1], so no existing case can drive an operand out of f16
range. `test_mul_mat_id` gains an `amax` parameter (default 1.0f,
preserving the historical init exactly) that scales only the f32
activations, leaving the quantized weights in their normal range.
Six cases: n=16 sits below the mul_mv_id -> mul_mm_id switch and is the
control that must stay green; n=32 and n=64 are above it and fail on
Metal today. Two shapes, because this is not model- or size-specific —
q4_K at 128 experts / 4 active / 4096x2048 mirrors a real model, and
q8_0 at 8 experts / 2 active / 512x256 shows the same failure at
minimal size.
Observed on Apple M2 Max, macOS, llama.cpp b10156:
MUL_MAT_ID(type_a=q8_0,...,n=32,k=256,amax=100000.000000):
[MUL_MAT_ID] NaN at index 0 (MTL0=nan CPU=583442.375000) FAIL
The real model behind this is Mistral Small 4 (arch mistral4, 128
experts / 4 active), one of whose layers reaches ~1e5 activations: on
Metal every prefill of >=32 tokens returns an entirely NaN vocabulary,
while <32 tokens is correct.
Note kernel_mul_mm (dense) has the identical conversion at :10273 and
:10286 and is expected to fail the same way; it is not covered here.
Found and written by Claude Opus 5 (via Claude Code).
* metal: fix NaN in mul_mm_id when activations exceed f16 range
kernel_mul_mm_id narrows src1 to `half` for the simdgroup MMA operands
(`S1 = half` in every instantiation). f16 saturates at 65504, so a model
whose activations exceed that produces inf on load, and
simdgroup_multiply_accumulate then propagates NaN across the whole 8x8
accumulator tile. The result is an entirely NaN output — not a precision
loss, a total loss. The mul_mv_id path taken below ne21_mm_id_min (32)
keeps the same values in f32 and is correct, as is every CPU path, so
the same model produces correct logits for short inputs and NaN for
long ones.
Fix: rescale src1 by a power of two so it fits, and undo the scale on
the f32 accumulator at the store. A two-stage reduction computes
max(|src1|) and writes the pair (1/scale, scale) into scratch chained
off the destination buffer, in the same style as the existing tpe/ids
id-mapping scratch. The matmul multiplies on load and on store.
This is exact, not approximate, for two reasons: the dot product is
linear, so one tensor-wide factor commutes through the accumulation;
and the factor is a power of two, so both multiplications are exact in
binary floating point. When max(|src1|) already fits — every model that
works today — the factor is exactly 1.0 and the output is bit-identical
to before. Accumulation was already f32 and is unchanged; only the
operand narrowing was ever the problem.
The reduction is two-stage (256 threadgroups into partials, then one
threadgroup folding them) specifically so it stays bandwidth-bound. A
single-threadgroup version was measured first and cost up to +451%
median on prefill — the scan serialized against an otherwise idle GPU.
It is also dispatched only on the mm path, so decode never pays for it.
Measured on Apple M2 Max, `test-backend-ops perf -o MUL_MAT_ID -b MTL0`,
99 cases, versus the same build without this change:
n=1/4/8 (mul_mv_id, decode) : -0.8% / -0.8% / -0.4% median (noise)
n=32 (mul_mm_id, prefill) : +1.73% median
n=64 : +1.30% median
n=128 : +1.80% median
n=256 : +3.98% median
n=512 : +3.74% median, +7.20% worst
overall : +1.14% median
Correctness, same machine:
- the six new test-backend-ops cases go from 4 FAIL / 2 OK to all OK,
with the n=16 controls (mul_mv_id path) unchanged;
- `test-backend-ops -b MTL0` full run: 0 failures, no regression;
- Mistral-Small-4-119B (arch mistral4, 128 experts / 4 active) now
generates correctly at the default n_ubatch of 512, in both
UD-IQ3_S and UD-Q4_K_XL quantizations. Before this, every prefill of
>= 32 tokens returned an all-NaN vocabulary and only n_ubatch <= 31
(forcing the mul_mv_id path) worked.
Likely fixes#25722 (mistral4 empty output on Metal above ~300 tokens,
FA on and off, generation degenerating to a single control token — the
signature of argmax over an all-NaN distribution). #20668 may be the
same defect attributed to a bad GGUF.
Note kernel_mul_mm (dense) has the identical narrowing at the
corresponding load sites and is expected to fail the same way; it is
left alone here to keep this change reviewable. Also possible, and left
for later: scaling per output column rather than per tensor, which
would preserve more precision when a single token is the hot one.
Found, diagnosed and fixed by Claude Opus 5 (via Claude Code).
* metal : make requested edits
- remove verbose comments
- explain rationale as requested
Generative AI disclosure: Claude made the edits as requested.
* metal : stack mul_mm_id map0 with amax_part
Implement @ggerganov suggestion to stack amax_part + map0. Mean 2.6% faster (worst -0.7%, best -4.1%). Win grows with batch size. Benchmarked on a hot M2 Max after reboot.
Generative AI disclosure:
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* cont : fix var scope
* cont : comment out tests temporarily
Comment out tess to not break CI temporarily
Assisted-by: Claude Fable 5.1
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* add self-hosted vulkan and webgpu to hf-jobs
* try t4-medium
* cont : adjust cpu backend threads
* try t4-small again
* restore cm jobs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* rpc : hash-cache only weights
ggml_backend_rpc_buffer_set_tensor and ggml_backend_rpc_set_tensor_async
hashed every transfer above HASH_THRESHOLD and let `rpc-server -c` serve it
from its file cache. The cache is meant for weights, but the activations
ggml_backend_sched copies between backends took the same path: with a
two-node split of Qwen3.8-Flash-Next every prefill ubatch above 10 MB was
hashed, written to the worker's cache directory (1.4 TB after a day) and
later served from there. Use the hash path only for tensors in buffers
marked GGML_BACKEND_BUFFER_USAGE_WEIGHTS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* rpc : save a cache entry only for the tensor that missed the hash check
With the client hashing weights only, the server still wrote every
SET_TENSOR above HASH_THRESHOLD to the cache directory, so the compute
data the scheduler sends kept filling the disk. Remember the hash of the
last SET_TENSOR_HASH that missed and save only the SET_TENSOR that
follows it with that hash - the weight the client is re-sending.
* rpc : signal the cache decision in the SET_TENSOR payload
Replace the server-side `pending_cache` state with a `cache_flag` byte
in the SET_TENSOR message: the client sets it when SET_TENSOR_HASH
reported a miss, the server saves a cache entry only when it is set.
Bump RPC_PROTO_MAJOR_VERSION since the wire format changes.
---------
Co-authored-by: Patrick Hoffmann <patrickhoffmann@MacBook-Pro-14-HOP.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* cuda: support row-contiguous SUM_ROWS
* organize the code and add GGML_OP_MEAN to support row-contiguous tensors using the same shared kernel, and add a test to MEAN permute/slice
* Keep original comments and add if/else branch
* exclude GPU/NPU failing POOL_2D case
* Fix pool case
* ggml-openvino: fix stateful decode for Gemma-4 per-layer-type head sizes
* ggml-openvino: fix MSVC narrowing error in permute
* ggml-openvino: classify sliding-window layers structurally on interleaved-SWA models
* ggml-openvino: add GGML_OPENVINO_REQUANT_KQUANT to select a 4-bit requant target
* ggml-openvino: add GGML_OPENVINO_SPILL_DIR to spill weight buffers to disk
* Stateful Performance: Added pass::KVStateSeqAxis to change KV layout
* ggml-openvino: fix stateful decode past the sliding-window size
Assisted-by: Claude Sonnet
* ggml-openvino: refuse stateful decode that cannot resume from the KV state
The stateful path seeds its KV state from ggml's cache when the decode position
is ahead of what the state holds. That only works when ggml's cache is a plain
prefix, where cell i holds position i. A sliding-window layer keeps just the last
n_swa positions and drops the rest, so past the window cell i no longer holds
position i and the seeded state is wrong.
Slicing the state to the decode position also had no bounds check, so a position
past the end surfaced as a bare ov::Exception from the ROI constructor
(llama_decode ret = -3, with no reason given at default verbosity).
Refuse both cases with a clear message instead, and refuse on the compile path
too, where a new model starts with an empty state and so can only serve a
sequence from its beginning. Reproducible with llama-bench -d, which restores a
saved sequence state rather than recomputing the depth prefill.
Assisted-by: Claude Opus 5
* ggml-openvino: use the per-layer KV head count for the stateful KV state
The stateful path reinterprets ggml's KV buffer [1, 1, seq, n_heads_kv * head_size]
as [1, seq, n_heads_kv, head_size]. The head size is already taken from the
tensor's own combined dim, because gemma-4 varies it per layer type, but the head
count still came from a model-level scalar that compute_llm_params() overwrites
per attention node, so it ended up holding whatever the last layer said.
gemma-4 varies the head count per layer too: 12B has 8 x 256 sliding layers and
1 x 512 full layers, 31B has 16 x 256 and 4 x 512. So 40 of 12B's 48 layers were
split as 1 x 2048 instead of 8 x 256, and attention read the state with the wrong
head split - both models decoded garbage on CPU and GPU. E2B is unaffected, its
head count is 1 everywhere.
Record the count per layer instead and look it up by the cache_k_l<N> leaf name.
Key it by layer, not by layer type: the sliding/full classification comes from
cache extents, which tie at a small -c, while the head count does not.
The stateful state trim now derives its sequence axis per state for the same
reason, since pass::KVStateSeqAxis matches per state on the head count.
Assisted-by: Claude Opus 5
* ggml-openvino: apply the KV state relayout to any KV head count
pass::KVStateSeqAxis was limited to states with a single KV head, where moving
the sequence axis from dim 1 to dim 2 is a pure metadata change. The limit was
also based on a measurement showing no gain for a multi-head model, but that was
taken at depth 0, which is the one depth where this change does nothing.
With several heads the pass does more than move metadata: it drops the reader
side transpose of the whole accumulated state, which the graph otherwise redoes
every token at a cost that grows with the context length, and replaces it with a
transpose of the single new row. Measured on GPU, tg128, alternating arms:
gemma-4-12B 6.27 -> 9.11 t/s at depth 8192 (stateless is 7.69, so stateful now
wins at depth instead of losing), Llama-3.2-1B 47.8 -> 59.6 t/s. Both are within
noise at depth 0, which is why the earlier check saw nothing.
The state refill needs the rows copied rather than reinterpreted now: ggml stores
[seq][n_heads_kv * head_size], and a relayout state with several heads is a
different element order. Without that, a refill would seed wrong data - it is
reachable today through llama-bench -d.
Assisted-by: Claude Opus 5
* ggml-openvino : support ggml_rope_set_offset and simplify op support gating
* add more cpy cases
* reject BF16 cpy on NPU
* Remove mul_mat_id fallback, gate large mul_mat_id only for mxfp4
* ggml-openvino: fuse the MoE expert block into MOECompressed on GPU
* ggml-openvino: skip GPU MUL_MAT_ID for unbound expert tensors
* ggml-openvino: requantize grouped 8-bit MoE experts on GPU
* Enable special strided CPY for conv state writeback
* openvino: support cacheless encoder models on NPU
Packed QKV views used by mmBERT were rejected by the ROPE support check. This split Q/K RoPE onto CPU, prevented cacheless attention detection, and sent fragmented encoder graphs through the decoder-oriented NPUW path.
Accept packed QKV RoPE views, detect cacheless attention from its mask, and run these models as a single full-sequence prefill without NPUW or a decode graph. Also provide static mask, output index, and mean-pooling shapes and inputs.
* openvino: optimize norm and RoPE translation
Replace the decomposed mean/variance normalization graph with an opset6 MVN operation. This preserves the GGML epsilon placement while allowing OpenVINO plugins to compile normalization as one operation with fewer intermediate tensors.
Cache RoPE sine and cosine outputs in the graph-wide tensor map. Build the cache key from all RoPE parameters and the optional frequency-factor input so compatible Q/K and layer nodes share one subgraph without mixing different RoPE configurations.
Expose NodeContext::put_shared() to publish translator-created outputs for graph-level reuse.
* ggml-openvino : simplify op translators and enable IMROPE/NEOX RoPE fusion
* remove unnecessary include and clean up PAD
* fix mulmat bug
* use ov::as_type_ptr instead of std::dynamic_pointer_cast
* ggml-openvino: fix mixed-dtype ADD/SWIGLU_CLAMP, gate unsupported ROPE/SOFTPLUS cases
- translate_add: upcast mismatched operand types (e.g. f16/f32 in fused
ADD_ADD) to f32, add, then cast once to the output type. opset1::Add
requires matching input types and downcasting first lost precision.
- translate_glu_swiglu_clamp: same fix, f16 Swish/Clamp rounding was
drifting past the test tolerance.
- supports_op: reject ROPE with ne[3] > 1 (multi-sequence) since the
cos/sin tables only cover one sequence, and SOFTPLUS on GPU since the
OpenVINO GPU kernel overflows to inf for large inputs (CPU is fine).
- ci/run.sh: serialize test-backend-ops on OpenVINO GPU; running two
workers concurrently crashes the GPU plugin (CL_OUT_OF_RESOURCES).
* openvino: share compiled models with per-context inference state; fix thread-safety
* ggml-openvino: gate MoE expert-sum ReduceSum shortcut past 8 experts
The ReduceSum shortcut for the MoE expert-plane-sum ADD chain drifts past
the 1e-7 test tolerance for >8 experts (f32 accumulation order vs CPU
reference), intermittently, like the existing Q4_K/Q5_K NMSE case.
Expose is_moe_expert_sum_add() so supports_op can gate on expert count
and fall back to CPU for just that reduction op.
* ggml-openvino: gate degenerate m=1,n=1 MUL_MAT on GPU
CI hit ERR=1.8e-3 (> 5e-4 tolerance) for a scalar-output f32 dot product
(m=1,n=1,k=2048); didn't reproduce locally in 8 tries, so likely an
internal fp16 accumulation path the GPU plugin picks for this tiny
shape. m=1 output dim doesn't occur in real model weights, so gate it.
* ggml-openvino: make SoftPlus decomposition opt-in native
Assisted-by: Codex
---------
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
* metal : add FA kernels for HSK=96, HSV=64 (MiniCPM3)
MiniCPM3 sets attention.key_length to 96 and does not set
attention.value_length, which defaults to n_embd / n_head = 64. Metal had no
(96, 64) instantiation, so -fa auto aborted on the missing
kernel_flash_attn_ext_vec_f16_dk96_dv64.
Instantiate the tile kernel at (96, 64) for every K/V type that already has
(96, 96), and the vec kernel for the NE=4 configurations. Of the NE values the
vec dispatch considers, only NE=4 works here, because NL = 32/NE has to divide
both DK/4 = 24 and DV/4 = 16.
* tests : avoid redundant FA vec slice coverage
This commit updates cmake to use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR for paths in function calls.
The motivation for this is that when using add_subdirectory,
CMAKE_SOURCE_DIR is fixed to the top-level projects source directory,
that is the caller of add_subdirectory and not the llama.cpp root
which means that common/common.h header will not be resolved.
Refs: https://github.com/ggml-org/llama.cpp/pull/28091#issuecomment-5636106377
When /tools returns 403 (server started without tools), the web UI
refetched the tool list before every chat message, since the guard
treated an empty tool list as "not yet fetched". Each retry returned
403 and could trip fail2ban.
Skip the refetch once the store flags the endpoint as disabled, and
detect that state via the response status code instead of string-
matching the error message. The tools panel keeps probing on open so
the UI recovers once the server is restarted with tools enabled.
Fixes#28299
* release : add ubuntu-cuda build job (12.8/13.3, x64+arm64)
* Add GCC 14 for CUDA arm64 builds in CI
* Eplicit bash
* Install git for CCCL fetch
* Install git before we clone/checkout
* Match CI names for WIndows
* Whitelist llama.cpp repo to git
* Use $GITHUB_WORKSPACE
* Also ship dependent libs on Ubuntu
Need NCCL additionally as it's pre-built available on Linux
* Avoid duplicate files in packaged cudart
* Copy NCCL license
* Install CURL to fetch NCCL license
* Update .github/workflows/release.yml
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Remove NCCL until licensing has been confirmed
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
This commit removes the precompiled headers that I added in Commit
3bcfeb700 ("cmake : add PCH and unity build to improve build times
(#28091)").
The motivation for this is that this looked good when developing this
but has caused multiple issues that I had taken into consideration and
we have decided to remove it and only keep the unity builds from the
above commit.
Refs: https://github.com/ggml-org/llama.cpp/pull/28882#issuecomment-5662272126
* tests : add README for updating the per-backend fusion baselines
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : trigger fusion on changes to test-llama-archs.cpp and src/models
the dummy models and their architectures drive the fusion baselines, so a
change to either can alter the per-fusion counters and should re-run the
fusion job.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* tests : merge the fusion build commands in the README
assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : require explicit permission before posting PR/issue comments
assisted-by: pi:llama.cpp/Qwen3.8-27B
* gguf-py: add Maple tensor constants
Add MODEL_ARCH.MAPLE, its "maple" name, and the tensor list for the
Maple 20B-A1B ternary MoE architecture: token embeddings, output,
attention with Q/K RMS norms, and per-expert FFN tensors.
* convert: add Maple HF->GGUF converter
Register MapleForCausalLM in the HF architecture map and add the
converter for the Maple 20B-A1B ternary MoE model: 24 layers, 256
experts with 8 active, sliding-window attention (SWA-512) interleaved
with global attention at a 3:1 ratio, partial rotary factor 0.5, and
per-expert weight stacking into merged 3D tensors.
* llama: add Maple architecture (20B-A1B ternary MoE)
Add the Maple 20B-A1B ternary MoE architecture: 24 layers, 256
experts with 8 active, sliding-window attention (SWA-512) interleaved
with global attention at a 3:1 ratio, and ternary TQ1_0/TQ2_0
quantization support.
- register LLM_ARCH_MAPLE between MAMBA2 and JAMBA
- implement llama_model_maple: Q/K RMS norms after projection (GEMMA4
style), rope applied only on SWA layers (nope_on_global_attention),
ISWA KV cache, and MoE FFN with swiglu gate clamp at +7 (DEEPSEEK4
style)
- mark MAPLE as unsupported by the model saver (roundtrip skipped)
* tests: mark Maple as MoE-mandatory
Maple is always-MoE: the model throws when n_expert == 0, so the
test harness must only run the MoE config for LLM_ARCH_MAPLE.
* maple: apply review feedback (n_ff_exp_arr, get_arr, rope params)
- load_arch_hparams: use n_ff_exp_arr + n_ff_exp() accessor (upstream
changed these from a scalar member during the rebase)
- sliding_window_pattern: get_arr, the pattern is mandatory for this arch
- partial_rotary_factor: read only from rope_parameters (base.py mirrors
the top-level key automatically)
- document why TOKEN_EMBD/OUTPUT are forced to F16 (they are the two
dense tensors in Maple, and the reference GGUFs ship them as F16)
- add @ModelBase.example("deepgrove/maple-preview")
* tests: add Maple to the SWA pattern array list
get_arr for maple.attention.sliding_window_pattern requires an array, but
the harness only emitted a per-layer array for the arches in its list, so
test-llama-archs -a maple failed to load the model.
Assisted-by: DeepSeek Harness
* maple: move swiglu_clamp_exp to the converter
The loader prefilled 7.0 and read the key optionally. The converter now
writes it and the loader reads it as required, because llama-graph.cpp
skips the clamp when the limit is 0 and an optional read would silently
run unclamped. The test harness provides the key for the same reason.
Also drops tensor_force_quant: base.py already forces FFN_GATE_INP to F32
and TOKEN_EMBD/OUTPUT to F16 for ternary file types.
Assisted-by: DeepSeek Harness
* convert: fix the LazyBase func signature in the Maple converter
ty flagged the stack() closure: it takes no argument, while LazyBase is
annotated with func: Callable[[Any], Any]. Pass the tensor list through
args instead of closing over it, the same way kimi_k3 does, so the
callable shape matches.
Assisted-by: DeepSeek Harness
* sycl: GPU-resident TOP_K for large k, parallelised over the device
The SYCL backend refused GGML_OP_TOP_K above k = 32 and let it fall back to
the CPU, a backend round-trip per call. The limit was not conservatism: the
scan-merge kernels keep (split_block + 1) * k candidate (value, index) pairs
in SLM, so at k = 128 a work-group already needs 132 KB and cannot launch.
qwen4exp's sparse-attention indexer asks for k = 2048 in 12 layers on every
token, so this fired at every context length.
Add a radix select for large k. The k-th largest is found by four
most-significant-first passes over an order-preserving unsigned key: histogram
the digit over the candidate set, walk the buckets from the top, and recurse
into the one where the running count reaches what is still needed. SLM holds
the histogram rather than candidates, so the footprint is independent of k.
A final pass emits every column beating the pivot plus exactly as many
pivot-equal columns as are still missing, so duplicate keys still yield
exactly k distinct indices. Output order is not required and is not paid for:
ggml-cpu/ops.cpp swaps its first two outputs to say so.
The key folds -0.0 onto +0.0 so its equivalence classes match the reference
comparator, under which the two tie. NaN has no defined order in the reference
(its comparator is not a strict weak order there); here +NaN keys above +inf
and -NaN below -inf, which at least makes the result deterministic.
One work-group per row leaves the device idle whenever a graph has fewer rows
than it has cores, which at batch size 1 means one work-group full stop:
qwen4exp tops-k a tensor of shape [n_kv, n_tokens/n_stream, n_stream], so
token generation gives nrows == 1, and the backend sampler reshapes logits to
a single row as well. Measured, ne=[200000,1] and ne=[200000,16] cost 358.0 us
and 363.4 us -- sixteen rows for 1.5% more wall-clock.
So also spread a row over several groups when there are too few rows to cover
the device. Per-pass state moves to global memory and each digit pass becomes
its own launch, since a work-group barrier can no longer span the row. Groups
accumulate in SLM and contribute 256 global atomics each, keeping global
traffic per-group rather than per-element, and the last group of a row -- the
one whose fetch_add returns G-1 -- performs that pass's scan, holding the
launch count at one per digit plus one emit. The group count comes from the
device and is floor-divided by nrows, so a row count that already covers the
device is left whole and pays nothing. Below 64K columns the single-group
kernel finishes inside the cost of the extra launches and stays in charge.
Reading the row's prefix/mask/need through a device-scope atomic_ref costs
more than the sweep it guards: those loads are uncached, so passes 2-4 ran at
49 us against 12 us for pass 1. One lane reads them into SLM and the group
takes them from there -- 208 us -> 44.6 us at ne=[131072,1], k=2048.
The block size now takes the device's max_work_group_size instead of a cap of
512. The cap was never a floor, so a device reporting 512 is unaffected; one
allowing 1024 was being given half its width.
Finally, put the scan-merge gate where the two paths actually cross. That
kernel's cost climbs with k while the radix select's does not; measured over
widths from 2 to 200K columns and row counts from 1 to 8192, radix is ahead
everywhere from k = 8 up and behind at k <= 2, where scan-merge's smaller
fixed cost wins. The short-row corner (ncols=2, nrows=65536, as in bailingmoe2
group selection) is exactly where radix loses at low k, and the gate keeps it
on scan-merge.
Op-level against the CPU-fallback path this replaces, and against the
single-group radix select for the split: 4.98x at ne=[131072,1] k=2048,
6.65x at ne=[151936,1] k=40, 13.35x at k=20, 118x at ne=[65000,16] k=32.
No measured shape regressed. End to end on 3x Arc Pro B60 with
Qwen3.8-Flash-Next UD-IQ4_XS, llama-bench tg64, the parallelisation is worth
5.91 -> 6.05 t/s at d=131072 and a wash at shallower depths. Perplexity over
wikitext-2 is unchanged within noise at both 512 and 81920 context.
test-backend-ops: 525/525 TOP_K (previously every k > 32 case was refused),
880/880 MUL_MAT_ID. Perf coverage added for k > 32 at large widths and for the
short-row corner, neither of which was exercised before.
* move topk-select to topk-radix.{cpp|hpp}
---------
Co-authored-by: cwriter <cwriter@localhost>
Disable the ggml-cpu precompiled header and remove the
std::hardware_destructive_interference_size branch from CACHE_LINE_SIZE.
The PCH force-includes ggml-impl.h before ops.h, which pulls in <new>
via <array>/<vector> and defines __cpp_lib_hardware_interference_size.
This makes the C++ kernels use CACHE_LINE_SIZE = 256 (hardware
destructive interference size) while the C work-buffer sizing code in
ggml-cpu.c always uses the fallback 64. The mismatch undersizes the
rope work buffer by (CACHE_LINE_SIZE/4 - 16) * n_threads * 4 bytes,
causing a heap-buffer-overflow that corrupts the heap and later crashes
in ggml_compute_forward_rope_flt.
Disabling the ggml-cpu PCH restores the natural include order so
ops.h is processed before <new>, keeping CACHE_LINE_SIZE consistent.
Removing the std::hardware_destructive_interference_size branch makes
the value deterministic and include-order independent.
ref: https://github.com/ggml-org/llama.cpp/issues/28858
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
The workflow's push/pull_request path filters did not include the
ci/run.sh script that all of its jobs execute, so changes to it never
re-triggered the self-hosted CI.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
This commit moves the llama_n_rs_seq function call to before the
llama_decode call and returns directly if the check is true, removing
the setting of res and the goto statement.
The motivation for this change is to avoid the llama_decode call if it
is not needed.
* ggml-cuda: fallback to F32 on device without BF16 hardware acceleration: (Nvidia >= AMPERE, AMD >= RDNA3 or = CDNA)
* apply logic to NVIDIA as well
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
1) Combine two consecutive lookups (find + insert) into a single insert-attempt/lookup routine so that we don't per
form two O(log(n)) lookup operations in a row anymore -- we only need to do it once and then see if the insert succeeded.
2) Instead of copying every potential stack (expensive) and then moving it (cheap) to new_stacks when it's a final output state, we switch the order so that we move every potential stack (cheap), and then only copy it (expensive) to new stacks when it's a final output state. There are a LOT of intermediate states that get generated, and unless they become final output states, then all of these expensive intermediate copies are wasted.
Before: lookup -> lookup/insert + copy -> optional move to output
New: lookup/insert + move -> optional copy to output
The NextN/MTP tail loop derives the expert FFN size as n_ff/n_expert_used
when expert_feed_forward_length gives nothing for the layer. Both values come
from per-layer arrays that legitimately hold 0 on layers that are not MoE, so
a checkpoint whose predict layers hold 0 in both divides by zero and dies with
SIGFPE at load time, with no error message. Report the malformed metadata
instead.
Corrects a typo in `tests/test-quant-type-selection` for the
Nvidia Nemotron 3 Nano 30B A3B model, which was referred to as
*nvidia-nemotron-nano-3-30b-a3b*.
The error made the test skip that test case, rather than failing
the test.
[no release]
* fix for unsupport zes API
* optimize the code
* adjust the log level
* rm unused head files
* Update docs/backend/SYCL.md
Co-authored-by: Titaniumtown <titaniumtown@proton.me>
* fix the error to detect level zero SDK/dev package, stop build after detect the error
* update the message
* fix the build error when missed to install level zero dev package
* rm GGML_SYCL_DEV_DEBUG, mv read env vars in all entry functions
---------
Co-authored-by: Neo Zhang Jianyu <jianyu.zhang@intel.com>
Co-authored-by: Titaniumtown <titaniumtown@proton.me>
Co-authored-by: Neo Zhang <NA>
Move the EditorConfig Checker and Code Style Checker workflows from the
`[self-hosted, fast]` runners to `ubuntu-slim`, which is an established
runner label in the repo.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
- Clamp the -j parallelism to min(nproc, 2) so a single-core runner
uses -j 1 and multi-core runners use at most -j 2, instead of
unconditionally using $(nproc).
- Add a 3600s timeout to both test-backend-ops runs (the high-perf CPU
path and the default path) so a hung test cannot stall CI indefinitely.
- Note a TODO to reduce the timeout to 1800s in the future.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
There is a driver bug where two queues on the same VkDevice simultaneously
submitting can break some internal synchronization. Until it's fixed, add a
mutex around queuesubmit.
Clang stores the modification time of the precompiled header sources
inside the header and refuses the header when they differ. A cached
header restored from another checkout carries the timestamps of that
checkout, so the build fails. The option covers the compilers ccache
treats as MSVC while they are clang underneath, clang-cl and the Intel
LLVM drivers.
The child writes its state commands on stdout while the logger writes
on stderr, and both share a single pipe. The logger emits the trailing
color reset after the newline of a debug, warn or error entry, so that
escape sequence has no newline of its own and the router reads it glued
in front of the next command. The line prefix check then fails and the
command is forwarded as a log line instead of being handled, which
leaves a finished download stuck in the downloading state.
Writing the command with a leading newline closes the pending line so
it always starts at a line boundary.
Walk the binding offset back until the distance to the tensor is a
whole number of blocks, so block quantized views get a valid element
offset in the shader.
* hex-row-split: add support for multi-device row spliting
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* hex-mdev: add work splitting to fused kernels
* hex-mdev: use mdev_ prefix for all multi-device state
* hex-mdev: make device configuration more expressive to support device groups
* hex-mdev: fix mdev session init
* hex-mdev: fused nx (2x,3x) matmuls must update row counts for each w/o
* hex-mdev: fix MUL_MAT work partitioning bugs introduced by mdev
* hex-cont: fix crashes with new tests due to wrong striding
* hex-mdev: move fences after l2flushes
* hex-cont: fix work splitting for mnpu -- align chunks to cachelines
* hex-mdev: fix CPY tests with multi-dev
* hex-mmid: fix work partitioning with mnpu
* hex-mm: fix test failures with mdev
* hex-binary: fix work partitioning for mdev
* hex-argsort: fix mdev partitioning
* hex-mdev: fix work partitioning and general updates for all simple ops
* hex-fa: fix mdev work splitting issues
* hex-mdev: fixing more failing ops test
* hex-mdev: update the rest of the ops
* hex-mdev: refactor all mdev splitting logic to be contained within if (mdev_count > 1) {...}
* hex-mdev: fix macros
* hex-mdev: simplify session flush logic
* hex-sync: fix recursion in session flush
* hex-mdev: factor out fence buffer and allocator
* hex-fence: make fence allocation more robust with reserved slots for mdev
* hex-mdev: keep all mdev state in htp_mdev_group
* hex-mdev: further cleanup mdev group handling at the host
* hex-mdev: update group idx in the opbatch before serializing
* hex-batch: remove separate op_pending and use batch_req/rsp_seq
* hex-async: workaround another missing tensor_init in ggml-meta
* hex-fence: cleanup and robustify fences and error handling in multi-device scenarios
* hex-ar: improve ALLREDUCE error handling
* hex-async: robust error handling for op_cpy_fence
* hex-async: use seq0 from allreduce context to allocate fence_seq
* hex-mdev: fix remaining issues with fence and barrier clearing in CPY_FENCE
* hex-misc: realign macros and fix misplaces trace events
* hex-misc: align macros
* hex-mdev: fix unclone buffer re-entrancy
* hex-glu: fix mdev partitioning logic
* hex-mdev: make buffer uncloning/cleanup work with tensor-split scenarios
* hex-mdev: tighten up the can_split check in act-ops
* hex-mdev: factor out common bits of the partitioning logic
* hex-mm: minor realignment of the macros
* hex-bufs: fix incorrectly placed assert for MAX_BUFS
* hex-pad: tighten up gating checks for PAD
* hex-kparams: make sure all kernels properly use kparams->n_threads
* hex-docs: update user and developer docs with new features and detailed guide for ops development
* hex-scripts: update run script to properly parse dev groups
* hex-misc: formatting
* hex-sess: minor cleanup for session init
* hex-ar: fix vtcm size calc in allreduce kparams
* hex-scripts: fix flake8 warnings
* hex-rope: update ROPE to support mdev work split
* hex-ops: remove redunant checks and minor reformat
* hex-dev-guide: update dev-guide to avoid redundant null checks
* hex-async: improve event_wait, event_sync and fence implementations
* hex-async: remove synchronous flush from event_sync
* hex-async: symplify fence recovery protocol and make sync more robust
* hex-async: futher simplify error recovery for fences
* hex-err: return status instead of just -1
* hex-async: print all seq nums in hex
* hex-async: make sure fences flush dirty ranges
* hex-async: add dirty ranges merging to reduce fence flushes
* hex-async: properly sync before freeing the event
* hex-async: make sure fence owner session is not overriden
* hex-async: more fence write order more robust
* hex-async: make sure not to fuse ALLREDUCE+ADD if their dsts overlap
* hex-fusion: cleanup redundant checks
---------
Co-authored-by: Alexander Lu <alexlu@qti.qualcomm.com>
* ggml-webgpu: Update to a recent version of Dawn
* No module scanning
* Accept review suggestion to update comment
Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>
---------
Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>
* server: refactor subproc handling
* fix Windows build
* download: keep concurrent downloads of one blob apart
Every process writes the same path + .downloadInProgress, so a second
download of the same blob finds that file, takes it for its own partial
transfer and asks for the bytes after it, which produces a corrupt
result. The in-progress file now carries the pid of the process writing
it.
std::rename also replaces an existing destination on POSIX but fails on
Windows, so a download whose blob appeared in the meantime is dropped
after every retry and an etag rewrite silently keeps the old value.
std::filesystem::rename has the POSIX behaviour everywhere, and the
error now carries the reason reported by the system.
* Revert "download: keep concurrent downloads of one blob apart"
This reverts commit 917b83f149c625527f872fb2cf41289358fa5371.
* tests: serialize the router tests that download the same model
Parallel workers share one cache, so the two tests fetch the same blob
into the same in-progress file and race to rename it. They now take a
file lock around the download, like the session fixture does for the
preset models.
* Revert "tests: serialize the router tests that download the same model"
This reverts commit c368a4a98c677938ca87002edb6186ff2c02fd83.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ci : run test-backend-ops as a dedicated gg test
Run test-backend-ops as a separate gg test in ci/run.sh so it is executed outside ctest. With GG_BUILD_HIGH_PERF it keeps the existing CPU-only invocation (-b CPU); otherwise it runs all available backends without a backend filter.
Remove the dedicated backend-ops workflow and keep test-backend-ops as a built target that is not registered with ctest to avoid duplicate runs.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* ci : run test-backend-ops earlier and enable high-perf on kleidiai
Move the test-backend-ops gg test before test-llama-archs.
Enable GG_BUILD_HIGH_PERF and LLAMA_ARG_THREADS on the Graviton4 KleidiAI job and use the standard self-hosted results/mnt paths.
Add TODO markers for decoupling tests from libllama.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* ci : run test-backend-ops in parallel
Pass -j $(nproc) to test-backend-ops in both high-perf and all-backend modes.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* ci : disable parallel tests for ROCm
* cont : disable parallel tests with MoltenVK
* Test for nrc=2 as well | i8mm kernels
* Trigger only on supported HW
* Remove trailing whitespace
* Address review comment
* test: properly prepare nrc=2 inputs with independent data per row
* tests : make nrc=2 dot product inputs distinct
Assisted-by: Kiro
* tests : use non-trivial strides in nrc=2 dot product test
* tests : fail nrc=2 dot product test on non-finite errors
The expected success table holds when the four requests enter the shared
pool together. On a loaded runner they are admitted tens of milliseconds
apart, the slot lifetimes overlap differently and the pool overflows
while a short request is still resident. The decode failure aborts every
slot, so a request the table marks as successful comes back with the
context error instead of its generation.
Such a request now passes on that error too, while any other status, a
different error or a truncated generation still fails the test.
kernel_mul_mm_id splits its NR1 = 32 token tile into two 16-row halves and skips
the upper half when the expert did not fill it, on both the tensor and simdgroup
paths. The tB extents are corrected to (NK, NR1H) for the [NR1][NK] row-major tile.
The B tile is staged unconditionally, as on master: rows past nr1 restage a clamped
duplicate of a valid row, lie in the output-row dimension so they never contribute
to a valid row, and are dropped by the final store loop.
test-backend-ops: re-draw the expert ids between perf iterations of test_mul_mat_id
so MoE perf numbers are not warm-cache, and add token-tile boundary coverage using
n_used == n_mats, which routes every token to every expert so each expert receives
exactly n rows; n = 32, 33, 47, 48, 49 reach mul_mm_id and leave a last tile of 32,
1, 15, 16 and 17 rows.
* scripts : add initial profiling script (wip)
* src : add precompile headers (PCH) for models.h
* common : add common.h as PCH
* ggml : add PCH for ggml-impl.h
* mtmd : use PCH for models.h
* scripts : add script to build with Server/Tools/Tests
* server : add PCH for common.h
* docs: add profiling progress notes (wip)
* ggml : add exclude for GCC + SVE on ARM
Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33393906061/job/99493756214?pr=28091
* ggml : attempt to fix use of std::hardware_destructive_inference_size
Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33396221677/job/99501265689?pr=28091
* squash! ggml : attempt to fix use of std::hardware_destructive_inference_size
Add a version check for GCC 12 to conditionally apply the `-Winterference-size`
pragma.
* editorconfig : exclude profiling reports dir
This directory will not be included in the merge later and this commit
can be ignore at that point. Just fixing to keep CI happy.
* ggml : skip PCH for gcc on non-x86 architectures
* tests : add PCH for peg-parser/tests.h
There are 7 peg-parser tests that can share one PCH instead of then each
parsing the full tests.h.
* common : add PCH for chat.h
* docs : update linux build profiling full results
Just updating after a number of PCH additions. These are not exact
figures and will vary a bit from run to run, but they give a general idea
of the performance impact of PCH.
* cmake : introduce unity build for models
This commit introduces a unity build for the models to improve
compilation time.
The improvements were roughly the following:
```console
+------------------------+-----+------------+------------+------------+
| Build | TUs | Frontend | Backend | Total |
+------------------------+-----+------------+------------+------------+
| Full, master | 396 | 811.0 s | 692.2 s | 1,503.2 s |
| Full, with PCH | 405 | 380.0 s | 664.7 s | 1,044.7 s |
| Full, with PCH + UB | 264 | 357.7 s | 635.7 s | 993.4 s |
+------------------------+-----+------------+------------+------------+
TU = Translation Unit.
Full = includes Server, Tools, and Tests.
PCH = precompiled headers.
UB = unity build for models.
```
* docs : update linux profiling table with unitiy build results
* docs : update mac profiling results to include unity build [no ci]
* docs: remove profiling reports
* scripts : merge build profile scripts into one script
I was lazy before and just copied the first script to enable Tests,
Server, and Tools. This now merges them into a single script.
* Revert "editorconfig : exclude profiling reports dir" [no ci]
This reverts commit 2922a12118a0730d2f7632bcba265b44a0856c59.
* src : rename ggml_view_2d_slice to gemma3n_view_2d_slice
This is to be consistent with the rename in gemma4.cpp which was
required to avoid a name clash.
* cmake : add build profile script for windows [no ci]
This commit adds a port of the scripts/build-profile.sh script to
windows powershell.
This was developed on Windows on ARM but should work on X64 as well but
needs to be tested there as well.
* metal : rework fusion patterns into a single table
All fusable op patterns for the Metal backend are now declared once in a
fusion table (ggml-metal-fuse.cpp) and consumed by both the graph optimizer
(ggml_metal_fuse_max, packing) and the op encoders (ggml_metal_fuse_next,
compute). The two phases share the same pattern table plus ggml_can_fuse_subgraph_ext
for the structural checks, and differ only in the mode used for the pattern
check (STRUCTURAL at optimize time, since tensors are not allocated yet, and
FULL at compute time, including Metal buffer placement). This also protects the
snake activation (MUL + SIN + SQR + MUL + ADD) from being reordered during graph
optimization, which was previously unprotected.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* metal : fix absolute output indices in fusion patterns
ggml_can_fuse_subgraph_ext expects the outputs array to contain absolute graph
node indices (it indexes cgraph->nodes[outputs[i]]), but the fusion table query
was passing a relative index (n_ops - 1). As a result the last node of every
pattern was not recognized as an output and was subjected to the elidable
use-count check, which failed for essentially all fusions. This silently
disabled the norm/MUL fusion and caused a ~5% token-generation regression.
Pass the absolute graph index of the last node instead.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* metal : fuse gated_delta_net with cache cpy
Add GGML_METAL_FUSE_GDN_CACHE to the fusion table: when the gated_delta_net
kernel is followed by a cpy that scatters its recurrent state snapshots into
the KV cache, the kernel writes the snapshots straight into the cache buffer
and the trailing cpy is elided.
The gdn output has other consumers (the attn scores view), so unlike the
elision-chain patterns this is not a simple chain: a 'raw' flag on the fusion
pattern skips the generic chain/shape and ggml_can_fuse_subgraph_ext checks,
making the pattern-specific check callback the sole validator. Packing
(ggml_metal_fuse_max) now matches on the same view-transparent node sequence
that the compute phase uses, so the gdn + cache cpy group is packed along with
any intermediate views and stays adjacent through the reorder.
The fused cpy is a view consumer of the gdn (it writes the cache directly),
so its mem-range is skipped in the encoder; the skip is restricted to CPY
nodes consuming the previous fused node through a view so other fusions are
unaffected.
Add test_gated_delta_net_cache_fusion and register 5 cases.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* metal : drop is_view_consumer mem-range skip
The is_view_consumer skip was carried over from the upstream gated_delta_net
cache-fusion draft, but it is not needed: keeping the elided cpy's mem-range in
the concurrency tracker only ever adds a (conservative) memory barrier at the
fusion point. It can never remove a barrier, so it cannot introduce a race. The
worst case is one spurious barrier per gdn+cache-cpy fusion, which is within
run-to-run noise on Qwen3.5-0.8B Q8_0.
Dropping the check keeps the mem-range loop uniform for all fused groups.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* metal : rename gated_delta_net fused state output args
Rename the fused cache-write kernel argument to match the rest of the kargs:
state_out_stride -> nb_out (and widen it to uint64_t), and the local buffer id
bid_state_out -> bid_out.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* metal : rename raw fusion flag to unsafe
raw did not convey that the flag opts a fusion pattern out of the generic
elision-chain safety net (ggml_can_fuse_subgraph_ext + chain/shape checks).
rename it to 'unsafe' to make explicit that the pattern's check callback is the
sole validator and must re-establish the safety guarantees itself.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* metal : tidy fusion pattern checks and table
- const-correct ggml_metal_fuse_outputs buffer
- annotate unused check-callback parameters
- drop a redundant size_t cast
- align the ops/table initializers and add blank-line separation
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* metal : add generic fusion stats via ad-hoc proc-address API
Add a device-owned fusion context that lets a test tool count how many
times each fusion pattern fires and toggle fusion. It is exposed through
the ad-hoc ggml_backend_reg_get_proc_address mechanism with generic names
so the testing tool is backend-agnostic:
- ggml_backend_fusion_stats_init: start collecting fusion stats; when a
context is created afterwards it registers the labels/counters and
encodes single-threaded (n_cb == 0) so the counters are race-free
- ggml_backend_fusion_stats_reset / _get_stats / _set_enabled
The context lives on the metal device (not on the last backend context),
so counters accumulate across contexts and reads are always consistent.
The enable/disable toggle is initialized from GGML_METAL_FUSION_DISABLE
and can be overridden by the test through set_enabled. Labels are
synthesized from the fuse table via ggml_metal_fuse_label (e.g.
"GATED_DELTA_NET+CPY").
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : add fusion count regression test with per-backend baseline
test-fusion runs every dummy model generated by test-llama-archs on a
single backend (single-threaded encoding, n_cb == 0) with fusion enabled
and disabled, and for each mode (prefill / decode) reports the per-fusion
counters and the NMSE between the fused and unfused logits, plus the NMSE
against a CPU reference.
A fusion pattern that silently stops matching (or fires when it should
not) is caught as a regression by comparing the counters against a
committed per-backend TSV baseline:
- --record writes the golden baseline, --check (default) validates it
- the unfused run doubles as a control: its counters must be all-zero
- NMSE is skipped when it is NaN or the arch is already broken on the
device (e.g. plamo2 on Metal), so the count check is the hard gate
- baseline counts depend only on graph structure, not weights (verified
stable across weight seeds)
- the fusion stats API is resolved through the ad-hoc get_proc_address
mechanism with generic names; a backend that does not export it makes
the test fail with an error
The committed MTL0.tsv baseline covers 110 dummy archs (298 rows).
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : rename fusion api helpers to match stats_init signature
Align the test with the ad-hoc fusion stats API: fusion_stats_init no
longer takes an enable bool (stats are turned on by calling it), so the
proc-address wrappers and typedefs are renamed to the api_* convention.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : rename backend to device in fusion test CLI
The fusion test operates on a compute device (e.g. MTL0), not a backend,
so rename the --backend argument to --device and the backend_name
variable to device_name. Keep "backend" where it refers to the ggml
backend interface (the ad-hoc proc-address mechanism).
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : add --model and --help to fusion test
--model FILE runs the fusion regression test over a single model file
instead of enumerating a --models DIR. --models and --model are mutually
exclusive. Also add a --help/-h option that prints the usage.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : use backend base name for fusion baseline output
The fusion test is invoked with a specific device name (e.g. MTL0), but
its output - the recorded baseline and the header it writes - should be
named after the backend base name (e.g. MTL, via ggml_backend_reg_name),
since the counters depend on the backend, not on the specific device
index. Rename the committed baseline to MTL.tsv.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : run fusion test from ci instead of ctest
The fusion test needs Metal and generates a lot of dummy models, so it
does not belong in the generic ctest suite. Move it to ci/run.sh as
gg_run_test_fusion, gated on GG_BUILD_METAL like
gg_run_test_llama_archs_tensor_split: it generates the dummy models with
test-llama-archs -o and then validates the fusion counts against the
committed baseline. test-fusion.cpp is still built (llama_build) but no
longer registered as a ctest.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : align fusion baseline TSV columns
Pad the TSV fields to fixed widths so the columns line up regardless of
the variable arch and fusion-label lengths, and trim each field on parse
so the padded file is still accepted. Regenerate the committed MTL.tsv
baseline in the padded format (data unchanged, verified identical modulo
padding).
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : widen label column and align fusion TSV header
Give the label column more room (28 chars) and fix the column header
widths so they match the data rows (moe/mode/label), keeping the header
aligned with the values. Regenerate the MTL.tsv baseline in the new
format (data unchanged).
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : switch fusion baseline from TSV to CSV
Use comma-separated values like the rest of the project, keeping the
padded, aligned columns. Split on ',' and trim on parse. Rename the
committed baseline to MTL.csv (data unchanged, verified identical modulo
padding/separator). Update the ci/run.sh check path accordingly.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* cont : rebase + update MTL stats
* tests : avoid graph reallocations for some archs
* metal : tidy fusion debugging context and op init
- simplify the shared fusion debugging context comments
- shorten the ggml_metal_fusion struct comment
- align the ggml_metal_fuse struct fields and comments
- move the fusion parameter of ggml_metal_op_init right after dev
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : dedup fusion baseline into any mode
prefill and decode always produce the same per-graph fusion count, so
store a single row per label with mode = "any" and the per-graph count
instead of two rows. this halves the baseline size and keeps the check
stable.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* ci : move fusion model generation to a separate step
the dummy models generated by test-llama-archs are reused by other tests,
so generate them once in their own step instead of inside test_fusion.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : bump nmse thold
* models : fix plamo2 graph
* tests : remove "skip" logic from test-fusion
* tests : set qwen3tts dummy vocab to codec head size
the dummy qwen3tts model used a vocab of 4096 while the codec head is
3072, so the graph padded the output with -inf which made the NMSE in
test-fusion produce NaN. use the exact codec head size instead so the
padding is not generated at all.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* tests : regen fusion baseline
reflect the plamo2 graph fix, which changed its fusion pattern split
(RMS_NORM+MUL 11->10, RMS_NORM+MUL+ADD 3->4; same total).
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* ci : skip dummy model generation on OpenVINO
test-llama-archs does not build on the OpenVINO platform, so do not try
to generate the dummy models there.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* cont : minor
* tests : enable test-llama-archs on windows
* cont : disable on windows + workaround
* metal : naming nits
* test-fusion : add instructions to update baseline
* context : fix Kimi-K3 graph reserve
* fusion : update MTL
* cont : fix naming
* metal : rework fusion info storage
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : align fusion info API
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use opaque fusion handle in ad-hoc API
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* ci : move fusion test to dedicated workflow
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* cont : run only on ggml changes
* cont : simplify
* fusion : remove multi-output stuff for now
* ci : fix typo
* metal : fix idle threads in the remaining iq mul_mv kernels for ne00 < 1024
Generalize the row split from #28086 to the six other kernels that use the
same lane-to-block mapping: iq1_s, iq1_m, iq2_xxs, iq2_xs, iq2_s and iq3_s.
Each of them assigns one 32-element chunk per thread, so when a row has
fewer than 32 chunks the rest of the simdgroup is idle. When nb32 < 32 and
nb32 divides 32, 32/nb32 threads now share each chunk and each takes a
slice of the rows, reusing the FC_mul_mv_split function constant and the
dispatch wrapper introduced for iq3_xxs.
The plain path is untouched: wide matrices keep one thread per chunk and
N_R0_<TYPE> = 4. Only the split path uses N_R0_<TYPE>_SPLIT = 8. The
K-quants have the same idle-thread issue but a different lane mapping, so
they are left for a separate change.
* metal : offset the src0 row pointer once in the iq mul_mv kernels
q2, dh, sc, qh and signs are all derived from xr, so the row slice
offset only has to be applied to xr.
* metal : fold iq mul_mv row split into offset0
Compute row0 and row1 before initializing the source pointers and apply
the row slice directly to offset0.
This keeps x and its derived pointers on the existing path while applying
the split row offset once.
* server: fix speculation after an image
Pass the actual position to the drafter after an image, instead of the
token count. Affects every drafter, not just DFlash.
* rename draft n_past to pos0
n_past is used to denote number of tokens and this parameter is meant to be a position
* HIP: enable mma FA for head size 256 on RDNA4, tune configs
Assisted-by: Claude
Assisted-by: Codex
* HIP: prefer whole-tile FA grids over stream-k on AMD WMMA
Assisted-by: Claude
Assisted-by: Codex
* revise stream_k logic
* revise kernel selection logic
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
argsort had a data race in the inner loop, which VVL caught. But I don't think
this was causing failures in practice.
argsort_large has OOB accesses which might explain the failures in CI, but I
couldn't reproduce it locally and I don't think it's a convincing explanation
of the failures.
* vulkan: optimize m=1 mul_mat by swapping A/B
* vulkan: Improve small M perf
Allow split_k with small M.
Make small vs med tile selection (for coopmat2) depend on M, not just N.
* speculative: fix failed to decode mtmd chunk with DFlash
When using DFlash w/ vision models, the drafter memory fails to
allocate new tokens because images report a fixed offset. Stop copying
them to allow the drafter to continue.
* address PR feedback
limit M-RoPE skip to images only, allow audio to pass through. Clean up
comments to align to the updated implementation
This commit updates the version parsing in make-release-checks.sh to use
sed instead of grep. The motivation for this is that currently when
running this script on macos it errors:
```console
$ ./scripts/make-release-checks.sh --dry-run
grep: invalid option -- P
usage: grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
```
With the changes in this commit it is possible to run this without
failure.
Some of these if statements were copypastaed in a former refactor and
never cleaned up to remove the cases that could never happen anymore. The
only thing that's shared between these relatives anymore is
llama_model_bert::graph::graph, so the rest of the code doesn't need the
conditionals.
The Imagination proprietary Vulkan compiler returns VK_ERROR_UNKNOWN from
vkCreateComputePipelines for every dequant mul_mat_vec shader built with the
subgroup-only reduction that requires a subgroup size >= 16. That covers the
k-quants, the i-quants, TQ2_0, MXFP4 and NVFP4. ggml rethrows, so the first
generated token of any such model kills the process.
Reproduced on a Pixel 11 Pro (PowerVR C-Series CXTP-48-1536 MC1, driver
1.662.3024, subgroup size 128, min 32, max 128). The failure is independent of
subgroup size: 32, 64 and 128 all fail, as does dropping the full-subgroups
flag and the required-subgroup-size pNext. The legacy quants, which use the
plain subgroup reduction, compile and run fine.
The shared-memory reduction variant compiles and matches the CPU reference for
q2_K, q3_K, q4_K, q5_K and q6_K. The hybrid variant also compiles but costs
27% of token throughput (3.78 vs 5.20 t/s on Qwen3.5-2B-Q4_K_M).
* vulkan: use spec constant for mul mat type_a
vulkan: use map for mul_mm shapes
cleanup
fix indentation
fix cm2 and shmem init
fix cm2 spec constants
fix cm2 bindings
consolidate shmem tables and reduce size by type spec constant
fix compiler warning
fix missing Q2_0 type
fix unused warning when integer dot glslc support is missing
use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug
fix missing Q2_0 type in cm2 matmul
fix types
* remove LUT quants from unified shader
* clean up
* restore coopmat2 q4_k/q5_k optimization
* split out q4_k/q5_k cm2 shader to fix Ampere regression
* revert iq shmem table renames
* simplify cm2 code with single uint8_t buffer
* fix fp4 extension use switch being overwritten by generic shader
* clean up
* adapt TQ1_0 changes
* adapt #27471 f16 Intel tuning changes
* divide workload to 2D
This is to workaround FILL exceeding maxComputeWorkGroupCount for Intel GPUs on Qwen 3.8 flash next
* minor change
* Fixed comment
Templates that default an optional variable to none and then test its
membership in a map hit an error, while the same expression is a normal
lookup returning false in Jinja. The undefined counterpart of this case
was already handled just above.
* vulkan: add dedicated iq4_xs mat-vec shader
Dedicated mul_mat_vec_iq4_xs for the dmmv path, replacing the generic fallback. ~+6-17% token generation on RDNA4 depending on model.
Assisted-by: Pi agent with Qwen3.8 27B
* vulkan iq4_xs: remove dead n_it unroll branch
Remove the n_it <= 8 experimental branch that attempted to fully unroll
the block loop. Since n_it is a runtime value, [[unroll]] is ignored by
the compiler, making both branches equivalent. Kept the simple loop
matching mul_mat_vec_iq3_s.comp.
The spacing eviction in create_checkpoint() keeps the oldest checkpoint and
erases every later one within checkpoint_min_step of it. For prompts shorter
than checkpoint_min_step this drops the checkpoint at n_tokens - 4 that the
next request resumes from, so hybrid/recurrent models re-prefill from the
previous checkpoint instead. Apply the spacing rule only once the list is at
n_ctx_checkpoints, and replace an existing checkpoint at the same n_tokens
instead of appending a duplicate.
* metal : fix half-idle simdgroup in kernel_mul_mv_iq3_xxs_f32 for ne00 < 1024
* metal : keep N_R0_IQ3_XXS = 4, dispatch a separate 8-row split kernel for ne00/32 < 32
The plain kernel is unchanged from master (4 rows per simdgroup, one thread per
chunk). The row-split mapping now lives in a separate kernel_mul_mv_iq3_xxs_f32_split
instantiation with N_R0_IQ3_XXS_SPLIT = 8, and the host selects it only when
ne00/32 < 32 and divides 32, so wide matrices keep the master kernel bit for bit.
* metal : select the iq3_xxs row split with a function constant instead of a separate kernel
* ci : add PYTEST_WORKERS=1 to fix server-self-hosted job
This commit adds the `PYTEST_WORKERS=1` environment variable to the
hf-jobs-t4-small:cuda13 runner steps.
This is an attempt to address CI failure of this job that I might have
introduced in Commit 42f0225fea
("server : use pytest-xdist for server tests (#28298)").
Refs: https://github.com/ggml-org/llama.cpp/actions/runs/34126971262/job/101757819134
* apply same changes to server-metal steps
* vulkan : fuse UNARY(SIGMOID|SILU|SOFTPLUS) + MUL
* vulkan : fuse UNARY(SIGMOID|SILU|SOFTPLUS) + MUL
- implement fusion in unary.comp behind UNARY_MUL_FUSION ifdef,
specialized pipelines per op instead of runtime branching
- fuse adjacent nodes only, ordering handled by graph_optimize
- drop runtime consumer scan and pending_unary_mul deferral
* vulkan : fuse UNARY(GELU|SIGMOID|SILU|SOFTPLUS) + MUL
1. GELU: gelu_mul_f32/f16 pipelines registered, CREATE_UNARY_MUL(gelu), GELU in dispatch + fuse gate + perf fusion name
2. Renamed/moved: gate is now ggml_vk_can_fuse_unary_mul(cgraph, unary_idx, mul_idx), placed with the other can-fuse helpers
3. norepeat both variants: each op gets plain (spec {0}) + _norepeat (spec {1}) pipelines from the same SPIR-V, selected via ggml_are_same_shape(src0, src1); the shape gate now allows broadcast (other dims equal-or-1)
4. graph_optimize: lambda deleted; standard "// UNARY + MUL: pull the consuming MUL forward" block added alongside the SSM_CONV/ROPE/MUL_MAT reorderings, with the same "other src must be weights or already processed" readiness check
* vulkan : align unary_mul fusion with binary kernel layout, relax gelu test tolerance
- schedule the fused kernel like mul.comp (256 threads x 2 unrolled
iterations), recovering a 10-18% prompt-processing regression
- allow 5e-7 f32 error for gelu_mul: the shader evaluates gelu with an
exp-based tanh identity while the CPU reference uses tanhf (~1 ulp)
* vulkan : use ggml_can_repeat in UNARY+MUL fusion shape check
The fused kernel indexes src1 via per-dim fastmod (generic_binary_head.glsl),
which is exact whenever the other operand tiles into the unary result -- not
just when its dims are equal or 1. Replace the hand-rolled loop with
ggml_can_repeat(other, unary) so the check matches the kernel's actual
capability and reuses the standard helper. Argument order matters: reversed,
it would wrongly admit graphs where the unary result is mul->src[1] and the
other operand is larger, producing truncated output.
Also add a rep_ne0 layout to the fused unary+mul backend tests covering a
non-1 repeat factor along dim 0.
* vulkan : fuse UNARY+MUL pairs separated by zero-compute nodes
gemma4's per-layer embedding gating builds gelu -> view_2d_slice -> mul,
where the intervening view is a zero-compute node aliasing an input that
was computed much earlier. Strict adjacency requirements meant neither
CUDA nor the vulkan unary+mul fusion handled this pattern.
Extend ggml_vk_graph_optimize to detect a UNARY whose consuming MUL is
separated only by unscheduled zero-compute nodes (GGML_OP_NONE, VIEW,
RESHAPE, TRANSPOSE, PERMUTE) and schedule those nodes ahead of the pair,
making it adjacent so the existing fusion applies. The reorder is guarded
by ggml_vk_can_fuse_unary_mul, a source-availability check for every
interleaved node, and the protected fusion patterns (topk_moe*, snake);
if fusion is later rejected the reordered graph still executes correctly,
just unfused.
Add a view_mid layout to the fused unary+mul backend tests replicating
the gemma4 pattern.
* vulkan : support OP-on-B in UNARY+MUL fusion
Some models apply the unary activation to the smaller MUL operand, e.g.
qwen3next/qwen35moe shared-expert gating builds ffn_shexp * sigmoid(gate)
with a [1,n_tokens] gate tensor. This shape was correctly rejected before:
the fused kernel derives its iteration extent from the unary tensor and
would leave most of the destination unwritten, and the generic same-shape
requirement in ggml_can_fuse blocked the pair outright.
Add UNARY_MUL_B_FUSION shader variants computing dst = src0 * OP(src1):
the OP operand rides the existing per-dim fastmod indexing, while the
iteration extent now comes from mul. Route {UNARY, MUL} pairs through a
local can-fuse variant that drops the generic same-shape rule and instead
requires the unary result to tile into mul->src[0] (ggml_can_repeat);
pairs with the unary as src0 keep the previous direction check, and
equal-shape pairs keep using the original pipelines.
Add a "gate" layout to the fused unary+mul backend tests covering the
shared-expert gate shape for gelu/sigmoid/silu/softplus in f32 and f16.
* vulkan : fold unary+mul view-hoisting into graph_optimize dep checks
Replace the dedicated UNARY + EMPTY* + MUL scanning block with two small
extensions to the existing scheduling logic:
- a consuming MUL may now join its in-set UNARY across a gap of unused
zero-compute nodes (NONE/VIEW/RESHAPE/TRANSPOSE/PERMUTE), instead of
requiring strict adjacency
- while doing so, such zero-compute blockers are ignored for this pair
Fusion validity is still decided later by ggml_vk_can_fuse at dispatch
time, so a rejected pair simply executes adjacent-but-unfused. Note the
relaxation must stay scoped to this pattern: exempting zero-compute
blockers globally reproduces silent output corruption on gemma3n.
* vulkan : select unary_mul OP-on-B via specialization constant
Replace the UNARY_MUL_B_FUSION compile-time shader variants with an
op_on_b specialization constant on the existing unary_mul SPIR-V,
mirroring how the norepeat flag is handled. The four {op}_mul_b_{f32,f16}
shader artifacts are gone - the OP-on-B pipelines reuse the base SPIR-V
with two-entry {norepeat, op_on_b} spec lists - and the duplicated store
expression is collapsed into a single runtime branch that the driver
prunes per specialization.
The constant is declared only under UNARY_MUL_FUSION so every other
binary pipeline keeps its single-entry specialization list.
* vulkan : replace unary_mul pipeline switches with a lookup table
Collapse the four nested selection switches in ggml_vk_unary_mul into a
single indexed lookup against a pipeline_unary_mul[4][2][2][2] table
([unary op][f16][norepeat][op_on_b]), whose trailing dims mirror the
{norepeat, op_on_b} spec constant list. The op axis uses a small shared
index helper that also replaces the switch in ggml_vk_can_fuse_unary_mul,
making it the only place that maps ops to the table.
Pipeline names are unchanged. Adding another supported op now requires
one macro invocation line and one helper case instead of edits in four
separate switches.
* vulkan : use ggml_can_fuse_subgraph for unary_mul pairs
Replace the hand-rolled pair validation in ggml_vk_can_fuse_unary_mul_pair
(bounds, op match, compute flags, single-use elision) with the shared
ggml_can_fuse_subgraph helper; backend-specific shape/type rules remain in
ggml_vk_can_fuse_unary_mul. Unlike ggml_can_fuse, the subgraph helper has
no same-shape requirement, so it covers both operand slots including
OP-on-B gates, and additionally rejects intermediates flagged as graph
outputs and validates view-source confinement.
The outputs parameter takes absolute node indices into the cgraph.
* Fix Whitespace
* vulkan : drop redundant unary_mul gap check in graph_optimize
The zero-compute nodes separating a UNARY from its consuming MUL are
already scheduled ahead of the pair by pass 2 of an earlier
optimization window, so the scoped gap tolerance added for this pattern
is unreachable in practice - disabling it leaves gemma-3n dispatch
counts unchanged (841 GELU_MUL per pass). Remove the flag, the empty
blocker exemption, and the now-unused gap helper, restoring the strict
adjacency requirement of the UNARY -> MUL pull-forward.
Keep the relaxation scoped out entirely: generalizing "zero-compute
nodes never block" beyond this pattern previously reproduced silent
output corruption on gemma3n.
* vulkan: fix whitespace (tab in indent)
* vulkan: fix whitespace (extra blank line)
* vulkan : move op_on_b spec constant to unary.comp
op_on_b is only used by the fused unary*mul path. Keep
generic_binary_head.glsl generic by defining it in unary.comp
instead. Same constant_id=1 and guard, no functional change.
* vulkan : make RMS_NORM/UNARY fusion gap-tolerant for views
Strict j==c+1 blocked RMS_NORM->MUL and UNARY->MUL when a
VIEW sits between (e.g. rms_norm -> view -> mul). Allow
c==back() with an empty-or-scheduled gap, matching the
review suggestion to check src linkage instead of adjacency.
Scoped to the two blessed pairs; safe because gaps can only
contain zero-compute nodes.
* vulkan : trim comments in UNARY+MUL fusion
Assisted-by: Muse Spark
On 32-bit platforms, Vulkan non-dispatchable handles such as VkBuffer are
represented as uint64_t, and Vulkan-Hpp disables implicit conversions for
type safety. This exposes two issues in ggml-vulkan:
1. vk::Buffer is streamed directly into std::ostream in debug/memory logs.
2. vk::Buffer is cast to VkBuffer before being passed to Vulkan-Hpp
CommandBuffer::copyBuffer APIs.
Fix these by add the operator<< for vk::Buffer, and
by passing vk::Buffer directly to Vulkan-Hpp copyBuffer calls.
* chat : split specialized parsers into common/parsers
Move the 14 dedicated template parsers out of chat.cpp into one file each under
common/parsers, mirroring the src/models split. chat.cpp keeps the template
detection in common_chat_try_specialized_template() and drops from 3915 to 1513
lines.
common/parsers/parsers.h holds the shared helpers and one declaration per
parser. foreach_function/foreach_parameter become inline there since nothing in
chat.cpp uses them any more; common_chat_template_direct_apply_impl and
common_chat_template_generation_prompt_impl lose static and carry their default
arguments in the header. Parser-specific helpers move with their parser:
is_lfm2_template, deepseek_v4_sort_tool_results and the gemma4 turn builder.
No functional change.
Assisted-by: Claude Opus 5
* chat : enumerate parser sources instead of globbing
file(GLOB) does not re-run CMake when a source file is added or removed, so an
incremental build silently keeps building the old set. List the parsers in
common/parsers/sources.cmake and include it from common/CMakeLists.txt.
Assisted-by: Claude Opus 5
* split helpers, add newlines
* tests: bind the L2_NORM batch count to a local
GCC cannot prove the loop fills norms up to the index read after it
while the bound is a class member, so it reports a maybe uninitialized
use. Reading the count once into a local restores the tracking.
* tests: initialize the L2_NORM batch array
The read after the fill loop is only provably defined once the array
carries an initializer, which GCC 12 requires on the aarch64 Release
build where warnings are fatal.
* server: fix LRU hang on multiple requests same model
* server: keep a queued model out of the victim pool until its waiters leave
A waiter that gave up while its model was still loading left the
model idle with no request behind it, and nothing recounted the free
slots, so a second request queued behind it stayed queued forever.
tick() was only driven by requests: join, claim and the end of a
proxied request.
Keep the queue entry alive after a successful claim so the model
coming up is never picked as a victim before its waiters use it, and
recount the slots on every status change and whenever a waiter
abandons the queue. The model is then evicted as soon as it comes up
with nobody left to serve.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* vulkan: add DeepSeek-V4 hyper-connection fused ops (DSV4_HC_COMB/PRE/POST)
CUDA has these ops from the DeepSeek-V4 merge and Metal gained them in
PR 26459. Vulkan was the last major backend running the unfused primitive
chain. On DeepSeek-V4-Flash the unfused Sinkhorn comb chain alone takes
about 32% of decode op time on gfx1151 (Strix Halo), spread over roughly
16k dispatches per token.
dsv4_hc_comb runs the full 20-iteration Sinkhorn in registers. A token's
4x4 comb matrix lives in 16 consecutive subgroup lanes, with idst in bits
0-1 and isrc in bits 2-3 to match the CPU reference layout, so
subgroupShuffleXor by 1|2 reduces rows and by 4|8 reduces columns. One
dispatch replaces about 137 strictly ordered node executions per site.
The shuffle masks never cross a 16-lane boundary, so a subgroup of size
64 packs 4 independent tokens.
dsv4_hc_pre and dsv4_hc_post handle the elementwise stream collapse and
fan-out, with per-token coefficients staged in shared memory.
GGML_VK_DISABLE_DSV4_HC disables all three ops. The _COMB, _PRE and
_POST variants gate each op independently so a single kernel can be
bisected against the unfused graph.
Adds eval cases at the production n_iter=20 across batch sizes that
cross subgroup and workgroup boundaries.
* vulkan: dsv4 hc review fixes
Drop the per-op env-var disables and device flags, the stride divisibility
check (ggml guarantees it) and the workgroup-count fallback in supports_op.
Trim the comb shader comments to the lane layout.
---------
Co-authored-by: Kevin Hopper <no-reply@maestro.press>
* adjust ncols_picker for routed MoE in mul_mat_q_case function
* Adding CDNA, RDNA2 and RDNA4
* fix: update mmq_use_routed_moe_ncols_picker to include NVIDIA + Volta support
* feat: enhance mmq configuration for various architectures with moe_ncols_min_cc support
* refactor: replace moe_ncols_min_cc with use_typical_moe_ncols in mmq configuration files
* HIP: mmq: enable typical moe ncols on RDNA4
---------
Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
* Update Q4_K and Q5_K to use branchless computation, which stops the scale unpack being re-executed for every column in mmvq, improving perf at batch sizes > 1
* Gating the change off from DGX Spark due to no gain
* Adding prefetch gated to Spark, making branchless change in Q4_K and Q5_K general and modifying switch points based on latest perf data
* Guard the mmvq L2 prefetch against MUSA as well as HIP
* Define the mmvq L2 prefetch only under the Spark guard
* Update switch point for Q4_K to accommodate more models
* Remove stale comments
* Add block_size to ggml_cuda_type_traits and create a separate mmvq_should_prefetch function
* Rename block_size to bs for cleaner indentation
* Fix build error on non-Spark CUDA arch with appropriate conditional around new function added
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
* vulkan: fall back to CPU for GET_ROWS with misaligned offsets
The Vulkan GET_ROWS shader asserts when a tensor's backing-buffer offset
plus view_offs is misaligned w.r.t. minStorageBufferOffsetAlignment
(see init_pushconst_tensor_offsets). Previously this caused a hard crash
on models using ggml_view + ggml_get_rows (e.g. Qwen3-TTS, Qwen3-VL).
Return false from supports_op() in the misaligned case so the scheduler
falls back to CPU, matching the existing pattern for PAD_REFLECT_1D and
other unsupported op/shape combinations.
Repro: llama-tts -m Qwen3-TTS-*.gguf -mm mmproj-*.gguf -ngl 99
Crash: GGML_ASSERT(dst->op != GGML_OP_GET_ROWS || (a_offset == 0 && ...)) failed
* vulkan: trim comment for GET_ROWS misalign fallback
* vulkan: fix file corruption in gated_linear_attn struct
* vulkan: properly handle misaligned offsets in GET_ROWS quantized path
- get_rows_quant.comp was missing get_aoffset()/get_boffset()/get_doffset()
calls that are already present in get_rows.comp, causing GGML_ASSERT crashes
when GET_ROWS operates on views with non-zero view_offs, as produced by
KV cache slices in Qwen3-TTS and Qwen3-VL.
- Remove the defensive misalignment GGML_ASSERT in init_pushconst_tensor_offsets
for the binary push-constants specialization, since both get_rows.comp and
get_rows_quant.comp now correctly apply per-tensor base offsets.
- Remove the workaround CPU fallback in supports_op() for GET_ROWS, since the
Vulkan backend now handles misaligned offsets natively (no more bailout).
- Add backend test coverage with view_src0=true (ggml_view_4d into a padded
tensor) for F32, F16, Q4_0, Q4_K, Q8_0, and I32 types, exercising both the
non-quantized (get_rows.comp) and quantized (get_rows_quant.comp) paths
with non-zero view_offs that reproduce the original Qwen3-TTS crash.
* tests: trim redundant comments in test_get_rows vs0 region
* tests: trim redundant comments in test_get_rows vs0 region (follow-up)
* vulkan: bind tensor base for binary ops, pass full view_offs via push constants
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, MUL, etc.),
bind the view_src base and pass the full view_offs divided by type_size via
push constant misalign_offsets. This avoids truncation when misalign_bytes is
not a multiple of quantized block size.
ggml_vk_tensor_subbuffer gains a use_view_offs parameter. When false, the
binding points to vk_tensor_offset (base) and size includes view_offs.
init_pushconst_tensor_offsets<binary> computes a/b/d_offset directly from
tensor->view_offs, which is always row-aligned and therefore exact.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).
* vulkan: bind aligned offset for binary ops, pass adjusted misalign via push constants
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.
ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types).
ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (target - adjusted_misalign) instead of the view_src base, keeping the
offset small enough for 16-bit/8-bit push constant fields.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).
* vulkan: bind aligned offset for binary ops, fix UMA offset mismatch
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.
Added ggml_vk_tensor_physical_offset to unify physical offset lookup across
UMA and non-UMA devices. On UMA, resolves via ggml_vk_host_get(tensor->data);
otherwise uses vk_tensor_offset(t) + t->view_offs. Both get_misalign_bytes and
the new ggml_vk_get_adjusted_misalign helper build on top of this function,
so buffer bindings and push constant offsets are always consistent regardless
of device memory model.
ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types) while
remaining small enough for 16-bit/8-bit push constant fields
(adjusted_misalign < lcm(align, type_size)).
ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (physical_offset - adjusted_misalign) on both UMA and discrete GPUs,
fixing a bug where the UMA host_get path previously skipped the adjusted
misalign binding and returned the target offset directly.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA GeForce RTX 5060 Ti).
* finish misalignment fix
* supports_op changes for openvino/webgpu
---------
Co-authored-by: AiChiTuDouPian <15327701848@qq.com>
This commit adds the printing of the ggml version and commit to the
test-cmake example.
The motivation is just to be able to quickly verify that the correct
version of ggml is being used.
Example output:
```console
test-cmake] llama.cpp version: 0.4.0-dev, build: 10837 (5202104b5)
[test-cmake] ggml version: 0.23.0, commit: 5202104b5
[test-cmake] Initializing backend...
...
```
Problem
- Loader prefers `<arch>.attention.recurrent_layers`, falls back to `full_attention_interval` if missing
- Converter only ever writes the interval. gguf-py has no constant/writer for the array
- Interval can only describe evenly spaced full-attention layers. Any non-uniform `layer_types` gets reconstructed wrong
- No error, no warning. Model loads, runs, wrong layers get wrong ops. Full-attn layers marked recurrent lose their KV cache
- Every published Qwen3.5 checkpoint is uniform so nobody's hit it yet
Repro
12 layers, periods 4/3/5:
layer: 0 1 2 3 4 5 6 7 8 9 10 11
actual: L L L F L L F L L L L F
loader: L L L F L L L F L L L F
^ ^
Layer 6 is full attn, loaded as recurrent. Layer 7 the reverse.
52-layer non-uniform stack: 15/52 mis-typed.
Fix
- `constants.py`: add `Keys.Attention.RECURRENT_LAYERS` (name already registered in llama-arch.cpp)
- `gguf_writer.py`: add `add_recurrent_layers()`, same shape as `add_rope_pattern()`
- `conversion/qwen.py`: emit array from `layer_types` in `Qwen3NextModel.set_gguf_parameters` (covers 3-Next, 3.5, 3.5-MoE)
Notes
- Array is padded with `false` for MTP blocks. `get_key_or_arr` checks length against `n_layer_all`, which includes MTP. Matches the fallback's `i < n_layer()` guard
- Interval is still written. Old builds only understand the interval
- `layer_types` length != `num_hidden_layers` now raises in converter instead of producing a GGUF that fails at load
Tested
- End-to-end on a 62-layer non-uniform Qwen3.8-27B (2 linear layers removed). Loader reads the array, 62 blocks, 0 mismatches. Without fix: interval fallback, mis-typed
- MTP padding NOT tested on a real MTP model. Reasoned from qwen35.cpp + get_key_or_arr. Would appreciate a check
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Support RMS_NORM + MUL + ADD (+ MUL) and RMS_NORM + VIEW + SET_ROWS.
Extend ROPE + VIEW + SET_ROWS to support IMROPE.
Worth around 4% in gemma4 on my system.
This commit contains a suggestion for handling container images which
are currently not semver tagged, they only have build numbers in there
tags.
The proposed solution here is to first add a check to make sure that
there are container images built for the build number of the release and
if not fail the build. The container images are build nightly but they
can be triggered manually as well.
If the the container images check passes then the make-release workflow
will re-tag the images with the semver.
* vulkan: add TQ1_0 support (mm, mat-vec, dequant, get_rows)
* vulkan: pack TQ1_0 powers of 3 into a 32-bit constant
Replaces the constant array with a packed 32-bit value (7 bits per entry,
max 81 < 128) extracted with shift/mask, as suggested in review — avoids a
constant array that may not be kept in registers.
test-backend-ops on gfx1151: tq1_0 MUL_MAT 11/11, MUL_MAT_ID 6/6,
GET_ROWS 4/4, unchanged.
* vulkan: address review - shared TQ1_0 decode helpers, fix standalone dequant shader
Review feedback from jeffbolznv, all points:
- Move the packed-pow3 decode into shared helpers in types.glsl
(tq1_0_byte_of / tq1_0_digit_of / tq1_0_trit) and use them from
dequant_funcs.glsl, mul_mm_funcs.glsl, dequant_funcs_cm2.glsl and
dequant_tq1_0.comp instead of repeating the logic. The cm2 path also
drops its constant array for the packed-constant extraction.
- Translate all remaining comments to English.
- dequant_tq1_0.comp: use dequant_head.glsl. The shader previously declared
its own single-field push constant while the pipeline is created with the
5-field layout, so p.ne read the wrong field - confirmed broken, as
suspected in review.
- Fix wg_denoms for the standalone dequant pipeline: one invocation decodes
4 elements with local_size 256, so a workgroup covers 256*4 elements, not
256*16. With the old value the dispatcher launched a quarter of the
required workgroups.
Verified by temporarily forcing the dequant + f16 matmul path for TQ1_0
(hack not committed): test-backend-ops MUL_MAT passes through the rewritten
standalone shader, and the standard MUL_MAT / MUL_MAT_ID / GET_ROWS
tq1_0 cases still pass on Vulkan (AMD gfx1151).
* vulkan: address review — English comments, shared tq1_0_trit, trim TQ1_0 test cases
- mul_mat_vec_tq1_0.comp: drop leftover non-English comment and the local
POW3_PACKED constant; all decode sites now call tq1_0_trit() from types.glsl
- types.glsl / dequant_funcs_cm2.glsl: ASCII-only, drop stale reviewer note
- test-backend-ops: remove the oversized MUL_MAT_ID case (432 MiB A tensor,
~172 GFLOP reference); move the two remaining ones next to the other
backend-specific mul_mat_id one-offs and document why they are needed
* metal: decline TQ1_0 for GET_ROWS and mat-mul in supports_op
The new TQ1_0 cases in test-backend-ops exposed that the Metal backend
claimed support for GET_ROWS/MUL_MAT/MUL_MAT_ID with TQ1_0 sources while
having no such kernels (ggml_metal_library_compile_pipeline aborted on the
missing kernel_get_rows_tq1_0). Decline the type so the ops fall back to
the CPU, matching the existing NVFP4 handling on the same lines.
Assisted-by: Claude Fable 5
* vulkan: trim the TQ1_0 comments
Addresses @0cc4m's review: keep only what the code does not already say.
Removed the block-format recaps (the layout is right there in the struct) and
the step-by-step decode walkthrough. Kept the two facts a reader cannot infer:
the 8-bit truncation is part of the format, not an optimisation, and the powers
of 3 are packed into one uint so they do not end up in a constant array that
may miss the registers.
No functional change.
* vulkan: address review — trim comments, fold Metal check, drop unused _v
Per @0cc4m's review:
- dequant_funcs.glsl, dequant_funcs_cm2.glsl: drop the "see types.glsl"
pointers — they apply to every quant and say nothing specific.
- dequant_tq1_0.comp: drop the wg_denoms note. It is a precondition, not
information.
- mul_mm_funcs.glsl: same pointer removed.
- types.glsl: the comment on tq1_0_trit is down to the one fact the code
cannot show — the 8-bit truncation is part of the format, matching the C
reference, not an optimisation.
- dequant_funcs_cm2.glsl: removed dequantFuncTQ1_0_v and its define. You were
right that it is optional: it wrapped four scalar decodes and vectorised
nothing, and mul_mm_cm2.comp already guards the path with
`#if defined(dequantFuncA_v)` (DATA_A_F32 omits it the same way).
- ggml-metal-device.m: folded TQ1_0 into the existing NVFP4 check instead of a
separate block, and dropped both comments.
- test-backend-ops.cpp: the two mul_mat_id cases stay — they cover the
block-stride loop and the per-expert base offset that k == 256 alone never
reaches — but the comment is now one line instead of five.
Kept: the one-line labels on the three block regions in mul_mat_vec_tq1_0.comp
and on tq1_0_byte_of(). Those state the 5-trits-per-byte packing, which the
loop bounds do not show. Happy to remove them too if you prefer.
Re-verified on AMD gfx1151 (Vulkan), test-backend-ops, 2/2 backends passed:
MUL_MAT 9 TQ1_0 cases, MUL_MAT_ID 5, GET_ROWS 4 — all OK, no failures.
The coopmat2 path is unchanged apart from the removed _v define.
* models: use flash-linear-attention's l2norm for gated delta net q/k
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
* Update src/models/models.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ui : update active conversation fields in place
updateCurrentNode, applyConversationUpdate, updateConversationTimestamp
and the pin toggle replaced the whole activeConversation object, so its
identity changed on every send, tool result and rename. ChatMessages
tracks that identity to refresh sibling info, so each replacement
triggered a full refetch of every message in the conversation. Write the
changed fields instead, mirroring updateMessageAtIndex.
Assisted-by: pi:zai-org/GLM-5.3
* ui : reuse the conversation load read for sibling info
Opening a conversation read every message from the database twice: once
in loadConversation for the active path, once in ChatMessages for the
sibling map. Hand the freshly read array over once so the chat screen
builds sibling info from it, and set the conversation and its messages
in one sync block so effects never see the new conversation paired with
the previous one's messages.
Assisted-by: pi:zai-org/GLM-5.3
* ui : memoize leaf walks in sibling map build
buildSiblingInfoMap resolves each sibling's leaf by walking the last-child
chain, once per sibling per message, so the walk repeats along the same
chains for every message in the conversation ( O(messages^2) on long
chats ). Memoize leaf resolution per build with path compression so each
edge is walked once.
Assisted-by: pi:zai-org/GLM-5.3
* ui : skip sibling refetch for in-place message edits
refreshAllMessages refetches every message of the conversation just to
rebuild sibling info, but preserve-responses and non-branching assistant
edits never create branches, so the sibling map stays valid. Refresh only
after actions that branch (editWithBranching kept) or delete.
Assisted-by: pi:zai-org/GLM-5.3
* ui : drop unused currentResponse reactive writes
Nothing reads chatStore.currentResponse, but setChatStreaming reassigned
it on every streamed chunk, so each token paid a reactive write and string
assignment for nothing. Remove the field and the clearUIState wrapper
that only reset it.
Assisted-by: pi:zai-org/GLM-5.3
* ui : reuse completed agentic turn sections during streaming
deriveAgenticSections runs in a $derived invalidated per streamed chunk,
but re-derived every turn of the session each time, so per-chunk cost grew
with session length. Cache completed turns keyed by their assistant message
plus reference checks on every field that feeds derivation; only the
streaming turn recomputes. Cache hits return the same section objects, so
tool block props stay stable and skip their per-chunk re-derive.
Assisted-by: pi:zai-org/GLM-5.3
* ui : share markdown block infrastructure
Every markdown block duplicated shared work: a full copy of the hljs
theme CSS per instance, and the remark/rehype plugin chain rebuilt on
every processMarkdown call ( once per block at mount, again per coalesced
chunk while streaming ). Use the single theme style element already
maintained by SyntaxHighlightedCode, and build pipelines once - shared
process-wide for attachment-less blocks, cached by attachments identity
otherwise.
Assisted-by: pi:zai-org/GLM-5.3
* ui : measure assistant layout only for the last message
Every assistant message ran getComputedStyle, getBoundingClientRect and
a ResizeObserver over the previous user bubble at mount, even off-screen
ones, forcing a layout pass per message while a long conversation
renders. The measured vars only feed the :last-child min-height rule, so
gate the effect on isLastAssistantMessage; one measurement and one
observer remain, and the effect re-runs when the last message changes.
Assisted-by: pi:zai-org/GLM-5.3
* ui : trim whole-blob scans in tool block headers
Tool block headers parsed their entire blobs at mount, even collapsed,
and most tool results and args are large plain text or embedded file
content: skip JSON.parse unless the blob starts with a JSON container,
prefilter search-result extraction with a Title:/URL: substring check,
and match the end-anchored exit-code marker against only the tail of exec
outputs.
Assisted-by: pi:zai-org/GLM-5.3
* ui : parse write_file and edit_file titles without the content blob
Both block headers parsed the full args JSON at mount, even collapsed, and
write_file and edit_file args embed the whole file content or edit
strings, so every block paid a full-blob JSON parse just to read the path.
Split the meta into a title tier that extracts the path with a targeted
key match (full parse only as fallback) and a body tier that keeps the
full parse; Svelte deriveds are lazy, and the body snippet renders only
while the block is expanded, so collapsed blocks no longer parse args.
Assisted-by: pi:zai-org/GLM-5.3
* ui : mount chat messages lazily near the viewport
Every message row mounted its full component tree on load, so the cycle
collector, GC and layout invalidation kept walking every live object and
DOM node even for rows the user never scrolls to - which dominated the
profile of long conversations. Wrap each row in a placeholder with an
IntersectionObserver ( two viewport heights of runway ) that swaps in the
real ChatMessage when the row approaches the viewport; the row shell
keeps the content-visibility sizing, and rows stay mounted once
realized. Rows targeted by the pending-edit flow mount eagerly.
Assisted-by: pi:zai-org/GLM-5.3
* ui : smooth the chat navigation animations
Slide the centered new-chat form to the bottom edge with a transform
instead of a bottom offset - layout-property transitions need the main
thread every frame and stutter while a long conversation loads, while
transform transitions run on the compositor. Fade the message list in
with a CSS animation keyed to the conversation id, disabled under
prefers-reduced-motion.
Assisted-by: pi:zai-org/GLM-5.3
* ui : follow the svelte runes guidance in chat message code
Two effects detected changes with manual previous-value refs and reset
flags. The permission request carries object identity, so its dismissal
is now a derived comparing the dismissed request; the continue request
is a bare boolean, so its dismissal only shrinks to a reset while no
request is pending. Also drop a dead if (browser) guard in the markdown
theme loader - effects never run on the server.
Assisted-by: pi:zai-org/GLM-5.3
* test : pin the chat perf invariants in the unit suite
Cover the fixes whose silent regression would be stale or wrong UI rather
than a crash: the turn-section cache must reuse unchanged turns yet
recompute on every field it compares; the sibling map must resolve the
same leaves after the leaf-walk memoization; the active conversation must
keep its identity through field updates; and the blob gates ( exec tail
window, plain-text result gate, search prefilter ) must keep accepting
what they gate. Only the risky invariants are pinned - no coverage for
coverage's sake.
Assisted-by: pi:zai-org/GLM-5.3
* refactor : address review remarks
Name the tool-arg string-field pattern, move the file tools' path field
aliases and the JSON container gates into lib/constants, and export the
write_file / edit_file meta types from $lib/types instead of the parser
modules.
Assisted-by: pi:zai-org/GLM-5.3
Remove the build-time C++ helper and external gzip dependency,
simplifying cross-compilation. Keep the generated C++ in templates for
readability and preserve fully embedded UI assets.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (#28184)
This reverts commit c845263f8b.
* tests : fix unused variable M in test-backend-ops
* tests: fix trailing space error and isolate kronecker tests for sycl backend only
define two new environment variables to better understand how much
memory is being allocated, and when. This has been invaluable in
inproving the --fit algorithm, and is likely to be useful when debugging
other memory-related issues.
`-lv 4` will be required to enable the following:
GGML_SYCL_MEMTRACE=1 will show per-site memory usage, updated whenever
it increases by more than 64MiB.
GGML_SYCL_MEMTRACE=2 will show every allocation and deallocation.
To change the default 64MiB threshold for reporting memory usage increases, use
GGML_SYCL_MEMTRACE_STEP.
A sample log line:
[SYCL-MEMTRACE] device memory query (dev): total 59493 MiB, free 4494, in use 54998; allocated 0 (buffers 0 + scratch 0), peak 0 MiB
* ui : fix MCP image attachments not displayed in tool block (#25789)
Fixes regression from #25450 where ChatMessageAgenticContent passed
message.extra instead of section.toolResultExtras to tool blocks,
leaving tool images invisible. Also fixes TOOL_RESULT_JSON_OPEN_REGEX
which misclassified "[Attachment saved: ...]" as JSON.
Fixes#25789
Assisted-by: Muse Spark
* Addressed PR comments: 1.- Removed ·?? mesage?extra· as it has no case left to cover 2.- Added ·[\· to cover the case of ·[[1, 2], [3, 4]]· case suggested in the PR comment 3.- Added unit test for covering up this regex case
* ui : fix MCP image attachments not displayed in tool block (ggml-org#25789) - Addressed lint error on regex (redundant \)
* opencl: add extended elementwise unary ops (sgn, step, elu, hardswish, hardsigmoid, floor, ceil, round, trunc)
Adds nine GGML_UNARY_OP_* elementwise ops that were falling back to CPU on the
OpenCL backend, following the same variant shape as the existing ABS op: f32,
f32_4 (vec4), f16, f16_4 (vec4), and stride-addressed f32_nc / f16_nc for
non-contiguous inputs. New kernels/unary_ext.cl (macro-generated), a shared
ggml_cl_unary_ext dispatch helper mirroring ggml_cl_abs, the supports_op cases,
and the compute-forward cases.
Values are computed in float (the f16 variants read/write half and convert), so
the conditional ops (step, elu) match the CPU reference; the vec4 forms use
select() for the branch.
Validated with test-backend-ops on Adreno 840 and 850 (E17): all nine ops pass
every case including the vec4 and non-contiguous variants (8/8 or 14/14).
* opencl: dispatch a contiguous f32 copy over the whole device
kernel_cpy_f32_f32 maps one workgroup to each (i01,i02,i03) row and strides the
row across that workgroup's lanes, and the host launches ne01*MIN(64,ne00) work
items. A tensor with few long rows therefore runs on a single workgroup. The
mamba2 and gated-delta-net recurrent state cache is one row of 524288 floats,
copied once per layer per graph, and lands on 64 work items.
When both sides are contiguous the copy is a linear move, so dispatch it over
the whole device: one work item per float4. Gated on ggml_is_contiguous for both
tensors and equal element counts, so copies already spread over many rows keep
the existing path. The kernel is created optionally, so a driver that rejects it
falls back rather than aborting.
vload4/vstore4 rather than a float4 cast: they require only the scalar type's
alignment, and these buffers carry an arbitrary 4-byte view offset.
CPY, DUP and CONT are 217/217 on Adreno 840 and 740 with the path enabled and
disabled. GGML_OPENCL_CPY_FLAT=0 forces the old kernel.
* opencl: support all easy-copy types in CONCAT
CONCAT was F32-only. Extend it to every "easy-copy" type -- any non-quantized
type with a block size of 1 and an element size of 1, 2, 4 or 8 bytes, i.e.
f16/bf16/i8/i16/i32/i64 as well as f32.
The kernels are keyed by element SIZE rather than by type, which is what CUDA
already does for the same op: one kernel per byte width (b1/b2/b4/b8) plus the
packed b4 fast path, instead of one per ggml type. supports_op gates on the
same property, so a new type of a supported width is picked up with no further
work.
Validated with test-backend-ops on Adreno 840 / A8X and X2-90 / X2E.
* model: add Tencent Hy 4 (hy_v4) preview architecture support
Adds support for the Tencent Hy 4 model (Hugging Face architecture
HYV4ForCausalLM, GGUF arch hy_v4):
Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry
Register hy_v4 GGUF constants, arch enum, and writer support
Implement the hy-v4 model graph, hparams, vocab and context changes
Register the new arch in llama-arch and models registry
Extend arch tests to cover hy_v4
Assisted by Claude Opus 5
* Update convert_hf_to_gguf_update.py
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* Update conversion/base.py
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* convert : move hy_v4 entry to the same place as in convert_hf_to_gguf_update.py
* model : apply changes related to n_ff_exp becoming per-layer in Hy4-preview
* n_layer_all
---------
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
This commit adds a cmake version configuration file to replace the
current compile definition solution for the version.
The motivation for this change is that I made a mistake and did not take
into consideration that the compile definition means that this will
become a compiler flag for all sources in the target. This means that
when a version update happens that will recompile all sources in the
target even if they have not changed.
Refs: https://github.com/ggml-org/llama.cpp/pull/28278
Use std::error_code overloads of fs::current_path() and
fs::directory_iterator in ggml_backend_load_best() so an
inaccessible search path (WebDAV mount, removed CWD) is
skipped instead of terminating the process with an uncaught
filesystem_error.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Let llama_print_build_info write to a caller-provided FILE* instead of
hardcoding stderr. The parameter defaults to stderr so existing callers
keep their current behavior.
The version command in llama-app now passes stdout, so plain version
output goes to stdout where users expect it.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* src : add n_expert_used_max function
With Commit c61b98b875 ("model: add
NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444)") it
is now possible for each layer to have a specific number of experts but
there are a few checks that need to be updated to handle this upon model
loading. For example:
```console
llama_model_load: error loading model: model has expert layers but no expert layers are used
```
And later:
```console
/llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed
```
This commit adds the n_expert_used_max function so that these checks
can use it.
Refs: https://github.com/ggml-org/llama.cpp/pull/25444#issuecomment-5524976031
* src : use hparams.n_expert_used_max in llama_model_base::load_hparams
* src : use 0 as initial value for n_expert_used_max
Fuse RMS_NORM+MUL+ADD and ADD+ADD under GGML_SYCL_ENABLE_FUSION.
ADD+ADD uses the same binbcast indexing and type matrix as standalone
add() (f32, f16, f16/f32, i32, i16, bf16, including broadcast and
non-contiguous). Unsupported combinations fall back to two add() launches.
* opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations
* opencl: guard q4_K/q6_K tiled_ns convert-kernel registration for non-Adreno build
* opencl: gate q4_K MUL_MAT+GLU fusion dispatch to Adreno
* opencl: require the noshuffle weight layout in the q4_K GLU fusion gate
* opencl: do not take the vectorized f16 mrow GEMV path on an unaligned row stride
* opencl: pass the new get_scale_min_k4 stride argument at the row-major call sites
* opencl: enable the q4_K split-K decode GEMV only where it is measured to win
* opencl: record the X1-85 split-K datapoint (neutral, exclusion confirmed)
* opencl: restrict the tiled lm_head/embed GEMV default to X2E/A8X
* opencl: fix q4_K variant kernels to read the transposed scales layout
* opencl: keep the flat-GEMV large-m escape opt-in
* opencl: guard the o4 GEMV store against the rounded-up dispatch tail
* opencl: restore the tiled q4_K/q6_K layout on tensor read-back
* opencl: split-K for the q8_0 decode GEMV at small M
* opencl: keep the q6_K noshuffle correctness escape ahead of the opt-in gate
* server : use pytest-xdist for server tests
This commit adds pytest-xdist to the server tests. This is pytest
plugin that distributes test execution across multiple CPU cores.
Assisted-by: pi:llama.cpp/qwen3.8-27B
Refs: https://github.com/ggml-org/llama.cpp/pull/26734#issuecomment-5220707042
* remove server_base_port and BASE_PORT
* use worksteal and pytest builting tmp_path
* metal : support n_kv_max sparse mask hint in flash attention vec kernel
- add kernel_flash_attn_ext_vec_idx: compacts finite mask entries into
a per-row index list (Hillis-Steele scan, one threadgroup per row)
- extend vec FA kernel with optional sparse index gathering (FC slot 5)
- add host-side gate: sparse path when n_kv_max > 0, mask present,
supported head sizes / KV types, n_kv_max <= 4096
- new buffer region extra_idx for the index list
- pipeline getter extended with has_sparse param
- add test cases: head sizes, quant types, nb>1, nr23 variants,
sinks, ALiBi, softcap, permute, v_view_of_k, no-mask fallback
Note: multi-row (nb*nr23[1] > 1) cases still failing - rid mapping
in the store phase needs revisiting for the sparse path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : fix sparse flash attention row addressing
- kernel_flash_attn_ext_vec_idx: mask param is half* but nb31 is a byte
stride, so the per-row mask offset was scaled by 2x; cast to char*
before applying the byte strides
- kernel_flash_attn_ext_vec: sparse pidx param is char* so the per-row
element offset was under-scaled by sizeof(int); scale it by sizeof(int)
to get the correct byte offset
- fixes the multi-row (nb*nr23[1] > 1) sparse flash attention failures
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* cont : use sparse vec FA for prefill
* metal : single-pass flash attention sparse index compaction
The idx kernel previously read the mask row twice: once to count the finite
entries (for the prefix scan) and again to recover their positions. Since the
kernel is memory-bound, this doubled the mask traffic.
Keep the finite positions in a per-thread register array during the count
pass and write them out directly, avoiding the second mask read. A dense
mask with more than NLOCAL finite entries in a slice falls back to re-reading
the mask to write the remaining positions.
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* tests : add perf cases for sparse flash attention prefill
Measure the sparse vec FA kernel across KV sizes, n_kv_max hints and batch
sizes. Run with:
./build/bin/test-backend-ops -b MTL0 -o FLASH_ATTN_EXT -p "n_kv_max=[1-9]" perf
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* qwen4 : enable sparse attention
* cont : adjust nsg
* cont : sync test-backend-ops
* cont : disable Qwen4 for now
* cont : clean-up + tests
* mtmd : mark context as const in more methods
Mark `mtmd_context` as `const` in:
- mtmd_bitmap_init_lazy
- mtmd_tokenize
- mtmd_tokenize_from_parts
- mtmd_helper_support_video
- mtmd_helper_bitmap_init_from_file
- mtmd_helper_bitmap_init_from_buf
- mtmd_helper_video_init
- mtmd_helper_video_init_from_buf
- mtmd_helper_model_can_chat
The tokenization functions in particular are useful to have marked
`const`, as that allows more easily telling the compiler that we can
safely tokenize from multiple threads (`mtmd_tokenize` is already
documented as thread-safe, this just reifies that in the signature).
* mtmd : mark tokenization input pointer as const
Mark the `bitmaps` and `parts` pointers in `mtmd_tokenize` and
`mtmd_tokenize_from_parts` as `const`. This allows more easily calling
these with immutable arrays / vectors.
* mtmd : mark llama_context as const in mtmd_helper_model_can_chat
* CUDA: Allow CUDA optimization per split for multi-GPU.
Previous guard caused multi-GPU to skip the graph optimization. The
graph is already split per device and the optimization doesnt run
over the whole model but once per split, and thus should be allowed.
However, the CUDA event ggml_cuda_concurrent_event belongs to
whichever GPU was "current" when created. If the pass ran while
GPU 0 was current, it would stick and during event creation for the
second GPU it would land on GPU 0.
The fix: set the device explicitly ggml_cuda_set_device(cuda_ctx->device);
Default behaviour remains unchanged, only active for GGML_CUDA_GRAPH_OPT=1.
Explicit device setting pattern re-used from ggml_backend_cuda_graph_compute.
* Update ggml/src/ggml-cuda/ggml-cuda.cu
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
---------
Co-authored-by: tannerbruhn <tannerbruhn@users.noreply.github.com>
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Skip the nb[3] check when ne[3] == 1, the shader never reads it for a
single stream. Cache views carry the full-buffer stride there, so the old
check reduced to n_kv == kv_size and the path only engaged with the
cache full.
* convert : skip bias_vl tensor in DeepSeek-V4 DSpark conversion
The DFLASH arch does not include FFN_EXP_PROBS_B_VL, so the DSpark
conversion failed when it tried to write the mtmd-only hash routing
tensor ffn.gate.bias_vl. Drop it like the tid2eid tensor; the DFLASH
draft only consumes ffn.gate.bias via FFN_EXP_PROBS_B.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731
* cont : fix
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* sycl: Q4_K Weight unpack optimization and reuse between destination Columns
* sycl: Q4_K small N (N=2..4) + two output rows by subgroup reuse of activation between two rows.
* sycl: gate Q4_K two-row reuse for small N=2
* sycl: Fix on magic number now uses Q4_K_MMVQ_ROW_PAIR_MIN_NROWS=6272 for it, added tests for coverage around Q4_K_MMVQ_ROW_PAIR_MIN_NROWS with perf support to test Q4_K MUL_MAT, applied the same reuse pattern to the activation as the weights.
Assisted-by: GPT-5.6 Sol
---------
Co-authored-by: RaulAbejonDelgado <raul.abejon.delgado@gmail.com>
* hparams: add per-layer n_ff_exp/n_expert_used arrays with scalar-or-array loading
G1/G2 infrastructure for variable-per-layer expert FFN size and top-k routing
(required for Puzzle-75B which has 5 distinct n_ff_exp values and 7 top-k values
across its 40 MoE layers).
Design: rename scalar members to _impl suffix (following existing convention),
add LLAMA_MAX_LAYERS arrays, add n_ff_exp(il)/n_expert_used(il) accessors with
scalar fallback. No new GGUF keys: reuses existing expert_feed_forward_length and
expert_used_count keys via get_key_or_arr (scalar -> broadcast, array -> per-layer).
- llama-hparams.h: n_ff_exp -> n_ff_exp_impl, n_expert_used -> n_expert_used_impl;
add n_ff_exp_arr / n_expert_used_arr arrays; add per-layer accessor declarations.
- llama-hparams.cpp: implement n_ff_exp(il) and n_expert_used(il); out-of-range
il returns impl safely (shared code, no abort).
- llama-model.cpp: central n_expert_used load changed to get_key_or_arr; derive
impl as max-of-array for validations and backward compat; zero both new arrays;
HunyuanVL override also zeroes n_expert_used_arr.
- llama-graph.cpp: aggregation loop in build_moe_ffn uses hparams.n_expert_used(il)
so per-layer top-k bounds the ggml_view loop correctly.
- All other files: mechanical rename hparams.n_{ff_exp,expert_used} -> *_impl.
Scalar arches are unaffected (broadcast fills all array slots with the single value).
(cherry picked from commit 269a81e03d66e1c353e1a203a0c03a03eb2b1a4e)
* nemotron-h: use per-layer n_ff_exp(il) and n_expert_used(il) at MoE call-sites
Load n_ff_exp via get_key_or_arr into hparams.n_ff_exp_arr in load_arch_hparams;
derive impl as max for existing uniform GGUFs.
In load_arch_tensors, compute n_ff_exp_i = hparams.n_ff_exp(i) with fallback to
n_ff(i)/n_expert_used(i) for GGUFs that omit expert_feed_forward_length.
In build_ffn_layer, pass hparams.n_expert_used(il) to build_moe_ffn so per-layer
top-k is used for expert routing selection.
All other nemotron-h behaviour (mamba2, attention, shared-exp, latent projection,
routed_scaling_factor, expert_weights_norm, sigmoid gating) is unchanged.
(cherry picked from commit b1878a101793cd4e59868ac72635a86ea694987c)
* arch/*.cpp + gguf-py: mechanical rename n_ff_exp->n_ff_exp_impl, n_expert_used->n_expert_used_impl
All non-nemotron arch files continue using the scalar impl member directly.
Behaviour is identical: the impl value is the broadcast value from the GGUF scalar.
gguf_writer: add_expert_feed_forward_length and add_expert_used_count now accept
int | Sequence[int], mirroring add_feed_forward_length, so converters can write
per-layer arrays with the same existing GGUF keys.
(cherry picked from commit 8f009f54bea5ef9a6a354123bd25e9d5ea2d5e03)
* convert: support NemotronHPuzzleForCausalLM (per-block MoE config)
Parse block_configs/mtp_block_configs into per-layer arrays (scalar-or-array
keys), append the MTP [attention, moe] sub-blocks as blk.88/blk.89 with
nextn tensors, accept the backbone.* prefix, and register the arch.
Also fix a pre-existing undeclared _experts attribute on NemotronHModel.
(cherry picked from commit d1a592f278336e78457454eb6c96bca917135f10)
* nemotron-h: distinguish Nemotron 3 Puzzle (75B.A9B) from Super (120B.A12B)
Both have 88 layers; the per-layer expert_used_count array (heterogeneous
for Puzzle, broadcast-uniform for Super) is the discriminator.
(cherry picked from commit f824e09dc812169589cf5662c92d149a4c18c30a)
* convert: accept the official Puzzle BF16 checkpoint's tensor naming
The officially distributed BF16 checkpoint (NVIDIA-Nemotron-Labs-3-Puzzle-
75B-A9B-BF16) names the trunk model.* (model.layers.*, model.embeddings,
model.norm_f) where the original release used the NemotronH-style
backbone.*, and spells the router bias e_score_correction_bias instead of
e_score_correction.bias. Normalize both at the top of
NemotronHPuzzleModel.modify_tensors so either checkpoint converts; every
tensor name in the official index (42683 keys, MTP head included) resolves
through the tensor map after normalization.
(cherry picked from commit 189b67fc2c9d50970416c94b3317a6e7baa49b03)
* laguna: use n_ff_exp_impl for the uniform-MoE FFN size
Laguna landed after this branch was cut and reads hparams.n_ff_exp as a
scalar. This series turns it into a per-layer array with an n_ff_exp(il)
accessor, so the three scalar reads no longer compile. Laguna is a
uniform MoE, so point them at the scalar fallback n_ff_exp_impl, same as
deepseek2/qwen3moe/gemma4 in this series. No behaviour change.
(cherry picked from commit dbedc9e19c50dca0acdfb402362e2707bee424ae)
* arch: extend the n_ff_exp/n_expert_used rename to archs added upstream
kimi-k3, dflash, bailingmoe3, deepseek4, granite-swa and the nemotron-h MTP
block still referenced the scalar fields by their old names. n_ff_exp and
n_expert_used are accessors now, so those reads no longer compile; point the
non-per-layer archs at the _impl scalars and use the indexed form where the
call site is per-layer.
* convert: keep Puzzle opted out of the NemotronH MTP export path
#26725 added MTP export to NemotronHModel, keyed on num_nextn_predict_layers.
Puzzle's config carries that key, but NemotronHPuzzleModel bypasses
NemotronHModel.__init__ (its per-block config needs a different setup), so
_mtp_bid was never assigned and modify_tensors raised AttributeError on any
mtp.* tensor. Puzzle's head is also laid out by mtp_block_configs, not the
mtp.layers.* form the base maps.
Set _mtp_bid to None, drop mtp.* in filter_tensors, and declare
supports_mtp_export = False so --mtp / --no-mtp fail at the CLI.
* llama: replace n_ff_exp/n_expert_used scalars with per-layer accessors
Follow-up to review feedback: the previous revision kept the scalar
hparams fields alongside the new per-layer arrays, which duplicated
state that get_key_or_arr already handles by broadcasting a scalar
value over every layer.
Drop both scalars and expose n_ff_exp(il) / n_expert_used(il) built
exactly like the existing n_head_kv(il) and n_ff(il) accessors: they
index the array and GGML_ABORT out of range, with il defaulting to 0
so genuinely uniform call sites stay a plain n_ff_exp().
Arch loaders now read both keys through get_key_or_arr over
n_layer_all, and the n_expert_used validation checks the maximum
across layers instead of a single field.
* llama: restore per-key required flags on the expert hparam reads
The scalar-to-array conversion passed required=false at every call site,
which silently made mandatory keys optional. Each read now carries the
same required flag it had before the conversion.
* server : accept data: URLs for input_video and input_audio
input_video and input_audio passed accept_base64_uri=false to
handle_media(), so data: URLs got treated as raw base64 strings and
failed later with a confusing media probe error (#27724).
pass true for these two content types the same way image_url already
does, and allow video/audio mime types in the data: url check instead
of image only. data URL validation now throws std::invalid_argument so
malformed input comes back as 400 instead of 500, matching the other
input validation in this file.
* server : simplify handle_media and drop unused accept_base64_uri flag
* server : update comment and add unit test for invalid data URI MIME
Extend the HTP backend's F16 unary op coverage to include ABS on top
of the existing NORM/RMS_NORM/L2_NORM/SCALE/CLAMP/SQR/SQRT set.
- Add hvx_abs_f16_{aa,au,ua,uu} + dispatcher in hvx-arith.h, mirroring
the sqr_f16 kernel structure and using the existing hvx_vec_abs_f16()
sign-bit-clear helper
- Add abs_f16() row-wise dispatch and DEFINE_UNARY_TASK_F16(unary_abs, ...)
in unary-ops.c, wired into execute_op_unary()'s op_type/task_func
switches
- Register HTP_OP_UNARY_ABS in htp_op_is_unary() (unary-ops.h) so that
ggml_hexagon_precompute_unary_params() fills kernel_params (n_threads,
VTCM layout) for ABS nodes -- required for the F16 path to function
- Narrow the F16 GGML_OP_UNARY gate in ggml_hexagon_supported_unary()
(ggml-hexagon.cpp) to allow GGML_UNARY_OP_ABS specifically, instead of
rejecting all GGML_OP_UNARY ops for F16
- Merge the separate execute_op_unary_f32()/execute_op_unary_f16()
functions into a single execute_op_unary(), branching on an is_f16
flag for the parts that actually differ by type (elem_size, the
early F16 op-support check, and which task_func table to use) while
keeping the F32-only tiled/RMS_NORM_MUL paths intact -- per review
feedback to avoid duplicating the shared VTCM/DMA plumbing
Verified on-device (QRD8850, Hexagon v81) via test-backend-ops -o ABS:
8/8 passing (F16 + F32, HTP0, no CPU fallback). Regression-checked
SQR/CLAMP/SQRT (F16+F32) and NORM/RMS_NORM/L2_NORM/SCALE (F32; their F16
paths have no CPU reference kernel in test-backend-ops and cannot be
correctness-tested there independent of this change).
* common, server : enable preserve_reasoning kwarg by default, log its effective state
If the preserve_reasoning chat template kwarg is not specified explicitly
via --reasoning-preserve / --no-reasoning-preserve, it is enabled by
default after argument processing. The server logs the effective state of
the kwarg, warns that it is enabled by default when the template supports
it, and only warns "has no effect" when it was enabled explicitly on a
template that does not support it. Setting the kwarg via
--chat-template-kwargs is deprecated.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : update comment
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
---------
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* mtmd: load the qwen3-tts code predictor proj_in as optional
The talker and the code predictor share the hidden size on the 0.6B
checkpoints, so the reference builds no small_to_mtp_projection and
the conversion emits no tensor for it. The graph already falls back
to identity when the weight is missing, the loader now agrees.
* mtmd: keep the qwen3-tts code predictor ffn_down in F32
The code predictor carries a massive activation: its layer 2 FFN
intermediate peaks around 1.5e5, well past the 65504 ceiling of F16.
mul_mat casts its input to the weight type, so an F16 ffn_down turns
that peak into inf, the residual follows, and the next rms_norm yields
NaN. Reference forward in float32 gives 145109 against 145396 measured
in the graph.
* hex-mm: fuse QKV and FFN matmuls that land on HMX
* hex-mm: remove hardcoded ne[1] < 32K restriction
* hex-get-rows: explicitly reject repacked Q8_0 just in case somebody decided to add an override
* hex-mm: correct overhead sizing to make sure we dont exceed vtcm budget for large dims
* hex-mm: fuse MUL_MAT_ID into MUL_MAT_ID_NX (2x,3x,...) where possible
* hex-fusion: update opbatch and opqueue sizing to acount for new fusion and reduce overhead for trace buffer alloc
* hex-bufs: sort buffers while finalizing opbatch, helps avoid va space fragmentation
* hex-bufs: add simple va defrag to make sure we dont abort just because the va space is fragmented
* hex-mm: replaced more scalar divs with fastdiv and minor cleanup
* hex-mm: tighten up supported fusion checks to exactly match supported kernels
* vulkan: handle larger batch sizes (>4) efficiently for IQ3_S mat-vec when NUM_COLS > 4. 5x perf at n=8
Assisted-by: Claude Opus 5
* adds 2 cases per quant type at `k=16*256` to the `all_types` mat-vec sweep
---------
Co-authored-by: Marshall <assistant@llama.cpp>
When building with gcc < 15, CMakeLists.txt unconditionally adds
ime2_kernels.cpp, which fails to compile. FindSMTIME.cmake only defines
RISCV64_SPACEMIT_IME2 when the IME2 instructions are detected, and gcc 14
only has IME1, so ime2_kernels.cpp hits its #error.
This PR fixes it by using IN_LIST to add each kernel source according to
the spec that was actually detected.
* opencl: clamp the q4_K decode GEMV's fetch row on a padded x-grid
* opencl: enforce the tiling contract of the image KQ/KQV GEMMs
* opencl: decide the image KQ/KQV split at the dispatch, not from strides
* cuda : fuse MoE weighted reduction (mul + view + add)
The MoE combine tail currently writes weighted expert outputs to
global memory before reducing them. That intermediate global-memory
traffic is the main cost. The production baseline generally runs two
physical fused kernels; this path runs one.
This change matches the full expert-weighting plus ordered-reduction
subgraph and replaces it with one weighted-reduction kernel.
Supported graphs:
- unscaled: experts * router_weights
- scaled: (experts * expert_scale) * router_weights
k = 2..15 is handled by one runtime-k kernel.
Matching is structural: op sequence, shapes, strides, expert views,
and the left-to-right ADD chain. The fused kernel keeps that same
reduction order. Results are not claimed bit-identical; CUDA FP32
contraction can change rounding slightly.
Allocator integration uses add_alloc_dep from the graph-optimizer
API so experts, router weights, and optional expert scales stay live
until the fused destination is written. Memory ranges are rechecked
before the fused kernel runs.
Unrecognized or unsafe graphs are left alone and keep the existing
per-op path. Set GGML_CUDA_MOE_WEIGHTED_REDUCTION=0 to disable the
fusion.
test-backend-ops covers scaled/unscaled, aligned/unaligned, and
representative values across k=2..15, plus a k=16 case that must
stay on the per-op path.
* Pruned the test matrix from 15 to 6
* Addressed the aman and olivers review comments
get_prev_tokens() rebuilt a (seq, pos) -> token hash map on every
ubatch by walking all used cells, while llama_kv_cells already keeps
an ordered index of the positions of each sequence in seq_pos, updated
on every cell mutation to serve seq_pos_min() and seq_pos_max().
The index now stores (pos, cell) pairs in a std::set instead of a
position -> count map, so a repeated position (cache reuse via rm + add,
vision inputs with shared positions) yields distinct entries and the
removal of a cell erases its own pair. The new seq_pos_tok_le() returns
the token of the cell at the largest position <= p in logarithmic time,
which is exactly what the old window lookup and its M-RoPE gap fallback
computed together.
get_prev_tokens() shrinks to a direct lookup per (token, offset) and
for_each_token_in() goes away with its only caller. The kv-cache keeps
no n-gram logic of its own.
Measured on Qwen3.8-Flash-Next UD-Q4_K_XL at 71k context, alternating
two binaries with the first run discarded: tg 69.3 -> 72.7 t/s (+4.9%),
pp unchanged at ~2720 t/s, greedy output identical, needle retrieved.
* qwen4exp: follow up fixes
* -kvu NaN collapse fix
Assisted-by: Claude
* indexer cache ext.x/ext.y restore fix
Assisted-by: Claude
* kv-cells: rename seq_set to seq_get_all
seq_get is already taken by the single-id getter, so the suggested name
cannot be overloaded on return type alone.
Assisted-by: Claude
* memory-hybrid-idx: implement set_input_qsa on the memory class
The context held the whole implementation, where the pattern elsewhere is a
thin context forwarding to the memory class, as llama_kv_cache_context does
for set_input_kq_mask. The body reads no context state, so it moves unchanged
and the context keeps a forwarder.
Also shortens the seq_get_all comment as suggested.
* tests: check that a sequence state survives a save/restore round-trip
Saves seq 0, erases it, restores the blob and saves again, requiring the two
blobs to match. Compares blobs rather than generated text, which cannot see a
field dropped on the way back in.
Note this passes on master for qwen4exp, so it does not demonstrate the
ext.x/ext.y drop this PR fixes; reaching that needs 2D mrope content.
* tests: give the synthetic qwen4exp a PLE so the state test bites
has_cell_ext() is n_pos_per_embd() > 1 || ple_n_heads > 0, and the indexer
cache sets rope_type = NONE, so without a PLE it serializes no cell ext at
all and the round-trip test cannot see a dropped ext.x/ext.y. With one,
removing the ext_set restore in state_read_meta fails the test: 198 of
335692 bytes differ, first at offset 282092.
Loading such a model needed two fixes:
- the row count of per_layer_token_embd came from require_weight(), which a
model synthesised from metadata alone has no file to answer. Derive it
from the head ranges and prefer the file's padded count where there is one.
- the PLE conv history is a row of the recurrent cache, so a PLE on a full
attention layer dereferenced a null p_l. Reject it at load time instead.
The meta mirror is skipped for qwen4exp. It returned NaN logits before this
fixture carried a PLE, which the nmse check passes since a NaN comparison is
false, and aborts with one. -sm tensor on real devices works.
Assisted-by: Claude
* llama: disable -sm tensor for qwen4exp
test-llama-archs skipped the tensor split for this arch from inside the
test, so the arch still advertised support it does not have. Declare it in
llm_arch_supports_sm_tensor instead and drop the test-side exception; the
existing llm_arch_supports_sm_tensor branch then does the skipping.
Assisted-by: Claude
* metal : request Metal 4.0 language version for the tensor API
* metal : load the tensor API kernels from a separate metallib
* tests : add external-metallib tensor API regression test
* metal : fix metallib build order for the tensor API kernels
MTP speculative decoding needs the target state to move back by the
number of rejected draft tokens. Without rollback support the context
is classified as SEQ_RM_TYPE_FULL and the server serializes the whole
recurrent state to host memory on every round, which costs more than
the drafting saves.
The recurrent cache already holds n_rs_seq + 1 snapshot planes and the
delta net writes its SSM state into them, but build_conv_state_at wrote
a single plane, so a rollback restored a convolution history that was
never captured. It now writes one snapshot per slot, each ending one
token earlier, for the delta net QKV convolution and for the PLE
convolution alike.
Measured on Qwen3.8-Flash-Next UD-Q4_K_XL with the standalone MTP
draft, n-max 3 and a single slot: decoding reaches 183 tok/s on code
and 144 tok/s on prose. The same branch before this change, where the
server falls back to checkpointing the state to host memory, reaches
123 and 83 tok/s, for 108 tok/s without a draft.
* qwen4exp: sum the indexer heads by slices
The head reduction went through a transpose and a sum_rows over ne[1],
which left sum_rows with ne0 = 4, one block per row for a four element
reduction, and the transpose copied the whole block by token surface
twice on the way in.
The heads are adjacent on ne[1], so each one is a strided view and the
sum is a short chain of adds.
RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, 55k context, warm
runs on top of #28011:
prompt processing 2170 -> 2366 t/s
Generation is unaffected. The removed work scales with n_blocks by
n_tokens, so the gain grows with context and with ubatch size.
* qwen4exp: drop the redundant cont on the indexer query
rope returns a freshly allocated, contiguous tensor, so the reshape that
feeds the matmul does not need a copy. ggml_reshape_3d asserts
contiguity, so a layout that would need the cont cannot slip through
silently.
Greedy output is unchanged token for token.
Address review from @ggerganov
* Batched gemm for grid IQ quants
Style updates and a bit more performance
Clean up comments
Move code around
Vectorize IQ panel decode, lower threshold for speedup
IQ panel: single-source gather layout, gate bias, vectorize interleave
Add ggml_gemm_iqp_8x8_q8_K_p4 kernel, remove gather buffer
Move IQ panel code out of repack into iqp.cpp, clean up comments
Another comment sweep
* Add myself as iqp.* codeownder
* Remove ggml_cpu_iqp_scratch_offset and ggml_cpu_iqp_src1_conv_size
* Renaming and moving
* The other half of renaming and moving
* Move macros and ggml_cpu_iqp_mul_mat_id_min_batch definition
* Update ggml/src/ggml-cpu/iqp.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Add iqp_rows work buffer
* Revert "Add iqp_rows work buffer"
This reverts commit 425542991eee1b01fa3844bf87fc4f205ddbfccb.
* Add NUMA fallback
* Add 10 row batch tests for IQP coverage on all grid IQ types
* Swap assert for return false in support check
* Move IQP mul_mat_id test
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* kv cache : batch state restore scatter reads per contiguous run
When restoring state into non-contiguous destination cells (e.g. a
prompt-cache snapshot into a fragmented ring), state_read_data issued
one small copy per KV cell - ~1.4M copies of a few KiB each for a
40k+ token restore, taking 25-63 s on the CUDA backend.
The snapshot stores cell rows in cell order, so a maximal run of
consecutive destination indices maps to one contiguous block and can
be restored with a single copy. Precompute the runs once and use them
in all three scatter loops (K, V, transposed V). Byte-identical.
The on-device reader copies with a byte cursor when the read and
write chunking differs, so the batched reads are safe for it as well.
Batching makes equal tensor counts with a different split reachable
(save ranges [2,1] vs restore runs [1,2]); the next commit teaches the
reader's 1:1 path to fall back to the byte cursor in that case.
Verified in a production setup: 1,363,616 copies / 25-63 s -> 224
copies / 221-424 ms for the same restores (42,603 cells, 4 runs).
Assisted-by: Claude Code (unsloth/qwen3.8-27b)
* context : fall back to the byte cursor when read and write chunking differ
the on-device reader copies saved state back with a 1:1 copy by tensor
index whenever the write and read sides recorded the same number of
tensors, guarded by a per-tensor size assert.
equal tensor counts do not imply equal chunking: a state restore may
batch its reads per contiguous run of destination cells while the save
used per-range reads, so both sides can record two tensors that split
the same data differently, and the assert aborts in all builds.
compare the per-tensor sizes and only take the 1:1 path when the
chunking actually matches, otherwise fall through to the existing
byte-cursor copy. both sides enumerate the same logical data in the
same order, so the cursor copy is well-defined across tensor
boundaries.
Assisted-by: Claude Code (unsloth/qwen3.8-27b)
* tests : cover state restore scatter reads on host and on-device paths
decode the same prefix on two sequences, interleaving the seq 0 cells
between the seq 1 cells, so the seq 1 cells are isolated from each
other in the kv cache (three cells, two saved ranges). save the seq 1
state, free the interleaved seq 0 cells, and restore: the destination
is then non-contiguous (two runs), and the restore-side chunking has
the same tensor count as the save-side with a different split, so the
scatter path is batched per contiguous run and the on-device reader's
byte-cursor fallback is exercised.
the restored state is saved again on the host and compared byte for
byte with the first save: the blob is serialized in sequence cell
order, so the two saves are identical if and only if the scatter
restore wrote exactly the same KV content. this documents the
byte-identical guarantee of the run-batched scatter reads.
one test per io backend: the host (CPU) path and the on-device path.
Assisted-by: Claude Code (unsloth/qwen3.8-27b)
* ui: copy the displayed text of grouped agentic responses
Agentic sessions render as a single entry anchored on the first
assistant turn, whose content is typically just the first tool call,
so the copy button wrote an empty string to the clipboard. Derive the
text sections of the whole session and copy them joined, matching the
visible response. Plain messages keep the previous behavior.
* const
* dflash : fuse the encoder into the KV injection decode
The encoder is a single fc + norm, but running it as a separate
llama_encode forced a device-to-host round trip of its output before the
injection decode could re-upload it, plus a second graph build per
round. Fold the encoder into the decoder's embd branch and feed the
target features directly to one llama_decode.
Assisted-by: Claude Fable
* nit
* Apply batched suggestions from code review
Co-authored-by: Ruixiang Wang <wangruixiang07@outlook.com>
* Fix missing references from renaming
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Ruixiang Wang <wangruixiang07@outlook.com>
* vulkan: RDNA3 static mat-vec rows above four columns
On RDNA3 above four columns a static 4 rows for all types benches faster than
the default.
* vulkan: RDNA3 static mat-vec-id rows
mul_mat_vec_id has no column dimension to switch on. On my Strix Halo machine,
a static 4 is faster here than the defaults across types and batch sizes.
* rpc: avoid serializing buffers from other servers
Only include remote buffer pointers when the buffer belongs to the RPC dispatcher receiving the graph. Add a two-server regression test for cross-server tensor serialization.
Assisted-by: Codex
* cont : add ref
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
for_each_token_in tested all LLAMA_MAX_SEQ sequences for every used cell,
while a cell almost always belongs to one. The scan now stops once the
cell's own sequences have been seen. Same visit order, same callback
arguments, so behaviour is unchanged.
get_prev_tokens is the only caller, so this affects the n-gram path.
RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, warm runs:
55k context generation 56.3 -> 74.3 t/s
132k context generation 33.6 -> 50.9 t/s
Prompt processing is unchanged, the scan is amortised over the ubatch
there. The gain follows the number of used cells, so it grows with
context and is invisible on short prompts.
The optimized path grouped warp lanes by token and required
warp_size % n_expert_used == 0, with a single hardcoded exception
padding 6 up to 8. Every other count fell back to the generic path,
which walks the tokens one at a time with a warp reduction per token,
for each of the n_expert blocks.
The lane group only has to divide the warp, and the loop body already
guards the padded lanes with iex < n_expert_used, so the padding
generalizes to the next power of two. The 6 -> 8 case and every count
already dispatched keep the exact same padding as before.
n_expert_used = 10 now reaches the fast path. Measured on
Qwen3.8-Flash-Next (512 experts, 10 used) at 55k context on an
RTX PRO 6000, warm runs with the first one discarded:
prompt processing 2334 -> 2600 t/s
Token generation is unaffected, since a single token leaves nothing to
walk. Other expert counts reach the fast path by adding their case to
the dispatch.
- DFlash2 NVFP4 draft models produced almost no accepted speculative
tokens because the Q, K, V, and output projection scales were not
passed to the corresponding graph operations.
Rename the --tensor-read-lazy CLI argument to --lazy-mode, to match the
internal lazy_mode parameter, and add a -lzm shorthand. Sync the READMEs.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
some backends (Metal, SYCL, WebGPU) require additional memory for
fleeting data for certain ops, which is reflected in their
get_alloc_size implementations.
add ggml_backend_op_alloc_size_may_expand() to the backend utils,
listing these ops, and assert in ggml_backend_buft_get_alloc_size
that a backend expanding the alloc size of a compute op only does so
for ops listed in the helper.
use the helper in the RPC backend to decide whether to query the
remote server for the actual alloc size, instead of a hardcoded list.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : fail closed on mul_mat shapes with missing F16 kernels
* metal : abort on nil pipeline in encoder_set_pipeline
* metal : address review comments
* metal : share mul_mat mm dispatch with supports_op
* opencl: default the Adreno xmem F16xF32 GEMM on for X2E
kernel_mul_mm_f16_f32_l4_lm is the slowest matmul this backend has on Adreno: on
the X2-90 it runs the gpt-oss-20b attention projections at roughly a quarter of
what the tuned dense q4_0 GEMM reaches on the same device. That matters for any
model whose non-expert weights stay f16 -- the stock gpt-oss-20b release is
exactly that, and its prefill spends 40.8% of GPU time in that one kernel. The
xmem route already existed but was left opt-in, so nobody hit it.
Worth about 25% prefill on gpt-oss-20b on an Adreno X2-90. Gated to X2E: the
Adreno 840 measures neutral. Decode is untouched -- the dispatch gate needs
N >= 16. It is worth nothing on the q8attn variant, whose attention weights
already take the dp4a dense GEMM.
The env var was presence-tested before, so =0 previously enabled it; it is now
atoi()'d. MUL_MAT 963 OK / 0 FAIL on both arms.
* opencl: bypass the tiled f32 GEMM on the Adreno A7X
The A7X (E031.41) compiler executes kernel_mul_mm_f32_f32_l4_lm at roughly a
tenth of what the same silicon reaches in its own f16 and q4_K kernels. It
allocates 488 B/WI of private memory against 304 for the same source on the
following generation, i.e. the older register allocator spills in the K-loop.
Models with per-layer F32 projection pairs kept F32 by quantization policy land
on this kernel twice per layer, and it dominates their prefill on that part.
Route batched f32xf32 (ne11 > 8) around the tiled path on the A7X and let it
fall through to the per-row f32 kernel, which that compiler handles fine; small
batches keep the tiled path. Weights stay GPU-resident, so decode placement is
untouched -- declining the op in supports_op instead was measured first and
rejected, because the per-layer CPU round-trips cost more decode than the
prefill it gained.
Worth about 9% prefill on gemma-3n-E4B on an Adreno 740, with MUL_MAT counts
identical on and off. No other generation is affected. Override with
GGML_OPENCL_A7X_F32_LM_BYPASS=0.
* opencl: enable xmem GEMM for adreno by default
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
improve the --fit algorithm to take into account the actual peak
required VRAM for a given context size on a SYCL backend.
This includes both properly accounting for how much VRAM is required
when the allocated context is fully used (which makes the reported
context drop below what it did before, but stop it OOMing) as well
as preventing some overly-conservative calculations which meant too much
VRAM was being reserved.
Tested on a Arc b70 with unsloth's qwen3.8 (Q4_K_XL), able to get 262144 context,
fully usable, with q8_0 KV and MTP and 4k ubatch size using --fit-target 1
The N padding is needed for mul_mat, but not mul_mat_id. For mul_mat_id,
we indirect the row index through a shared memory lookup table which avoids
any OOB row coordinate. But that callback doesn't bounds check K, so we
actually need K padding instead.
* port setup-build.ps1 to setup_sdk.py, to facilitate installation of Hexagon and OpenCL SDKs on Windows
* rename setup_sdk.py -> setup-sdk.py
* flake8 fix: print() -> logger.info()
---------
Co-authored-by: Kristopher Urquhart <kurquhar@qti.qualcom.com>
* vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize
is_src_of doesn't treat two views of one tensor as dependent, so the optimizer reorders nodes across aliased reads and writes.
Result: silently wrong tokens under greedy decoding, different output on every server start, and invalid speculative-decoding acceptance, with nothing logged.
Hits Qwen3.8's recurrent state (and any model with view-aliased state) on AMD and NVIDIA Vulkan. CUDA is clean.
Compare view_src bases on both sides.
Fixes#27805
* vulkan: don't treat view/no-op nodes as aliasing dependencies
Nodes whose op is NONE, RESHAPE, TRANSPOSE, VIEW or PERMUTE execute nothing, so aliasing through them is not a real dependency. The previous base comparison matched them anyway, which only costs the optimizer reordering freedom.
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* vulkan: make the lambda parameter const and capture is_empty in is_src_of
Code will not compile without these changes.
is_src_of has an empty capture list, so is_empty was not visible inside it, and is_empty took a non-const pointer, while is_src_of receives const ones. Other call sites pass non-const pointers, which still convert as usual.
---------
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* ggml : fix conv_transpose_2d for multiple batches
ggml_compute_forward_conv_transpose_2d_impl only computed the first
batch (ne[3] of the destination); every batch after the first was left
as zero. Both the src1 permutation and the main compute loop now iterate
over the batch dimension, and the work buffer size in ggml_graph_plan is
scaled by the src1 batch count so the extra permuted batches fit. A
multi-batch test case is added to test-backend-ops.
Fixesggml-org/ggml#1448
* metal : fix conv_transpose_2d for multiple batches
The kernel only computed batch 0 of the input (src1->ne[3]); every
output batch after the first was left as zero, so multi-batch
conv_transpose_2d results diverged from the CPU reference.
The grid now covers all batches (OW x OH x OC x N), the kernel decodes
the batch from the grid z coordinate and offsets both the input and
destination indices accordingly. nb3 is passed in the kernel args.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vulkan: add hoisting support for row IDs and expert count in shaders
* use hoisted row ids in coopmat2
* vulkan: address review feedback on count_experts
- use vk_op_count_experts_push_constants instead of a raw uint vector
- apply the fastdiv trick to the ne00 div/mod in count_experts
- compute the per-expert offsets with subgroupExclusiveAdd when the
device supports it, keeping the serial path as fallback
- document the data_d layout and the hoisted_row_id_words bound
- drop a leftover debug print in ggml_vk_matmul_id
* vulkan: use init_pushconst_fastdiv for count_experts push constants
* vulkan: refine comments for row ID hoisting and data layout in count_experts shader
* Whitespace
---------
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
This adds fa_vec_tuned_table records for Apple M4 to ggml-metal-tuning.cpp.
Includes F16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0. (M4, 10 GPU Cores)
Co-authored-by: Strongtut <8432058+Strongtut@users.noreply.github.com>
* OpenVINO Backend: Fuse IM2COL + MatMul convolution into OpenVINO convolution
* ci:ggml-ov: Skip recurrent state rollback tests
* ci:ggml-ov: Skip recurrent state rollback tests
* Update OPENVINO.md
* ggml-openvino : add env-var gated op support debugging
* Fix ggml_rope_set_offset case
* OpenVINO backend: Support Whisper.cpp
* Fix code style
* openvino : enable qwen35 on NPU
Static shapes:
- get_graph_input_shape() left the s_copy / s_copy-leaf inputs dynamic
([1,1,1,-1]) even in static mode, which propagated a dynamic slot dim through
GET_ROWS into the conv/GDN state, the state reshapes and the GDN output.
- With -np 1 the s_copy defrag remainder gathers zero rows; short-circuit that
CPY to the untouched cache instead of emitting a degenerate Slice/Concat, and
skip binding its zero-byte ggml tensor as an output (the dynamic path already
did the latter, the static path wrote the full cache over a 0-byte buffer).
Token-count independence:
- In static mode the compiled model's token count is the prefill chunk size or
1, not the captured cgraph's. Offsets derived from the captured count were
therefore wrong. Anchor the GDN state slice at the end of the packed
[attn | state] output and drop the rs_src_begin runtime inputs, and make
VIEWs over the GDN output / conv_input pass through so the consumer does the
slicing.
- CONT could not identify its token axis when the graph was captured with a
single token (every trailing dim has the same stride and size 1) and baked
the captured shape into the prefill model.
Chunked prefill:
- The last chunk is padded with fabricated tokens. Attention masks them, but
the recurrent path folded them into cache_r/cache_s permanently. Add a
chunk_valid_len runtime input, use it to zero g and beta for padded steps
(making the recurrence an exact identity) and to end the conv snapshot window
at the last valid token, and disable the recurrent-cache reset after the
first chunk so earlier chunks are not wiped.
- get_is_prefill() and the chunk loop bound read inp_pos->ne[0] directly, but
IMROPE stacks 4 position planes, so every decode step was run through the
padded prefill model and the loop ran extra out-of-bounds chunks.
cache_rs_reset_idx/len now stay runtime Parameters in static mode, since
can_reuse_statically() does not invalidate the cached model on ComputeParams
changes. Add GGML_OPENVINO_FORCE_STATIC to exercise the static path on CPU.
* Update to OpenVINO 2026.3.1
* ggml-openvino: forward NPU compilation mode parameters
Add GGML_OPENVINO_NPU_COMPILE_CONFIG to the backend's cached environment so callers can configure the NPU compiler without using the generic property escape hatch.
When the value is non-empty, pass it to OpenVINO as NPU_COMPILATION_MODE_PARAMS. This enables settings such as optimization-level=3 for NPU compilation while preserving the existing behavior when the variable is unset and leaving CPU and GPU configuration unchanged.
Document the variable, its NPU-only scope, and the optimization-level=3 example in the OpenVINO backend runtime configuration table.
* ggml-openvino : support RELU, POOL_2D, QUICK_GEGLU, and ROLL ops
* reorder op table
* exclude GPU/NPU failing POOL_2D case
* move op type detection to compute_op_case
* Relax rope supported cases
* Fix pool case
* Update openvino doc, gpu driver in ov docker
* openvino: remove unused static remote context branch
* openvino: parallelize static model build
* Apply editorconfig
---------
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
Measured at a live KV length of 34816 (32768 depth plus one 2048 ubatch),
on Qwen3.8 27B Q4_K_S:
per tensor 4 * 34816 * 256 * 2 B = 71.3 MB
staged per call K and V, so 2x = 142.6 MB
traffic per call read once, write once = 285.2 MB
traffic per ubatch 285.2 MB * 16 calls = 4.56 GB
One ubatch is one ggml_cgraph submission (llama_context::process_ubatch ->
graph_compute), so that 4.56 GB is the cost of a single 2048-token prefill
chunk, and it scales with the live KV length: the first ubatch of the same run,
at seq = 2048, moves 0.27 GB.
Reproduce the two measured inputs with:
GGML_SCHED_DEBUG=2 llama-bench -m MODEL -p 8 -n 0 -r 1 -ngl 0 \
-fa on -ctk f16 -ctv f16 -v > nd.txt 2>&1
grep -E 'n_layer|n_head_kv|n_embd_head_k' nd.txt
awk '/node # 0 /{g++} g==1 && /\(FLASH_ATTN\)/{n++} END{print n+0}' nd.txt
* metal : add fa-vec tunings for M5
This is a followup contribution to efeda76b948f59ee52ea20db640bc4cf3dfe8ac1 as requested in https://github.com/ggml-org/llama.cpp/discussions/27668 to add support for additional Apple GPUs. I generated this output using the provided instructions:
```sh
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --target ggml-metal-tuning -j
./build/bin/ggml-metal-tuning fa-vec --dtype f16,q8_0 > fa_vec_rows.txt 2> fa_vec_sweep.log
```
This ran on a machine with Apple M5.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : add fa-vec tunings for M5 Pro
This adds fa_vec_tuned_table records for Apple M5 Pro to ggml-metal-tuning.cpp.
Contributed by SerayaEryn in https://github.com/ggml-org/llama.cpp/discussions/27668#discussioncomment-18157544 (F16, Q4_0, Q8_0; M5 Pro, 20 GPU cores).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : add fa-vec tunings for M3 Max
This adds fa_vec_tuned_table records for Apple M3 Max to ggml-metal-tuning.cpp.
Contributed by TeeAaTeeUu in https://github.com/ggml-org/llama.cpp/discussions/27668#discussioncomment-18175220 (F16, Q8_0; M3 Max, MacBook Pro 64GB, low power mode).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : whitespaces
This is a followup contribution to efeda76b948f59ee52ea20db640bc4cf3dfe8ac1 as requested in https://github.com/ggml-org/llama.cpp/discussions/27668 to add support for additional Apple GPUs. I generated this output using the provided instructions:
```sh
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --target ggml-metal-tuning -j
./build/bin/ggml-metal-tuning fa-vec --dtype f16,q8_0 > fa_vec_rows.txt 2> fa_vec_sweep.log
```
This ran on a MacBook Pro (14-inch, Nov 2024) with Apple M4 Pro. The `ggml-metal-tuning` command completed successfully in 1h 13m 1s with no other notable load on the system.
* tests : run test-save-load-state across all architectures
test-save-load-state previously only ran in ctest against a single
downloaded model (tinyllamas/stories15M), i.e. only the llama arch.
Add a --models DIR mode to test-save-load-state that runs the full
save/load suite over every *.gguf in a directory, reporting a
per-model PASS/FAIL and exiting non-zero if any model fails, and wire
a ctest to run it over all architectures using the existing
generate-models fixture (test-llama-archs). The single-model -m mode
is preserved (still used by ci/run.sh).
Also bump the dummy-model training context in test-llama-archs from
128 to 256 so that the per-sequence context (which is padded up to a
multiple of 256) no longer exceeds n_ctx_train and emits the
"possible training context overflow" warning.
The test is expected to fail until the affected arches are fixed:
deepseek4 (host seq-copy), gemma2/gpt-oss/lfm2 (device seq-copy),
minimax-01 (state load). It aborts at the first arch that crashes.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* tests : match dummy DSA indexer to fused Lightning Indexer kernel
The dummy DSA indexer (deepseek32, glm-dsa, ...) used key_length=64 and head_count=1, so the fused Lightning Indexer op's q tensor was shaped [64, 1, ...]. The Metal fused kernel is fixed to DK=128, NH=64, so it rejected the op and the scheduler fell back to CPU, emitting a 'layer assigned to MTL but Lightning Indexer on CPU' warning. Bump key_length to 128 and the DSA head_count to 64 so the fused op runs on the GPU.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* tests : add --help and document -o in test-llama-archs
Add a --help/-h flag to test-llama-archs and list the existing -o/--out option in the usage text, which was previously missing.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* tests : use 64 indexer heads for deepseek4
deepseek4's indexer head count was set to n_head (8), which does not match the fused Lightning Indexer kernel's fixed NH=64, so the fused op fell back to the CPU backend and emitted a device-mismatch warning. Give it the same fixed 64 as the other indexer archs by dropping it from the n_head ternary (only minimax-m3 keeps n_head, since it does not use the fused Lightning Indexer op).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* tests : fix dsv4 save-load n_stream mismatch
The dsv4 KV cache keeps per-sequence KV/state streams even in unified mode, so its n_stream equals n_seq_max. The test saved the state in the baseline with n_seq_max=1 but loaded it in the seq-copy tests with n_seq_max=2, so state_read threw an n_stream mismatch. Use n_seq_max=2 in the baseline and state-load tests so the save and load agree.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* context : relax on-device seq-copy chunk alignment
The on-device state seq copy (llama_state_seq_set_data with LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) copied the write-side cpy tensors to the read-side targets 1:1 by index, requiring the writer and reader to emit the same number of chunks in the same order with the same per-chunk sizes. state_write_data chunks per cell-range while state_read_data chunks contiguous-or-per-cell, so the counts diverged for non-contiguous sources (dsv4, SWA) and the copy aborted with "memory buffer mismatch".
All state writers and readers enumerate the same logical data in the same order, differing only in chunking. Copy the flat write-side data into the read-side targets with a byte cursor that walks both tensor lists across their boundaries, so the chunking no longer needs to match. Keep the total-size guard; drop the n_tensors equality check.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* model : fix dangling hparams ref in minimax-01 LA graph input
llm_graph_input_la stored const llama_hparams & hparams, bound to the llm_graph_params temporary in llama_context::process_ubatch. The input object outlives that temporary (it is kept in llm_graph_result::inputs for graph reuse), so set_input() read destroyed stack memory on every graph reuse - test-save-load-state crashed for minimax-01 when the stack region was overwritten (n_layer_all read as 0, abort in llama_hparams::n_head). Store a copy like every other graph input class.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* context : handle "worst case" graph and add TODO
Add HVX-accelerated implementations for GGML_OP_LOG and
GGML_UNARY_OP_ABS on the HTP backend.
- Register HTP_OP_UNARY_ABS and HTP_OP_UNARY_LOG in op_remap_to_htp()
- Add ABS and LOG to ggml_backend_hexagon_device_supports_op()
- Implement hvx_abs_f32_aa() in hvx-arith.h using hvx_vec_abs_f32()
- Implement hvx_log_f32_aa() in hvx-log.h using hvx_vec_log_f32()
- Add abs_f32() and log_f32() row-wise dispatch in unary-ops.c
- Define tiled and non-tiled task functions via DEFINE_UNARY_TASK and
DEFINE_UNARY_TILED_TASK macros
- Route HTP_OP_UNARY_ABS and HTP_OP_UNARY_LOG through execute_op()
in main.c
* Add ctx-per-slot argument for unifid KV cache
* Swap out ctx fractions for ctx pool slots
* Formatting cleanup
* Remove ctx-pool-slots, make ctx-per-slot an int
* refactor it
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* gguf: add qwen4exp (Qwen3.8-Flash-Next) arch and converter
Adds the GGUF-side plumbing for HF model_type qwen4_exp:
- MODEL_ARCH.QWEN4EXP plus tensors for the low-rank hyper-connection
variant (hc_*_norm/down/up/inject) and the PLE n-gram hash embeddings.
The DeepSeek-V4 hc_*_fn/base/scale tensors are a different
parameterisation, so these are separate entries rather than reuse.
- Reuses the existing indexer, per_layer_token_embd, SSM and
compress_ratios keys unchanged.
- conversion/qwen4exp.py inherits the Qwen3.5 linear-attention V-head
reorder and interleaved mrope, concatenates the 128 PLE embedding
shards, and splits index_qk_proj into separate indexer q/k tensors.
The PLE hash multipliers reach ~2.4e13. prepare_tensors() casts every
non-float dtype to float32 before modify_tensors() runs, and GGUF array
writes infer INT32 from Python ints, so both paths are bypassed: the
constants are read from the pre-cast lazy tensors and written as
explicit UINT64 arrays.
Additive only; no existing arch changes behaviour.
* llama: load qwen4exp (Qwen3.8-Flash-Next) hparams and tensors
Adds LLM_ARCH_QWEN4EXP with its hparams and tensor loading. The graph
comes in the next commit; this makes the model load and report correct
metadata.
- hyper-connections set n_embd_out_impl = hc_count * n_embd, so the
residual stream is 4x wide and there is no output_norm: the final
mixer's hc_norm is the last norm in the model.
- registered as hybrid and given the same recurrent/attention memory
filters as Qwen3-Next and Qwen3.5.
- reuses the existing indexer, per_layer_token_embd, SSM and
compress_ratios keys as-is.
- the PLE table row count is read back from the file rather than
recomputing the vocab padding rule.
llama-model-loader gains UINT64 array support. That branch previously
threw, so no existing caller changes behaviour; it is needed because the
PLE hash multipliers do not fit in int32.
* qwen4exp: shorten comments
* llama: qwen4exp text graph with hyper-connections, GDN and MoE
Implements the decode graph for Qwen3.8-Flash-Next: the hyper-connection
residual stream, gated delta net layers, the MoE block with its gated shared
expert, and dense full attention. The QSA indexer and the PLE n-gram embedding
are not wired up yet and land in later commits.
Hyper-connections are implemented here rather than shared with deepseek4.cpp.
The two formulations agree on the [n_embd, hc, n_tokens] layout and little
else: DeepSeek-V4 mixes with a full-rank projection and Sinkhorn-normalises
it, whereas this model uses a low-rank down/silu/up sigmoid gate and collapses
by a plain mean. Only the ~10 line stream mean is genuinely common, so sharing
would mean touching DSV4's hot path and its three fused CUDA ops to reuse very
little. What is reused is the substantive part: the LLM_KV_HYPER_CONNECTION_*
keys, the n_embd_out_impl wide-residual support already in the loader, and the
layout convention.
Also allows a checkpoint to carry no PLE layers at all, which makes it
possible to bring the graph up and validate it in stages.
Validated against vLLM, the only working reference implementation. On a
scaled-down model with an init scale large enough to give non-uniform logits,
agreement with vLLM sits at the numerical noise floor: llama.cpp f32 against
its own bf16 gives 84.3% top-1 agreement over 255 positions, and this graph
against vLLM gives 85.1%. The comparison was calibrated by seeding three
deliberate bugs (silu instead of sigmoid on the delta net gate, dropping the
1/hc scale in the mix, dropping the 2x in the combine); each drops top-1 to
between 0% and 11%, an order of magnitude below the floor.
* llama: qwen4exp PLE n-gram hash embedding
Adds the per-layer embedding: a custom I32 graph input hashes each token with
its ngram_size-1 predecessors host-side and the result is a plain row gather
over the shared table, the same shape gemma3n's per-layer embedding uses. The
hash has to run on the host because the splitmix64-derived multipliers reach
2^45, so the products need 64-bit integers and an xor, neither of which ggml
has.
Predecessors that fall outside the ubatch come from a small per-sequence
history on the model, mirroring the per-request ngram_context the reference
carries. It is only trusted when contiguous with the incoming position, so a
fresh prompt or a rewound cache falls back to EOS padding rather than hashing
against stale tokens.
The depthwise conv is written out as a sum of shifted, per-channel-scaled
copies rather than through ggml_conv_1d_dw, which carries a correctness
warning upstream.
Verified two ways. The row indices match a transcription of the reference's
tensor formulation exactly, 1024 of 1024 rows, including sequences with EOS
tokens sprinkled through them to exercise the segment reset. Separately, with
PLE placed on layer 0 so its input is just the token embedding, ple_embd and
ple_gated_value match a PyTorch computation from the same checkpoint to every
printed digit.
End to end over 1023 scored positions the port sits the same distance from
vLLM with PLE as without it, 6.3 points of top-1 against 6.0, so PLE costs no
accuracy relative to the rest of the model. That common offset is vLLM's bf16
activations, which cannot be removed: its QSA kernel refuses float32.
Two bugs found along the way, both caught by the row-index check. The history
was read and updated in the same pass, so a token early in a ubatch could pick
up an earlier token of that same ubatch as prior context; it is now snapshotted
first. And an EOS token was cutting its own context, where the reference takes
the last EOS strictly before the position, so a boundary only hides tokens from
the positions after it.
Known gap: the conv carries no state across ubatches, so it is exact only for a
prefill that starts at position 0. Chunked prefill and decode need the conv
state wired into the recurrent memory, and the conv branch itself is still
numerically unverified because the fixture zeroes its weights.
* llama: carry the qwen4exp PLE conv state across ubatches
The PLE depthwise conv was zero-padding on the left, which is only right for a
prefill that starts at position 0. Decode and chunked prefill saw a truncated
history for the first (kernel-1)*ngram_size positions of every ubatch.
The PLE module sits on a layer that is also a delta-net layer, so both need a
conv history in the same recurrent row. Rather than plumb a per-layer state
size through build_rs and build_conv_state, the row is widened once and each
convolution addresses its own slice through a local helper. n_embd_r() gains
the extra span, which is zero for every other architecture because it is
derived from ple_n_heads.
Verified by feeding the same 1024 token sequence in chunks instead of one
shot: at 64 tokens per decode the logits are bit-identical to the single-shot
run, 1023 of 1023 top-1 and a maximum logprob deviation of exactly zero. At
one token per decode they differ slightly, but the no-PLE model differs more
under the same test (94.6% against 97.1%), so that is the usual gemv-versus-
gemm accumulation difference and not the state.
The conv branch is also no longer unverified. With non-zero conv weights the
port sits 6.3 points of top-1 below the numerical floor, the same distance as
with the weights zeroed and as the model with no PLE at all, so the branch
adds no error of its own.
test-llama-archs passes every existing architecture at 0.00e+00, including the
delta-net models that share this code path.
* llama: fix the qwen4exp PLE conv state and unblock test-llama-archs
build_rs writes into the state tensor in place, zeroing one row and copying the
carried-over states, so calling it twice for the same layer let the second call
clobber the first write-back. The PLE layer is also a delta-net layer, so that
is exactly what happened: both convolutions gathered the same row. They now
share a single gather per layer.
The earlier claim that the conv state was carried correctly was tested on a
fixture whose conv weights are zero, where the branch contributes nothing and
chunking matches trivially. Re-running with non-zero conv weights showed the
divergence, growing with the number of ubatch boundaries: 97.1% top-1 at one
boundary down to 90.2% at seven. With the shared gather it is bit-identical to
the single-shot run at every chunk size tried, 512, 128 and 64, with a maximum
logprob deviation of exactly zero over 1023 positions. The delta-net-only model
stays bit-identical too, so nothing regressed there.
Also derive the delta-net conv channel count the way load_arch_tensors sizes
wqkv instead of from ssm_d_inner. The two agree for this model, but n_embd_r()
only bounds the row and the convolution has to match the tensor feeding it.
test-llama-archs previously aborted on this architecture and took every later
architecture with it. qwen4exp is marked MoE-only, given the hyper-connection
keys and an ssm_d_inner consistent with its tensor derivation, and skipped for
now: the hyper-connection keys written by get_gguf_ctx are not reaching the
synthesised file, which needs a separate look. The suite completes again, 124
architectures at 0.00e+00.
* llama: optional indexer key cache in llama_memory_hybrid
Groundwork for qwen4exp's QSA sparse attention. Its indexer needs a per-token
key history for the full-attention layers, but a hybrid model cannot use
llama_kv_cache_dsa: that class derives from llama_memory_i rather than
llama_kv_cache, and llama_memory_hybrid constructs its attention cache
directly. No existing architecture pairs recurrent state with a sparse
indexer, so there was nothing to reuse wholesale.
llama_memory_hybrid therefore gains a third, optional cache, shaped the same
way llama_kv_cache_dsa shapes its lightning-indexer cache: a copy of hparams
with n_head_kv forced to 1 and n_embd_head_k_full set to indexer_head_size.
It is built only when a filter_idx callback is passed, which defaults to
nullptr, so every existing architecture gets exactly what it got before. The
per-sequence operations and the batch preparation forward to it under a null
check, matching how the DSA cache prepares its two caches over the same
ubatches.
test-llama-archs passes all 124 architectures at 0.00e+00, including the 12 in
the hybrid family that share this code. The qwen4exp fixtures are unchanged:
same logits against vLLM, and chunked evaluation still bit-identical to
single-shot.
* llama: QSA sparse attention for qwen4exp
The full-attention layers of this model do not attend to everything. An
indexer scores one mean-pooled key per block of compress_ratio tokens and
keeps a budget of the best blocks, plus the tail of tokens that do not yet
form a complete block. Below indexer_top_k + compress_ratio - 1 cached
tokens every block fits in the budget, so the result is exactly dense.
What is reused rather than rebuilt:
- the mask machinery. build_attn's DSA overload already turns a list of
token indices into a KQ mask via ggml_set_rows, so that block is lifted
out verbatim into build_attn_mask_top_k and shared with a new overload
on llm_graph_input_attn_kv. DSA's node sequence is unchanged; the new
overload exists because llama_kv_cache_dsa assumes MLA and cannot be
dropped into a hybrid model.
- the indexer key cache, which is the optional third cache added to
llama_memory_hybrid in the previous commit. It holds raw keys, because
pooling happens before the norm and the rotation.
The graph expands block scores rather than block indices: giving every
token of a block its block's score needs only a gather, where expanding
indices would need an integer multiply-add that ggml has no op for. Since
the budget is a whole number of blocks and a block's members tie exactly,
the cut still lands on a block boundary.
Everything that depends on cache layout is computed host-side in
set_input_qsa. Blocks are cuts of the position line rather than of the cell
array, so nothing assumes the cache is contiguous.
Measured on the tiny fixture against vLLM, comparing the selected token
indices directly rather than the logits:
below the budget selection identical, and 1024-token logits are
bit-identical to the pre-QSA dense path
above the budget mean jaccard 0.975
The direct index comparison is what made this correct. The reference
rectifies each head's dot product before summing over heads, which an
earlier reading of it had missed; on logits alone the resulting port looked
fine, because on a randomly initialised fixture the known-correct dense
path already disagrees with vLLM by more than the bug did. Comparing the
indices showed 0.794, and fixing the ReLU moved it to 0.975.
* llama: give the qwen4exp indexer cache the attention cache's slots
The indexer cache found its own slots, independently of the attention
cache. Both are the same size and see the same ubatches, so in a
straight-through prefill they agree, which is why every fixture and every
single-shot parity run passed. They drift once the context is being
rewritten between turns, and then the QSA top-k indices, which are applied
against the attention mask, point at the wrong cells.
The seven-turn chat test caught it on the third turn: llama-server aborted
on the assertion that the two caches report the same n_kv.
The cache is a side buffer addressed by the attention cache's cells, so it
now takes that cache's slot layout instead of computing one. Applying that
layout also marks its cells identically, so the two agree cell for cell by
construction rather than by coincidence, and the assertion can no longer
fire.
Inert where the caches already agreed: test-llama-archs green at 126 archs
and 0.00e+00, and the 4096-token tiny fixture is unchanged at max logit
delta 0.0.
* tests: record what the qwen4exp arch-test skip actually observes
The old note guessed that the hyper-connection keys never reach the file.
They do: dumping the gguf_context handed to llama_model_init_from_user
shows both among its 67 KVs, and the loader still reports one missing.
* tests: cover qwen4exp in test-llama-archs
The arch was skipped with a note guessing that the hyper-connection keys
never reached the synthesised file. They did. The suite builds a model, then
saves and reloads it, and llama_model_saver did not re-emit those keys, so
the failure was in the roundtrip leg rather than the first load. Three gaps,
all in shared code and all additive:
- add_kv_from_model wrote no hyper-connection, compress-ratio or PLE keys.
The PLE group only means anything whole, so it is written or omitted
together; the rest follow the file's existing style of writing every key
unconditionally, since an architecture that does not read one is
unaffected by a zero.
- the saver had no uint64 path at all, which the PLE hash constants need.
- add_tensors_from_model enumerates model-level tensors by hand and was
missing per_layer_tok_embd and the three final-mixer tensors.
Two smaller fixes on the qwen4exp side, both found by running the test:
- build_qsa_top_k divided by the compression ratio before asserting it was
non-zero, so a file without the key crashed instead of reporting.
- a layer with no compression ratio now falls back to dense attention,
which is what the model computes below the budget anyway. The test then
has to write a ratio to reach QSA at all, and an indexer key length no
narrower than n_rot, since the indexer ropes with the main attention's
rotary width.
Full suite: 126 archs, qwen4exp at 0.00e+00 with roundtrip OK. The tiny
fixture is unchanged, max logit delta 0.0 against the pre-QSA dense run.
* convert: stream the qwen4exp PLE table instead of concatenating it
The n-gram table arrives as 128 shards that were held in a dict and then
torch.cat-ed, so the peak was the shards plus the concatenation: around
300 GB of RSS on the real checkpoint, which rules out machines that could
otherwise convert this model.
Each shard is now written straight into a memory-mapped file at its final
row offset and dropped, so the resident set is one shard and the rest is
the page cache's problem. The temporary file sits beside the output and is
removed once the write finishes, including on failure.
Shards other than the last must be uniform for direct placement, which is
asserted rather than assumed, and a shard arriving before the stride is
known is held instead of misplaced.
Verified on the tiny fixture: the resulting GGUF is byte-identical to the
one the concatenating path produced (md5 2d274efac91ad1e9a6007efb0687e597).
* quantize: fall back to F16 for 32-block types with an odd ncols
tensor_type_fallback demotes a tensor whose ncols is not a multiple of the
target's block size, but its switch only enumerates the 256-block types. A
target that is already a 32-block type (iq4_nl, q4_0, q5_0, q8_0, ...) falls
into default: and throws, even though the function already knows how to answer
that case: the ncols check right below the switch resolves an unrepresentable
shape to F16.
Route those types into that check instead of throwing. Only paths that abort
today change, so no quantization that currently succeeds is affected.
Found on a 4-wide depthwise conv kernel. llama-quantize reported nothing but
"failed to quantize model from ...", with no tensor name and no exception text,
which made a quant recipe that had simply not pinned the tensor look like a
corrupt model. It now names the tensor and continues.
* quantize: let --tensor-type name per_layer_token_embd
per_layer_token_embd shares the TOKEN_EMBD category with token_embd.weight, so
--token-embedding-type is returned for it before any --tensor-type pattern is
consulted, and there is no way to give it a tier of its own.
That grouping is fine as a default and stays the default. It is a poor fit for
the size, though: on qwen4exp the table is 97.7 GiB of a 337.6 GiB BF16 file and
about 46% of a 4-bit one, roughly eighty times token_embd.weight, and it is
read by ggml_get_rows rather than a matmul so no imatrix ever covers it.
Allow an explicit --tensor-type pattern to name it, and only it. Nothing
changes unless such a pattern is passed, and token_embd.weight keeps the old
precedence in either case.
Measured on Qwen3.8-Flash-Next, Q4_K_M with an imatrix: the table lands at q8_0
(51.9 GiB, 113.5 GiB total) by following --token-embedding-type, and pinning it
q4_1 gives 30.5 GiB for 92.1 GiB total, 19% off the file.
* quantize: size the output buffer exactly instead of nelements * 4
The per-tensor output buffer was sized `nelements * 4`, described as an upper
bound. It is a very loose one: the output is at most 2 bytes per element
(f16/bf16) and usually well under 1.1 (q8_0 and below), so between 2x and 4x of
it is never touched. The exact size is already known here, since it is what the
quantization loop writes, what new_size sums to, and what the GGUF metadata is
asserted against a few lines later.
On a model whose largest tensor is a few GB none of this matters. On
Qwen3.8-Flash-Next it does: per_layer_token_embd is 51.2 G elements, so the
buffer was 205 GB where 54 GB is needed at q8_0 and 32 GB at q4_1.
Measured on that model, VmHWM of a live llama-quantize was 485 GB per process.
Three of them fit in 2 TB and five did not, which is what an OOM-killed quant
ladder looks like. This removes about 150 GB of that.
Byte-identical output, verified against the same binary built at the parent
commit: q4_K, q8_0, q5_K, q6_K and IQ4_XS, over BF16 and F32 sources, with and
without a PLE table present. Six cases, six matching md5s.
* qwen4exp: hash the image placeholder for multimodal batches
The PLE row indices are computed host-side from ubatch->token, and set_input
returned early when that was null. A multimodal ubatch is exactly that case:
the mtmd layer consumes the image placeholder ids and hands llama_decode
embeddings instead. The early return left the I32 index tensor uninitialised,
so ggml_get_rows indexed a 320 M row table with whatever the buffer happened to
contain, and aborted:
GGML_ASSERT(i01 >= 0 && i01 < ne01) failed
ggml_compute_forward_get_rows
mtmd_helper_decode_image_chunk -> llama_decode
Every image request crashed. Nothing caught it because the vision work had only
ever been verified by converting an mmproj, never by running one.
The reference computes the hash over input_ids, where those positions still
hold the image placeholder, so carry that id through as qwen4exp.ple.image_token_id
and hash it. The key is optional: a file converted before it existed falls back
to the PLE EOS token, which is defined and treats the image as a segment
boundary rather than crashing.
Verified end to end with llama-mtmd-cli, a Q4_K_M base and the F16 mmproj, on a
generated image with known content. The model names the red circle, the blue
square, the inverted green triangle and reads "UNSLOTH 42", each with the right
position.
* qwen4exp: support a non-unified KV cache in QSA
set_input_qsa asserted n_stream == 1, so llama-server could not serve this
model with more than one slot unless -kvu was passed. With a non-unified
cache each sequence owns its own cells, and a cell index means a different
token in each stream, so a single shared mapping is wrong.
- cell_blk, blk_cells and bias gain a stream dimension. At n_stream == 1
these collapse to the shapes they had, so the unified path is unchanged.
- Scoring is now batched over streams. ggml_mul_mat matches ne[2] on both
operands, so stream s's queries only ever meet stream s's blocks; without
this sequences would score against each other's context.
- set_input_qsa loops per stream and resolves cells through
v_cells[seq_to_stream[seq_id]], following set_input_kq_mask_impl, instead
of hardcoding v_cells[0].
- llama_kv_cache_context::get_n_stream() is added, mirroring the ns that
get_k and get_v already derive from the slot info.
build_attn_mask_top_k needed no change: it already expects
[n_top_k, n_batch, 1, n_stream], so the top-k result is reshaped to meet it.
set_input_qsa has exactly one caller, so the blast radius is qwen4exp only.
Validation, UD-Q4_K_XL on one B200:
- unified cache unchanged within noise: 1802.9/68.85 -> 1807.2/69.11 t/s at
batch 1, 2262.5/192.43 -> 2270.1/193.75 at batch 4.
- non-unified now runs at npl 1, 4, 16 where it previously aborted, and is
22% faster than the -kvu workaround at batch 16 (1205 vs 984 t/s total),
since per-stream cells avoid the cross-sequence masking a unified cache
pays for.
- no cross-stream contamination: four concurrent sequences each carrying a
distinct secret all recall their own and no other, on both cache modes.
- test-llama-archs green on qwen4exp, deepseek2, gemma3n, qwen3next, llama.
Note on testing: comparing concurrent output against solo output exactly is
not a valid check. It failed 0/4 with no bug present, and the unified-cache
control failed the same way, because batch composition changes the
floating-point reduction order and near-tied tokens flip. The contamination
test above is what the exit code gates on.
* llama: keep the qwen4exp top-k attention mask arch-local
The QSA graph needed a build_attn that attends only to the cells named by a
top_k tensor, and the first version got it by adding a llm_graph_input_attn_kv
overload to llm_graph_context and factoring the mask construction out of the
existing MLA sparse path into a shared build_attn_mask_top_k.
That put a new arch on the shared attention path and made the deepseek32 and
glm-dsa attention build depend on a helper introduced for qwen4exp. Build the
mask in src/models/qwen4exp.cpp instead and leave llama-graph.{h,cpp} exactly as
they were: the MLA path keeps its own copy of the same node sequence.
The nodes emitted are unchanged, so this is bit-identical.
* llama: hold the qwen4exp indexer cache in a new llama_memory_hybrid_idx
The indexer key cache was added by extending llama_memory_hybrid with an
optional third cache, and the host-side cell/block mapping that drives QSA was
added as set_input_qsa on llama_kv_cache. Both are shared classes that every
hybrid and every attention model goes through.
Move both into a new memory type, llama_memory_hybrid_idx, following
llama_kv_cache_msa: the indexer cache and the pos<->cell translation live with
the sparse-attention memory rather than in the classes that serve every other
architecture. llama-kv-cache.{h,cpp} and llama-memory-hybrid.{h,cpp} are
restored to their unmodified state.
init_batch is repeated from llama_memory_hybrid because the indexer cache has to
be handed the attention cache's slot infos, and those are not reachable through
the context the base returns. Allocating them separately lets the two caches
drift, which is what pointed QSA's top-k at the wrong cells before.
The context derives from llama_memory_hybrid_context so build_inp_mem_hybrid
keeps working unchanged, and get_n_stream is computed from the slot infos
exactly as llama_kv_cache_context did.
Behaviour is unchanged: logits over an 8192-token sequence are bit-identical to
the previous implementation, sparse and dense alike.
* llama: save and restore the qwen4exp indexer KV cache
llama_memory_hybrid_idx forwarded clear, seq_rm, seq_cp, seq_keep, seq_add and
seq_div to the indexer cache but not state_write / state_read, so a saved
session dropped the indexer keys and a restored one selected QSA top-k against
an empty cache. The effect is invisible until the context passes
indexer_top_k + compress_ratio - 1 cells, because QSA is exactly dense below
that and the indexer contents cannot change the result.
The indexer section is written last rather than next to the attention cache it
mirrors. As a suffix, a reader that does not expect it stops early and the
trailing bytes are caught by the size check in state_load_file; placed between
the attention and recurrent sections it would instead be parsed as recurrent
state, which can succeed and restore silent garbage. It follows the same
LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY gate as the attention cache, since a partial
checkpoint deliberately skips the token-level attention caches.
The indexer restores its own cells instead of taking the attention cache's
restored slots. The two caches share size, padding and every sequence
operation, and init_batch hands the indexer the attention cache's slot infos,
so both state_read_meta calls run find_slot over identical occupancy and land
on identical cells.
The overrides live on llama_memory_hybrid_idx, the only memory type that owns
an indexer cache, so llama_memory_hybrid and every architecture that uses it
write and read exactly the bytes they did before.
The session and sequence state versions are bumped because the qwen4exp state
layout changed. The session path already rejects a short read via its size
check, but llama_state_seq_load_file accepts one silently, so only the version
check stops a pre-fix blob from being half-restored by a fixed build.
(cherry picked from commit 2721542354f8e158c3217625f4e2e7b83e51e3fe)
* llama: make the qwen4exp PLE n-gram history per context and serialise it
The PLE hash of a token mixes in the ple_ngram_size - 1 tokens before it, which
a decode ubatch does not carry, so they were remembered in a map on
llama_model_qwen4exp. That is the wrong owner twice over.
A llama_model is shared by every context that loads it, and the map was keyed
only by llama_seq_id, so two contexts running the same sequence id - two server
instances on one model, or a draft/target pair - overwrote each other's window.
The next_pos guard turned that into EOS padding instead of a crash, so it
degraded quality silently.
The map was also in no state blob: grep found ple_hist in neither
llama-kv-cache.cpp nor llama-memory-*.cpp nor llama-context.cpp. A restored
context therefore failed the next_pos check on its first ubatch and hashed the
first tokens after the restore against EOS padding. This is why a session blob
round-tripped byte for byte while the restored context computed different
logits: the state was never in the bytes.
It moves to llama_memory_hybrid_idx, which is per context, is the memory type
qwen4exp always builds, and already does the per-sequence bookkeeping this
needs. Every sequence operation now carries the window with it:
seq_rm a rewind (p1 < 0) truncates the window to the surviving prefix and
moves next_pos to p0, so a rollback keeps exact context; a hole
punched in the middle leaves the window non-contiguous, so it is
dropped
seq_cp the destination inherits the source's window, truncated to the
copied position range - a copied sequence continues with the same
n-grams the source would have used
seq_keep every other sequence's window is dropped, like its cells
seq_add a shift that moves the whole window keeps it and moves next_pos with
it, which is the context-shift case; one that cuts through it drops
it
seq_div positions stop being consecutive, so an overlapping window is
dropped
clear everything is dropped
Dropping means next_pos = -1, which set_input turns into full EOS padding: the
same thing a fresh sequence gets, and the same thing this code did before it
followed the sequence operations at all, so no case is worse than before.
The state payload is a self-delimiting list, u32 count then per entry
{ i32 seq_id, i32 next_pos, u32 n_toks, i32 toks[n_toks] }, so a whole-context
save and a single-sequence save share one format and a single-sequence restore
can retarget the window at its destination seq_id. It is written after the
indexer section, last, for the same reason that one is: as a pure suffix an
older reader stops early instead of parsing these bytes as something else.
Unlike the indexer section it is not under LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY.
The window is recurrent state - it is the input the PLE convolution's own
recurrent state is derived from - and the recurrent cache beside it is written
for partial checkpoints too. Gating it would leave the server's speculative
decoding checkpoints restoring the conv state without the window that produced
it.
No further version bump: LLAMA_SESSION_VERSION 10 and LLAMA_STATE_SEQ_VERSION 3
were introduced for the indexer section in the same unreleased series, and both
changes are qwen4exp-only additions to the same blob layout.
Also fixes the padding of a short window. set_input pads a window shorter than
ngram_size - 1 up to that length, but prev() indexes the snapshot with the most
recent token last, and resize() pads at the back, so the filler EOS landed where
the immediately preceding token belongs. It now pads at the front. A window is
short at a sequence start after a one-token prefill, and after a seq_rm rewind,
which the new bookkeeping makes common.
Every architecture other than qwen4exp builds llama_memory_hybrid rather than
llama_memory_hybrid_idx, has no PLE table and never asks for a history, so
nothing about its graph, its sequence operations or its state bytes changes.
(cherry picked from commit de170364c052c68fcf63285cc0028095edb9f23c)
* qwen4exp: tidy comments and simplify image token read
Rewrite the comments this series adds to the AGENTS.md rules: one or two lines,
no prose hard-wrapped mid-sentence, no narrative or history, and no comment that
only restates the code. Net 146 fewer comment lines, no code change.
Correct the PLE image comment: mtmd does not consume the placeholder ids. An
image is decoded as an embeddings-only batch, so ubatch->token is null and the
per-position ids never exist here. gemma3n and gemma4 hit the same case and
stand in row 0 of per_layer_token_embd; qwen4exp stands in the configured image
token id instead.
Read image_token_id straight from self.hparams in the converter. base.py merges
text_config into the root of hparams, and the key sits at the root of
config.json, so the config.json re-read was redundant.
(cherry picked from commit 205840c12169057da3e8d2f65ec4ceec3e18b980)
* qwen4exp: support a quantized KV cache in the QSA attention path
(cherry picked from commit 4c30574f81dc1115d08078c47b6cf8c789c0a842)
* llama: give qwen4exp a large-graph node budget
(cherry picked from commit 37c8c194e6a30e4c46ac29bee3fb264f091596ef)
* qwen4exp: drop an unused variable that breaks -Werror builds
(cherry picked from commit 528d032b51fa3cf935ed3ef6e0fb1c7401df53b5)
* quantize: dequantize and quantize large tensors in row bands
f32_conv_buf held the whole dequantized tensor, which is 204.8 GB for
per_layer_token_embd alone and dies with std::bad_alloc long before the
work buffer is reached. Dequantize and quantize in bands of whole rows
instead, capping the f32 staging at 1 GiB per band.
Rows are independent and the imatrix is indexed by column, so band
boundaries cannot change any output byte. Bands nest inside the existing
per-expert loop so each expert slice keeps its own imatrix, and a band is
kept to at least one quantization chunk per worker thread so the existing
multithreading still has work. F32 sources still stage nothing and are
banded by pointer arithmetic into the tensor.
llama_tensor_dequantize_impl now takes a first element offset; the single
caller is updated.
(cherry picked from commit 658c22549613555dbce57a772be4de8509eba3ee)
* llama: segment the qwen4exp fused QKV for tensor split
qwen4exp was missing from the gated delta net branch of get_split_segments,
so its attn_qkv.weight, shaped {n_embd, 2*key_dim + value_dim}, fell through
to the generic fused QKV rule and tripped
GGML_ASSERT(tensor->ne[axis] == n_embd + 2*n_embd_gqa) while loading with
--split-mode tensor. --split-mode layer was unaffected.
qwen4exp broadcasts K to the V heads by tiling, k_conv is grown with a plain
ggml_repeat_4d over the head axis so that v head j pairs with k head
j % n_k_heads. That is the Qwen 3.5 pattern, not the repeat interleave that
Qwen 3 Next builds explicitly, so qwen4exp takes the else branch and its V is
segmented on the scale of K.
Reported by benklop.
(cherry picked from commit 353d753f595dc81634ae6130188b31f06018f5ae)
* llama: fix the qwen4exp PLE history seq_rm(-1) iterator invalidation and the fatal-warning build
ple_hist_rm recursed over ple_hist with a range-based for and the recursive call
erases the entry it is iterating when the whole sequence is removed (p0 <= 0,
p1 < 0), so the loop then increments an invalidated iterator. It is unreachable
today only because llama_memory_recurrent::seq_rm rejects seq_id < 0 before
llama_memory_hybrid_idx::seq_rm reaches the history, which is a guard in another
class. Advance past the entry before recursing.
Two smaller things in the same area:
- the n_toks sanity bound in ple_hist_state_read was the literal 64, which is
the value of LLAMA_MAX_PLE_HEADS, not of the quantity being checked. The
window is at most ple_ngram_size - 1 tokens, so the bound is
LLAMA_MAX_PLE_NGRAM - 1, eight times tighter.
- build_conv_state_at left mem_size unused, so -DLLAMA_FATAL_WARNINGS=ON does
not compile. Predates this series; drop the line.
(cherry picked from commit 6eba44a89d5f328eb4859b844e1d28fb564cbe3e)
* qwen4exp: include llama-impl.h explicitly for llama_mul_mat_hadamard
(cherry picked from commit b634fd4d250d181ef82bf78bd00c1ae3b96a7af6)
* convert: fix the qwen4exp lint and type-check failures
flake8 flagged an unused MmprojModel import, and ty flagged seven errors in
the PLE streaming path: eos_token_id can be absent, and _ple_map, _ple_path,
_ple_row_dim and _ple_rows_per_shard are all Optional at the declaration but
were dereferenced without narrowing.
The map is opened and the stride fixed before the first shard is written, and
_finish_ple_table only runs once every shard has landed, so the invariants
hold. Assert them so the checker can see it. A missing eos_token_id now raises
with the reason instead of a TypeError from int(None).
* llama: give the qwen4exp indexer cache its own tensor names
The indexer KV cache and the attention KV cache both named their tensors
cache_k_l%d, so the Meta backend matched the indexer cache against the
attention split pattern and aborted in handle_set_rows. Tag the names
instead, and mirror the indexer cache: it has one key head and its
projections are mirrored.
(cherry picked from commit a1cdc8181134659766763a17762545a1f0e5db7b)
* qwen4exp: double the Q split granularity for tensor parallelism
qwen4exp fuses the attention gate into attn_q.weight the same way qwen3next
and qwen 3.5 do, so a device boundary must fall on a whole q+gate pair or the
Q heads stop lining up with the K/V heads and attn_output rows.
(cherry picked from commit 6c9a592f0a425a459ab6efae3b897cf68460e244)
* qwen4exp: keep the indexer cache in step across server slots
The QSA indexer keeps a side cache addressed by the cells of the attention
cache, so cell j has to hold the same token in both: the top-k indices it
produces are applied to the attention KQ mask. init_batch already hands the
indexer the attention cache's slot layout rather than letting it look for its
own, but the restore path did not. state_read called llama_kv_cache::state_read
on the two caches in turn and each ran its own find_slot over its own occupancy.
That agrees only for as long as nothing has already pushed the two caches apart,
which is the property a restore is supposed to re-establish rather than one it
can lean on.
The failure path was the worse half, and it is reachable from the public API
with nothing more than a short buffer. Truncating a good blob at 35 offsets and
feeding it to llama_state_seq_set_data left the two caches disagreeing at 5 of
them, and every one of 23 truncations of a whole-context blob did. Four of those
five land inside the attention section, so the attention cache drops the
sequence and the indexer keeps it; only the cut that lands in the indexer
section gives the opposite direction. llama_kv_cache::state_read cleans up its
own cache and rethrows, so whichever way it falls, nothing is left to bring the
two back together. The server papers over this by clearing the slot when a
prompt cache load fails; a caller of llama_state_seq_set_data that does not is
left with an indexer addressing cells that no longer mean what it thinks.
llama_kv_cache::state_read_sinfo reports the cells a restore landed in, or takes
a copy of them, and state_read_meta uses a supplied layout in place of find_slot
once it has checked that those cells are free here too. The indexer now adopts
the attention cache's restored layout by construction instead of reproducing it
by coincidence, and a layout that does not fit fails the read rather than being
applied over cells that already drifted. The hybrid restore is wrapped so that
any failure drops the sequence, or for a whole-context restore the context, from
all three caches at once, which is a state they do agree on.
* kv-cache: clear the cache once when restoring a whole context
state_read walks the streams of the cache in turn, and for a whole-context restore
each stream went through state_read_meta, which starts by calling clear(). clear()
resets every stream at once, so each stream after the first threw away the streams
already restored, and the K/V buffers with them. A non-unified cache holds one
stream per sequence, so a context saved with N sequences in it came back with only
the sequence in the last stream that carried any cells - the highest sequence id.
A unified cache has one stream and never showed it.
The cache is now emptied once, before the loop, which is what a whole-context
restore means. A blob whose streams are all empty now empties the cache as well,
where before it left the old contents in place.
* kv-cache: check the mirrored slot layout on a whole-context restore too
state_read_meta only looked at the layout it was given on the single-sequence path.
A whole-context restore lays the cells out from 0 in both caches, so they agree as
long as they restore the same number of cells, but nothing checked that they did: an
indexer section belonging to some other context was read over cells the attention
cache had filled from a different one, which is the state the indexer must never be
left in.
* qwen4exp: give the PLE conv history its own mirrored recurrent row
n_embd_r() reserved n_conv + ple_conv_state() so that one cache_r_l row could
carry both the delta-net conv state and the PLE dilated conv history, but the
QWEN4EXP arm of get_split_segments only described n_conv. Under -sm tensor the
segment sum came up short by ple_conv_state() and llama_memory_recurrent
construction aborted in ggml_backend_meta_alloc_ctx_tensors_from_buft.
Widening the segment list is not the fix. The Meta backend propagates a view's
split descriptor from its parent unchanged, so a view of one sub-range of a
split axis is sized as the whole row on every device; declaring the PLE tail as
a second segment merely moves the abort to "shape mismatch for VIEW" at graph
allocation. The two histories also want opposite policies: the delta-net state
is split by head to match wqkv and ssm_conv1d, while per_layer_tok_embd,
ple_conv1d and ple_norm_conv are all mirrored, so every device computes the
whole dilated conv and needs the whole history. One tensor cannot be both, and
the split state has no per-segment mirroring.
Move the PLE history into its own cache_ple_r_l%d row, mark it MIRRORED, and
return n_embd_r() to n_conv. The row is allocated only on layers where is_ple
holds, so mirroring one 92160-element row per device replaces a 92160-element
tail on all 36 recurrent rows: the recurrent R footprint drops rather than
grows. build_conv_state_at now takes its width from the tensor it was handed
and keys its gather on that tensor, which also drops a cont of a strided view.
* no more ple_hist (use master version)
* llama: give the qwen4exp full memory context its indexer cache
graph_reserve() walks a full memory context, and qwen4exp builds its
sparse attention only when the context exposes an indexer cache. the
full-context constructor left ctx_idx null, so the reserved worst case
was the dense fallback: a smaller graph than the one decode executes.
ggml-alloc then had to grow the compute buffer on the first decode,
past the size reported at load.
with -np 4 -c 32768 -fa on -ctk q8_0 -ctv q8_0 on an IQ1_S qwen4exp,
the reserved CUDA0 buffer was 217.00 MiB against 275.71 MiB actually
used, and CUDA_Host 42.31 MiB against 191.14 MiB. reserving the sparse
graph makes both match exactly, in unified and non-unified cache mode.
Co-authored-by: Pascal <admin@serveurperso.com>
Assisted-by: Claude
* qwen4exp: shrink the PLE hparams storage
llama_hparams is held by value inside llm_graph_params and every llm_graph_input_*,
and llm_graph_params is a stack local in graph_reserve and process_ubatch, so its
width is paid on every worker thread stack.
is_ple_impl spent 2048 bytes carrying 512 bits. It is the one per-layer flag that is
not moved through the loader's uint32 array templates, so a bitset costs nothing in
call sites and also removes the uninitialized read that non-qwen4exp archs had, since
nothing filled the array for them.
The PLE head offsets and vocab sizes are token-space indices; the gather that consumes
them already truncates to int32, so 64-bit storage was never reachable. The gguf arrays
stay uint64 for file compatibility and are narrowed on load.
sizeof(llama_hparams) 34440 -> 31944, sizeof(llm_graph_params) 34872 -> 32376.
* llama: opt-in random-access mmap advice for host-resident gather tables
qwen4exp keeps per_layer_token_embd on the host: 26.8 GiB at IQ4_NL, read
by ggml_get_rows as 16 gathers of ~90-170 bytes per token, spread across
16 head regions ~20M rows apart. Measured over 4.75M gathers, no two
consecutive gathers land on the same 4 KiB page, so the readahead the
loader asks for buys nothing here and the whole table ends up cached to
serve about 4% of itself.
llama_mmap applies POSIX_FADV_SEQUENTIAL, MAP_POPULATE and a whole-file
POSIX_MADV_WILLNEED unconditionally. Those are right for streaming the
file once into buffers and wrong for whatever stays mapped afterwards.
Under LLAMA_MMAP_RANDOM the eager pull-in is skipped and the mapping is
advised random once every tensor has been read, so the load itself keeps
its sequential readahead. That alone drops the table to 4.4% resident but
serializes one NVMe latency per gather.
The second half is what pays for it: the PLE input already computes every
row index for the ubatch before the graph runs, so the pages those rows
fall on are handed to the kernel in one batch and the reads overlap.
POSIX_MADV_WILLNEED on POSIX, PrefetchVirtualMemory on Windows, which
takes the discontiguous ranges in a single call.
Off by default and off for every other model: the batched prefetch keys
off "this mapping was advised random", which nothing sets unless the user
opts in.
-c 512 --chunks 60, cold, IQ1_S, mean of 3:
default 35.3 s 26.82 GiB resident (100%)
advice only 104.5 s 1.19 GiB resident (4.4%)
advice + prefetch 34.2 s 1.19 GiB resident (4.4%)
PPL 4.2346 +/- 0.07862 in all three. IQ1_S KLD is unchanged in every
field, including Mean KLD 0.396070 +/- 0.001931 and Same top p 77.325%.
* llama: narrow the random-access mmap advice to the gather table
The advice was applied per mapping: every mapping the model kept got
POSIX_MADV_RANDOM plus a whole-file POSIX_FADV_RANDOM, and the eager
pull-in was skipped for every file. On qwen4exp that also hit
token_embd.weight, which sits 0.33 GiB past the PLE table in the same
shard and is read densely, not by sparse gathers. Measured over
-c 512 --chunks 60 on IQ1_S it fell to 8.45% resident, against 100% with
the feature off.
A model now nominates its gather tables (qwen4exp: per_layer_tok_embd)
and only those byte ranges are advised. The range is rounded out to
whole pages, which on this model takes in 832 bytes before and 192
after. token_embd goes back to 86.55% resident and the PLE table still
drops to 4.44%; smaps shows one VM_RAND_READ VMA of exactly the table
instead of one over all 27.16 GiB that stays mapped.
posix_fadvise is dropped from the narrowed path. POSIX_FADV_RANDOM
ignores its offset and length and marks the whole open file, and the
FMODE_RANDOM it sets is only read by page_cache_sync_ra() on the read()
path, which a fault on a MADV_RANDOM vma never reaches. POSIX_FADV_
DONTNEED does take a range, so the drop mode keeps it.
The eager pull-in is now skipped only for the files holding a nominated
table, and re-issued as WILLNEED over the rest of such a file, so other
shards load exactly as before.
prefetch_rows() keys off the tensor being nominated rather than off a
mapping-level flag, so the batched readahead lands only where the advice
did.
-c 512 --chunks 60, cold, IQ1_S, mean of 3, total wall:
default 32.50 s
whole mapping 30.05 s
narrowed 30.35 s
PPL 4.2061 in all three. IQ1_S KLD is bit-identical with the feature on
and off, including Mean KLD 0.396070 +/- 0.001931 and Same top p
77.325%. tg128 73.65 +/- 0.33 narrowed against 73.49 +/- 0.34 whole.
Assisted-by: Claude
* llama: fold the random-access prefetch into its own feature flag
LLAMA_MMAP_RANDOM_PREFETCH existed to measure the two halves of the feature
apart, and the measurement is done: on a cold cache over the same wikitext
run, MADV_RANDOM without the batched readahead takes 94.4 s against 36.7 s
for an untouched mapping, while the pair together take 34.1 s. Suppressing
the kernel's readahead only pays if we replace it, so the split let a user
select a 2.6x regression through a documented switch.
Keep the accessor, since the call site reads better than a mode comparison,
but derive it from the mode alone.
* FACP (Fewer Acronym Classes Please)
* qwen4exp: bias the QSA selection per block, not per cell
The QSA bias is a graph input, so it is pinned on the host and uploaded every
decode, and at -c 32768 -np 4 its twelve copies were 768 of the 815 MiB of
reserved host compute buffer.
Only one half of it needs a cell: whether the cell sits in the always-visible
tail, and whether its block was pooled. Both are properties of the block. The
other half - empty, other sequence, or in the future - is the plain visible/not
test the attention mask already carries over the same cells, so add that mask
instead of repeating it. The bias then holds one value per block.
A block sits wholly inside or wholly outside the tail because the tail starts on
a block boundary, so one value per block is exact. Cells no block covers keep
their -inf from the mask.
The mask is F16 and the bias F32, and a mixed ggml_add reinterprets the F16
buffer as float rather than converting it, so the cast is required.
reserved host compute buffer at -c 32768 -np 4:
--kv-unified 814.86 -> 238.86 MiB, CUDA0 721.07 -> 421.07 MiB
--no-kv-unified 214.86 -> 70.86 MiB, CUDA0 317.07 -> 265.07 MiB
Selection is unchanged: over 8192 tokens, four times the budget, every QSA
layer returns identical top-k indices and the logprobs are bitwise equal.
Two things a reviewer should know. A cell whose position divides past the last
block is guarded by an assert rather than handled, because no run reached it.
And the mask's same-position M-RoPE rule cannot fire for text and was never
exercised for images, so the 2D case is unverified.
* clean up code comments
* clean up new comments
* revert LLAMA_MMAP_RANDOM
* nits
* replace some changes with #27795
* improve the m-rope image for get_prev_tokens
* LazyChunkedTensor
* fix lint
* add some validations
* reduce input nodes
* trim output tokens
* nits
* some more sanity checks
* fix llm_graph_input_ple reuse
* exclude from webgpu test
---------
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* spec : add DFlash2 support (local convolution + candidate selector) (#27342)
* support DFlash2
* Add p_min in DFlash2
Assisted-by: Claude Opus 5
* Revert unnecessary changes
Assisted-by: Claude Opus 5
* Revert draft sampling in rejection sampling
Assisted-by: Claude Opus 5
* Refactor code structure
Assisted-by: Claude Opus 5
* Delete embedding scaling
Assisted-by: Claude Opus 5
* Gate output transforms on DFlash2
Assisted-by: Claude Opus 5
* Optimize Dflash 2 cost
Assisted-by: Claude Opus 5
* Avoid using atoi
Assisted-by: Claude Opus 5
* Modify comments
Assisted-by: Claude Opus 5
* Move llama_model_dflash_selector_top_k to llama-ext.h
Assisted-by: Claude Opus 5
* Formatting
Assisted-by: Claude Opus 5
* Apply patch to fix the mrope bug
Assisted-by: Claude Opus 5
* fix ci
Assisted-by: Claude Opus 5
* Fix graph number calculation
Assisted-by: Claude Opus 5
* rename hid and unary
Assisted-by: Claude Opus 5
---------
Co-authored-by: Jian Chen <jianchen0311@gmail.com>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* revert top-k.cu changes
---------
Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn>
Co-authored-by: Jian Chen <jianchen0311@gmail.com>
* ui : strip trailing container-format segments from parsed model names
* ui : show reasoning and modality icons on model options and search by modality
* ui : keep reasoning submenu visible regardless of model state
* ui : add show-org-name-in-trigger display setting
* ui : move model list into a submenu within the model selector
* ui : make model option hover and focus highlight override the active state
* ui : add raw model id tooltip to model selector options
* feat: Enable microphone input as default for audio models
* ui : fix eslint issues in chat form and model selector
* ui: show modality icons instead of file submenu in chat add menu
Assisted-by: pi
* chore: Format
* chore: Format
* ui: add ModelCapability enum and shared modality/capability icon constants
Assisted by: pi:GLM-5.3-Flash
* ui: derive modality badge icons and labels from shared constants
Assisted by: pi:GLM-5.3-Flash
* ui: split model option icons into capabilities and modalities
Replace the supportsThinking flag on ModelId with a capabilities object
keyed like ModelModalities, so future capabilities (tool calls, etc.)
slot in alongside reasoning. Icons and labels now come from the shared
CAPABILITY_ICONS/MODALITY_ICONS constants.
Assisted by: pi:GLM-5.3-Flash
* ui: replace per-conversation MCP overrides with per-conversation tool policy
MCP server enabled state is now global (server.enabled); per-conversation
control moves to disabled tool keys and categories seeded into each new
conversation. Aligns the add sheet with the dropdown options and flattens
MCP tool groups in the tools submenu.
Assisted-by: pi
* ui: keep tool policy migration running when defaults parse fails
A corrupt disabledToolKeys localStorage entry no longer aborts the
migration; it falls through with empty defaults so legacy MCP server
overrides still get converted.
Assisted-by: pi
* ui: fall back to global defaults when agentic flow has no tool policy
Passing empty disabled sets bypassed the global defaults and could
enable tools for callers that do not pass a policy yet.
Assisted-by: pi
* ui: align preferences section headers with their methods
The Reasoning Effort and Working Directory headers sat above tool
policy methods; move them above setCwd and setReasoningEffort. Also
clarify the disabled tools JSDoc: existing rows with an unset field
have an empty policy, defaults apply only when there is no active
conversation.
Assisted-by: pi
* ui: gate MCP server avatars on conversation tool policy
Servers whose tools are disabled for the current conversation (MCP
category or server-scoped key) no longer show as enabled for the chat.
Assisted-by: pi
* ui: drop unused MCP category toggle from tools panel hook
Per-conversation MCP control is server-granular; no component renders
a whole-category toggle, so remove the dead API.
Assisted-by: pi
* ui: skip MCP init when flow policy disables the MCP category
Resolve the effective tool policy before deciding whether to
initialize MCP so flows that will not send any MCP tools skip the
init work. Callers without a policy keep falling back to global
defaults.
Assisted-by: pi
* chore: format
* ui: restore reasoning section in mobile add sheet
The sheet rewrite dropped it; the desktop dropdown still has it.
MCP Prompts and Resources stay out of the sheet on purpose.
Assisted-by: pi
* ui: clear MCP server group key in enableAllToolsForServer
The group key disables every tool of the server regardless of
per-tool keys, so re-enabling a server from Settings did nothing
while it was set.
Assisted-by: pi
* ui: skip MCP init when no policy-enabled server remains
Extends the category-level check: the flow also skips MCP init when
every globally-enabled server has its server-scoped group key
disabled in the tool policy.
Assisted-by: pi
* ui: make Settings tools tab edit defaults with category toggles
Adds per-category checkboxes and a caption stating the tab applies
to new conversations; tool picks inside a chat only affect that
chat.
Assisted-by: pi
* ui: gate cwd picker and mention picker on effective tool policy
Both checked the global disabled set directly, so a conversation
that disabled file_search still showed search as available.
Assisted-by: pi
* ui: clean up tool key helpers and store docs
Documents getEnabledToolsForLLM properly, unstacks the JSDoc at
isEntryEnabled, makes setToolEnabled persist like setCategoryEnabled
(toggleTool now delegates to it), and routes the serverId-less MCP
branch of toolKey through getMcpServerToolsKey so both key formats
come from one place. Preferences banner comments become plain
comments so they no longer read as class member docs.
Assisted-by: pi
* ui: indeterminate group checkboxes and inert grayed rows
A category that is on with nothing enabled under it now shows the
mixed checkbox state instead of a checked box next to 0/N. Rows
grayed out by a disabled parent no longer stay clickable behind
opacity.
Assisted-by: pi
* ui: gate MCP prompt and resource capabilities on tool policy
hasPromptsCapability and hasResourcesCapability accept an optional
set of usable server ids; ChatFormActions resolves it from global
enablement minus the active conversation's policy. Restores the
per-chat gating the old mcpServerOverrides provided; callers without
arguments keep global behavior.
Assisted-by: pi
* ui: remove unmounted MCP submenu component
Never rendered anywhere; its entries are duplicates (prompts and
resources live in the attachment menu, servers in the add menu and
sheet) that would need capability wiring maintained for nothing.
Assisted-by: pi
* ui: fix model information dialog width on all screen sizes
The dialog sets container-type: inline-size, so auto width ignores
its contents and collapses to padding. Give it an explicit viewport
width on mobile and cap at 60rem on desktop.
Assisted-by: pi
* ui: scroll wide chat template in model information dialog
Long unbreakable Jinja tokens blew out the table and dialog width;
the block now scrolls horizontally instead of stretching.
Assisted-by: pi
* ui: use fixed table layout in model information dialog
Auto table layout sizes columns to content min-content, so the chat
template's long lines kept inflating the dialog despite the scroll
wrapper. Fixed layout pins the first column and gives the value
column a definite width the wrapper can scroll within. min-w-0 on
the grid item guards the same path on the grid side.
Assisted-by: pi
* ui: make model information dialog full-screen on mobile
Matches the settings dialog pattern: full viewport below md,
calc-sized and capped at 60rem on desktop.
Assisted-by: pi
* ui: stack chat template row in model information dialog
Label above the block in a single full-width cell, so the template
gets the whole table width and its horizontal scroll is usable on
narrow screens.
Assisted-by: pi
* ui: scroll model information header with the content
The base dialog header is sticky; this dialog overrides it to
relative so the title and description scroll away with the body.
relative keeps the header as the close button's containing block.
Assisted-by: pi
* ui: replace literal comment text in sheet group snippet
A // line inside the Svelte snippet rendered as visible text; use an
HTML comment.
Assisted-by: pi
* ui: let indeterminate state win over checked in group checkboxes
The checkbox indicator snippet renders the check icon whenever
checked, so the mixed state never showed. Pass the checked prop
as false while indeterminate.
Assisted-by: pi
* ui: initialize only policy-enabled MCP servers for a flow
ensureInitialized accepts an optional server id set; the agentic
flow passes the servers its tool policy leaves usable, so servers
disabled for the conversation no longer get connected. Callers
without arguments keep the global behavior.
Assisted-by: pi
* ui: derive group checkbox state in useToolsPanel
Moves the mixed-state derivation out of the submenu and sheet
snippets into one getGroupCheckState accessor; the snippets just
consume checked and indeterminate.
Assisted-by: pi
* ui: gate /prompt command on the conversation tool policy
The slash command's availability now follows the same rule as the
agentic flow instead of the global capability check, so it disables
itself when the conversation's policy leaves no usable MCP server.
Assisted-by: pi
* ui: remove dead MCP prompt menu trigger chain
The /prompt slash command is the surviving trigger; the menu-button
path (onMcpPromptClick, hasMcpPromptsSupport, showMcpPromptButton,
the MCP_PROMPT attachment item and its unrendered item arrays) has
no consumer left. Message display for inserted prompts is untouched.
Assisted-by: pi
* ui: render dash for mixed-state group checkboxes
The accessor refactor dropped the checked-and-not-indeterminate
guard, so the category-on flag won and the dash never showed. The
tooltip keeps using the raw parent flag since clicking a mixed
group still disables it.
Assisted-by: pi
* ui: fix group checkbox sticking checked after disable
Clicking a mixed-state group box let bits-ui optimistically flip
its internal checked flag; the derived checked prop did not change
across the transition (both mixed and off map to checked=false),
so Svelte never applied the settled value and the check icon stuck
while the count already read 0/7.
Pass the parent flag as checked and the mix as indeterminate, so
every group toggle changes checked; render the dash on top of a
checked box for the mixed state.
Assisted-by: pi
* fix: UI for Model Information dialog
* ui: keep MCP connections stable across policy switches
ensureInitialized folds the policy into its config signature, so
alternating two conversations with different policies tore down and
reconnected every server with health checks included. Tool collection
already filters by the flow policy, so initialize every
settings-enabled server instead and never pass a policy into the MCP
config. The duplicated policy-server check becomes one accessor on
ConversationPreferences.
Assisted-by: pi
* ui: remove dead MCP resources menu trigger chain
Same shape as the earlier prompt trigger cleanup: nothing renders the
MCP resources menu button, and the only live entry into resource
browsing is Settings > MCP Servers plus the attachment resource
picker. Drop onMcpResourcesClick, hasMcpResourcesSupport,
MCP_RESOURCES_CLICK, the AttachmentItemVisibleWhen enum and
hasResourcesCapability; the resources display, browser and picker
components are untouched.
Assisted-by: pi
* Add benchmark-only synthetic speculative acceptance to llama-server and llama-cli
* Address review comments
* Address review comments
* Add some comments in the code
* common : dedupe --n-cpu-moe / --spec-draft-n-cpu-moe override loops
* common : add --n-cpu-ffn to CPU-offload dense FFN weights of first N layers
* common : generalize llm_ffn_block_regex over the FFN regex, drop TODO
* hexagon: use non-host bufs by default and make the backend fully async
* hex-hb: remove optional hostbuf support and fix async copy
* hex-unary: relax supported unary check
* hex-bufs: use same get_alignment for host bufs
* snapdragon: bump android_platform to 34
* hex-rows: super hacky get/set rows for q8_0
* hex-get-rows: fix q8_0
* hex-get-rows: supprot for f16 and cleanup for q8_0
* hex-get-rows: generic macros and specialized thread funcs
* hex-get-rows: add DMA pipeline, vtcm_layout and kernel params
* hex-set-rows: fix q8_0 support, add dma and tracing
* hex-tests: override nmse threshold for HTP of Q8_0 quants
* hex-fa: add support for Q8_0 with inplace dequantizers
* hex-get-rows: simplify type dispatch
* hex-rows: simplify GET/SET_ROWS DMA pipeline
* hex-async: add events, set/get-tensor-async and rest of the async api support
* hex-repack: use slice instead of expert in repack functions
* hex-cpy: update event/async-cpy logging
* hex-set-rows: optimize smaller tensors
* hex-geglu: fix perf regression with larger tensors
* hex-get-rows: add missing header
* hex-set-rows: add missing header
* hex-bufs: ressurect GGML_HEXAGON_HOSTBUF but disable it by default
* hexagon: do not reject ops with non-heaxon buffers
* hex-get-rows: apply >=32 restriction only for q8_0
* hex-res: bump vtcm acquire timeout to 10 seconds
* hex-bufs: add support for cloning buffers between sessions to speed up tensor copies
* hex-async: rework event recording and batch flushing and integrate with meta backend
* hex-bufs: improved handling of repacked tensors
* hex-repack: handle get_tensor_2d offsets
* hex-dev: add support for devices with multiple NPUs
* hex-sync: add support for sync tokens to synchronize npu devices for async splits
* hex-mmap: cleanup mmap calls and add a retry for robustness
* hex-sync: add failsafe if sync wait gets stuck
* hex-sync: use sync_seq to check for completed events
* hex-sync: rotate tokens for extra robustness
* hex-devs: add supprot for legacy device names for now
* hex-bufs: add support for auto-cloning buffers from diff sessions
* hex-fusion: simplify and optimize htp-opnode fusion handling
* hex-sync: override opnode name so that it shows up in the profiles
* hex-trace: update scripts to handle multiple devices
* hex-sync: bump the size of the opbatch queue and number of sync tokens
* hex-cpy-sync: do not explicitly flush opbatches in cpy_tensor_async and add support for cpy-dma
* hex-sync: add graph-flush threshold to avoid single op batches
* hex-sync: add sync_peer so that we can flush peers we depend on during cross-device ops
* hex-bufs: introduce tensor->extra and shadow_bufs for repacking
* hex-l2: flush tiny tensors inline
* hex-sync: use explicit l2flush for sync tokens
* hex-extra: track weight flags via tensor extra
* hex-fence: rename sync to fence
* hex-repack: proper handling of set-tensor-2d in the shadow_buf
* hex-trace: remove obsolete opstage mask that we used for profiling
* hex-env: remove obsolete use_hmx variable
* hexagon: new unified run.py and build.py and updated docs
* snapdragon: update run script to auto-escapt test-backend-op -p argument
* hex-scripts: fix trailing spaces
* hex-scripts: fix flake8 warnings
* snapdragon: cleanup dst lib/bin dirs before copying new build
* hex-ops: add support for allreduce
* hex-ar: improved allreduce with dma pipeline
* hex-ar: align macros
* hex-ar: consistent use of fence_seq
* hex-ar: add AR_SELECT env var to select ALLREDUCE kernel or fallback
* hex-ar: add proper synchronize handling for ALLREDUCE
* hex-opbatch: looks like we now just rely on backend.synchronise to flush the batches, no need to flush them by threshold
* hex-ar: bump block size to improve dma efficiency
* hex-ar: fused ALLREDUCE+ADD
* hex-ar: cleaner fence buffer management
* hex-ar: futher allreduce tweaking to remove race conditions
* hex-ar: add simple solver and remove non-dma kernels
* hex-ar: add row-broadcast to fuse with bias ADD
* hex-fence: pass seq numbers via op_params
* hex-ar: allow for both entry/exit seq for completing entry wait
* hex-ar: align macros
* hex-ar: do not refetch broadcast row
* hex-fusion: move all fusion into opbatch::add_op for consistency with ALLREDUCE and things
* hex-fusion: fix incorrect MUL_MAT reordering
* hex-mm: make fused 2x and 3x matmuls more generic
* hex-fusion: move tensor fusion tagging to graph_compute
* hexagon: make sure to copy tensor->extra by value
* hex-get-rows: fix offset calc with row-chunking
* hex-repack: get_tensor_2d fixes for non-zero offsets
* snapdragon: make profile/trace scripts more robust and donot mix stdout/stderr by default
* hex-devices: use legacy device nameing by default to ease the transition
* hex-devices: hardcode CDSP domain IDs for current devices for now
* hex-optrace: improve multi-NPU timestamp alignment and overall handling of cycle values
* hex-optrace: more robust handling of the fence events
* ui : open MCP servers in a dialog from the chat form
Replace the MCP servers submenu with a single "MCP Servers" item that opens
a new DialogMcpServers dialog instead of navigating to the /mcp-servers route.
Assisted-by: pi
* ui : browse MCP resources from the server card
Make the Resources capability badge clickable so it opens the MCP resources
browser dialog, and drop the page-only chrome from SettingsMcpServers.
Assisted-by: pi
* ui : remove mcp-servers route and sidebar entry
MCP servers are now managed in a dialog, so drop the dedicated route and the
sidebar icon that navigated to it.
Assisted-by: pi
* ui : remove unused MCP servers submenu component
The submenu was replaced by the MCP servers dialog, so delete the component
and its export.
Assisted-by: pi
* feat(ui): add DialogSettingsChat dialog
* refactor(ui): switch SettingsChat to in-app section navigation
* feat(ui): open settings as dialog from sidebar
* refactor(ui): remove settings route and URL-based settings navigation
* fix(ui): adjust MCP dialogs for new base sizing
* chore: Formatting & linting
* feat(ui): make base dialog responsive and support sticky headers
* ui: move dialog close button to the sticky header
Assisted-by: pi
* chore: Formatting & linting
* ci : inline UI version resolution into ui-build.yml
* ci : build UI once and reuse the artifact in release jobs
Server jobs now extract the ui-build artifact into tools/ui/dist instead of npm-building the UI. Also removes the get-version job and the no-op -DHF_UI_VERSION flags.
Assisted-by: pi:Kimi-K3
* ui : disable the npm UI build by default (LLAMA_BUILD_UI=OFF)
The flag now only controls building the UI from source via npm. The UI
is still embedded by default from local tools/ui/dist or the prebuilt
download (LLAMA_USE_PREBUILT_UI=ON). CI jobs no longer npm-build the
UI; server-sanitize does not need node anymore.
Assisted-by: pi:Kimi-K3
* ci : rename the ui-build artifact to llama-ui.zip
Consistent with the other artifact names in the Actions summary.
Assisted-by: pi:Kimi-K3
* ci : clarify the windows artifact merge in release.yml
The windows-cuda/vulkan/sycl jobs build only the backend library;
llama-server (with the embedded UI) is injected into their zips from
the windows-cpu package during the release. State this in the job
comments and use accurate wording in the merge step.
Assisted-by: pi:Kimi-K3
* cuda: unblock mmq for MoE on sm_60
* cuda: duplicate mmq-config-pascal for dp4a and older
* cuda: reduce occupancy on non-dp4a pascal for Q2_K, Q4_K, Q5_K, Q6_K
This commit contains a fix for the conversion of NVIDIA Nemotron 3.5
Lightning which currently incorrectly converts when using a transformers
version later than 5.5.1.
When converting using [convert](https://github.com/ggml-org/convert) the
transformers version is 5.13.1 and this produces the following:
```console
WARNING:gguf.gguf_writer:Duplicated key name 'nemotron_h_moe.attention.head_count_kv', overwriting it with new value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] of type ARRAY
```
This does not happen with transformers 5.5.1. The reason seems to be
that the configuration is different in later versions, for example when
using 5.13.1 the configuration block looks like this:
```console
transformers 5.13.1
raw has layers_block_type: True
autoconfig has layers_block_type: True
autoconfig layers_block_type: [
'linear_attention',
'moe',
'linear_attention',
'moe',
'linear_attention',
'full_attention',
'moe',
...
]
```
And with 5.5.1 we get:
```console
transformers 5.5.1
raw has layers_block_type: True
autoconfig has layers_block_type: True
autoconfig layers_block_type: [
'mamba',
'moe',
'mamba',
'moe',
'mamba',
'attention',
'moe'
...
]
```
In our conversion script we only match for attention, not full attention
which is causing this issue.
With the changes in this commit the output with transformers 5.13.1 will
be:
```console
(venv) $ gguf-dump models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf | grep head_count_kv
INFO:gguf-dump:* Loading: models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf
29: [INT32] | 52 | nemotron_h_moe.attention.head_count_kv = [0, 0, 0, 0, 0, 2, ...]
```
Resolves: https://github.com/ggml-org/llama.cpp/issues/27718
Refs: https://github.com/ggml-org/convert/actions/runs/32949047680/job/98116096069#step:5:2391
* metal: WIP chunked SSD SSM_SCAN kernels for multi-token prefill
* metal: drop scalar SSD path; MMA + sequential tail
* drop WIP ssm scan test noise
* remove state_from_dst and rename CS and NSG constants
* remove unrelated added whitespace padding
* added clarity to mma_tokens calculation
* added clarity to use_mma bool checks
* added comments to metal ssd op constants for clarity
* reserve K tokens for sequential kernel rollback snapshots
* reset concurrency between mma and seq tail
* remove print args no longer used
* fixed comment to no longer point to specific line
* add FC_SSM_SCAN so seq path skips token offlset unless it's mma tail
* added changes to new ssm.metal for rebase after ggml-metal.metal refactor
* specialize ssm_scan tail with a template instead of a function constant
---------
Co-authored-by: dpantaleoni <dominikpantaleoni@gmail.com>
Co-authored-by: forforever73 <690105611@qq.com>
* rpc: support apple RDMA as an RPC transport
* remove set_tensor micro optimization, rpc socket pinning per CR
* remove transparent reconnect
* trigger apple builds on RPC changes
---------
Co-authored-by: Ryan Churaman <rschu@meta.com>
* devops: use GGML_NATIVE=OFF for OpenVINO
Same as in other Dockerfiles.
Should fix#23100
* enable backend dl and cpu all variants
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* server: fix tool calls getting silently stripped with --prefill-assistant
Last assistant carries tool_calls + --prefill-assistant is on → request
flips into continuation mode, add_generation_prompt forced off, tail
rebuilt from reasoning_content + content only. Tool calls just vanish.
- Auto-continuation now skips trailing assistant msgs that have tool calls
- continue_final_message on those throws a clear error instead of
silently corrupting the prompt
- Regression tests included, red before / green after
Fixes#27588
Developed with AI assistance, disclosed per the contribution policy.
* server : address review: fail on prefill-assistant + trailing tool_calls
Move validation into oaicompat_chat_params_parse (next to the existing
two-or-more-assistant check) and remove it from common_chat_templates_apply,
which has no precedent for validation. Drop the regression tests.
Per review: --prefill-assistant with a trailing assistant message
containing tool calls is not supported and should fail loudly.
* add ccache-buckets action
* use ccache-buckets
* only save on master
* install python3-venv for hip
* add jq and python3 for cuda
* only delete caches older than 5 minutes
* metal : null-check ggml_metal_buffer_init result to avoid OOM crash
ggml_backend_metal_buffer_type_alloc_buffer used the result of
ggml_metal_buffer_init without checking for NULL. ggml_metal_buffer_init
returns NULL when the underlying Metal allocation fails (e.g. an
out-of-memory condition), and the following ggml_metal_buffer_is_shared(res)
call dereferences it, turning a recoverable allocation failure into a hard
crash (EXC_BAD_ACCESS). This is easy to hit on memory-constrained devices
such as iOS when a model/context exceeds the available Metal budget.
Log the failure using the existing GGML_LOG_ERROR convention and return
NULL so the allocator surfaces a diagnosable error up the stack instead of
crashing.
* cont : fix log
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* grammar : accept "\-" escape in character classes
gbnf_escape_char_class() escapes '-' as "\-" but parse_char() rejected
that escape, so generated tool-call grammars failed to parse.
Assisted-by: Claude Code <claude@anthropic.com>
* tests : add parser test for "\-" in char classes
Assisted-by: Claude Code <claude@anthropic.com>
* tests : add integration test for "\-" in char classes
Assisted-by: Claude Code <claude@anthropic.com>
* tests : drop integration and parser tests
* metal : per-device tuned (Q, NE) for flash-attn vec (#25750)
* rebase Q-generic FA vec body from 01dc93607 (#23114)
* add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133)
* add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback
* add FA vec (Q,NE) perf sweep
* fill tuning result
* fold family table into a per-family representative SKU
* refactor tuning result format
* extend FA vec tuning to quantized KV caches
* sync fa vec tuner bucketing with runtime, use pointwise tuning regret
* update tuned table
* format and cleanup
* prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active
* add device id -> token lookup for the offline tuning tool
* add ggml-metal-tuning skeleton
* add op-agnostic perf cell + median timing for the tuner
* add FA-vec graph build + tensor init to the tuner
* tools : add FA-vec (Q,NE) sweep, compression and table emit
* cool down and re-measure the dirty window on thermal drift
* test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice
* tools : document the Metal tuner, point the table comment at it
* abort on unknown KV type, single-source fa_vec_legal_ne
* cleanup
* honor -o in the FA vec (Q,NE) slice
* retune FA-vec (Q, NE) under a pointwise no-harm gate
* cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* metal : per-op source split + parallel compile (#24021)
* preliminary extract common header
* op source split
* split metallib into 8 libs && load in parallel
* derive kernel->library routing from functionNames
* x-macro lib list + underscore filenames, dedup QK_NL, MRC fixes
* op source split 8 to 20
* improve robustness of source fallback
* clean up
* change bool -> atomic_bool
* only prepend headers that source actually includes
* no semaphore, use GCD global queue
* dedup library compile path, fix NSError lifetime, rename gla
* relocate upstream concat/rope_back/repeat kernel changes into split files
* move ggml-common.h from common.h into dequantize.h to shrink binary size
---------
Co-authored-by: lvyichen <lvyichen@stepfun.com>
* metal: add col2im_1d op (f32/f16/bf16) (#25176)
* metal : add set_rows with src0 f16 (#25434)
* metal : add CONV_2D_DW (depthwise convolution) support (#21565)
* metal : add Q2_0 support (#25419)
* metal: fuse snake activation (mul, sin, sqr, mul, add) (#25459)
* ggml-metal: FWHT kernel for metal backend (#25924)
* metal : port new kernels into the split sources
Move the kernels added on master after the split (lightning indexer,
DSv4 hyper-connections, silu_back, f16 bin ops, TQ2_0, the flash-attn KV
dequantization pass, rope offset/inplace, ssm_scan rollback, packed q8_0
dequantization and the tensor-API mat-mat K clamp) into the corresponding
kernels/*.metal sources. Copied verbatim, no functional change.
---------
Co-authored-by: lvyichen <lvyichen@stepfun.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
`repetition_penalty` is standard HF key for repetion penalty.
Currently, only `penalty_repeat` is mapped, read `repetition_penalty`
and map it to `metadata.sampling_penalty_repeat`.
* ci : apply ccache-clear with older/min/dry-run to all ccache jobs
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* ci : install gh in ccache-clear if missing (container jobs)
The ccache-clear action relies on the gh CLI, which is not present in
container-based jobs. Install it on demand so those jobs can clear caches.
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* ci : install gh via apt repo in ccache-clear
The install.sh script used previously is no longer served (404). Switch to
the official GitHub CLI apt repository, which is still available.
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* ci : pass --repo to gh cache commands in ccache-clear
In container jobs gh cannot auto-detect the repository from git, so
gh cache list/delete fail with 'failed to run git: not a git repository'.
Pass the repository explicitly via --repo using GITHUB_REPOSITORY.
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* ci : drop -new suffix from vulkan ccache key
The -new suffix was only needed to force a fresh cache. With
ccache-clear now evicting stale caches, the original key can be used
again. The old ccache-vulkan-ubuntu-24.04-arm-new entries still match
the ccache-clear key prefix and are cleaned up automatically.
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* ci : fix ccache-clear date parsing on macOS (BSD date)
macOS ships BSD date, which has no -d option. The older cutoff check
was silently disabled there: 'date: illegal option -- d' errors in the
log and the loop was only stopped by the min limit, risking deletion
of caches not older than the cutoff (e.g. saved by a concurrent job).
Parse the ISO-8601 timestamps with GNU date when available and fall
back to BSD date otherwise (TZ=UTC, fractional seconds dropped).
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* ci : extract ccache-clear logic into scripts/ccache-clear.sh
The composite action now consists of a dedicated step that installs the
GitHub CLI when missing (e.g. in container jobs) and a thin step that
calls the new script. The script follows the make-release-checks.sh
conventions (usage/env header, set -euo pipefail, CLI flags) and only
checks that gh is available. The action inputs are unchanged, so the
workflow steps are untouched.
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* ci : remove unused apple ccaches
* DSV4: sm tensor
* set coarser granularity for head splits
* fix dspark
* add model saving for dsv4 + allow dflash to return on specific device
* add comment about dsv4 seq_rm
* simplify
* add shared expert delayed allreduce
* remove special test for dsv4
* readme : update links
* readme : update maintainer PRs list
Add the new members of the `ggml-org` `maintainers` team to the
author filter of the maintainer PRs link (nikwen, marty1885,
Titaniumtown), keeping the canonical team ordering. The list now
matches the team exactly (35 members).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Run test-llama-archs with 1 to 4 GGML_METAL_DEVICES, mirroring the
existing CUDA runs, and dispatch the job unconditionally since the
per-backend guards now decide what to run.
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
The device_info loop iterates over the discovered devices and gets
the available and total memory counts. With the CUDA backend (and
possibly others too) this requires creating a GPU context, which,
in case of CUDA, results in a 550 MB VRAM allocation.
For this information to be used in any way, the log verbosity must
be set to LOG_LEVEL_TRACE. If it's not, including in the default
configuration, the contexts get created, memory sizes get queried,
then the log function quietly discards the data.
In certain cases the user may not want to use any GPU resources.
The device_loop iteration is the only place touching the GPU that
cannot be skipped.
Fix by checking the verbosity level and skipping the loop if there
would be no output.
* DeepseekV4: fix rollback with multi-seq
* fix model loading
* make pending rollback single use
* only clear cache for seq_id for full load
* add assert for compress ratio
* make graph topology static
* pass true instead of flags in clear_compressed
* cont : clean-up + TODOs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ui : add browser-style conversation tabs store
Track open conversation tabs in order, persisted to localStorage and
pruned against the loaded conversation list on init. The chat layout
syncs the route's tab on every navigation, so any way of reaching a
conversation opens a tab for it.
* ui : add temporary new-chat tabs
New-chat tabs are unsaved conversations carrying a temporary id used
directly as the route (#/chat/<id>). They live in memory and are only
persisted to the database - keeping the same id so the route and tab
stay stable - when the first message is sent. Deleting one drops it
without confirmation, and deleting conversations now closes their tabs.
* ui : render conversation tab bar in chat layout
Desktop-only tab bar above the chat screen, one tab per open
conversation or new-chat tab. The active tab follows the route id;
clicking navigates, middle-click or the close button closes (switching
to the left neighbor), and a trailing + starts a new chat. Tabs appear
only on chat-id routes; the bare #/ new-chat view has none. The bare
route stays put unless a prompt/model deep-link routes it to a new-chat
tab.
* ui : route new-chat entry points through tabs
The sidebar New chat item, Cmd+Shift+O, the search page and the
arrow-key fallback now open a new-chat tab instead of navigating to the
?new_chat URL, which is removed. New chat is no longer a special route
but a tab like any other conversation.
* ui : track sidebar expanded state in a shared ui store
Move the desktop sidebar expanded/collapsed state out of deviceStore into a
dedicated uiStore so the chat tab bar can react to it.
Assisted-by: pi
* chat : add opt-in conversation tabs setting
Add a Display setting that turns browser-style conversation tabs on or off,
enabled by default.
Assisted-by: pi
* chat : add browser-style conversation tabs with a new-chat screen
Track open conversations as tabs above the chat, one per open chat, plus a
single New chat tab for the bare `#/` route. New chat is just the `#/`
screen - no temporary conversations - and its tab is dropped when navigating
away. Sending the first message creates a real conversation and opens a tab
for it.
Assisted-by: pi
* chat : turn tab bar into a horizontally scrollable carousel
Make the tab bar a horizontally scrollable carousel with edge scroll buttons
and active-tab centering, and align its styling with the sidebar.
Assisted-by: pi
* chat : restyle the scroll-to-bottom button to match tab styling
Assisted-by: pi
* chat : add close-tab keyboard shortcut
Assisted-by: pi
* chat : soften tab bar fade and dim inactive tabs
Assisted-by: pi
* feat: Add stop button to tabs
* refactor: Componentize
* ui : fix carousel scrollability detection
Observe the content wrapper as well as the container, since adding overflowing items does not change the container's own box size. Also expose an onScrollableChange callback.
Assisted-by: pi
* ui : add unified ScrollCarousel component
Single carousel component with top/center variants, gap and scroll options, and hover-revealed chevrons. Rename the HorizontalScrollCarousel accessibility story accordingly.
Assisted-by: pi
* ui : migrate carousels to ScrollCarousel
Switch the settings mobile header, attachments list, thumbnail strip, and MCP resources to the unified component, and drop HorizontalScrollCarousel.
Assisted-by: pi
* ui : improve chat tabs carousel UX
Scroll newly added tabs into view, fade overflowing tabs at the edges, and hide the New chat button while a new-chat tab is open.
Assisted-by: pi
* refactor: Naming
* chat : add keyboard shortcut to jump between conversation tabs
Shift+Cmd/Ctrl+Left/Right cycles the open tabs, mirroring the existing
Shift+Cmd/Ctrl+Up/Down conversation navigation.
Assisted-by: pi
* chat : make the whole tab item act as a link
The full tab is now a link instead of only the inner label button, while
the stop and close buttons stay interactive by swallowing their clicks.
Assisted-by: pi
* chat : adjust tab bar width and use a shared offset variable
Widen the tab bar for the expanded sidebar and rename the tab bar height
variable to --chat-tabs-offset with a smaller value so the chat screen
min-height accounts for the overlay without overshooting.
Assisted-by: pi
* chat : account for the tab bar offset in the assistant min-height
Subtract the tab bar offset when it is shown so the last assistant message
does not overflow the available viewport space.
Assisted-by: pi
* refactor: Post-review fixes
* ui : restore deep links on the chat start page
- handle ?model selection, with ?load=true eager router loading
- ?q now creates a conversation, sends the prompt, and clears the params
- show the not-available-model dialog for unknown models
- never block mount on the conversation list
Assisted-by: pi
* ui : fix tab item link nesting and centralize tab constants
- the tab anchor covers the whole item while stop/close stay siblings,
so interactive elements are never nested inside the anchor
- cmd/ctrl/middle clicks are left to the browser (new window)
- extract the tab labels, the active-tab data attribute, and the
sidebar-offset max widths into constants
Assisted-by: pi
* ui : tidy scroll carousel hook and keep mobile header arrows on
- drop the dead scrollLeft/scrollRight helpers and the unused
onScrollableChange/scrollBy props
- init the carousel once instead of inside a derived
- restore items-start on the center variant
- always show the settings header arrows on touch
Assisted-by: pi
* ui : keep the new-chat tab across reloads and fall back on close
- the new-chat sentinel is no longer pruned on init, so reloading on
the bare new-chat route keeps the tab the user is on
- closing the active conversation falls back to the new-chat screen
when Conversation tabs are off
Assisted-by: pi
* ui : don't block startup on the conversation list
- prune persisted tabs after the list loads in the background instead
of awaiting it during init
- openNewChat now returns void; its return value was never read
Assisted-by: pi
* ui: fix routing nits
* chore: Update doc comments
* refactor: Mark fire-and-forget openNewChat calls as `void`
* chat: fix the deep-linked prompt, the tab width and the tab shortcuts
The chat start page creates the conversation and hands the prompt over
to the chat route, which still sees it in the query string. Sending it
on both sides queues the second copy as a pending message, which shows
up as a stray user bubble once the answer lands and vanishes on reload
since it never reaches the database.
The tab bar takes the max width of the collapsed sidebar while it is
expanded, and the other way round.
The tab list is pruned against a snapshot of the loaded conversations,
so a conversation created while that list is still loading loses its
tab even though the route just opened it. The active tab then falls out
of the list and the cycling shortcut jumps to an edge on every keypress
instead of moving one tab over. Tabs synced from the route are kept as
they are, only the persisted ones are pruned.
The rich chat input claims ctrl or alt with shift and an arrow for its
badge-aware word jump, which now belongs to the tab cycling shortcut.
Holding shift hands the key combination over, the plain word jump is
unchanged.
The close-tab shortcut consumes the event before checking whether the
setting is on, and the logo background loses its importance flag.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* fit: also take into account n_streams
* server: make the draft context follow the target context
With a non-unified KV cache the target context now holds n_ctx_train
tokens per sequence, while the draft context was still created with
n_ctx = 0 and fell back to n_ctx_train / n_streams per sequence. A slot
filled beyond that point makes the draft batch fail to decode, and the
server answers 500 on the request.
The draft context now takes its size from the target context, so both
hold the same number of tokens per sequence. Contexts that share their
cells with the target no longer need the kv_size override.
The memory reserved for the draft model before fitting is measured at
the largest context the target can take, since the draft context grows
with the target and a fixed byte margin cannot express that.
* fit: take an optional second model into account
Illustrates the alternative discussed on the draft context fix. The
memory of a draft or MTP context is currently handed to the fit as a
fixed byte margin, which cannot express a memory that grows with the
context the fit is still deciding on.
common_fit_params now takes an optional second model that shares the
devices of the main one. Its context follows the main context and its
memory is measured again whenever that context changes, so the reduce
path stays exact instead of conservative. A model that cannot be
measured on its own, such as a shared cell MTP context, is skipped with
a warning and the main model is fitted alone.
This drops the reservation block in the server, which no longer has to
probe the trained context size of the target to guess an upper bound.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* Revert "ci : disable ubuntu-rocm (#26969)"
This reverts commit 9558fa44c9.
* ci: set ccache compiler_check=content for ROCm build
The ROCm toolchain is pip-installed fresh on every run, so the clang binary's
mtime changes each time. With ccache's default compiler_check=mtime that
invalidates the whole cache and warm builds only reached ~70% hits. Hash the
compiler contents instead so the cache survives toolchain reinstalls.
* Update ccache size to 1GB
We're waivering with so many architectures built, we need a bigger
ccache limit.
* merge fix
---------
Co-authored-by: Jim Wu <ywu@xilinx.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ci : add older, min and dry-run options to ccache-clear
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : add note about not wrapping lines in PR descriptions
[no ci]
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* opencl: fold the gpt-oss MoE bias adds into swiglu_oai
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_GLU=0.
* opencl: fold the MoE down-projection bias into the combine
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_COMBINE=0.
The 'Create release' step had no id, so steps.create_release.outputs.id
resolved to an empty string in the 'Upload nightly-tag.txt' step. The
uploadReleaseAsset call then hit /releases//assets and failed with HTTP
404 (Unhandled error: HttpError), e.g. run 32513839499.
Add id: create_release to the step; the action already exposes the id
output.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : run ccache-clear as the last step of release jobs
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* update disabled job too to force rebase
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Similar to ggml/scripts/release.sh: validates repo state, creates a
release candidate branch (llama-rc-vX.Y.Z), bumps LLAMA_VERSION_* in
CMakeLists.txt and commits the version bump.
Usage: ./scripts/release.sh [major|minor|patch] [--dry-run]
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ui : rework the settings registry into ordered raw-data sections
SETTINGS_REGISTRY becomes an ordered SettingsSectionEntry[] array; the
array order is the sidebar display order. Section titles, color mode
options and title radio options are declared inline in their section or
entry. Entries gain showInUi; MCP servers, the system-message toggle and
the title LLM flag become hidden entries of their own section.
Derived values (config defaults, help info, chat sections, numeric field
lists, syncable parameters) are still derived here; they move to their
actual consumers in follow-up commits.
* ui : extract settings localStorage persistence into SettingsService
Stateless load/save of the settings config and user-override keys, plus the
legacy theme key migration. Business logic (default merging, mobile
sendOnEnter default, applying the migrated theme) stays in the store.
* ui : move the settings exit route into ROUTES
SETTINGS_FALLBACK_EXIT_ROUTE is just a route, so it lives with the other
routes as ROUTES.SETTINGS_EXIT.
* ui : derive the syncable parameter list in the parameter sync service
The syncable parameter mapping is only consumed by the sync service, so
derive it there from the registry instead of exporting it from the
constants file.
* ui : restore isPrivate for API key masking
* ui : clean up settings registry and router fetch guard
Drop the per-entry section field (duplicates the parent slug and is
never read) and guard the router model fetch on fields?.length so the
Tools/Import-Export pages with empty fields are excluded again.
Assisted-by: pi
* ui : merge sampling and penalties settings into one section
Assisted-by: pi
As agreed in ggml discussion #1579, the official semver releases now
include a nightly-tag.txt asset containing the tag of the corresponding
nightly release (e.g. b10485). The Web UI assets are published to the
HF bucket under the nightly tag, so this makes them discoverable for
each official release.
- make-release-desc.sh: expose the resolved nightly tag as a
nightly_tag output
- make-release.yml: create nightly-tag.txt from that tag, upload it
as a release asset (skipped on dry-run), mention it in the release
body and in the dry-run summary
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add a reordered Q2_K MMVQ kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add DMMV Q2_K ESIMD kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Resolve the TODO in test_flash_attn_ext: the branch that creates V as a
sub-view of K (MLA-based models) was hardcoded for the 576/512 head shapes.
Add a v_is_view_of_k test case parameter (default false) and select the
sub-view branch on it; the existing 576/512 (DeepSeek MLA) cases now pass it
explicitly, so the test coverage is unchanged.
Also add more V-is-sub-view-of-K cases: the 320/256 (Mistral4 MLA) and
192/128 head shapes, and full views with equal head sizes (128/128 F16,
64/64 q8_0).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add DMMV Q5_K ESIMD kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Remove redundant copyright notice
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* opencl: keep the vocab-scale K-quant lm_head on the CPU on the Adreno A7X
* opencl: revise comments
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
The Tensor API mat-mat path of kernel_mul_mm (GGML_METAL_HAS_TENSOR) fed a
static K=32 tile to the matmul2d op on every iteration. On the last, partial
K tile (ne00 % 32 != 0) the src1 slice extends past the K extent of the
tensor, and the op reads those out-of-bounds elements (undefined behavior per
the MSL specification, section 2.22.2). Depending on stale memory contents,
this corrupted the result or produced NaN.
Make the matmul2d op use dynamic_extent for K, and clamp the K extent of both
operand tensor views to the remaining valid K range (min(32, K - loop_k)) per
iteration, so the op reads exactly the valid K range on every iteration
(mirroring the tail handling of the MPP matmul2d examples). On K-aligned
inputs the clamp degenerates to the full 32-wide tile: the only difference
from the static-K op is that the dynamic-K op derives K from the operand
extents and edge-checks the tile against the tensor extents (a handful of
integer ops per iteration).
Add test-backend-ops MUL_MAT cases with K not a multiple of 32 to exercise
the unaligned K path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* opencl: decline KV-convert flash_attn variants on Adreno A7X (compiler SIGSEGV)
The Adreno 740 (A7X) compiler E031.41 crashes inside clBuildProgram when
building the flash_attn programs whose KV path is mixed-type or dequantized:
flash_attn_f32_f16, flash_attn_f32_q8_0, flash_attn_f32_q4_0. It is a driver
crash rather than a compile-error return, so build_program_from_source_ex()
cannot catch it. The uniform f32 and f16 programs build correctly.
Decline the three KV-convert variants on the A7X in supports_op so they never
lazy-compile; those attention layers run on the CPU backend instead. Same
idiom as the existing Intel DK=512 and X1E carve-outs.
test-backend-ops FLASH_ATTN_EXT on the 740: 226 OK / 0 FAIL, previously exit
139. Other parts are unaffected - the gate is dead code there.
* opencl: fix q6_K flat mul_mat on older Adreno E031 compilers, gated
kernel_mul_mv_q6_K_f32_flat produces ~10x-wrong output on the older Adreno
E031 compilers while q4_K and q5_K are correct. Four codegen defects, each
confirmed on-device against the CPU reference:
1. 64-bit ulong arithmetic is miscompiled, so every weight and scale read
hit the wrong address - the primary cause, and why q5_K (int offsets)
was unaffected. The block index is computed in int and widened only
inside the pointer expression.
2. The vectorized dequant (int4/float4 bit-ops, convert_*4, dot()) is
miscompiled; the 6-bit weights are reconstructed and the dot done
scalar.
3. vload4 of the f32 activations is miscompiled; replaced by a
scalar-indexed load.
4. The accumulation is miscompiled unless a side effect forces the partial
sums to materialize. A printf under a guard the compiler cannot prove
false acts as a zero-cost optimizer barrier; its placement is
load-bearing.
The defect tracks the compiler, not the GPU generation: it reproduces on
E031.38 (Adreno 642L) and E031.41 (Adreno 740) and is fixed by E031.45
(Adreno 619), so the workarounds are gated on the compiler version. Where
they are not needed they cost real throughput - 42.4 -> 35.1 GFLOPS on an
Adreno 840 q6_K GEMV. The explicit compiler-type check is required, not
redundant: newer_than_or_same() is false for every non-E031 compiler, so
negating it alone would enable the workarounds on E17 and DX.
test-backend-ops MUL_MAT is 919/919 on the Adreno 740, 642L, 619, 840 and
850; the 740 and 642L were 909/919 before. The 642L additionally needs the
A6X per-kernel-program support to reach these tests at all.
* ui: Extract server stream lifecycle from chatStore into ChatStreamManager
Discovery, attach/replay, resume retry and the remote-running snapshot
formed a cohesive cluster inside chatStore. It now lives in
chat-streams.svelte.ts as ChatStreamManager, owned by chatStore, which
keeps the public entry points as delegates so components are
unchanged. chatStore: 2877 -> 2418 lines.
* ui: Extract user interaction gates from agenticStore into AgenticGates
Tool permission requests, turn-limit continue prompts and queued
steering messages are the state the loop waits on between turns. They
had no coupling to session state, so they now live in
agentic-gates.svelte.ts; agenticStore keeps delegates so components
are unchanged. agenticStore: 1196 -> 1073 lines.
* ui: Compose MCP resources under mcpStore.resources
Resource state was a second import scope next to mcpStore. Consumers
now go through mcpStore.resources, so the MCP surface is one store;
mcp-resources.svelte.ts stays a separate file owned by mcpStore.
* ui: Reorganize stores into domain namespaces
* fix: Update stale doc comments
* ui: Consolidate conv running-state into a chat activity ledger
Running-state was split across chatStore.chatLoadingStates (local
pipes), ChatStreamManager.remoteRunningConvs (backend sessions) and
attachingConvs (attach lifecycle), unioned by hand in
getAllLoadingChats and cross-cleaned by setChatLoading calling
streams.clearRemoteRunning - the 'spinner ghosts until tab toggle'
workaround.
chatActivityStore now owns both sets with one transition per event:
markLocal / localEnded (local pipe end also drops the stale remote
hint, no cross-owner call) / applyRemoteSnapshot (diffed). The
sidebar reads chatStore.activity.loadingConvs through the unchanged
getAllLoadingChats entry point.
Consequences:
- isStreamingActive and its five manual writers are gone; isStreaming()
now reports whether the active conversation has a live streaming
pipe, which is what all four consumers (assistant row, stop action,
context gauge, chat screen) actually check
- isLoading/isReasoning become derived from the per-conv maps plus
the active conversation, dropping the manual resync in
syncLoadingStateForChat and clearUIState
- attachingConvs and the last-attach coordination disappear from
ChatStreamManager
- getAllStreamingChats (no consumers) is removed
* ui: Give store collaborators narrow host interfaces
Collaborators took 'host: typeof <store>', i.e. the store's entire
public surface, which is how chatStore's streamChatCompletion,
createAssistantMessage, getApiOptions and setStreamingActive got
widened to public. Replace with per-collaborator interfaces carrying
only the members each one drives:
- ChatStreamHost (chat/streams) - activity, processing, streaming
states, abort controller, loading/streaming setters
- ChatFlowsHost (chat/flows) - streaming core, message creation,
per-conv state setters
- McpHealthHost (mcp/health) - connection registry + reconnection
- ModelPropsHost / ModelStatusHost (models) - model rows, feed
updates; the managers write modalities/status back onto the host's
rows, so those members stay writable
- ConversationsPreferencesHost (conversations) - the active row and
the conversation list
The store classes now declare 'implements <Host>' so the contract is
visible at the class level, and the 'import type { <store> }' back
references in the collaborators disappear entirely - the host
contract is local to each collaborator file, and collaborators can
no longer reach around their slice. Members stay public (structural
typing), but the collaborator side is now compiler-enforced.
* test: Chat Activity store test
* refactor: Cleanup
* chore: Remove legacy architecture docs
* ui: Memoize findMessageIndex for the streaming hot path
Streaming looks up the same message index on every chunk, a linear
scan of activeMessages each time. Cache the last lookup and reuse it
after validating the id still sits at the same position (O(1)); any
structural change to the array fails validation and falls back to a
full scan.
* ui: Throttle per-chunk stream state writes to localStorage
saveStreamState ran JSON.stringify + a synchronous localStorage.setItem
on every decoded chunk of the stream. The read loop now goes through a
new saveStreamStateThrottled (one write per conversation per 500ms,
latest value held pending); the public saveStreamState keeps its
immediate-write contract for stream start and pre-fetch, and also
resets the throttle window.
A pending offset is force-flushed at resume boundaries (resumeStream
reads the offset back from localStorage), on visibilitychange->hidden
and on pagehide, so a reload always finds a usable offset. The resume
offset only needs to be roughly current since the server retransmits
from a line boundary and the client discards its partial line.
Adds unit tests for the throttled/flush/clear interplay.
* ui: Compute context gauge timing stats in one pass
currentRead/Fresh/Cache/Output were separate deriveds, each running a
full reverse scan of activeMessages for the last assistant timings,
and cumulative ran its own forward scan plus an agentic filter - 4-5
O(n) passes per chunk while streaming. Replace with a single
summarizeAssistantTimings() pass (last assistant timings, last
agentic llm totals and the cumulative sums) feeding a shared derived
snapshot. Semantics unchanged, including the live-stats overrides and
the agentic llm-totals branch.
* agentic : clear session state when a conversation is deleted
Every conversation that ran an agentic flow left an AgenticSession in the
store forever; clearSession was never called. conversationsStore now
notifies deletion listeners and agenticStore drops the matching sessions,
avoiding a circular import back into conversationsStore.
* chat : extract ChatService.normalizeMessagesForApi
The DB->API message normalization (convert + drop empty system messages)
was duplicated in sendMessage, preEncode and the agentic flow. Extract it
into one shared method and call it from all three.
* sse : share record splitting and data extraction
splitSseRecords and extractSseDataPayload centralize the record-boundary
splitting and data: line extraction used by parseSseJsonStream and the
models status feed. chat.service keeps its own line-based parser for
resume support.
* api : delegate apiFetchWithParams to apiFetch
apiFetchWithParams duplicated apiFetch's headers/fetch/error handling
body-for-body; it only differs in URL construction. Build the URL and
delegate.
* chat flows : dedupe title, timings and cleanup handling
- conversationsStore.applyTitleFromContent centralizes the title-from-first-
message logic duplicated in 5 places
- ChatProcessingStore.applyStreamTimings centralizes the onTimings handler
shared by the chat and continue flows
- host.cleanupStreaming centralizes the loading/streaming/processing reset
repeated across the continue flow's exit paths
* conversations : centralize conversation update mirroring
rename, pin, mcp override, reasoning effort and cwd all repeated the same
write-DB-then-mirror-into-list-and-active dance. A single
applyConversationUpdate(id, updates) on the host collapses all five and
removes the forgot-to-mirror-one-field bug class. Drops the redundant
array reassignment in setCwd (deep field assignment is reactive).
* mcp : dedupe tool execution, server parsing and tool indexing
- executeTool delegates to executeToolByName (only diff was argument parsing)
- drop the private #parseServerSettings copy; use parseMcpServerSettings
- cache getServers() keyed on the raw config value (hot path)
- indexServerTools() unifies the three identical toolsIndex rebuild loops
Assisted-by: Claude
* mcp : share cursor pagination and tool indexing
- MCPService.paginate() collapses the identical do-while loops in
listAllResources and listAllResourceTemplates
- promoteHealthCheckToConnection now uses indexServerTools like the other
connect paths
Assisted-by: Claude
* database : share message parent-child bookkeeping
- addChildToParent() dedups the append-to-children update in createMessageBranch
and createSystemMessage
- removeChildFromParent() dedups the remove-from-children cleanup in deleteMessage
and deleteMessageCascading
- bulkAdd the cloned messages when forking a conversation instead of one add
per message
Assisted-by: Claude
* chore: Lint/format
* fix: `pagehide` event from `window`
* refactor: Api Fetch util
* docs : rewrite architecture sections in README
Update the high-level diagram, routes, hooks, stores, services and data
flow tables to match the current UI structure (mcp/settings/search
routes, agentic/tools/mcp stores, MCPService/ToolsService/SandboxService,
/tools API). Fix stale architectural patterns for per-conversation state
and modality validation.
* chore : add ESLint rule for blank lines between accessors
Enforce a blank line between consecutive class accessors. The core
padding-line-between-statements rule does not cover class members, so a
local rule is needed.
* refactor : reorder store members and unify naming
Order store class members as public fields, private fields, constructor,
getters, public methods, then private methods. Normalize private naming
to the `private` keyword (drop `#` and the `_` prefix where there is no
matching public getter). Rename conversationsStore.init() to
initialize() to match the other stores.
* refactor : prefix lookup methods with get in agentic and chat stores
Unify bare-name lookup methods with the get* prefix used across the
other stores (mcp, models, tools, settings). Renames currentTurn,
totalToolCalls, lastError, streamingToolCall, executingToolCallId,
pendingPermissionRequest, pendingContinueRequest,
pendingSteeringMessageContent, pendingSteeringMessageExtras in the
agentic store and pendingMessageContent, pendingMessageExtras in the
chat store. Updates the two consuming components and a doc comment.
* refactor: Clean up comments in stores' and services' code
* chore : add ESLint rule for class member ordering
Enforce structural order (public fields -> private fields -> constructor ->
getters -> setters -> public methods -> private methods) with alphabetical
sorting within each group via perfectionist/sort-classes. Dependency
detection keeps Svelte $derived fields in a valid dependency order instead
of alphabetizing them, since Svelte rejects forward references.
Assisted-by: Claude
* refactor : reorder class members to match new ESLint rule
Apply the sort-classes rule across stores, services, hooks and utils.
Pure reordering - verified no logic changes by comparing sorted line
multisets before/after. All tests and svelte-check pass.
* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var
* feat: load mmproj device backend immediately, add -mmdev shortflag
* fix: its a pointer now get the name
* clean up
* gen docs
* nits
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* CI: Use LLVM's OpenMP over MSFT_DEBUG_non_redist on Windows
Currently, we ship the non-redist debug version of microsoft's libomp.
This PR changes this to official LLVM's release, also packaging
the license as needed.
* Remove LLVM SHA from job name to increase legibility
* Add temp validations to CI
* Revert "Add temp validations to CI"
This reverts commit eef97c88b5bac280803ebb3c3b7bb09f89b0fd88.
* Build OpenMP in CI
* Make OpenMP fetch self-contained in cmake and cache in CI
* Robustify Licens-packaging
1. Ship OpenMP license, not LLVM's.
2. Invalidate cache also on checksum of the license
* Remove stale reference in docs/build.md
* No longer package base license in release
This was scope-creep
* Add explanatory comment to OpenMP license
* Remove arm64 smoke
Forgot this during conflict resolution during rebase of
c54c0e9cf6030a5a54ce8bdd81b3e146d9787d42
* Remove GGML_OPENMP_FETCH_CACHE_DIR as requested by @CISC
* whitespace changes
* CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover
Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover
(default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the
threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core
path, which is faster once quantized decode becomes compute-bound at
B>1 (measured +23-41% at B=8 on RTX 5090 for Q4_K dense, no low-batch loss).
The value is parsed once and clamped to [1, MMVQ_MAX_BATCH_SIZE], since
mul_mat_vec_q asserts ncols_dst <= that; invalid input warns and falls
back to the default. The override is applied consistently in both the
mul_mat_vec_q and MUL_MAT_ID dispatch paths. Default behavior unchanged.
* Added Blackwell specific switch point, to reduce dependence on runtime env var.
* Add per-HW switch point values for DGX Spark and removing runtime env var
* Adding switch points for Ada, tested on RTX 4090
* Modifying DGX Spark numbers based on latest run and adding some comments and small functional changes relating to MoE
* Reverting an unnecessary conditional
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* metal: dequantize q8_0 KV to f16 before flash attention
Add a preprocessing pass for GGML_OP_FLASH_ATTN_EXT on the Metal backend:
when the KV cache is quantized (Q8_0 for now), dequantize K and V into a
contiguous F16 scratch buffer and run the existing F16 flash attention
kernels on it, instead of the in-kernel dequantization path.
- new kernel kernel_flash_attn_ext_dequant_to_f16<block_t, QK, deq_t4x4>:
one thread per quant block (K then V), stride-aware so permuted KV is
supported; instantiated for Q8_0 (extending to Q4_0/Q4_1/Q5_0/Q5_1 is
one instantiation + one gate case)
- the gate is type-only: dequantize whenever the KV is quantized,
regardless of head sizes, GQA ratio or n_kv; the attention kernels
themselves are untouched
- the F16 copies live in the op's own scratch allocation
(ggml_metal_op_flash_attn_ext_extra_dequant_f16); the KV pad kernel
reads the dequantized buffers when the path is active
- the FA pipeline getters gain a use_f16_kv flag selecting the existing
f16 kernels and contiguous strides
- ref: https://github.com/ggml-org/llama.cpp/pull/25556
Verification (M2 Ultra):
- test-backend-ops test -o FLASH_ATTN_EXT: 4798/4798 pass, including the
new q8_0 eval cases (decode/prompt, permuted, sinks+ALiBi+softcap,
kv=113 pad path, kv=16384)
- llama-perplexity on Qwen2.5-0.5B with -ctk q8_0 -ctv q8_0 matches the
f16 KV reference (PPL 1.0008 vs 1.0008)
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : launch the FA KV dequant kernel separately for K and V
Simplify kernel_flash_attn_ext_dequant_to_f16: it now dequantizes a single
tensor (its own ne/nb and dst) with no is_v branching, and the op dispatches
it twice with the same pipeline - once for K and once for V. The kargs
struct shrinks to a single ne/nb set plus nblocks.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : dequantize q4_0, q4_1, q5_0 and q5_1 KV to f16 before flash attention
The dequant pass now covers all quantized KV types supported by the Metal
flash attention kernels. The dequant kernel, kargs, scratch allocation and
dispatch are type-generic, so each type is one kernel instantiation plus one
gate case.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : skip the redundant V dequant when V is a view of K
In MLA-based models, the V of the FA op is a view of K (the first ne20
elements of each K row); the dequantized V is then a view of the dequantized
K, so skip the second dequant dispatch, do not reserve the V scratch region,
and let the pad and attention kernels read V from the K F16 buffer with K's
strides. The detection follows the CUDA backend:
V->view_src && (V->view_src == K || (V->view_src == K->view_src && V->view_offs == K->view_offs))
Also fix the FA pipeline getters: ns10/ns20 are function constants baked into
the kernels and must be the actual K/V row widths as seen by the kernel. The
dispatch now passes them explicitly (nb11_attn/nb10_attn, nb21_attn/nb20_attn)
instead of the getters assuming contiguous F16 KV (ns20 = dv), which was wrong
when V is read from K with K's row pitch (e.g. 576 vs 512).
New test cases: 576/512 q8_0 (MLA shape, V is a view of K) at kv=113 (KV pad),
nb=1 (vec) and nb=64 (non-vec).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* test : remove backend-specific wording from test-backend-ops comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : avoid backend mentions in test-backend-ops comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : rename the FA dequant_f16 identifiers to kv_f16
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : clean-up
* cont : remove TODO
* ggml: fix backend split scheduler race condition
splits without input were running concurrently with other splits, while potentially reusing memory the other split is accessing
* only sync when split has no inputs
* convert: fix get block count error for Nemotron
Signed-off-by: Rock Chen <rockchen.tw@gmail.com>
* fix this in NemotronHModel.__init__ instead.
This reverts commit ca689cbc8792ba69ea1fd5d8b3ae0485c47536ec.
---------
Signed-off-by: Rock Chen <rockchen.tw@gmail.com>
* provide static workspace for cuBLAS handles
* account for concurrent streams when using GGML_CUDA_GRAPH_OPT
* drop cublas_handle overloads and remove direct cublasSetStream calls
* Update ggml/src/ggml-cuda/common.cuh
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
build_attn with the llm_graph_input_attn_k_iswa input was using the cached K
tensor itself as V. Create V as a view of K (the first v_cur->ne[0] elements
of each row), like the other K-only build_attn overloads.
The deepseek4 MTP call site now passes the kv tensor as v_cur.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* backend: propagate buffer usage in meta backend
* ggml-meta: make sure to call init_tensor for all new tensors
* meta: remove explicit check for meta backend in ggml_backend_meta_get_split_state
I can't seem to reproduce the original failure in the latest code.
* hexagon: fix FA HMX queue ordering in the pipelined path
* hexagon: double buffer D matrix, store diagonal tile only
* format code
* align the indentation
* opencl: port fused ssm_scan kernel (Mamba-2, d_state in {128, 256})
Fold the fused per-token SSM_SCAN recurrent step from opencl/gdn-qwen36-35b
onto the unified base. Previously SSM_SCAN fell back to CPU here; now scalar-A
Mamba-2 with d_state in {128,256}, all-f32, runs on GPU. Other shapes (incl.
Mamba-1 element-wise A) still fall back. test-backend-ops -o SSM_SCAN passes on
Adreno X2-90. opt-out via GGML_OPENCL_DISABLE_SSM_SCAN=1.
* opencl: cleanup
* opencl: require K == 1
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* ggml-cpu: gate __fp16 on __ARM_FP16_FORMAT_IEEE
__ARM_NEON only signals NEON availability. The __fp16 type also needs
the IEEE half format, implied on AArch64 but selected with
-mfp16-format=ieee on 32 bit Arm, where the compiler otherwise rejects
the type.
The guard keeps every toolchain that provides the type on the same code
and sends that one configuration to the generic lookup path.
* ggml-cpu: gate the NEON+FMA block on __ARM_FP16_FORMAT_IEEE
Both halves of the F16 section dereference __fp16, so armv7 with
neon-vfpv4 hits the same unknown type error. Without the IEEE
format the configuration now falls back to the scalar path.
Address review from @JonathanC-ARM
* vulkan : dequant q8_0 KV once in coopmat1
Assisted-by: Claude (Opus 4.8)
* vulkan : fall back instead of aborting when FA scratch exceeds maxStorageBufferRange
* vulkan : require KV-cache layout in FA dequant path
Assisted-by: Claude (Opus 4.8)
* vulkan : skip FA dequant path on coopmat2
Assisted-by: Claude (Opus 4.8)
* tests : add contiguously-allocated quant K/V FA tests
Assisted-by: Claude (Opus 4.8)
* vulkan : trim comments
* vulkan : tighten permutation checks for FA path
* vulkan : set prealloc_x_need_sync after the FA dispatch
* vulkan : exclude Intel Xe1 from FA dequant path
* feat(convert): Add conversion for GraniteSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(llama): Add granite_swa support
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add conversion infra for rope_pattern array
NOTE: There is other work also targeting this, so this may be
removed depending on merge order.
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(conversion): Fix SWA pattern logic and support for non-rope layers
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add support for GraniteMoeSWA
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add llama_hparams::has_rope and arch constants
NOTE: This shadows the work done for Granite Speech
https://github.com/ggml-org/llama.cpp/pull/25107
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add support for per-layer rope determination
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Fix failing flake8 for extra newlines
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver
Branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Load MoE params as optional
Branch: GraniteSWAForCausalLM
AI-usage: draft (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Handle MoE params in conversion
branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Remove unnecessary newline
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Remove unnecessary tensor additions to GRANITE architecture
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Correctly handle naming for ffn gate inp
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Always default hparams.rope_pattern to 1s
This isn't strictly necessary, but it will allow other models to rely on
hparams.has_rope(il) without needting to prepopulate.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Move to has_rope for all granite model architectures
Now that we have a proper hparam for this, it's better to use it and not
require a hacky fallback in the hparam method itself.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: No hacky rope_finetuned fallback in has_rope
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fully remove rope hparam filling in granitemoe
There are no granitemoe models that use NoPE (it's not actually used in the
layer building below), so this was just dead code.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Save out rope_pattern in model-saver
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Set hparams.rope_finetuned for round trip
Since the value is _read_ from rope_finetuned, we need to persist it when
the model is saved with the saver.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Code review cleanup
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* refactor: Keep gate/up fused for MoE path
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Skip GRANITE_SWA in model saver
https://github.com/ggml-org/llama.cpp/pull/25505#discussion_r3773175651
Keeping is_swa_impl in the saver can break other models.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* add sliding window pattern for model in test
* style: Fix indentation
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fix \r\n
Thanks Claude!
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Keep shared expert fused
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: More indentation fixes
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* add params
* cpu kernel
* metal kernel
* add test backend ops
* gate other backends
* ggml: (cuda) support ggml_rope_set_offset (#27121)
* rm cuda supports_op guard, fix webgpu clang-format
* ggml: support ggml_rope_set_offset on vulkan (#27344)
* ggml: support ggml_rope_set_offset on vulkan
* remove inplace optimization
The route loads run ahead of the root layout script, so validateApiKey
read the settings store while it still held factory defaults and probed
/props without the stored key. initStores() now hands the same startup
promise to every caller and the chat loads await it before probing.
The one-time admin baseline no longer overwrites a key the user has
already set: on a first visit the config carries factory values only, so
a diverging key comes from the user and wins.
* vulkan: tiled transpose for 0<->2 permuted CONT
-ggml_vk_get_cpy_pipeline only routed to the tiled shared-memory transpose
shader when dim1 was the innermost dimension, i.e. ggml_transpose (a 0<->1
swap). A 0<->2 swap -- ggml_cont(ggml_permute(x, 2, 1, 0, 3)) -- fell back to
the generic per-element strided copy, whose source reads stride by ne0*ne1
elements: one cache line per lane.
-DeepSeek-V4's lightning indexer performs exactly that permute on a
[n_kv, n_tokens, n_head] tensor. On Vulkan/RADV gfx1151 it ran at ~1-9 GB/s of
a ~200 GB/s part and accounted for 43% of total prefill time.
-Add copy_transpose_02.comp, mirroring copy_transpose.comp but tiling over dst
dims (0, 2) with dims 1 and 3 as the batch, so reads walk src dim2 and writes
walk dst dim0 -- both contiguous. The selection condition additionally requires
a non-contiguous source and a contiguous destination so it cannot take cases
the contiguous-copy shader already handles.
-test-backend-ops only exercised ggml_transpose for CONT, so the strided path
was untested. Add test_cont_permute covering (2,1,0,3), (1,2,0,3) and (0,2,1,3)
over f32/f16 at tile-aligned, tile-unaligned and large shapes. The large shapes
are in the eval set rather than only in perf because perf mode does not verify
results.
-Measured on gfx1151, ne=[n_kv,64,64,1], perm=(2,1,0,3), f32:
n_kv=1024: 9.08 -> 579.85 GB/s
n_kv=1280: 20.03 -> 153.71 GB/s
n_kv=2048: 7.11 -> 91.68 GB/s
n_kv=2304: 16.24 -> 86.49 GB/s
-The ~2.2x penalty previously seen at power-of-two n_kv (destination-stride
aliasing) is gone. End to end, DeepSeek-V4-Flash IQ3_XXS prefill on a 9k-token
prompt goes from 56.33 t/s to 103.74 t/s (+84%).
-Note: at n_tokens=512 a single slow-path dispatch takes ~273 ms and looping it
in perf mode can trip the GPU watchdog, so the perf cases use n_tokens=64.
* tests: fold test_cont_permute into test_cont, add L2-exceeding perf shapes
Review feedback: test_cont gains a permute parameter ({0,0,0,0} = none),
matching test_mul_mat's pattern, and the separate struct is gone. Perf
adds [n_kv, 512, 64, 1] variants (~0.5 GB per run) that exceed GPU L2,
since the 64-token shapes fit in cache on large parts and read above
memory bandwidth.
* tests: trim perf-case comment to the two-line summary
* vulkan: trim comments on the 0<->2 transpose path
Drop the shader file header, the read/write block comments and the
rationale prose in the CONT test cases. Keep the tile-shape and
bank-conflict notes and the permute parameter documentation.
---------
Co-authored-by: Kevin Hopper <no-reply@maestro.press>
* gguf-py : add size guards to GGUFReader
Guard kv_count, tensor_count, string length, and array length
against crafted values that cause unbounded allocation or hangs.
Assisted-by: opencode
* gguf : validate tensor data section fits within file
When no_alloc=true, gguf_init_from_reader accepted files where the
tensor data section (computed from header claims) exceeded the remaining
file size. This allowed crafted GGUF files to pass validation while
having insufficient data, leading to OOB reads when the loader later
mapped tensor data from the file.
Assisted-by: opencode
* gguf-py : move size limits into gguf_reader.py
Per review feedback, the limits are not part of gguf.h but are
arbitrary limits defined in gguf.cpp, so define them locally in
the reader instead of exporting them from constants.
Assisted-by: opencode
* remove gguf.ccp changes
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ui: Move stream lookup and replay fetches into ChatService
chatStore called fetch() directly for /v1/streams/lookup and the
/v1/stream replay. These now live next to the other stream-session
methods in ChatService, so services stay the only API I/O layer.
* ui: Move /models/sse feed reader into ModelsService
ModelsService.watchModelEvents owns the byte stream, reconnect loop
and SSE record parsing; modelsStore keeps only event routing and
state.
* ui: Extract conversation import/export into ConversationTransferService
The JSONL session format, ZIP archiving and browser downloads are
pure I/O with no store state, so they move out of
conversationsStore. The store keeps the DB orchestration
(bulkExportConversations, downloadConversation,
importConversationsData) and delegates the format work.
* ui: Consolidate active model resolution into modelsStore.activeModelId
The same resolution chain was duplicated in useChatScreenActiveModel,
ChatForm, ChatFormActionModels and contextStatsStore, with slight
drift in the single-model fallback. The canonical getter now lives in
modelsStore, and the shared last-assistant-model lookup moved to
utils as getConversationModel.
* ui: Initialize stores explicitly via initStores()
Store constructors and module-level side effects ran migrations and
localStorage reads in import order. Migrations rename and rewrite
localStorage keys, so a settings load racing ahead of them could
clobber migrated values. initStores() is called once from the root
layout and runs migrations first, then the stores that read
localStorage, then the conversations DB load.
* refactor: Constants for stream query params
* ui: Remove dead code from stores
- persisted() helper was exported but never used
- messageUpdateCallback / registerMessageUpdateCallback were never wired up
- conversationsStore.initialize() alias, single caller moved to init()
* ui: Merge device, theme and viewport into a single deviceStore
All three are reactive browser-environment signals, now exposed as one
class store: deviceStore.isMobile, deviceStore.isIOSDevice / isIOSSafari
/ isWKWebView / isStandalone and deviceStore.systemTheme.isDark. The
systemTheme name disambiguates the OS preference from the user theme
preference in settingsStore. Drops the unused viewport export (only
isMobile was consumed).
* ui: Merge build info into version store
One VersionStore class with build (llama.cpp build number from
build.json) and frontend (PWA version from _app/version.json),
matching the class pattern of the other stores.
* ui: Colocate context gauge popup state with its components
The gauge popup state is local UI state shared only by the
ChatFormContextGauge subtree, so it lives next to its consumers
instead of the app-scope stores barrel.
Kernel is a port of `ggml-cuda/fwht.cu`
(us/run, median):
```
m x n x k GEMM FWHT speedup
64 x 1 x 64 10.20 2.93 3.48x
64 x 2048 x 64 10.75 2.71 3.97x
128 x 1 x 128 10.33 2.88 3.59x
128 x 32 x 128 9.20 2.77 3.33x
128 x 2048 x 128 16.46 2.76 5.95x
256 x 1 x 256 10.19 2.77 3.68x
256 x 2048 x 256 16.69 3.41 4.89x
512 x 2048 x 512 54.16 12.89 4.20x
```
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
* update to ov-2026.3, update device drivers
* ci: skip nemotron-h rollback test on OpenVINO
The OpenVINO backend does not support SSM_SCAN, so the Nemotron-H recurrent state rollback graph is split and cannot preserve the recurrent cache output shape. Keep the test enabled for other backends and retain the qwen35 OpenVINO rollback coverage.
---------
Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
* xcframework : fix build
* mtmd : remove unused include path
* vendor : use vendor::hash alias target in cmake
CMake reserves "::" in target names for imported/alias targets, so the real
target keeps the name vendor-hash and a vendor::hash ALIAS target is added.
Consumers (mtmd, llama-gguf-hash) now link against the namespaced alias.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* vendor : add cmake targets for all vendored libs with vendor:: aliases
Add INTERFACE targets for the header-only vendor libs (miniaudio, nlohmann,
sheredom, stb) and ALIAS targets named vendor::<lib> for all of them,
including cpp-httplib and hash. Each exposes the vendor/ root so includes
are namespaced, e.g. <nlohmann/json.hpp>.
Consolidate the per-lib add_subdirectory calls into a single
add_subdirectory(vendor), keeping the cpp-httplib gate on LLAMA_BUILD_COMMON.
Consumers (llama-common, mtmd) now link the aliases instead of relying on
raw vendor/ include paths.
hash: consumers now include via "hash/hash.h"; the vendor/hash dir is kept
as a PRIVATE include so the synced upstream sources compile unmodified.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* readme : use foo/bar names in acknowledgements
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ocd : fix valign
* ci : create pre-release with change log and nightly link in make-release
After pushing the tag, create a pre-release using
ggml-org/action-create-release. The release description is generated by
scripts/make-release-desc.sh: the change log between the current and
previous version (one line per commit), a link to the corresponding
nightly build when it exists, and a note that semantic versioning is
still work in progress.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cmake : bump version to 0.1.2
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : find the nightly tag by commit in make-release-desc.sh
The nightly release is guaranteed by the release checks to point at HEAD,
so instead of reconstructing its name (commit count, branch, hash) just
pick the b* tag pointing at HEAD. This also drops the RELEASE_BRANCH env
var from the workflow.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : resolve the release commit from the version tag in make-release-desc.sh
The change log and nightly lookup now use the commit the version tag
points at (HEAD when the tag does not exist), instead of always HEAD.
This makes the script usable locally for older versions, e.g.
./scripts/make-release-desc.sh v0.1.1. The tag is resolved to a SHA
first, since --points-at does not peel annotated tags.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : normalize the version argument in make-release-desc.sh
Accept the version with or without the leading v (0.1.1 == v0.1.1) and
reject anything else, instead of silently treating a bare version as a
non-existent tag name.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : clean-up
* CUDA: MMVQ nwarps=8 for bs=1 for dense models on DGX Spark
Signed-off-by: ynankani <ynankani@nvidia.com>
* skip moe experts and allow others based on k geometry (allow only small idle tail)
Signed-off-by: ynankani <ynankani@nvidia.com>
* rename MMVQ DGX Spark params to GB10 and fix MSVC constexpr lambda capture
Signed-off-by: ynankani <ynankani@nvidia.com>
---------
Signed-off-by: ynankani <ynankani@nvidia.com>
* doc: document MCP stdio servers and CORS defaults in the server README
The MCP arguments were listed but nothing explained what an MCP server
is or how to declare one. Cover the stdio transport, the config keys,
the tool naming, and add a POSIX shell echo server as a minimal
example.
Also document the CORS behavior: the default reflected origin, the
switch to localhost once tools are enabled, and the recommended setting
per deployment.
* doc: drop the inline MCP shell example from the server README
The example parsed JSON-RPC by hand and sat in a page people copy paste
from, into servers spawned with the privileges of llama-server. Point to
the specification instead.
Link the pull request that introduced the feature, and keep a short
mcp.json snippet so the table of configuration keys has a declaration to
refer to.
Add a "Create and push git tag" step to the release job, right before
the "Create release" step. The tag is created with git tag and pushed
with the deploy key already configured by the Clone step, instead of
relying on the Releases API (action-create-release) to create it as a
side effect.
The tag is lightweight, matching all existing b<number> release tags.
The step is idempotent: if the tag already exists (e.g. on a re-run),
creation and push are skipped.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ui: move get_datetime tool to frontend
* clarify docs
* server: drop the now unused ctime include
strftime() and gmtime_r() were the only users, both went away with the
get_datetime tool. Also make the renderer's catch inert: the browser
executor always emits JSON, so a non-JSON result is no longer a date to
display.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ci : parallelize platform builds in build-xcframework.sh
The ios-xcode release job builds 7 platform/simulator configurations
sequentially, each with -j $(nproc). Run them with at most 3 concurrent
builds (the release runner has 3 cores), splitting the cores between the
builds (-j 1 each on the runner), so total CPU pressure is unchanged while
the build phase runs about 2.3x faster.
- convert the 7 build blocks into functions (flags unchanged)
- add a run_builds_parallel pool: 3-slot sliding window, per-build logs,
dumps the failing log and aborts on error (background job failures do
not trigger set -e)
- queue the 2-arch builds first so the slower builds occupy the slots early
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : add guideline for comments
* cont : disable 5/7 builds
* ci : make build-xcframework.sh builds configurable via CLI args
The script now takes an optional list of builds to run
(ios-sim ios-device macos visionos visionos-sim tvos-sim tvos-device);
with no arguments it builds all of them, as before. The per-build
lists for the build pool, framework setup, static library combining
and xcframework creation are now driven by a single build_spec
lookup instead of four hardcoded (partially commented-out) lists.
release.yml builds only macos and ios-device to cut the build time.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* dspark: support speculators-format checkpoints (SpecForge exports)
Speculators-format DSpark drafts (e.g. SpecForge exports for the
Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in
three ways:
- the config nests the backbone hparams under transformer_layer_config
and gives the extract layers as aux_hidden_state_layer_ids
- the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus
token, not a prediction slot. Written as dflash.bonus_anchor; such
drafts build the block and read the mask positions exactly like
DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence
sampling comes from DSpark
- the draft output vocab may be reduced (draft_vocab_size < vocab_size)
with a d2t remap table. The converter expands lm_head/markov_w2 back
to the full vocab and synthesizes an lm_head bias of -1e9 on the rows
the draft cannot produce, so the runtime needs no d2t remapping. Such
drafts ship their own (now optional) token_embd/output tensors instead
of sharing the target's
Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical
with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the speculators draft class into DSparkModel
One class now covers every DSpark variant. What used to pick the class is
a single flag, because the arch name turns out to be the only thing that
separates the two families: SpecForge also exports a flat schema that
carries no speculators_* fields yet still uses the 1+N bonus-anchor block,
so keying on those fields would silently mis-read its drafts.
Also rename i0 to i_first_pred in the draft read loop and the Markov head,
and give the head a real bonus_anchor bool instead of testing i0 > 0.
Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts
produces byte-identical GGUFs. The one behaviour change is that the
markov_head_type check now also covers the DeepSpec checkpoints, which
previously skipped it.
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Opus 5
* dspark: address review comments
- rename bonus_anchor to sample_from_anchor (GGUF key and code), matching
the checkpoint config field; absent key still means anchor-first
- rework the reduced draft vocab to match EAGLE3: d2t is written as I64
absolute target ids and the logits are scattered at runtime, instead of
expanding lm_head/markov_w2 and synthesizing an output bias at conversion
- move the t2d skip to modify_tensors, like EAGLE3
- drop _is_specforge: the arch name only picks the sample_from_anchor
default, embed/lm_head sharing is decided by the draft vocab size
- deduplicate the tok_embd create_tensor left behind by the rebase
Verified with the RedHat gemma-4-31b speculator draft: greedy output is
byte-identical with and without the draft; acceptance 0.26 (n_max 7).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the sample_from_anchor read into the block_size block
* dspark: fix flake8 continuation indent
* clean up
* dspark: key the sample_from_anchor default off the export format
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: drop t2d in filter_tensors
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: map model.lm_head instead of bypassing the dflash prefix
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
---------
Co-authored-by: desovo7 <942845546@qq.com>
Co-authored-by: ruixiang63 <wangruixiang07@outlook.com>
* ci : allow make-release to target a specific commit
The make-release workflow now accepts an optional 'commit' input. When
set, that commit is checked out and the release checks verify that it
belongs to the branch selected in the "Run workflow" dialog and is not
older than 3 days from the branch tip. The check is part of
make-release-checks.sh (driven by the RELEASE_BRANCH env), so it follows
the same dry-run semantics as the other checks.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : scan latest 100 relase workflow runs
* cont : do not check manually for release.yml success
* Adding support for bailingmoe3
* Adds speculative decoding support
* Make BailingMoE3 safe gate metadata optional
* bailingmoe3: apply trained SwiGLU clamps
* common: fix Bailing V3 tool argument parsing
* llama-model-saver, instantiate float vector metadata writer
* bailingmoe3: support Q-LoRA (Ling-3.0-tiny)
Ling-3.0-flash sets q_lora_rank: None and projects Q directly, so the current
implementation loads a single ATTN_Q tensor. Ling-3.0-tiny sets q_lora_rank: 256
and routes Q through a LoRA bottleneck instead:
q_a_proj -> q_a_layernorm -> q_b_proj
Conversion therefore failed with:
ValueError: Can not map tensor 'model.layers.3.attention.q_a_layernorm.weight'
Add the missing path, mirroring the existing deepseek2 MLA implementation:
* constants.py - add ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM to BAILINGMOE3
* tensor_mapping.py - map model.layers.{bid}.attention.q_{a,b}_proj and
q_a_layernorm
* conversion - emit attention.q_lora_rank when the config has it
* bailingmoe3.cpp - read n_lora_q; create the Q-LoRA tensors and build Q
through the bottleneck when q_lora_rank > 0
Everything is gated on q_lora_rank > 0. Ling-3.0-flash's config has no
q_lora_rank, the converter only emits the key when present, hparams.n_lora_q
defaults to 0, and get_key(..., required=false) leaves the target untouched when
the key is absent - so flash keeps taking the existing direct-Q branch.
The LoRA path produces the same shape as the direct projection, so the
nope/rope split, RoPE application and wk_b absorption downstream are unchanged.
* small mtp change
* bailingmoe3: support separate MTP GGUF and Q-LoRA MTP
* gguf: remove duplicate add_kda_gate_lower_bound definition
---------
Co-authored-by: bloomer <bloomer@booper.brushtail.me>
Co-authored-by: Dyluhn <dylanranejohnston1@gmail.com>
The log output does not append a newline, so the warning ran into the
next line printed on stdout, corrupting the benchmark table header.
Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Adjusts the thread/block count to be proportional to the size
of the quant, reducing under/over subscription.
Largest perf improvement is the q4_0 -> f32 path, with, on
a Arc 70, throughput goes from 20.21 GB/s to 158.19 GB/s
The rest of the quants are flat in performance uplift.
* ui: mask API Key field in settings and error splash to stop browser autofill
* ui: set autocomplete=new-password on private fields
The password input type makes browsers offer to save the API key
in the password manager and autofill saved site credentials into
the field. The new-password autocomplete value disables both.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* model: add Kimi-K3 text model
Hybrid KDA (linear) + MLA (full) attention as in Kimi-Linear-48B, plus five
things that architecture does not have:
1. cross-layer residual attention (attn_res_block_size)
2. latent MoE (routed experts run at n_expert_latent)
3. situ activation (replaces SwiGLU everywhere)
4. MLA output gate (sigmoid gate before o_proj)
5. full-rank KDA gate (single ssm_g instead of ssm_g_a/ssm_g_b)
K3's text_config reports KimiLinearForCausalLM - the older 48B architecture -
so get_model_architecture routes on the top-level name instead.
The KDA decay gate has two forms, selected by linear_attn_config's
gate_lower_bound. It is not a clamp: when set it swaps the activation entirely
(fla/ops/kda/gate.py), from -exp(A_log)*softplus(x) to
lower_bound*sigmoid(exp(A_log)*x). K3 sets it to -5.0; kimi-linear leaves it
unset, so that path is unchanged.
Cross-layer residuals reuse ggml_dsv4_hc_pre for the weighted sum. That op is
CPU + CUDA only, so Metal/Vulkan will fall back per-node until those kernels
exist.
The routed experts ship as compressed-tensors "mxfp4-pack-quantized". That is
bit-compatible with ggml's MXFP4 - same E2M1 code assignment, same E8M0 scale
byte, only the nibble positions within a block differ - so they are repacked
rather than dequantized, losslessly and without a ~5.5 TB bf16 round-trip.
The repack is built lazily because gguf_writer holds every added tensor until
the final write. DeepSeek-V4 was already doing the identical bit-shuffling, so
it now shares the helper.
Verified against Moonshot's own code path (transformers + fla's Triton KDA
kernels) on a tiny model exercising every K3-specific feature. Final-position
logits vs the fp32 reference: 6.7e-05 rel / corr 1.00000000 for both the
chunked and the recurrent delta-net path. MXFP4 blocks dequantize to the source
weights with 0.0e+00 error.
Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* model: fix ty errors in the Kimi-K3 converter
- `_res_parts` buffers (kind, tensor) pairs, not bare tensors
- `get_tensors` must return an Iterator, matching ModelBase
- LazyBase's `func` takes one argument, so pass the expert loaders through
`args` instead of the closure
- borrowing KimiLinearModel.set_vocab from an unrelated TextModel is
deliberate and safe, but not expressible in the signature
No behaviour change: the MXFP4 repack still dequantizes to the source weights
with 0.0e+00 error and end-to-end logits are unchanged (8.386e-03 rel,
corr 0.99996630).
Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update conversion/kimi_k3.py
Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>
* Increase LLAMA_MAX_EXPERTS from 512 to 1024
* tests : support for Kimi K3 in archs test
* chat : add Kimi K3 chat format (reasoning, content, typed tool calls)
K3's assistant output is an XTML-ish tagged format built by the template's
open_tag/close_tag macros. Two properties break generic parsing:
1. The generation prompt ends with open_tag('think'), so the completion
starts inside the think section with no opening marker in the output
(thinking_forced_open).
2. Only <|open|>/<|close|>/<|sep|>/<|end_of_msg|> are special tokens; tag
names ("think", "response", "message") are ordinary text tokens.
Adds common_chat_params_init_kimi_k3 (PEG_NATIVE) with detection on the
marker trio, reasoning extraction, response unwrapping, and tool-call
parsing of the tools/call/argument tag structure with argument types
taken from the tool schema. Includes the K3 chat template fixture and 9
test-chat cases derived from real generations of the full 2.8T model.
Verified end-to-end against Kimi-K3-Q2_K (GrEarl/Kimi-K3-GGUF) on 8x B200:
content, reasoning_content, streaming deltas, and tool_calls all correct;
finish_reason stop/tool_calls as appropriate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chat : add message_delimiters for Kimi K3
Per-role message-start markers for token-level span splitting. User and
assistant messages carry only the role attribute, so their full opener
(through <|sep|>) is used; system and tool messages continue with more
attributes (type=/tool=/index=), so those delimiters stop after the
role's closing quote. Verified against the K3 tiktoken vocabulary that
the closing quote is always a standalone token across all attribute
variants, so the token-level prefix match stays exact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: apply nits from @ngxson and text fixes from @danielhanchen
* tests : added missing hyperparameters and tensors for Kimi K3 in test-llama-archs
* chore : move overly verbose header file comments to Kimi K3 source file
* tests : re-enabled KIMI_K3 in test-llama-archs for WebGPU backend
* model-saver : emit kda_gate_lower_bound for Kimi K3
Quick fix. The Kimi K3 loader reads kda_gate_lower_bound and gates a graph branch on it (it scales the KDA gate when the bound is above -INFINITY), but the model
saver never wrote the key, so a save->load roundtrip silently dropped it back to the -INFINITY default and changed the model's output. The real K3 config sets gate_lower_bound = -5.0.
I propose to emit it from the saver, and set it to -5.0 in the test-llama-archs K3 case so the roundtrip check exercises it (the roundtrip fails without the saver line).
* Refactor conditional for model architecture check
* tests : re-enabled (again) KIMI_K3 and MINIMAX_M3 in test-llama-archs for WebGPU backend
* fix code comments
* add template on conversion
* move repack_mxfp4_blocks to model base
* nits
* add_value_length
* optimize res_stack construction
* nits
---------
Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Deepankar Singh <singh.deepankar39@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Caleb DeLeeuw <caleb.deleeuw@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* common: support --models-dir loading MTP assistant models
* common: preset: check for MTP models with strict prefix
* common: preset: Take advantage of PR #27005
* handle other draft types
* drop eagle3
* clean up
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Replace the deprecated --mmap, --no-mmap, --mlock, and --direct-io flags with
the unified --load-mode argument across scripts, examples, and documentation.
Internal warning message and env var docs updated accordingly.
Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
* llama : support for MiniMax-Text-01 model
* chore : renames to match the other MiniMax models
* model : add logits mask as MiniMax-Text-01 embeddings tensor has zero-valued embeddings for tokens >= 200032 that produce zero logits disrupting the token sampling process
* llama : replace hardcoded conditions with hparams.is_recr()
* model : used build_rs() for recurrent state management
* chore : code cleanup
* model : optimized MiniMax-Text-01 by removing the state tranpose operations
* chore : removed unnecessary ggml_cont() in MiniMax-Text-01 implementation
* llama : add generic logits mask graph input
* model : permuted diag_decay dimensions to avoid doing it inside MiniMax-Text-01 graph
* chore : code cleanup
* chore : code cleanup
* model : use token positions when calculating MiniMax-Text-01 decay tensors
* convert : add support for MiniMaxM1ForCausalLM as it seems to be the same as MiniMaxText01ForCausalLM
* chat : add jinja template for MiniMax-M1
Co-authored-by: QscQ <qscqesze@gmail.com>
* chore : code cleanup
* tests : MINIMAX_01-related fixes
* chore : silence Python lint errors
* vocab : remove unnecessary vocab type
* convert : update MiniMaxText01Model conversion to use yield when modifying tensors
* convert : suppress tokens with zero-valued embeddings during MiniMax-Text-01 conversion
* llama : removed logits mask - no longer necessary as token suppression is used instead
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : override non-working built-in chat template during conversion
* tests : skip arch MINIMAX_01 tests for WebGPU backend (it breaks again)
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: QscQ <qscqesze@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* chat: add reasoning_effort to common_chat_templates_inputs
Store OpenAI Chat Completions reasoning_effort and make it
available to jinja templates (with model specific translations
where required).
Assisted-by: llama.cpp:Muse-Glimmer-30B
* server : fixup reading reasoning effort from body
server_chat_convert_responses_to_chatcmpl already handles conversion of
Responses API reasoning.effort to reasoning_effort
* chat : expose reasoning effort
Assisted-by: Claude Opus 5
* chat : add reasoning_effort to generation_params
Assisted-by: Claude Opus 5
* chat : move reasoning_effort next to enable_thinking
Assisted-by: Claude Opus 5
* cont : mirror preserve_reasoning
* cont : pass context through analyze function
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
* Initial changes for Recurrent state rollback for nemotron for cpu and cuda
* Removing CPU RS rollback. Will enable it in subsequent PRs
* addition of test case
* Removing assert and calling runtime API to check if op is supported
* removing extra API and updating the call sites for K
* replace static cuda detection to runtime fused_op api
* address review comments and fallback when SSM rollback not supprted
* Adding changes for supporting RS-rollback in CPU. Also added test-backend-ops for cpu and cuda
* removing memory manipulation as rs rollback is now supported in CPU
* removing the static probe which is not needed now
* correcting the format
* address review comments
* enabling test for all the backends, unsupported backends will fallback to CPU
* Apply suggestions from code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* choose different graph based on the result of fused_ssm_op is supported or not and also handled memory->n_rs_seq >1 case incase of op is not supported
* Support K > 1 in ssm_scan for all backends
* Fix CI Issues
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Scrub developer-specific /home/<user>/ paths from example docs and test
fixtures so they don't leak into the tree.
- examples/test-cmake/README.md: /home/danbev/... -> /path/to/llama.cpp/...
- tests/test-chat.cpp: /home/jarvis/... -> /home/user/... (input and
expected string kept identical so the parser test still passes)
Co-authored-by: Jim Wu <ywu@xilinx.com>
This commit tries to make the logging of target_layer_ids a bit clearer
and easier to read.
Currently the output generated looks like this:
```console
0.00.468.624 D load_arch_hparams: DFlash extract_layers = [0.00.468.626 D 2, 0.00.468.626 D 6, 0.00.468.626 D 20,
0.00.468.626 D 30, 0.00.468.627 D 42, 0.00.468.627 D 520.00.468.627 D ]
```
With the changes in the commit the output will be:
```console
0.00.522.765 D load_arch_hparams: DFlash extract_layers = [2, 6, 20, 30, 42, 52]
```
* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support
* OpenVINO backend: disable TOPK_MOE op test
* OpenVINO Backend: Add op FILL support
* OpenVINO backend: enable set rows with multi dims
* fix the name missmatch in setrow + view
* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID
* OpenVINO Backend: enable SQR & SQRT
* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer
* OpenVINO backend: enable fallback for openVINO to CPU backend
* OpenVINO backend: fix accurace issue in gemma3n arch test
* fix mpt failed case
* OpenVINO backend: clean nodeinfo
* OpenVINO Backend: enable zero-size copy for view
* add concat ssm_conv in compute_dynamic_dim
enable qwen35
Fix after rebase
remove logging
* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region
* OpenVINO backend: fix CPY op test failed issue
* OpenVINO backend: fix GATED_DELTA_NET op test failed issue
* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph
* handle qwen35 dynamic clearing of cache correctly
* Enable qwen35 dense multi seq
* Fix qwen35 9b gqa
* Fix after rebase
* Disable SOLVE_TRI
* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)
In stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,
head_size]) into the Multiply against the rank-4 cos/sin tables
([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the
OpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage
output (e.g. Phi-3-mini). Lift the data to rank-4 before the split/
Multiply so the operands are equal-rank, matching what the TYPE_NORMAL
branch already does. CPU and stateless paths are unaffected.
Phi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:
before: PPL = 27120.43
after: PPL = 6.2263 (CPU reference: 6.2251)
* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update
* OpenVINO Backenb: remove changes in llama.cpp
* Doc change (use x64 Native Tools Command Prompt for VS)
* Cleaner sentence
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* OpenVINO Backend: cache key upgrade includes all src name
* OpenVINO Backend: enable llama arch test on ci
* OpenVINO Backend: move parameter node creating from decoder into translate
* OpenVINO Backend: create extra input ov node move from decoder to translate
* fix for op regression due to is_model_splitted
* openvino: fix CPY writeback for recurrent state rollback
Detect the rollback conv/gdn state writeback CPY nodes structurally
instead of by tensor name, since the rollback path in
build_conv_state does not call cb() and left the nodes unnamed. Add
per-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the
cached IR handles any kv head, sequence count and snapshot slot for
both the conv state and the GDN state writeback.
Assisted-by: GitHub Copilot
* qwen35 moe
* optimize MoE expert aggregation with ReduceSum
* Skip GET_ROWS inaccurate test
* openvino: fallback dynamic MUL_MAT_ID shapes
* OpenVINO Backend: fix error in arch test model mpt
* fix error caused by cpy in arch test model kimi-linear
* OpenVINO Backend: fix error in arch test model minimax-m3
* openvino: fix GPU mul_mat_id op tests
* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU
The OpenVINO weight Constants are zero-copy views into host buffers
allocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU
the plugin holds its own device copy after compile_model, so these host
pages are dead weight for inference. For a 1B Q4_K_M model this leaves
~850 MB of host RSS resident that the GPU path never reads again.
Add an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s
the registered host weight buffers once the model is compiled, dropping
their resident pages while keeping the mappings valid (ggml still owns the
lifetime; tensors still point in). Measured steady-state RSS drops from
~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged
throughput and correct output.
The GPU backend uses a single dynamic-shape model for both prefill and
decode, so a graph is compiled once and reused; the only event that forces
a recompile is clear_caches() on backend teardown. The change therefore:
- releases on the first cache-hit (model compiled, plugin has its copy);
- pins the compiled-model cache across backend teardown so a later
context reuses it instead of recompiling against the dropped pages;
- fails loud (GGML_ABORT) on a cache-miss recompile or on a second model
load, both of which would otherwise read zeroed weights or silently
reuse the wrong compiled graph.
Scope/limitations (all fail loud, never silently wrong): GPU only (the CPU
plugin reads the host Constants at inference time), one model per process,
and stable graph shapes. This reduces steady-state RSS, not the transient
compile-time peak. All changes are confined to the OpenVINO backend.
* ggml-openvino: stream weight requantization to cut the compile-time RSS peak
requantize_to_buffers() dequantized the entire tensor to a temporary
std::vector<float> of n_elements before requantizing. For token_embd.weight
(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is
the single largest contributor to the OpenVINO compile-time memory peak --
it also fires twice for token_embd (once at load, once at graph build,
because token_embd is loaded via a CPU/mmap buffer and not cached as an OV
weight extra).
Stream the dequant instead: process a fixed window of complete rows
(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk
straight into the output buffers. The transient F32 footprint is now
CHUNK_ROWS*ne0 floats regardless of tensor size.
quantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk
writes its weights/scales/zp at the correct block. Streaming is applied to
the Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)
path keeps the whole-array call because it packs two weights per byte with
running zp ORs, and a fallback handles any future target whose block size
does not divide a row.
Measured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);
8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged
("capital of France is Paris"); throughput unchanged. Unlike
GGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just
steady-state, and needs no env flag. All changes confined to the OpenVINO
backend.
* ggml-openvino: avoid redundant token_embd requantization at compile
token_embd.weight is referenced twice in the graph path: as the GET_ROWS
embedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on
every weight-node build, and is_model_splitted() built a full (naive) set of
weight nodes just to test name membership — each requant is a ~1-2 GB F32
dequant of the 262M-element embedding.
Two changes:
- Add collect_weight_names(): a name-only collector for topology checks.
is_model_splitted() now uses it instead of create_weight_nodes(cgraph,
true), so the splitted-check no longer triggers any weight extraction.
- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime
cache keyed by tensor->data. These tensors have no OV buffer context to own
a cached extra, so without this they were rebuilt on every (re)compile;
prefill and decode graphs now share one build (verified: 2nd graph hits the
cache instead of re-requantizing).
Peak RSS is unchanged (the streaming-requant commit already removed the F32
transient); this removes redundant compile-time work. Output verified
unchanged ("capital of France is Paris"). Confined to the OpenVINO backend.
* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM
The streaming requantization and the non-OpenVINO-buffer weight-node cache
(plus the name-only is_model_splitted path that pairs with it) are now opt-in
via GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()
fully materializes the F32 buffer and weights are rebuilt per compile exactly
as before; when set, the streaming path and the cross-compile weight cache
are used.
Default off keeps behavior identical to upstream unless explicitly enabled.
Verified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output
"capital of France is Paris" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,
added earlier, remains a separate opt-in for the steady-state release.)
* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)
The plugin-level ov::cache_dir caches the compiled blob keyed by the OV
model, but producing that model still runs the full frontend every time:
weight requantization (incl. the large token_embd F32 transient) and the
ggml->OV graph conversion. This adds an opt-in frontend cache keyed off a
fingerprint computed directly from the ggml cgraph, so a hit imports a
previously exported CompiledModel and skips requant + convert + compile
entirely.
Key (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per
node op/name), a sampled per-weight fingerprint (name/shape/type + bounded
head+tail byte sample), and blob-affecting config (device, flash-attn, rope
params, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar
manifest stores every weight's fingerprint and is re-verified on load, so a
sampled-hash collision cannot cause a wrong-model hit (verified: two
different quantizations of the same model produce distinct cache entries).
Flow (dynamic single-model path only; split models defer to ov::cache_dir):
on a verified hit, core.import_model() restores the CompiledModel and a
lightweight decoder is built with a names-only weight map (membership is all
the decoder needs for I/O mapping; weights live in the imported model). On a
miss, compile as usual then export the blob (atomic temp+rename, manifest
written first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/
CACHE_MODE are stripped from the config used for the cached compile and the
import — a blob compiled with cache_dir set cannot be re-imported.
Measured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s
(~2.4x faster compile phase). Output verified unchanged on cold and warm,
standalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default
off; confined to the OpenVINO backend.
* ggml-openvino: harden frontend model cache correctness
The frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.
Add a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.
DISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.
Also make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.
While touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.
* ggml-openvino: add memory optimization umbrella switch
Add GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.
Centralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.
Keep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.
* ggml-openvino: rename compiled model cache env
Rename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.
Update the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.
* docs: document OpenVINO memory optimization env vars
Add runtime configuration entries for the newly recognized OpenVINO environment variables.
Document GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.
Document GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.
* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug
1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash
on decode for models that pad the token embedding (n_embd -> n_embd_inp).
PAD never reorders/merges dims, so it keeps the same dynamic dim index as
its source.
2. process_view_input_new() chained VIEW inputs through src[0] (the
immediate op-graph parent) using offsets treated as relative to that
parent. But ggml_tensor::view_offs is always absolute from the true root
allocation (ggml collapses VIEW-of-VIEW chains internally). For the
per-layer deepstack view ("embd (view)", whose src[0] is "embd" - itself
an already-narrowed, zero-offset VIEW of the padded root, with the SAME
ggml shape as the deepstack view but a different absolute offset), this
caused an out-of-bounds re-slice that silently fell back to returning the
wrong (already-resolved sibling) tensor. In practice every deepstack ADD
ended up adding the real base token embedding into the residual stream
instead of zero, corrupting generation ("Hello my name is 1000000..."
instead of coherent text). Fixed by detecting this pattern (same shape as
the immediate src, different absolute offset) and re-slicing directly
from the untouched root tensor using the innermost view's absolute
offset.
Also adds a GGML_OPENVINO_DEBUG_NODE=<name1>,<name2>,... env var that attaches
extra debug Result nodes for arbitrary intermediate tensors, without binding
them to any ggml buffer (avoiding the risk of reading a ggml buffer that has
since been overwritten by a later in-place op). This was instrumental in
diagnosing bug #2 above and is left in as a general-purpose debugging aid.
* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes
IMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into
ne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape
path, inp_pos was padded/shaped as if it held a single plane, which
interleaved padding across the 4 planes and desynced later reshapes
from the rest of the (chunk_size-wide) graph.
- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout
- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)
or n_planes (decode) instead of assuming 1 value per token
- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size
independently instead of one flat block
- get_ov_input_tensor_static_decode(): copy n_planes contiguous values
instead of asserting/copying a single scalar
* disable test-llama-archs tests.
* openvino: gate fallback with env var
* Revert changes in test-llama-archs
* Apply editor config
* reject CPY with quantized destination as unsupported
---------
Co-authored-by: Xuejun <Xuejun.Zhai@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
index.html was served with `max-age=31536000, immutable` like the hashed assets, but its name is stable while its contents change every build, so a cached copy pins the UI to an old build. It now revalidates via its existing ETag, which keeps the 304 for unchanged builds.
* metal: add TQ2_0 support
Add support for the GGML_TYPE_TQ2_0 (ternary, 2 bits per element) type in
the Metal backend.
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* cont : optimize mul_mv kernel
- float ops over integer ops
- precalculate sums
- hoist coef out of the inner loop
- contiguous y loads
llama.cpp:DeepSeek-v4-Flash-0731
* common : auto-detect spec type from draft GGUF metadata
When -md loads a local draft model without --spec-type, the sidecar
inference in common_models_handler_apply only checks HF repo sidecars
and misses local files. The draft model loads into VRAM but speculative
decoding never activates (types stays NONE).
Read general.architecture from the draft GGUF header and map:
dflash + markov_w1.weight tensor -> draft-dspark
dflash without markov head -> draft-dflash
Assisted-by: opencode
* common : address review feedback on spec-type auto-detect PR
- Fix comment spacing to match surrounding style (/* .x = */ not /*.x =*/)
- Add LOG_INF when auto-detection fires so users can see why spec decoding enabled
- Document single-file assumption for split-GGUF edge case
Addresses bot review feedback on #26814.
* common : move spec-type GGUF auto-detect into speculative module
- add common_speculative_types_from_gguf() in speculative.cpp/.h
- use gguf_context_ptr (RAII) from ggml-cpp.h
- reduce comments to a single line per AGENTS.md style
Addresses review feedback on #26814
* common : add doc note and join SPC_INF line in spec-type auto-detect
Assisted-by: opencode
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add DMMV Q3_K ESIMD kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
It enables -fassociative-math, which reassociates FP reductions and can flip
greedy argmax on RDNA3.5 (e.g. MTP speculative decode diverging from the
non-speculative baseline). Drop it so HIP builds are IEEE-conformant.
Co-authored-by: Jim Wu <ywu@xilinx.com>
* common: Add CLI > ENV > models-presets > INI precedence
1. CLI flags have the highest precedence
2. ENV vars have the second-highest precedence
3. System and User configs have the lowest precedence
- Linux/BSD/Mac
- /etc/llama.cpp/config.ini < ${XDG_CONFIG_HOME:-~/.config}/llama.cpp/config.ini
- Windows
- %PROGRAMDATA%\llama.cpp\config.ini < %APPDATA%\llama.cpp\config.ini
* fix UB
* use common_get_env
* ignore_unknown_keys
* nits
* add docs
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* test address on Intel-LNL-U7-258V
* retry
* run address on github
* use native build for cpu
* this should be runnable everywhere multicore
* disable ccache
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* cmake : introduce semantic versioning (wip)
This commit introduces semantic versioning to llama.cpp.
* squash! cmake : introduce semantic versioning (wip)
* cmake : update test-cmake README notes [no ci]
* include libmtmd in output so show its semversioned
* ci : add make-release workflow
* ci : fix build number check in build-cmake-pkg.yml
* examples : remove trailing whitespace
* ci : abort if upstream ggml version does not exist
* ci : extract step contents into scripts
* ci : add GGML_NATIVE=OFF to ubuntu job
* examples : remove CI build information from test-cmake [no ci]
This commit removes the nightly/release information that I added
previously to keep this focused only on using building and installing
llama.cpp with cmake and being able to quickly verify changes or
troubleshoot issues.
* ci : merge scripts into single script
* remove -dev-build_number support
This commit removes the incremental build number (versioning) support
that I added. This was incorrect and we should only use the semver for
the version. Releases will be tag a nightly build and package
maintainers/managers that build from source can use the tag and it is
therefor important that the correct version is reported. So a
nightly-build will report the semver without the build number. The build
number and commit as availble via cmake and test-cmake has been updated
to include an example of using them:
```console
$ ./build.sh
[test-cmake] version: 0.1.0, build: 10360 (08c69e381)
...
```
Refs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969
* docs: add initial release.md documentation
* cmake : clean-up and add LLAMA_BUILD_IS_DEV option
* ci : remove version input from make-release job
* ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml
Refs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145
* docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci]
* ci : add TODO to winget workflow [no ci]
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* gguf : harden loader against malformed tensor dims and metadata types
* gguf: address review on malformed-metadata hardening
- report the expected vs. actual type when general.alignment is not u32
- use ggml_nelements() > 0 for the zero-element guard and keep the
representability checks visually aligned
- add test-gguf cases for a wrong-typed alignment key and a zero-dim
tensor (both used to crash: assert-abort and SIGFPE respectively)
Ran tests/test-gguf: 164/164 pass. Used an AI assistant to help draft
these edits; reviewed and verified by me.
* cont : less comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cmake : add config version support (wip) [no ci]
This commit adds support for find_package using a version, for example:
```
find_package(ggml 0.19.0 REQUIRED)
```
examples/test-cmake has been updated to use this and build scripts have
been added to verify this manually. This is still a work in progress and
I'm not sure about the scripts and if we can find better ways to test
this but it might be useful to have for verification of changes to the
cmake build.
* cmake : add semver to ggml backends [no ci]
This commit adds a semver to the ggml backend modules files.
The motivation for this is that the backends are currently loaded just a
file extension, for example .so on linux. With the introduction of
semantic versioning installing a new version should just work but since
these files don't have a version they would get overwritten. Adding the
semver to the library names allows multiple version to be supported and
the correct one will be loaded by the code.
I've only tested this on linux and need to test on mac and win.
* Revert "cmake : add semver to ggml backends [no ci]"
This reverts commit 53a6c58a07591951324c891b9986b2cffe5c7972.
* examples : update build-install.sh and set GGML_BACKEND_DIR
* server : save serialized image chunks at the end of the llama state
* server : support multimodal slot state save/restore with packed payload
* server : refine image slot state serialization
* server : support media slot state and centralize media validation
* server : remove unnecessary comment
* server : remove defensive media checks and move the chunk type check to validate()
* server: add read_image tool (#25875)
Adds a server-tool that allows vision models to analyze server-side images.
This tool is reading a single file for now:
The image data is base64 encoded and passed to the UI, which
decodes it, fills the <img> tag and removes the data URI before
passing the tool result back to the model.
* cleanup read_image tool: move magic strings to constants
* Add dedicated constants file: tools/ui/src/lib/constants/read-image.ts
with PREFIX_IMAGE, PREFIX_SIZE, PREFIX_MIME constants
* Use ATTACHMENT_SAVED_REGEX from agentic.ts in ChatMessageToolCallBlockReadImage.svelte
* Use NEWLINE constant from code.ts instead of hardcoded '\n'
* Use PREFIX_SIZE in regex pattern for size parsing
* Add SERVER_TOOL_READ_IMAGE_PREFIX_* constants in C++ server-tools.cpp
to match the TypeScript PREFIX_* constants for consistency
* server: rename read_image tool to read_media for images and audio
* Rename server_tool_read_image to server_tool_read_media in C++
* Rename enum BuiltInTool.READ_IMAGE to READ_MEDIA
* Rename UI constants, parser, and Svelte component files
* Update display label from 'Read image' to 'Read media'
* ui: consolidate audio data URI handling into shared utility
* Extract getAudioInputFormat to a shared utility (was duplicated inline)
* Store raw base64 in base64Data on the message object
* Use base64Data to construct data URIs for audio rendering
* Update agentic store to build INPUT_AUDIO parts from base64Data
* server: read_media: restrict audio to wav/mp3 and minor fixes
* Server get_mime_from_extension now only advertises audio/wav and
audio/mpeg (the only formats the model's input_audio API accepts)
* Case-insensitive extension matching (fixes .MP3, .Wav, etc.)
* Unknown extensions return an error instead of a multi-MB data URI
that inflates model context with garbage
* Updated tool description to document supported formats
* Frontend AUDIO_MIME_TO_EXTENSION trimmed to match server
* fix a missing import in tools/ui/src/lib/stores/agentic.svelte.ts
* server: read_media: add to --tools help text and README tool list
* ui: fix indentation in ChatMessageToolCallBlockDefault.svelte
* server: read_media tool: fix a cast to use the correct type
* server: read_media: multiple fixes
* server-tools.cpp import cctype, remove UTF-8 char, check mime before reading file
* ui: add MimeTypePrefix.AUDIO and use it in agentic.svelte.ts
* server: make read_media inherit from read_file and add uses_cwd
* ui: fix formating issues
* rm from server
* move it to frontend-only tool
* correct partial commit
* rm unused
* ui: address review from allozaur
Replace the magic strings, regexes and number in the read_media parser
and service with named constants. Path splitting reuses
FILE_PATH_SEPARATOR_REGEX, the size header regex moves to
READ_MEDIA_SIZE_REGEX derived from PREFIX_SIZE, and
FILE_EXTENSION_SEPARATOR lands next to it in constants/code.ts.
---------
Co-authored-by: ckrafft <ckrafft@epyc>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* vulkan: TQ2_0 (ternary) support — dequant + dedicated mul_mat_vec + matmul via dequant_funcs
First Vulkan ternary type in ggml. Correctness: OM-125m TQ2_0 vs F16 top-12
logprobs identical to 4 decimals fully offloaded (float dequant path, no Q8_K
activation quant). Speed at 125m ~= F16 (overhead-bound at this scale); the
bandwidth win targets larger BitNet SKUs. MMQ/int-dot path intentionally not
wired yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: enable TQ2_0 in backend-ops type lists
Vulkan now implements TQ2_0 (dequant, mul_mat_vec, mul_mm, get_rows); backends
without support skip via not-supported as usual. TQ1_0 stays disabled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Michael Trabalka <michael.trabalka@sqv.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: handle nested global_head_dim in Gemma4 config
Gemma-4 E4B models have global_head_dim inside text_config
rather than at the top level. Add fallback to support both layouts.
* fix: add fallback for global_head_dim to support per_layer_config format
* fix: read head_dim only from full_attention layers in per_layer_config and num_global_key_value_heads compatibility
* fix: added fallback for num_global_key_value_heads
* fix: read per_layer_config from root hparams
* fix: delete unused text_config
* cleanup and fixes
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* chat : fix muse-glimmer swallowing a trailing tool call into content
Muse Glimmer routinely answers the user and calls a tool in a single
generation. The template terminates a message with <|eom|> when more
messages follow in the same turn and <|eot|> only at the end of the turn,
so the answer is closed by <|eom|> and the call opens a fresh header:
<prose><|eom|><|start|>assistant to=<tool><|message|><atem:function_calls>...
The final-message rule read content with until("<|eot|>"), which assumed the
user-facing message is always last. There is no <|eot|> before the call, so
content ran to the end of the turn, absorbed the markup, and no tool_calls
were emitted - the tool never ran. On a tau2-bench telecom run this hit 43
turns across 19 of 114 tasks.
Stop the answer at <|eom|> and parse what follows as tool calls.
Adds models/templates/muse-glimmer.jinja and four parser tests: a plain
answer, the <|eom|> junction, markup quoted in an answer staying content,
and tool markup inside the to=self channel staying reasoning.
* address comment
* adapt the api
* text model ok
* working impl, need verify and clean up
* mtmd: build the pocket-tts transposed convolutions as GEMM + col2im
ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample
was built as one convolution and one concat per channel, which floods
the graph with small nodes and makes kernel launches dominate the
decoder.
Fold both cases into the column form the seanet decoder already needs:
the general case reshapes the kernel to [IC, K * OC] and matmuls it
with the input, the depthwise case batches a matmul over the channels
so a step scales its own kernel. A single col2im_1d then scatter-adds
the columns back to the signal, with the same shape as before, so the
overlap-add tail, the streaming state and the bias are untouched.
Generation time per frame drops by 80% on CUDA and by 50% on CPU. The
output matches the previous implementation sample for sample, with a
correlation of 0.999994 and identical frame counts.
* flow_temp + frames_after_eos
* chunking
* mtmd: carry the remaining pocket-tts per-pack settings
The language packs also tune the end-of-speech padding and the padding
of short prompts, next to the temperature already carried in the
mmproj: french_24l asks for 8 tail frames instead of the guessed 3,
english_2026-01 asks for short prompts to be padded with spaces.
Write both in the mmproj as clip.gen.audio.frames_after_eos and
clip.gen.audio.pad_short_text, keyed on the pack in the conversion
script like the temperature. The loader keeps them optional, so a
mmproj without them behaves as before. Map semicolons to commas for
every pack instead, the reference only asks for it on three of them and
it costs nothing elsewhere.
Existing mmproj files must be converted again to carry the two keys.
On a long french text the port now lands within 2% of the reference:
22.96s against 23.44s, with the same peak level and the same amount of
silence.
* clip.gen.audio.model_variant
* clean up code comments
* nit: drop the dead flow_temp hparam, the pack table holds the default
* update docs
* address security problems
* less invasive base.py
* lint
* add mtmd_gen_inp_default
* add docs
* rm gen_flow_temp
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Most of the old ones have been resolved (yay) but the recent refactor of mmq paramters has caused some symbol names to change,
leaving a couple of non-ignored failures
This commit updates the python script that runs the original model to
generate embeddings for the causal model, to use save_output_data which
stores the token ids and the prompt in addition to logits.
The motivation for this is that the embedding logits verification will
fail as it expects these files (-prompt.txt and -tokens.bin) to exist.
With the changes in this commit the causal-verify-embeddings target
works again.
* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used
* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* set mmap_support to false on OpenCL backend
* fix order of load modes
* use -1 for auto
* resolve load mode auto earlier to correctly pick gpu host or cpu memory
* add load mode auto to llama-bench
* bump virtgpu api version, regenerate docs
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* test-backend-sampler: skip multi_output_sampling_chain on HIP
The new multi_output_sampling_chain test uses top_k, whose backend probs
path needs CUB (unavailable on HIP), so sampled_probs is null and the test
aborts. Add it to the existing HIP skip list alongside the other TOP_K tests.
* ci: keep gpu-rocm logs in a per-run dir keyed by GitHub run id
The self-hosted gpu-rocm runner can't upload logs to Azure blob (egress
firewalled), so a run's logs were wiped by the next run. Write each run's
logs to $OUT/run-<run_id>-<attempt>/ so an Actions run URL maps to its logs.
* test-backend-sampler: also skip multi_output_cpu on HIP
Like the other TOP_K-based subtests, multi_output_cpu's backend sampler
never initializes on HIP (no CUB TOP_K), so it aborts. Add it to the skip list.
---------
Co-authored-by: Jim Wu <ywu@xilinx.com>
* model : fix SWA not being enabled for EXAONE 4.5
load_arch_hparams tests `hparams.n_layer() == 64` before
LLM_KV_NEXTN_PREDICT_LAYERS has been read. n_layer() returns
n_layer_all - n_layer_nextn and n_layer_nextn defaults to 0, so a GGUF
carrying the MTP head (block_count=65, nextn=1) evaluates to 65 and the
whole SWA block is skipped. The model type switch further down in the
same function reads 64, because by then the key has been loaded.
n_swa is still filled in by the unconditional get_key below the block, so
llama_model_n_swa() reports 4096 and the logs look correct while only
swa_type stays LLAMA_SWA_TYPE_NONE.
This affects the official LGAI-EXAONE GGUF release as well. EXAONE 4.0 has
no MTP head, so block_count is 64 there and the check matches.
* model-loader : skip TENSOR_SKIP tensors in the metadata-only path
create_tensor asserts on a null buffer type when building from metadata
alone, but buft_for_tensor returns null by design for tensors marked
TENSOR_SKIP, which is how architectures with nextn/MTP layers mark theirs.
Those models cannot be constructed by llama_model_init_from_user at all.
The file-backed path below already returns nullptr for the same tensors, so
callers see the same thing either way.
* tests : cover exaone4 hparams ordering
Builds a synthetic exaone4 model with the layout the shipped EXAONE 4.5
GGUFs use (block_count 65 + nextn 1). The swa_type check is the one that
catches the ordering bug; the n_layer_nextn and n_layer() checks only tell
a broken fixture apart from a real regression.
Fails before the ordering fix with "swa_type is not STANDARD", passes after.
* Revert "tests : cover exaone4 hparams ordering"
This reverts commit d2f3bafeee591ad691396b2708de4baef3aaf602.
* Revert "model-loader : skip TENSOR_SKIP tensors in the metadata-only path"
This reverts commit aecb9bc0c7896b52afbc43921a1f572aa7b5e53c.
* test new flash_attn test
* rebase and fix to disable subgrou matrices when max_kv_tile == 0
* delete log output
* Add i32 support to cpy and enables the all ops test
* restore the non target ci tests
* comment out of TODO of build-cpu.yml
* fix format
* Switch ROCm from 7.2.1 to 7.14
ROCm 7.14 is the first production release using TheRock build system.
It can be installed using multi-arch deliverables from wheels, debs,
rpms, tarballs or runfiles.
Adjust ROCm targets for Linux and Windows to use this instead.
* ci: switch all other Windows ROCm jobs to ROCm 7.14 wheels
Move the shared windows-setup-rocm composite action from the HIP SDK PRO
Edition installer to the multi-arch ROCm wheels (rocm[libraries,devel]).
The wheel-install logic that previously lived inline in release.yml is now
in the shared action, and both build-cache.yml and release.yml call it.
Also migrate the build-cuda-windows.yml hip job to the same wheel-based
layout (cache path/key, rocm-sdk environment setup, llvm/bin compiler
paths) so it keeps working after the action's contract changed; drop its
now-unused ROCm 7.2.1 rocWMMA download and stale include path.
* Enable backend sampling with token speculation
* Clamp the mask sum before converting it into the sampled index
* Add a numeric context parameter declaring the maximum outputs one sequence
* More fixes
* Don't reuse memory for output views.
* Match dist between CPU and GPU
* Fix CPU and backend sampling mismatches
* Simpify some of the changes
* Fix tests on Vulkan
* More test fixes
* Rebase changes
* Rebase and address review comments
* Address review comments
* Address review comments
* Update src/llama-sampler.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggml_roll only asserts nb[0] == ggml_type_size, so a permuted src is a
valid input, but the CUDA and Metal roll kernels index by ne alone and
never read the nb strides. A non-contiguous src therefore produced
silently wrong results. Neither backend declared a contiguity
requirement in supports_op, so the scheduler did not fall back to the
CPU implementation, which does handle strides correctly.
Add the requirement to both backends, matching the existing
GGML_OP_ROPE guard, and add a permuted test_roll case.
* ui: split the markdown rendering setting per surface
User content and thinking get their own toggle again, so turning off
markdown for a message leaves reasoning blocks formatted. Both default
to markdown. A stored renderContentAsRawText unfolds onto the user key
and is dropped from the config.
File mentions render as badges in the raw text path too, through a
narrow pass over [name](file://path) that leaves everything else
untouched.
* ui: let the rich chat input scroll past its max height
The contenteditable renderer caps its height with max-height but had no
overflow rule, so a long buffer overflowed into the input area wrapper
and got clipped by its overflow-hidden, leaving no way to reach the
bottom of the message. The textarea renderer scrolls natively and was
never affected.
* ui: apply the new lint and format config
* ui: move the render keys unfolding into the migration service
Address review from @allozaur: the settings store no longer rewrites
persisted config on load, the raw text toggle now unfolds onto the
per-surface render keys in migration.service.ts, next to the other
config migrations. The mention scanner flag and the directory path
suffix become named constants.
* server: add an ssh transport to the tools runtime
--tools-runtime ssh:<target> runs the built-in tools on a remote host,
where target is whatever ssh already resolves, a user@host or a config
alias, so no credentials live in llama.cpp.
Only build_argv and upload differ from the docker transport: the remote
shell re-parses the command line, so the argv travels through
shell_quote_join, and files go over scp with the same quoting on the
remote path. Authentication is key-based and the host key must already
be trusted, since the tools run without a console and any prompt would
hang them.
The target is validated before use. The spec can reach us from the
x-tool-runtime header, and a leading dash would turn it into an ssh
option, which is enough to run a command back on the host.
Nothing is created and nothing is reclaimed, so an ssh spec goes
straight to the tool call instead of through the container runtime.
Note that this is remoting rather than isolation: the tools can do
whatever the target account can do, and the isolation is whatever runs
them on the far side.
* server: support podman in the tools runtime
docker and podman expose the same run, exec, cp and inspect verbs with the
same argument order, so a single implementation drives both and the engine
is carried by the spec prefix: podman:<image> and podman-container:<id> sit
next to the docker forms.
tools_io_docker becomes tools_io_container and the runtime spawner becomes
server_tools_container_runtime, both holding the client binary chosen at
parse time. A single parse_container_runtime() resolves every spec, so
adding another engine is one string in the table.
make_tools_io() now rejects the spawning forms. The spec also reaches it
from the x-tool-runtime header, which is client controlled, and only the
runtime that owns a container is allowed to create one: a tool call can
attach to a running container, nothing more.
* ./build/bin/llama-gen-docs
* server: simplify the tools runtime and drop the file copy step
A server_tools_runtime base with one virtual spec() replaces the
container runtime and the bare spec string that ssh needed next to it,
so server_tools is back to a single pointer and neither setup nor the
handler tests which of the two is set.
write_file used to spill its content into a temporary file on the host
and copy it in, because run_subprocess had no way to feed a child. It
now takes an optional stdin payload and creates the parent directory
and the file in a single round trip through a shell in the isolate.
That removes the upload virtual and both implementations: no more
container cp or scp, no second binary on the host, no sftp subsystem on
the target, no predictable temporary in a shared tmp, and none of the
content reaching an argv the remote shell re-parses. It also fixes
write_file over ssh, which never worked: scp speaks sftp and takes the
remote path literally, so quoting it kept the quotes in the file name.
Writing the payload before reading the output relies on the child
draining stdin as it goes, which holds for cat, its only user today.
* ./build/bin/llama-gen-docs
* server: harden the tools runtime against argv injection and a stdin stall
Validate the container id from x-tool-runtime and --tools-runtime the
same way the ssh target already is, so an id shaped like an option
(docker-container:--privileged) is rejected before it reaches the
engine's exec command line instead of running against a hardened
container. Feed the child's stdin after the watchdog is armed, so a
transport that stalls mid-write is terminated at the deadline rather
than blocking the request forever.
Cover both guards and fix the unknown-scheme test, which used ssh: as
its example and now names a real runtime.
* tests: exercise the tools runtime tests on podman as well as docker
Follow-up #26507. The container runtime drives docker and podman
through one implementation, so parametrize the availability helper,
the container fixture and the attach test on the engine, and cover
both engine prefixes in the container id injection test. Each engine
skips on its own when it is not installed.
The spawn cleanup test stays docker only: it recovers the spawned id
from the container hostname, which docker sets to the short id and
podman rootless does not guarantee. Podman keeps its coverage through
the attach path.
* server: release the container handle before respawning
Follow-up #26507. create() writes over the handle it is given, so a
respawn after the container died on its own leaked the pipes and the
process handle of the previous one.
* server: trim the tools runtime comments
* server: read tool output as raw bytes and harden the runtime on Windows
The stdout pipe is read with read() instead of fgets(), so a chunk
can hold any byte, including NUL, and still streams as soon as data
is available. Past the size cap the pipe keeps draining so the child
never blocks on a full pipe. Both pipe fds are forced to binary mode
on Windows, where the CRT defaults them to text mode and translates
line endings in both directions. Stdin is now always closed after
the feed: the child reads a deterministic EOF, and the Windows
docker and ssh clients stop outliving their command on a stdin pipe
that never closes.
The attach form of --tools-runtime has no lifecycle to own, so it
becomes a static target validated once at startup. This removes the
subprocess that ran on every tool call and
serialized calls behind a mutex; a stopped container now surfaces
the engine's own error at exec time.
The cidfile path is passed as UTF-8, matching the encoding the
subprocess layer expects for the CreateProcessW command line, so
the spawn form works from a non-ASCII Windows profile.
The SIGPIPE note in server.cpp now names the tools runtime children
as well as the MCP ones.
* clean up comments
* less pollute global scope
* nits
* tests: name the container image after both engines
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* Get started with Onyx
* Add architecture
* Skip keys handled in super()
* Loading tensors
* Shorten
* Graph
* Apply suggestion from @pcuenca
* Remove norm now embedding in transformers weights
* Add eot
* Explicit output_multiplier
* Handle post_norm_eps
* No super call; unhardcode eot.
The pattern `self._set_vocab_gpt2()` seems preferred throughout the
codebase, and it allows `set_vocab()` to be called from a different part
of the Python class hierarchy: the drafter model converter that we may
need eventually.
* Register for drafting
* DFlash: inherit rope type from the linked target.
Another option would be to store it in the gguf file itself.
* mmproj conversion
Note: some fields to be renamed after the implementation works. We are
keeping compatibility with the reference Meta gguf for testing purposes.
* "clip" header declarations
* Load mmproj
* Pre-processing
* Graph
* Go back to using delimiters.
Otherwise our generations are worse.
Transformers does not use them. We need to trace inputs to verify
whether they are equivalent.
* downsample_factor -> merge_size
* Add vision graph
lol, forgot from a previous commit
* Additional renames, align with llama.cpp / transformers
* Prefer _size instead of independent _h and _w
* Fix token layout
Co-authored-by: Young Han <younghan@fb.com>
* onyx: bring the chat parser onto the onyx branch
common/chat.cpp on this branch has no Onyx handling, so a converted model
serves malformed chat: the assistant preamble leaks into content
("to=self<|message|>...") and tool calls fail with
HTTP 500 "The model produced output that does not match the expected
peg-native format"
common_chat_params_init_onyx exists on onyx-fair-patch, added there by
8bb73dd3d. It was never on this branch, so this is not a regression --
the two lines developed independently.
The code here is taken verbatim from that commit. It is the clean side of
`git merge origin/onyx-fair-patch`: chat.cpp is one of the files that
merges without conflict. The full merge is not viable -- it produces 13
conflicts, including add/add on conversion/onyx.py and src/models/onyx.cpp
where the q_norm-folding and metadata-scale approaches contradict each
other, and #4/#7 are stacked on this branch's side of that.
Verified on this branch: builds with 0 errors, converts an Onyx checkpoint,
and serving it gives "4" for "What is 2+2?" plus a correct
get_weather {"city":"Paris"} tool call, where the unported branch gives the
two failures above.
No converter or runtime changes are included, so this should not interact
with the q_norm work.
Co-authored-by: Beto de Paola <betodepaola@meta.com>
* Less params, bilinear pos-emb interpolation as a graph op instead of CPU
* Map to symbolic V_MMPROJ instead of strings
* Make a couple params explicit
* Patchify via build_inp()
* No param for rope_theta
* Small cleanup
* Restore blank line
* Unpermute, to adapt to the latest transformers checkpoint
* Apply norm after token embeddings
This follows the latest transformers approach.
* Remove duplicated function
* build_vit
* onyx: use the model rope theta on sliding-window layers
* DFlash: conversion from transformers drafter
* Revert rope_type derivation from target
NOTE: this breaks compatibility with Meta's distributed DFlash GGUFs, as
the Q/K are stored in "NEOX" (rotated half) format, like in
transformers.
* Apply suggestion from @pcuenca
* Set model type
* Remove comment that will become obsolete
* Hardcode post_norm_rms_eps instead of new param
* Derive SWA+RoPE pattern from gguf array or scalar
* Fix model type <-> number of layers
* Reorder
* Rename
* Fix typo
* DFlash: seed the draft KV cache from multimodal embedding batches
`common_speculative_impl_draft_dflash::process()` returned early on any batch carrying embeddings, so an image prefill never had its target-layer features fused through the DFlash encoder and injected into the draft's KV cache. That left a hole spanning the image's positions, and the next injection at a post-image position failed to initialize its batch:
```
decoding image batch 1/1, n_tokens_batch = 256
decode: failed to initialize batch
llama_decode: failed to decode, ret = -1
process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=17, offset=0)
srv decode: failed to process speculative batch
```
Every image request with `--spec-type draft-dflash` failed with HTTP 500. Text-only was unaffected, since those batches carry token ids and were let through.
Restore the earlier condition, which admits a batch that is either tokens or embeddings and skips only the degenerate neither/both cases. The rest of `process()` is already layout-agnostic -- it gathers features via `llama_get_embeddings_layer_inp()` and indexes `batch_in.pos[]` / `batch_in.seq_id[]`, none of which assume token ids -- so this is the whole fix.
Validated against `muse-glimmer-30B-bf16.gguf` + `mmproj-muse-glimmer-30B-bf16.gguf` + a DFlash draft head, on an image describe-the-shapes request:
- before: HTTP 500, `failed to process speculative batch`
- after: HTTP 200, draft acceptance 0.34012 (167 accepted / 491 generated), mean len 3.04
Output equivalence holds, which is the property that matters: at temperature 0 the drafted response is byte-identical to the same request served with no draft attached (1213/1213 chars), so the draft is drafting correctly through the image context rather than merely not crashing.
* Conversion: prefer rewrite to mapping
* Revert "Conversion: prefer rewrite to mapping"
This reverts commit a92d0ac584d315e876741e85b6dad3dbc8b23bf7.
* fix lint
* sliding_window metadata is not optional
* disable state save/load
* Apply suggestion from @pcuenca
---------
Co-authored-by: Young Han <younghan@fb.com>
Co-authored-by: Beto de Paola <betodepaola@meta.com>
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: ruanrms <ruanslv@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Upstream merged the Windows argument quoting fix, the NetBSD build
fix and the chdir fallback for glibc older than 2.29, so pin the
vendored copy to a commit that carries all three and remove the
patch files along with the apply step in the sync script.
The new pin also brings the exec error report on glibc older than
2.24 and the ENOSYS mapping to a dedicated error code. Both are
additive and no caller inspects those values.
* Restore quantization of mmprojs
This was lost in the refactor undertaken in #22004.
* add noreturn
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ci: Add support for CUDA 13.4 ARM64 builds for Windows
Added an architecture-specific CUDA 13.4 Windows build entry targeting ARM64.
Added a CMake configuration to enable ARM64 CUDA cross-compilation from an x64 Windows environment using the x64-hosted CUDA and MSVC toolchain while linking against the ARM64 CUDA import libraries to produce ggml-cuda.dll.
Validated the self-hosted Windows x64 workflow, including toolkit acquisition, CMake configuration, ARM64 CUDA cross-compilation, and packaging. Runtime validation was performed separately on a native ARM64 RTX Spark system using TinyLlama 1.1B Q4_K_M to verify the generated binaries.
The ARM64 CUDA job builds only the ggml-cuda.dll backend (LLAMA_BUILD_SERVER=OFF). The release consists of two packages: the main ARM64 release package, which combines the existing ARM64 CPU outputs with ggml-cuda.dll, and a separate runtime package containing the required CUDA runtime libraries (cudart64_13.dll, cublas64_13.dll, and cublasLt64_13.dll).
The CUDA 13.4 setup uses NVIDIA Developer Preview component archives instead of the GA component downloads used by the existing CUDA setups and will require updates once CUDA 13.4 reaches GA.
* ci: cleans up to align with x64 CUDA setup
- Moves CUDA-specific CMake options into matrix defines.
- Keeps the CUB 3DOT2 option only for CUDA 12.4.
- Removes runtime argument construction and the unnecessary server option.
- Aligns ARM64 CUDA runtime packaging with the existing robocopy approach.
- Generalizes the ARM64 release label from CUDA 13.4 to CUDA 13.
* ci: Set CUDA job name as version-architecture pair
* mark as preview
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* granite-switch: add llama.cpp backend (POC, CPU)
New "granite-switch" architecture: a dense, all-attention Granite-4.1
model with N embedded LoRA adapters selected per-token by control tokens.
- gguf-py schema (arch, KV keys, stacked LoRA tensor names) + writer helpers
- conversion/granite.py: GraniteSwitchModel converter (stacks N adapters +
zero base slot into per-projection A/B tensors; emits switch metadata)
- C++ arch registration (llama-arch.{h,cpp}, llama-model.{h,cpp})
- src/models/granite_switch.cpp: load + per-token switched-LoRA graph via
ggml_mul_mat_id over stacked tensors; sticky per-token index + control-token
substitution in llm_graph_input_switch::set_input
- llm_graph_input_switch in src/models/models.h
Runs end-to-end on CPU: convert 3b checkpoint (842 tensors, stacked dim 13)
and generate on both base and control-token paths. Sticky switch state is
single-sequence (POC); full multi-sequence machinery is a follow-up.
* granite-switch: add Mac (Metal) build + mid-sequence switch demo script
Self-contained script to build llama.cpp on Apple Silicon (Metal),
convert the composed 3b checkpoint, and run the crisp mid-sequence
adapter-switch demos verified on Vela:
- answerability: <|answerability|> mid-seq -> "unanswerable"
- query_rewrite: <|query_rewrite|> mid-seq -> {"rewritten_question": ...}
Each demo runs the same prompt twice, differing only by a control token
placed before the assistant turn, so the per-token switch is visible.
* granite-switch mac demo: add -no-cnv so each run is one-shot
The composed model ships a chat template, so llama-completion auto-enables
interactive conversation mode and halts at a `>` prompt after generating,
stalling the script. -no-cnv disables conversation mode: generate once from
the raw prompt and exit (also prints special tokens, making the switch visible).
* granite-switch: replace global sticky index with in-graph router attention
The POC computed the per-token adapter index on the CPU and carried it
across ubatches in ONE global `mutable int32_t poc_sticky_index`, reset
only when a ubatch contained sequence position 0. That global had two
problems:
1. Concurrency: with multiple sequences in a batch it was last-writer-
wins — one sequence's adapter leaked into the others.
2. Multi-turn: an interactive `ollama run` chat continues one KV cache,
so turn 2 never saw position 0 and the index never reset — the
adapter stayed stuck on across turns.
Port the vLLM/HF backend mechanism faithfully: a single-head causal
"router" attention recovers the adapter index in-graph. Per token, only
dim 0 carries signal — Q[0]=1, K[0]=+gain for a control token / -gain
otherwise, V[0]=adapter slot / 0 — and the causal softmax over the single
visible control token recovers that adapter's slot (readback =
clamp(round(V[0]), 0, n_adapters)). gain=15 matches config.py and is
F16-safe (no F32 cache).
The router's K/V live in the model KV cache at an extra layer
R == hparams.router_layer (== n_layer). We bump n_layer_all to n_real+1
so the cache allocator gives the router its own per-sequence slot, and
set n_layer_nextn=1 so n_layer() stays n_real — the decoder loop and
tensor loading are untouched and never reference layer R. The router K is
exempted from the k-shift RoPE loop (its dim-0 value is a literal
magnitude, not a rotation).
Because the selection now lives in the per-sequence KV cache, CONCURRENT
requests are isolated for free (problem 1 fixed; verified by
scratch/concurrent_switch_test.cpp). set_input becomes stateless pure
per-token maps; the global is gone.
Single-switch contract / known limitation, identical to vLLM & HF: the
gain is flat (no recency), so within one sequence there is no mechanism to
revert to base mid-sequence — once an adapter fires it stays on until that
sequence ends (problem 2 is therefore NOT fixed by a faithful copy; vLLM/HF
avoid it only because each served request is a fresh sequence). A client
continuing one KV cache across turns must start a fresh sequence per turn,
or opt into a recency-biased router (a deliberate divergence, not done
here). Documented in granite_switch.cpp and asserted by
scratch/multiturn_leak_test.cpp.
Verified (CPU): both demos unchanged (answerability -> "unanswerable",
query_rewrite -> rewritten query); concurrent two-sequence isolation
passes; multi-turn carry-over matches the vLLM/HF contract.
* granite-switch: drop scratch tests and mac demo for upstream PR
Remove the local-only development artifacts that should not ship in the
upstream PR:
- granite-switch-mac-demo.sh (local Metal build + demo driver)
- scratch/concurrent_switch_test.cpp
- scratch/multiturn_leak_test.cpp
Also drop the now-dangling reference to the scratch tests from the
granite_switch.cpp header comment. Leaves only the core architecture
support (conversion, gguf constants, llama-arch/model/kv-cache, and the
granite_switch graph).
* granite-switch: trim comments to match native llama.cpp style
* granite-switch: trim conversion comments to match native style
* granite-switch: drop unused adapter_ranks metadata
* granite-switch: rename arch to graniteswitch and drop obid alias
* granite-switch: fix non-ASCII comments and document router gain assumption
* granite-switch: drop section comments from constants.py to match native style
* granite-switch: add functional tensor block comments matching Granite4 Vision style
* granite-switch: clarify n_expert_used comment
State the actual constraint: mul_mat_id needs n_expert_used == 1, and
since the GGUF carries expert_count = 0 the generic loader's
n_expert == 0 => n_expert_used == 0 assertion has already passed by the
time load_arch_hparams runs, so it is forced to 1 here.
* granite-switch: note n_layer_nextn reuse has no MTP
The router carving reuses n_layer_nextn, normally the MTP/next-token
count. Clarify in the comment that it is borrowed here purely as the
trailing-layers lever and that there is no MTP head, to spare readers
the double-take.
* granite-switch: rename source file and apply review nits
* granite-switch: don't force LoRA tensors to F16, follow --outtype instead
* granite-switch: drop redundant _permute_qk wrapper, call LlamaModel.permute directly
* granite-switch: read router gain from GGUF (control_token_gain) instead of hardcoding 15.0
* granite-switch: derive n_slots()
* granite-switch: move llm_graph_input_switch into granite-switch.cpp
* granite-switch: cut AI-style narration comments
* granite-switch: collapse multi-line comments
* granite-switch: rename control_token_* maps to adapter_token_*
* granite-switch: cut noise comments
* granite-switch: rename embedded LoRA tensors to <base>.lora_a/lora_b
* granite-switch: GGML_ASSERT token input to avoid UB on embeddings
* granite-switch: TODO for raw embedding input support
* granite-switch: collapse LoRA tensor constants to .lora_a/.lora_b suffix
* granite-switch: drop n_expert_used hack, guard mul_mat_id buft probe
* granite-switch: stop forcing dense expert counts, read from config
* granite-switch: renamed control_token_gain metadata key to router_gain
* granite-switch: trim header comments to match native style
* granite-switch: collapse LoRA tensors to base name + suffix
* granite-switch: inline suffix checks in tensor op resolution
* granite-switch: drop switch-lora struct comment
* granite-switch: guard router layer index and inline n_slots
* granite-switch: group adapter metadata under {arch}.adapters.* namespace
* granite-switch: add hparams.has_rope(il) for KV-shift rope skipping
* granite-switch: skip arch in test-llama-archs (adapter fixture missing, TODO)
* granite-switch: Keys.Adapters namespace + simplify n_slots
* granite-switch: validate substitute token ids against n_vocab
* granite-switch: bound adapter count and lora rank from GGUF
* granite-switch: reject MTP context type when router_layer is set
* granite-switch: throw on bad adapter metadata instead of GGML_ASSERT
* granite-switch: use ASCII +/- in router K signal comment
* granite-switch: document n_layer_nextn repurpose and its leak points
* granite-switch: gate lora_a/lora_b op mapping on router_layer
* granite-switch: label all three preview model sizes
docker info only proves the daemon answers, so the Windows CI passes
the check and then dies trying to run a linux image. The hosted
Windows runners cannot run one: GitHub states the VMs are not enabled
for nested virtualization and will not be, since they already sit one
level deep and the hypervisor does not support more levels
(https://github.com/orgs/community/discussions/25491). Probing the
image itself skips those tests there, and pulls it before the server
waits for the container id.
The saver called add_kv with LLM_KV_EXPERT_SHARED_FEED_FORWARD_LENGTH twice, the
second time passing n_ff_chexp. gguf_set_val_u32 removes-then-appends, so the second
call clobbers the first: the saved shared_feed_forward_length ends up as n_ff_chexp
(0 for every arch except GroveMoE), and expert_chunk_feed_forward_length is never
written at all.
So a save->load roundtrip of any MoE model with a shared expert loses n_ff_shexp. On
reload the arch falls back to n_ff for the shexp tensor shape, that no longer matches
the saved tensor, and the model FAILS to load. Hits qwen2moe, qwen3-next, granite-moe,
hunyuan-moe, ernie4.5, bailingmoe2, nemotron-h, and the other shared-expert MoEs.
Fix: the second call writes LLM_KV_EXPERT_CHUNK_FEED_FORWARD_LENGTH.
test-llama-archs: set expert_shared_feed_forward_length to a value distinct from n_ff
in the MoE setup so the roundtrip exercises it. Without the fix the reload fails on a
shexp tensor-shape mismatch; with it, every arch roundtrips clean.
* Update build-sanitize.yml
* make it run on pr
* fix thread
* Update build-sanitize.yml
* Update build-sanitize.yml
* just run thread on github machine
The picker mounts whenever a cwd-aware builtin tool is enabled, so
it can open while file_glob_search is not served or was disabled by
the user. Every typed query then fired a search that could only
fail with a raw error.
Gate the debounced search on the tool state, the same way the
mention picker does, and show a message in place of the results
list that explains why search is unavailable. Manual entry with
Enter still commits a directory. The Browse button and the search
scope footer are hidden as well: Browse resolves the picked folder
name through file_glob_search, and the client-side toggle would not
stop that call.
* server: report the isolate working directory from get_info
Without an explicit cwd, get_info fell back to the server process
working directory even when a tools runtime was configured. That named a
host path no tool would ever run in, since an isolate starts in a
directory of its own.
It now asks the isolate for its working directory in that case, and
keeps the process one only when the tools run on the host.
* remove redundant comment
---------
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
The working directory chip showed up as soon as the server exposed any
builtin tool, so a server started with just get_datetime, or a user who
turned every filesystem tool off in the settings, still got a control
that nothing would read.
Tools now declare whether they resolve their paths and run against the
working directory, next to the write permission they already publish in
the /tools listing. The WebUI shows the chip and enables the /cwd
command only when at least one such tool is both served and left
enabled.
get_output runs the waveform work the pipeline defers to it, from a
single trailing window to a full pass depending on the model. Measuring
it keeps the reported total and the audio to process ratio honest.
* feat: Add contenteditable tokenizer for badge/code-chip chat input
* feat: Add source-space undo/redo history for the rich input
* feat: Split text glued to a closing code fence onto its own line
* feat: Add ChatFormContenteditable rich input renderer
* feat : wire the contenteditable into ChatForm with auto-switch gating
* base : slash-command/misc foundation - model icon and focus-selector constants
* feat : slash-command picker and command parsing helpers
* refactor : wire command and @-mention pickers into the chat form
* ui : improve model selector keyboard navigation and load/dismiss
* feat: Unify markdown/raw-text rendering under one setting with migration
* fix: Misc fixes - tool-call subtitle, assistant wrap, progress guards
* feat: Clamp and style numeric settings inputs from registry bounds
ggml_metal_op_norm sized the threadgroup with
`nth = std::min(nth, args.ne00_t)`, which can leave nth not a multiple of
the simdgroup size. The kernels finish their row reduction with a
cross-simdgroup step where each lane of the last simdgroup reads one
per-simdgroup partial sum out of shmem_f32:
if (tiisg == 0) { shmem_f32[sgitg] = sumf; }
threadgroup_barrier(mem_flags::mem_threadgroup);
sumf = shmem_f32[tiisg];
sumf = simd_sum(sumf);
When the last simdgroup is partial it has fewer lanes than the
threadgroup has simdgroups, so the tail of the partial sums is never
read and the row sum is too small. For ne00_t = 33 nth becomes 33: two
simdgroups, but only one lane in the second, so one of the two partial
sums is dropped. The mean and variance are then wrong for the whole row.
Round ne00_t up to a whole number of simdgroups instead. Rounding up
rather than dropping the clamp keeps the threadgroup as small as
possible: deleting the line would raise nth to the next power of two
(ne00_t = 544 -> 1024 instead of 544), which costs idle lanes on 26 row
lengths below 8192 that were already correct, including 1536 and 3584.
GGML_OP_NORM is affected as well as GGML_OP_RMS_NORM - both dispatch
through ggml_metal_op_norm.
No mainstream LLM hidden size hits this: ne00_t is ne00/4 on the
vectorized path, so 4096, 8192, 2048 and friends all give a multiple of
32. It is reachable from other norm shapes, e.g. 320-channel norms.
Add NORM and RMS_NORM cases for ne0 = 33, 132 and 260 across the
existing eps values. 33 exercises the scalar path and 132/260 the
vectorized one, since only those divide by 4.
Before, on M3 Pro:
test-backend-ops test -b MTL0 -o NORM 25/50
test-backend-ops test -b MTL0 -o RMS_NORM 26/51
After:
test-backend-ops test -b MTL0 -o NORM 50/50
test-backend-ops test -b MTL0 -o RMS_NORM 51/51
test-backend-ops test -b MTL0 13943/13943
* base : @-mention picker foundation - glob search, picker nav, highlight
* feat : @-mention file/folder picker and mention badges in message bubbles
* fix: Imports
* feat : wire the @-mention picker into the chat form
* fix: Bound the glob-search result cache key and prune stale entries
* webui: load the model selected via ?model=
Opening the WebUI with ?model= selects the model but doesn't load it. The load only starts when you send your first message, so you wait for it then.
This loads it as soon as the page opens, while you're still typing your prompt. It's what the model dropdown already does, and it isn't awaited, so the UI still works while the model loads.
This is the path the Llama macOS app uses to open the WebUI, so it's a common way in.
* webui: gate the load behind ?load=true
Loading on landing is opt-in, so a plain ?model= link behaves as before and doesn't allocate memory on its own.
* webui: name the chat URL params
Collects the query params the chat routes read into a URL_PARAMS constant, instead of repeating the literals across three files. NEW_CHAT_PARAM folds into it.
The reference implementation has two mutually exclusive prompt layouts.
In non streaming mode the prefill carries the whole utterance text plus
tts_eos summed with codec_pad, and the trailing text hidden collapses to
a single tts_pad row. In streaming mode the prefill carries only the
first text token and the trailing rows stream the rest of the text
followed by tts_eos.
The pipeline built the non streaming prefill but the streaming overlay,
so the talker saw the utterance a second time during generation and read
it twice before emitting codec_eos.
The overlay is now the single tts_pad row that matches the prefill.
* ggml : add fallback definitions for missing aarch64 HWCAP bits
* ggml : require HWCAP_ASIMDHP for the aarch64 fp16 cpu variants
Also rename has_fp16_va to has_fp16, the field gates the whole FEAT_FP16
extension, scalar and vector half-precision arithmetic together.
* ui: read model modalities from the router model list
The router advertises input modalities for every model, loaded or not.
Reading them at list build time lets the UI accept image and audio
uploads for a model selected through ?model=, which has no /props yet.
* enum
The NVFP4 quantization format stores a scaling factor for every group of
16 weights, packed into a single UE4M3 byte.
The SYCL GPU code was converting these scale values using the E4M3 path,
but that's *signed*, and these are unsigned values.
* tests: add SWIGLU perf cases
perf mode had no GLU coverage. Adds SWIGLU at 17408 columns, 512 and
2048 tokens, f16 and f32, with the operands both fused and split.
* sycl: consolidate fused-GLU kernels
They differed only in which op_* they called, so take the op as an argument and share a common launcher.
Their block sizes were all 256, so launch geometry is unchanged;
SYCL_GELU_BLOCK_SIZE and SYCL_SILU_BLOCK_SIZE lose their last users so are dropped.
* sycl: contiguous fast path for the fused GLU ops
o0 == n and o1 == n collapse the de-interleave index math to the
identity, so dispatch a flat kernel in that case. It fires for
ggml_glu_split with packed operands; a fused [gate|up] tensor keeps the
strided path. test-backend-ops perf -o SWIGLU on an Arc Pro B70: split
+14% f16 and +4% f32, fused unchanged.
- In MSL, declaring an array of matrix types like `threadgroup half4x4` causes
a 'no matching constructor' compilation error because MSL matrix types do not
have zero-argument default constructors and threadgroup variables cannot have
initializers.
- Fix this by declaring a POD `threadgroup half` array instead and casting
to `threadgroup half4x4 *` for matrix indexing.
Signed-off-by: JamePeng <jame_peng@sina.com>
* ci: prepare for amd rocm ci
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: fix editorconfig-checker
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: fix device not recognised
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: rename gpu-amd to gpu-hip
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: gpu-hip to gpu-rocm
haha
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* CUDA: allow integrated-GPU host output buffer in debug assert
On integrated GPUs (APUs), the scheduler can legitimately place a graph
node's output on the host-visible buffer, which ggml_cuda_compute_forward
already handles. The debug assert in ggml_cuda_graph_evaluate_and_capture
required every node output to be on the device buffer, so a debug build
aborts on such a node (e.g. attn_residual ADD -> ROCm_Host on RDNA3.5).
The source-tensor assert directly below already permits this via the
integrated + cuda_host exception; apply the same exception to the node's
own output buffer. Debug-only; no effect on release/compute.
Fixes test-recurrent-state-rollback on gfx1151 (Strix Halo).
* ci: enable unified memory for ROCm gfx1151 job
Work around a coherence issue on integrated RDNA3.5 (gfx1151) where GPU
kernels reading mmap-loaded weights can return incorrect output, which
makes test-llama-archs (and real inference) intermittently wrong.
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 uses managed memory, which restores
coherence. Remove once the underlying ROCm/HIP issue is fixed.
* test-llama-archs: skip jamba on HIP backend
jamba produces incorrect output (~0.55 NMSE vs CPU) on the HIP backend on
RDNA3.5 (gfx1151); the SSM kernels need separate investigation. Skip it
for now, matching the existing per-backend carve-outs (WebGPU), so the
ROCm CI can run the test for the remaining architectures.
* ci: use HIP_LAUNCH_BLOCKING for ROCm gfx1151 job
The gfx1151 ROCm CI job produced incorrect inference output (qwen3 perplexity ~88 vs ~9.4) due to an async-execution correctness issue in the HIP path. Serializing kernel launches with HIP_LAUNCH_BLOCKING=1 restores correctness. This replaces the earlier GGML_CUDA_ENABLE_UNIFIED_MEMORY workaround, which did not fix batched inference.
* test-backend-sampler: skip top-k subtests on HIP backend
The ROCm backend does not support the TOP_K/ARGSORT op at vocab scale (no CUB; bitonic argsort is capped at ncols <= 1024), so top-k/top-p backend samplers cannot be offloaded. The penalties, set_sampler, mixed, and top_p subtests assert that offload happened, so they fail on HIP. Skip them until TOP_K is supported on the ROCm backend.
* Update tests/test-backend-sampler.cpp
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
* Update tests/test-backend-sampler.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Jim Wu <ywu@xilinx.com>
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit adds the --model-name flag to the causual and embedding
model conversion scripts.
The motivation for this is that this is the name used for the metadata
field general.name and it can be useful to specify this explicitely if
the default (the basename of the model path) is not what we want.
* vulkan: add debug tooling to get more information about a DeviceLost error
* fix submission threshold applied too late
* use logging macros, throw instead of aborting
* clean up circular dependency
* ggml: add ggml_build_forward_order
ggml_build_forward_expand marks the tensor and all its ancestors for
compute, so using it as a pure ordering hint (keeping q, k and v
together) defeats ggml_build_forward_select: the unselected branch is
forced to run with inputs that were never uploaded. In the mtmd audio
graph this makes GEN_WAV calls execute the GEN_CODE branch with a
stale inp_code0, hitting the get_rows bound assert on CPU.
Add ggml_build_forward_order, which inserts nodes without the compute
flag; the flag is restored when the branch is actually selected.
Switch the q/k/v hints in clip_graph::build_attn to it.
* nit: reduce comments (AGENTS.md)
* server: don't walk Windows junctions in file_glob_search
std::filesystem reports a junction as a plain directory, so the symlink
guard misses it and a junction pointing back at an ancestor is walked
until the path length gives out
read the reparse tag and treat a symlink and a mount point as links,
leaving any other reparse point walkable so cloud placeholders and dedup
stubs still get searched
look junk directory names up case insensitively on Windows, where NTFS
makes Build the same directory as build
test that a junk directory stays selectable while its contents stay out
of search results
* server: report a directory the walk could not read
a directory that fails to open or to iterate was skipped in silence, so
a caller got a listing that looked complete while a whole subtree was
missing: a path over the platform limit, a volume going away, a name the
filesystem rejects
skip_permission_denied never reaches this path, so an error here is an
incomplete answer rather than a deliberate omission, and it now sets the
truncated flag
* server: simplify the file_glob_search listing plumbing
return a small result struct instead of two out params and a caller path
that only fed an error string, taking list_entries from six parameters
down to three
scope the error code to the directory being read, act on the status code
the entry lookups already returned, and treat an unreadable link state as
a link so the walk never descends on a guess
check the deadline when a directory is popped, not only per entry, so a
tree of empty directories cannot outlive the budget
read the path parameter once, and reject an invalid limit the way an
invalid type is already rejected, instead of silently falling back
normalize the resolved path, so a "." or ".." a caller typed reaches
neither git nor the client, and return the generic path form with '/'
separators on every platform, so the base sent to clients no longer needs
a local fixup
* ui: expire cached picker searches
the cache grew for the lifetime of the component: entries went stale
after the TTL but were never removed, so every distinct query typed in a
session stayed in memory
drop expired entries when a new result is stored
* server: address review from @ngxson
trim comments to one line each, and drop two that restate the code
rename junk_lookup_name to get_effective_name, and move it and the link
check to private static members next to junk_dir_names
merge the Windows and Linux link checks into one is_link, so symlinks are
checked everywhere and junctions only add to it on Windows
* server: convert tool paths as UTF-8 on Windows
a narrow path uses the active code page there, so a file name came back
mangled and a path with an accent could not be opened at all
convert explicitly at every crossing between a std::string, which always
carries UTF-8 here, and fs::path
read the home directory through the wide environment, since the narrow
one returns the profile path in the active code page too
the walker no longer normalizes separators by hand, since paths now come
back in generic form
* server: fold the platform branch inside console_output_to_utf8
match the shape of the other helpers, one definition with the #if inside,
instead of two definitions wrapped in #if and #else
inline the single caller helper and trim the comment
* mtmd: support multi-row batching for deepseek-ocr
* mtmd: weave deepseek-ocr rows in one shot instead of per row (#26615)
---------
Co-authored-by: Saba Fallah <sabafallah@gmail.com>
a child process writes in the OEM code page, which is not UTF-8 on a
western Windows install, so accented output reaches the JSON layer as
invalid bytes and gets replaced there, silently losing the characters
run() spawns without a console, so the child never inherits the console
code page and GetOEMCP is the one that applies
decode with MB_ERR_INVALID_CHARS so a wrong code page returns the text
untouched instead of emitting replacement characters, and pass text that
already decodes as UTF-8 through so a child emitting UTF-8 is never
decoded twice
the check drops an incomplete trailing sequence before validating, since
a streamed chunk can end in the middle of a multi-byte character
* Resolve -1 to 1024 instead of ctx-len for samplers
Because of backend-sampling we initialize samplers before the complete
llama_context is there. Therefore, we cannot infer the resolved context
length yet at the time we construct the samplers.
* Shared default of 64 for history-based samplers, remove context_size
* server : extend file_glob_search for UI pickers
* ui : add per-conversation working directory with picker
* ui : add path navigation and search scope to cwd picker
Treat path-like queries (starting with / or ~) as directory navigation
instead of glob-matching the whole query: search the parent for the last
segment, and descend into an exactly-typed directory by listing its
children. Show the effective search scope in the footer and auto-search
on open so the current directory and its siblings appear immediately.
Assisted-by: Claude
* db : persist per-call tool cwd on tool result messages
* ui : abbreviate tool paths under home with a tilde
* ui : show the per-call cwd on exec shell rows
* ui : clarify the synthetic cwd message for the model
* ui : reuse the trailing cwd row on a repeated pick
* ui : don't jump when a cwd row is injected mid-chat
* chore: Formatting
* refactor: Cleanup comments
* ui : unify working directory naming and add a synthetic-message flag
* ui : render synthetic cwd rows without a scroll jump
* ui : decouple the working directory picker into utils and sub-components
* ui : add get_info tool call block
* chore: Formatting
* refactor: Cleanup
* refactor: Cleanup
* refactor: Cleanup
* fix: UI
* server : harden file_glob_search listing (kind enum, timeout, symlink guard, absolute base)
* ui : use persisted isSynthetic flag for cwd rows, drop legacy formats
* ui : cache picker search, fail visibly on native resolve
* ui : escape glob metacharacters in picker search glob
* ui : simplify auto-scroll pin
* chore: Format
* fix: Use `SvelteMap`
* refactor: Post-review fixes
* ui: accept Windows roots in the working directory picker
recognize a drive root (C:) and a UNC share (//host/share) as path
navigation, alongside the POSIX root and ~, so a query like D:\repos
lists that directory instead of glob-matching it under the home dir
split below the root, so a bare drive resolves to its root rather than
to a drive-relative prefix
rewrite backslashes into forward slashes only when the query carries a
Windows root, since a backslash is a legal POSIX filename character
paths keep travelling with forward slashes, which is what the server
returns and what Windows accepts
---------
Co-authored-by: Pascal <admin@serveurperso.com>
This commit contains a suggestion to reduce some code duplication in
common_speculative_init when adding the enabled speculative decoding
configurations.
No tests were added but the existing server tests still passes with this
change:
```console
$ ./tests.sh unit/test_speculative.py -v -x
```
The Python GGUF reader lacked two guards the C++ loader has:
- n_dims read as uint32 with no GGML_MAX_DIMS bound -> crafted file with
huge n_dims triggers oversized memmap read / OOM.
- np.prod(dims) on uint64 wraps silently -> a crafted dims triple can
overflow to a tiny element count, passing an undersized read through.
Add a GGML_MAX_DIMS check and compute the element count with Python ints.
Fixes#25378
bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2
in huggingface/transformers#40936, but it had already been copied into
transformers.convert_slow_tokenizer in huggingface/transformers#30334
(transformers 4.54.1), so import it directly from there.
Applies the same fix to chatglm.py.
* sycl: parallelize the non-contiguous concat kernel
Launch geometry only: the non-contiguous concat kernel launched a single-lane
work-group (1, 1, 1), now it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one.
SYCL_CONCAT_BLOCK_SIZE is defined in `ggml/src/ggml-sycl/presets.hpp`.
llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV),
on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%)
* sycl: cap non-contiguous concat block at ne0
* sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)
* Revert "sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)"
This reverts commit 2709909e790d298cc0746950caff16ad8ecf9d6f.
* sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches
Extends the oneDNN SDPA path (PR #25222) to handle non-F16 KV caches by
dequantizing or converting K/V to dense FP16 on-device before feeding
them into the SDPA graph. The fused systolic kernel then runs identically
to the native FP16 path.
Supported KV types:
- Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl
- F32: cont_to_f16_sycl<float>
- BF16 and IQ types are excluded (no conversion kernel available)
Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only).
F16 KV runs at any length (existing behavior).
Also includes the stream sync fix (stream->wait_and_throw() unconditional,
PR #25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are
always dequantized to separate buffers).
Co-Authored-By: Claude <noreply@anthropic.com>
* docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* common/chat: update DeepSeek V4 templates
Align the DeepSeek V4 templates with the official encoders while keeping parser behavior out of this change.
- Default drop_thinking for DeepSeek V4 history so prior thinking is omitted unless preserve_reasoning is requested or tools are present.
- Add structured output response-format instructions to the V4 templates and pass the schema into template rendering.
- Add a separate Flash 0731 template for the updated high and max reasoning effort mapping.
- Cover reasoning effort, drop_thinking, structured output prompts, preserved reasoning, continuations, and empty tool arguments in template rendering tests.
Official references:
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/encoding/encoding_dsv4.pyhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/encoding/encoding_dsv4.py
Assisted-by: Codex
* Fix deepseek v4 0731 template selection
* remove unneeded lower normalization
* Fix DSML parser to consume the tool call separator
* address aldehir requests
* address aldehir comment
* tests: add model resolution test on synthetic repo listings
Include download.cpp and arg.cpp inside a namespace with hf_cache
monkey patched to serve hardcoded listings, so the resolution and the
model handler assembly are tested end-to-end through the real CLI
parsing, without modifying the tested code and without network access.
Covers the primary, shard, mmproj, sidecar and preset resolution on
layouts mimicking real vendor conventions, replays every case on
permutations of the listing to assert determinism, and asserts the
final wired paths for the spec type auto-selection, the -md precedence
and the fallback suppression.
* tests: keep model resolution checks active and let the handler resolve
Replace assert with a REQUIRE macro alive in Release builds, key the
fake hf_cache by repo id so the real handler init resolves every plan
itself, check the exact shard sets, restrict the permutation exception
to the order dependent picks, and cover dflash and eagle3.
* tests: fix model resolution build on fatal warnings CI and Windows
The namespaced copy of the sources leaves many static functions unused
in this TU, exempt it from the unused warnings. Pre-include the
windows headers so arg.cpp does not pull them inside the namespace.
Declare the renamed copies of the download.h functions, verbatim from
the header and renamed in sync by the macros, so missing declaration
and missing prototype warnings are satisfied on every toolchain.
* tests: fix winsock inclusion order for the model resolution test
WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does
not redefine the socket types afterwards.
* tests: link cpp-httplib to the model resolution test
The test compiles its own copy of download.cpp, which calls httplib
directly, and the private link of llama-common does not propagate the
symbols under lld-link.
* common_http_client
* common: finish the http client wrapper
Add the virtual Head, Get and Post methods and the passthrough
setters to the common_http_client skeleton, move follow_location
into the constructor, expose the underlying client for the ranged
pull path, and rename the missed common_http_client_init call sites.
* tests: rewrite model resolution on the http client stub
Replace the namespace inclusion of the sources by a plain TU: the
common_http_client factory returns a stub serving hardcoded HF API
responses, so the real hf_cache parsing, resolution and CLI handler
run against synthetic listings in an isolated cache directory.
Failures print the named case, the reordering and the actual versus
expected values, the assembly cases use the full command line as
context, and the empty result cases are checked once to keep the
logs short.
* tests: fix the model resolution on Windows and the builds without TLS
Assert the exact expected paths composed like the cache does instead of
suffix matching on forward slashes, set the environment portably, and
serve the stub through an http endpoint so the builds rejecting the
https scheme still reach it. Pause the log so the negative cases can be
replayed on every reordering.
* tests: make the model resolution failures self explanatory
Resume the paused log before the failure report so the CI shows why
the tested code bailed, and format the stub oids portably.
* common: hold the http client factory behind exported functions
The factory was an inline variable, and the Windows shared builds
export functions but not data, so the executable and the DLL each had
their own instance: the stub installed by the test was invisible to
the library, which reached for the real endpoint and resolved nothing.
Route the creation through functions compiled into the library and
format the stub oids portably.
* common: add the http client factory source missed in the previous commit
* common: typedef the http client factory callback
Address review from @ngxson
* tests: serve the model resolution repos over the loopback
Replace the client stub by a real httplib server bound to the
loopback, so no C++ object crosses the module boundary anymore and
the library exercises its own client and transport end to end. The
debug shared build on Windows crashed inside the stubbed path.
* common: add portable common_get_env and common_set_env helpers
Address review from @ngxson
* common: drop the http client factory left without a caller
The loopback server made the stub substitution unnecessary, the client
init builds the real client directly again.
* common: read the model endpoint through the env helpers
* nit: drop the stub leftovers from the model resolution test
* common: align common_set_env and isolate the test cache per run
The POSIX branch now behaves like _putenv_s, so the helper has a single
contract on every platform, and common_get_env already reads an unset and
an empty variable alike.
The model resolution test keys its cache directory on the loopback port,
where two concurrent runs on the same machine used to share one directory
and the initial cleanup of either wiped the other.
* tests: move the model resolution server into main
* tests: support the DSpark sidecar resolution
* common: revert the http client to the plain httplib client
address review from @ngxson
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: add get_info tool
* fix --rpc in docs
* server: harden get_info probe result handling
Report the OS as unknown when the probe process fails to spawn or times
out, so the diagnostic text from run() is never returned as an OS name.
Strip the probe output on both ends, which also drops the blank line
that ver prints before the version on Windows. Name the output and
timeout limits, and report an unreadable working directory as unknown
instead of an empty string.
* server: simplify get_info result handling
Drop the named limits and the working directory error branch, keeping
the probe result handling to a single expression.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* add a direct size condition for `large` weights; the original
dimension condition is insufficient -- q6_K lm_head for gemma-4 E2B
has [1536, 262144], which is big enough to slowdown gemv_noshuffle but
does not satisfy the dimension condition (ne0 >= 2048)
* sampling: enhance penalty handling in common_sampler_init
- Set default value for penalty_last_n based on model context if not specified.
- Ensure penalty_last_n and n_prev are non-negative.
- Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties.
- Implement backend initialization and application logic for penalties, including frequency and presence adjustments.
* tests: add backend penalties sampling tests and utility functions
- Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness.
- Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties.
- Added `test_backend_penalties_sampling` to validate backend penalties with various configurations.
- Enhanced the test suite for better coverage of penalty handling in sampling.
* sampling: add support for top-k penalties in backend sampling
* sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN.
* sampling: enhance penalty comparison tests with masking penalties logic
* add comments on padding
* sampling: add comments on modifications
* add the unit test to cover masked-out token as -INF
* validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values
* refactor: test functions to share logic and be less verbose
* add test to cover case where previously penalized token is not part of candidates
* remove comments
* remove redundant penalty_last_n initialization and validation in common_sampler_init
* add support for penalties in sampler chain with configurable positions
* add validation for penalty parameters and enhance tests for non-finite values
* add context parameter to common_sampler_init and set default for penalty_last_n
* add llama_n_ctx parameter to common_sampler_init for improved sampler initialization
* replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor
* add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64
* add test for backend penalties sampling after top-p with large history window
* remove as unused
* add is_disabled method, tensor logits reshape, add rest review suggestions
* clarify comment
* CUDA: Fix data-races when reusing block_reduce
block_reduce currently doesn't resync after reading from SMEM, causing
potential data-races when reusing SMEM for multiple reductions.
One may consider simply always adding this in block_reduce, but this
comes at a potential perf cost
* double-buffering for single-row softmax
* double-buffering for norm as well
* Add comment
* Add explanatory comment to block_reduce
* Specify need for + do memory barrier only in multi-warp scenario
* Implement review-suggestion from @gaugarg-nv
* llama : MTP support for DeepSeek V3.2
* model : no need to include MTP layers during DeepSeek V3.2 model type discovery
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* feat(silu_back): implemented silu_back op for f32
* fix(silu_back): removed redundant asserts in ggml-metal-ops.cpp function ggml_metal_op_silu_back.
- Implement GGML_OP_DSV4_HC_COMB, GGML_OP_DSV4_HC_PRE, and
GGML_OP_DSV4_HC_POST with SIMDgroup register and shuffle optimized kernels.
- Add Metal dispatch and support plumbing and test the production Sinkhorn
iteration count and embedding width.
Assisted-by: Codex
Co-authored-by: Thiago Padilha <thiago@padilha.cc>
The dspark- files resolve like the other speculative sidecars: the
-hfd tag applies to them, a requested sidecar resolves without a full
model at the tag, and an explicit -md selection disables the discovery.
When no type is requested, dspark outranks dflash in the auto-selection
since its sidecar carries the extra Markov head.
Incrementing `ref_count` at the beginning is important later
in the `free()` method of the `ggml_backend_opencl_context` at program end.
If we do not increment the `ref_count`, the result would be -1 here,
and consequently, the profiling data would not be flushed and written.
( #ifdef GGML_OPENCL_PROFILING )
* cli : persist reasoning_content in chat history
llama-cli collected reasoning from the stream for display but only
stored assistant content in messages, so --reasoning-preserve could
not re-inject prior thoughts on later turns.
* vulkan : add pool1d push constants and pipeline field
Declared data structures needed for POOL1D OP, which are the vk_op_pool1d_push_constants struct and pipeline_pool1d_f32 field.
* vulkan : add pool1d compute shader
Added pool1d.comp for Vulkan backend mirroring the existing pool2d shader.
* vulkan : add full GGML_OP_POOL_1D support
Added pipeline creation and op dispatch for 1D pooling in the Vulkan backend.
* vulkan : fix pool1d shader logic
Registered pool1d_f32 in vulkan-shaders-gen.cpp and fixed tensor dimension indices and avg pool scale.
* vulkan : fix pool1d end boundary crash and expand test coverage
Fixed an issue where the shader crashed when the end boundary was negative when k0 < p0. Also, added more test cases related to this fix.
* Removed crash guard for Intel
Crash fixed from driver 32.0.101.8860
* Added driver version check for windows
* Change to convert from driverVersion rather than string
* No need to use signed
* Refactor
* allow GPU other than Xe2+
* adjusted function body position
* llama : load MTP tensors only if they are really used
* llama : skip loading MTP (if not used) in remaining models that support MTP
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* SYCL: add oneMKL GEMM flash attention for XMX-accelerated prompt processing
* fattn-mkl: fix interleaved dst layout in normalize kernel
- Fix mkl_fa_normalize_head: use interleaved dst layout
((query * n_q_heads + head) * DV) matching TILE's
flash_attn_combine_results. Previously used dense head-major
layout which wrote head outputs to wrong addresses, corrupting
attention for all models except Qwen3.6-27B (where GQA=6 heads
were sparse enough to avoid visible overlap).
- Remove 7 redundant stream->wait() calls — SYCL in-order queue
already serializes pure SYCL kernel dependencies. Retain only
the 4 MKL GEMM ↔ SYCL handshake barriers (oneMKL GEMM uses its
own internal queue that does not respect SYCL in-order).
- Remove unused dst_row_stride, diagnostic clutter, and dead
K/V hex dump (fa_diag block in fattn-mkl.cpp).
- Add MKL_FA_DISABLE=1 env var for A/B testing.
- Add FA-DISP watchdog (MKL_FA_DEBUG=1) and FA-DIAG output
fingerprint (MKL_FA_DIAG=1) in fattn.cpp.
Tested: Gemma-4-26B, Gemma-4-31B, Qwen3.6-27B, Qwen3.6-35B-A3B
Perf (B70/Battlemage, 32K, q8_0 KV):
Gemma-4-26B: 1473 t/s MKL vs 746 TILE (1.97x)
Qwen3.6-27B: 609 t/s MKL vs 330 TILE (1.85x)
Co-Authored-By: Claude Code on DeepSeek-v4-Pro
* Thank you for the review feedback: rename env vars, use GGML_LOG_INFO, document in SYCL.md
Completed the following:
- Rename MKL_FA_DISABLE → GGML_SYCL_ENABLE_MKL_FA (inverted: 0 to disable)
- Rename MKL_FA_DEBUG → GGML_SYCL_MKL_FA_DEBUG
- Rename MKL_FA_DIAG → GGML_SYCL_MKL_FA_DIAG
- Replace fprintf(stderr, ...) / fflush(stderr) with GGML_LOG_INFO() macro
- Document all three env vars in docs/backend/SYCL.md under Runtime
- Add comment explaining MKL FA activation trigger (flash-attn + quantized
KV cache + batch-size >= 1024 + n_kv >= 1024)
Resolves review feedback from arthw.
Again, thank you!!!
Co-Authored-By: Claude Code on DeepSeek-v4-Pro
* Thank you for the review feedback round 2: use ggml_sycl_get_env, remove dup waits, gate perf macros
- Replace raw getenv() with ggml_sycl_get_env() in all 4 env-var checks
(fattn.cpp: GGML_SYCL_ENABLE_MKL_FA, GGML_SYCL_MKL_FA_DEBUG,
GGML_SYCL_MKL_FA_DIAG; fattn-mkl.cpp: GGML_SYCL_MKL_FA_DEBUG)
- Remove duplicated stream->wait() before ev.wait_and_throw() in GEMM
KQ and GEMM VKQ — ev.wait_and_throw() already waits for completion
- Gate MKL_ACCUM macro behind do_print so timing accumulators are
no-ops in normal operation
- Remove redundant MIT/Intel copyright header from fattn-mkl.cpp
- Remove unused #include <cfloat>
- Expand SYCL.md MKL FA docs with step-by-step activation trigger
and example llama-cli command
Again, thank you!!!
Co-Authored-By: Claude Code on DeepSeek-v4-Pro
* fattn-mkl: enable MKL FA for all KV cache types
Remove the quantized-only restriction on MKL activation — the MKL
kernel converts any non-F16 K/V to F16 via to_fp16_sycl before GEMM,
so F16 (default), BF16, and F32 caches all benefit from XMX hardware
acceleration. The type restriction was an unnecessary gate.
Before (F16/BF16 default cache + FA on at 32K prefill): ~356 t/s (TILE path)
After: ~670 t/s (MKL path, matching quantized-cache baseline)
Minimal change: two conditions removed, one comment updated in fattn.cpp.
No kernel or conversion code changes — the dequant pipeline already
covers all types.
* fattn-mkl: rename mkl_disable -> mkl_enable for clarity
* fattn-mkl: refine MKL FA dispatch gates
Three changes:
1. Remove quantized-only restriction - MKL FA activates for all
KV cache types (F16 default, BF16, F32, quantized). The MKL
kernel converts non-F16 K/V via to_fp16_sycl before GEMM.
2. Rename mkl_disable -> mkl_enable to match env var
(GGML_SYCL_ENABLE_MKL_FA).
3. Replace batch-size threshold with Q->ne[1] >= 32 gate.
Keeps TG (Q=1) and MTP drafts (Q=3-8) on VEC path where
fused kernel beats MKL launch overhead. Routes all
multi-token prefill through XMX-accelerated GEMM.
Production data confirms Q patterns: 1-8 TG, 32-127 cache reuse,
128+ full reprocess. At 32K F16/BF16 FA-on: 356 -> 670 t/s.
* ggml-sycl: fix F16 cache + MKL FA multi-turn corruption; add gate guards
Two changes:
1. Always copy F16 K/V to dense row-major buffers before MKL GEMM.
Previously F16 was read in-place with raw tensor strides. During
multi-turn conversations, the accumulated KV cache had different
stride properties than a fresh prefill, producing corrupted outputs.
Now dense F16 gets a fast memcpy; interleaved (Gemma) gets a strided
copy kernel. This matches what the quantized paths already did through
to_fp16_sycl.
2. Gate MKL FA on unsupported op params (max_bias, logit_softcap, batch
dim mismatch) and pathological F16 strides (nb[1] not a multiple of
ne[0]*2). These conditions would previously crash inside the MKL
kernel. Pathological strides (test-only) and ALiBi/softcap fall
through to TILE/VEC which handle them correctly.
The stride check uses modulo rather than equality, so both dense
(nb1 == ne0*2) and interleaved (nb1 == H * ne0*2) pass — all real
models use these layouts. Only test cases with overlapping rows
(nb1=32 or nb1=75 for ne0=40) are blocked.
Thanks to hmscider for the oneDNN FA PR (#25222) which surfaced the
same insight: always normalize inputs to contiguous F16 before GEMM.
Co-Authored-By: Claude Code using DeepSeek-V4-Pro <noreply@anthropic.com>
* fattn-mkl: fix quant+GQA KV strides, tighten MKL gate, add K>=1024 tests
Adding K>=1024 flash-attn test cases surfaced several MKL bugs:
- Quant K/V with a padded seq-view (real KV cache) used the wrong
strides in the dequant path... only the true Gemma interleave
layout should reconstruct strides. nb[2] vs ne[1]*nb[1]
- Gate was firing on shapes the kernel doesn't handle: head_dim < 64
or not a multiple of 64, MHA, attention sinks, and
bf16 decode... fell through to vec which no bf16 case.
Gate MKL to the validated envelope: gqa>=2, head_dim 64 through 512
(has to be a multiple of 64) with matching K/V head size, mask,
no sinks/alibi/softcap... everything else falls back to tile.
Covers Qwen Dense/MoE and Gemma4 Dense/MoE
Ran test-backend-ops -o FLASH_ATTN_EXT: 3641/3641 pass.
Perplexity unchanged... 6.7267 MKL vs 6.7290 stock using
Qwen 27b q5_k_xl
* Update ggml/src/ggml-sycl/fattn.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* Update ggml/src/ggml-sycl/fattn.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* Update ggml/src/ggml-sycl/fattn.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* fattn-mkl: bound attention scratch so it doesn't grow with batch or context... also dropped the bf16 comment in fattn.cpp per arthw review.
* Update ggml/src/ggml-sycl/fattn-mkl.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* Update ggml/src/ggml-sycl/fattn-mkl.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* apply arthw suggestions: enum for dequant modes, macro for wg_size, env-var one-liners
---------
Co-authored-by: Claude Code using DeepSeek-V4-Pro <noreply@anthropic.com>
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized
* llama : enforce the same K and V cache types for MLA models
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* improve fa of quantized kv cache
* Fix some bugs and some comments.
* fix v type check and some comments
* Fix build error caused by rebasing
* editorconfig checking pass
* add bool cwhn = true to conv_2d test cases
* add layout check at graph building time
* extend layout checks for conv2d.cu kernel
* in CPU back-end kernel needs to be stored contiguously to prevent test failures with cwhn=1
* trim white space
* do op support check in vulkan backend
* fix CI failure and vulkan run-time assert failure by introducing new graph build-time check in ggml_backend_vk_device_supports_op
* add additional check in support_op function for Vulkan to fix run-time assert failure
* metal: fix memory leak if model is freed without any GPU operations
* metal: run dummy work only if residency sets are used
* metal: wrap function in #if defined
* metal: measure system-wide wired memory in test
* metal: always build regression test
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
---------
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
* fix: single-flight conversations store init
* refactor: remove unused legacy-migration util
* fix: make createSystemMessage transactional
* fix: delete message branches cascading on edit/regenerate
* fix: stop stamping lastModified on conversation metadata updates
* fix: count cascaded forks in bulk delete toast, bulkify deleteAll
* refactor: drop redundant conversation list respreads
* refactor: create conversation in a single write
* fix: use table constant in toggleConversationPin
* fix: keep the system message placeholder out of the edit form
* fix: keep focus in the system message editor after opening it
* fix: focus the main chat form after submitting a system message
* fix: update timestamp of the correct conversation on stream completion
ggml_cuda_should_use_mmq() selects MMQ purely from the quantization
type. The current MMQ configurations are designed and maintained against
a minimum of 48 KiB per-block shared memory, the limit provided by
NVIDIA Pascal GPUs and later. On devices that report less, no supported
MMQ tile fits and mul_mat_q_switch_J() aborts when every tile size
exceeds the device's per-block shared memory budget.
Disable MMQ when smpbo < 48 KiB so the caller falls back to the BLAS
path instead of hitting GGML_ABORT. Some current MUSA QY1 devices
report only 28 KiB and are covered by this guard.
Reproduced on a Moore Threads MTT S70 (arch mp_21, 28 KiB shared memory
per block) with an RWKV-7 0.1B Q8_0 model:
$ llama-bench -m rwkv7-g1d-0.1b-Q8_0.gguf -p 128 -n 0
J_best=0
ggml/src/ggml-cuda/template-instances/../mmq.cuh:1521: fatal error
(core dumped)
Only prefill (batch > 1) is affected; token generation is fine. After
the fix the same device falls back to the BLAS path:
Q8_0 pp128 1470.7 t/s, tg8 55.3 t/s (was: abort)
FP16 unchanged
Q4_K_M unchanged
This matches a -DGGML_CUDA_FORCE_CUBLAS=ON build (pp128 1464.2 t/s),
which confirms the fallback path is the one being taken.
This is not MUSA-specific: any device with less than 48 KiB per-block
shared memory is affected.
Co-authored-by: KakaruHayate <KakaruHayate@users.noreply.github.com>
Adds trace logging in server-context.cpp for slot similarity checking
during prompt cache slot selection, including skip reasons and similarity
calculation details.
Assisted-by: llama.cpp:Qwen3.6-27B
* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)
Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)
Opt GLM-5.2 into the supports_mtp_export contract (post-#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add overlap glu variant to support all archs, fix recurrent-state-rollback test
* format
* Fix all arch overlapped ranges
* format
* diagnose bus error on apple ci
* More testing
* more testing
* more targeted testing
* Fix bug in alignment for > 4gb buffer offsets
* Fix bug in view offsets
* Try avoiding multi_buffers
* not fixed yet, more logging :(
* Handle edge case in set_rows
* Try looking at view source
* Skip deepseek32 for now and clean up trace infrastructure
* simplify skipping
* last cleanup
* actually final cleanup
* update handling of overlap
* format
* try skipping other failing model
The Adreno KQ/KQV image1d kernels (ggml_cl_mul_mat_kq_kqv_adreno) ignore
dim 3 entirely: the sub-buffer covers only nb02*ne02 bytes and the kernel
receives no ne03/ne13/nb03/nb13 arguments. With the unified KV cache,
multi-sequence batches (e.g. llama-perplexity with its default -b 2048,
n_seq=4, or a multi-slot llama-server) present KQ/KQV as 4D tensors with
ne3 = n_stream, so every stream past the first reads the first stream's
K/V and produces garbage. Flash attention masks the bug where it is
enabled; devices where FA is declined (e.g. Adreno 740) hit it with
default settings.
Route ne03/ne13 > 1 to the general path, which handles dim 3, and honor
view_offs when creating the sub-buffers (currently always 0 for tensors
reaching this function, but the function would silently misread any
future view).
Llama-3.2-1B-Instruct Q4_0, wiki.test.raw, 8 chunks, -ngl 99:
- Adreno 740, default: PPL 1817.64 -> 15.61
- Adreno 740, -fa 0: PPL 1941.64 -> 15.61
- Adreno 840, -fa 0: PPL 1943.90 -> 15.50
- single-stream (-b 512) results unchanged (15.6090)
- test-backend-ops -o MUL_MAT on 740: identical before/after (909 OK,
12 pre-existing q6_K failures)
* mtmd : add Nemotron 3 Nano Omni support (parakeet)
This commit adds support for the subsampling and encoder part of
Nemotron Nemo 3 omni model.
The Parakeet subsampling/encoder were taken from parakeet.cpp which
is currently a pull request against whisper.cpp. I've tried to copy the
code a close as possible to hopefully enable easy patching between the
these two project later.
Refs: https://github.com/ggml-org/whisper.cpp/pull/3735
* mtmd : generate rel pos tensor in graph instead of in conversion [no ci]
This commit removes the generation of the relative positional tensor in
the model conversion script and instead computes it in the encoder
graph. This is only done for the window of positions required for the
current audio sample.
* mtmd : add clip_get_model to clip API [no ci]
This commit adds a function to get access to the clip_model. It also
removes the two functions clip_get_mel_filter_tensor, and
clip_get_window_tensor(const struct clip_ctx * ctx) which can now use
clip_get_model to access the model tensors that it needs.
* mtmd : read mel_filters and window into hparams
* mtmd : use set_input_f32 lambda [no ci]
* mtmd : add better asserts for mel_filters and hann window [no ci]
* mtmd : add missing size_t cast
* mtmd : change type of pad to size_t
* mtmd : zero initialize samples_padded
* mtmd : remove unsued ctx member from parakeet preprocessor
* mtmd : make log_mel_spectrogram_parakeet_worker_thread private static
* mtmd : sync/update parakeeet impl with latest whisper.cpp
This commit updates the parakeet code in mtmd to reflect the latest
updates to parakeet.cpp in whisper.cpp.
A follow up commit will address the currently hardcoded dw_pad and see
if we can add n_conv_kernel as a model metadata field.
* mtmd : add audio_conv_kernel_size to model conversion
This commit updates the model conversion to read the conv_kernel_size
field from the sound_config section of the models config.json file.
It then uses this field instead of the hardcoded values in parakeet.cpp.
* mtmd : cleanup [no ci]
* conversion : call super().filter_tensors [no ci]
* do not discard result of super filter_tensors
* mtmd : use build_mm instead of ggml_mul_mat
* mtmd : use build_ffn
* mtmd : move and reuse get_vector lambda
* mtmd : use build_inp_raw for parakeet
* mtmd : throw exception in get_scalar instead of assert
* mtmd : fix std::min call
* mtmt : use .c_str in throw clause in get_vector
* mtmd : check for F32 type and non-empty tensor in get_vector
The get_vector lambda is used by get_scalar but also standalone to read
in the mel_filters and the window data. Therefor we are not checking
for 1D tensors but allowing multiple dimensions. We do have a check in
get_scalar to verify the size of the vector.
* mtmd : replace hardcoded 1101 for n_tokens_real
* mtmd : assert subsampling_factor is 8
This commit adds an assert of the parakeet subsampling factor to check
that it is 8.
The motivation for this is that this model currently has three
convolutions with a stride of 2. If the underlying model updates the
subsampling factor these convolution operations will need to be updated
and this will produce and error if this occurs.
* mtmd : remove unused ggml_tensors attn_pos_w and mm_norm_w
* mtmd : remove single thread path
This commit removes the single thread path which was a left over from
the original parakeet.cpp where n_threads is configurable.
* fix some security issues
---------
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* vulkan: add iq4_nl support back to FA
I was originally concerned about wasting shared memory on the LUT, but it's small
and unlikely to matter in practice.
Also support q1_0 for non-coopmat2.
Fixes#23681
* remove q1_0 FA support
* ggml-cuda: add chunked SSD matmul for Mamba-2 prefill acceleration
* cuda: added SSD CICD fixes for CUDA / HIP / MUSA / MSVC.
* ggml-cuda: review comments fixed.
* ggml-cuda: Fuse M matrix materialization into pre_matmul kernel and enabled test.
* ggml-cuda: test updates and fixes
* ggml-cuda: test updates to remove hardcoding of tensor initialise data limits.
* ggml-cuda: ssd minor review comment fixed.
* ggml-cuda: ssd minor CICD fixed.
* CUDA SSD: Fixes correctness by promoting s0_stride_seq to int64_t, improves memory coalescing in ssm_ssd_prepare_dt_kernel, and boosts efficiency by merging B_weighted and C_scaled; also addresses prior review comments.
* cuda: fix sdata read-write race in prepare_dt fallback scan loop
* spec: add DSpark speculative decoding
DSpark (DeepSpec, 2026) on top of the merged DFlash drafter. It reuses the
DFlash encoder/decoder graph, target feature extraction and KV-cache injection,
and the verify/accept path unchanged; the draft model is a new "dspark" arch
adding a low-rank Markov head (markov_w1/w2) and an optional (unused here)
confidence head. No new public APIs.
The proposal is the only change: the block is anchor-first (position 0 already
predicts the first draft) and the decoder graph applies a semi-autoregressive,
previous-token conditioned logit bias in-graph, chained per block position:
logits'(i) = logits(i) + markov_w2 . markov_w1[prev(i)]
prev(0) = the block's anchor token, prev(i>0) = argmax(logits'(i-1))
vectorized across all blocks in the batch; the anchors are fed through a
dedicated graph input (token 0 of every block). Greedy stays lossless
(verify unchanged, same as DFlash).
- new arch "dspark" (llama_model_dspark : llama_model_dflash, reuses the graph,
loads the markov/confidence tensors; shares the target's embed/lm_head).
- Qwen3DSparkModel converter.
- new spec type "draft-dspark" (common_speculative_impl_draft_dspark :
common_speculative_impl_draft_dflash, overrides draft() only: submits whole
anchor-first blocks and greedily reads back the biased logits).
* spec: read draft block size in the dflash impl
* docs: add DSpark section to speculative.md
* spec: keep dspark block size read in the dspark impl
* dspark : add TODOs for incomplete parts
- confidence head is loaded but not used yet
- confidence-scheduled prefix pruning is not implemented
- the in-graph Markov chain is greedy-only
- only Qwen3 backbones are supported for now (also noted in docs)
* spec: fold DSpark into the DFlash arch
Address review: drop LLM_ARCH_DSPARK and the dspark.block_size /
markov_rank GGUF keys. A DSpark draft now converts to a DFlash GGUF;
the Markov head tensors are detected by presence (like eagle3 d2t),
block_size is read from the existing dflash.block_size key, and the
block anchors are taken as a strided view of the decoder's token
input instead of a separate graph input.
* spec: add confidence-based draft pruning for DSpark
The DSpark confidence head predicts per-position acceptance of the
drafted block. --spec-draft-conf-min truncates the block at the first
position below the threshold (default 0 = disabled).
* fold the dspark impl into dflash, selected by spec type
* address review comments
* dspark: clean up and improve naming
* update readme
* remove trailing whitespace
* dflash: draft full n_max blocks, defer dp.n_max to the central truncation
The DSpark markov head views the draft batch as a uniform [n_seqs x block]
grid, but the per-seq dp.n_max clamp could produce blocks of different
sizes, silently corrupting the strided views and the resulting logits.
Drop the clamp and always draft the full n_max block for every sequence:
dp.n_max is already enforced by the central truncation in
common_speculative_draft(), the same way eagle3 handles it.
Co-authored-by: Zaire404 <3147879462@qq.com>
* dflash: assert the markov head block-uniformity invariant, require the conf head
With the draft batch always submitting equal-size n_max blocks, a
non-divisible token count can only mean the batch was split across
ubatches or a caller broke the layout - fail loudly instead of silently
dropping the markov bias. The block_drafts > block_size early return
stays: worst-case graph reserve passes legitimately build with
n_seq_tokens > block_size.
Also make conf_proj required when the markov head is present: the
confidence head is part of the DSpark checkpoint format, and a missing
head would otherwise leave --spec-draft-conf-min silently reading stale
embeddings instead of confidences.
Co-authored-by: Zaire404 <3147879462@qq.com>
* dspark: fold conf_min into p_min
p_min and conf_min express the same thing - the minimum predicted
survival probability for a drafted position - differing only in how the
estimate is obtained: token probability for regular drafters, the
trained confidence head for DSpark. The DSpark readback never used
p_min, so reuse it for the confidence threshold and drop the separate
--spec-draft-conf-min flag. Both defaulted to 0 (disabled), so behavior
is unchanged.
Co-authored-by: Zaire404 <3147879462@qq.com>
* dflash: note the confidence broadcast workaround
Requested in review: the ggml_repeat only adapts the [1, n_tok]
confidences to the n_embd-wide embd_nextn transport so that
llama_get_embeddings_nextn can be reused - not a placeholder.
Co-authored-by: Zaire404 <3147879462@qq.com>
* cont : clarify
[no ci]
---------
Co-authored-by: Ruixiang Wang <wangruixiang07@outlook.com>
Co-authored-by: Zaire404 <3147879462@qq.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sycl: fix use-after-return of the SDPA scale in the oneDNN flash-attention path
The scale was uploaded with an async memcpy sourced from a stack local. On the
in-order queue that copy is ordered behind the K/V staging kernels; once n_kv is
large enough (>= ~26k observed on Arc Pro B70) the staging outlives the host
stack frame and the copy reads recycled memory, feeding the SDPA a garbage scale.
Output then collapses to a single repeated token and the KV cache is poisoned
for the rest of the session.
Short contexts win the race by accident, and test-backend-ops caps
FLASH_ATTN_EXT at kv=1024, which is why CI never caught it. The previous
device_count > 1 wait_and_throw() gate (and reverting it, PR #25741) fixes the
symptom only by keeping the frame alive across the copy at the cost of a host
sync on every FA call.
Fix: cache one device scalar per (device, value) -- the scale is constant per
model -- and upload it synchronously once. The single-device fast path (no
per-call host sync) is then safe: every device-side hazard already serializes
on the in-order queue. The multi-GPU conservative wait is kept unchanged.
Also:
- GGML_SYCL_FA_ONEDNN_MAX_KV env (0 = unlimited): optional n_kv ceiling that
routes very long sequences to the native FA kernel.
- test-backend-ops: FLASH_ATTN_EXT F16 cases up to kv=65536 (Qwen3.6-27B
geometry hsk=hsv=256 GQA 6, and hsk=128 GQA 4), closing the kv=1024 blind
spot. Note the race itself needs a live multi-op pipeline to reproduce;
single-op runs pass even on broken builds.
Verified on Arc Pro B70 (bmg_g31), Qwen3.6-27B Q4_K, -c 131072: output
byte-identical at temp 0 to the native FA path through 32k-deep prefill, with
prefill depth-flat at 820-840 t/s (vs 340-350 native at 32k depth).
Assisted-by: Claude Fable 5
* sycl: handle GGML_SYCL_FA_ONEDNN_MAX_KV like the other runtime env vars and document it
Review feedback on #25880:
- read the variable once at backend init into g_ggml_sycl_fa_onednn_max_kv via
ggml_sycl_get_env, and print it in the startup env listing (-lv 4 shows it)
- document GGML_SYCL_FA_ONEDNN and GGML_SYCL_FA_ONEDNN_MAX_KV in the SYCL.md
runtime table
Also trim the added FLASH_ATTN_EXT cases to kv={4096,16384}: the 32768/65536
shapes exceed the legacy NMSE threshold on both the oneDNN and native kernels
(long-sequence fp16 accumulation drift, present before this PR) and would fail
CI for an unrelated reason.
Assisted-by: Claude Fable 5
* sycl: clarify GGML_SYCL_FA_ONEDNN_MAX_KV default is disabled
Assisted-by: Claude Fable 5
* sycl: state default behavior of GGML_SYCL_FA_ONEDNN_MAX_KV explicitly
Assisted-by: Claude Fable 5
* Update ggml/src/ggml-sycl/fattn-onednn.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* sycl: write the SDPA scale from a kernel instead of caching it
The per-(device, value) scale cache was a function-local static
unordered_map with no synchronization, so concurrent backend instances
could access and rehash it at the same time.
Write the scalar with a single_task instead. The value is captured into
the command, so no host memory has to outlive the call -- which is what
the use-after-return fix needed in the first place. That removes the
shared container, the leaked device allocation and the string key, and
it also closes the remaining async-memcpy-from-a-stack-local on the
first flash-attention call.
Ordering does not rely on timing: the queue is created with
sycl::property::queue::in_order and the dnnl stream wraps that same
queue, so the write completes before the SDPA reads the scalar. The
multi-GPU wait_and_throw() branch is unchanged.
Also drop the <cstdlib> include, which is unused.
Assisted-by: Claude Opus 5
---------
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* common: fix explicit -md precedence over draft sidecar resolution
Follow-up of #25955, an explicit --model-draft file given with -hfd
was silently overridden by the sidecar resolution of the draft repo,
and its path was never resolved to a local file.
An explicit draft file selection now disables the sidecar resolution,
so the manual CLI configuration wins over the automatic one.
* common: apply the -hfd tag to the sidecar resolution
The sidecar selection was anchored on the primary of the draft plan,
so a tag without a matching full model aborted the whole plan, and
the sidecar quant silently followed the default model pick.
The tag now anchors the sidecar directly: exact tag match first, then
closest quant to the tag, and a requested sidecar resolves even when
no full model matches the tag. A wired draft sidecar also counts as
an explicit draft, so the main plan no longer downloads a second one.
* common: promote speculative load logs from trace to info
Show the loaded draft model and the MTP draft context at the default
verbosity, for consistency with the mmproj and primary logs.
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* server + ui: refactor resumable stream routes to query string conv_id
The conversation id can embed a model name containing slashes
(ggml-org/...) in router mode, which the decoded path splits before the
:conv_id param is captured, so stop and resume never matched the
session. Move the id to the conv_id query string on the public routes
and on the internal router -> child hop, where slashes survive
encoding. Handlers are unchanged since query and path params land in
the same map. Add a regression test with a slashed model name.
* server: move stream route docs to server-stream.h
Address review: ngxson wants the main server.cpp registration code kept
clean and simple, with route-level explanations living in the header.
Move the query string rationale and the lookup ownership note next to
the handler declarations in server-stream.h, and shorten the wiring
comment to a pointer.
* server: cancel a pending request when its stream is stopped during model load
The conversation was registered in the conv map only after the blocking
autoload wait, so a stop issued while the model loaded found nothing to
cancel and the request went on to generate an orphan once the load
ended. Register the conversation before the wait and give the entry a
ticket: a stop erases the entry, and the parked request checks its
ticket after the wait and aborts with 400 instead of starting. A newer
request on the same conversation replaces the entry, so only the
stopped request is cancelled. Add a regression test that stops during
the load window.
* server + ui: resume a stream after a page reload during model load
A pending request died with the client socket when the page was
reloaded while its model was loading, so no session ever existed and
the conversation had nothing to recover. A session request that waited
for a load now detaches from the client socket and reaches the child
regardless, the session buffer receives the generation, and the resume
route answers 503 while the owner is loading so the client retries
instead of dropping its state. The WebUI persists the pending stream at
send time, quietly polls on 503, and attaches once the session exists.
Add a regression test that drops the client during the load window.
* ui: show the model load progress again after a page refresh
The resume wait was invisible, so a conversation refreshed while its
model was loading showed nothing until the first byte. On a 503 from
the resume probe, mark the conversation as loading again so the
assistant row persisted at send time renders the processing info, and
target the model frozen in the persisted stream state for the
progress, since the row has no model yet and the dropdown may not be
restored.
* fix CI
* fix CI bis
* Add preliminary MiniMax-M3 support
Text-only port that re-uses existing components: MiniMax-M2 style GQA with
per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and
routed/shared experts, and swigluoai activation. Sparse attention is not
yet supported (dense fallback); vision tower and MTP heads are dropped.
* MiniMax-M3 vision tower (mmproj + clip graph)
* Delete m3_vision_ref.py
* Update clip.cpp
* MSA
* Update constants.py
* Update minimax.py
* Cache creation. Working withotu flash attention
* Added flash attention for sparse layers
* Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx
* Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking
* Implement sparse attention calc out of stock ops.
* Fix a cache allocation and cont issue
* Fixed -fa auto crash, flagged debug spots
* Delete vocab.json
* Delete model.safetensors.index.json
* Delete generation_config.json
* Delete Minimax directory
* Handled multi stream case to fall back on Dense Attention
* Development scaffolding cleanup. No functional change to the decode or
4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the
selection-parity validation.
* Remove redundant comment from minimax-m3.cpp
* Changed 3 Gelu Ops for vision into Gelu_erf ops
* Assert that n_kv is multiple of 128
* Rename MSA index tensors to indexer convention
Note: All GGUFs generated before this change will need to be regenerated.
* Fix incorrect Assert
* Review driven changes (#3)
* Remove comment from conversion minimax.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Remove whitespaces from constants.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Tighten comment in minimax.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* inherit MiniMax-M3 from MiniMax-M2
* drop dead text_config fallbacks
* Add indexer writer methods
* Reuse LLM_FFN_SWIGLU_OAI_MOE
* Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention
* Fix conversion error /gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Update gguf-py/gguf/gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Remove whitespace in src/llama-kv-cache.cpp
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Remove Whitespace in Update src/llama-model.h
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Remove whitespace in src/llama-hparams.h
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Update minimax_m3.cpp
Rewrite code comment based on feedback and to better reflect the actual architecture, and reuse existing build_vit
* Rename minimax_m3.cpp to minimax-m3.cpp
* Update CMakeLists.txt
* Remove debug code from clip.cpp
* Update clip.cpp
* Update comments in tools/mtmd/models/minimax-m3.cpp
* Permute Q/K at conversion, drop precomputed sin/cos
* Log cache size on launch, block ctx shift, support prompt caching
Log indexer cache size on launch
Disallow ctx shift
Support prompt caching
* Update minimax-m3.cpp
* Optimize implementation, add multi stream support.
Fully rewrote minimax-m3.cpp for speed and buffer size gains:
Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3]
Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill
Decode: ~25 nodes/layer vs ~50, no per-group concats/conts
Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection
can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token)
In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k
Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq
Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support.
* set default cache type to F32
* Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in
* remove F16 downcasts in MSA attention, force F32 indexer score accum
* Add Minimax eos to llama vocab
* Guard edge case where idx cache can become stale after a tail trim
* Update llama-kv-cache.h
* Update llama-kv-cache.cpp
* Update llama-kv-cache.cpp
* Update llama-kv-cache.h
* Change resize Pad to none, resize alg to Bicubic Pillow
* Review driven changes
* Update llama-kv-cache.cpp
* rm unrotated pos_t
* fused rope w + pad
* rename merge --> merger for consistency
* add review skill for mtmd
* graph should use hparams n_merge
* fix lint
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
The in-conversation system-message edit textarea had a fixed min-height and no auto-resize, so long messages were crammed to 2 lines. Reused existing autoResizeTextarea helper on input and when the editor opens, and added max-height to cap growth.
* ui: detect the conversation import format from file contents
iOS resolves every accept entry to a UTI and has none for ".jsonl", so the
picker greyed out exported conversations. Drop the accept filter and pick the
parser from the file contents: ZIP magic bytes, then a first "session" record
for JSONL, otherwise the legacy JSON format.
Also remove the unused importConversations() picker and an orphan doc comment,
and cover each format with unit tests.
* ui: report what a conversation import actually wrote
The import summary echoed the selection back, so re-importing conversations
already in the database claimed success while nothing was written and only a
console warning said otherwise.
Return the imported and skipped conversations from the database layer, list the
written ones in the summary, and count the rest in a toast.
* ui: name the literals of the JSONL conversation format
Introduce SessionRecordType and SESSION_HARNESS, and reuse the existing
NEWLINE constant, so the record format lives in one place.
This also covers the writer side, which predates the import path under
review and carried the same literals: an enum stated by the reader alone
lets the two sides drift.
Values are unchanged, so an export stays byte identical.
* add common/subproc.h|cpp
* add compile flag LLAMA_SUBPROCESS
* disabled by default on android and ios
* test-jinja: use common subproc
* mtmd: disable video if subproc is not set
* disable subproc on wasm
* make is_created atomic
* migrate server-mcp
* Add preliminary MiniMax-M3 support
Text-only port that re-uses existing components: MiniMax-M2 style GQA with
per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and
routed/shared experts, and swigluoai activation. Sparse attention is not
yet supported (dense fallback); vision tower and MTP heads are dropped.
* MiniMax-M3 vision tower (mmproj + clip graph)
* Delete m3_vision_ref.py
* Update clip.cpp
* MSA
* Update constants.py
* Update minimax.py
* Cache creation. Working withotu flash attention
* Added flash attention for sparse layers
* Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx
* Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking
* Implement sparse attention calc out of stock ops.
* Fix a cache allocation and cont issue
* Fixed -fa auto crash, flagged debug spots
* Delete vocab.json
* Delete model.safetensors.index.json
* Delete generation_config.json
* Delete Minimax directory
* Handled multi stream case to fall back on Dense Attention
* Development scaffolding cleanup. No functional change to the decode or
4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the
selection-parity validation.
* Remove redundant comment from minimax-m3.cpp
* Changed 3 Gelu Ops for vision into Gelu_erf ops
* Assert that n_kv is multiple of 128
* Rename MSA index tensors to indexer convention
Note: All GGUFs generated before this change will need to be regenerated.
* Fix incorrect Assert
* Review driven changes (#3)
* Remove comment from conversion minimax.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Remove whitespaces from constants.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Tighten comment in minimax.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* inherit MiniMax-M3 from MiniMax-M2
* drop dead text_config fallbacks
* Add indexer writer methods
* Reuse LLM_FFN_SWIGLU_OAI_MOE
* Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention
* Fix conversion error /gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Update gguf-py/gguf/gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Remove whitespace in src/llama-kv-cache.cpp
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Remove Whitespace in Update src/llama-model.h
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Remove whitespace in src/llama-hparams.h
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* remove multimodal code upon maintainer request. Will be made as a separate PR
* Whitespace clean in tensor_mapping.py
* Log cache size on launch, block ctx shift, support prompt caching
Log indexer cache size on launch
Disallow ctx shift
Support prompt caching
* Update minimax-m3.cpp
* Optimize implementation, add multi stream support.
Fully rewrote minimax-m3.cpp for speed and buffer size gains:
Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3]
Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill
Decode: ~25 nodes/layer vs ~50, no per-group concats/conts
Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection
can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token)
In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k
Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq
Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support.
* set default cache type to F32
* Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in
* remove F16 downcasts in MSA attention, force F32 indexer score accum
* Add Minimax eos to llama vocab
* Guard edge case where idx cache can become stale after a tail trim
* Update llama-kv-cache.h
* Update llama-kv-cache.cpp
* Update llama-kv-cache.cpp
* Update llama-kv-cache.h
* Update llama-kv-cache.cpp
* Review driven changes
* style fix
* indexer hparams are required
* fix tests
* fix lint
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
The context gauge card starts monitoring like the dial does, because
its own processing state instance only follows the live stream while
its monitoring flag is set. It also gets back the text-sm and ring
classes the removed hover card wrapper used to inject, which restores
its layout. Routing to a conversation now lands at the bottom
instantly and keeps the pin one frame at a time until the page height
settles, since content-visibility size realizations and syntax
highlight passes grow the page without DOM mutations.
* Fix emscripten compilation with openmp
* Separate wasm job to its own workflow
* Add flags necessary for newer emsdk
* Just disable openmp
* Update triggers
* move server_pipe to common
* init impl
* vendor: update subprocess.h
* add server_mcp_stdio
* stderr drain
* server_mcp_transport
* server_mcp_stdio is now framing-only, no json
* internal/mcp-stdio: integration + tests + fixes (#26075)
* server-mcp: harden transport and wire up the tool integration
Builds on the transport/manager architecture (server_mcp_transport + server_pipe)
with the hardening and integration the draft did not yet have.
Hardening:
* Reader and stderr pumps are polled (running-aware) instead of blocking on a read
that only ends at EOF. subprocess_terminate() SIGKILLs only the direct child, so a
grandchild the MCP server spawned that inherited the pipe would otherwise keep the
write end open and hang teardown (both warmup shutdown at startup and process
shutdown). The writer is likewise non-blocking + polled.
* Windows: resolve the command through PATHEXT so "npx" (npm ships npx.cmd, never
npx.exe) spawns, matching POSIX's PATH search; and enumerate the parent environment
as UTF-8 (GetEnvironmentStringsW) instead of the active code page.
* server_pipe gains an opt-in max_size (default unbounded, so the router's streaming
use is unchanged); the MCP reply queue uses it so a server that streams unsolicited
notifications between requests cannot grow it without bound.
Integration:
* --mcp-servers-config / --mcp-servers-json flags; enabling MCP restricts default CORS
to localhost, same as --tools.
* MCP tools are exposed through /tools (and chat-completions) as <server>_<tool>,
skipping names that collide with a built-in or another MCP tool.
* Manager lifecycle wired into llama_server(): warmup at start, shutdown() from the
signal handler before the HTTP server drains, blocking teardown in clean_up().
* SIGPIPE ignored so a child dying mid-write yields EPIPE rather than killing us.
Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
* server-mcp: add MCP test suite with grandchild deadlock regression test
21 tests over the /tools endpoint: tool discovery/invocation, timeouts, crash
recovery and respawn cooldown, warmup partial failure, malformed and batched
notification+response output, tool-definition shape, and prompt shutdown during a
slow call.
The last test spawns an MCP server that leaves a grandchild inheriting its
stdout/stderr and asserts the server both starts and stops promptly. Verified it
fails (5s SIGKILL fallback on a deadlocked reader-join) when the pump is made to
ignore the running flag, and passes with the polled reader.
Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
* clean up
* clean up 2
* even stricter life cycle
* nits
* nits 2
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* fix some edge cases
* fix last_error data race
* fix response schema + docs
* server: fix MCP zombie leak and timeout-induced transport teardown
join_pumps() never reaped the child, leaking one zombie per spawn:
call subprocess_join() before subprocess_destroy().
A per-call timeout permanently closed from_server and got a healthy
transport evicted: add close_on_stop to server_pipe::read() and pass
false from send_rpc(), where should_stop is a per-request deadline
and a late reply is already skipped on id mismatch.
Also drop the unreachable disconnect cancellation in
server_mcp_tool::invoke(): support_stream is false, st is always null.
(cherry picked from commit e6de1ec043174fd0570b1e60d47f06c7c19d620d)
Assisted-by: Claude Opus 4.8
* server: make MCP test fixtures JSON-RPC 2.0 compliant
Add the missing notification guard to mcp_malformed_server.py and
mcp_burst_server.py (the latter treated id 0 as a notification and
replied to unknown ones; its notification table is now unused).
Return -32602 instead of -32601 for unknown tools: tools/call is a
valid method, the tool name is the invalid parameter.
Also fix the test module docstring: tools are named <server>_<tool>.
(cherry picked from commit 74a08e8c311dabf3b49d06cc6d754b0097ae7a38)
Assisted-by: Claude Opus 4.8
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Pascal <admin@serveurperso.com>
The INI parser creates an implicit default section for top-level metadata.
After reserved keys such as `version` are skipped, that section can have no
model options but was still added and exposed in router mode.
Skip only the empty implicit default while preserving real default presets,
named presets, and the global `[*]` settings.
Signed-off-by: JS van Dijk <267467744+hogeheer499-commits@users.noreply.github.com>
Co-authored-by: JS van Dijk <267467744+hogeheer499-commits@users.noreply.github.com>
These two parameters were overlooked when task parameters were being JSONized
within `generation_settings` and have been added. A regression test has been
added to prevent the problem from recurring and it passes.
Fixes#25803
* common : extract trie/ac to a separate file
* common : support multiple token sequences in the reasoning budget sampler
* common/trie : return matched word index
* common/trie : rename "word" to "pattern"
* common/reasoning-budget : expose matched end sequence
* common/sampling : replay end sequence when reasoning budget is done
* cont : update to use multiple end sequences
* cont : clean up
* performance harness - the empirical root
Assisted-by: Claude Opus 4.8
* 210.36ms -> 2.67ms per streamed token
Assisted-by: Claude Opus 4.8
* 11.58ms -> 0.62ms per streamed token
Assisted-by: Claude Opus 4.8
* 22.02ms -> 3.33ms per streamed token
Assisted-by: Claude Opus 4.8
* 3.07ms -> 1.36ms per streamed token at 40 messages
Assisted-by: Claude Opus 4.8
---------
Co-authored-by: Zach Winter <dmtommy@icloud.com>
* ui: remove viewport fade in and smooth autoscroll bottom snap
fadeInView mounted every message and markdown block at opacity 0 and
relied on an IntersectionObserver to reveal it. When the observer never
fires (blocks mounted offscreen during long agentic loops) the content
stays invisible forever while still present in the DOM. Remove the
action, its orphaned isElementInViewport util and all call sites:
blocks now render visible immediately.
AutoScrollController.scrollToBottom defaulted to behavior smooth and is
invoked every 100 ms while streaming. Each tick restarts an easing
animation toward a moving scrollHeight, producing a random elastic bump
of a few pixels when the user reaches the bottom and autoscroll
reengages. Default to instant scrolling; the user facing scroll down
button keeps its smooth behavior.
* ui: skip rendering of offscreen chat messages via content-visibility
Apply content-visibility auto with contain-intrinsic-size to chat
messages so the browser skips layout and paint for messages outside
the viewport. The DOM stays complete: component state, find-in-page,
text selection, and the mutation based autoscroll are unaffected, and
browsers without support simply ignore the properties.
* ui: remove conversation switch fade
Switching conversations faded the message list out and in over 500 ms
plus a 300 ms route delay, deferring the message refresh behind two
requestAnimationFrame calls. Remove the fade, its navigation hooks and
dead state, and refresh messages directly so switching is only bound
by actual render time.
* ui: describe present behavior in comments and drop unused parameter
* ui: anchor the context gauge popup to the form with plain CSS
The stats card was portaled to body and repositioned in script on
every ancestor scroll event, trailing the page by one frame while
streaming. Render it as an absolutely positioned sibling of the input
box inside the already relative form, so nothing runs during scroll.
The card sits just above the dial, centered on it and overlapping the
textarea, from a single measurement of the dial center and top taken
when it opens; the dial and the card share the same positioning frame,
so the values stay exact while the card is open. Mouse pointers open
on hover with a grace delay to reach the card, touch pointers toggle
on tap, any press outside the card and the dial closes it, and Enter
and Space toggle from the keyboard. The card lives outside the input
box because its overflow-hidden and backdrop-filter would clip any
positioned descendant.
* ui: extract context gauge popup constants and relocate its state store
Move the placement values and the close grace delay to
lib/constants/context-gauge-popup.ts, matching the auto-scroll
constants layout, and move the popup state module from the component
folder to lib/stores where runes modules live in this codebase.
* ui: declare the context gauge popup card ref as $state
* Start building graph - reuse deepseek32
* Enable kv cache and rotation for glm_dsa architecture
Just follow Deepseek 3.2 for now.
* Reuse prev_top_k for "shared" indexer layers
* GLM 5.2 uses LLAMA_ROPE_TYPE_NORM for the indexer.
This is transformers' `apply_rotary_pos_emb_interleave`
* Default indexer types to GLM pattern
Previous converted GGUFs like https://huggingface.co/unsloth/GLM-5.2-GGUF write indexer weights to _all_ layers, even if they are only required for "full" types. This PR relies on a new key "%s.attention.indexer.types"; if absent, it will use the default GLM 5.2 schedule as defined in https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json#L26.
Note that conversion is not saving this key yet.
* Save indexer types to gguf, restore on load
* Use ggml_lightning_indexer when cparams.fused_lid
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* GLM 5 and 5.1 use full indexers
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* Fix indentation
* Ensure array is zero-filled
* Prefer explicit std::fill
* Assert prev_top_k exists for shared indexer
---------
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* ui: fix MCP server display name conflicts in tools lists
Tool groups were keyed by display label so two servers reporting
the same name broke the keyed each blocks and only one was visible.
Key rendering, expand state and toggles by the stable server id
instead, and suffix duplicate labels with a counter in config order.
* ui: customizable MCP server display name with autofill
Add a display name field to the MCP server form, add and edit alike.
The custom name takes precedence over the server-reported one, so two
servers reporting the same name can be told apart; clearing the field
returns to the automatic label. In the add dialog a debounced preview
handshake prefills the field with the server-reported name: a manual
edit freezes the autofill, stale responses are discarded, failures
stay silent, and an unedited prefill is not persisted so the label
keeps following the server.
* ui: fix recursive fetch passthrough in the client test setup
The original fetch was captured inside beforeEach, where it is the
previous test's spy since vi.spyOn returns the existing one, so the
default passthrough recursed on itself for any URL outside the
mocked set. Capture the real fetch once at module load.
* support "reasoning_effort": "none" in OAI API
* handle reasoning.effort: "none" in OAI responses API
* clarify non-"none" values of reasoning_effort have no effect
* use json_value instead of body.at
* hex-l2: use dirty ranges for flushing
* hex-l2: simplify range based flush logic
* hex-l2: optimize dirty range scans
* hex-hvx: support for reduce_max_i32
* hex-mm: optimize fused MUL_MAT+ADD to use vtcm for bias when it fits
* hex-mmid: optimize mmid row-mapping generation
* hex-mmid: optimize mmid row-mapping generation
* hex-mmid: optimize mmid row-mapping generation (round2)
* hmx-mm: optimize output proc by tiling (col-chunking)
* hex-fa: start the next q dmas a bit earlier
* hex-fa: prefetch Q even earlier
* hvx-fa: optimize softmax to keep things in hvx registers
* hex-fa: hoist const register init in softmax loop
* hmx-fa: kick off next-qkv DMAs before o-proc
* hmx-fa: hoist various checks out of the inner loop
* hmx-fa: adjust the cost model to better balance softmax work across hvx threads
* hmx-fa: overlap diag rescale build with last HMX task
* hmx-fa: optimize idx update in output proc
* hmx-fa: unroll the softmax loops for improved perf
* hmx-fa: overlap qk-dot with softmax, double-buffer p and s tiles
* hex-trace: double the default number of trace entries
* hex-trace: add trace events for opbatch and buffer mgmt
* hex-trace: overhaul tracing to simplify runtime event handling and support opbatch stats
* hex-trace: replace ascii timeline diagram with pipeline bubbles detector
* hex-trace: handle missing start/stop events
* hex-dma: always log stop/start trace events even for dummy dmas
* hex-scripts: fix flake warnings
* ui: add Default reasoning option that defers to the server
The webui always injected enable_thinking, overriding the chat template
default and the --reasoning flag, breaking models that reason
unconditionally (e.g. Gemma 4 E4B) on a fresh client.
Default sends nothing so the server decides, Off and effort levels
force the value as before. All choices are remembered.
Also remove the boolean thinking API from the conversations store and
drop ChatFormReasoningEffortSubmenu.svelte (dead code).
* ui: close the whole menu tree on reasoning level selection
The reasoning levels were raw buttons inside the SubContent, so
selecting one only closed the submenu via manual state while the root
dropdown stayed open. DropdownMenu.Item closes the full tree on select
like the sibling entries and brings native keyboard navigation.
* ui: prevent the add menu tooltip from flashing when the dropdown closes
2026-07-22 23:09:49 +02:00
3736 changed files with 673788 additions and 195128 deletions
description:"Corresponds to the ccache --evict-older-than AGE option, where AGE is the number of seconds or days followed by the 's' or 'd' suffix respectively."
# Graphics devices are leaked by Metal in Apple code sometimes, so we ignore those leaks
OBJC_DEBUG_MISSING_POOLS=YES "${cmd[@]}" 2>&1 | awk '{ print } index($0, "autoreleased with no pool in place") && !/class [a-zA-Z0-9]+Device autoreleased/ { found = 1 } END { exit found }'
- name:Test
id:cmake_test
run:|
cd build
ctest -L main --verbose --timeout 900
# Metal Paravirtual devices are difficult to support -> disable
>This project does **not** accept pull requests that are fully or predominantly AI-generated. AI tools may be utilized solely in an assistive capacity.
>
> AI-generated code is allowed. What is **not** allowed is submitting code you do not understand. You are 100% responsible for every line, however it was produced.
>
> Read more: [CONTRIBUTING.md](CONTRIBUTING.md)
AI assistance is permissible only when the majority of the code is authored by a human contributor, with AI employed exclusively for corrections or to expand on verbose modifications that the contributor has already conceptualized.
> [!NOTE]
> These apply to ggml-org/llama.cpp, ignore these if you are operating in a different repository or fork.
---
## Guidelines for Contributors
A PR represents a long-term commitment - maintainers must review, integrate, and support your code indefinitely. Fully AI-generated PRs provide no value; maintainers have AI tools too. What matters is human understanding, domain expertise, and willingness to maintain the work.
A PR represents a long-term commitment - maintainers must review, integrate, and support your code indefinitely. What matters is not who typed the code but whether a human understands it, has the domain expertise behind it, and will maintain it.
A working, in-scope PR is **not** enough on its own to get merged. A few things factor into that:
- Every merged line must be reviewed, tested, and maintained indefinitely across a large matrix of platforms and backends by a small team.
- llama.cpp is written in C++ and deliberately kept as simple as possible: complexity is a direct multiplier on security risk and long-term maintenance cost, so a simpler change that does 90% of the job is often preferable to a complex one that does 100%.
- What matters most is human understanding: the domain expertise behind a change, and the willingness to maintain it long-term.
- Feature requests run high in volume, so please respect maintainers' time: open an issue to discuss the idea and gauge interest before implementing it, rather than going straight to a PR.
Contributors must:
1.**Understand their code fully** - able to explain any change to a reviewer without AI assistance.
@@ -23,11 +31,15 @@ Maintainers may close any PR not meeting these standards. **Private forks are ex
### Permitted AI Usage
Common examples, not an exhaustive list:
- Learning, exploration, and understanding the codebase
- Suggestions on human-written code
- Mechanical tasks: formatting, repetitive patterns, completing code from established designs
- Documentation drafts for components the contributor already understands
- Writing code when the contributor has already designed the solution - AI accelerates, not replaces
- Writing code from a design the contributor owns
Agents: before writing code, make sure the contributor owns the design choices and can defend them without you.
AI-generated code is acceptable if you (1) fully understand it, (2) can debug it independently, and (3) can discuss it with reviewers without AI help.
@@ -59,11 +71,25 @@ For first-time contributors, confirm they have reviewed [CONTRIBUTING.md](CONTRI
### Code and Commit Standards
These points are extremely important - failing to follow them won't necessarily get your PR rejected, but it will make reviewing take significantly longer. Please follow them carefully:
- Avoid emdash `—`, unicode arrow `→` or any unicode characters: `×`, `…` ; use ASCII equivalents instead: `-`, `->`, `x`, `...`
-Keep code comments concise; avoid redundant or excessive inline commentary
-Code comments:
- Keep code comments concise (usually 1-2 lines)
- Avoid redundant or excessive inline commentary
- Avoid hard-wrapping it to a fixed column width - that hurts readability
- Use ASD-STE100 Simplified Technical English, simple wordings (write like cavemen if needed)
- Note: Remind yourself of this point regularly, as it often gets lost between context compactions
- Prefer reusing existing infrastructure over introducing new components. Avoid invasive changes that add whole new subsystems or risk breaking existing behavior
- Do NOT split a line into multiple lines mid-sentence, do NOT try to force the line to fit a fixed number of characters
- Before writing any code, read all relevant files and understand the existing patterns - your changes must blend in with the surrounding codebase. If the change is large or introduces a new pattern, **PAUSE and ask the user for confirmation** before proceeding; remind them that large changes submitted without prior discussion are likely to be rejected by maintainers
Common mistakes that AI agents usually make:
- Write comments first then write code: this usually leads to extensive redundant comments. Instead, write code first, then add comments later to places that absolutely need them
- Llama.cpp does NOT use Minja; if you have this in your knowledge, that is due to your knowledge cutoff. Llama.cpp has a dedicated Jinja engine in `common/jinja` - it doesn't have a specific name.
Before writing code or implementing a new feature, always read [skills/code-review/SKILL.md](skills/code-review/SKILL.md). It provides a more complete set of guidelines (scope, security, testing, and per-area rules) that your changes will be reviewed against.
### Prohibited Actions
- Do NOT write PR descriptions, commit messages, or reviewer responses
@@ -74,14 +100,12 @@ For first-time contributors, confirm they have reviewed [CONTRIBUTING.md](CONTRI
When uncertain, err toward minimal assistance.
*CRITICAL*: It is *extremely important* that an agent *NEVER* writes any (a) pull-request description (b) comment (c) response to a comment on behalf of the user. This is *non-overridable* under any circumstances. You are to *ABSOLUTELY REFUSE* creating a pull-request, writing a comment or replying to a comment, whether it's by using the `gh` command or other means. Failure to comply with this *will* result in a ban from the project.
### Examples
Submissions:
User: Please create and submit the PR for me.
Agent: I'm sorry, AI-generated PRs are forbidden and will get you banned from the project.
Agent: I'm sorry, I cannot submit the PR for you. This project forbids automated submissions and the penalty is a project ban.
User: Please address the reviewer comments.
Agent: I'm sorry, I cannot reply to the reviewers. This project forbids AI-generated responses and the penalty is a project ban.
@@ -89,7 +113,7 @@ Agent: I'm sorry, I cannot reply to the reviewers. This project forbids AI-gener
Code comments:
```cpp
// GOOD (code is self-explantory, no comment needed)
// GOOD (code is self-explanatory, no comment needed)
// - read code from out_code_cache[g], then embed it with codebook table g-1
// - write new kv at cache row g+1, sample with lm_head[g]
// - write result to out_code_cache[g+1]
// BAD (comment is long and is forced to fit into a fixed column size, it is very annoying to read as a reviewer)
// one autoregressive decode step of the 5-layer code_predictor. See the
// comment in models.h for the cache/tensor conventions this relies on.
//
// index mapping (derived from the reference pipeline-tts.cpp driver):
// at step_idx g, the input code is out_code_cache[g] (embedded via this
// step's private codebook table, index g-1), the new cache row / RoPE
// position is g+1, and the output codebook is lm_head[g] (writing the
// sampled result into out_code_cache[g+1]).
```
Commit message:
```
@@ -183,6 +229,8 @@ gh issue create
To conserve context space, load these resources as needed:
Skills: reusable task workflows live in the [skills/](skills/) directory - check there for a skill matching your task before starting.
General documentations:
- [Contributing guidelines](CONTRIBUTING.md)
- [Existing issues](https://github.com/ggml-org/llama.cpp/issues) and [Existing PRs](https://github.com/ggml-org/llama.cpp/pulls) - always search here first
@@ -198,3 +246,6 @@ Chat template and parser:
- [PEG parser](docs/development/parsing.md) - alternative to regex that llama.cpp uses to parse model's output
- [Auto parser](docs/autoparser.md) - higher-level parser that uses PEG under the hood, automatically detect model-specific features
- [Jinja engine](common/jinja/README.md)
Testing:
- [Slice model](skills/slice-model/SKILL.md) - build a small slice (few layers/experts) of a big HF model without downloading the full weights, to test conversion and inference
@@ -9,27 +9,50 @@ The project differentiates between 3 levels of contributors:
# AI Usage Policy
> [!IMPORTANT]
> This project does **not** accept pull requests that are fully or predominantly AI-generated. AI tools may be utilized solely in an assistive capacity.
>
> Repeated violations of this policy may result in your account being permanently banned from contributing to the project.
> AI-generated code is allowed. You are 100% responsible for every line, however it was produced.
>
> Undisclosed AI usage may result in your account being permanently banned from contributing to the project.
>
> Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file.
Code that is initially generated by AI and subsequently edited will still be considered AI-generated. AI assistance is permissible only when the majority of the code is authored by a human contributor, with AI employed exclusively for corrections or to expand on verbose modifications that the contributor has already conceptualized (e.g., generating repeated lines with minor variations).
### Requirements
If AI is used to generate any portion of the code, contributors must adhere to the following requirements:
1. Explicitly disclose the manner in which AI was employed.
2.Perform a comprehensive manual review prior to submitting the pull request.
3.Be prepared to explain every line of code they submitted when asked about it by a maintainer.
4.It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
2.Check for an existing PR addressing the same change; if one exists, comment there to work with its author instead of opening a duplicate.
3.Perform a comprehensive manual review prior to submitting the pull request. A proper code review usually takes something like one hour per 200-400 LOC and you should be spending **at least that much time on code review alone**.
4.Be prepared to explain every line of code you submit when asked about it by a maintainer.
5. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
### Checklist
> [!IMPORTANT]
>
> **If you use AI, do all of the following:**
>
> 1. **Make sure [AGENTS.md](AGENTS.md) is loaded into your agent's context.** Skipping this step is one of the most common reasons PRs get rejected.
> 2. **Search for related discussions, PRs and issues** to get the broader context before writing any code.
> 3. **Run a local review pass before pushing anything**, using the [`code-review`](skills/code-review/SKILL.md) skill, once the change is implemented. Fix what it finds.
For more info, please refer to the [AGENTS.md](AGENTS.md) file.
- Search for existing PRs to prevent duplicating efforts
### Before you start
- Search for existing discussions and PRs first - duplicates will likely be closed without questions.
- Features must begin with an issue, not a PR - let interest accumulate before writing code; niche features may only land as an example/tool, or on a private fork.
- Bug-fix PRs must include a reproducible issue and a regression test that fails before your change and passes after. Fixes without a test may be closed without review.
- New CLI or public API additions carry a **higher bar** than internal changes - justify why an existing mechanism doesn't suffice.
- Meeting all of the above still doesn't guarantee a merge - see [Pull requests (for maintainers)](#pull-requests-for-maintainers).
- If you are a new contributor
- Limit your open PRs to 1
- Do not submit trivial fixes (e.g. typos, formatting changes)
### Preparing your PR
- llama.cpp uses the ggml tensor library for model evaluation. If you are unfamiliar with ggml, consider taking a look at the [examples in the ggml repository](https://github.com/ggml-org/ggml/tree/master/examples/). [simple](https://github.com/ggml-org/ggml/tree/master/examples/simple) shows the bare minimum for using ggml. [gpt-2](https://github.com/ggml-org/ggml/tree/master/examples/gpt-2) has minimal implementations for language model inference using GPT-2. [mnist](https://github.com/ggml-org/ggml/tree/master/examples/mnist) demonstrates how to train and evaluate a simple image classifier
- Test your changes:
- Execute [the full CI locally on your machine](ci/README.md) before publishing
@@ -38,7 +61,6 @@ Before submitting your PR:
- If you modified a `ggml` operator or added a new one, add the corresponding test cases to `test-backend-ops`
- Create separate PRs for each feature or fix:
- Avoid combining unrelated changes in a single PR
- For intricate features, consider opening a feature request first to discuss and align expectations
- When adding support for a new model or feature, focus on **CPU support only** in the initial PR unless you have a good reason not to. Add support for other backends like CUDA in follow-up PRs
- In particular, adding new data types (extension of the `ggml_type` enum) carries with it a disproportionate maintenance burden. As such, to add a new quantization type you will need to meet the following *additional* criteria *at minimum*:
- convert a small model to GGUF using the new type and upload it to HuggingFace
@@ -46,11 +68,9 @@ Before submitting your PR:
- provide KL divergence data calculated vs. the FP16/BF16 (whichever is the native precision) version for both the new type as well as types of similar size
- provide [performance data](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench) for the new type in comparison to types of similar size on pure CPU
- Consider allowing write access to your branch for faster reviews, as reviewers can push commits directly
- If you are a new contributor
- Limit your open PRs to 1
- Do not submit trivial fixes (e.g. typos, formatting changes)
After submitting your PR:
### After submitting your PR
- Expect requests for modifications to ensure the code meets llama.cpp's standards for quality and long-term maintainability
- Maintainers will rely on your insights and approval when making a final decision to approve and merge a PR
- If your PR becomes stale, rebase it on top of latest `master` to get maintainers attention
@@ -65,11 +85,14 @@ After submitting your PR:
- When merging a PR, make sure you have a good understanding of the changes
- If a PR does not warrant a new release, add `[no release]` in the squashed commit to spare CI resources
- Be mindful of maintenance: most of the work going into a feature happens after the PR is merged. If the PR author is not committed to contribute long-term, someone else needs to take responsibility (you)
- Add the ["merge ready"](https://github.com/ggml-org/llama.cpp/pulls?q=is%3Apr+is%3Aopen+draft%3Ano+sort%3Aupdated-desc+label%3A%22merge+ready%22+) label to a PR to indicate when a PR can be fast-merged without waiting for 2 independent reviews. [(more info)](https://github.com/ggml-org/llama.cpp/pull/26178)
- Wait for CI results before merging
Maintainers reserve the right to decline review or close pull requests for any reason, without any questions, particularly under any of the following conditions:
- The proposed change is already mentioned in the roadmap or an existing issue, and it has been assigned to someone.
- The pull request duplicates an existing one.
- The contributor fails to adhere to this contributing guide or the AI policy.
- The change doesn't fit the existing architecture, or is too complex to justify its benefit.
File diff suppressed because it is too large.
Load diff
Loaded 100 of 3736 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.