Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last
user message (and of earlier ones on a fresh prompt) so that a context
checkpoint lands there. On a hybrid/recurrent model with MoE experts on the
host, that turns a typical chat turn (previous answer + new user message, under
one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the
host experts again.
When the rest of the prompt from the current batch start fits in one ubatch,
skip the user-start break. The near-end rule still checkpoints the batch start
and the end-4 rule the prompt end, so a later prompt that diverges inside the
last user message re-processes less than one ubatch more, in the same ubatch.
Prompts longer than one ubatch, and batches after a media chunk, keep the
upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
What and why (#32755 and the 6.5 GiB of shared expert copies), the knob,
F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the
pod validation plan (reference with the loop experts on the GPU, views on,
views off).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
inkling2_loop runs the core layers (6..17 on Inkling-2-Mini) three times, and
post_arch_tensors makes every cache slot share the tensors of its physical
layer. With the experts in host memory (--n-cpu-moe, -ot exps=CPU) the
scheduler makes one device copy per (host tensor, backend) per graph, at the
first split that reads it, and keeps it until the last reader. So:
- the expert copies of all 12 loop layers are alive at once (about 6.5 GiB on
UD-Q4_K_XL), which is most of the ~7 GiB Vulkan0 compute buffer;
- sched_copy_experts runs only for the first reader. When it reads the ids back
(ubatches of 32-63 tokens: MUL_MAT_ID is offloaded from 32 tokens, the F1
skip starts at 4*n_expert ids), it uploads only the experts that pass 0
selected, and passes 1 and 2 read stale VRAM for the others (upstream issue
#32755, "GGGG" output).
Add a post_load_tensors hook to llama_model (called after the weights are
loaded, and on the no_alloc path after the dummy buffers are set). In
inkling2_loop it replaces, for each slot of a later pass, every tensor in a
host buffer with a view of it (name "<name>.visit<pass>", buffer and data of
the original). The scheduler hashes tensors by pointer, so each pass gets its
own copy, freed after that pass, and the copy callback sees the ids of that
pass. Weights in device buffers and CPU extra buffers (CPU_REPACK, not host)
stay shared. The pointer-keyed weight maps (w_scales, prec_policy, hdmd.rot)
get entries for the views. The views live in a model-owned context outside
ctxs_bufs, so the F4 release/restore, tensor counts and the memory breakdown
do not see them; they point only at host buffers, which F4 never releases.
LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors (A/B knob).
Not built or run yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32740 at head 808fd1c4e,
its four commits (1fa93f902, d29f9d489, ff91e4f84, 808fd1c4e) squashed.
Inkling's per-query bias table is applied inside flash attention
instead of a dense [n_kv, n_tokens, n_head] bias tensor, which cuts
Inkling's compute buffers and long-context prefill time on the RX 580.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").
One conflict, ggml-vulkan.cpp use_mask_opt: kept the fork's GCN
`HSK >= 256 || HSV >= 256` clause and its comment (F2) and upstream's
`!use_sparse`, added the PR's `!bias &&`.
Original commit trailer: Assisted-by: Codex
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32735 (head 0ea4ffd00),
cherry-picked unchanged. The Vulkan backend runs SCALE on a
non-contiguous source instead of rejecting it (CPU fallback). Carried
together with #32740 and evaluated as a pair.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").
Original commit trailer: Assisted-By: Opus 5.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
HF_UI_VERSION must use the tag after the '|' in .ui-stamp; bundle path
for agents told not to push; build backup and copy-based rollback;
status, pointers and last-run record for 3bf6b55b3; q35.py build path
note; disk-cache restore test wording; open questions.
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
Upstream #30591 added its own tiled-transpose concat pipelines, selected
for any dim-0 concat whose src1 is transposed. Behind the fork's fast path
(which already takes ne0 >= 32 and honours GGML_VK_CONCAT_FASTPATH) that
selector still caught the decode case (ne0 < 32), bringing back the
decode regression the TILE_DIM gate fixed (FORK.md F3), and
GGML_VK_CONCAT_FASTPATH=0 no longer gave the generic kernel.
Apply the same gate and the knob to upstream's selector.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort
with the production mmproj), the fix, the new mmproj cases (15 PASS), and
that the earlier 35B numbers were measured with --no-mmproj. The 35B run with
production's exact command line (diskcache-window3.sh) is ready, not run, and
a precondition for deploying --cache-disk-path.
Runbook:
- tests must use production's exact model command line, mmproj included;
- the validated build has all three targets;
- the deploy mtime list includes libllama-server-impl.so and
libllama-common.so (llama-server is a stub);
- deploy and rollback steps run as one setsid-wrapped script on the pod,
never as a backgrounded cat && sh -n && setsid chain from kubectl exec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production runs the 35B and gemma with an mmproj, and the first deploy of the
disk prompt cache aborted on that path. Every case had used --no-mmproj. Now
--no-mmproj is in DC_MM, separate from DC_ARGS, and two cases run with the
0.8B's mmproj (--no-mmproj-offload, arbiter env, LLAMA_MMPROJ_ONDEMAND=1),
against their own reference:
- restore-mmproj: turn 1 is saved on SIGTERM and restored after the restart,
prompt_n equal to the reference, output identical;
- mmproj-image: an image request (no restore, answer "red"), then the text
chat (restored, identical), then an image again, then SIGTERM with media in
the slot (no state copied, exit 0, the file count unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deploy of cb48bf224 with --cache-disk-path was rolled back: the first
request to the production 35B aborted in disk_cache_load().
server_tokens::get_tokens() has GGML_ASSERT(!has_mtmd), and has_mtmd is true
whenever an mmproj is loaded, even for a text-only prompt. Production's 35B
and gemma-26B always load one (-hf pulls it). disk_cache_collect() and
server_prompt_disk_cache::save() made the same call, so the shutdown save
would have aborted too. Every test had used --no-mmproj, which hid it.
disk_cache_text() takes the list from get_text_tokens(). A prompt is
text-only when that list is as long as size(), because media positions hold
LLAMA_TOKEN_NULL. So a text-only prompt is saved and restored with an mmproj
loaded, which is the main case (OpenWebUI chats on the 35B and gemma), and a
prompt with media is never saved or restored. When the slot itself holds
media, its reuse estimate is server_tokens::get_common_prefix(). These four
calls were the only get_tokens()/mtmd-unsafe accesses in the feature.
fork-validate --diskcache with the new mmproj cases, beside production:
15 PASS, 0 FAIL, GPU legs included. With the 0.8B's mmproj loaded, a text
chat is restored: prompt_n 58 of 596, output identical to the reference.
An image request is answered "red" with no restore, the text chat after it
is restored, and a SIGTERM with media in the slot copies no state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.
Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.
Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F11 in the per-feature format: design, invariants, the measured costs (CPU
only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at
30 s). Also: the recommended config (not applied), the commit map entries, the
F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache,
the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run
yet). Runbook: --diskcache in phase 1, the pod window script, the deploy
notes (every deploy empties the cache, /root free space, a slower stop), a
post-deploy check, and the cache dir.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs beside production. The model is the small hybrid (Qwen3.5-0.8B), CPU only
(-dev none -ngl 0, no VRAM), with chat requests and greedy decoding. A
reference server that never restarts runs turn 1 and turn 2. The candidate
saves turn 1 on SIGTERM and restarts. Turn 2 must then be restored from disk
with the same prompt_n as the reference (only the suffix processed) and the
same answer. Cases:
- restore plain, with the arbiter env warm, and forced cold by a fake waiter
on want.lock (the save from a released state). Each case records the exit
time and SIGTERM-to-token-free, also once without the cache;
- no restore for another conversation, or with another n_ctx, KV type or
model path (other key);
- a truncated file is removed at startup and a byte-flipped one at load,
never used;
- the size limit evicts the oldest file;
- a write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored: EFBIG) is logged, the
exit stays 0 with no file left, and the next start serves;
- the same restore on the GPU (-ngl 99, arbiter env, warm and cold) when
>= 1.5 GiB of VRAM is free, else SKIP.
start_server gains SRV_WRAP, a command prefix that execs the server.
First run (2026-10-25, beside production): 11 PASS, 0 FAIL; the GPU leg
SKIPped (1219 MiB free). The first run found the CPU_REPACK release crash
(exit 139 in the cold case).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama-swap SIGTERMs a model when the owner switches models (only one large
model fits in RAM), and switching back re-prefills the whole conversation:
about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR,
each slot with a prompt is saved to DIR on shutdown. After the restart, a
request whose prompt starts with a saved prompt is restored from the file,
so only the suffix is processed. Off by default.
Save, in server_context::start_loop() after the loop returns:
- disk_cache_collect() copies the state of each slot with >= 256 text tokens
to host memory (llama_state_seq_get_data_ext, flags NONE) while the device
buffers still exist. A cold server reads its released state through
llama_context_restore_memory_host(), with no device use and no extra copy.
- vram_share_exit() then frees the VRAM and releases the GPU token as before,
so only the D2H copy delays the token release, never the disk IO.
- disk_cache_flush() writes the files. A slot is skipped if its KV does not
cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not
decoded) or if the same token list is already on disk.
File (server_prompt_disk_cache, server-task.cpp):
- A 48-byte header (magic, version, key hash, sizes, payload hash), then a
payload of 8-byte padded sections: tokens, checkpoint table, state, the
checkpoint states. The hash is FNV-1a over the 64-bit words of the payload.
- Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read
is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked
the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file.
- The index is built at startup from the headers. A bad magic, a size that
does not match the header, or an inconsistent table means the file is
removed. A hash mismatch at load also removes it, and nothing is restored.
Other keys and other versions are kept. A .tmp older than 1 h is removed.
- --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the
least recently used *.lpcd files of every model in DIR are removed until the
new file fits; a restore updates the file mtime. Entries the new prompt
extends are dropped after the write and do not count against the limit.
A write also needs its size + 1 GiB free on the file system. A write error
or a full disk is logged; the file is not saved and the exit stays clean.
Key: the build commit, the model path + size + mtime, LoRA and control
vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the
K/V types, the flash-attn type, and the RoPE/YaRN parameters.
Restore, in get_available_slot() after the RAM cache:
- Only while warm (holding the GPU token), with cache_prompt on and no media
tokens. The file must reuse at least 256 more tokens than the slot holds.
- Reusable means one of two things. Either the saved token list is an exact
prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint
at or below the common prefix can be used. A plain-KV model can be cut back
to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the
template rewrites the previous assistant turn (the reasoning is dropped), so
the saved list is almost never a prefix of the next turn. The checkpoints
are exact snapshots, and update_slots() rolls back to one exactly as it does
for the RAM cache. They are restored with id_task -1, so one that fails to
load means a re-prefill, not an abort.
- The slot's current prompt goes to the RAM cache first. A set_data failure
clears the slot and removes the file.
Not supported with speculative decoding (warning, disabled): the draft state
is not saved. Only the slots are saved, not the RAM cache entries.
Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention):
- 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms,
exit 314 ms (125 ms without the cache). After the restart: read 166 ms,
restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full
prefill. The answer is identical to a server that never restarted.
- 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS
+162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms,
turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill.
- The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node
overlay) stalled once: 5.8 s for 64.5 MiB.
llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a process that is still there after 30 s. So the exit
save adds to the model switch time: about 1 s per GB on the PVC.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After llama_context_release_device(ctx, true) the KV cache and the recurrent
state live only in host shadows, and their tensors have no buffer, so
llama_state_seq_get_data cannot read them. The disk prompt cache must save
a cold server's slots at exit, and a shutting-down server must not restore
to the device (it may not hold the GPU token, F7).
llama_context_restore_memory_host() makes the state readable without the
device. restore_device_buffers(to_host = true) wraps each shadow as a
CPU_Mapped buffer (ggml_backend_cpu_buffer_from_ptr) and points the tensors
into it: there is no copy and no allocation. The first version copied into a
CPU buffer instead, which needs the full KV allocation a second time, about
2.7 GB for the 35B at ctx 131072; the node has had an OOM-controller kill
before. The buffer base is rounded down to the 32-byte alignment that
cpu_buffer_from_ptr asserts (a large std::vector is only 16-byte aligned);
those leading bytes are never accessed. With to_host the shadows are kept,
because the buffers point into them.
It sets device_discarded, so restore_device() aborts afterwards. Exit only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A model with weights on the CPU device (-dev none / -ngl 0) segfaulted in
llama_model::release_device_weights() the first time the VRAM arbiter put
it to sleep: CPU_REPACK buffers are not host buffers, so load_tensors marked
them releasable, and capturing the shadow called their NULL get_tensor.
Reproduced with the production build 2fc9f9c35 (Qwen3.5-0.8B, -dev none
-ngl 0, LLAMA_SLEEP_VRAM_ONLY=1 LLAMA_SLEEP_EVICT_KV=1, force a yield with a
waiter on want.lock): exit 139 at the release, before any SIGTERM. The
faulting call (rip 0) returned into release_device_weights.
Only buffers on a non-CPU device are releasable now. These buffers are in RAM
already, so there is nothing to free. Production is not affected: every
arbiter model there has its weights on the GPU, and a releasable CPU_REPACK
buffer would have crashed it. Found by the --diskcache check, which runs
its arbiter cases CPU only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Feedback from the 2026-10-25 deploy, which used only these docs:
- status: production is 2fc9f9c35; F10 deployed and verified on a 35B
GPU load; --moe-exact and median --perf have run on the GPU; last
full fork-validate run (37 PASS) and its timings; the first-candidate
-arm VRAM spike seen twice is an open question.
- runbook: a no-rebase redeploy path; --arbiter beside production only
if 1536 MiB VRAM is free, else in the swapdown window; full --moecache
model path; page-cache eviction before and after phase 2; deploy
steps reordered so the memory check follows stopping llama-swap;
in-place rebuild and system_fingerprint check; mtime check through the
versioned symlinks; rollback pointer numbering and one format; disk
space; nothink alias and a PPM image helper for post-deploy requests;
power1_cap via sysfs; restart and OOM checks; harmless UI warning.
- fork-validate.sh header: note the --arbiter VRAM requirement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With -lm none the GGUF is read() into our own buffers, but the kernel keeps
the file in page cache afterwards, charged to the pod's cgroup: about 20 GB
after a 35B load. On 2026-10-24 the Talos OOM controller killed the pod right
after a load under exactly that pressure. Those pages are pure waste once the
data is copied.
llama_file::drop_cache() wraps posix_fadvise(POSIX_FADV_DONTNEED) (Linux only,
errors logged at debug). load_all_data() drops each copied tensor's range right
after reading it, so the cache never grows past about one tensor, and drops the
whole file at the final cleanup to catch the metadata and partial pages at
tensor edges. Both only when the loader holds no mapping of the files
(mappings.empty(): not mmap, not mmap+mlock, no lazy tensors), so nothing that
is mapped is ever advised away. Split GGUFs are covered (every file in files).
clip.cpp always copies the mmproj via ifstream, so it drops the whole file
after loading it.
Measured on the pod, gemma-4-E2B UD-Q4_K_XL (2.5 GiB) + mmproj BF16 (0.9 GiB),
CPU only, -lm none, cold cache, same binary with the drops disabled as the
control (4 rounds, rotated order):
- file pages resident after load (mincore): 2431-2444 / 940 MiB -> 0 / 0
- peak model-file residency during load: 2.5 GiB -> 1.2-1.5 GiB (largest tensor)
- pod cgroup file after load: +3.37 GiB -> +0; host MemFree after load
0.6 GB -> 2.5-2.8 GB (MemAvailable counts clean cache, so it cannot show this)
- load time: 5.87 s -> 6.82 s median (+0.95 s); a post-load-only drop is
6.56 s (+0.68 s). The cost is synchronous page freeing that reclaim would
otherwise do later; -lm dio (direct IO) costs the same (+0.9 s) and does not
cover metadata or the mmproj.
- greedy output byte-identical; -lm mmap keeps the model file fully resident
and mapped (2498 MiB, no refaults); -lzm on skips the drop (file mapped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 1 now covers the VRAM release/restore check and the handoff latency
A/B against --ref (which must be the production build); phase 2 adds
--moe-exact on the 35B and keeps its scratch dir. Explains median gating and
--perf-report, that a pod restart empties /tmp, that the manual checks are
now automated, and that gdb is not in the pod image (lost in the 2026-10-24
restart). States the commit trailer convention and what llama-swap does with
a child that exits vs one that wedges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README fork section still said "the two code changes below are auto-on"
(the fork has three prefill changes plus the VRAM sharing and server fixes)
and recommended --n-cpu-moe 28 on the 35B, which FORK.md section 4 records as
worse at depth and unable to hold a long context. Replace it with a short
accurate summary that points to FORK.md, the runbook and the scripts. The
Laguna time profile, the iSWA KV note and Laguna's --parallel 1 saving were
only in the README and move to FORK.md sections 4 and 5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- status: no hardcoded fork-commit count (the list includes docs and
scripts; count git log origin/master..HEAD); what is after the production
commit; the commit-trailer convention of this private fork vs upstream's
AGENTS.md rules
- commit map: the commits after the production commit
- F1: validated by the new --moe-exact check and the LLAMA_MOE_IDS_SKIP knob
- F3: every graph that builds concat(state, transpose(x), 0) (bailingmoe3,
clef, kimi-linear, kimi-k3, glm5-next, mamba-base, plamo2, lfm-base,
qwen4exp besides delta-net); only the delta-net path is measured
- F4/F6: the new v-vram-* and a-handoff-latency checks with first results
- F5: encode_mtmd_chunk was split out of process_mtmd_chunk, not renamed
- F6: the vram_cold ordering rule restated from the code (the measured
failure was under the doorbell design; with the inline probe should_sleep
cannot run inside the restore; keep it, and A/B handoff latency before any
change) and why vram_acquire_for_load sets the flag early
- F7: what llama-swap does when a child dies, verified with a scratch
llama-swap: "upstream exited unexpectedly", restart on the next request
- F9: new flags, phases, median gating, 2026-10-24 reference values, the
/tmp and gdb loss on the pod restart, new grep targets and gaps
- section 7: stale comments fixed, what is not yet run on the GPU
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F1 (the routing-ids readback skip in sched_copy_experts) had no automated
check. --moe-exact MODEL reads expert_count / expert_used_count from the GGUF
header, computes the skip threshold ceil(4 * n_expert / n_expert_used) tokens
per ubatch (128 on the 35B), and runs one greedy request with a prompt of
about 2x that in a single ubatch, every expert on the host, so the prefill
takes the skip and each decode step takes the readback. The candidate's 64
tokens must equal the reference's byte for byte; with the LLAMA_MOE_IDS_SKIP
knob in the candidate, also the candidate with the skip forced off (=0), which
proves the skip exact even when the reference carries it, and =log must show
skips, all at >= 4 * n_expert ids. Without the knob skip-engaged SKIPs with
the batch-size argument. It needs the GPU to itself (same guard as --perf,
now shared as gpu_exclusive); FV_TEST_SHARED_GPU=1 bypasses the guard for
script tests against stub binaries only.
Also unset inherited LLAMA_MOE* variables.
Not run on the GPU yet (needs a swapdown window). Dry-run on the pod
against stub servers and the real 35B GGUF header: threshold 128, all-PASS
when outputs match, FAIL on a differing reference, SKIP without the knob,
FAIL for a dense model and while llama-swap runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing checked that a release actually frees VRAM. --arbiter now takes a
baseline of mem_info_vram_used before any scratch server starts, then with A
warm and alone on the scratch arena makes it release through a fake waiter
on want.lock. v-vram-release requires the card-wide delta to drop by at
least 70% of what A added warm; v-vram-restore requires it back (and the
output equal to the reference) after the next request. Production's own
share is re-read at every point from the DRM fdinfo of every process the
script did not start (deduplicated by drm-client-id); if it moved by more
than 64 MiB the check SKIPs instead of blaming the candidate.
Run beside production on the pod (reb1024 vs itself): warm +546 MiB, cold
+4 MiB, restored +546 MiB, production steady at 5053 MiB; 16 PASS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--arbiter only reported the candidate's handoff latency, while the fork's
process rule (two production deploys lost) is to A/B the ordinary handoff,
not only the pathological cases. After the arbiter cases, a-handoff-latency
starts fresh A/B server pairs of each build on the scratch arena and runs
blocks ref, cand, cand, ref of 16 alternating handoffs each. It FAILs if the
candidate median exceeds the reference median by more than max(25%, 200 ms)
and reports both medians, maxima and per-block medians.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--perf averaged the two per-arm llama-bench means of each build, so one
catastrophic sample (CPU contention from the ~60 other pods on the node) moved
the result, against the fork's own rule of medians, never means. llama-bench
now writes -o jsonl (every repetition sample) next to the markdown table, and
the gates use the median of all samples of a build pooled over its two arms
(4 per prefill cell, 6 for decode); the table shows min-max and n.
--perf-report TSV re-aggregates a saved <scratch>/perf.tsv without a GPU.
Checked offline on the pod: the 2026-09 A/B samples in results.txt give
pp2048 reference 250.24 (old mean) vs 247.76 (new median), candidate 280.15
vs 276.93; a synthetic 2026-10-24-like set with one 180 t/s candidate sample
read -9.3% (FAIL) the old way and -0.2% (PASS) with medians.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The llama-bench sweep measured decode as tg32 at the default depth and the
README called it the continuity metric, while tg32 at n_ctx 32 is launch /
barrier bound on this card and is not a valid decode metric (it once read
-36% between builds at parity at tg256). lbsweep.sh now runs prefill
(-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as
two invocations and labels depth cells (tg256@d4096). summarize.py marks
old depth-less tg rows as invalid-metric history. The README examples and
the reference values are updated to the 2026-10-24 production numbers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LLAMA_MOE_IDS_SKIP=0 forces the ids readback in sched_copy_experts (the
upstream path) and =log reports each skip on stderr, so validation can show
that the skip engages at prefill and that output is byte-identical with and
without it in one build. Read once (function-local statics); unset changes
nothing. Not set in production.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments only, plus one log string:
- the arbiter block comment said the KV cache is never evicted; with
LLAMA_SLEEP_EVICT_KV it is (and the scheduler and scratch with it)
- vram_go_cold said vram_ensure_warm() rebuilds the encoder; only
encode_mtmd_chunk() restores it, and only for an encode
- vram_ensure_warm and the mtmd init block named process_mtmd_chunk; the
restore lives in encode_mtmd_chunk
- handle_sleeping_state said waking happens lazily before decode; it
restores eagerly on the wake transition, before update_slots()
- the vram_cold ordering note now says why it still holds with the inline
want-lock probe
- the sleep log line said "keeping KV cache" even when LLAMA_SLEEP_EVICT_KV
evicts it; it now names what actually happens (the prefix
"server entering sleeping state" that fork-validate.sh counts is kept)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FORK.md documents every fork feature (what and why with measured numbers,
where, how it works, invariants, past failures, rebase hazards, validation),
the dead ends measured on this hardware, the production configuration and
the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation,
deploy, rollback and GPU-recovery procedure plus an operations reference.
The README fork section links both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checks the op kernels, the concat fast path, the VRAM arbiter incl. every
shutdown path that has caused a production incident, and prefill/decode
against a reference build. Prints PASS/FAIL per check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GGML_VK_CONCAT_FASTPATH=0 forces the generic path and =log reports each
fast-path dispatch, so validation can prove the path fires at prefill and
not at decode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:
- SIGTERM with a request in flight: terminate() stopped only the task queue.
The HTTP handler kept waiting for results the loop would never send, the main
thread blocked joining it, and the token was released only in the destructor
after that join - i.e. never, until the client went away. Terminate the
result queue too (readers now get "server is shutting down": a 503, or an SSE
error event), and let wait_until_no_sleep give up once stopping.
- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
inside the wake callback, holding mutex_tasks, so terminate() waited for the
current holder to finish and then for a pointless restore of our weights.
Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
stays cold, skips the restore, and the loop exits before update_slots.
Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.
Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A SIGTERM that lands while the start_loop() thread is inside a sleeping
callback deadlocks the server. The callbacks run under
server_queue::mutex_tasks, and the kernel delivers the signal to that same
thread; the handler calls terminate(), which locks mutex_tasks again, and the
thread waits on itself forever.
With the VRAM arbiter that window is wide - a whole device release, or a
blocking wait for the GPU token - and llama-swap sends SIGTERM to evict
models, so it hit production: an evicted model wedged mid-release, never
released token.lock, and every other model queued behind it. SIGTERM could
not stop it either; only SIGKILL did.
Make the handler only write the signal number to a self-pipe, which is
async-signal-safe, and run shutdown_handler on a dedicated thread. terminate()
there simply waits until the loop thread drops mutex_tasks, after which the
loop exits normally. The thread is joined before the objects the handler uses
are destroyed. If pipe() fails the old direct call is kept.
Reproduced 6/6 off production (SIGTERM during release and during the token
wait); with the fix 8/8 clean exits, and handoffs, image requests and greedy
output are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream #28334 removed --mmap/--no-mmap/--mlock and llama-bench's -mmp, so the
harness flags no longer parse: the server would refuse to start and llama-bench
would silently produce no rows. Switch both harnesses to -lm none, the same
load mode, and note --moe-cache-mib as a server-only option worth measuring.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The MoE cache (--moe-cache-mib, #29887) keeps the most recently used host
experts in device banks. The on-demand VRAM path (release_device /
restore_device, used by the server VRAM arbiter and the on-demand mmproj path)
did not know about it, so its banks were never freed on handoff. Restoring them
by reallocation alone would also be wrong: the banks come back zeroed while the
slot maps still mark experts as cached, so the model would silently read zero
weights.
Free the banks on release and empty the cache instead of shadowing it: re-init
every LRU, re-point its slot-map pointers at the host slot maps and clear those
to -1. The banks are reallocated (which also re-inits their views) and cleared
on restore, before the scheduler is rebuilt, and the cache refills from the
host experts as before. Release runs in both modes, weights-only and evict_kv.
device_resident() now covers the cache too, for the early returns in
release/restore, the auto-restore in decode() and
llama_context_weights_resident().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
release_device(evict_kv) drops the scheduler, so a context evicted by the VRAM
arbiter and then destroyed reaches the destructor with a null sched. The upstream
"compute buffer size matches expectation" loop calls
ggml_backend_sched_get_buffer_size() unconditionally, whose GGML_ASSERT(sched)
then aborts: the server exits 134 instead of 0 on every shutdown taken while
cold, which under llama-swap makes an ordinary stop look like a crashed child and
leaves a ggml backtrace in the log each time. It also skipped the rest of the
destructor, so the cold teardown path had never actually run to completion.
Guard at the call site rather than relaxing the assert - the assert is right, and
every other caller reserves the scheduler first. Same shape as the null guards in
synchronize(), memory_breakdown() and the released-buffer iteration.
The loop is a diagnostic size comparison, already reported by sched_reserve() at
load, so skipping it when cold loses nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the
holder's warden turned the inotify event into a queue flag. A ring carried no
notion of "still wanted", so it was lost whenever the holder was not already warm
and listening - during its model load (the watch does not exist yet, and the
kernel does not queue events for a watch that is not there) or inside
restore_device - and any ring that survived past the point its sender had been
satisfied caused a spurious release. Lost rings wedged the waiter permanently,
because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder
had no other reason to release. The wait blocks under mutex_tasks, so the whole
server stopped answering while /health still returned 200.
Express the request as kernel state instead. A waiter holds <arena>/want.lock
shared while it waits and drops it once it owns the token; the sleep decision
probes that lock non-blocking and releases the GPU while anyone is waiting. The
probe needs its own fd - flock treats two open file descriptions of one file
independently, so probing on the waiter's fd would convert our own lock rather
than conflict with it. Nothing can be missed, nothing goes stale, and a waiter
that dies is cleaned up by the kernel.
Probe from should_sleep() on the loop thread rather than from a warden thread.
Routing it through a flag is what made the first attempts fail: start_loop()
holds mutex_tasks from should_sleep() through the callbacks to the wait, so a
warden's request_yield() blocks on that mutex and is admitted only after the flag
has been consumed, latching a release for the next wake. Reading live state where
the decision is made has no edge to latch, and drops the warden's poll latency.
Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but
the loop clears req_stop_sleeping on the way in, so a loop that slept again
before the waiter ran stranded it forever - re-ask on every wake. And a task
queued after the waiter saw us awake could not wake us by itself, so sleep now
also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the
request that woke us is not queued yet, and yielding at once only sends it round
again.
Measured on the RX 580 pod, two servers contending, 60 alternating handoffs:
0 stranded, median 0.309 s against the doorbell's 0.314 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a
loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of
40 layers) build_conv_state emits a transposed concat per recurrent layer, so
the fast path runs 30 times per decoded token.
At one token the transposed source is a single column, so there is no
uncoalesced stride left to fix, but the path still costs two dispatches and an
unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes
the command stream. Measured -17% tg256 on RX 580; the barrier, not the
half-empty transpose dispatch, is nearly all of it.
Require the transposed source to be at least one TILE_DIM wide. That source is
qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the
fast path, decode and small speculative drafts take the generic one.
pp8192 +11% retained, tg256 back to parity, generation byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.
When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.
Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.
Assisted-by: Claude
Add clip_release_device/clip_restore_device (and mtmd_release_device/
mtmd_restore_device wrappers over the vision + audio contexts) that free
the multimodal encoder's device weight buffer to a read-only host shadow
and rebuild it on demand, using the same shadow/free/reallocate pattern
as llama_model weights. No-op for a CPU-backed encoder. This lets the
server drop the ~hundreds-of-MiB vision encoder from VRAM when it is not
encoding an image.
Assisted-by: Claude
On device release with KV eviction, also call ggml_backend_free_scratch()
on each backend so a cold model drops its Vulkan compute preallocations
(reallocated lazily on the next compute via restore).
Also guard llama_context::memory_breakdown() against a freed scheduler:
evict_kv release resets sched to null, so a cold model that is then torn
down (e.g. terminated by the process manager) hit GGML_ASSERT(sched) in
ggml_backend_sched_get_buffer_type. Skip the compute-buffer accounting
when sched is null.
Assisted-by: Claude
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.
Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).
Assisted-by: Claude
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left
resident when a model's device buffers were released for on-demand VRAM
sharing - llama_memory_recurrent::release_device_buffers() was a no-op
default. Implement it (and restore_device_buffers) with the same
capture-host-shadow / free / reallocate pattern as llama_kv_cache, so
llama_memory_hybrid now evicts both its attention KV and its recurrent
state. The state is read-write, so its shadow is recaptured on every
release.
Assisted-by: Claude
Under LLAMA_SLEEP_EVICT_KV a cold model still held the scheduler's worst-case
compute buffer (hundreds of MiB, e.g. ~700 MiB for gemma-26B) plus the resident
experts. release_device(evict_kv) now also frees the sched (sched.reset() +
sched_need_reserve), and restore_device() rebuilds it via sched_reserve(), so a
fully-evicted model holds essentially no VRAM (gemma cold: 7147 -> 51 MiB).
Trades a heavier re-warm (weights+KV H2D + sched reserve) for the freed VRAM;
weights-only mode is unchanged (fast switch, keeps KV+compute).
Assisted-by: Claude
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.
Assisted-by: Claude
release_device_weights()/release_device_buffers() leave a null buffer in
ctxs_bufs while the device memory is released for on-demand VRAM sharing.
The memory-breakdown / total_size / clear paths iterated these and called
ggml_backend_buffer_get_size()/get_type()/clear() on the null buffer, tripping
GGML_ASSERT(buffer) and aborting on shutdown of a cold model. Skip null buffers
in llama_model::memory_breakdown and llama_kv_cache::{memory_breakdown,
total_size,clear}.
Assisted-by: Claude
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.
- add vram_arena_open() (idempotent flock/doorbell setup) and
vram_acquire_for_load(), called from load_model() before
common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
request without a re-warm (drop the init-time go_cold cold-start).
Assisted-by: Claude
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.
Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.
Assisted-by: Claude
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.
- llama_memory_i: add release_device_buffers()/restore_device_buffers()
(default no-op). Implemented in llama_kv_cache (D2H shadow of the live
ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
buffers when requested; restore_device() rebuilds them. Public API
llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
since the KV shadow adds a D2H/H2D copy of the live cache each cycle.
Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.
Assisted-by: Claude
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.
- llama-model: release_device_weights()/restore_device_weights() capture a
compact host shadow (stable iteration order, view-skipping) and free then
realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
(not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
enforces "resident iff holds token"; an inotify doorbell forces the holder
to release on contention. The warden thread only touches the task queue, so
releases run on the loop thread and never race a decode.
Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).
Assisted-by: Claude
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).
Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).
Assisted-by: opencode
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.
Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.
Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.
Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.
Assisted-by: Claude
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.
Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.
Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.
The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.
Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.
Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Used angelX to optimize SYCL/drama 3 coordination, results are as follows
Each output reads only inputs with i*s0 <= idx < i*s0 + K, in the same ascending order as before.
Arc Pro B70, oneAPI 2026.1, test-backend-ops perf, base 9b56ccf00 vs patch, median of 3 same-boot A/B/A rounds, A/A 1.000-1.001:
| input [ne0, ne1] | kernel [K, OC, IC] | s0 | base us | patch us | speedup |
|---|---|---|---|---|---|
| [16, 1536] | [16, 768, 1536] | 8 | 3910.57 | 1348.59 | 2.90x |
| [128, 768] | [16, 384, 768] | 8 | 37735.00 | 1665.26 | 22.66x |
| [1024, 384] | [8, 192, 384] | 4 | 270371.75 | 1471.74 | 183.71x |
| [4096, 192] | [4, 96, 192] | 2 | 534614.63 | 752.74 | 710.23x |
Shapes are the drama 3 codec decoder upsampling stack, added as perf cases. Chained 4-layer stack at 1/4/24 input frames: base 66.3/847.4/28638.5 ms, patch 2.3/6.7/30.7 ms. Kernel numbers, not end-to-end TTS latency.
Correctness: test-backend-ops -o CONV_TRANSPOSE_1D 116/116 on base and patch, and 116/116 rebased on 3f03b1772. Seeded output dump (7.46 MB) byte-identical to base.
Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857). With only those cases excluded: 20145/20173 on base and patch, and every patch-run failure also fails on base.
Co-authored-by: Titaniumtown <titaniumtown@proton.me>
* mtmd: support hibiki realtime duplex
* fix and clean up
* fix long session quality
* fix ci
* tests : add the moshi fusion baseline for Metal
The Metal fusion check found 5 RMS_NORM+MUL fusions in the new moshi
arch and no baseline for them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ui: open the context gauge panel on click as a popover
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the context gauge trigger the ghost button hover
Assisted-by: Claude (Opus 5.5 Medium)
The loop slots copied the physical layers at the end of load_arch_tensors,
before the generic pass attaches the optional .scale tensors, so the later
passes through the core ran NVFP4 experts without their per-expert scales.
Add an optional post_arch_tensors hook that runs after the scale pass and
share the slots there.
Fixes https://github.com/ggml-org/llama.cpp/issues/31680
The matmul_tiled() implementation in tinyBLAS_Q0_PPC requires nc to be a multiple of 8 to perform matrix multiplication correctly.
The previous alignment logic contained a special case for n == 4, which allowed n_aligned to be 4. Consequently, the tiled path could be selected with nc = 4, violating the kernel's tile-width requirement.
This caused incorrect matrix multiplication results, observed as garbled inference output when running Flint Q4_K_M and Gemma 3 27B Instruct.
The fused lightning indexer is always used now, so drop the cparams flags, the auto probe and the non-fused fallbacks.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* Conversion
* Separate model_type
* Add gate_rank to allow building from metadata while testing
* Base model
* Loop graph
* Support the new is_varloop attribute
* Adopt naming scheme from https://github.com/ggml-org/llama.cpp/pull/32588
* Rename to varying
* hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1)
* hip: write the RDNA1 MMVQ values explicitly instead of recursing into the generic table
* hip: list the RDNA1 MMVQ table before RDNA2 in the enum
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ci : run the OpenVINO GPU job only for OpenVINO changes
The self-hosted OpenVINO GPU job ran for most PRs and every master push
on a shared runner, so it queued for hours.
- ci-self-hosted-openvino.yml (GPU): runs only for changes to
ggml/src/ggml-openvino.
- build-openvino.yml: keeps its triggers (OpenVINO changes on PRs, every
source change pushed to master). The Linux job runs on a GitHub-hosted
runner and tests the OpenVINO CPU device only.
* address review comments
* ci : limit the OpenVINO PR CMake trigger to ggml
* ci : drop the OpenVINO Ubuntu job
* ci : trigger OpenVINO PR runs on tests/CMakeLists.txt
* convert: qwen3-asr: put "language ..." output into thinking block
* convert: qwen3-asr: read the chat template from models/templates
The converter embedded its own copy of Qwen3-ASR.jinja. Read the
bundled file instead, as the Kimi-K3 and Kokoro converters do, so
the GGUF and test-chat share a single template.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Llama Chat compiles @ggml-org/llama-voice from ui/lib/voice/src, but the
CMake stage and the Docker UI stages copied only ui/lib/ui, so their npm
build could not resolve the voice imports. Stage ui/lib/voice/src next to
Llama UI and track it as an input of the CMake build.
Assisted-by: Claude (Opus 5.5 Medium)
* mtmd: drama3 voice cloning
The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3:
- conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2
- fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj,
builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2)
- drama3_gen tower: teacher forcing when the caller gives the codes of the next frame
- drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the
reference frames go through the backbone as codec rows and through the tower with their codes
- mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state
Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX
* fix correctness
* clean up
* rm redundant code
* fish_s2: rm unused import
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui: add a voice engine for capture, peaks and transcription
Voice input needs a live waveform, a clip that can be sent and drawn,
and a transcript, whether the model streams text while the user speaks
or not. This adds the engine for it in src/lib/voice, with no imports
of app stores or components, so it can move into a package later.
- VoiceCapture: one microphone start gives live levels (RMS on a dB
curve, fixed history, newest last), PCM16 mono chunks at 16 kHz from
an inline AudioWorklet (ScriptProcessor fallback), and on stop a
16-bit mono WAV clip with its duration and peaks. Tracks, nodes and
the context are released on stop, cancel, a failed start and page hide.
- computePeaks / computePeaksFromAudio: normalised peaks of samples, a
buffer, or a stored base64 or Blob file.
- transcribeClip: POST /v1/audio/transcriptions, with the server's own
error message on failure.
- RealtimeTranscriber: the /v1/realtime?intent=transcription socket,
queueing audio until session.created, a final commit, and a typed
RealtimeUnsupportedError when the model cannot stream or the socket
is refused (a browser websocket cannot send the API key header).
- startVoiceSession and VoiceRecorder: try realtime, fall back to one
HTTP transcription of the clip, and expose levels and live text.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add voice waveform and transcript components to Llama UI
Voice input needs three visuals that the app can feed with its own
recording and transcription logic: the live waveform while recording,
a playable waveform of a recorded clip, and a clip with its transcript
that opens on hover, keyboard focus or a tap.
VoiceWaveformLive draws input levels as many bars as fit the width,
newest on the right, and holds still with reduced motion.
VoiceWaveform plays a clip through its own audio element, fills the
played part in the primary colour, seeks by press, drag and arrow keys
(a slider) and pauses any other clip that was playing.
VoiceTranscriptReveal keeps the transcript in the DOM for screen
readers while it is closed, and shows transcribing and error states.
The waveform math (resampling peaks, bar capacity, seek and time
formatting) is in src/lib/voice with unit tests. The components are
exported from the package root and from @ggml-org/llama-ui/voice, which
Llama Chat resolves to the sources like the other entry points.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add stories and tests for the voice components
Storybook gets UI/Blocks/Voice: the live waveform fed by a fake
microphone meter in a chat form row, the clip waveform in its sizes and
states, and the transcript reveal in each state and on touch. The demo
audio is a WAV synthesised in the story, so no binary file is added.
The play functions check seeking by keys and by press, the reveal on
hover, keyboard focus and a tap, and a clip without audio. Pressing play
needs a real click before a browser plays audio, so a client test covers
play and pause and one clip pausing another.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: store voice messages with their transcript and send it to models that cannot hear
A voice message recorded in the chat form is an audio attachment like an
uploaded file, plus what the app needs to draw and read it. The audio
attachment gets four optional fields: voice (recorded in the app),
transcript, peaks and durationMs. Stored rows, export and import carry
them as they are, so no migration is needed and older messages keep
working.
The transcript is for reading. A model with audio input still gets the
audio alone; a model without it (the user switched models later) gets the
transcript as a labelled text part in place of the audio. When the
model's modalities are not known yet, or there is no transcript, the
audio goes as before.
A transcript often comes back after the message was sent. The new
voiceTranscriptsStore follows each pending transcript by the clip's name,
hands it to a send that happens after it arrived, and otherwise writes it
into the stored attachment of the message it went out with, in the open
chat or, if the user left it, in the database.
Copying a message now includes its voice transcript.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add Voice settings and resolve what the microphone does per model
The user should not have to think about which kind of voice input is in
use, so the app decides it from the conversation's model and two
settings. resolveVoiceMode is a small pure function: with Voice input on
Automatic and a chat model that has audio input, a recording is a voice
message, transcribed by that same model so a router swaps nothing.
Otherwise it is dictation, with the chat model when it can hear, else,
in router mode, with the chosen transcription model. When no model can
transcribe, the microphone is unavailable and says why.
Settings get a Voice section of their own rather than more rows in
General, which is already long: these four belong together and one of
them lists the server's models.
- Voice input: Automatic (default) or Always dictate.
- Transcription model: Same as chat model (default) or, in router mode,
any model with audio input; on a single-model server it only says
whether that model can transcribe. It notes that dictating with
another model loads it beside the chat model.
- Transcribe voice messages: on by default.
- Transcription language: empty for automatic.
The keys live in the settings registry like the others, so defaults,
storage, reset and export need nothing new. The model picker is drawn by
SettingsChatVoiceTab, the way Extensions and Tools draw their sections.
Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: record voice messages and dictation from the chat form
The microphone now does what the conversation's model allows, without
the user choosing a mode: its label says "Send a voice message" or
"Dictate", and when no model can take audio it is disabled and says why
(shown in router mode only when some model there has audio input, so the
reason can be acted on; a single-model server without audio keeps the
send button as before). The microphone errors keep their warning state
and toast, now also for the voice engine's own capture errors.
Recording goes through the voice engine's VoiceRecorder instead of the
MediaRecorder flow. While it records, the actions row becomes a
recording row: cancel, the live waveform, the elapsed time and stop.
Escape or cancel drops the recording. The form keeps its two rows while
recording, so the message stays visible on desktop and the row fits the
phone layouts and the expanded sheet.
- Dictation writes the live transcript into the message at the caret
as it streams. DictationSpan replaces only the dictated stretch, moves
it when the user types before it, keeps what is typed after it, and
hands text the user edits over to them. When nothing streams, the row
shows Transcribing... after stop and the text goes in once it is back.
Cancel takes the dictated text out again. No audio is attached.
- A voice message streams its live text, if any, under the waveform,
not into the message. On stop the clip is attached as a voice file
with its peaks and length, and its transcript is followed by the voice
transcripts store (unless Transcribe voice messages is off).
Realtime transcription is skipped for servers reached through a tunnel
or with an API key, which a browser websocket cannot carry. The first
dictation with a separate router model shows a one-time note that it
loads beside the chat model.
The form takes an optional voiceRecorder, so tests and stories can run
it without a microphone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw audio attachments as playable waveforms with their transcript
A voice message should read as a clip, not as a file chip. Audio
attachments now render with Llama UI's VoiceWaveform inside
VoiceTranscriptReveal, through a direct branch in the core attachment
item (voice is core, so it does not go through the extension renderers):
the removable sm chip in the chat form, and the md clip in a sent
message, on the user bubble's surface.
- The audio plays from an object URL made from the stored base64 (or the
uploaded file's data URL), revoked when the clip leaves the screen.
- A clip stored without peaks, an older voice message or an uploaded
audio file, gets them computed from its audio once per session, in
memory only; uploaded files keep their name under the waveform.
- The transcript opens under the clip on hover, keyboard focus or a tap,
and shows Transcribing... while the voice transcripts store waits for
it, Transcription unavailable when it failed, and nothing when none
was asked for.
- The full-screen preview keeps its player and shows the transcript.
Tests run the chat form with a fake recorder: the microphone per mode,
the recording row, dictation around typed text, the Transcribing...
state, cancel, and a voice message chip whose transcript arrives later
and goes with the send. Stories show the form recording each kind, a
recorded chip, and sent voice messages with the transcript, pending,
and computed peaks.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the old audio recorder and proxy realtime transcription in dev
The voice engine replaced AudioRecorder, so it goes. The dev server now
proxies WebSockets on /v1, so realtime transcription works under vite dev.
The VoiceCapture test waits for the tone to be recorded instead of a fixed
700 ms, which a loaded test run could miss.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: dictate with a model that is already loaded when the chat model cannot hear
The Transcription model setting's Automatic choice now falls back to a
router model with audio input that is already loaded, so dictating with a
text-only chat model works without picking a model and loads nothing.
A picked model still wins, and the empty value is a named constant.
Idea from upstream PR 27567, by NickM-27.
Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: fill the recording waveform, and open voice message transcripts
- The live waveform fills the row from the start: a recording shorter
than the row is stretched over all its bars, a longer one scrolls with
the newest level on the right. No dotted placeholder.
- A voice message's transcript is open under its clip, in the chat form
while it comes in and in the sent message, with the toggle beside it
to fold it.
- Settings > Voice > Collapse transcripts (off by default) keeps them
folded instead, opening on hover, keyboard focus or a tap.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 2a93f2f948ed976e40f693ce5d3acf1168284b05)
* ui: record voice without ducking other audio, from the first word
- Voice input asks for the raw microphone signal: no echo cancellation,
noise suppression or auto gain. Echo cancellation made the system treat
a recording as a call, which lowers or interrupts other playback, and
auto gain faded the first words in.
- One AudioContext serves every recording, with the capture worklet
loaded once, suspended in between and, where the browser allows it,
without an output device, so recording never touches the speakers.
- The chat form prepares it when it gets focus (no microphone, so no
recording indicator), and a mouse starts recording on press.
- A recording starts counting once audio flows: until then the bar says
Starting... instead of running the timer. Development builds log how
long each step of a start took.
- Settings > Voice > Microphone: Automatic (the system default, or the
built-in microphone when the default is a Bluetooth headset, which
would switch to its low-quality call mode), System default, or a
listed microphone.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 76643e7210d21617d734fdfde677e84af7273153)
* ui: show voice text live in the message box, and follow transcripts as spoken
- A model that hears but has no live ASR (e.g. LFM3-Omni) now gives text
while you record: the realtime client commits the audio piece by piece,
at the pauses the levels show, one piece at a time, and the final
commit takes only what is left.
- A voice message's live text shows as the empty message box's
placeholder. Clicking into the box makes it text to edit there, and
what is said next follows it. On stop the text leaves the box and goes
with the clip; when it was edited, the edit is the clip's transcript.
- Llama UI gains VoiceSpokenText and spokenLength(): spoken words in the
full foreground colour, which they keep, the rest muted. A clip's
transcript follows its playback (estimated over the parts with speech,
as there are no word timings), and live text brightens as it arrives.
- The recording bar has a send button that stops and sends in one go.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit ce773017f902f8a5690ab9ecca368cecb6427422)
* ui: let the capture test start with silence
A capture starts with its first audio block, which can come before a
stream's first sound, so the steady tone is checked from where it begins.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit a41ee5337b83fef685a5cbaff176925f1647ce9d)
* ui: show live voice text sooner, and keep silence from becoming "yeah"
- A model that transcribes per commit (LFM3-Omni) gets shorter pieces:
cut at a 0.35 s pause after speech, or at the next quiet moment once a
piece reaches 3 s, so the text follows the speaker within a few
seconds in dictation, the message box and voice mode.
- When the recording stops, the audio after the last piece is committed
only if speech was heard in it. LFM3-Omni transcribes a commit of pure
silence as "Yeah." (or "はい。"), which ended up at the end of the text.
A recording whose only piece is that rest still commits it.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 43e52cbab0fba9d8e84b9491890a406e3c6c90f3)
* ui: transcribe a recording cut at pauses once more as a whole
Live text comes piece by piece, cut at the speaker's short pauses, and a
model reads each piece as a sentence of its own: "I went. To the store."
When the recording ends, one pass over the whole clip gives the final
text, punctuated by meaning, in dictation and in voice mode alike. The
pieces' text stays the live text until then, and is kept when that pass
fails or the recording is longer than two minutes.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 359dd9632e6c88000a1d23c2575bd1e6fcdd9ec1)
* ui: give live text over HTTP for models without a realtime session
A model that hears but cannot hold a realtime session, such as
Qwen3-ASR, was transcribed only after the recording, so dictation
showed no text while speaking. The recording is now cut at the
speaker's pauses as with a realtime session, and each piece sent to
/v1/audio/transcriptions on its own, one at a time, its text joined
live. The audio is kept from the start, so what is said while the
socket answers goes with the first piece. The whole clip is still
transcribed once more at the end, and the pieces' text is kept if
that fails. A server a websocket cannot reach (an API key, a remote
server) gets live text the same way.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say when the browser processes the microphone despite the raw request
The browser keeps one processing setting per microphone: while a call,
another tab or voice mode has it open with echo cancellation, a
dictation that asked for the raw signal is processed too, sounds
distant, and the system may lower other sound. The capture now warns
when it gets processing it did not ask for, and its development log
names the microphone, its rate and what processing is on.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the microphone beside the stop and send buttons
While a reply streams, or once the message has text, the microphone no longer
gives way to the main button: it stays beside it as a quieter, secondary
button, so more can be dictated without clearing the message or waiting for
the reply.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show the voice components' stories with Llama UI
VoiceWaveform and VoiceTranscriptReveal are Llama UI components, so their
stories move to the library's Storybook, under Llama UI/Voice, with the demo
audio they play. Llama Chat's Storybook shows them there too, and its own
voice stories take the demo audio from the library. VoiceWaveformLive stays
with Llama Chat, as its demo records through the chat form's microphone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a noise suppression setting for voice input
Recordings stay raw by default. Settings > Voice can now switch the
browser's noise suppression on; echo cancellation and automatic gain stay
off either way, as they interrupt other sound and fade the first words in.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep live voice text coming when the room is never quiet
Pieces were cut only at a quiet moment, so with music or talk in the room
none was ever cut and no text showed until the recording ended. A piece half
as long again as its longest length is now cut where it is.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add echo cancellation and automatic gain settings for voice input
Settings > Voice now has a switch for each kind of microphone processing the
browser offers. All three stay off by default, so recordings are raw; each
one says what switching it on costs.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: put the microphone after the stop button, filled like it
While a reply streams, the microphone is the last button of the row and
looks like the stop and skip buttons beside it, instead of a bare icon
before them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say when dictated text is being refined, and send it refined
After a recording with live text, the whole clip is transcribed once more
so pauses do not end sentences. The form now says so while that pass is
out, and a send asked for meanwhile waits for it, so the message goes out
with the refined text rather than the pieces.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the send button in place and put the microphone beside it
The send button no longer gives way to the microphone on an empty
message. The microphone stands before it as a quieter, tertiary button,
with the same tooltip, warning and recording states as before.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let a recording be sent while a reply streams, once it has words
While a reply streams, the recording row now shows its send button as
soon as the first transcribed words are in the message, so the
recording no longer has to be stopped first. The microphone also stays
beside the send button once a voice message is attached.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: raise quiet speech for transcription, and say when it is too quiet
A quiet microphone gave the model near silence to transcribe. A new
Voice setting, Boost quiet speech (on by default), raises speech to a
level speech models hear well, up to eight times louder: the audio
streamed for live text as it goes, and the whole recording in one pass
when it stops, so its first words get the same gain as the rest. Only
speech is measured, never the silence around it, loud speech is left
as it is, and a soft limiter keeps raised peaks from clipping. When
speech stays quiet even at the highest gain, the recording row says so.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the voice engine free of app code
The engine in lib/voice imported the microphone messages from the app's
utils and the WAV type from its enums, the only two things keeping it
from moving into a package of its own. The microphone checks and
messages move into the engine (voice/support.ts), next to the capture
that needs them; the uploaded-file helpers stay in the app's utils.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: start voice sessions from one voice service
How a recording reaches the server (routes, API key, whether the
realtime socket can be used) and how the Voice settings become capture
and transcribe options were built inside the chat form's voice hook,
and copied into every other feature that records. VoiceService builds
them in one place: a feature asks it for its session options and adds
only its own callbacks.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tell speech from the room in one place
The pause detector, the speech gain and (in voice mode) the listener for
speech over a reply each followed a noise floor with their own numbers.
A SpeechDetector now does it for all of them, in dB, with one set of
settings in VOICE_SPEECH.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let any text field take dictation
Writing dictated text into a field (at the caret, around the user's
typing, replaced by the whole recording's text when it comes) moves out
of the chat form's hook into a plain createDictation(). The hook keeps
what is the chat form's own: voice messages, toasts and the hint.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move the recording row into Llama UI
VoiceRecordingBar draws the row shown while recording (cancel, waveform
and time, send, and a caption or warning under it) from data alone, so
voice mode or a notes page can show the same row. The chat form passes
its labels and its microphone button as the stop button.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the voice engine's Svelte recorder behind its own entry
VoiceRecorder, the one part of lib/voice written with runes, moves to
lib/voice/svelte, so the rest of the engine is plain TypeScript any app
can use without Svelte.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move voice input into its own package, Llama Voice
The voice engine (capture, the speech signal, transcription over the
realtime socket or HTTP) and dictation into a text field move from the
app to ui/lib/voice as @ggml-org/llama-voice, so another app, such as a
notes app, can use them. The Svelte recorder is its ./svelte entry.
Llama Chat compiles the sources, as it does Llama UI's, and keeps its
server and settings wiring. The package has its own type check, tests
and build, and a CI job for them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show which model does what in voice input
The model picker and the Models manager mark each model that hears:
Dictation on the one that transcribes dictation, Speech to text on
transcription models such as Qwen3-ASR, and Voice on chat models that
take voice messages. A loaded model kept only for dictation sits apart
from the chat models, a model can be made the dictation model from its
row, and Settings > Voice groups the transcription models apart from
the chat models with audio input. A voice store now resolves what the
microphone does, for the chat form and the model lists alike.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: warn of a quiet microphone inside the recording row
The too-quiet line under the waveform made the form grow and jump. The
warning now stays inside the row: the waveform turns the warning colour,
an icon sits by the time, and the advice opens above the row in a
tooltip for a few seconds, then on hover or focus. Screen readers still
hear it when it appears.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the microphone while the pointer is on its button
The first seconds of a recording could come out quiet while the
microphone settled, and the too-quiet warning showed for them. With the
pointer or the focus on the microphone button, the microphone now opens
ahead of the press (only when the page may already use it, so nothing
asks), and the recording takes it at once; it is let go shortly after
the pointer leaves without a press. The warning also waits until the
speech has stayed too quiet for two seconds.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tidy voice input after review
Fix a microphone left open when a recording is cancelled while it starts
or the pointer leaves while it opens, a capture that is still letting go
silencing the next one, and live text stopping after a realtime error.
Share the dictation model choice, the transcript lookup, base64 decoding,
PCM joining and stopping a stream instead of repeating them, and export
only Llama Voice's public API.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin Llama Voice's dev dependencies to Llama UI's versions
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let demos speak into a synthetic microphone
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the microphone for voice input, off the audio files entry
The attach menu's Audio Files entry used a microphone, which now sits
beside the form's own microphone button. The microphone stands for
recording and dictation only; a model's audio modality is headphones, and
the audio files entry is a file with headphones.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: set voice input on the slimmed design tokens
Voice input's components use the type roles, the one corner scale and
shadcn's utility names for the neutral colours, as the rest of Llama UI
and Llama Chat do since the design tokens were slimmed down.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name voice states, platform strings, patterns and tunings
Addresses the review on the voice PR:
- Llama Voice: VoiceCaptureState, VoiceRecorderStatus, RealtimeState and
RealtimeUnsupportedReason are enums (src/enums.ts, exported); DOMException
names, AudioContext states, transcription form fields, socket protocols,
device kinds, worklet messages, error messages, regexes and tuning values
are named constants.
- Llama UI: VoiceTranscriptStatus, VoiceWaveformSize and
VoiceRecordingBarStatus are enums; KeyboardKey, MouseButton and PointerType
join its UI enums; the voice components' sizes, labels, timings, selectors
and patterns move to constants/voice.constants.ts. The voice unit tests
move from src to tests/unit.
- App: ChatFormVoiceState is an enum, the recording bar's status comes from a
map, and the voice code uses the enums above, BooleanString, HEADERS and
new voice constants in place of inline strings, regexes and numbers.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep all of Llama UI's tests under tests
The Model UI formatting spec moves to tests/unit and the button spec to
tests/client, next to the voice specs; the test projects now read only
tests/unit and tests/client.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a transcript's words without the model's own format
Qwen3-ASR names the language before its text (language French<asr_text>...) and
VibeVoice-ASR answers with timed segments as JSON, which the server passes on as they
are. The voice engine now takes both off, over HTTP and over the realtime socket, also
while segments still stream, so the message box and the clip show the words alone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: dictate when a speech-to-text model is picked for the conversation
A speech-to-text model has audio input, so on Automatic the microphone recorded a voice
message for it, which such a model has nothing to answer with. It now dictates, as the
model's badge already says it is for.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say why a live-only model cannot transcribe behind an API key
A browser websocket cannot send the API key, so the realtime socket is left out on a
server that needs one, and a model that only transcribes over it (Voxtral Realtime)
answers the HTTP route with a 501. The clip and the dictation toast now say so instead of
only that transcription is unavailable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pick the dictation model with a microphone on its row
The model that transcribes dictation was marked with a filled Dictation badge. It is now
marked by a red microphone among the row's actions in the model picker, beside the heart,
and every other model that hears shows the same microphone on hover: a click makes it the
transcription model, and a click on the chosen one hands the choice back to Automatic. The
badge keeps saying what kind of hearing model a row is, and the Models manager marks the
dictation model with a small microphone beside its name.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: search message content and Discover models from the search dialog
The search dialog matched chat titles only. It now also scans what was
said in each chat (content, reasoning, tool calls and results, attachment
names and text), newest chat first, stopping at the first match per chat
and after 20 chats, and shows a snippet around the match. Ids, timestamps
and base64 payloads are not searched.
With Discover on, it also lists matching catalog models and, once typing
settles, Hugging Face repos; picking one opens Discover on that repo.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the search dialog one width whatever it lists
The shared dialog sizes itself to its content from md up, so the search
was narrow while empty and widened as results came in. It now takes the
full max-w-2xl there, less a margin on narrower windows.
Assisted-by: Claude (Opus 5.5 Medium)
The unscoped llama-ui on npm is a 0.0.1 placeholder. After the scoped
publish, the workflow renames the same build and publishes it again at
the same version, with a readme note pointing at the scoped package.
Assisted-by: Claude (Opus 5.5 Medium)
llama-tts built the mtmd context from mtmd_context_params_default(), which hardcodes 4 threads, and never copied n_threads over, so -t never reach the mtmd side.
Copy params.cpuparams.n_threads into mtmd_params.n_threads, as mtmd-cli and the server do; with no -t this now follows the resolved default.
Verified: -t 1 now runs single-threaded, -t 4 is unchanged, and the produced WAV is identical.
Assisted-by: Claude
* fix(server-bench): prevent float bin-count TypeError in histogram generation
Symptom: benchmark crashes while generating gen_rate histogram.
Root cause: np.ceil(...) returns numpy.float64, and that value was used where an integer count/shape argument is required.
Fix: cast ceil result to int before +1, ensuring bin count is an integer.
Result: histogram path completes and gen_rate.png is generated reliably.
* Update scripts/server-bench.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ui: load highlight.js on first use
highlight.js and every lowlight language (~900 KB raw) moved out of
Llama Chat's first load into a chunk that loads with the first code block.
- @ggml-org/llama-ui/code: highlightCode stays synchronous. Until
highlight.js has loaded it returns the code escaped as plain text and
starts the load; a $derived or $effect that called it runs again once
it has loaded. New loadHighlighter() and isHighlighterLoaded().
- Markdown: rehype-highlight runs through a lazy rehype step, like
KaTeX's: the first message with a code block waits for it, every
other one renders in the same pass.
First-load JS of the chat page: 3.47 -> 2.56 MB raw, 1.05 -> 0.75 MB
gzip, 0.84 -> 0.60 MB brotli.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load each highlight.js language on first use
highlight.js now loads one language at a time instead of all ~190 at
once with the first code block: every language is its own chunk, and
the core loads with the first one.
- @ggml-org/llama-ui/code: highlightCode looks a language or alias up
in a generated table (scripts/generate-highlight-languages.mjs), loads
it with the languages it embeds (CSS and JavaScript inside HTML, say)
and returns plain text until then; a $derived or $effect that called
it runs again once it has loaded. Auto-detection picks among lowlight's
37 common languages, loaded the first time something auto-detects.
New loadHighlightLanguages(), isHighlightLanguageLoaded(),
resolveHighlightLanguage() and getLoadedHighlightLanguages() replace
loadHighlighter() and isHighlighterLoaded().
- Markdown: rehypeLazyHighlight highlights with lowlight's core and the
languages a message uses, waiting only for one it is the first to use.
rehype-highlight is no longer in the pipeline, since it bundles
lowlight's common languages.
First-load JS of the chat page stays at 2.58 MB raw, 0.76 MB gzip,
0.61 MB brotli: no highlight.js code, only the 14 KB language table.
The first code block now loads its language (1-12 KB for most) and the
20 KB core instead of 900 KB.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the code-block class names and shell language as constants
Assisted-by: Claude (Opus 5.5 Medium)
A printable key pressed on the chat screen while no field, editor,
dialog or menu has focus now moves focus to the chat form, and the
character lands there.
Assisted-by: Claude (Opus 5.5 Medium)
* model: load `ffn_gate_up_exps` scales
The generic scale pass loads per-expert scales for ffn_gate_exps,
ffn_up_exps and ffn_down_exps but not for the merged ffn_gate_up_exps.
Load ~ffn_gate_up_exps_s there~ and expose it as ffn_up_exps_s ~when the layer
has no separate up tensor,~ since build_moe_ffn applies up_exps_s to the
merged gate_up output. Inkling 2 now passes the expert scales to
build_moe_ffn like the other MoE models.
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-model.h
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama : move the indexer key rotation choice into the model classes
The KV cache forced a Hadamard rotation for the lightning indexer keys
based on a list of architectures. The virtual llama_model::indexer_k_rot()
now returns true for the same models. DeepSeek4.1 inherits from DeepSeek4
but was not in the list, so it overrides it back to false.
* llama : rename indexer_k_rot() to lid_k_hdmd()
* ui: drop component tokens that only alias a semantic token
A component token now exists only for what a component decides itself:
a value no semantic token has, or one that only this component changes
between themes. Components that take a semantic token as it is read it
directly, so 88 of the 118 component tokens go, with no change to what
is drawn.
Also drops semantic tokens that repeat another one (status-* is fill-*,
elevation-tooltip is elevation-surface), the unused info lamp tokens and
14 primitives nothing reads, and adds fg-soft for the label of an option
that is not the picked one (tabs, toggle group).
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a short summary on top of DESIGN.md
The rules most changes need, in one screen: components, colour, type,
shape, sizes, surfaces, tokens and no arbitrary values. The sections
below keep the detail.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: derive design tokens by rule and drop the alias layers
Semantic tokens are either a pick (a primitive per theme) or a derivation,
mixed from another semantic token by a fixed rule and declared once: hover
fills, status surfaces and lines, dark shadows (--elevation-strength), the
middle of a matte edge, a lamp's rim and glyph.
Primitives keep colours only. shadcn's colour names stay as Tailwind
utilities and no longer exist as variables; neutral semantic utilities that
duplicated them are gone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: set all text through type roles and move off-scale values onto the scale
Tailwind's text sizes and bracketed sizes become the type roles, with a new
text-caption-sm (10px) for tags and counts. Body and caption roles no longer
set a weight, and line heights are written as Tailwind writes its own, so the
swap from text-sm and text-xs changes no pixel. A unit test keeps old sizes
from coming back.
Arbitrary values with a scale equivalent use it (z-20, max-w-4/5, min-w-32),
and the two easing curves repeated across the app are ease-out-quint and
ease-sheet. DESIGN.md is updated and regenerated.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: bump @ggml-org/llama-ui to 0.5.0
Removed and renamed tokens are breaking for an app that reads them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a display type role for the biggest headline
The greeting on a new chat keeps its 30px from md up, now as text-display
instead of a Tailwind size.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: one corner scale, one set of shadows and three more derived tokens
Six corner roles named values Tailwind's rounded-md to rounded-4xl
already have, and components used the Tailwind names nine times out of
ten. The roles are gone; the whole scale is computed from --radius.
Tailwind's shadow sizes were restated next to the elevation tokens and
did not follow the theme, so they were five times fainter in dark. The
five uses take elevation tokens and a unit test keeps both from coming
back.
A button group's seam is the button's own text colour at 25%, the dark
status text is the dark status fill, and the pressed shadow scales with
--elevation-strength like the inset one.
Assisted-by: Claude (Opus 5.5 Medium)
The model modality icons match the ones the Llama apps use: glasses for
vision, headphones for audio. The attach menu's list of what the model
takes reads the same icons instead of its own microphone and picture.
Assisted-by: Claude (Opus 5.5 Medium)
graph_max_nodes() picked the nodes-per-tensor factor from lists of
architectures. Each model class now gives it through a new virtual,
llama_model::graph_nodes_per_tensor(), which defaults to 8. The
architectures that need 32 or 64 override it in models.h, and DFlash
keeps its hparams-based rule in dflash.cpp.
LoRA adapter nodes are now added for every architecture, not only for
the ones on the default factor.
* ui: move the reasoning effort into the model picker
The chat form's row no longer holds the reasoning button: the model
picker ends with a Reasoning row, in place of Manage models, that shows
the level in use and opens the levels under it. A phone's model drawer
has the same row. A desktop server with a single model, whose selector
opens the manager instead of a picker, keeps the button beside it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a reasoning effort that is not the default on the model selector
Next to the model on the closed selector, a set level shows as its word,
and reasoning that is off as a small crossed-out lightbulb. At the
default, which leaves it to the model, nothing more shows.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the model picker's footer in place while it opens in Safari
WebKit pins a bottom-sticky element against the scaled box of a parent
that zooms in, so the Reasoning row rode up over the model list until
the opening animation ended. The desktop picker now scrolls its search
and list in a box of their own, with the footer under it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep Pele diagrams inert
Pele turns a node with `click ... href` into a live link, where Mermaid's
strict security level left diagrams without links. Unwrap every link in
Pele's output, keeping its children, so the inline block, the live-preview
editor, the preview dialog and the downloaded svg stay inert. The Diagrams
story's flowchart now carries click directives and its play test checks
that no link or href is left.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open diagrams full screen, zoom around what is on screen
The diagram preview dialog fills the window on every device. The preview
places the diagram from its top left corner, so a zoom keeps a screen point
fixed: the wheel and a trackpad pinch zoom around the pointer, the buttons
and +/- keys around the middle of the screen, wherever the diagram has been
panned to. Two-finger pinch, double-click and arrow keys work too. The
diagram opens fitted and keeps fitting until panned or zoomed; Reset view
fits it again.
Adds MermaidPreview and DialogMermaidPreview stories with play tests and a
unit test for the zoom math.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a diagrams trace to the demo fixtures
Assisted-by: Claude (Opus 5.5 Medium)
The MoE cache adds a slot lookup to every MoE layer, and with
--moe-cache-cpu-share a second expert path, but graph_max_nodes() did
not account for it. Models on the default 8*n_tensors budget with few
tensors per layer (cohere2moe: 443 tensors, 49 layers) overflow the
graph context when the cache is combined with a quantized KV cache
(K/V rotation), aborting in ggml_new_tensor with GGML_ASSERT(obj_new).
Fixes#32584
Assisted-by: Claude Opus 5.5
* tests: add CUDA-only MUL_MAT/MUL_MAT_ID cases with results above the fp16 range
IQ1_M weights have no MMQ kernel, so once the batch is bigger than the MMVQ
limit the path is dequantize + cuBLAS. These cases produce results around
1.3e5, past the fp16 max of 65504, so an fp16 result buffer turns them into inf
while the CPU reference stays finite. They are only added for the CUDA backend.
Vulkan deliberately accumulates quantized matmuls in fp16 at default precision
and would fail them by design.
* CUDA: use an fp32 result for quantized src0 in the cuBLAS fallback
On the F16 compute path, quantized weights went through cuBLAS with fp16
accumulation and an fp16 result buffer that was converted to fp32 afterwards.
Anything above 65504 became inf and the next layer turned into NaN (see #30951:
gpt-oss-120b with IQ1_M experts at -ub 512). Set prefer_f32_output for
quantized src0 so the GEMM still uses fp16 operands but accumulates and stores
in fp32, the same mode already used on Volta, RDNA4 and CDNA. Every other
quantized path already accumulates in fp32.
Diagnosis and fix by denizyuret in #30951.
* CUDA: keep the MUL_MAT_ID precision in the per-expert fallback
The per-expert dst_slice kept op = GGML_OP_NONE, so ggml_prec_get_acc returned UNDEFINED and a requested precision was ignored. Set the op to GGML_OP_MUL_MAT. Also revert the fp32 result change and remove the tests.
---------
Co-authored-by: JonSnow1807 <JonSnow1807@users.noreply.github.com>
On Adreno, subgroupAdd() after divergent code returns wrong sums. The
fused ADD+RMS_NORM partial-sum path hits this, so the decode-step norm
is garbage and models emit stray tokens after long prompts.
Clear subgroup_arithmetic for the Qualcomm proprietary driver, which
also turns off add_rms_fusion and the other subgroupAdd paths.
test-backend-ops on Adreno 850 and X2-90: ADD_RMS_NORM 20/25 -> 25/25,
RMS_NORM_MUL_ADD 33/45 -> 45/45. llama 3B Q4_0 on Adreno 850: pp512
and tg128 unchanged.
load_arch_hparams multiplies n_hrm_layers_per_stack, n_hrm_h_cycles and
(n_hrm_l_cycles + 1) into a uint32_t, then asserts the product equals
block_count.
Any factors whose true product is block_count + k*2^32 still pass the
assert; 1073741825 * 2 * 2 = 2^32 + 4 compares equal to block_count = 4.
load_arch_tensors iterates the same three values in int and writes past
the end of the block_count-sized layers vector.
* opencl: support int8 KQ for DK = 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
* opencl: allow query chunk up to 96 for int8 KQ
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
* opencl: extend `build_program_from_source` to allow non-fatal failure
Assisted-by: Codex
* opencl: add int8 KQ for decomposed FA with DK <= 256
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA)
* cuda : SET_ROWS from BF16 into quantized types and F8_E4M3
* cuda : fuse BF16 rope into a BF16 cache
* tests : trim the BF16 rope cases to one per code path
- Allow cls.output.weight to be absent from the mmproj. It is an optional
Qwen3 reranking head that the YuE2 NAR decoder never uses.
- Without this fix, mtmd_init_from_file() fails with
"tensor cls.output.weight is missing from the mmproj", preventing
audio generation from starting.
* ggml-openvino: round the GPU KV length up to a power of two
Review notes (follow-ups, not blocking):
- convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
copies it into the ov::Tensor. Writing the padded rows straight into
padded.data() avoids the second allocation and copy (up to 16 MB per
call at a 16K context).
- The padding is skipped when the mask is not contiguous, and the KV
inputs are then sliced to more rows than the mask has columns. Masks
are always contiguous today; a GGML_ASSERT would make that explicit.
- compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
is_stateful_enabled() is local to ggml-openvino.cpp.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Remove solution C and keep A as default
* ggml-openvino: match the output count before reusing a compiled NPU prefill
The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.
* ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding
* ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix
- compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
ran before the new one and still exported result_norm, so the
result_norm exclusion had no effect. Keep only the new block.
- compute_node_dynamic_dims checked is_activation_input twice in a row.
Remove the second copy.
- process_view_input_new created the dynamic Split on split_dim. With
stateful execution the input drops the leading size-1 axis, so use
split_axis, which already accounts for the rank difference.
* ggml-openvino: hash the output flags into the model-cache fingerprint
The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.
* ggml-openvino: support f16 SCALE and f16/bf16 TRI
test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.
SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.
* ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
layer
The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.
* enable mamba based models on OV CPU-GPU
* revert changes in tests
* docs : add Falcon-H1 to the OpenVINO validated models
Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.
* ggml-openvino: simplify the SSM_SCAN build check and clean up
- Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
always provides the op. Drop the no-op LANGUAGE CXX source property.
- Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
with -Wextra-semi, which fails builds with fatal warnings.
- Add ggml_openvino_is_static() for the repeated NPU or
GGML_OPENVINO_FORCE_STATIC check.
- Share the f16 and f32 KQ mask padding code.
- Drop the redundant inline on in-class static functions.
---------
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
* cuda: add scatter for batched tensor writes
* update test case to mirror get_n test case
* save restore timing logs
* refactor code into batch-copy.cu file
* Adds initial scheduler stress tests
* Adds new sleep op to simplify scheduler testing
* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321
* Adds experimental vulkan GGML_OP_SLEEP
* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.
* Cleanup: Removes CUDA specific comments and nvtx-markers
* Fix: forgot to add sleep specific test case
* Fix oversights identified by CI
* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior
* Fix: Early exit in CPU-only settings (cause for graviton CI fails)
* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds
* Exclude ACCEL devices from new tests due to incompatibility
* Fixes vulkan test failure in llvmpipe
* Adds asserts, improves comments and some checks
* Fixes nitpicks in white-space and variable naming
* Adds comments that certain vk drivers could time OP_SLEEP incorrectly
* Drops limitiation to CPU/GPU in test-sleep
* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)
* Implements feedback for OP_SLEEP to work in place (no copy)
* Apply non-code suggestions from JohannesGaesslers code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail
* Make clang-tidy happy
* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)
* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name
* Drops unnecessary shader_clock.comp test and guards
* Replaces sched_ prefixes with helper_ prefix
* Moves test suite to new RAII types for better memory management
* Adds sleep option to test_chain_all_backends
* Adds sleep option to first-run arm of the y-shaped test
* Fix switchcase order for legibility in stress_test_dag
* Simplfies stress_test_linked_list via double for-loop
* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)
* Fix RPC Proto patch due to GGML_OPS bump
* Adds ggml-cpp.h treatment to test-sleep.cpp
* Adds TODO pointers for subsequent additional tests.
* Apply suggestion to ggml.h
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)
* Apply batched suggestions from gg code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Completes callsite change of gg-suggestion
* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : give Inkling2 its own relative-position attention input
The Inkling2 graph built its flash attention mask from the hybrid iswa
input and passed it to build_attn through a new kq_mask argument. That
argument let a caller pair its own dense mask with the kv_rows of
llm_graph_input_attn_kv_iswa, whose mask follows the per-sequence cell
lists whenever kv_rows is enabled.
Add llm_graph_input_attn_rel_pos instead. It owns dense KQ masks and the
query-to-cell distances, fills both from the same cells, and its
build_attn overload adds the per-head bias to its own mask and never
passes kv_rows. The iswa build_attn is back to its previous signature;
its body moves to build_attn_iswa_impl, which both overloads call.
Inkling2 now folds the per-query sink into the bias with and without
flash attention, so build_attn_mha no longer needs the per-query sink
path.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* inkling2 : make the expert weight scale and final logit softcap optional
Load both keys as optional and skip them when they are zero, as other
models do, instead of rejecting zero at load.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : build kv_rows masks for the hybrid iswa memory
llm_graph_input_mem_hybrid_iswa checked its masks with
can_reuse_attn_mask, which expects kv_rows whenever they are enabled,
but built them dense. With a unified cache and flash attention the
check always failed, and hybrid iswa models (LFM2, LFM3, Inkling2)
never reused a graph. Build the masks with build_attn_inp_attn_mask,
like the plain iswa input.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : make the relative-position input a hybrid iswa input
Address review: llm_graph_input_mem_hybrid_iswa_rel derives from
llm_graph_input_mem_hybrid_iswa and owns the attention and recurrent
inputs, so Inkling2 builds one memory input instead of two. Its
attention input is built with dense masks, can_reuse checks them as
dense, and its build_attn asserts that the mask has no kv_rows (TODO:
a kv_rows variant). build_attn_iswa_impl is now a static helper.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : compose the hybrid iswa rel input, format inkling2
Address review: llm_graph_input_mem_hybrid_iswa_rel owns its attention
and recurrent inputs like the other composite inputs instead of deriving
from llm_graph_input_mem_hybrid_iswa.
Format src/models/inkling2.cpp like the other model sources: grouped
and aligned key and tensor loading, descriptive per-layer names, blank
lines and section comments between the steps, and cb() names for the
intermediate tensors. The graph is unchanged.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
llm_graph_params no longer carries hparams, prec_policy and hdmd, since the model reference already provides them. llm_graph_context sources them from the model and keeps prec_policy as a reference.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal: add a tensor API conv_2d kernel and split long reductions
The conv_2d matmul kernel runs on simdgroup matrices only, and gives
each threadgroup the whole reduction of its output tile. On M5 this
leaves the tensor cores unused, and convolutions with few output
positions and a long reduction (IC * KW * KH in the thousands) run a
handful of threadgroups on an otherwise idle GPU.
Add kernel_conv_2d_mm_tensor for devices with the tensor API: like
kernel_mul_mm, it gathers the patches of 64 output positions into
threadgroup memory as the left operand and reads the kernel rows
from device memory as the right operand. Devices without the tensor
API keep the simdgroup kernel.
Both kernels can now split the reduction until there are enough
threadgroups: each split writes its partial sums to a buffer placed
after the output, as flash attention does, and a small kernel adds
them in a fixed order, so the result stays deterministic. CONV_2D is
marked as an op whose allocation may exceed its output size.
test-backend-ops -o CONV_2D passes, with and without
GGML_METAL_TENSOR_DISABLE.
CONV_2D, F16 kernel, us/run on an M5 Max:
[19,19,256,16] k [4,4,256,4096] 19008 -> 5235
[16,16,128,8] k [3,3,128,512] 333 -> 124
[1536,1536,32,1] k [3,3,32,64] 12086 -> 6512
[33,1,1280,1] k [3,1,1280,1280] s2 659 -> 33
[2054,1,768,1] k [7,1,768,768] 3161 -> 699
* metal: walk the conv_2d patches incrementally, size splits by output
The patch gather of both conv_2d matmul kernels spent most of its
time on address math: 64-bit multiplies, divisions and four bound
checks per value. Walk the reduction index with 32-bit byte offsets
inside the image, updated by additions, with one unsigned compare per
bound; the host checks that an image fits. The tensor kernel no
longer gathers the reduction indices past the end of its split.
The number of splits now targets about 1M partial sums, with at least
3 reduction steps per split, instead of counting the tiles of one
kernel.
CONV_2D, F16 kernel, us/run on an M5 Max:
[19,19,256,16] k [4,4,256,4096] 4998 -> 4512
[1536,1536,32,1] k [3,3,32,64] 6721 -> 5354
[4096,1,1024,1] k [7,1,1024,1024] 2701 -> 2236
[2054,1,768,1] k [7,1,768,768] 1050 -> 871
with GGML_METAL_TENSOR_DISABLE:
[4096,1,1024,1] k [7,1,1024,1024] 9047 -> 8283
* nit: share the CONV_2D simdgroup count between host and kernel
kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per
threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel
constant and res.nsg, so the dispatch and the kernel cannot drift apart.
* metal: fall back to the generic conv_2d kernel for images past 4 GiB
The matmul conv_2d kernels walk the patches with 32-bit byte offsets
inside an image. Instead of asserting on nb13, compute the largest
offset actually read from ne/nb, which also covers strided views of
the input, and route images that do not fit to the generic kernel.
* cont : comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ui: import Lucide icons by path in modules that run in Node
Assisted-by: Claude (Opus 5.5 Medium)
* ui: register highlight.js languages once, from lowlight's modules
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load the pdf.js worker and nerdamer as files of their own
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw Mermaid diagrams with Pele
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin dependency versions and use one Svelte across the UI packages
Assisted-by: Claude (Opus 5.5 Medium)
* ui: build Llama Chat as split chunks
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load KaTeX and pdf.js on first use
Assisted-by: Claude (Opus 5.5 Medium)
* llama : fall back to buffered I/O on EIO
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Preserve bytes already read, resumes at the current file offset
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* vulkan: Implement stream-k MUL_MAT
Stream-k tries to split chunks of 256 K elements evenly across all SMs.
Workgroups that finish a tile write to dst, others write to scratch, and
there's a resolve shader similar to split-k.
The shader code is implemented for scalar+cm1+cm2, but only currently enabled
for cm2. The heuristic for enabling on coopmat2 is pretty simple due to the
large tile size (one workgroup per SM), but would be more complex for the tile
sizes being used for cm1, and it doesn't make sense for me to tune that on
NVIDIA devices.
Assisted-by: Codex
* address review feedback
The MMQ kernels computed the dst tile offset (jt*J*stride_col_dst) in
32-bit int. For a dst with more than 2^31 elements, column tiles that
start past INT_MAX/nrows wrap around and write out of bounds.
This happens with an all-logits output head, e.g. llama-perplexity
-c 32768 -b 32768 -ub 16384 on a 248320-vocab model: the q8_0 output
matmul has a 248320 x 16384 dst (4.07e9 elements) and aborts with an
illegal memory access.
Widen the offsets in mul_mat_q and the stream-k fixup to int64_t, and
add a (disabled by default, ~9 GB) test-backend-ops case.
Assisted-by: Claude Opus 5.5
n_mudd_taps comes from limite.mudd.tap_count, a scalar read separately
from the length-checked tap_indices array. mudd_tap_idx is a fixed
std::array<int32_t, 512> indexed at [i*n_taps] in both the tensor load
and the graph build, so a crafted GGUF with n_layer * n_mudd_taps > 512
reads past the array.
Reject that at hparams load; the one check covers both index sites.
* llama-bench : size the batch to the prompt for models without memory
Assisted-by: Claude Opus 5.5
* server : read up to 16 lfm3-d1 questions in one pass
Assisted-by: Claude Opus 5.5
* cuda: select cuBLAS precision by GPU SKU
Assisted-by: Codex
* cuda: fix remaining GB10 macro reference
* cuda: review comment addressed, store accumulation preference in device info
* Apply batched suggestions from code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* addressed Oliver's comment regarding initialize NVML once and install headers in Windows CI
* move precision selection helper into init
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* model:add Zaya1 8B (Zyphra)
- Added the support for Zaya.
- Fixed the SSM_CONV case, so the model runs on cuda without abort.
Please enter the commit message for your changes. Lines starting
* Made the following changes based on the bot reviewer feedback:
1) Added runtime asserts when loading wrong KV head size, unsupported conv kernel size.
2) Removed the static cast usage.
3) Fixed double load of tokenizer.
4) Updated res_scale_in to be input tensor.
* - Moved the res_scale_in back to layer 0 to fix the cuda bug
* i) Removed the EOS addition.Team agreed that updating Tokenizer is the right move.
ii) Fixed formatting
* Fixed the comment to make it clear
---------
Co-authored-by: varun <varun_subramanian@zyphra.com>
* ui: install @ggml-org/llama-brand from npm
Both packages take llama-brand 0.1.0 from the registry instead of a
codeload tarball, so npm 12 installs them without allow-remote, which
leaves Llama Chat's .npmrc.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin @ggml-org/llama-brand to 0.1.0
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat publish waits for the registry, bin paths without ./
The first publish of @ggml-org/llama-chat took about four minutes to show
up on the registry, so the 60 seconds the check after publishing waited
failed a release that had gone through. It now waits up to ten minutes.
npm normalizes "./bin/cli.js" to "bin/cli.js" on publish and warns that
the bin was "invalid and removed", which reads as if the command was
dropped. Both package.json files now give the paths as npm writes them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the registry check after llama-chat publishes
npm publish already fails when the registry refuses a package, and the
check after it only failed a release that had gone through, while npm
was still processing it. Nothing waits on the registry any more.
Assisted-by: Claude (Opus 5.5 Medium)
The tool permission card took the fields' border (white 30% in dark), which
outweighed everything else on the page; it now has its own token, white 18%
in dark, the focused chat form's edge. The split-button divider had the
keys' own rim colour (white 12% over a white 9% fill) and vanished between
them; it now has its own token, white 20% in dark. Light mode is unchanged.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide Models and MCP behind settings, off by default
Two new settings in General, "Show Models" and "Show MCP", both off by default, like "Enable Discover Models". The WebUI is often the first thing people new to local models see, and most of them don't need a models manager or know what MCP is.
"Show MCP" only hides the way to manage MCP; servers already added keep working. Turning MCP off for real would mean gating every place that reads the server list (health checks, tool groups, prompts, avatars). Anyone who already has servers saved gets MCP shown, so they don't lose the way to manage them.
With Models hidden, every entry point to the manager goes: the sidebar, search, the shortcut and "Manage models" in the selector -- it opens the same full manager. The /models page still opens by its URL, which apps link to. On a single-model server the selector chip used to open the manager; with it hidden, the chip is a plain label.
* ui: keep Models shown while Discover Models is on
Discover opens only from the Models page (its header button and the manager's row actions), so with Models hidden by default, checking Discover showed nothing. Checking Discover now shows Models too and locks its checkbox, the same way it already locks the Hugging Face Hub API one; unchecking Discover leaves Models as it is. Saved configs with Discover on get Models turned on when they load, so people who use Discover don't lose it.
* ui: run Llama Chat with npx @ggml-org/llama-chat
Makes @ggml-org/llama-chat publishable: drops private, adds bin,
files, exports, repository and a prepack build. Its CLI serves the
prebuilt dist on localhost (3000 by default) and forwards every other
request to a llama-server (--server, default http://127.0.0.1:8080),
with the headers llama-server sends for the UI. A thin unscoped
llama-chat package in alias/ forwards to it, so npx llama-chat works.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat revalidates service-worker.js and asks for a build before packing
SvelteKit 3 adds service-worker.js next to sw.js, and llama-server
revalidates it, so the CLI does too. The folder's .npmrc sets
ignore-scripts, so prepack never ran: it goes, and the publishing steps
build dist explicitly and check the packed file list.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat keeps llama-server's DNS rebinding and cross-origin guards
The CLI called llama-server with its own Host and dropped every Origin,
so a rebound host name or another page got past the checks llama-server
makes when --cors-origins is not *. It now drops only its own page's
Origin, passes other Origins and Sec-Fetch-Site through, and, when it
listens on loopback, answers only to localhost and loopback addresses.
`start` on Windows gets its empty title verbatim.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat build writes the app to a folder for any static server
`npx @ggml-org/llama-chat build [--out <dir>] [--force]` copies the
prebuilt app, without source maps, to ./llama-chat or the given folder,
so a plain file server can host it next to the CLI serving it. It
refuses a folder that is not empty unless forced, and the serving flags
and build flags refuse each other's command. The README shows serving
the folder with http-server forwarding the API to llama-server.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: publish Llama Chat to npm from CI when its version is bumped
ui-chat-publish.yml publishes @ggml-org/llama-chat and its llama-chat
alias the way ui-lib-publish.yml publishes @ggml-org/llama-ui: the
version is set by hand, and a commit that reaches master with a version
npm does not have builds dist, checks the packed files and publishes both
with trusted publishing. PRs that change the packages get a dry run.
npm-version-check.mjs takes the package folder, so both workflows share
it. ui/README.md says how releases work, and Chat's AGENTS.md says not
to bump versions unless asked to release.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: call the app Llama Chat in its README
The README's title and a few lines still called the app llama-ui or
Llama UI, the name of the library it is built on. They now say Llama
Chat, and the intro points to Llama UI in ui/lib/ui.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat connects to llama-server's default port, 9931
The CLI's --server default and its examples used 8080, which is not
where llama-server listens. They now use 9931, its default port, so
npx llama-chat reaches a llama-server started without --port.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: a shorter Llama Chat README that matches the code
The README grew to nearly 900 lines, and its tables of components,
hooks, stores and services had drifted from the code. It now keeps what
a reader needs: running it with npx and self-hosting, a short feature
list, development commands, the architecture (the line between Llama UI
and the app, a table of the layers and where they live, a flowchart and
one sequence diagram for sending a message), a few patterns worth
knowing, cross-origin isolation, how the build is embedded in
llama-server, the project tree and publishing. The folders now carry the
detail the tables used to. ASCII replaces the unicode arrows.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat serves only static files, without the API proxy
The CLI forwarded every request it had no file for to llama-server, from
loopback. With --host 0.0.0.0 that let any LAN client reach a llama-server
started on 127.0.0.1 as a local request, tools included, which llama-server
on its own refuses without an API key. The browser now talks to each
llama-server directly, so llama-server's own rules apply. --server is gone,
and anything that is not a file in dist is a 404, or a 405 for other methods.
Assisted-by: Claude (Opus 5.5 Medium)
The deploy failed on master with a 403: HF_TOKEN_UI_STATIC_OUTPUT is
scoped to the UI bucket and cannot write ggml-org/llama-ui-storybook, so
hf upload tried to create the Space. The upload now uses
HF_TOKEN_UI_STORYBOOK when that secret exists, and the job runs on
hf-jobs-cpu-upgrade instead of a GitHub-hosted runner. That image gets
python3-venv from apt for the hf CLI, as the other HF Jobs workflows do.
Assisted-by: Claude (Opus 5.5 Medium)
This commit adds a lambda to check if a specified byte is not a valid
continuation byte in a multibyte UTF8 sequence.
The motivation for this is to improve readability and avoid some code
duplication.
* vocab : extract bpe_rank loading into function
This commit extracts the BPE rank loading logic into a separate function
named `load_bpe_ranks`.
The motivation for this is that the came code is currently used by both
gpt2/hybriddna/whitespace tokenizer models and gemma4 and this avoids
the duplication of code to improve readability and maintainability.
* change signature of load_bpe_ranks
* vulkan: expand bf16 support for unary and binary ops and rms_norm
* cpu: add bf16 support for scale, clamp, leaky_relu, swiglu_oai, swiglu_clamp, tri and diag
* fix sycl fill supports_op condition
* restore CUDA op order
* ui: generate DESIGN.md's token sections from the CSS
Add scripts/design-md.js in ui/apps/chat (npm run design:tokens, and npm
run check:design in CI), which writes the colors, typography and rounded
sections of DESIGN.md's front matter from llama-brand's colors.css and
Llama UI's primitives, semantic tokens and font stacks. The front matter
had drifted: it gave Matte's surfaces as the light base and system-ui for
every text role. Twins are now written wherever a token resolves
differently in dark mode (-dark) or in Matte (-matte).
Describe the floating panel's rim as the CSS draws it, on the border,
and say which parts of DESIGN.md are generated and which are kept by hand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say the Storybooks show accessibility findings without enforcing them
Both Storybooks keep a11y test 'todo': under 'error' 98 of the stories
fail, most on colour contrast of the muted foreground and on nested
interactive controls, which need design changes. Reword the Uses tile
and record the reason next to the setting.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list every Llama Chat group in the Introduction's Sections figure
The figure said it was in sidebar order but left out Forms, Actions,
Badges, Misc, Servers and PWA. Its list moves to sections.ts, and a unit
test checks it against storySort in .storybook/preview.ts, which has to
stay inline there because Storybook parses it as written.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load the package's Llama Mono in both Storybooks
The library's Storybook declared Llama Mono a second time from
tests/stories/brand/fonts, a 72276-byte file that differs from the
76896-byte one the package ships, so its pages showed a Llama Mono that
does not ship. Drop both Storybooks' copies: theme.css already loads the
package's font.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: take Llama Chat's shared Storybook helpers from Llama UI
Llama Chat kept copies of the Editorial, Chapters, StateGallery,
StateGalleryItem, svx.d.ts and docs.css that live with the library's
stories. Its stories now import them as $llama-ui-stories/..., an alias
its Storybook config and tsconfig.json map onto ui/lib/ui/tests/stories,
the way it already compiles the library from source; docs.css and
svx.d.ts are referenced by path. The library's Editorial takes Llama
Chat's newer list numbering. An empty svelte.config.js next to the
stories keeps Llama Chat's svelte-check from loading the library's
vite.config.ts, as src/svelte.config.js does for the sources.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the library's Storybook to Llama Chat's versions
ui/lib/ui used ^10.6.2 with @storybook/addon-mcp at 10.6.2, while
ui/apps/chat pins 10.6.6, so one tree had two Storybooks. Pin the
library's Storybook packages exactly to Llama Chat's: 10.6.6,
addon-svelte-csf 5.1.6 and @chromatic-com/storybook 5.5.1.
Assisted-by: Claude (Opus 5.5 Medium)
* ggml-openvino: fix SWA mask detection
When the SWA cache size matches the full cache size, identify SWA
layers by the number of layers using each mask instead of cache size;
the larger group is SWA, and a tie leaves the layers unclassified.
* ggml-openvino: fix SWA KV cache write indices
Give SWA KV cache index tensors unique input names (a _swa suffix, like
the masks) to prevent them from being merged with the full-attention
cache indices.
* ggml-openvino: classify SWA layers only when the mask shows the window
Two attention masks do not imply SWA: DeepSeek sparse attention uses two
full-attention masks with the same name. When cache extents match,
classify the larger mask group as SWA only once its mask has fewer keys
per row than the other mask. Keep the classification per KV cache buffer
so new sequences reuse the same graph, and include swa_layers when
checking graph reuse.
* ggml-openvino: classify SWA layers from the first graph
Waiting for the window to appear in the mask caused a mid-sequence graph
recompile (~9 s for Gemma-4 at -c 1024 after position 512).
SWA classification already gives each mask group its own mask, KV write
indices, and attention size, which is correct whether or not the mask is
a sliding window. Stateful mask rebuild recovers the window from the
mask, or the causal mask for full attention. Classify the larger of two
mask groups from the first graph and drop the per-KV-buffer bookkeeping.
* ggml-openvino: mark SWA fallback as a model-level heuristic
Add a TODO above the equal-extent fallback to document that the larger
mask group is currently assumed to be the SWA group. Since stateless
graphs only require the two groups to remain distinct, this heuristic
may only be necessary for stateful execution and could potentially be
restricted to that path.
* ggml-openvino: derive attention sizes from the KQ mask shape
Dynamic CPU/GPU graphs previously passed n_seq_active and
attention_size(_swa) as separate inputs. Derive these values directly
from the KQ mask shape using ShapeOf + Gather inside the model. This
allows the GPU plugin to infer the Q reshape and KV slice shapes from
shape inference rather than runtime input values. Q's head size also
becomes static, enabling the fused SDPA kernel.
GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous
separate-input behavior.
The fused SDPA path also requires two supporting changes:
- Clamp the GPU mask to -30000. A key block that is entirely -inf for
a row produces NaNs in the fused kernel.
- Restate the head count and head size after the KV slice. The plugin
loses all slice dimensions when the end index is dynamic, causing
SDPA to fall back to the reference kernel.
Additional graph and quantization improvements:
- Skip the identity reshape of rows in SET_ROWS.
- Share a single Split for one-element view slices in dynamic GPU
graphs (e.g. Gemma-4 per-layer embeddings), replacing one
StridedSlice per layer.
- Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve
the file's native quantization for token_embd/output instead of
requantizing to Q8_0_C.
- Include both environment settings in the graph and model cache keys
and document them.
* ggml-openvino: only reuse Split outputs for exact chunk-aligned views
The dynamic GPU path reused a Split output for any single-element view
by deriving the chunk index from offset / stride. This produced
incorrect results for views that were not chunk-aligned or whose
remaining dimensions did not preserve the source strides, as seen in
test-backend-ops CONT with use_view_slice.
Restrict Split reuse to cases where the view exactly matches a split
chunk, meaning the offset is aligned to the split stride and the
remaining non-unit dimensions preserve the original source strides.
Gemma-4 per-layer embedding slices continue to use the shared Split
path.
* ggml-openvino: initialize ModelExtraInputInfo shape fields
Initialize the newly added `shape_source` and `shape_axis` fields in
the `ModelExtraInputInfo` brace initializer used by
`add_extra_inputs()`.
Without initializers for these fields, GCC emits
`-Wmissing-field-initializers`, causing CI to fail with
`LLAMA_FATAL_WARNINGS`.
* ggml-openvino: remove GGML_OPENVINO_REQUANT_EMBD
The option identifies embedding and output weights by their tensor
names, which are defined by core llama.cpp. Remove it from this PR;
preserving the file quantization of these weights will be addressed in
a separate change that identifies them by their roles in the graph.
* ggml-openvino: keep manual GQA attention 4-D
The manual GQA attention path reshaped Q, K, and V to 5-D so that
query heads sharing a KV head could broadcast against it. The GPU
plugin cannot fuse this graph: it materializes the attention scores
and copies the transposed K and V caches for every layer.
Instead, stack the query heads for each KV head along the sequence
axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask
accordingly. This keeps the graph 4-D from the MatMuls through the
Softmax, allowing the GPU plugin to fuse the attention into its SDPA
kernel and fold the K and V transposes into the fused operation.
Clamp the mask to -30000, matching the SDPA path, because the fused
kernel can return NaN when an entire block is masked with -inf.
Models where the K or V head size differs from the Q head size now
use the SDPA path, since the manual path assumes one head size.
* ggml-openvino: use a 4-D stateless KV cache on GPU
In stateless GPU mode, bind the entire KV cache as a
`[1, ctx, n_heads_kv, head_size]` tensor, with one token per row,
instead of binding the first `n_kv` rows as a
`[1, 1, ctx, n_heads_kv * head_size]` tensor.
Write cache updates with `ScatterUpdate` along the token axis, and
read the cache through `VariadicSplit` and `Transpose`. This allows
the GPU plugin's `StatelessKVFusion` to replace cache reads and writes
with in-place `StatelessKV` nodes, avoiding full-cache copies on every
decode step and repeated memory pool allocations as the bound cache
size grows.
Set `GGML_OPENVINO_DISABLE_KV_4D=1` to retain the previous layout.
* make embedding to have dynamic shape for OV GPU
* ggml-openvino: make the 4-D stateless KV cache opt-in
* add support for Q5_0 data type in OV GPU backend
* resolve CI test error
* ggml-openvino: fix -fa off and optimize grouped-query attention
With -fa off, llama.cpp stores the V cache transposed, with value
elements as rows and tokens along each row. The KV slicing code assumes
one token per row and binds the wrong data when n_kv < kv_size. Detect
transposed caches with GgmlOvDecoder::is_kvcache_token_rows() and skip
slicing for them.
For grouped-query attention, avoid repeating K and V for each query
head. Stack the query heads for each KV head along the row dimension
and let SOFT_MAX apply the scale, mask, and softmax so the GPU plugin
can fuse the operation with the MatMuls.
Read only the current layer's V-cache rows when layers use different
cache sizes. Reject -fa off in stateful mode with a clear error and
document that it requires stateless mode.
* ggml-openvino: fall back to stateless execution when -fa off
With `-fa off`, the graph writes the V cache in transposed form, which is
not supported by the stateful KV cache. Detect this pattern in
`compute_llm_params()` as a `SET_ROWS` into a KV cache through a `[1, n]`
reshape.
When stateful execution is enabled, log a warning and fall back to
stateless execution instead of failing graph conversion.
Update the `-fa off` note in `OPENVINO.md` to document this behavior.
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
* ggml-openvino: preserve flat KV cache layout when -fa is off
With -fa off, the 4-D KV cache layout introduced in #344 breaks MUL_MAT
attention for Phi-3.5 mini and Phi-4 mini, causing oneDNN to fail with
"could not construct a memory descriptor using strides".
Preserve the [1, 1, ctx, n_heads_kv * head_size] layout when the graph
writes a transposed V cache.
* ggml-openvino: extract Q5_0 weights when requantization is off
Q5_0 weights are requantized to Q4_0, but test-backend-ops builds naive
graphs, where requantization is skipped. Q5_0 then reached the normal
weight extraction, which had no Q5_0 case, so every Q5_0 MUL_MAT and
GET_ROWS op test failed with "Unsupported quantized type: q5_0".
Extract Q5_0 as symmetric i8 weights (q - 16) with one f16 scale per 32
weights, the same layout as Q8_0. Models are unchanged: they still
requantize Q5_0 to Q4_0.
* ggml-openvino: Optimize GDN output pre-scale for the NPU static path
The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero.
* Refine the code as review comments
1. Add envs desc into doc
2. Refine the code with removing duplicated env
* ggml-openvino: reuse graph bindings between decode calls
Reduce host overhead outside `infer()` from 0.5–1.1 ms to 0.14–0.38 ms
per token on Panther Lake.
- Reuse KV cache tensors and skip redundant port bindings.
- Use persistent USM host tensors for small GPU inputs and outputs.
- Defer graph-wide consumer lookup until after shape checks.
- Return model and compute params by const reference.
- Preserve tensor extras when memory, shape and type are unchanged.
* ggml-openvino: Add npuw dump envs for profiling
* Add the comments of new npu dump envs into doc
* ggml-openvino : ensure all leaves have unique names, remove input_srcs from graph key
* openvino: slice NPU prefill KV cache by active length
Add GGML_OPENVINO_NPU_KV_SLICE as an opt-in NPU prefill optimization. When enabled, static prefill keeps the graph's attention dimension at the active prompt/KV length instead of expanding it to the full context capacity, and the KV-cache input/output tensors are bound with a shortened context axis over the same backing storage.
This reduces the amount of KV cache and attention-mask data imported by the NPU plugin and lets the prefill graph plan smaller attention buffers for early prompt chunks. For example, with a 4096-token context and a 256-token prefill chunk, the first chunk can expose KV/mask dimensions at 256 instead of 4096, avoiding work over padded cache rows.
The trade-off is graph stability: NPU static shapes now depend on the aligned active attention length for prefill, so a cached prefill model is reused only when the attention sizes match. Decode keeps a fixed full-context graph to avoid recompiling on every generated token, and SWA/ring-cache cases still fall back to full-size binding when the live prefix is not a simple leading slice.
* openvino: use L0 host tensors for NPU buffers
Add GGML_OPENVINO_NPU_L0_HOST_TENSORS as an opt-in NPU buffer allocation mode. When enabled, non-remote OpenVINO backend buffers on NPU are allocated through the NPU default context with create_host_tensor(), and ggml tensor data points directly at that OpenVINO-owned Level Zero host allocation.
The optimization targets the data path rather than the graph math: KV-cache and compute tensors can be passed back to the NPU plugin as OpenVINO tensors backed by importable Level Zero host memory. That avoids an extra ordinary-host-memory-to-plugin-staging copy on each infer boundary, which is especially useful for prefill where large KV/cache buffers dominate input and output traffic.
If Level Zero host allocation fails, the backend logs a warning and falls back to the existing allocation path. The destructor tracks OpenVINO-owned storage so it does not free memory owned by the host tensor. The trade-off is that this is NPU-specific, consumes Level Zero host allocation resources, and improves transfer/import overhead without changing attention compute complexity or tensor shapes.
* openvino: fill NPU prefill masks directly
Add GGML_OPENVINO_NPU_FAST_MASK as an opt-in NPU static prefill input-packing optimization. When enabled, the prefill attention mask is built directly in the OpenVINO input tensor instead of first materializing a padded std::vector and then copying that whole buffer into the tensor.
The new fill_prefill_mask() helper writes each row once: it copies the valid mask prefix, fills padding columns with -inf, handles padded rows, and restores the causal diagonal to zero. This keeps the final mask numerically identical to the existing pad_input() plus set_zero_diagonal() path while removing one intermediate allocation and one chunk_size-by-context_size host copy per prefill chunk.
The optimization targets CPU-side staging overhead, not attention math. It composes with GGML_OPENVINO_NPU_KV_SLICE: KV slice reduces the mask dimensions, while FAST_MASK reduces the work needed to construct whatever mask shape remains. The trade-off is a second mask construction path to maintain, with limited benefit for very small prompts or contexts.
* openvino: make NPU requant layout configurable
Add GGML_OPENVINO_NPU_REQUANT_POLICY as the NPU-side selector for the default weight requantization layout. The existing NPU behavior remains the default: unset or group-128 maps quantized weights to Q4_0_128, while channel-wise selects the experimental Q4_0_C layout. Unknown policy values fail fast during model loading instead of silently compiling a graph with unintended weight packing.
The optimization target is NPU weight bandwidth and compressed-weight code generation. Q4_0_128 keeps weights in a 4-bit symmetric layout with one scale per 128 values, reducing both payload bytes and scale metadata versus smaller groups while preserving a more conservative quantization granularity than channel-wise. Q4_0_C uses one scale over the innermost dimension, which can further shrink metadata and simplify dequant broadcast, but may cost accuracy because the quantization block is much larger.
Include the selected NPU requant policy in the OpenVINO model cache discriminator. The same cgraph compiled with group-128 and channel-wise has different weight constants and dequant shapes, so reusing a cached decoder across policies would be incorrect. This keeps cache reuse stable within a policy while forcing recompilation when the layout policy changes.
Trade-off: policy selection gives a performance/accuracy knob for NPU experiments, but all options still pay the load/compile-time requantization cost. The default remains group-128 for a safer balance of bandwidth, metadata, and accuracy; channel-wise is intentionally documented as experimental.
* openvino: centralize NPU plugin config overrides
Add GGML_OPENVINO_NPU_CONFIG as the recommended catch-all override for OpenVINO NPU plugin and compiler properties. The value is parsed as comma-separated KEY=VALUE pairs and applied after the built-in defaults and named NPU aliases, so advanced users can override any exposed property without adding a new llama.cpp environment variable for every plugin tuning knob.
Keep the common knobs as named aliases for discoverability and compatibility: GGML_OPENVINO_NPU_COMPILER_TYPE maps to NPU_COMPILER_TYPE, GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL maps to NPUW_FUNCALL_FOR_ALL, GGML_OPENVINO_NPUW_UNFOLD_IREQS maps to NPUW_UNFOLD_IREQS, and GGML_OPENVINO_COMPILATION_NUM_THREADS maps to COMPILATION_NUM_THREADS. GGML_OPENVINO_NPU_COMPILE_CONFIG is retained as a legacy alias for NPU_COMPILATION_MODE_PARAMS, while the documentation now recommends expressing it through GGML_OPENVINO_NPU_CONFIG.
This keeps the public tuning surface compact while preserving high-signal aliases for settings with known performance or stability implications. NPU_COMPILER_TYPE can select the driver compiler, which currently generates faster prefill kernels on the tested NPU stack. NPUW_FUNCALL_FOR_ALL and NPUW_UNFOLD_IREQS expose NPUW graph lowering trade-offs around dispatch overhead, memory use, and long-context stability. COMPILATION_NUM_THREADS gives a way to cap compiler worker parallelism to reduce peak host memory during large graph compilation.
Trade-off: a free-form KEY=VALUE string is less type-safe than one env var per option and invalid plugin properties will only be diagnosed by the OpenVINO stack. Applying it last is intentional: it gives one escape hatch for bisecting driver/compiler behavior and for testing new plugin options without rebuilding llama.cpp, while the named aliases continue to document the most important performance knobs.
* openvino: add scatter-elements KV update option
Add GGML_OPENVINO_KV_SCATTER_ELEMENTS as an opt-in lowering for non-stateful single-row KV cache writes. When the flag is enabled and the set_rows indices are the simple decode case, translate_set_rows uses ScatterElementsUpdate with broadcasted rank-4 indices instead of ScatterUpdate.
The motivation is NPU decode performance. The default ScatterUpdate path updates one KV row but the NPU plugin does not perform that update in place; it effectively copies the full destination tensor to write a small slice. As context length grows, that full-tensor movement dominates the per-token KV update cost. ScatterElementsUpdate expresses the same single-row write at element granularity and is measurably cheaper for this shape.
Keep the optimization guarded. Stateful execution keeps its existing concat/update path, and multidimensional indices already use the broader ScatterElementsUpdate lowering. The new flag only changes the common non-stateful decode write, so the default behavior and graph shape remain unchanged unless the user opts in.
Trade-off: ScatterElementsUpdate introduces extra ShapeOf, Reshape, and Broadcast nodes to build indices matching the update tensor. That overhead is worthwhile only when it avoids the NPU plugin's full-destination ScatterUpdate copy, so the path is controlled by an environment variable rather than becoming the unconditional lowering.
* openvino: gather token embeddings from compressed rows
Add NPU token-embedding controls for models whose token_embd.weight would otherwise be widened from Q6_K to FP16. GGML_OPENVINO_TOKEN_EMBD_I8 keeps the embedding table on the normal int8 compressed-weight path, while GGML_OPENVINO_TOKEN_EMBD_I4 requantizes it to the group-128 int4 layout used by the NPU weight path.
The performance target is tied embeddings and large vocabulary tables. When token_embd.weight is also consumed by the lm_head matmul, a normal Gather from the dequantized table gives the dequantization subgraph a second consumer and can force the full table to be materialized. The new gather_compressed_rows helper recognizes the dequantization chain emitted for int4/int8 constants, gathers only the selected packed rows from the leaf constants, and rebuilds the convert/subtract/multiply/reshape chain on those rows.
This reduces memory pressure and avoids dequantizing or materializing a full embedding table when inference only needs the token rows for the current input. The helper is deliberately conservative: it only rewrites static 2D tables above a size threshold, walks only the known compressed-weight node patterns, limits recursion depth, and falls back to the original Gather when the subgraph is not recognized.
Trade-off: int8 and especially int4 token embeddings can change accuracy and may not be faster for small untied embedding tables, because the rewrite adds extra Gather/Reshape nodes and only pays off when it avoids a large full-table dequantization. The default behavior is unchanged; the NPU-specific I8/I4 paths are opt-in through environment variables.
* openvino: stream u4 requantization chunks
Extend the GGML_OPENVINO_REDUCE_COMPILE_MEM streaming requantization path to Q4_0/u4 targets. The previous streaming path intentionally skipped u4 because Q4_0 packs two weights per byte and the zero-point tensor stores two 4-bit values per byte, so the original quantize_q4_0 helper wrote as if every call started at block zero.
Teach quantize_q4_0 to accept a destination block offset and use the absolute block index for weights, scales, and zero-point writes. Streaming requantization emits whole-row chunks in increasing order, and the target block size is required to divide the row width, so no target block straddles a chunk boundary. That keeps packed weight bytes aligned and preserves the even-block assignment before the odd-block zero-point OR for each shared zero-point byte.
With that indexing in place, reduce-compile-memory mode can dequantize a chunk of rows into the scratch buffer and immediately quantize it into the final u4 buffers instead of materializing the entire tensor as temporary F32. This extends the peak-memory reduction already used by Q8/F16 targets to the NPU group-128/channel u4 layouts used for compressed weights.
Trade-off: the u4 path is more sensitive to block alignment than Q8/F16 because of nibble packing, so streaming remains guarded by the same row-divisibility check and only runs when GGML_OPENVINO_REDUCE_COMPILE_MEM is enabled. When the flag is off, the full-materialization path and quantized output layout remain unchanged.
* openvino: fail fast on static recompile after weight release
Extend the GGML_OPENVINO_RELEASE_WEIGHTS safety check to the static graph cache-miss path. Once host weight buffers have been released, any later compile would read invalid/zeroed host pages and bake the wrong constants into a new OpenVINO model.
The dynamic path already rejected this situation. Static execution can still miss the decoder cache when graph shapes or model parameters change, so it needs the same guard before erasing cached infer requests and rebuilding prefill/decode models.
This is a correctness and diagnosability change rather than a throughput optimization. It preserves the memory-saving release-weights mode for stable graph shapes, but turns unsupported recompilation into an explicit abort instead of silently producing a corrupted compiled model.
* openvino: add experimental NPU Q4_0 keep option
Add GGML_OPENVINO_NPU_KEEP_Q4_0 as an opt-in escape hatch for NPU weight handling. When enabled for native Q4_0 tensors, the NPU requant path returns nullopt and keeps the original Q4_0 extraction instead of regrouping the tensor through the selected NPU requant policy.
The motivation is to avoid unnecessary work for weights that are already stored as 4-bit values. The default NPU policy maps quantized weights to layouts such as Q4_0_128, which can reduce scale metadata but requires round-tripping Q4_0 through the requantization path. Keeping Q4_0 native can save that conversion work and preserve the original group-32 packing.
Keep the option experimental and disabled by default. Current NPU driver/plugin stacks can compile the resulting group-32 graph, but inference may hang or fail with ZE_RESULT_ERROR_DEVICE_LOST. The default behavior therefore continues to use the safer NPU requant policy; this flag is mainly useful for debugging driver behavior, measuring the true cost of regrouping, and testing future NPU stacks.
Trade-off: native Q4_0 avoids regrouping overhead and keeps the original quantization blocks, but it gives up the more regular group-128 layout expected by the default NPU compressed-weight path and may be unstable on current hardware/software combinations.
* openvino: unfold NPUW only for decode graph
Compile the static prefill and decode models with separate OpenVINO config maps so the decode graph can opt into NPUW_UNFOLD_IREQS without forcing the same lowering on prefill. The build_static_model helper now receives the config map explicitly, and the decode path gets a copy of the base config with NPUW_UNFOLD_IREQS=YES when the user has not already set it.
The performance target is NPU token generation. Decode executes a long stream of single-token infers, where folded NPUW function-call dispatch overhead becomes visible in every token. Unfolding the NPUW calls into separate infer requests removes that repeated dispatch cost and measured about +20% token-generation throughput on the tested stack, matching the behavior seen in OpenVINO GenAI.
Prefill keeps the original folded config. Prompt processing is a batched workload dominated by larger matmuls, and the folded form is faster there while also using less memory. Splitting the config lets prefill and decode use the lowering that matches their workload shape instead of treating NPUW_UNFOLD_IREQS as a single global compromise.
Trade-off: unfolding can increase memory and request-management overhead, so the automatic default is limited to the NPU decode model. If a user explicitly provides NPUW_UNFOLD_IREQS through the named env alias or GGML_OPENVINO_NPU_CONFIG, that value is respected because the decode override is only applied when the property is absent.
* openvino: tidy NPU tuning helper ordering
Collect the remaining small cleanups from the NPU tuning series. This reorders the NPU KV-slice documentation and environment-variable cache entry next to the other NPU opt-in toggles, adds a short header comment for the centralized NPU helper declarations, and makes the static prefill attention-size assignment in the decoder a little more explicit.
There is no intended behavior change in this commit. The purpose is to make the final source layout match the split commits that introduced the NPU tuning knobs, so future readers can find the related flags and helpers in one place without mixing that mechanical cleanup into the functional performance commits.
* openvino: normalize NPU compile env overrides
Move the default NPU_COMPILER_TYPE=DRIVER setting into the NPU compile_config initializer so the default NPU plugin options are declared in one place.
Route GGML_OPENVINO_NPU_COMPILE_CONFIG and GGML_OPENVINO_NPU_COMPILER_TYPE through the same set_compile_option_from_env helper used by the other named NPU compile option aliases. This keeps the legacy NPU_COMPILE_CONFIG mapping to NPU_COMPILATION_MODE_PARAMS intact while making all optional env-to-compile_config overrides follow the same non-empty-value rule.
GGML_OPENVINO_NPU_CONFIG is still parsed last, so explicit comma-separated KEY=VALUE overrides continue to have the highest precedence over both defaults and named aliases.
* openvino: stream NPU weight requant by default to cut compile peak
Default the streaming requantization path on for NPU so the compile-time F32 dequant transient (~1-2 GB for token_embd) is capped without an env flag. Output is byte-identical to full materialization; GGML_OPENVINO_REDUCE_COMPILE_MEM / MEMORY_OPTIMIZE still override (set to 0 to force off). Kept the g_nonov_weight_cache opt-in to avoid holding token_embd resident (which regressed decode steady memory). pp1024_d0: PrefillPeakPrivWS 5974->4664 MiB (-22%), decode steady and pp/tg throughput unchanged.
* openvino: add self-contained NPU model cache
Add an opt-in embedded-weight cache mode for the static NPU backend. GGML_OPENVINO_SELF_CONTAINED_BLOB=1 exports separate prefill and decode compiled models with ENABLE_WEIGHTLESS=false, uses isolated NPUW weight banks, validates raw GGUF weight manifests, binds imported ports by deterministic frontend order, and serializes the large NPUW compile/export/import operations.
Preserve raw model bytes during cold cache construction by requantizing into separately owned OV tensors instead of rewriting the backend model buffer. This keeps later graph fingerprints and manifests stable when llama-bench creates independent prompt and generation contexts.
Add strict GGML_OPENVINO_SELF_CONTAINED_MMAP=1 warm-import mode. The backend exposes a non-owning buffer_from_host_ptr wrapper over the GGUF mmap, uses raw tensor allocation sizes, avoids allocating/copying the private OpenVINO model-weight buffer, and stages the few mmap-backed runtime constants into small importable NPU host tensors. Invalid or missing cache entries fail clearly instead of trying to requantize into read-only mapped storage.
The normal path and existing weightless MODEL_PTR cache remain unchanged when the new flags are unset. Verified with phi-4-mini: chunk512 pp3072/tg512 cold 1123.10/27.60 t/s and warm mmap 1124.72/27.16 t/s; warm mmap removes roughly 3.9 GiB peak commit in the measured prefill workload. Self-contained blobs consume about 8.34 GiB for the separate pp and tg contexts.
* openvino: add lazy import for self-contained mmap cache
Add GGML_OPENVINO_SELF_CONTAINED_LAZY_IMPORT to make the low-working-set strict mmap path opt in instead of changing the default warm mmap behavior.
When the flag is set with SELF_CONTAINED_BLOB=1 and SELF_CONTAINED_MMAP=1, the static NPU path imports only the current phase blob: prefill imports the prefill blob, decode imports the decode blob later. This avoids the pair-shaped shared compiled-model cache for the lazy path and caches only the current phase infer request.
The regular warm mmap path remains eager and continues to prepare both prefill and decode compiled models. Non-mmap self-contained builds remain serial, and the regular non-self-contained path keeps the existing parallel build behavior.
Also register the new environment variable, document the memory/latency tradeoff, and add profiling-only import timing around model-cache imports.
Validated with ReleaseOV build: cmake --build build/ReleaseOV --target llama-bench llama-simple --config Release. Lazy warm mmap run measured PrefillPeakWS around 5759 MiB versus the eager warm mmap path around 7897 MiB.
* openvino: release GGUF mmap resident pages before blob import
Add GGML_OPENVINO_SELF_CONTAINED_RELEASE_MMAP_PAGES as an explicit opt-in for self-contained compiled-model cache hits.
Before importing a valid self-contained blob, recursively walk the graph to find unique external GGUF mmap-backed weight buffers and drop their resident pages while preserving the mapping address and tensor pointers.
On Linux use page-aligned madvise(MADV_DONTNEED). On Windows, where there is no reliable per-range mapped-file equivalent, call VirtualUnlock for the identified range and trim the current process working set with SetProcessWorkingSetSize(-1, -1). Pages required later can fault back on demand.
Keep the behavior disabled by default and document the physical-memory versus commit-memory tradeoff. Profiling reports the submitted GGUF range and model import duration.
Validated with ReleaseOV llama-bench/llama-simple build and pp3072+tg512 warm mmap lazy import. With page release enabled, PrefillPeakWS dropped from about 5758 MiB to 3053 MiB while throughput remained comparable; PeakCommit stayed about 3331 MiB.
* openvino: add GGML_OPENVINO_NPU_SHARED_BANK to share prefill/decode weights bank
When set, prefill and decode self-contained blobs use one shared NPUW_WEIGHTS_BANK name (ggml-sc-shared-<pfp>-<dfp>) instead of separate per-phase banks. Since both decoders are built from the same model_weights map, NPUW dedups them into a single in-process bank; on warm import the identical (uid,tensor) pairs are content-verified and reused (patched NPUW weights_bank). Cuts resident compiled-weight footprint ~2GB (pp3072 warm: In-Use 7954->5871 MiB, WS 7891->5765) with prefill throughput unchanged.
* openvino: drop SELF_CONTAINED_LAZY_IMPORT and RELEASE_MMAP_PAGES
The shared weights bank (GGML_OPENVINO_NPU_SHARED_BANK) supersedes both: it deduplicates the prefill/decode compiled-weight copy directly, so lazy per-phase import and GGUF-page release no longer reduce steady memory. Remove the two env vars, the self-contained lazy-import branches (defaulting to the eager path that imports both blobs), the release_external_weight_pages helper, and the now-unused ggml_openvino_buffer_is_external / _release_external_pages buffer APIs.
* openvino: demand-page large NPU compute buffer in strict warm-mmap mode
In strict self-contained warm-mmap mode weights come via buffer_from_host_ptr,
so every alloc_buffer is graph scratch/KV. The large prefill compute buffer is
never touched on the host by the on-device fused OV model, yet the NPU L0 host
tensor (create_host_tensor + memset) makes all ~2.4 GiB resident. For buffers
>= 1 GiB in this mode, reserve demand-zero pages (no L0, no memset) so they stay
non-resident; KV/small buffers keep the L0 host tensor for zero-copy import.
pp3072 warm mmap + shared bank: prefill peak WS 5765 -> 3314 MiB (-43%),
In-Use delta ~5940 -> ~3232 MiB; prefill 1256.7 -> 1252.9 t/s and decode
28.27 -> 27.96 t/s (both flat). Commit charge unchanged (demand-paged).
* perf improvement success
* openvino: size static NPU sliding-window masks to the window cache
The static graph built the sliding-window KQ mask at the full context length,
while the windowed layers bind their (smaller) window KV cache. With gemma-4 at
-c 4096 -ub 1024 the window cache is 1536 cells, so SDPA shape inference failed
([1,1,1024,4096] mask vs [1,8,1536,256] K/V) and every prompt errored out.
With GGML_OPENVINO_NPU_KV_SLICE the prefill mask was meant to use
attention_size_swa, but the check matched "swa" in the ggml tensor name, and
llama.cpp names both masks "attn_inp_kq_mask"; the windowed layers got the
full-attention sliced length instead ([1,1,1024,256] vs [1,8,1024,256]).
Identify the sliding-window mask by tensor identity (is_swa_mask) and size it to
ctx_per_seq_swa, both in the graph input shape and in the runtime mask padding.
* ggml-openvino: create the model cache dir under a Windows drive root
make_dirs() mkdir'ed every path prefix, including the drive root "D:", which
fails with EACCES (not EEXIST) on Windows. Any absolute cache path therefore
logged "could not create model cache dir ... caching disabled" and silently
ran without GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. Skip the drive root.
* ggml-openvino: split device-specific passes and NPU config out of shared code
Mechanical refactor, no behavior change (step 1 of the NPU/MoE architecture plan):
* openvino/pipelines/: TranslateSession::apply_transformations() now registers
common passes, then the device's passes, then the execution-mode passes
(stateful KV, static shapes), selected by a per-graph GraphTraits. The GPU MoE
fusions and the GPU eltwise-rank workaround move to pipelines/gpu.cpp; CPU and
NPU pipelines are empty placeholders for device lowerings. Pass order is
unchanged.
* npu/npu_config.*: the NPU (NPUW) compile config built at backend init, the
NPUW strip for cacheless graphs and the per-graph static prefill/decode
variants (HFA prefill, decode NPUW_UNFOLD_IREQS) move out of
ggml-openvino-extra.cpp and utils.cpp.
* npu/npu_quant_policy.*: the NPU requantization rules (requant policy,
token_embd Q6_K, keep-Q4_0) move out of ggml_openvino_get_requant_type().
Verified identical output before/after on OV-CPU, GPU and NPU (gemma-4 E4B).
* ggml-openvino: one KV-slice switch for dynamic and static graphs
KV-cache slicing (bind the KV cache, and size the KQ masks, at the attended
length instead of the full context) existed twice: on by default for dynamic
graphs (GGML_OPENVINO_DISABLE_KV_SLICE to turn it off) and opt-in for static
prefill (GGML_OPENVINO_NPU_KV_SLICE). The static mode is not NPU-specific; it
applies to any static graph, including CPU/GPU with GGML_OPENVINO_FORCE_STATIC.
Describe it as one mechanism with two modes behind GGML_OPENVINO_KV_SLICE
(unset = each mode's default, 0 = off, 1 = on) and two predicates,
ggml_openvino_{dynamic,static}_kv_slice_enabled(), used by the KV binding,
the static graph shapes, the mask padding and the cache keys. The two old
variables remain as deprecated aliases.
* ggml-openvino: initialize all ModelExtraInputInfo fields for the HFA inputs
dev_backend_openvino added shape_source/shape_axis to ModelExtraInputInfo; the HFA past-KV and mask
initializers still listed four fields, which GCC rejects under -Werror=missing-field-initializers.
* ggml-openvino: make new NPU defaults opt-in and fix static model conversion
- Convert the static prefill and decode models one after the other, both
before either is compiled. The decoders share the weight nodes;
converting on two threads crashed intermittently inside libopenvino,
and converting decode after the prefill compile gave wrong output on
Bonsai-8B.
- Compile prefill and decode in parallel only with
GGML_OPENVINO_NPU_PARALLEL_COMPILE=1, since it raises the host memory
peak.
- Unfold NPUW function calls on the decode graph only with
GGML_OPENVINO_NPU_DECODE_UNFOLD=1. The unfold default gave wrong output
on gemma-4-E4B and slowed decode on several small models.
- Stream NPU weight requantization only with
GGML_OPENVINO_REDUCE_COMPILE_MEM or GGML_OPENVINO_MEMORY_OPTIMIZE, as on
the other devices.
- Fix a -Werror=missing-field-initializers build error in utils.cpp.
- Declare ggml_openvino_get_npu_requant_type() only in
npu/npu_quant_policy.h.
- Docs: document the new switches, deprecate stateful execution, show
stateless-only results in the validated models table, and update the
validation system.
* ggml-openvino: honour the -1 skip id in MUL_MAT_ID
Upstream made an id of -1 in MUL_MAT_ID skip the slot and zero its row of
the result. The translator passed -1 straight to GatherMatmul or Gather,
so 169 of the 743 MUL_MAT_ID op tests on CPU (all skip=1) failed.
A skipped slot now looks up a valid expert, and the result is multiplied
by a 0/1 mask of the kept slots. For the packed MXFP4 path (Gather and
MatMul), clamping the id to expert 0 is enough. GatherMatmul computes a
wrong row when a token names the same expert twice, so the general path
gives each skipped slot an expert that no other slot of the token uses.
TopK indices are never -1, and model graphs take the ids from the
router's TopK. Those graphs skip the handling and stay unchanged, so the
GPU MoE fusion still matches them.
* ggml-openvino: Gather rows before dequantizing a quantized embedding table
1. Optimize gather to fetch row before dequant
2. add NPU token_embd quantization option
* ggml-openvino: select the NPU token_embd type with GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT
GET_ROWS had two rewrites that gather the rows of a quantized table before
dequantizing them: gather_compressed_rows, which runs on every device, and
gather_before_dequant, enabled on NPU by GGML_OPENVINO_GATHER_BEFORE_DEQUANT.
Both match the int4 and int8 chains built by make_int4_weights() and
make_int8_weights(). Keep gather_compressed_rows and drop the second rewrite
and its switch.
The NPU type of a Q6_K token_embd.weight was set by two booleans,
GGML_OPENVINO_TOKEN_EMBD_I8 and GGML_OPENVINO_TOKEN_EMBD_I4, and by
GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT. Keep only the string variable, with the
values f16 (default), q8_0_c, q4_0_128 and q4_0_c, so the int8 and group-128
int4 settings stay available. An unknown value aborts, as for
GGML_OPENVINO_NPU_REQUANT_POLICY, instead of silently using f16. The variable
replaces the two booleans in the NPU compiled-model cache key.
* align PAD pads to input rank in stateful mode
* Gate the changes under is_stateful check to address review comment
* ggml-openvino: lift Q tensors for non-RoPE layers to rank 4 in stateful mode
* ggml-openvino: unify KV slicing under GGML_OPENVINO_KV_SLICE, on by default
Replace ggml_openvino_dynamic_kv_slice_enabled() and ggml_openvino_static_kv_slice_enabled() with a single ggml_openvino_kv_slice_enabled(). KV slicing binds the KV cache, and sizes the KQ masks, at the attended length instead of the full context.
GGML_OPENVINO_KV_SLICE now defaults to 1 on CPU, GPU and NPU; set it to 0 to disable. Dynamic graphs (CPU/GPU) bind a view over the cache without recompiling. Static graphs (NPU or GGML_OPENVINO_FORCE_STATIC) compile prefill at the aligned attended length and recompile when it changes; decode keeps its full-context graph.
Remove the deprecated aliases GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_NPU_KV_SLICE from the env var list, the model-cache settings, docs/backend/OPENVINO.md and the backend README.
Behavior change: static KV slicing is now on by default on NPU, so prefill compiles one graph per distinct n_kv (multiples of 256).
* ggml-openvino: keep stateful execution with -fa off
Keep the V cache token-major in stateful mode instead of falling back to
stateless execution. SET_ROWS appends to the state, and PERMUTE reads it
back in the layout attention needs, so MUL_MAT does not transpose it.
Dropping a reshape in grouped attention lets OpenVINO fuse it into SDPA.
For padded V caches, slice after the transpose so the cache is read in
place, without a copy that is recompiled for every KV length.
This removes the stateless fallback for -fa off.
* ggml-openvino: Add the stateful check to make the gate safer
* ggml-openvino: support gemma-2 soft-capped attention
Gemma-2 caps its attention scores as softcap * tanh(scores / softcap)
before the softmax. FLASH_ATTN_EXT now applies the cap on the manual
attention path, as SDPA has no soft-capping, and the decoder looks
through the SCALE, TANH, SCALE chain of -fa off when it matches
attention.
One helper finds the K cache of an attention for both the
sliding-window pre-pass and the main pass. The pre-pass copy did not
see through the chain, so the sliding-window layers used the
full-attention sizes once the context exceeded n_swa + n_ubatch.
Soft-capped attention keeps the flat KV cache layout: with the 4-D
layout the GPU plugin fails in its oneDNN path.
* ggml-openvino: enable NPU Level Zero host tensors by default
Back NPU buffers with Level Zero host memory unless GGML_OPENVINO_NPU_L0_HOST_TENSORS=0.
The plugin imports these buffers without a host-to-device copy each infer; on phi-4-mini
Q4_0 (pp3072+tg512) decode goes from 7.2 to 10.6-10.8 t/s. Allocation failures still fall
back to plain host memory.
* ggml-openvino: enable NPU fast prefill mask by default
GGML_OPENVINO_NPU_FAST_MASK now defaults to 1; set it to 0 to fall back to pad_input() + set_zero_diagonal().
The fast path builds each static prefill mask chunk directly in the OpenVINO input tensor (row memcpy of the valid prefix, -inf padding, zero diagonal), avoiding a temporary chunk_size x context_size vector and a full-tensor copy per chunk. The result is bit-identical to the fallback path.
* ggml-openvino: add opt-in exact zero points for asymmetric weights
Asymmetric quantized weights (Q4_K, Q4_1, Q5_1, and non-requantized
Q5_K) represent each group as `w = d * q - m`. The backend currently
converts these to u4/u8 weights with an f16 scale and an integer zero
point `round(m / d)`, which can shift all weights in a group by up to
half a quantization step.
Add `GGML_OPENVINO_EXACT_ZERO_POINT=1` to preserve the zero point in f16
instead, using the same path already used for MoE expert weights.
This is disabled by default because, with current OpenVINO releases, an
f16 zero point moves many GPU fully-connected layers off oneDNN,
resulting in slower prompt processing.
* ggml-openvino: rename npu/ to device/ and openvino/pipelines/ to openvino/pass_registry/
npu/ holds device-specific backend behaviour (NPU compile config and
requant policy), and pipelines/ only registers passes per device; the new
names describe what the folders contain. No functional change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs: document NPU self-contained model cache and shared weights bank
Add GGML_OPENVINO_SELF_CONTAINED_BLOB, GGML_OPENVINO_SELF_CONTAINED_MMAP and GGML_OPENVINO_NPU_SHARED_BANK to the OpenVINO environment variable table, and a note describing the export/import workflow and the requirement to keep NPU settings identical between runs.
* ggml-openvino: remove experimental GGML_OPENVINO_NPU_KEEP_Q4_0
Keeping native Q4_0 (group-32) on NPU compiles but hangs the device at inference (ZE_RESULT_ERROR_DEVICE_LOST), so the option was never usable. NPU Q4_0 weights now always go through GGML_OPENVINO_NPU_REQUANT_POLICY. Drop the env var, the requant bypass, its self-contained blob fingerprint term and the docs entry.
* ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares
All changes of https://github.com/ravi9/llama.cpp/pull/358 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.
Commits of the PR:
- Update fix to just maintain fp32 precision instead of modifying the graph
- Limit the fp32 precision spreading in the graph by fencing it with Convert. Prevents perf degrade.
- Update OPENVINO.md
- Update flash_attn_ext.cpp
- Update utils.cpp
* enable flash_attn with sink input
* ggml-openvino: fix FLASH_ATTN_EXT input detection and sink scale
translate_flash_attn_ext read get_input_names()[3] and [4] on every
FLASH_ATTN_EXT node, also on nodes with only 3 or 4 inputs. The
out-of-bounds read crashed test-thread-safety and
test-speculative-rejection-sampling. The inputs are q, k, v, then the
optional mask and sinks, and supports_op rejects sinks without a mask,
so the input count tells which inputs are present.
With sinks, the scale was replaced with 1.0. llama.cpp passes the
attention scale in op_params[0] and adds the sinks separately, so keep
op_params[0]. The "scale" input branch is removed: ggml's src[4] is
only ever the sinks.
Only SDPA applies the sinks. The manual GQA path and the NPU HFA
prefill path drop them, so attention with sinks now always takes the
SDPA path. SDPA has no logit soft-capping, so supports_op rejects sinks
together with soft-capping.
* ggml-openvino: accept flash attention on layers without RoPE
On layers without RoPE (SmolLM3: every 4th layer), Q reaches
FLASH_ATTN_EXT through PERMUTE -> VIEW -> RESHAPE. The support check
rejected this pattern, causing these layers to fall back to CPU and the
split graph to fail with -fa on/auto.
* ggml-openvino: keep only weight-divisor DIVs on CPU
The GPU DIV guard for per-channel scale vectors (`MPT`'s `act_scales`) currently checks only the divisor's shape. During single-token Qwen2-MoE decode, the shared-expert gate `silu(x) / x` has a `[1,1,1,1]` divisor that matches this shape, causing its DIVs to be kept on the CPU.
As a result, every decode step incurred CPU execution and Qwen1.5-MoE decoded at only ~2 tok/s.
Apply the guard only when the divisor is a model weight, so runtime tensors such as the Qwen2-MoE gate divisor can remain on the GPU.
* ggml-openvino: Qwen MTP and multi-seq optimization with new OV ops
All changes of https://github.com/ravi9/llama.cpp/pull/370 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.
Commits of the PR:
- openvino : support Qwen MTP decoding
- openvino: Optimize TopK
- openvino: remove single-slot recurrent cache shortcuts
- openvino: fuse generic recurrent caches and remove shape waits
- openvino: include dense Q8 weights in all INT4 requantization
- openvino: use two GDN outputs for snapshots
- docs: explain native OpenVINO recurrent cache contracts
- docs: clarify Qwen3.5 purpose of native OpenVINO cache ops
- llama: skip unallocated hybrid graph inputs
- quantize: preserve Qwen hybrid layer metadata when pruning
- openvino: emit fused recurrent computation and rotate device caches
- openvino: limit KV slicing to attention caches and share compiled blobs
- tools: add natural-text server benchmark and OpenVINO profile summary
- docs: record fused-cache branch split and UD validation
- server-bench: preserve native token counts and save request timings
- openvino: document MTP regression and host USM alias recovery
- openvino: record large-context allocation diagnosis and default-batch validation
- server-bench: count buffered native tokens and reject stream errors
- openvino: document UD split and unified server benchmarks
- openvino: record Vulkan long-context baseline and GPU-fit controls
- openvino: finalize UD benchmark report and MTP comparison
- openvino: use stable in-place recurrent caches and optional separate convolution
- openvino: accept matching empty axes in DIV repeat translation
- openvino: remove the obsolete cache rotation environment setting
- openvino: validate imported KV results in their four-dimensional layout
- docs: record the bounded in-place KV and convolution benchmark rerun
- docs: complete default-batch OV unified-KV 32K benchmarks
- openvino: remove the separate recurrent convolution path
- openvino: handle zero-logits batches and changing sampler sequences
- docs: compare default and large Qwen3.6 benchmark batches
- docs: use the historical Vulkan default-batch reference consistently
- server-bench: use an integer histogram bin count
- benchmarks: summarize current OV unified MMLU timings
- docs: record cache-only unified KV MMLU results
- openvino: preserve the TopK index port in GET_ROWS
- openvino: reuse the MTP sampling model across active slots
- openvino: expose selection flags and offsets in graph dumps
- docs: consolidate Qwen OpenVINO defaults and performance evidence
- docs: retain continuous-request MTP benchmark results
- docs: compare stable MTP with the non-MTP serving baseline
- openvino: isolate GDN snapshots and skip naive cache fusion
- ggml: document Qwen3.6 GenAI benchmarks
- ggml-openvino: keep PR changes inside the backend folder
* ggml-openvino: drop the extra semicolons after OPENVINO_OP
OPENVINO_OP expands to member function definitions that end in a closing
brace, so a semicolon after it is an empty declaration inside the class.
GCC 15.2 reports it with -Wextra-semi, which ggml enables for GCC 8.1 and
newer, so a build with LLAMA_FATAL_WARNINGS=ON (the default in ci/run.sh)
fails in gated_delta_net.hpp. The CI compilers do not report it.
recurrent_cache_update.hpp has the same pattern twice. It is compiled only
when the OpenVINO package provides the recurrent cache-update ops.
* ggml-openvino: make q8_0_c the default NPU token_embd requant type
Default of `GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT` changed from `f16` to `q8_0_c`.
Applies only to NPU and only to a Q6_K `token_embd.weight`; other devices and
quantization types are unchanged.
* ggml-openvino: do not treat the recurrent state gather as out_ids
The output_len scan now runs for every model and takes the first GET_ROWS
whose index is a graph input. On Qwen3.5 that is the recurrent state gather
of build_rs, which indexes the inp->s_copy leaf directly, so output_len
became 1. In static (NPU) mode the prompt model then returned one logits
row per chunk, and llama-perplexity read stale rows (PPL about 240000 on
Qwen3.5-2B). The same predicate also gave the s_copy input the out_ids
shape and data in static mode.
is_output_idx now skips GET_ROWS that read a recurrent state cache. The old
predicate stays as is_get_rows_input_idx for the dynamic-dim bookkeeping,
so the s_copy leaf keeps its dynamic length there.
* ggml-openvino: prefer the size-preserving dim when a RESHAPE repeats a stride
The RESHAPE dynamic-dim rule matches the source's dynamic stride against
the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4
E2B has one KV head, so [head_size, n_tokens] -> [head_size, 1, n_tokens]
has the same stride on dims 1 and 2. The first match picked the size-1 head
dim, so the K/V views lost their dynamic token dim and became static
reshapes with the first graph's token count. Every later graph with another
token count failed.
Prefer the stride match that keeps the dynamic size, and fall back to the
first stride match for reshapes that change it.
* ggml-openvino: correct the RESHAPE dynamic-dim comment for size-1 dims
The comment above the RESHAPE stride match in compute_node_dynamic_dims()
still said the match is unambiguous because result strides are strictly
monotone. A size-1 dim has the same stride as the dim after it
([head_dim, T] -> [head_dim, 1, T] with one KV head), so it is not.
Explain the size-1 case in the block comment, drop the duplicate comment
inside the loop, and name the source dim's size once.
* examples: add llama-chat-replay
Replays JSONL prompts like OpenVINO GenAI chat.py (-pf, -i chat history, greedy, -o, -pc metrics). The first prompt is prefilled plus one decode step during load (Warmup time) and the memory is cleared, so compile/import is counted in Load time as in GenAI.
* ggml-openvino: make the NPU lazy host threshold configurable, default 2 GiB
With a large context the KV cache (~1 GiB at ctx 8K) crossed the 1 GiB threshold, lost its Level Zero host tensor in strict warm-mmap mode and made decode copy the KV every token. Default to 2 GiB so only the larger prefill compute buffer stays demand-zero, and allow overriding it with GGML_OPENVINO_LAZY_HOST_THRESHOLD_MB.
* ggml-openvino: fix NPU HFA prefill with ordinal-suffixed KV input names
KV cache inputs are now named with a graph-ordinal suffix (cache_k_l0#l12), but flash_attn_ext looks the HFA past input up as ggml_hfa_past_cache_k_l<layer>, so GGML_OPENVINO_NPU_HFA_PREFILL failed with 'ggml_hfa_past_cache_k_l0 not found in tensor map'. Name the past inputs after the bare cache name and resolve the source cache by that name when binding.
* ggml-openvino: keep the NPU prefill model on cache hits after decode
With KV slicing, decode calls overwrite the shared decoder's attention size with the full context, so the next prefill compared that against its sliced size, missed the cache and rebuilt the decoder and both infer requests (plus a cold first chunk). Remember the attention sizes the prefill model was built with and compare against those.
* ggml-openvino: bind NPU HFA past KV zero-copy in the cache layout
The HFA prefill past K/V inputs were [1, heads, past, head_size] copies of the cache, relaid out per chunk on the host and allocated per decoder (256 MiB on phi-4-mini at 3k). Declare them in the ggml cache layout [1, past, heads, head_size], bind the first past rows of the cache directly and transpose in the graph; NPUW HFA reads the tiles from the BSHD past (needs the matching NPUW HFA support). HFA input ABI v4.
* ggml-openvino: size the NPU decode KV window by GGML_OPENVINO_NPU_DECODE_BUCKET
Static decode attended the whole context, so every token read and wrote the full KV cache. With a bucket base set, decode attends the smallest base*2^k rows covering the next position (clamped to ctx): the KV inputs and masks are sliced to the window, the compiled pair's decode graph is built at the window the prefill reaches, and other windows are built or imported from the self-contained cache on first use with the pair's settings and bank. With self-contained blobs, keep the token_embd weight node for the later window models so they reuse the pair's bank entry instead of embedding a second vocab copy. The static build/convert lambdas become functions shared with the window path.
* ggml-openvino: allocate the NPU KV cache on demand with GGML_OPENVINO_NPU_KV_ON_DEMAND
The NPU KV buffer was committed for the whole context up front. With the option set, the KV buffer is swapped for demand-zero memory at its first KV tensor and every KV tensor gets its own storage in a registry, grown in base*2^k rows (base = decode bucket or 1024) as the static graphs reach further. Host accessors, KV inputs and outputs, sliced KV views and the HFA past bind that storage; when it moves, idle requests bound to the old storage are recreated so it is released.
With decode buckets and KV slicing the storage also takes the layout GenAI uses on NPU: K is [heads][rows][head_size] and V is [heads][head_size][rows], each bound whole to prefill and decode. SET_ROWS scatters new K rows on axis 2 and V rows on axis 3, the attention permute only cuts the window, the HFA past is the storage itself, and attention multiplies the probabilities by V^T with transpose_b instead of SDPA. A chunk-sized scatter on the innermost axis is slow on NPU, so the HFA prefill returns each chunk's V^T rows and the host copies them into the cache. Host reads, writes and fills go through a layout-aware copy, storage growth re-packs each head, and the layout is part of the model cache fingerprint.
* ggml-openvino: NPU weightless blob cache with one shared weights file
GGML_OPENVINO_NPU_WEIGHTLESS_BANK=1 (with GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR) exports weightless prefill/decode NPUW blobs plus one <prefill>-<decode>.weights.bin. Weight Constants are laid out once by data pointer and their WeightlessCacheAttribute is retagged with the real offsets. Warm import uses ov::weights_path, and NPUW dedups both models into one bank through the LazyTensor hashes, so the OpenVINO weights_bank duplicate-uid patches are not needed. Decode KV-window models resolve the shared weights through the pair's file, or get their own file when the pair was imported. Also drop the old blob and manifest before publishing a re-export: std::rename does not replace an existing file on Windows.
Based on anikulk/llama.cpp 4ab270aa9.
Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com>
* ggml-openvino: prebuild all NPU decode KV windows with GGML_OPENVINO_NPU_DECODE_PREBUILD
Decode windows were built lazily when the sequence first grew into them, which stalled generation for a compile without a cache and made a strict-mmap import fail for windows the export run never reached. GGML_OPENVINO_NPU_DECODE_PREBUILD replaces GGML_OPENVINO_NPU_DECODE_BUCKET: 0 (default) keeps decode over the whole context; 1 sizes decode by 1024*2^k KV windows and the call that builds or imports the prefill/decode pair also builds or imports the decode graphs of every other window up to the context, from the pair's graph and compute params. Requests are still created on first use. Window building is split out of the request lookup so both paths share it, and window ports are checked against the pair's decode ports.
* Revert "examples: add llama-chat-replay"
This reverts commit 4eded2c52c2df9fa1251673a523f8c7cded20d7c.
* ggml-openvino: initialize the new compiled-graph and build-args fields
ov_compiled_graph and static_build_args gained members, but three brace
initializations still listed only the old ones. GCC reports
-Wmissing-field-initializers there, and CI builds with warnings as errors.
Set the fields by name, and pass {} for weights_path, which the window
build sets afterwards.
* ggml-openvino: fix gemma-4 when only the output rows reach the KV-shared layers
With one sequence, gemma-4 now drops the rows that are not output before
its first KV-shared layer. Those layers take their positions through
GET_ROWS(inp_pos, out_ids), and the backend mishandled three things:
- The positions are reshaped from [n] to [1, n]. That matched RESHAPE
case 1, which marks the wrong dim dynamic. Use case 3, which keeps
the source dim as dim 1.
- ROPE shared one sin/cos table per frequency setting, so the KV-shared
layers reused the table built from the full inp_pos. Add the position
input to the cache key.
- compute_llm_params took input_len from the last attention op. That is
now a KV-shared layer with only the output rows, so the static (NPU)
path padded prefill to the wrong length. Take the largest query length.
Stateful execution still fails on the per-layer embedding view and is
not covered here.
* ggml-openvino: keep the inp_pos layout for gemma-4's gathered positions
A batch can have no output rows, for example a prompt chunk that
llama-cli processes before its last tokens. gemma-4 then reshapes an
empty position gather for the ROPEs of its KV-shared layers. The size
checks of the RESHAPE cases cannot place a dim of size 0, so the
positions got the wrong layout and the ROPE multiply failed on CPU and
GPU. A RESHAPE that feeds the ROPE positions now always takes the
inp_pos layout [1, 1, 1, n].
* ggml-openvino: enable stateless GPU execution for MLA models
The per-head MLA weights (`attn_k_b` / `attn_v_b`) are 3D quantized tensors. Their `MUL_MAT` operations were falling back to the CPU, splitting the graph at every layer and causing GPU execution to fail.
Accept these tensors in the GPU support check and address the following issues exposed by enabling GPU execution:
* Quantization: Use exact zero points only for MoE expert weights. U4 weights with F16 zero points fail in batched GPU MatMul.
* Dynamic VIEW dimensions: Select the correct dimension when a size-1 dimension shares its stride (e.g. `k_pe`), and match KV cache views by stride.
* KV cache V view: Translate the V view of the K cache as a reshape while preserving the dynamic cell axis.
* Flash Attention: Accept the MLA Q (`CONCAT`) pattern and V views derived from another view.
Limitations: The GPU plugin fails on batched compressed weights with static shapes. Keep these unsupported in op tests and `GGML_OPENVINO_FORCE_STATIC`.
* openvino : use writable recurrent cache inputs
Assisted-by: Codex
* ggml-openvino: fix decode crash for Qwen3.5 4B
When the dynamic size is 1, the code now keeps the innermost match. When the size is greater than 1, it still uses the MLA rule, so the MLA k_pe fix still works for multi-token graphs.
* ggml-openvino: keep the GDN state in the OpenVINO layout
Adds an opt-in switch, GGML_OPENVINO_GDN_NATIVE_STATE. When it's on, the GATED_DELTA_NET recurrent state stays in the layout the OpenVINO GatedDeltaNet op uses, [B, H_v, key_dim, value_dim], instead of the ggml layout [B, H_v, value_dim, key_dim]. That removes the two state transposes the translator adds on every call.
* ggml-openvino: support Ling-mini-2.0 group-routed MoE
Ling-mini-2.0 selects experts by group, but the group-selection graph
used ops that were not supported by the backend. This left 370 nodes on
the CPU and reduced decode to ~1.5 tok/s.
* Translate the batched 4D `GET_ROWS` gather used for group scores.
* Support `SET_ROWS` for compute tensors, not just KV caches.
* Preserve the dynamic token dimension in `FILL`.
* Skip the `DIV` repeat when dimensions already match.
* Use the tracked token dimension for `RESHAPE` / `GET_ROWS` op cases,
so a graph compiled during one-row warmup also works for ubatches
with more output rows.
* ggml-openvino: disable output staging
The backend stages small outputs (up to 4 MB) in a USM host buffer
and binds larger outputs directly to the ggml buffer. When the number
of outputs changes between calls to the same graph, as in
llama-perplexity, an output port can switch from USM host memory to
plain host memory.
The OpenVINO GPU plugin mishandles this switch for some dynamic
outputs, causing out-of-bounds writes on subsequent calls. This
results in `CL_OUT_OF_RESOURCES`. For example,
`llama-perplexity -c 2048 -ub 512` fails on the fourth ubatch with
Gemma-2, Gemma-4, Phi-3.5, Phi-4-mini, and Qwen3.5.
Disable output staging until the plugin issue is fixed, so each output
is always bound the same way. Input staging remains enabled.
* ggml-openvino: match attention masks by identity
Gemma-4 prunes non-output rows before its KV-shared layers. These
layers select the corresponding mask rows with GET_ROWS(mask, out_ids).
Mask parameters were named by their consumers, but GET_ROWS is not
recognized as a mask consumer. As a result, the sliding-window layers
selected their rows from the full-attention mask instead of the
sliding-window mask. This either caused the mask add to fail
broadcasting beyond the sliding-window cache (d1024 and above with
-ub 512) or allowed attention beyond the sliding window at shorter
contexts.
Track graph-input attention masks by identity and assign them the
same parameter name for every consumer. The SWA pre-pass now keeps
each layer's own attention mask, as the KV-shared layers read the
same cache with mask rows derived from it.
* ggml-openvino: preserve input mask names and window sizes
KV-shared layers appear at the end of the graph, and their derived
masks are not graph inputs. Processing these masks overwrote the
names and sliding-window sizes recorded from the original input
masks with empty values.
Preserve the names and window sizes of graph-input masks so the
attention dimensions continue to be derived from the original
mask shapes.
* ggml-openvino: accept empty tensors in stride checks
In Gemma-4 batches without output rows, the attention tensors of
KV-shared layers are empty. Their strides beyond the empty
dimension are zero, causing has_strides_on_element_grid() to reject
them.
This sent SOFT_MAX, the KQV MUL_MAT, and CONT to the CPU with
-fa off. It also caused the per-layer input crossing the graph
split to be bound with the wrong layout.
Accept empty tensors in the stride check. Their contents are never
read, so the remaining strides do not need to satisfy the element
grid constraints.
* ggml-openvino: classify transposed V caches using v_trans
With -fa off, the V cache is transposed. When the attended length
is the whole cache, as with a prompt that fits in one ubatch, its V
view is contiguous and was mistaken for the K cache layout.
The K layout path does not keep only the rows of the layer's own V
size. Gemma-4 allocates 512 V rows in every layer, the size of its
widest layer, and its sliding-window layers use the first 256. These
layers then attended with all 512 rows, and prompts fitting in a
single ubatch failed with a broadcast error.
Use v_trans to identify the V cache layout instead of relying on
contiguity.
* ggml-openvino: use per-layer Split in stateful execution
Stateful execution was excluded from the per-layer Split path
used for Gemma-4's per-layer embedding slices. The generic
re-slicing path therefore cut the token axis instead of the layer
axis.
Allow stateful execution to use the per-layer Split and adjust
its axis by the rank dropped during stateful graph construction.
* ggml-openvino: centralize stateful KQ mask lookup
Stateful execution rebuilds the KQ masks from positions. The
existing replacement logic applied to every mask input of
FLASH_ATTN_EXT and SOFT_MAX, including masks derived through
GET_ROWS.
For Gemma-4's KV-shared layers, this replaced the row-selected mask
with the full rebuilt mask, losing the output-row selection. As a
derived mask's name does not name the sliding window, these layers
also got the full-attention mask instead of the sliding-window one.
Add get_kq_mask_input() to replace only the actual mask parameters.
Use it for FLASH_ATTN_EXT, SOFT_MAX, and GET_ROWS, so the derived
masks select their rows from the rebuilt masks, and the lookup is no
longer duplicated.
* ggml-openvino: Removed duplicated if clause.
* fix flash_attn with sink conversion to OV SDPA
* ggml-openvino: drop the unused sink-without-mask path in FLASH_ATTN_EXT
supports_op already rejects FLASH_ATTN_EXT with sinks but no mask, so the
translator never sees a 4-input node whose fourth input is the sink; the
name check could only reach the assert after it. Restore the input-count
detection, fetch the sink where it is used, and format the sink reshape.
* ggml-openvino: fix dimensions bug in stateful recurrent-state remainder writeback
* ggml-openvino: move scalar scales before weight MatMuls on GPU and NPU
GPU and NPU run the graph in f16. Granite multiplies the attention and FFN
outputs by residual_scale right after their output projections. For some
tokens the ffn_down output is far above the f16 maximum (65504), so it
becomes inf and every later layer turns NaN: granite-4.0-350m and
granite-4.0-1b answer with repeated tokens on GPU, and llama-perplexity
reports NaN.
Add MoveScaleBeforeMatMul. It rewrites Multiply(MatMul(x, W), s), with a
scalar constant |s| < 1 and a constant W, as MatMul(Multiply(x, s), W). The
result is the same up to rounding and the MatMul output stays in range.
Register it for GPU and NPU; CPU runs in f32.
* ggml-openvino: translate convolution rollback snapshots
Assisted-by: Codex
* ggml-openvino: run GDN rollback snapshots in OpenVINO on CPU
Speculative decoding that rolls back recurrent state (MTP, EAGLE3, DFlash)
builds GATED_DELTA_NET with K = n_rs_seq + 1 state snapshots and copies them
into the recurrent cache. supports_op accepted the snapshots only on GPU, so
on CPU the GDN ran in ggml while the copy, whose destination is the OpenVINO
cache buffer, ran in OpenVINO. The copy read the packed ggml GDN output
wrongly and stored wrong states without an error: test-llama-context
rollback failed and MTP accepted no drafts.
translate_gdn_snapshots builds plain GatedDeltaNet ops on token prefixes and
does not depend on the device. Accept the snapshots on every device with
dynamic shapes and keep rejecting them only for static execution (NPU and
GGML_OPENVINO_FORCE_STATIC). Update the two doc sentences that said the
snapshots need the GPU.
* ggml-openvino: abort when GDN snapshots would run in ggml
The snapshot copy writes into the recurrent cache in the OpenVINO buffer, so
it always runs in OpenVINO. When OpenVINO rejects the GATED_DELTA_NET that
produces the snapshots (per-key gating as in Ling-3.0, a permuted v, static
execution), the GDN runs in ggml and the OpenVINO side reads its packed
output wrongly. Depending on the shapes, this silently corrupts the cache or
fails inference with a reshape error. Abort when the graph is scheduled,
with the reason.
* ggml-openvino: share one weights file and NPUW bank across all NPU prefill/decode pairs
With GGML_OPENVINO_NPU_WEIGHTLESS_BANK, every prefill/decode pair (one per KV window, e.g. llama-server's startup probe and a long prompt) wrote its own weights file and used its own NPUW bank, so import kept the weights resident once per pair (+2.1 GiB for phi-4-mini). Pairs now use a model-level <wfp>-<wfp>.weights.bin and bank; the weights file records a layout/content hash and is reused when identical, with a per-pair file and bank as fallback. Old caches are invalidated.
* ggml-openvino: export and preload NPU prefill blobs for every KV window
With head-major KV the prefill also runs at a KV window picked by prompt length, so an import-only run failed for prompts whose window was never exported. After a pair is exported, also export the prefill model of every other 1024*2^k window (weights resolved through the pair's shared weights file), and on import preload them into a process-wide cache of imported blobs that pair and decode-window builds reuse. Cache the source fingerprint of weights outside OpenVINO buffers, which made each model fingerprint cost ~9 ms. A window switch now costs ~60 ms instead of ~250 ms.
Rename GGML_OPENVINO_NPU_DECODE_PREBUILD to GGML_OPENVINO_NPU_BLOB_PREBUILD since it now covers prefill as well as decode windows.
* ggml-openvino: drop invalid prefill blobs and clear blob caches on backend free
The prefill preload imported each KV window's cached blob without a try, so
one blob that failed to import failed the graph compute, and the blob stayed
in the cache for every later run. Remove the blob and its manifest and log a
warning, as the pair import does; the window is built when a prompt first
needs it.
The imported-blob map and the fingerprints of weights outside OpenVINO
buffers lived until the process exited. A process that loads models one
after another kept every compiled model resident, and a later model whose
weight landed at a freed weight's address reused that weight's fingerprint.
Clear both with the shared compiled-model cache when the last backend is
freed.
Also apply clang-format to the changed lines.
* ggml-openvino: enable NPU KV windows by default
With GGML_OPENVINO_NPU_BLOB_PREBUILD unset, every NPU decode step ran a
static graph that attends the whole context, so decode was slow for any
sequence much shorter than -c. On a Lunar Lake NPU, Llama-3.2-1B decodes at
0.79 t/s with -c 4096 and 0.20 t/s with -c 16384, against 11.1 and 12.0 t/s
with KV windows. Each window adds one decode graph to compile at load: with
-c 4096, load goes from 11.1 s to 14.6 s for Llama-3.2-1B and from 61.2 s to
81.6 s for Llama-3.1-8B, whose decode goes from 0.20 to 0.82 t/s.
Default GGML_OPENVINO_NPU_BLOB_PREBUILD to 1; 0 keeps the full-context
decode graph. Update the env table and the Boolean-flag note, which said an
unset flag is always disabled.
* ggml-openvino: add POOL_1D and allow F32 EXP on GPU
* skip empty tensors
* ggml-openvino: detect the gated LFM2 conv input as a recurrent sequence
A reshape whose output reaches SSM_CONV through concat(conv_state,
transpose(x)) gets op_case 6, which keeps the active sequence count apart
from the token count, as the conv state does. The LFM2 and LFM3 short
convolutions gate x first, so the transposed tensor is
mul(view(x), view(x)) and the check missed it. The reshape got op_case 5
instead, and:
- in stateful mode, which drops the leading dim for op_case 5, TRANSPOSE
got a rank-3 input with a 4-axis order and graph conversion failed;
- with more than one sequence, the concat with the conv state failed.
Accept the gated input too. The output for a single stateless sequence
does not change.
* ggml-openvino: always stream weight requantization
Requantizing Q6_K/Q5_K weights to 8-bit (or F16 for some tensors) previously converted the entire tensor to f32 first. For large output or embedding matrices, this required 2–2.5 GB of temporary memory.
The chunked conversion path already processes 256 rows at a time and produces identical output. It was previously enabled by `GGML_OPENVINO_REDUCE_COMPILE_MEM` or `GGML_OPENVINO_MEMORY_OPTIMIZE`; use it by default whenever the target block size divides a row. Q4_1_64, which has no chunked quantizer, still converts the whole tensor.
The variable retains its other effects.
* ggml-openvino: allocate host buffers as untouched OS memory
Host buffers (weights, KV cache and compute) were allocated with `ggml_aligned_malloc` and explicitly zeroed with `memset`. This made the entire weight buffer resident before any tensors were loaded.
Allocate host buffers using `mmap(MAP_ANONYMOUS)` on Linux and `VirtualAlloc` on Windows instead. Both provide zero-initialized memory without touching the pages, allowing the OS to bring them into memory as tensors are written.
This preserves the zero-initialization required by ggml-org/llama.cpp#20857 while avoiding unnecessary memory residency during model loading.
Remove the cache-only allocation branches, the corresponding `ggml_aligned_free` path from the destructor and the `data_owned_by_ov` member it used, since they are no longer needed.
* ggml-openvino: release GGUF file pages once a weight is converted
Once `set_tensor` converts a weight into the backend buffer, its source pages in the mapped GGUF file are no longer needed for that buffer. Release these clean pages using `madvise(MADV_DONTNEED)` on Linux and `VirtualUnlock` on Windows. The mapping itself remains intact, and pages can be loaded again from the page cache or file if needed.
Only release whole pages within tensor ranges belonging to read-only file mappings. Skip buffers that wrap the mapping itself.
Keep locked pages resident. On Linux, `MADV_DONTNEED` fails on mlocked memory. On Windows, check page lock status with `QueryWorkingSetEx` and skip ranges containing locked pages, since `VirtualUnlock` would otherwise unlock them.
* ggml-openvino: key weight constants by name for the NPU model cache
The NPU compiled-model cache (GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR without the
weightless bank) imports a blob with the freshly converted model as ov::hint::model,
and NPUW finds each constant by its WeightlessCacheAttribute bin_offset. The keys came
from a per-model counter, and a constant that already had a key kept it. Weight nodes
are shared between models: between the NPU prefill and decode models, and across the
process through the tensor extras. So a weight kept the key from whichever model
converted it first.
The decode model for a KV window is built both as the decode half of the
prefill/decode pair and as a standalone window model, under one fingerprint, and the
two builds got different keys. Importing the blob of the other build failed with
"Couldn't find Constant in cache!", and the decode model was recompiled and exported
again on every run. With GGML_OPENVINO_NPU_BLOB_PREBUILD on by default, this happens
whenever a prompt reaches a larger KV window.
Key weight constants by a hash of the weight name and their position in the weight's
subgraph, above the range of the counter. Other constants are created per model and
keep the counter.
* ggml-openvino: drop CACHE_MODE=OPTIMIZE_SIZE from the NPU plugin cache
With GGML_OPENVINO_CACHE_DIR set, the NPU compile config added
CACHE_MODE=OPTIMIZE_SIZE next to NPUW_CACHE_DIR. With it, the NPU plugin
compiles the NPUW submodels with weights separation, which fails for them:
with NPUW folding, "No WeightlessCacheAttribute has been found in any of the
model's Constant nodes"; without folding, a driver compile error. So every
NPU run with GGML_OPENVINO_CACHE_DIR failed on its first compile.
Keep the default cache mode. NPUW_CACHE_DIR still caches the compiled
submodels.
* ggml-openvino: load each self-contained NPU blob once per process
A self-contained blob (GGML_OPENVINO_SELF_CONTAINED_BLOB) embeds its NPUW
weights bank, and NPUW cannot load a blob into a bank that already holds
those weights: Bank::read_and_add_tensor asserts. Two cases did this on
every run that imported the cache:
- Pairs that differ only in the prefill model share the decode model and
its bank. A long prompt builds such pairs, and each one imported the
decode blob again. The import failed, so the decode model was compiled
and exported again for every pair, until the run ran out of memory.
- The decode model of each KV window uses the bank of the pair's decode
model, so the import of a window blob failed and stopped the run.
Keep each self-contained model in the process once it is imported or
compiled, and reuse it for every pair with the same fingerprint. Turn the
KV windows off for self-contained blobs; the weightless bank keeps them.
* ggml-openvino: update the Docker image GPU drivers to 26.40.40196.8
The OpenVINO image pins the Intel GPU drivers at compute-runtime
26.35.39758.10. With that driver, gemma-4-E4B hangs on the Lunar Lake
GPU. With 26.40.40196.8 it runs.
Take the GPU driver pins from the compute-runtime 26.40.40196.8 release:
IGC v2.42.6, compute-runtime 26.40.40196.8 and gmmlib 22.10.2.
* ggml-openvino: fix long NPU prompts failing with KV slicing
With GGML_OPENVINO_KV_SLICE, on by default, a static prefill is compiled
at the attended KV length that llama reports for the ubatch. That length
grows with every ubatch of a long prompt, and each new length compiled
another prefill/decode pair, with GGML_OPENVINO_NPU_BLOB_PREBUILD (on by
default) also its decode KV windows. The shared compiled cache keeps
every pair until the backend is freed.
Each loaded prefill model keeps NPU memory for its inferences. On a Lunar
Lake NPU, a prefill model loaded next to two or three others cannot get
it: DRM_IOCTL_IVPU_BO_CREATE fails with "Buffer size is too big", and
pfnAppendGraphExecute returns ZE_RESULT_ERROR_UNINITIALIZED. With -c 4096
and the default -ub 512, a 2200-token prompt failed on smollm2-1.7B,
Phi-4-mini, Ministral-3b and gemma-3-4b.
- Round the attended length of a sliced static prefill up to a power of
two, clamped to the context. A growing sequence now compiles a prefill
model when its attended length doubles, not at every ubatch. The added
rows are masked, so the output does not change.
- When only the prefill attention size changes, the new pair takes over
the decode models (base and KV windows) of the entry's current pair,
because they do not depend on that size, and the current pair is
dropped before the new prefill model compiles. A pair that another
backend may use stays cached. Self-contained pairs are exported as one
blob pair, so they keep their own decode models.
- The taken-over KV window that covers the next decode position becomes
the base decode model, as in a pair built at that length, so the first
decode step does not have to create an infer request for it.
* ggml-openvino: pre-scale the GDN q input by default on the NPU
The NPU GatedDeltaNet kernel writes its output in FP16 and flushes
values below 6.1e-5 to zero, and the RMS norm after it amplifies the
loss. GGML_OPENVINO_GDN_Q_SCALE lifts the output above that floor, but
it was off by default. On a Panther Lake NPU with the compiler bundled
in OpenVINO, Qwen3.5-2B scores a perplexity of 14.77 without it and
13.00 with it (12.93 for the same weights on CPU), and its greedy chat
no longer loops while thinking.
- Enable the pre-scale by default with the factor 64.
GGML_OPENVINO_GDN_Q_SCALE=0 turns it off.
- Scale the eps of the norm only when the GDN that feeds it is
pre-scaled. A norm that reads the GDN output directly got the scaled
eps on every device, while q is scaled only on the NPU. With the
variable set, Qwen3.5 gave wrong output on CPU and GPU.
- Add the factor to the model cache fingerprint of NPU graphs with a
GATED_DELTA_NET, so a cached model is not reused across settings.
- docs: update the GGML_OPENVINO_GDN_Q_SCALE row and note.
* openvino-ggml: Decompose the GDN for LNL
1. Automatically decompose LNL GDN prefill into chunked MatMuls; retain the native op for decode.
2. Isolate lowering selection, fix positive-gate handling, and add overflow guards.
3. Update cache identity and add chunk-boundary regression cases.
* ggml-openvino: fix GDN decomposition in decode and on long prompts
Two problems remained with the GATED_DELTA_NET decomposition on a Lunar
Lake NPU.
The decomposition covered only multi-token (prefill) graphs, and decode
kept the GatedDeltaNet op. Under Linux the decode kernel loses the
device as well: after a successful prefill, the first generated token
failed with ZE_RESULT_ERROR_DEVICE_LOST in llama-completion, llama-bench
and llama-server.
The chunk masks were new constants in every converted model. The NPUW
weights bank is shared by all compiled models (NPUW_WEIGHTS_BANK=shared)
and identifies a constant by its data pointer, also after the model
that owned it is gone. When a longer sequence needs a second prefill
model, its masks can get the addresses of freed masks of the first one
and are then read as those. The Qwen3.5-2B perplexity over 512 tokens
changed between 12.99 and 56.94 from run to run, and a 1900-token
prompt gave garbage.
- Lower a single-token GATED_DELTA_NET to one recurrence step when the
decomposition is on: the chunk formulas for a chunk of one token,
where P = 1 and the decay is 1.
- Keep one instance of each chunk mask for the whole process.
- test-backend-ops: add a single-token case with two sequences, repeated
heads and a positive gate.
- docs: GGML_OPENVINO_GDN_DECOMPOSE covers single-token graphs.
* openvino-ggml: Enhance the safe env parsing
* ggml-openvino: keep the GDN q pre-scaling on when its variable is unset
ggml_openvino_gdn_prescale_enabled() returned false when
GGML_OPENVINO_GDN_Q_SCALE is unset, which switched the default-on
pre-scaling off. Qwen3.5-2B on a Panther Lake NPU went from a perplexity
of 13.00 back to 14.77.
- Treat an unset variable as enabled. Zero, negative and non-numeric
values still turn the pre-scaling off.
- Comments: ggml_openvino_getenv_int() no longer goes through std::atoi.
- docs: a boolean flag with a non-integer value is no longer always
disabled; a flag whose default is 1 stays enabled.
* ggml-openvino: use the PLUGIN compiler for GDN graphs on the NPU
The NPU driver compiler, the default, gives wrong output for graphs
with GATED_DELTA_NET (Qwen3.5). Qwen3.5-2B on a Panther Lake NPU answers
with unrelated text and has a perplexity of 16.77, against 13.00 with
the compiler bundled with OpenVINO. Other models stay on the driver
compiler, which builds faster prefill kernels.
- Compile static graphs with GATED_DELTA_NET with
NPU_COMPILER_TYPE=PLUGIN, unless GGML_OPENVINO_NPU_COMPILER_TYPE or
NPU_COMPILER_TYPE in GGML_OPENVINO_NPU_CONFIG sets the compiler type.
- Add the choice to the model cache key of these graphs, so blobs from
the driver compiler are not imported.
- docs: describe it in the GGML_OPENVINO_NPU_COMPILER_TYPE row.
* ggml-openvino: fix gemma-4 E2B on the Lunar Lake NPU at small contexts
The NPU 4000 compiler computes ScaledDotProductAttention wrongly when K
has one KV head and is square, that is when the attended rows equal the
head size. gemma-4 E2B has one KV head, 512-wide full-attention heads
and 256-wide sliding-window heads, so a context of 512 tokens or less,
and a prefill that attends 256 or 512 rows, gave wrong output.
For that shape on NPU 4000, build the attention from MatMul, Softmax and
MatMul, as the transposed-V path already does. Other shapes and devices
keep the op.
* ggml-openvino: support multiple GPUs with -sm layer
* ggml-openvino: check GGML_OPENVINO_* variables and reorganize the docs
The backend ignored a misspelled GGML_OPENVINO_* name and a value that
was not valid, and the docs listed every variable in one table.
Code:
- One table holds the known variables and their types. A value that is
not valid stops the backend with a message that names the variable and
the accepted values. An unknown GGML_OPENVINO_* name gives a warning.
- With log verbosity 4, one line shows the variables in effect.
- GGML_OPENVINO_GPU_CONFIG and GGML_OPENVINO_CPU_CONFIG pass plugin
properties to the GPU and the CPU, as GGML_OPENVINO_NPU_CONFIG does
for the NPU. If the plugin rejects a property, the error names the
variable.
- GGML_OPENVINO_STATEFUL_EXECUTION gives a deprecation warning.
- The key of the compiled model cache covers the two new variables,
GGML_OPENVINO_KV_SCATTER_ELEMENTS and
GGML_OPENVINO_DISABLE_ELTWISE_RANK_ALIGN.
- GGML_OPENVINO_ENABLE_CACHE, which no code read, is no longer
registered.
Docs:
- docs/backend/OPENVINO.md: Quick Start, collapsible blocks, runtime
configuration in groups (basic, plugin properties, advanced tuning,
compiled model cache, deprecated), updated validated models and known
limitations, corrected stale entries.
- dev_doc/OpenVINO-ENV-VARS.md: the variables for development and the
NPU properties that the backend sets.
* docs : add LFM2.5 and Ling-mini-2.0 to the OpenVINO validated models
* ggml-openvino: rebuild converted weights in get_tensor
set_tensor converts a quantized weight in place into the extracted or
requantized layout, so get_tensor returned bytes that are not in the ggml
format. A reader such as the CPU reference in test-backend-ops then
computed on garbage and produced NaN (MUL_MAT cases with weights=1).
get_tensor now rebuilds the ggml bytes for such a weight: it evaluates the
weight node on the host and quantizes the values back to the tensor type.
The result is close to the original data, not bit-exact. Other reads are
unchanged, and the inference path does not call get_tensor on weights.
* ggml-openvino: drop the GDN test cases from test-backend-ops
Restore tests/test-backend-ops.cpp to the upstream version.
* docs : improve the OpenVINO backend docs
* docs : restore the OpenVINO intro and the development note
* ggml-openvino: fix slow performance of MoE models with requantize and workaround to use fused MOE Compressed
* ggml-openvino: document the fused GPU MoE op limits
Converting K-quant MoE experts to Q4_1_64 on GPU lets the plugin use its
fused MoE op, which runs in f16 only. granitemoe models give wrong output
with it, and an f32 inference precision hint fails to compile. Document
GGML_OPENVINO_MOE_OP=0 as the way to turn the op off, and that
llama-perplexity with MoE models needs -b equal to -c.
Also split the long TODO comment over two lines.
* ggml-openvino : FILL for in-place clear of recurrent cache, sticky NaN fix
* restore this part
* ggml-openvino: bump the model-cache version for the recurrent-cache reset change
Recurrent-cache resets now clear slots with Select instead of Multiply,
so a NaN in a cleared slot no longer survives. Compiled models cached
before this change keep the old graph, so bump the fingerprint version
to rebuild them. Also note how SCALE and FILL store their parameters.
* ggml-openvino: read rope_factors_short/long as weight constants
create_weight_nodes() registers the ROPE frequency factors as
"rope_freqs.weight", but set_input_output() and compute_model_inputs()
kept the tensor's own name. For models whose factors are named
rope_factors_short or rope_factors_long (Phi-3.5, Phi-4, MiniCPM with
LongRoPE) the factors became a Parameter fed from the host weight buffer.
That buffer is empty in cache-only mode and after the host weights are
released, so these models gave garbage output with
GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY, GGML_OPENVINO_RELEASE_WEIGHTS
and GGML_OPENVINO_MEMORY_OPTIMIZE.
Use the same name in all three places, so ROPE always reads the weight
Constant. The graph changes, so bump the model-cache version to 7.
* ggml-openvino: import cached models from a memory-mapped tensor
A stream import reads the whole compiled blob into a heap buffer while
the plugin builds its own copy of the weights. On CPU this held the blob
twice at the peak. Import from ov::read_tensor_data() instead, which maps
the file. This lowers the CPU cache-hit and cache-only peak memory; GPU
memory is unchanged.
* ggml-openvino: do not write CPU model-cache blobs for MoE models
The CPU plugin exports its MoE op (GatherMatmulCompressed) but cannot
import it again with OpenVINO 2026.4 and 2026.5 ("unsupported opset").
Every later run then failed the import, compiled again and rewrote the
blob. Skip the blob for such models on CPU and print a warning.
In cache-only mode no blob exists for them, so the generic advice to run
once without cache-only cannot help. Print an error that says cache-only
does not work for MoE models on CPU instead.
* ggml-openvino: fix GGML_OPENVINO_DEBUG_INPUT on GPU
GPU inputs can be device (USM) tensors, whose data() throws, so every
GPU compute failed with GGML_OPENVINO_DEBUG_INPUT=1. Print a host copy
of remote tensors instead.
* docs : add OpenVINO memory optimization tips and MiniCPM-V-4.5
Add a "Tips for Memory Optimization" section to OpenVINO-ENV-VARS.md
with examples for the spill directory, the compiled-model cache, host
weight release on GPU, CPU MoE compile memory and the NPU weightless
cache, and link it from OPENVINO.md. Add MiniCPM-V-4.5 to the validated
models.
* ggml-openvino: run MoE models with F16 and BF16 weights on GPU
On GPU, supports_op rejected MUL_MAT_ID with non-quantized expert weights
and GET_ROWS with BF16 data, so the scheduler ran those ops on the CPU.
For F16 MoE models the CPU MUL_MAT_ID then read the strided top-k view as
expert ids and crashed. For BF16 models the token embedding came from the
CPU and reached the OpenVINO graph as a static-shaped input, which failed
once the token count changed. Both ops translate correctly on GPU, so
remove the two rejections.
* ggml-openvino: fix BERT embedding models on CPU and GPU
BERT models (bge-m3, nomic-embed-text-v1.5) failed on every device:
- A VIEW of a weight with a GGML_OP_NONE source was marked as having a
dynamic token dimension, a rule meant for KV cache views. The token type
row view of BERT was then never sliced, and the whole two-row table was
added to the activations. Skip the rule for model weights.
- A row range view of a 2D weight now becomes a Slice of the weight rows,
as GET_ROWS already does.
- Attention without a KV cache pads K and V up to the mask width, which
depends on the token count. PAD baked the amount from the first graph,
so later token counts gave mismatched shapes. Compute the end padding
from the mask width at run time.
The NPU still fails for these models.
* docs : add bge-m3 and nomic-embed-text-v1.5 to the OpenVINO validated models
---------
Co-authored-by: Sabnis, Omkar <omkar.sabnis@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yu, Zijun <zijun.yu@intel.com>
Co-authored-by: Proshunin, Nikolay <nikolay.proshunin@intel.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
* ui: Llama UI's own Storybook in ui/lib/ui
The library gets its Storybook (`npm run storybook`, `npm run
build-storybook`) with a story for everything it holds, under Llama UI:
the foundations, the 29 base components (as actions, forms, display,
overlays and navigation), the app components (ActionIcon, SearchInput,
SyntaxHighlightedCode), the Model UI and the native app mockups. The
stories import the package by its published names, aliased onto src/lib,
run as tests under `npm test` (the storybook project in vite.config.ts),
and are type-checked with the rest of the package. `npm run dev` starts
Storybook next to the SvelteKit check page; the toolbar switches the
color mode and the surface theme, and pages that are mostly prose are
.svx, through mdsvex.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: a Storybook for Llama Chat
Llama Chat's Storybook gets an introduction, Agents › Design guidelines
(DESIGN.md), Llama UI's stories as the library has them, and Llama Chat:
its own foundations (safe areas, the splash screens), then one section per
src/lib/components/app/<area> (Chat with the chat screen, form, messages,
attachments and tabs; Content, Models, MCP, Navigation, Settings, Dialogs,
Forms, Actions, Badges, Misc, Servers and PWA), each accessibility story
next to its component's. It replaces the flat pages it had (chat message,
chat form, markdown content, the PWA refresh alert and their fixtures,
and the ChatForm single-row exploration).
The stories run on the new tokens and surfaces with the color mode and
surface theme in the toolbar (by mode-watcher's key, the ColorMode and
SurfaceTheme enums and applySurfaceTheme), and pages that are mostly
prose are .svx in an editorial frame (AGENTS.md says how to write them
and where a story goes). The story tests run Llama Chat's stories and
leave the library's to the library. The brand and communications pages
are in ggml-org/llama-brand; the logo files come from @ggml-org/llama-brand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let the ChatScreen story store its reply before the next one starts
The ContinueConversation story ended while its reply was still being stored,
and the next story reset the stores under it. It now waits for the reply to
finish.
Assisted-by: Claude (Fable 5.2)
* ui: publish the Storybook to its Hugging Face Space on every UI change
A new UI Storybook workflow builds ui/apps/chat's Storybook whenever
ui/apps/chat or ui/lib/ui changes on master (or on demand) and uploads it
to the static Space ggml-org/Llama-UI-storybook, with a Space card naming
the commit it was built from. Files the new build no longer has are
removed; the Space's .gitattributes stays. It runs only in
ggml-org/llama.cpp, takes the UI's Hugging Face token
(HF_TOKEN_UI_STATIC_OUTPUT), and the Space can be changed with the
HF_SPACE_UI_STORYBOOK variable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: run the library's tests, its stories among them, in CI
The Library checks job type-checked and packed ui/lib/ui but never ran its
tests, so the library's stories, which now hold every library component's,
went untested in CI. It installs Chromium and runs `npm test`.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw a floating panel's rim on its border, so its edge is one line
Menus, popovers, dialogs and the other matte-float panels carry a 1px border that the
surface made transparent, and drew their inner highlight as an inset shadow. An inset
shadow sits inside the border, so the 1px of fill between it and the outer ring showed
as a second edge. The border now draws the highlight, right inside the ring.
Assisted-by: Claude (Opus 5.5 Medium)
The implicit GEMM conv2d kernel recovers the channel and the kernel
position of each reduction index with three integer divisions by
runtime divisors, once per loaded row of every tile. Outside the 3x3
and 1x1 specializations this integer division bounds the kernel:
removing it halves the time on long reductions.
Divide with the precomputed magic numbers of fastdiv instead. The 3x3
and 1x1 specializations keep their compile time divisors.
test-backend-ops -o CONV_2D: 2006/2006 on CUDA0.
CONV_2D, F16 kernel, us/run on an RTX PRO 6000:
[2054,1,768] k [7,1,768,768] 221 -> 138
[16390,1,384] k [7,1,384,384] 433 -> 266
[65542,1,192] k [7,1,192,192] 420 -> 271
[8240,1,256] k [11,1,256,256] 165 -> 108
[19,19,256,16] k [4,4,256,4096] 1672 -> 1076
* server: fix lora and recurrent
* tests: run the hybrid SWA checkpoint test with one token per ubatch
With and without the cache, the prompt was not split into the same
ubatches, and the generated tokens are decoded one at a time while the
uncached run processes them in a batch. On CUDA this moves the top
logprobs of the tiny model by up to 0.03 and swaps near ties, so the
test failed while the cache behaved as expected.
With one token per ubatch both runs go through the same kernels and
the output is compared exactly.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ui: publish @ggml-org/llama-ui from CI when its version moves
ui/lib/ui reaches npm only when someone remembers to publish it by hand, which let 0.3.0
sit a day behind master. The version in its package.json becomes the release: a commit
landing on master with a version npm does not have gets built, linted and published with a
provenance attestation, through OIDC rather than a stored token, as release-publish.yml
already does for GitHub releases. A library change carrying no bump publishes nothing and
says so, on the PR and on the run.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: bump @ggml-org/llama-ui to 0.4.0
The library on npm, published before the design system PR merged, holds no entry points for
the Model UI, the app components or the shared constants and enums, and no dependency on
@ggml-org/llama-brand, whose colours the primitives now build on. Consumers import the new
subpaths and get ERR_PACKAGE_PATH_NOT_EXPORTED.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: name no environment for the Llama UI publish job
Creating a repository environment needs admin rights the UI maintainers do not hold, so a job
that names one cannot be set up by them. The job works without it: npm's trusted publisher
matches on the repository and the workflow file, and its environment field stays blank to
match. Nothing else in the job depends on an environment.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: address review on the Llama UI publish job
- the advisory version check leaves ui.yml for the publish workflow, on a pull_request trigger
whose path filter keeps it off PRs that touch only Llama Chat: a chat-only PR was told to
bump a library it did not change
- publish and verify run on master only. The trusted publisher matches on the repository and
the workflow file, not the branch, so a dispatch from a PR branch - anything with write
access can start one - could publish code no one reviewed. With no environment, this was the
only branch protection in place.
- the registry read after a publish retries six times rather than sleeping once, so a release
the registry is slow to serve is not reported as a failure
- a build metadata suffix (1.2.3+build) is not a prerelease, and versions order by semver's
prerelease rules, so 0.4.0-beta.1 behind a released 0.4.0 is refused instead of shipped to next
- the summary step passes its values through env like the others, actions: read is gone, and the
concurrency comment claims only what the setting does
- npm older than 11.5.1 is refused up front, as it cannot publish with an OIDC token
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: note the two trusted-publishing traps in the Llama UI README
A publisher names a workflow file that must already be in .github/workflows, so it is added
after this workflow lands rather than before, and one created now publishes only through
staged publishing until npm publish is permitted for it. Provenance comes free with either,
and only on a GitHub-hosted runner.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* Update .github/workflows/ui-lib-publish.yml
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* jinja: impl scope-chain key lookup
* add test for call_statement lambda
* chat: add tts_dict for kokoro G2P template
* more testing
* value_float_t range check
* no json-in-meta, add llama_model_meta_arr_str/_n
* mtmd: add kokoro tts support
* detailed guide about G2P and text norm port
* refine G2P
* clean up
* migrate batch api
* nits
The fast CONV_2D path (implicit GEMM with MMA, or cuBLAS for
pointwise convolutions) required an F16 kernel. F32 kernels fell
back to the naive direct convolution, which is much slower than
im2col + mul_mat on large channel counts.
Convert F32 kernels to F16 in a pool buffer, as is already done for
the input, and take the same path with F32 accumulation. The Metal
implicit GEMM kernel already stages both operands as F16.
* Added support for Qwen3-TTS ICL/reference mode
Assisted-By: Claude Opus 5.5
* fix correctness
* attach transcription to bitmap
* nits
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* sampling : fix crash when both ignore_eos and grammar are on
* server : rework, throw 400 when grammar and ignore_eos are on
* sampling : move the check into the sampler (review suggestion)
* tests : remove redundant tests
As discussed in #31955, the live preview is now opt-in. With it on, the editor renders blocks as widgets, and moving the cursor down from the first line of a prompt with a few blocks (a heading, a list, a code block) jumps to the last line; with it off, the cursor moves one line at a time. Users with a saved config keep their current value, since the full config is stored in localStorage.
The message input also gets spellcheck, autocorrect and autocapitalize back. CodeMirror turns them off by default, so they were off with the preview on or off; the former textarea and contenteditable inputs had the browser's defaults.
* llama : keep the k-pool layout prefix on sequence edits
Assisted-by: Claude Code
* llama : address review of the k-pool layout update
* Append and edit share one branch, using stale > sq.pos_min.
* LLAMA_KPOOL_VERIFY=1 checks every update against a full rebuild. The scan moved into kpool_scan() so the check can reuse it.
* The test turns the check on for its rollback context, skips -sm tensor, and drops kimi-k31.
* Two comments that were no longer true are fixed.
Assisted-by: Claude Code
* llama : check the k-pool layout update in debug builds instead of LLAMA_KPOOL_VERIFY
* llama : remove the k-pool layout verification
* llama : inline the k-pool scan again
* remove dangling new line
* do not search twice
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
* ui: bring @ggml-org/llama-ui into ui/lib/ui
The design system package (tokens, fonts, surfaces, components and Model UI), so far in the temporary ggml-org/llama-ui
repository at edaf518, moves to its home in ui/lib/ui, built with @sveltejs/package and published as @ggml-org/llama-ui
from here. ui/lib holds the packages, one folder each. Its references to tools/ui now name ui/apps/chat, its
package.json points at this repository, and the UI workflow type-checks and packs it. ui/README.md describes the
layout.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: Llama UI design system: tokens, surfaces and brand
The design system from allozaur/docs/storybook-overhaul: three tiers of
tokens, matte glass surfaces, Llama Sans and Llama Mono, and the base
components on the new tokens (Tabs and GlassKey among them). It also
restyles the stack's features with it, replacing the plain stand-ins they
used until now. DESIGN.md is generated from the tokens. The Storybook that
documents it is the next PR.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: have Llama Chat use @ggml-org/llama-ui for its base components, tokens and fonts
Llama Chat kept its own copy of the design system: 29 base components, the
three tiers of tokens, the surfaces, the theme and the fonts, next to the 12
components ui/lib/ui held. The copy goes, and the library gets what it lacked.
ui/lib/ui
- the 17 base components the app had and the library did not (alert,
alert-dialog, button-group, checkbox, collapsible, context-menu, dialog,
drawer, dropdown-menu, empty, hover-card, radio-group, scroll-area,
skeleton, table, tabs, textarea), with vaul-svelte for the drawer
- the inset-group and selected-row tokens and surfaces
- sources import each other by relative path instead of $lib, which in an
app points at the app's own src/lib
- theme.css no longer imports tw-animate-css: it is a peer dependency that
the consumer imports before the theme, as it does Tailwind
- 0.3.0
ui/apps/chat
- imports the base components as @ggml-org/llama-ui/ui/<name> and cn/tv as
@ggml-org/llama-ui/utils; app.css imports the library's theme.css
- compiles the library from source, with no install or build of it:
scripts/vite-plugin-llama-ui.ts resolves its entry points and what it
imports, and svelte.config.js gives TypeScript the same paths
- src/lib/components/ui, styles/tokens, styles/surfaces.css and assets/fonts
are removed; components.json moves to the library
Builds
- CMake stages ui/lib/ui next to the app, keeping the layout of ui/, and
rebuilds when a library source changes
- the Dockerfiles copy the library's sources; the Nix derivation takes ui/
as its source
The built CSS has the same rules in the same order, apart from one
zero-specificity rule and the @font-face rules moving, plus ten utilities
and two tokens that only the library's own components use.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: AGENTS.md places Llama UI in ui/lib/ui
Llama Chat's AGENTS.md says where each part lives: Llama UI is
ui/lib/ui, and Chat imports its tokens, fonts and base components
from there.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: Llama Sans and Llama Mono from llama-brand's latest, and shell commands marked in bash code
The fonts in ui/lib/ui come from llama-brand's latest, with optical sizes
from 10 to 36. Bash code blocks mark shell commands and flags.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open submenus outside the menu they come from
A submenu was rendered inside its parent menu, so the menu's scroll box
clipped it: the glass menus' backdrop filter, and the filter and transform
of the open animation, make the menu the containing block even for the
fixed-position submenu. Dropdown and context submenus now render in a
portal, like the menus themselves.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw submenus like the menu they open from
Submenus had smaller corners than their menus, and context menus had none
of the dropdown's glass. Context-menu parts now match the dropdown's, and
both submenus take the menu's corners and height cap, with their first row
level with the row that opens them, 4px past the menu's edge. Sub-triggers
take the items' radius and pointer.
The submenus' height cap and zoom origin named the dropdown's CSS variables,
which bits-ui does not set on submenus; they now use its --bits-menu-* ones.
The chat group submenu takes the actions menu's z-[999999] layer, as it no
longer renders inside that menu.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a late current-node write off a closed or switched conversation
updateCurrentNode checked the active conversation, awaited the database write, then wrote currNode to whatever was active after it. When the conversation closed during the write it threw, and the reply ended in a streaming error dialog. When another conversation opened, that one got the wrong node.
Assisted-by: Claude (Fable 5.2)
* ui: have Llama Chat use the library's Model UI and app components
Llama Chat still kept its own copies of what the library holds above the
base components. It now takes them from @ggml-org/llama-ui too, and its
copies go.
ui/lib/ui
- ModelId gains what Chat's had: the model's own draft kind, draft
sidecars with their quant (ModelDraftSidecars, new), aliases, hideName
and a tooltip, in the same badge order
- entry points for them, like ui/<name>: @ggml-org/llama-ui/app/<group>
(actions, content, forms, misc) and @ggml-org/llama-ui/models
ui/apps/chat
- ActionIcon, SearchInput, SyntaxHighlightedCode, Logo,
ModelCapabilityIcons, ModelDownloadProgressBar and ModelDraftSidecars
come from the library; Chat's copies and the two constants only they
used are removed
- ModelId and ModelOrgAvatar stay as thin wrappers: ModelId parses the raw
id and applies the user's display settings, ModelOrgAvatar hides the
avatar while Hub metadata is off
- the svelte/* TypeScript path falls back to Svelte's types/index.d.ts:
svelte/reactivity has no file of its own, so a library source importing
it got its types from ui/lib/ui's own install when there was one
Assisted-by: Claude (Opus 5.5 Medium)
* ui: Llama UI takes the brand's colours and logo from llama-brand
@ggml-org/llama-brand is now a dependency of the library, by the same GitHub
tarball Llama Chat already installs, which now names it @ggml-org/llama-brand
too.
- primitives.css drops the 25 tokens it held copies of (the neutral
greys, sand, the skies and cinder) and builds on llama-brand's
css/colors.css instead; the skies' gradients point at llama-brand's.
Apps import that file before theme.css, as they do Tailwind and
tw-animate-css: the library's stylesheets import no package, so an app
that compiles them from source resolves it from its own node_modules
- Logo draws llama-brand's icon, in currentColor, and assets/logo.svg goes
- Llama Chat's TypeScript paths resolve llama-brand from its own
node_modules, like the library's other dependencies
The fonts stay as the library's WOFF2 copies until llama-brand ships
WOFF2 builds of them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: have Llama Chat take the library's constants, enums and helpers
Llama Chat redefined most of what the library exports next to its
components. Where the two were the same, Chat now re-exports the library's
from the files that held them, so its own imports stay as they are.
- the library gets entry points for them: @ggml-org/llama-ui/constants,
@ggml-org/llama-ui/enums and @ggml-org/llama-ui/code, and
@ggml-org/llama-ui/models carries the model data with the Model UI
- from the library now: the shared CSS classes and code-block constants,
the byte units, the memory-fit constants and minMemoryTierGb, the bit
depth labels, the capability and modality icons, labels and order,
ColorMode, TooltipSide, BooleanString, the model enums, ModelModalities,
formatParameters, highlightCode and trimCodePadding
- the library's modality order follows Chat's: vision, video, audio
- Chat keeps its own where they differ: UI_DATA_ATTRS and
ModelDownloadProgress are supersets of the library's (the latter extends
it now), and formatFileSize counts in 1024s
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the chat form's add key and model selector a tighter shadow
In light, the two keys inside the chat form, white on its white glass, took
the control elevation, whose soft spread blended into the form. They take a
new elevation-key instead: a 0.5px hairline all round and a 1px drop. In
dark they keep the control elevation. Chosen from the options in
Scrapbook › UI › Composer keys.
- semantic tokens: --elevation-key (light: the hairline and the drop; dark:
the control elevation)
- component token: --chat-form-key-shadow, for the add key and every model
selector trigger
- DESIGN.md names it next to the other elevations
Assisted-by: Claude (Opus 5.5 Medium)
Browsers already have tabs, so a second tab bar inside the page adds a layer most people don't need. The setting stays, so anyone who wants tabs can turn them on. Users with a saved config keep their current value, since the full config is stored in localStorage.
* ui: replace deprecated packages and fix npm audit findings
Swap the mdast stub for @types/mdast, move the katex override to 0.19.0,
and override glob, source-map and html-encoding-sniffer under
workbox-build and http-server so installs no longer pull deprecated
packages. Apply the non-breaking npm audit fixes to the lockfile.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: migrate Llama Chat to SvelteKit 3 and its built-in service worker
Move to SvelteKit 3, Vite 8, TypeScript 6 and Storybook 10.6. The
SvelteKit options move from svelte.config.js into sveltekit() in
vite.config.ts, $lib imports become #lib subpath imports from
package.json, and $app/environment becomes $app/env. Modules named
server.* or living under a server/ folder are server-only in SvelteKit 3,
so the llama-server components, store and enums move to llama-server
names.
@vite-pwa/sveltekit and Workbox give way to SvelteKit's own service
worker in src/service-worker, registered by usePwa(), which precaches the
app files and leaves API requests to the network. A small plugin emits
manifest.webmanifest. static/sw.js removes the worker of older builds, and
llama-server serves a self-removing service-worker.js as well as sw.js
when the built-in UI is off or replaced.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: use SvelteKit 3's update detection and Svelte 5's current patterns
A page the service worker doesn't control now learns of a new build
from SvelteKit itself: `updated` from $app/state checks
_app/version.json on navigation, on focus and every hour. It replaces
the version Llama Chat kept in localStorage and compared by hand.
Reload on the update prompt pulls in the new worker and activates it
before reloading, so one click lands on the new build, whether or not
a worker controlled the page.
Following Svelte's current guidance:
- usePwa keeps its state in $state rather than a store, and no effect
checks `browser`
- nearViewport, autoResizeTextarea and the custom CSS injection are
attachments ({@attach}) rather than actions (use:)
- class lists use arrays rather than class: directives
- the chat contexts come from createContext, typed, in place of
setContext/getContext with string keys
- ActionIconCopyToClipboard takes $props() rather than `export let`
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move to ESLint 10
ESLint 9 is out of support. Every plugin the app uses supports 10:
typescript-eslint, eslint-plugin-svelte, perfectionist,
simple-import-sort, eslint-plugin-storybook and eslint-config-prettier.
@eslint/js and @eslint/compat follow.
ESLint 10's recommended rules ask for two things the code now does:
errors rethrown from a catch keep the caught error as their `cause`,
and no variable starts with a value that every path overwrites.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: override minimatch to clear npm audit's last finding
ESLint 10.14, @eslint/config-array 0.23.5 and typescript-eslint 8.72
depend on minimatch 10, which GHSA-g3hv-75wg-m249 (denial of service
via globparts amplification) affects below 11.0.1. Their fixed releases
move to minimatch 11 but are all younger than the repo's 7-day
min-release-age, so an override pins minimatch 11.0.5, the newest the
rule allows. The three only call new Minimatch(), match() and
minimatch(), which minimatch 11 keeps. npm audit now finds nothing.
Assisted-by: Claude (Opus 5.5 Medium)
This commit removes a call of the CUDA_SET_SHARED_MEMORY_LIMIT macro as
the fallback decision was moved out of launch_mul_mat_q.
launch_mul_mat_q now has the fallback template parameter which was added
in Commit 6eddde06a4 ("CUDA: refactor MMQ
kernel configuration (#24127)").
This code was then refactored in Commit
c49edf0963 ("CUDA: fuse FFN gate/up
matmuls and GLU in MMQ with interleaved gate/up rows (#29948)") which
added the has_fusion template parameter but it kept both macro calls.
launch_mul_mat_q only launches one type of kernel now (this is now
decided earlier in the templated call chain) and each specialization
will set the limit itself.
The motivation for this is to simplify the code and improve readabilty.
* fix the SYCL fa with mkl error of a mask broadcast over heads or batches; add argument cache-ram to start-svr.sh
* add argument cache-ram to win-start-svr.bat
* fix format issue, remove --cache-ram in script: move to another PR
* Only run webgpu CI on my fork
* Add webgpu only workflow
* resolve merge conflicts
* save state
* save kernel
* add kernel back in
* fix merge issues with kernels
* updates
* update tests
* change invocation count and workgroup to be u32
* clang format
* clang format
* update docs
* format
* remove extra file
Replace the per-element modulo of i0 with fast_div_modulo in the ravel and
unravel bin-bcast kernels, drop the now-unused ne10/ne20 kernel arguments,
and reject divisors that do not fit in 32 bits when the magic values are
computed, as the CUDA implementation does.
Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
* mtmd: support YuE2-3B
* improve speed
* add CFG
* fix streaming
* add cfg control from api
* check ctx size
* fix n_predict
* add flow_steps config
* nits
* fix batch api
* nits
* adapt
* mtmd: yue2 samples from its own saved logits
The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.
* mtmd: let the yue2 snake chain fuse
The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* opencl: release the WMM flash-attention mask fallback image
The fallback image is created on every call of the bin flash-attention
kernel over a sub-buffer of the K cache and was never released, so each
call kept a K-cache sub-buffer alive. Every call with more than one query
row takes this path, including each speculative-decoding verify step, so
the objects pile up and the server slows with every request.
* opencl: keep q8_0 K/V at head size 512 off the WMM flash-attention kernel
With q8_0 K/V at DK=512 the bin flash-attention kernel gives wrong results
whenever V holds different values from K (test-backend-ops FLASH_ATTN_EXT
hsk=512 q8_0, max error 2.04 on X2-90); f16 K/V at DK=512 is correct. The
backend's own flash-attention kernels already decline q8_0 at DK=512, so the
shape is now handled the same way with or without the kernel library.
* opencl: decline WMM flash-attention prefill beyond the device's 3D image limits
The WMM path repacks Q/K/V into 3D images; V is n_kv wide, so past the device's
maximum 3D image width (about 16k tokens of context on X2-90) clCreateImage
failed and the call asserted. Such shapes now take the regular flash-attention
path.
* ui: move tools/ui to ui/apps/chat
The chat app moves to its own root folder, ui/apps/chat, as the first step of splitting the UI into one library
(ui/lib) and the apps built on it (ui/apps). The code is unchanged; every path that pointed at tools/ui now points at
ui/apps/chat: CMake adds it from tools/, and the Dockerfiles, the Nix package, the workflows, the labeler, CODEOWNERS,
.editorconfig, the git hooks, the dev and demo scripts and the docs follow it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name Llama Chat's package @ggml-org/llama-chat
It was still called llama-ui, the name of the library that now sits beside it
in ui/lib (@ggml-org/llama-ui). The package stays private; the app's display name
and the llama-ui.zip build artifact are unchanged.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: call the built-in web UI Llama Chat in docs, help text and comments
The app in tools/ui is Llama Chat; Llama UI is the component library it is
built from. Prose that said "Web UI", "web UI" or "WebUI" for the app now
names it: the READMEs, SECURITY.md, the --ui help text, the CMake option
description, one server log line, code comments and the Storybook intro.
Names that others depend on are unchanged: the deprecated --webui flag
aliases, the LLAMA_*_WEBUI CMake and environment names, the webui_settings
field, the storage migration keys and the Nix attributes.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name tools/ui as a former path of Llama Chat in the code-review skill
Changes under the old tools/ui path, as well as tools/server/webui, still
get the Llama Chat checklist.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name Llama Chat's path in the security policy's exclusions
Llama Chat no longer lives under tools/server/, so the exclusion listed
under it now names ui/apps/chat. What is covered stays the same.
Assisted-by: Claude (Opus 5.5 Medium)
The flash attention shaders indexed the mask without its head
dimension, so masks with ne[2] > 1 gave wrong results. The mask offset
and the vec block states now include the head, broadcast as
head % mask->ne[2] like the other backends. The params buffer grows to
256 bytes to fit the two new mask fields.
The shared memory estimate dropped kv_shmem when only one of K and V
is read directly, while the shaders keep it unless both are, which made
the K f16 / V q8_0 cases exceed the workgroup storage limit. The vec kv
tile now takes both flags: the memory estimate drops kv_shmem only when
both are direct, and the tile still divides the KV padding when either
is. get_alloc_size and supports_op use the same rule as the pipelines,
and supports_op accounts for the register tile fallback.
* ci : use vX.Y.Z tags as docker tags
When the Docker workflow is manually dispatched from a vX.Y.Z tag, use that tag verbatim for the image tags instead of appending the build number and commit hash.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ci : require a tag for manual docker runs
Manual dispatches of the Docker workflow must now use a tag ref, so the image tags are derived from the selected tag. The check can be skipped with the new skip_ref_check input. Scheduled nightly builds are unchanged.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* vulkan: make the radix top-k tie selection deterministic
The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.
Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.
* optimize shader for AMD, Intel and Nvidia
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.
Fixes#30576
Assisted-by: GitHub Copilot
Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.
Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.
Also trim the output_config.effort test to a single level.
* ggml-cpu: use VNNI for Q2_0 dot products
Keep the upstream AVX2 unpacking, accumulation, and fallback. Use VNNI for the integer dot product on supported x86 CPUs.
Assisted-by: Codex
* ggml-cpu: use VBMI for Q2_0 unpacking
Use multishift to unpack Q2_0 weights on CPUs with AVX-512 VBMI and VL. Retain the existing shift-and-mask unpacking on other CPUs.
Assisted-by: Codex
* Update ggml/src/ggml-cpu/arch/x86/quants.c
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* fixed support for DFlash2 for Ripple 65B
* spec: Add NoPE handling to DFlash
(cherry picked from commit 89ea8e3e77e3359d17902a2db136022d5d419d88)
* spec: Fix NoPE not being picked up by DFlash
(cherry picked from commit 43393b95fda2ce7e7d243e2650bf7b1cadf1c732)
---------
Co-authored-by: James Pickering <james_pic@hotmail.com>
* Fail tests on missing or unsupported type
* Fix output
* Remove whitespace
* Fail if no tests ran regardless of backend support
* Revert unnecessary cosmetic change
* lfm3 : put the normed tensor first in residual adds to enable norm fusion
Assisted-by: Claude Opus 5.5
* tests : update Metal fusion baseline for lfm3
Assisted-by: Claude Opus 5.5
* convert : add Inkling2 text GGUF conversion
Assisted-by: Codex
* llama : implement Inkling2 text inference
Assisted-by: Codex
* tests : cover Inkling2 text support
Assisted-by: Codex
* conversion : dequantize packed Inkling2 experts in row chunks
Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.
Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.
Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.
Assisted-by: Codex
* convert : add Inkling2 vision mmproj support
Assisted-by: Codex
* mtmd : implement Inkling2 still-image encoder
Assisted-by: Codex
* inkling2 : scope image loading
Assisted-by: Codex
* inkling2 : enable flash attention with folded bias and sinks
Assisted-by: Codex
* cuda : route per-head flash attention masks through vec
Assisted-by: Codex
* mtmd : support Inkling2 temporal frame means
Assisted-by: Codex
* style : restore shared image comment
Assisted-by: Codex
* tests : enlarge Inkling2 context fixture
Assisted-by: Codex
* inkling2 : load per-layer expert feed-forward widths
Assisted-by: Codex
* tests : add Inkling2 Metal fusion baselines
Assisted-by: Codex
* inkling2 : group and align registry entries
Assisted-by: Fable
* convert : regenerate the Inkling2 tokenizer hash with the update script
Assisted-by: Fable
* conversion : revert the print_registered_models change
Assisted-by: Fable
* conversion : sort the Inkling2 model map entries
Assisted-by: Fable
* convert : map Inkling2 vision checkpoint names in the tensor table
Assisted-by: Fable
* convert : map Inkling2 text checkpoint names in the tensor table
Assisted-by: Fable
* tests : document the Inkling2 packed expert conversion test
Assisted-by: Fable
* tests : cover Inkling2 checkpoint and transformers tensor names
Assisted-by: Fable
* llama : place Inkling2 fields and list entries by file convention
Assisted-by: Fable
* gguf-py : align Inkling2 tensor mapping comments
Assisted-by: Fable
* mtmd : place Inkling2 entries by file convention
Assisted-by: Fable
* tests : place Inkling2 cases after the existing ones
Assisted-by: Fable
* cuda : index per-head flash attention masks in the tile and mma kernels
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : project the relative bias before the gather, drop arch branches
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* convert : require Inkling2 hparams, add inkling2 pre-tokenizer
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* mtmd : cap Inkling2 image size, drop video frame averaging
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : fix CI build, Meta device assert and Windows conversion test
Assisted-by: Claude
* tests : use the default sliding window for the Inkling2 test model
Assisted-by: Claude
* simplify mtmd part
* give _v arch for vision
* remove alpha channel handling, it should be a follow up
* explicit rule about audio + vision models
* add audio support
* rm audio
* rm gguf mtmd rope base
* address review comment
* nits
* do not normalize embd from mtmd
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* common : reject out-of-range grammar_triggers type instead of aborting (#30835)
* tests : remove the invalid grammar trigger type regression test (#30835)
* vulkan: Explicitly include SPIRV-Headers
Distros that don't put the headers in a system include path need
to specify it to the compiler. So we use the appropriate CMake
target to include it.
Assisted-by: Codex
* vulkan: Add explicit headers location to workaround OSX CI runner conf
* vulkan: optimize Flash Attention cm1 for RDNA3 and RDNA4
* reduce shmem use through buffer reuse
* clean up and fixes
* limit q_in_regs to prefill
* vulkan: add flipped FA path, each subgroup owns 16 Q rows
* cleanup
* move flip shader into own file
* replace kvsep with direct load mode
* fix nvidia hang, tune AMD
* only store pv in sfsh if necessary
* fix skip condition
* nvidia tuning
* reduce occupancy on RDNA4 for large attention head sizes
* fix sparse FA fp16 underflow bug
Template soft_max_f32_parallelize_cols_single_row to keep column
elements in registers when they fit, reducing memory traffic from
5 global memory passes to 2 passes.
Assisted-by: Gemini 3.8 Flash
* vulkan: optimize matrix multiplication for narrow N
Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.
* Attempt to handle wave64
* add n=16 tile for cm1
* apply RDNA split_k heuristic to narrow_n generally
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* chat: allow tool calls inside reasoning blocks for Ministral-3 and Qwen3
Fixes#29248Fixes#20837
* chat: address review feedback for Ministral-3 reasoning end
- Remove <function= from Qwen3 (intentionally not used outside outdated coder models)
- Ministral-3: use choice between [/THINK] and peek([TOOL_CALLS]) instead of optional
* chat: add [TOOL_CALLS] to Ministral-3 thinking_end_tags
* chat: simplify Ministral-3 reasoning end choice
* chat: restore peek lookahead for Ministral-3 tool call start
* chat: keep [TOOL_CALLS] only in parser, not in thinking_end_tags
* cont : revert tool call tag removal and update tests
---------
Co-authored-by: Alen Požun <kukirin-ui@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
The thought channel is normally closed with <channel|>, but the model
may open it and go straight to a tool call without closing it. Stop the
reasoning at <channel|> OR <|tool_call>, so an unclosed thought does
not swallow the following tool call.
Fixes: #32231
Assisted-by: Qwen 3.8 27b
The qwen2vl graph passed NULL for the ffn_up/gate/down scales, so the
per-tensor NVFP4 scales loaded by the generic pass were never applied
and the model produced garbage (PPL ~1e6 on CUDA, NaN on CPU).
Pass the scales to build_ffn like qwen2 does.
Fixes#32101
* mtmd : allow custom video decoder
* also adapt to wllama requirements
* clean up
* add test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* dflash : move the DSpark markov head and backbone graphs to dspark.cpp
Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and
graph_mistral_large4 move verbatim. build_inp_target_features and
build_dspark_markov_head become static members of llama_model_dflash so
both files can use them.
* dflash : detect the head and the backbone once, split the loaders per file
load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2,
DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4).
The head, backbone and base layer loaders become member helpers in
dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph
dispatches on the backbone.
The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through
a helper in dflash2.cpp, right after their head tensors instead of in
the base layer loop; the other tensors keep their creation order. The
DFlash2 loader drops its copy of the conv group size check, which master
never reached.
* dflash : share the injection input, token embeddings and lm head lookups
build_inject_inp (target features + fc + encoder norm), get_tok_embd and
get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral
Large 4 graphs. build_inp_target_features is file-static again.
* dflash : split the DFlash decoder into a graph per head
graph_base creates the shared inputs and provides build_inject,
build_inp_tokens, build_layers and build_logits. graph_dflash
(dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and
graph_lilicorr (lilicorr.cpp) assemble them and add their head; the
selector, scorer and markov head builders move to the file of their
graph, and the DFlash2 local conv run by build_layers to dflash2.cpp.
build_arch_graph dispatches on the backbone, then on the head.
The op order and the tensor names are unchanged.
* speculative : split the DFlash driver into a class per head
common_speculative_impl_draft_dflash stays the DFlash implementation and
becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its
protected constructor only reads the draft metadata; init() then sets up
the contexts with the options of the draft (draft length, backend
sampling, device features, masked h_nextn). Each derived class overrides
draft() with the path of its head from the former draft(). DFlash2 also
overrides begin() to reseed its generator per request, and LiliCorr
overrides process() to store the context features of each injected
chunk, which its noise block carries as its state.
The factory keeps the current routing: draft-dspark picks DSpark,
draft-dflash picks DFlash2 or LiliCorr from the draft metadata and
DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs
the DSpark reader instead of a mix of both. sample_from_anchor is logged
by DSpark only.
* speculative : add the draft-dflash2 and draft-lilicorr types
draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts,
which draft-dflash keeps running too: it picks them from the draft
metadata. draft-dflash2 and draft-lilicorr must match the draft model,
draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a
DFlash draft is rejected, and a run uses at most one DFlash family type;
these errors leave speculative decoding uninitialized.
The type detected from the draft GGUF is now the concrete one, and
draft-dflash2 and draft-lilicorr download the dflash sidecar.
* docs : describe the DFlash2 and LiliCorr drafts
Add the DFlash2 and LiliCorr sections and the new types, and explain
that draft-dflash also runs DFlash2 and LiliCorr drafts.
* speculative : drop the unused DFlash is_mrope
Nothing reads it since the migration to common_batch (#29385), which
picks the number of M-RoPE position rows from the rope type of the draft.
* doc: update spec dec
* address review comments
* requirements : support Python 3.15
torch 2.11.0, numpy 2.4, matplotlib 3.10, pandas 2.3, pillow 11.3 and
cffi 1.16 have no Python 3.15 wheels, so the CI runner on 3.15 cannot
install the requirements.
Python 3.15 gets the first releases that ship 3.15 wheels through
version markers, and every version below 3.15 stays as is.
pyproject.toml now allows 3.15 and picks torch 2.15.0 there.
* requirements : apply review suggestions
numpy 2.5.4 from Python 3.14, matplotlib 3.11.2 and pandas 3.0.6 from
3.11, cffi 2.1.1 and pillow 12.3.0 for every version. cffi 1.16.0 has
no 3.14 wheel and fails to build there, so this also fixes the
tokenizer test requirements on 3.14.
* hex-bufs: combine queue and fence buffers
* hex-bufs: use n_threads for queue sizing
* hex-mmap: optimize buffer stacking at the host to avoid iova fragmentation
* hex-mmap: add simply retry for spurios mmap failures
* hex-mmap: use eager mode for extended mmap retries
* vulkan: add TOP_K_MASK
Assisted-by: Claude Opus 5.5
* vulkan: fix the TOP_K_MASK barrier and use the common offset helper
Assisted-by: Claude Opus 5.5
* vulkan: declare ggml_vk_top_k_mask in the common header
Assisted-by: Claude Opus 5.5
* spec : skip p_min for sampled MTP and DFlash2 drafts
Under rejection sampling the acceptance at a position is 1 - TV(p, q),
which the draft's top probability does not tell, so the p_min cutoff
dropped draft tokens the target would have accepted. Skip it when
MTP or DFlash2 samples the draft. draft-simple keeps it: each extra
draft token costs a full draft model pass.
* spec : replace the sampled-draft p_min skip with a warning
p_min behaves as on master again for every drafter and sampling mode.
Instead, log a warning when --spec-draft-p-min > 0 is used with
--spec-draft-sampling probabilistic, suggesting p_min 0: under
rejection sampling the draft's top probability does not tell how much
of the draft the target accepts. The cut is not skipped because with
MTP at long drafts it can still pay off.
* spec : warn about p_min only for drafters that use it
The n-gram drafters ignore p_min, and DSpark uses it as the cutoff of its confidence head, so the warning is skipped when only these are configured.
---------
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* grammar : limit the number of alternatives
* grammar : introduce an AST for grammar parsing
* grammar : revert AST refactor and alternatives limit
This reverts commits:
- 84f3b57c4d6c482e7187a30735d16b02de3784fa
- 393a707ebb7e6f798abdb8af34cb1bc102f4aaa6
* grammar: optimize big grammar matching by using left-factorization
* grammar : preserve character class boundaries during left factoring
* grammar : prevent infinite expansion during left factoring
* grammar : use ASCII characters in comments
* grammar : test empty suffixes after left factoring
* grammar : deduplicate left factoring logic
* grammar : add comments
* grammar : use number parameters and clarify RSS output
* grammar : clarify character class test names
* grammar : remove performance benchmark
* grammar : share immutable rules
---------
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
The kernel launches one block per row and 256-wide tile, so the
{2, 16777217} test case asks for 16777217 blocks of 256 threads. HIP
refuses a grid dimension above 2^32 threads and the ROCm jobs fail
with "invalid configuration argument".
The host now launches the grid in chunks of at most UINT32_MAX / 256
blocks and passes the chunk offset to the kernel. Real sizes fit in a
single chunk and keep their timings.
This commit removes the llama version test-cmake and instead uses the
version that is supplied by CMAKE_PREFIX_PATH.
The motivation for this is to avoid issues when bumping major versions.
The D=256 vec flash attention kernel with F8_E4M3 K and V uses 257
VGPRs (255 + 2 spilled) on gfx908, one over the limit of the check.
The spill is a perf issue only, like the Q4_0 D=256 vec kernel
already in the ignore list.
Both variants of the kernel join the ignore list.
[no ci]
* server: read a small temp file in the read_file tool test
test_tools_builtin_read_file read its own file, which grows with every
added test. On Windows the CRLF checkout puts it at 16664 bytes, over
the 16384 byte read limit, and the tool refuses it.
The test now writes a small file of its own and checks the exact
content read back.
* server: use a temp cwd in the x-tool-cwd tool test
test_tools_builtin_cwd_header read test_tools_builtin.py through the
x-tool-cwd header, which hits the same 16384 byte read limit on the
Windows CRLF checkout, and wrote its shell marker into the source tree.
The test now uses a temp directory as cwd, reads a small file from it
and checks the shell marker lands there.
The vision tests download their images from huggingface.co, which
resolves to a private address on the runners inside the HF network.
The media URL check refuses it with "host is not a public address"
and the server-cuda job fails.
The vision fixture enables media_allow_private_urls. The private
address check keeps its own coverage in
test_vision_image_url_private_address.
The same-origin tests over the LAN address block forever on the Metal
runners: the request to the host's own LAN address never returns and
has no timeout, so the job runs until the 6h limit.
get_lan_ip() now probes /health on that address with a short timeout
and returns None when the host cannot reach it, which skips the test.
The test requests themselves use the default request timeout so a
hang fails the test instead of stalling the job.
* fix(hexagon): make host build work with clang-cl on Windows ARM64
Two CMake fixes for building ggml-hexagon natively on Windows-on-ARM64
with x64-hosted clang-cl (validated on Snapdragon X Elite, discussion in
qualcomm/GenieX#1390):
- Some clang drivers (e.g. x64-hosted toolchains used on ARM64) don't
predefine the arch macro, which makes winnt.h fail with "No Target
Architecture". Define _M_ARM64/_ARM64_ explicitly for the host TUs.
- The Hexagon SDK's incs/stddef include (AEEStdDef.h and friends) poisons
the MSVC header chain when it leaks onto dependent TUs' include paths
via htp_iface's PUBLIC includes. Keep it PRIVATE to the interface stub
and system-level for the backend's own TUs (which include <AEEStdErr.h>
directly).
Originally validated 2026-08-26 as reported in the issue; applies
cleanly to 15759e0.
* hex-cmake: trim down excessive comments
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* llama : remove deprecated `llama_set_warmup`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : remove remaining deprecated API
Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* fit-params : remove stale `llama-ext.h` include
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : document `llama_model_dflash_selector_top_k`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : promote the NextN / speculative API from `llama-ext.h`
Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : polish the NextN API promotion
Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : prune deprecated API
* cont : bump RPC version
* cont : minor
* ggml : remove outdated documentation from ggml.h
The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:
- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`
The block also ended with 5 empty `TODO` sections.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)
* rpc : bump RPC protocol major version
The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.
The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: losewayy <1134285612@qq.com>
* CUDA: few-row MMQ for 2-24 src1 columns
New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.
* nit fixes
* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat
* few-row quantize: DS4 sum from the quantized values, as in #31247
* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build
* few-row MMQ: dispatch order MMVQ, few-row, MMQ
With --sleep-idle-seconds, a handler passes wait_until_no_sleep()
while the server is awake, then keeps using the server state, for
example to parse the body and tokenize. If the idle timer expires in
that window, the main loop enters sleep and the callbacks destroy the
model: the handler reads freed vocab memory, and a task posted after
that stays in the queue with nobody awake to process it.
Count the requests that passed wait_until_no_sleep() and are not done
yet, and do not enter sleep while the count is positive. The response
generator releases its count in its destructor, so every return path
of a handler gives it back.
Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
HunyuanVL resamples its position embedding on the host into a float
buffer sized for F32, then reads ggml_nbytes() of the tensor into it.
The graph never uses the tensor, so any type loads, and an F64 tensor
overflows the buffer at the first image request.
Require the position embedding to be present and F32 when loading the
HunyuanVL tensors.
The MiniCPM-V resampler ends with build_mm(mm_model_proj), whose output
width is ne[1], while n_mmproj_embd() returns ne[0]. Both are equal in
real models, the projection is square. A non square projection passes
the text model n_embd check and only aborts at the encode, on the
output buffer size check.
Return ne[1], so the width matches the graph output.
* mtmd: require the MLP projector and patch embedding tensors
The loader marks the patch embedding and the LLaVA projector tensors
as optional, while the graphs and n_mmproj_embd() dereference them
unconditionally. A GGUF without them crashes on a null tensor during
the warmup.
Load the patch embedding as required for the vision encoders that
start with it, which is all of them except GEMMA3NV, DEEPSEEKOCR and
DEEPSEEKOCR2. Load mm.0 and mm.2 as required for MLP, and mm.0, mm.1,
mm.3 and mm.4 for the Yi-type MLP_NORM variant, selected by mm.3.
The graph no longer tests mm.2 and the yasa2 assert on the patch
embedding is covered by the loader.
* mtmd: keep the Yi-type llava TODO
The Yi-type variant is still detected by the presence of mm.3, the
TODO on that detection stays.
The Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Ling3-VL graphs build their
input with a temporal merge that assumes a 2x2 patch grid, and the
Qwen2-VL merger reshapes by 4. Any other spatial_merge_size from the
GGUF reaches a GGML_ASSERT at the first image request and aborts the
server.
Reject a spatial_merge_size other than 2 when loading the hparams.
The SpacemiT compute case was commented out, so the op stayed on the scalar CPU path. Type dispatch stays with the other ops, and use_ref keeps the reference path on the generic CPU kernel.
Assisted-by: Codex
Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
* vulkan : Intel customized FA shader for prefill usage
* vulkan : adding check of max_bias/logit_softcap for Intel FA kernel path
* vulkan : add m16n16k16/SIMD32 support for Xe FA prefill shader
* vulkan : Intel FA shader and host refinement, with accuracy issues fixes
* vulkan : Limit the SLM usage to 48 KiB for Intel prefill FA shader
* vulkan : Update Intel FA shaders to handle cases with large head dimension K, unaligned Q head stride and multi head mask cases
* vulkan : Intel FA shader and host refinement to align with existing design, extreme cases handling, shape check relaxing.
* vulkan : Intel FA shader and host update to support q as tensor view, unaligned batch stride q. Add env variable for enforcing XE FA path for other vendor GPUs with 16x16x16 CM1 and SIMD32 support
* vulkan : Inte FA shader and host update to fix the potential SLM over 48 KiB issue, enhance the buffer access robust and add asymmetric qk/pv ration support.
* vulkan : Inte FA shader and host update to add SLM 48 KiB size check, remove unnecessary extensions and finetune the unroll/dont_unroll loops
* vulkan : Intel FA shader and host update to add SLM size check function, use real device max SLM size in shader specialization constants.
* vulkan : Intel FA shader and host update to consolidate host helper function, remove redundant buffer sync, bypass unused pipeline and buffer, fix subgroup shmem conflict and add negative case handling.
* vulkan : Intel FA host update to correct get_xe_fa_tuning_params() function log message, and add workgroupMemoryExplicitLayout/workgroupMemoryExplicitLayout16BitAccess prerequisite feature check.
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back
* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ
* ggml-cuda: Tests for the fused scale
* ggml-cuda: reuse existing graph to test fusion in mmq epiloque
* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency
* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions
* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
* ggml-backend : prefetch only inputs computed by earlier splits
The copy callback of a split runs after the prefetch and can write tensors that a later split reads, e.g. the MoE cache uploads missed experts into its bank. A prefetch of such a tensor reads it before the write.
* llama : skip the MoE cache when its device cannot run the cached mul_mat_id
Otherwise the scheduler runs the matmul on another backend and copies the whole bank every ubatch.
The 32x256 pack already keeps a separate uint8 zero point. Complete the M1 path and use the same HP kernel as Q4_0.
Assisted-by: Codex
Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
* ui: add search_chats tool
* add from_timestamp / to_timestamp
* ui: search_chats review fixes
- keep search_chats disabled on import, copy tool policy on fork
- stream messages without attachment payloads, one transaction for time ranges
- keep the output within budget, cut whole messages from the older end
- reject impossible date-times, quote titles and attachment names
- typed readOnly tables, move localIsoString to formatters as formatLocalIso
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
npm 11.20 refuses git dependencies by default (EALLOWGIT), so npm ci
failed on fresh installs. Pin llama-brand to the same commit through
an HTTPS tarball, which also adds an integrity hash to the lockfile.
Assisted-by: Claude (Opus 5.5 Medium)
Each graph result reserves metadata for max_nodes, but many graphs use
only a fraction of that capacity. The vector resize initializes the
whole reserve and makes otherwise unused pages resident. Allocate the
reserve with new uint8_t[] instead.
* fix aot for >4g garbage output issue; add --aot in build script
* correct the aot flag
* support no-xmx build, update the code for no-xmx building, rename nossl to no-ssl
ggml_sycl_set_main_device() runs at the start of every graph compute. Its
early-return check called dpct::get_current_device_id(), which locks the
device manager mutex and looks up the thread-to-device map. Use the
thread-local cached id that the rest of the backend already uses.
Assisted-by: Claude Code
Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
* CUDA: skip empty MoE tiles in MMQ stream-k fixup
mul_mat_q_stream_k_fixup loads ids_dst[col_low + j] for all j < J before
checking j_sup. mul_mat_q returns early on col_low == col_high, the fixup
kernel did not. mm_ids_helper emits one extra tile per expert that may be
empty; for trailing experts col_low == ncols_dst, so the load reads one
element past the J_best-1 padding added in #29953. The kernel returns at
j >= j_sup before storing anything, so results are unaffected.
Return early when j_sup <= 0, like the main kernel. The return is
block-uniform (tsj is loaded once per block) and precedes __syncthreads().
Add test_mul_mat_id_unused_tail to test-backend-ops: only the first
n_mats/2 experts are selected, so the trailing experts have zero rows and
their tiles are empty; m=33, k=512 keeps tile efficiency low enough that
stream-k splits along K and launches the fixup kernel.
* CUDA: drop the extra MUL_MAT_ID test case and the explanatory comment (review)
test_router_restricted_cors_same_origin_reaches_child still named
ggml-org/tinygemma3-GGUF, which the router no longer finds since the test
models moved to ggml-org-dev, so the Server test job fails on master.
Assisted-by: Claude (Opus 5.5 Medium)
* opencl: add decomposed FA
Assisted-by: Claude Opus 5.5
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: build FA MQ kernel on first FA call
* so allows warmup to early build neccessary kernels
Assisted-by: Claude Opus 5.5
* opencl: handle inf for fully masked row
* opencl: refine condition for using fa decomp
* opencl: support dp4a for PV mm
Assisted-by: Codex
* opencl: add direct PV for FA decomp to avoid V transpose for small batch
Assisted-by: Codex
* opencl: enable direct pv by default
* opencl: expand X2E decomposed FA to smaller heads and batches
- Lower the default head dimension threshold from 256 to 128 on X2E.
- Allow DK64 with P8, no attention sinks, GQA >= 2 and n_kv >= max(1024, 2*n_q).
- Allow P8 with V transpose for 2-31 queries when direct PV is unavailable, DK >= 128 and n_kv >= 1024.
Assisted-by: Codex
* opencl: adjust format
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* ui : keep downloads in Discover sidebar during search, sort base-org repos first
- Show the sticky Downloading section regardless of an active search query
- Order search results so repos published by their base model's org lead
Assisted-by: Claude (Opus 5.5 Medium)
* server : throttle download progress per file
Files of a download run in parallel threads, but progress updates were
throttled with one shared timer. A small draft sidecar reports far more
often than the main weights, so it took nearly every slot and the main
file's updates, including the first one carrying its total size, were
dropped until the draft finished. The UI then showed the draft's progress
first and restarted from 0% for the model.
Throttle each file on its own and always pass a file's first update.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list a downloading repo once and match base orgs case-insensitively
A search hit that is downloading already shows under Downloading, so the
results leave it out. Base model orgs compare without case, as the hub
treats them.
Assisted-by: Claude (Opus 5.5 Medium)
* server : tell clients a download stopped on request
When a client paused or cancelled a download, every other client following
/models/sse got a bare download_failed and could not tell it from an error,
so the web UI showed "Download failed" for a pause made in the Llama app.
A stop asked for through unload or delete now fails with {"stopped": true}.
Deleting a paused download, which has no instance left, now sends
model_remove as deleting a running one does, so other clients drop its row.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : follow downloads other clients start, pause or cancel
A download another client started (e.g. the Llama app) already showed up
from the feed; it now also picks up the sidecars the list reports for it.
A stop another client asked for reads as a pause instead of a failure, and
a removal clears the download row even when the list does not carry it yet.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : take the logo and favicons from llama-brand
The web UI's logo mark and favicons now come from llama-brand, added as a
dev dependency from GitHub and pinned to a commit, so the UI draws the
same icon as the README cover instead of its own copy of an older one.
Logo.svelte inlines the brand's icon and fills it with currentColor, and
the PWA asset generator starts from the brand's web favicons, which carry
their own margin and optical centring, so the UI's colouring and padding
script, its test and its logo.svg go.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : follow llama-brand's master instead of a pinned commit
Assisted-by: Claude (Opus 5.5 Medium)
* server: fix --cors-origins localhost handling
* server: validate Host on loopback connections against DNS rebinding
A page whose own name resolves to 127.0.0.1 reaches the server with
that name in Host and a matching Origin, so the same-origin shortcut
let it through. On a loopback connection, Host must now name loopback
or the host of an allowed --cors-origins entry. The LAN tests reach
the server through a real interface instead of a forged Host.
* fix python type check
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Allocate gf_res_reserve and both gf_res_prev entries once in the llama_context constructor. Add llm_graph_result::set_max_nodes() and reset the existing instances in sched_reserve() instead of recreating them. get_gf_res_prev() now only selects the arena for the current batch.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
- remove build_delta_net_chunking and build_delta_net_autoregressive
- always use the fused gated delta net
- make graph_max_nodes independent of the ubatch size
- drop the fused GDN probes and fused node tags
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
ggml-sycl.dll built with oneAPI 2026.2 imports mkl_intel_thread.3.dll and libimalloc.dll, which are not part of the release package. LoadLibrary fails and the SYCL backend is silently skipped, llama.cpp falls back to CPU.
Adding both DLLs fixes the load: measured on Arc B570 with LFM2.5-8B-A1B Q4_K_M, pp4096 goes from 404 t/s (CPU fallback) to 2232 t/s (SYCL) with the extracted b12244 package.
Assisted-by: GLM (z-ai/glm-5.3-flash)
Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com>
* CUDA: re-instantiate graphs when the cluster dims of a kernel change
* CUDA: keep the cluster dims with the graph instance, check them only on Hopper+
Assisted-by: Claude Opus 5.5
* CUDA: use CUDA_CHECK and a helper struct for the graph cluster dims
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CUDA: add a TODO for the graph cluster dims test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call
Assisted-by: Claude Opus 5
* ROCm/HIP: corrected by only setting the workspace on ROCm 7.0 or newer
* cpu: accumulate flash attention V in F32 in the one-chunk kernel
The one-chunk kernel accumulated an F16 V cache in an F16 running sum
that is only normalized at the end, which overflows past 65504 on long
contexts: inf, then NaN on every following token.
Every V type now accumulates in F32, like the tiled kernel. F16 rows go
through ggml_vec_mad_f16_f32, which converts and multiplies in the same
register (F16C/AVX512, NEON, RVV) with no round trip through a temporary
F32 row, so decode keeps master speed on x86 and loses about 4% at 4k
context on ARM, where master used native F16 FMA.
* cpu: widen the F16 V rows like ggml_vec_dot_f16_f32
ggml_vec_mad_f16_f32 required F16C and FMA, so AVX builds without FMA
fell back to a scalar loop and ran 3x slower than master. It now
follows the guards of ggml_vec_dot_f16_f32: AVX512, then AVX with F16C
through GGML_F32x8_FMA, then NEON and RVV, then the scalar loop.
* additional FA test cases
Assisted-by: Codex
* vulkan: allow dequantized KV optimization with CM2
Enable the KV dequantization optimization with coopmat2, guarded by a heuristic
based on query tokens*GQA ratio.
While doing this we saw very large speedups for certain head sizes, and it was
because those head sizes were incorrectly treated as unaligned. So also fix
those to be treated as aligned, which should help for when the dequant
optimization doesn't apply.
* gqa uses nbq2 for the stride, so we need to check that value for alignment
* tests : use VARS_TO_STR18 for flash attention
Assisted-by: Codex
CodeMirror takes text input through EditContext on Chrome for Android
only, and there the chat editor never brings up the virtual keyboard.
The editor now always takes its input through contenteditable, as on
desktop and iOS.
A press in the input box outside the text also focuses the editor, in
the collapsed form too, so the whole box is a tap target again as the
textarea was.
MMF tiles 32 rows per block, so a BF16 matrix with few rows (e.g. the
qwen4exp hyper-connection projections, 320 rows with K = 10240) runs only
a handful of blocks and leaves most SMs idle at batch sizes 2..8. MMVF
runs a block per row and wins there when K is long enough.
MMVF's cost grows with rows x batch while MMF's stays roughly flat in that
range, so use MMVF only while rows * ne11 * 256 <= nsm * min(K, max_k) and
K >= 1024. The budget scales with the SM count, which keeps GPUs with few
SMs (e.g. the A16) on MMF. Blackwell allows up to 8 columns and
max_k = 6144; older GPUs stop at 5 columns and max_k = 4096.
Add eval cases at 128x2560, 512x2560 and 320x10240 for n = 2, 3, 4, 5 and 8.
* CUDA : add sm70 FlashAttention config case for the 256x256 head
* CUDA : tune sm70 FlashAttention for 256x256 ncols=32
Assisted-by: Codex
---------
Co-authored-by: Jirka Svítil <mistrjirka@users.noreply.github.com>
* cuda : add MMA FlashAttention support for head sizes 40 and 72
Head sizes 40 and 72 (used e.g. in vision towers of Gemma 3/4 and Qwen3-VL) were excluded from every tensor-core dispatch branch and fell back to the tile kernel.
The fp16 MMA fragment covers 16 halfs in the head direction, so the in-kernel geometry is padded to the next multiple of 16 (40 -> 48, 72 -> 80). Register arrays, shared memory tile strides and MMA loop bounds use the padded size, while global memory loads and stores the real one.
Enabled on NVIDIA Turing and newer while Volta and AMD keep the tile kernel. The RDNA/CDNA config rows and the mma_tile_sizes specialisations are needed because the explicit instantiations are compiled for every target of a translation unit.
* tests : add the head size = 40 encoder perf shape
* cuda : address review comments on the head size 40 and 72 kernels
* Initial CUDA WIP version
* Use fp8 intrinsics
* Do W8A16 for FP8 in GEMV case
(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB
| model | size | params | backend | ngl | n_ubatch | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 | 2781.53 ± 8.48 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 | 37.56 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 @ d32000 | 1616.31 ± 2.61 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 @ d32000 | 34.11 ± 0.03 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 | 2542.00 ± 0.51 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 | 37.36 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 @ d32000 | 1762.86 ± 1.33 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 @ d32000 | 34.09 ± 0.02 |
* Optimize GEMV kernel for FP8
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2
Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099
* Remove trailing whitespace
* Add supported for strided copies to/from FP8 in CUDA
* Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh
* Fix odd-rows for convert/getrows, possible with FP8
* Fix FP8 fallback in cuBlas
* Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8
* Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte
* Add todo about accumulator/precision honoring
This is beyond the scope of the stacked PR
* Avoid races in fp8_fallback
* Allow Hopper and other BW SKUs for FP8
* Use intrinsic over explicit manual conversions
* Resolve TODO
As per spec, we don't have to preserve signs
* Add commend on cublaslt workspaces
* Use convert_unary_cont_cuda over block dequant in convert.cu
* Use get_rows_cuda_float for fp8 kv cache
* Make FP8 copies work on all IHV's
* More cleanup
* Simplify by falling back to shared cuBLAS helper
* Fuse weight_scale multiplication for FP8 also
* More code-cleanup
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4
Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix
* Use F32 activations in fallback mmvf path
* Explicitly link against cublaslt
* Move FP8 conversion intrinsics into ggml_cuda_cast
Cleaner this way
* WIP src1/prec for fp8 in MMVF
* Remove comment
* Keep only f8 x f8 KV cache for now
* Just do FP8 x FP32 x FP32 in MMVF as its fast enough
* Update ggml/src/ggml-cuda/ggml-cuda.cu
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
* Add batched FP8 GEMMS in cublas as suggested by code review
https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644
* Extract fp8x2 -> float2 load into helper
* Searapte copy checks to/from fp8
* Compiler massaging to avoid repeated address offsets
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Modify `ggml_sycl_op_clamp` to use the unary helper function
`ggml_sycl_detail::ggml_sycl_op_unary` to handle non-contigous or
strided views. The fix uses a pattern that matches other stride-safe
unary ops, and is a simpler execution path.
Deleted dead template `unary_op_clamp_kernel`, the flat `clamp` kernel
launcher from element_wise.cpp, and deleted (now) unused
`SYCL_CLAMP_BLOCK_SIZE` from presets.hpp.
The hidden desktop strip relied on inherited `visibility: hidden` to keep
its content from catching clicks. Since the sidebar's sections fold
(#31936), an open section's CollapsibleRegion sets `visibility: visible`
again, and its rows' context-menu triggers carry an inline
`pointer-events: auto`, so they caught clicks meant for the page.
Make the hidden strip's content inert, which no descendant can undo, and
add a client test that hit-tests the hidden sidebar's area.
Fixes#31971
Assisted-by: Claude (Opus 5.5 Medium)
* media : remove the legacy llama0 and llama1 brand assets
The brand assets live in ggml-org/llama.brand; none of these is referenced
in the repository.
Assisted by: Claude (Opus 5.5 Medium)
* readme : use the new llama.cpp cover from llama.brand
Assisted by: Claude (Opus 5.5 Medium)
* ui: show audio in Discover for models whose mmproj has an audio encoder
Discover took audio only from the pipeline tag, so omni models tagged
any-to-any, such as Gemma 4 and LFM 3 Omni, showed vision but never audio,
though their mmproj carries an audio encoder.
Read the encoders from the mmproj's GGUF metadata instead: a ranged read of
its first 64 KB, once per repo, for clip.has_vision_encoder and
clip.has_audio_encoder. The Discover list and details now show audio when
either the pipeline tag or the mmproj says so.
Assisted by: Claude (Opus 5.5 Medium)
* ui: take Discover modalities from the mmproj's encoders in one place
Review fixes for the mmproj header read:
- drop vision for audio-only mmprojs: the converter writes
clip.has_vision_encoder only when true, so a header read in full without
it has no vision encoder; a header cut short leaves vision as it was
- accept only a 206 to the ranged read, so a server that ignores Range
never streams the whole mmproj
- encode the mmproj path in the resolve URL
- derive the list and details modalities with one HuggingFaceService
helper, so both views agree
- point the key comment at tools/mtmd/mtmd-impl.h, where the keys live now
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a gated repo's mmproj from being read again on every mount
A gated, private or missing mmproj answers 401, 403 or 404 to every read,
so caching nothing made each mount of its Discover row request it again and
log an error. Cache it as unknown instead: no audio, and vision from the
mmproj being there, as before the header was read.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move Models, MCP and Settings to a sidebar footer
The sidebar's actions split in two: New chat, Search and Projects stay at
its top, and Models, MCP and Settings move to a footer at its bottom, under
a thin rule, in the expanded sidebar and its collapsed strip alike. On a
phone the footer keeps clear of the home indicator, and Models stays
desktop-only as before. The search dialog still lists every action.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : group the sidebar footer in an inset surface
The design system gains inset-group-surface, for rows grouped inside a glass
panel: white on the light glass and a faint tint of the text colour on the
dark one, with the sidebar's own ambient edge, no inner highlight, a faint
wide shade under it and the card radius. It is matte-glass pointed at its
own component tokens, like the sidebar and chat form surfaces, and reads as
part of the panel: no tight contact shadow or solid border like a button,
no inset shade like a field.
The sidebar footer uses it in place of the rule above Models, MCP and
Settings. DESIGN.md and the Surfaces page document it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : give the sidebar's logo button the add button's key look
Whenever the sidebar shows, open or peeking, its logo button takes the same
key as the chat form's add button (matte-key, white in light mode, a light
tint in dark), in place of its own muted fill. While the sidebar is hidden
the logo has no fill and no shadow, and hover and focus never change the
fill. It keeps its round shape and its own size.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : put search in a round key at the top of the phone sidebar
On a phone the sidebar's Search row moves to a round key at the right end
of its top row, across from the logo, with the same key look as the logo
and the chat form's add button. It opens search as the row did. Desktop
keeps the Search row.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : size the phone sidebar for a thumb and list its footer with the actions
On a phone the sidebar's actions and rows are 48px tall at 17px, flush one under
the other with 16px corners, its section headings 15px and grey, and its groups
28px apart. MCP and Settings join the top actions there: the footer is a desktop
pattern.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : mark the selected sidebar row with a tinted inset surface
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the phone sidebar's round keys white on the grey page
Assisted-by: Claude (Opus 5.5 Medium)
* ui : open Settings and MCP as pages on a phone, dialogs elsewhere
On a phone the sidebar opens them at #/settings and #/mcp-servers, with the menu key and a
title like any phone page. Pinned under the status bar, Settings' section tabs fill the
screen above them in the page's colour.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : open a phone sidebar's pages behind it, settled before they slide back
A tap in the phone sidebar now opens its page while the current one is still aside and slides
back once it has rendered, instead of swapping pages mid-slide. The slid page's fixed bars are
pinned again as the page under them changes, the lock ends with the slide itself, and a
phone's new chat shows its form and greeting at once, without the move and entry they have
on a desktop.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style the sidebar footer group and the selected row with master's classes until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add GlassKey with master's styles for the phone sidebar's round keys
Assisted-by: Claude (Opus 5.5 Medium)
* ui: center the top edge of a new chat's form on the screen
The form rested 42px below the middle up to 2xl and 6px above it past
that; one drop, from the resting card's height, now puts its top edge on
the middle at every width from md up.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: round the sidebar's action rows concentric with the footer group
The sidebar stacks four surfaces: the page, the sidebar, the footer group
inset in it and a row in the group. app.css now names the group's radius
and padding, and derives the row radius from them (16px less 4px), so the
footer's rows run parallel to its corners; the actions at the top of the
sidebar share that radius. The footer moves to its own component, the
README documents the pattern, and a client test holds the rule on the
rendered footer.
A selected action row was bg-accent, the same grey as a hovered row in
light and darker than the footer's fill in dark, so it read as hovered or
not at all; it is now raised and edged, while a hover only tints.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: import demo chats through the Settings page on a phone
A phone opens Settings as a page now, not a dialog, so the demo harness
waited for a dialog that never came and every phone demo with traces
failed; it now imports through whichever of the two opens.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: count the sidebar's rows from their items in the surface test
The test expected three footer rows and two top actions, which the
stacked PRs change as they add Projects, Library and Servers; it now
expects as many rows as the sidebar has items.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a line on Meta+Enter off macOS too
The chat form takes Meta+Enter as Ctrl+Enter and leaves the newline to
the editor, but the editor bound only Mod-Enter, which is Ctrl off
macOS, so on Linux Meta+Enter neither sent nor added a line. This failed
the client test on CI's Linux runners, on master too.
Assisted-by: Claude (Opus 5.5 Medium)
This commit removes the unused ids and type_src0 parameters from
quantize_row_q8_1_cuda.
The motivation for this is that these parameters were required in the
past because this function was used in combination with the function
pointer quantize_cuda_t, which has been now been removed, and this
function no longer needs to adher to the function pointer signature.
Refs: https://github.com/ggml-org/llama.cpp/pull/31884.
* Add FP8 KV-cache support
* Add tests for scaled, FP8 KV-cache to test-backend-ops
* Fix get_*_scale mapping and dequnatize FP8 also before rope shift
* Normalizes cache-scale names independently of projection layout.
While fused KQV is currently impractival for non-shared scales, this is
an improvement for future use regardless
* Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs
* Update comment wording
* Move E2E FP8 KV cache tests to test-llama-archs
They belong there as they are not just a test for ggml's backend ops
* Increase support surface, remove seed
* make scaled kv generic
* Fix CI failings for test-kv-cache
https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392
* CI: execute kv-cache-scale tests
* exclude fp8-kv tests from OpenVINO
* Fix linting errors of test_kv_cache_scales
* Disable kv-ffload=false + multi-GPU tests for FP8
See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413
* Adjust test label for test-kv-cache.cpp
* Fix conflict-resolution mistakes in test-kv-cache.cpp
https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392
* Move KV tests to test-fp8.cpp
https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4410518158
* Remove tmp file for test-kv-cache-scales.py on Windows as well
* Fix fp8 kv cache test by population kv_source_impl
* Exclude models that consume KV cache directly
* Remove obsolete comment
* Allow BF16 scales as well for KV cache
* Rename *_cache_scale to *_cache_s
https://github.com/ggml-org/llama.cpp/pull/28392#discussion_r4517194999
* Move `is_type_scaled()` to `llama-kv-cache.cpp`
* ui: let a menu action open a submenu
Group and project pickers open as flyouts from a chat's menus. Master's menu
list has no submenus, so bring them back next to its tooltips, with a close
callback for picks made inside them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : store groups and let conversations join them
Groups live in their own IndexedDB table (schema version 2) with a name.
groupsStore creates, renames and deletes them; deleting a group keeps its
chats, out of any group. A conversation records the group it is sorted
into, and a group picked before the first message is kept pending and
applied when the conversation is created.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : show each group as a sidebar section
Every group is a sidebar section next to Pinned and Recent, folding,
showing its first six chats and dragged into place like them. A chat
dropped on a group joins it and leaves Pinned; dropped on Recent, it
leaves its group. Recent is called Ungrouped once a group exists, and its
menus create a group in a small naming dialog; so does a right-click
anywhere in the list that no row or heading takes, down to the empty space
under the last section. A group's heading starts a new chat in it, and its
menus rename or delete it; deleting keeps its chats, back in Ungrouped.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move chats into groups from their menus
A chat's menus, in the sidebar and on right-click, move it into a group
or take it out of its group. Desktop menus show a searchable group flyout
whose New group entry names a group in the group dialog and moves the chat
into it; a phone gets a picker dialog instead, which creates a group from
what is typed.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the group picker from a phone's hold menu
The hold menu is anchored to the row, so it showed the desktop flyout,
which ran off the left edge of the screen and whose New group closed
without naming one. On a phone, Move to group now opens the picker
dialog, as intended.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open a new chat in a tab of its own
With conversation tabs on, New chat from the sidebar, a group's heading
or the shortcut replaced the open chat's tab. It now adds a tab, as the
tab bar's "+" does, and the open chat keeps its own.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list a phone's groups under the picker's search
On a phone the picker fills the screen, and its grid spread the rows
over the height, so the list sat in the middle. It is now a column: the
list starts under the search and scrolls in the rest of the screen.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fix gaps in chat groups found in review
- A phone shows a group heading's New chat and menu, and the Ungrouped
heading's menu, which were reachable only by hover or right-click.
- A pinned chat moved into a group from its menu leaves Pinned, as a
drag onto the group already did, so the move shows.
- Only the sidebar's New chat, a group's New chat and the new-chat
shortcut open a new tab; stepping past the last chat, search and the
models drawer show the new-chat screen in the current tab again.
- An export carries each chat's group, and an import sorts the chat
into the group of that id or name, or creates it; a group the file
does not carry is dropped.
- The group picked with a group's New chat survives leaving the
new-chat tab and coming back.
- A phone's menus skip any submenu in the shared menu list, not only
the group one.
- Deleting a group forgets its section's place and fold.
- Groups load newest first by creation, so a rename no longer moves one
after a reload.
- groupsStore looks groups up in a map and answers which live group a
chat is in, for the three places that checked it by hand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a forked chat in its source chat's group
A fork copied the working directory but not the group, so it landed in
Ungrouped.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move a chat with its whole fork family between groups
Moving a chat into a group, or out of one, takes its parents and its
children along, so a fork tree never splits across sidebar sections. A
pinned member that joins a group leaves Pinned with the rest. Group
sections render the tree with its depth, as Ungrouped does.
* ui: show the group name prompt as a small centered card on phones
The new and rename group prompt opened full screen on a phone, like a
large dialog. It is now built on the alert dialog, as the chat rename
prompt is, so a phone shows it as a card in the middle of the screen.
Its Cancel is a plain button, so Enter in the name field saves.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: space a phone's group sections like Ungrouped
On a phone a group or Pinned section put its first chat right under the
heading, where Ungrouped leaves a row gap. They now leave the same gap,
and the sections sit a little further apart than the rows inside them.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Pascal <admin@serveurperso.com>
This commit removes the unused quantize_cuda_t from quantize.cuh.
The motivation for this is that the usage of this typedef as a function
parameter was removed in Commit 74976e1aef
("CUDA: remove -sm row, refactor cuBLAS (#24216)") and there is no longer
any usages of it.
The sampling chains are read on the host after the graph compute, and the
graph addresses are reused across builds with different selected rows: the
token-generation reserve graph has one output row per sequence, while the
actual block draft has n_max rows per sequence. The reserve allocates the
unselected chains without keeping them, so their buffers alias, and the
actual graph then reuses those addresses by node index.
set_outputs only marked the tensors stored for the selected rows, so the
unselected per-(sequence, row) chains were freed and reused. Collect every
chain tensor in t_sampling_all and mark all of them as outputs there.
Reproduces with -np 2 (non-unified KV) and 2 concurrent requests for
draft-dspark, draft-dflash and draft-eagle3; with different prompts one
sequence's acceptance collapses (eagle3 0.15 vs 0.40). After the fix:
dspark 0.71/0.78, dflash 0.56/0.44, eagle3 0.35/0.42.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ui: expand the chat form into a large editor for longer messages
The chat form expands into a large editor once a message has markdown
beyond plain paragraphs or reaches four lines, and a Maximize button in its
top-right corner toggles it by hand. The expanded form starts around half the
screen height and grows with the text up to the space below the tabs bar,
animating its height by mirroring the editor's content height.
On a new chat the expanded form keeps the collapsed form's top edge and the
greeting only fades out, waiting for a collapse to settle before it fades
back in. In a conversation the messages follow the form as it grows, so the
latest one stays in view above it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: make the expanded chat form a full-screen sheet on phones
In the expanded editor Enter adds a line and Cmd/Ctrl+Enter sends.
On a phone the expanded form is a full-screen sheet with no corners or
border. The page around it and Safari's bars take its colour, and the
sidebar's logo button and the working-directory chip step aside while it is
open. It moves by relative offsets and a clip on the main thread, so its
edges, text, actions and chip stay in step in every engine, and its content
keeps the collapsed card's places and widths, so nothing reflows when it
settles. Expanding does not focus the editor on a phone.
On a new chat on desktop the form is lifted by its sticky bottom offset
instead of a translate, and the expanded size is built on the collapsed
card's real height, so its top edge holds still through any sequence of
quick toggles.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the phone menu toggle and the sheet's focus edge while expanded
On a phone the menu toggle rides on the page, above the chat form: it now
fades out while the expanded form covers the screen, and comes back when
it collapses. The expanded sheet also drops the glass rim and inner ring,
so focusing the editor no longer outlines the whole screen.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the expanded phone sheet clear of the status bar
The expanded chat form fills the screen on a phone, under the status bar
an installed app draws over the page. Its text now starts below the bar
(pt-safe) and its collapse button sits below it too (top-safe-2.5).
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the phone's context gauge while the chat form is expanded
The gauge pinned to the phone's top right sat over the expanded sheet's
collapse key. Like the menu toggle across from it, it now fades out while
the expanded form covers the screen and comes back when it collapses.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: lift the expanded phone sheet's actions above the iOS keyboard
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move the chat form with a short, quick drift, without sliding after it lands
Its position transition came back on before the browser had laid out the new position,
so it slid the form there after the fade; the drift is now 10px each way and quicker.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style the expanded chat form with master's surfaces until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tighten the expanding chat form after review
- Expand on markdown only in finished lines: a lone `-`, `#` or `1.` on the
line being typed no longer opens the large editor.
- Show "Cmd/Ctrl+Enter to send" by the send button while expanded.
- Move the expansion state, the phone sheet and the held collapsed height
into useChatFormExpansion, and the sheet's geometry into pure functions.
- Drop the working-directory chip's phone animation, which never ran since
the phone chip moved to the + menu.
- Clearing or sending mid-move cancels the sheet's animation.
- A click only continues at the end of the text when it lands below it.
- Reuse ActionIcon for the toggle and INPUT_CLASSES for the card, derive the
centered form's offset from its drop, name the shared easing and fade.
- The greeting owns its fade behind the expanded form.
- MarkdownEditor's view is reactive, so following its content starts once it
exists.
Assisted-by: Claude (Fable 5.2)
* ui: keep the expanded chat form to the message being written
- The form expands on its own only for what is written or pasted, and
another conversation starts collapsed: a restored draft no longer opens
the large editor, or the phone sheet, on arriving at a chat.
- In the expanded editor the pickers open by the line being written, above
it or, short of room, below, instead of above the form's top edge.
- The sidebar's swipe is off while the phone sheet covers the screen.
- The send hint shows only beside the send button, where the row has room.
- The inner area keeps master's focus shadow, except on the phone sheet.
- The card and the form's position honour reduced motion.
- The phone greeting's place accounts for the wrapper it is counted from.
- The sheet's animation tolerates the form going away mid-move.
Assisted-by: Claude (Fable 5.2)
* ui: turn the typed pickers off on phones, and scroll long messages
- On a phone, typing `/` or `@` opens no picker for now: each mobile
browser moves the page its own way as the keyboard opens, so a list by
the form can not stay clear of the text being written. A note marks it
for later.
- The editor scrolls inside the form again: MarkdownEditor's own box sat
between the input and the editor without taking its height, so a message
longer than the form ran under the actions instead of scrolling.
Assisted-by: Claude (Fable 5.2)
* ui: settle the expanding chat form's review findings
- A phone expands the form only by hand: the sheet switching in mid-typing
moves the text under the keyboard.
- The typed pickers are off wherever a finger is the main pointer, not
below a viewport width, so a narrow desktop window keeps them and an
iPad loses them. The prompt picker opened from the + menu floats by the
sheet's actions on a phone.
- A message put back after a failed send expands the form again.
- Only the left button moves the caret from a click below the text.
- The sheet's fill is one token, used by the sheet, the page and Safari's bars.
- The collapsed-height hold is its own module and no longer finds the
editor's elements by class name.
- Shorter comments.
Assisted-by: Claude (Sonnet 5.5)
* ui: keep a one-line message on the chat form's actions row
On a desktop the main chat form is a single pill while its message is one
line: the add button, the text, then the model selector and send. A second
line, or a line that wraps beside the actions, brings back the two rows,
and they stay until the text is shorter than where it wrapped, so the
wider row can not flip it straight back. The editor stays mounted: the
actions join the text's row by CSS alone.
The single row's corners are `--chat-form-single-row-radius` in app.css, a
pill by default; the "Scrapbook / ChatForm single row" story shows the real
form with other radii.
Assisted-by: Claude (Opus 5.5)
* ui: give the one-line chat form 1.25rem corners
Assisted-by: Claude (Opus 5.5)
* ui: pick the working directory from the + menu on desktop too
The chip under the chat form is gone: the + menu offers the working
directory everywhere but a phone's chat, whose header shows it. The menu
item names the directory once one is set, and the picker gains a row to
clear it, which the chip's X used to do.
Assisted-by: Claude (Opus 5.5)
* ui: put the single-row chat form behind a Display setting
"Single-row chat form for short messages", on by default.
Assisted-by: Claude (Opus 5.5)
* ui: send the chat form on Enter, and add a line with Shift or Ctrl/Cmd
Enter sends wherever the caret is, the expanded editor and code blocks
included; Shift+Enter and Ctrl/Cmd+Enter add a line, continuing lists and
quotes. With "Send message on Enter" off the two swap as before. The
expanded editor's send hint and the code-block rule are gone with it.
Assisted-by: Claude (Opus 5.5)
* ui: start the chat form on the row layout it will settle on
Assisted-by: Claude (Opus 5.5)
* ui: show a set working directory under the chat form again
On desktop the chip under the form is back once a directory is set, and
the + menu offers the directory only until then.
Assisted-by: Claude (Opus 5.5)
* ui: add a line on Enter with an on-screen keyboard
A phone's Return has no Shift, so on a touch screen Enter adds a line and
the send button sends, as "Send message on Enter" turned off does; on a
desktop Enter still sends.
Assisted-by: Claude (Opus 5.5)
* cuda : Lowering VMM pool reservation size to 8 GB on GPUs with combined memory due to CUDA out of memory errors
Assisted-by: Gemini Argon
* cuda : Halve VMM pool reservation size and retry on failure
Assisted-by: Gemini Argon
* cuda : only shrink VMM pool on CUDA_ERROR_OUT_OF_MEMORY
Assisted-by: Gemini Argon
* cuda : log warning when VMM pool reserves less than max size
Assisted-by: Gemini Argon
ggml_cuda_graph, ggml_cuda_concurrent_event and ggml_cuda_stream_context
are only used by ggml-cuda.cu, but keeping them in common.cuh meant any
change to them rebuilt every CUDA translation unit. Move them to graph.cuh
and leave forward declarations in common.cuh. The context constructor,
cuda_graph() and any_cuda_graph_*() are now defined in ggml-cuda.cu, and
the stream context is held through a unique_ptr.
Also define ggml_cuda_kernel_can_use_pdl once in ggml-cuda.cu instead of
as a static function in the header, so there is one PDL cache instead of
one per translation unit, and fold the GGML_CUDA_PDL env check into it.
* ui: use a live-preview markdown editor for the chat form input
Replace the contenteditable rich input with a CodeMirror editor that shows
the markdown source of the block being edited and renders every other block
through the MarkdownContent pipeline and stylesheet, so they match chat
messages. Plain text types exactly as before, the value stays the exact
markdown source, and mention links show as atomic badges.
The editor lives in a shared MarkdownEditor component, and the highlight.js
theme loader is shared between MarkdownContent and SyntaxHighlightedCode.
The old rich input, its tokenizer, source history and their tests go away.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the production build more memory, note what mdsvex is for
The single-bundle build now includes CodeMirror and runs past Node's default
heap limit on CI runners. Raise it in the build script, so local and CI builds
behave the same on every platform.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the live preview's blocks while only the cursor moves
Moving the cursor re-walked the syntax tree and rebuilt every block's
decoration. Keep the blocks with the decorations and rebuild only when the
text changes or the cursor enters another block.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: point the chat form story at the editor and drop the engraved placeholder until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the markdown editor for screen readers
CodeMirror's content is a textbox with no name of its own, so the chat form
input was announced as an unnamed text box, and could not be found by its name
as the old input could. It takes an aria-label, its placeholder by default,
which makes the chat form's "Type a message..." again.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: format the chat form input's classes
Assisted-by: Claude (Opus 5.5 Medium)
* ui: share the markdown styles and diagram drawing between messages and the editor
The editor and MarkdownContent each @imported markdown-content.css into their own
<style>, so the bundle carried two scoped copies of it. Both now render into a
MarkdownSurface, the one element that carries the stylesheet; the few rules for
MarkdownContent's own template elements are global under .markdown-content.
Drawing mermaid and svg blocks and the actions of their header buttons move to
markdown-diagrams.ts, so the editor's rendered blocks can draw diagrams the same
way messages do.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: harden the live preview editor after review
- onChange skips only the editor's own syncs from `value`, marked by an
annotation, instead of reporting only edits that carry a user event
- Enter and the mention badges ask the syntax tree whether they are in code, so
~~~ fences and indented code count, and a closing fence ends a block
- a picked mention is one undoable edit, through replaceText
- rendered blocks draw mermaid and svg diagrams with their header buttons, a $$
math block with blank lines renders as one block, and a link reference
definition stays raw instead of rendering to nothing
- rendered blocks render again when the mention path settings change
- a drag in a rendered block selects its text and copy takes it, across blocks;
a click still opens the block
- the block margins are the ones a chat message renders (each heading is a first
child there), checked by a test; the blank lines after a raw block are its gap
- the tree comes from syntaxTree, without forcing a full parse on every edit
- placeholder, label, extensions, mention widgets and render key follow prop
changes after mount, through compartments
- the editor uses the markdown language alone, which drops the html, css and
javascript parsers markdown() pulls in for html tag completion (-70 KB gzip)
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the CodeMirror packages and set the build heap in .npmrc
The new dependencies are pinned like the rest of package.json. The build heap
moves from a node call on vite's bin path to npm's node-options, so the build
script is plain `vite build` again and the setting reaches it on every platform.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: update the live preview block by block and estimate block heights
An edit used to build every block's decorations and widgets again, which the
editor then compared one by one. The previous decorations, and the mention badges
of raw blocks, now move through the edit, and only the blocks whose range, text,
raw or rendered state, or gap changed get new ones; the texts of the blocks an
edit left alone are reused instead of sliced again. A test drives random edits,
cursor moves and focus changes and checks each result against a full build.
Rendered blocks report the height they were last measured at, or a guess from
their text, so CodeMirror's height map for blocks off screen in a long document
stays close and scrolling stays steady.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: skip the editor's redundant syncs, reconfigures and theme reloads
- syncing `value` compares it with the text the editor last reported or synced, so
a keystroke coming back through onChange no longer serializes the document again
- the prop effects reconfigure the editor only when a prop changed, not once more
right after it is created (which rebuilt the whole preview)
- loadHighlightTheme leaves the theme's style element alone while the mode stays,
so a message or an editor mounting no longer restyles the whole page
Tests check that typing in one block leaves the other rendered blocks without a
single DOM mutation, and that mounting reconfigures nothing.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: simplify the live preview to a plain rebuild in three small modules
Rebuilding the decorations of every block on an edit measures the same as
patching only the changed ones (0.09 ms against 0.11 ms per keystroke at
200 blocks, 0.9 ms against 1.1 ms at 2000), and equal widgets keep their
DOM either way, so the incremental update and its test go.
- markdown-blocks.ts collects the document's blocks, markdown-block-widget.ts
renders one, markdown-live-preview.ts decides which ones show raw text
- one bounded cache holds a block's HTML and its measured height
- the render key is a plain option: a new one reconfigures the preview,
which keeps the editor's focus
- MarkdownEditor follows its props with one helper instead of five effects
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep setext headings raw in the live preview editor
`test` over `---` is a setext heading, so the preview rendered it as an h2
and the underline vanished once the cursor left the block. The editor now
shows a setext heading's text as typed at all times, like a link
reference definition.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a setting to turn the live markdown preview off
"Live markdown preview in the message input" (General) is on by default.
Off, the input shows the plain markdown text, with file mention badges
still in place; switching it takes effect without a reload.
Assisted-by: Claude (Opus 5.5 Medium)
normalize_quotes_to_json copies raw control characters verbatim while
converting pythonic tool arguments to JSON, so calls like
python(code='a<LF>b') produce invalid JSON in the arguments returned to
clients; replaying that history then fails hard. Escape C0 control
characters inside quoted strings via escape_json_string_inner.
* sycl: Add device-side MoE scheduler
Assisted-by: Codex
* Add width-heuristic override bit and Q2_K and Q3_K types
Assisted-by: Codex
* tests: cover skipped MUL_MAT_ID slots on the device MoE schedule
Assisted-by: Claude Opus 5.5
* sycl: keep graphs on for lazily reordered MUL_MAT and reject BN=64 combos on the device MoE schedule
The graph check disabled SYCL graphs until every MUL_MAT weight with a
reorder extra was reordered, which never happens for some (dense MXFP4,
the fused GLU plain path), so graphs stayed off and the check reran on
every compute. Upstream already reorders those during recording with
the async memory extension, so only MUL_MAT_ID weights are reordered
ahead of recording now.
ggml_sycl_grouped_dequant_gemm_device_supported() accepted any XMX
combo, but the device schedule only runs with BN == GGML_SYCL_FG_BN, so
a 32x64 combo (GGML_SYCL_XMX_GATHER_SHAPES=4 or 8) made the graph check
pass and the compute fall back to the host sync path while recording.
Assisted-by: Claude Opus 5.5
* sycl: keep the ggml_sycl_mul_mat_id_needs_sync name for the graph check
Assisted-by: Claude Opus 5.5
* fix: stop erroneously reordering host-offloaded tensors
With -ncmoe, prompt processing offloads the CPU-resident expert matmuls
to the GPU: the scheduler uploads the experts into a copy in the compute
buffer before every run. init_tensor gave that copy a reorder record, so
the MUL_MAT_ID gather path reordered every offloaded expert tensor on
every ubatch, and the next upload overwrote the result.
Qwen3-30B-A3B Q4_K_M, 1x B60, llama-perplexity -c 4096 -ub 2048
-ncmoe 48, 8 chunks, on master ac8e1694:
master 116 s, PPL 7.739
master XMX_GATHER_TYPES=0 67 s, PPL 7.738
with this change 67 s, PPL 7.729
Refs ggml-org/llama.cpp#31262
Assisted-by: Claude Opus 5.5
* Remove SYCL-specific test comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sycl : name the MUL_MAT_ID path predicates after what they decide
ggml_sycl_mul_mat_id_mmvq_ok -> ggml_sycl_mul_mat_id_should_use_mmvq
ggml_sycl_mul_mat_id_mmvq_shape_ok -> ggml_sycl_mul_mat_id_mmvq_eligible
ggml_sycl_mul_mat_id_grouped_shape_ok -> ggml_sycl_mul_mat_id_device_sched_eligible
node_is_device_local -> graph_node_is_in_local_device
ggml_sycl_gg_rows -> ggml_sycl_grouped_gemm_rows
ggml_sycl_gg_tile -> ggml_sycl_grouped_gemm_tile (from #29245, renamed to match)
"eligible" is a necessary condition; the type and width checks follow
elsewhere. "should_use" means the path runs. No functional change.
Assisted-by: Claude Opus 5.5
* sycl : reorder Q3_K MoE experts and load them as vectors in the grouped GEMM
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: cwriter <cwriter@localhost>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vulkan: tiled transpose for dim-0 concat with a transposed source
The delta-net conv-state path does ggml_transpose() straight into a dim-0
ggml_concat(), so the generic concat kernel reads src1 with the transposed
stride, one element per thread, and neighbouring lanes land on different cache
lines. Stage a 32x32 tile through shared memory instead, which keeps both the
load and the store coalesced. 33 columns pads away the bank conflicts.
Gated narrowly: dim 0, 2D, unquantized 4-byte elements, contiguous src0/dst
rows, and src1 actually transposed. Everything else falls through to the
existing kernel.
Qwen3.6-35B-A3B on gfx1151/RADV, 30 gated-delta-net layers: the CONCAT op goes
11877 us -> 957 us, and pp2048 at ub2048 goes 1224 -> 1628 t/s. Generated text
is unchanged.
The approach follows the tiled transposed concat Gaetan Puleo wrote for the HIP
backend in his Strix Halo fork.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* enable by default, add smaller shader sizes
* clean up comments
* improve shader
* Add non-contiguous concat tests
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
* handle x or y axis overflow
---------
Co-authored-by: Nathan Wilson <nath.flagman151@passmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
* ggml-rpc: fail gracefully on unreachable endpoints and failed handshakes
rpc_dispatcher::start() aborted the whole process (GGML_ABORT -> SIGABRT) when
the endpoint could not be parsed, the transport failed to initialise, the socket
could not connect, or the HELLO handshake failed. A typo in --rpc or a
temporarily down node killed the host application, and the reason went to a log
stream the host may never see.
Make the connect-time failure reportable instead of fatal: start() returns bool
(LOG_ERROR + false; the dispatcher thread only starts after a successful
handshake), negotiate_hello() returns false on a half-open socket,
get_dispatcher() returns nullptr and does not cache failed attempts, and every
consumer treats nullptr as "endpoint did not answer" with the conservative
behaviour of its domain (allocation -> nullptr, alloc-size -> lower bound,
device memory -> 0/0, events -> no-op, device count -> 0, and add_server()
already maps 0 devices to nullptr).
Mid-stream failures keep the existing RPC_STATUS_ASSERT by design (fail fast
against silent corruption) and are intentionally out of scope.
Field motivation: on Android (llama.rn + phone RPC cluster) an unreachable
worker crashed the app; the crash buffer showed
abort <- ggml_abort <- rpc_dispatcher::start <- ggml_backend_rpc_add_server.
* tests : add a regression test for unreachable rpc endpoints
Calls ggml_backend_rpc_add_server with a socket that nothing listens on and with a
malformed endpoint, and expects nullptr from both. Without the fix in this branch
the first call aborts the process, so the test cannot even fail cleanly.
* ggml-rpc remove low-value comments
* tests : drop the internal device-count check after the rpc registry refactor
---------
Co-authored-by: figkcoci-debug <figkcoci-debug@users.noreply.github.com>
With head sizes above 128 and KV >= 1024, scalar flash attention uses
16 rows per workgroup. On the Qualcomm Adreno driver this variant
spills registers to scratch memory (264 bytes per invocation on the
X1-85, none at 8 rows) and runs about 40x slower. The largest
test-backend-ops case then becomes a single dispatch of several
seconds, past the Windows GPU timeout, and the test never completes on
the X1 Elite.
Use at most 8 rows per workgroup on Qualcomm when hsk or hsv > 128.
Smaller heads keep 16 rows, which is faster for them on the X2 Elite.
Adreno X1-85, FLASH_ATTN_EXT hsk=256 kv=4096: nb=64 711 ms -> 17 ms,
nb=512 no result -> 125 ms; test-backend-ops completes. Adreno X2-90:
Gemma 4 E4B pp512 at depth 4096 with flash attention 81-90 -> 117 t/s.
On a device without mmap support, --lazy-mode auto resolved to off even
with --load-mode mmap, so llama_prefetch_rows never ran and the rows of
the large tensors faulted in one page at a time. Resolve it to off only
when mmap is off; the default iGPU load path is unchanged.
Fixes#31943
* server: add oai-compat /v1/audio/speech endpoint
* add e2e pytest
* server: share the s16 rounding, type speech form fields
The WAV writer of the audio helper truncates while pcm_f32_to_s16
rounds, so wav and pcm differ by one LSB. Both now round.
pcm_f32_to_s16 takes a pointer and a count and converts the TTS
bytes without an intermediate copy.
/tts and /v1/audio/speech parse their body with parse_tts_body, so
multipart fields such as speed get their type before the speech
conversion checks them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* mtmd: support gemma 4 input audio stream
* use 480ms step
* server: enable realtime embeddings only for audio generation
The realtime slot forced embeddings on, which marks every row of a
batch as an output. Only a model that generates audio reserves
n_batch outputs, so a chat model with an audio stream (Gemma 4)
aborted on its first batch of audio rows. The hidden state is only
read by the audio generation pipeline.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
A cold dev server reloads the page once it has optimized its
dependencies, and that reload can land right after the harness opened
the phone sidebar, closing it again before Settings is clicked. Repeat
the pass over the menu keys up to three times while Settings is not
reachable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: reveal the mobile sidebar by sliding the page aside
On a phone the sidebar now stays put under the page, and the page slides
right to uncover it, as a rounded card with the card border. The menu
button rides on the page; a swipe from the left edge opens the sidebar,
and a swipe back or a tap on the page closes it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: polish the mobile sidebar reveal and add a hold menu to rows
Holding a conversation row on a phone opens its action menu anchored to
the row, in place of the hidden three-dot trigger. The page no longer
widens the document when it slides aside, so iOS stops panning it on the
first open.
The page and sidebar backgrounds start in the page color and blend into
the sidebar tint as the sidebar opens, following a drag frame by frame,
and a theme-color tag follows them for the browser bars. The sidebar's
contents fade in on a log curve and scale up slightly, and sit at the
same inset as the menu toggle, which now uses a two-line icon in a
filled circle.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the context gauge on phones and lock the slid-aside page
On a phone the context gauge leaves the chat form for the top right of
the chat screen, in a round button across from the menu toggle. Both
take the scroll-down button's frosted look.
While the sidebar is out, and until the page is fully back in place,
the page neither scrolls nor takes taps; a tap or drag on it still
closes the sidebar.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the mobile sidebar with a swipe from anywhere
A rightward swipe from anywhere on the page now opens the phone sidebar
too, beside the edge swipe that competes with the browser's own back and
forward gesture. It starts after a short sideways move, skips text
fields and content that scrolls sideways, and keeps the page from
scrolling under it. Letting go opens or closes the sidebar past 30% of
the way or on a flick.
The top buttons and the chat form now hold their place on screen while
the page is slid aside, instead of jumping to the far ends of a
scrolled conversation.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the model org name on mobile unless a setting keeps it
A phone's model selector is narrow, so it now drops the organization
from model names by default. A new Display setting, "Show organization
name on mobile", brings it back; it applies only while the organization
name is shown at all.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: close the mobile sidebar before following a tap in it
On a phone, picking a conversation, New chat, Search, MCP or Settings
in the open sidebar now slides it shut first and acts 180ms later, so
the page or dialog appears once the sidebar is out of the way. With the
sidebar already closed, or on a desktop, the action runs at once.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: make the app feel native on phones and run it under the status bar
Applies the mobile-native checklist: no tap highlight or landscape text
inflation, instant taps on buttons and links, no label selection or
callout on a held control, hover styles only where a pointer can hover,
and 16px textareas on phones so focusing them does not zoom the page.
The page now runs under the notch (viewport-fit=cover), so the installed
iOS app tints its status bar from the page and the sidebar. The menu
toggle, the context gauge, the sidebar and the top of the chat and
models pages pad themselves clear of it, and the locked, slid page
reaches down by the status bar's height, which iOS leaves out of it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a viewport height readout for debugging on phones
With VITE_DEBUG set, a dev build shows the window, viewport unit and
safe-area heights, and the elements that reach past the window's bottom
and make the page scroll. It is loaded on demand, so production builds
leave it out.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fill the screen from launch in the installed iOS app
iOS 26 gives an installed app with viewport-fit=cover and a translucent
status bar a viewport short of that bar (WebKit bug 301108), clipping
the bottom of the page and lifting the chat form, but only while the
page is shorter than the screen. In the installed app the page now
reaches the screen plus the status bar, capped at the screen, so it
gets the full screen from the start without ever scrolling.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: raise the phone's corner controls in the installed app
The installed app's status bar already sets the menu toggle, the context
gauge and the open sidebar's logo apart from the screen's edge, so they
sit just below it rather than a full 1rem lower. They share one value,
which stays at 1rem below the inset in a browser tab.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add safe-area utilities for what is pinned to a screen edge
The page runs under the status bar and over the home indicator, which an
installed iPhone app always shows. New pt-safe, pb-safe, top-safe and
bottom-safe utilities, each with a spacing step (pt-safe-12 is 3rem plus
the top inset), keep pinned content clear of them instead of hand-written
calc()s. A --safe-area-bottom joins --safe-area-top, and the chat screen and
Models page use pt-safe-12.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open search in a full-height drawer on phones
A phone used to leave for the /search page. It now opens the same search
as the desktop dialog, in a bottom drawer that fills the screen below the
status bar. Keyboard shortcuts, the Enter hint and the keyboard highlight
stay on desktop, and so does the models manager, as in the phone sidebar.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show every dialog full screen on phones, clear of the status bar
A phone now shows every dialog full screen, whatever size it asks for on a
desktop, instead of each full-screen dialog repeating its own classes. Its
top and bottom borders take the status bar's and the home indicator's
height, transparent, so an installed app's dialogs keep their content clear
of both with no change to their padding. The attachment preview, which has
its own full-screen content, pads itself and lowers its close button the
same way.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fade the page out under the status bar of the installed app on phones
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show search in the full-screen dialog on phones instead of a drawer
Assisted-by: Claude (Opus 5.5 Medium)
* ui: start a phone page's title a row below the corner controls
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the slid page's bottom bars against the screen in the installed iOS app
Assisted-by: Claude (Opus 5.5 Medium)
* ui: cast a soft shadow from the slid page onto the phone sidebar
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the scroll-down key at the real bottom on iOS and show it only well above
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fade the status-bar fade out as the page slides aside for the sidebar
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw the slid page's shadow above its rounded frame, so it follows the corners
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a phone chat's title and working directory at the top, and pick it in a drawer
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give iOS Safari a solid top and fade the bottom of the screen on phones
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let iOS Safari find the top strip, which it skips when it takes no taps
Assisted-by: Claude (Opus 5.5 Medium)
* ui: lengthen the top fade and lift the chat form a little in phone browsers
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give iOS Safari's bottom toolbar the page's colour, not the chat form's
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw the phone's context gauge key like the menu key across from it
Assisted-by: Claude (Opus 5.5 Medium)
* ui: confirm opening the phone sidebar, its hold menu and its rows with a haptic tap
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the screen-edge fades on the page as it slides aside for the sidebar
Fading them out with the slide made the page flicker; the rounded frame over the
slid page already clips them at its corners.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the working directory chip optional for hosts with their own trigger
The phone layout opens the picker from the chat's header, so ChatForm hides the
chip there. Master's version of the chip has no such switch; bring it back.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style layer 5's additions with master's tokens until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the design system's stylesheets and fonts from app.css until it lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep master's font stacks until the design system brings its fonts
Assisted-by: Claude (Opus 5.5 Medium)
* ui: restore master's colour palette in app.css until the design system lands
Resolving the safe-area commit took the design system's app.css, which drops
the palette for its token files and aliases the shadcn names to semantic
tokens. With the tokens gone those aliases point at themselves, so every
colour went invalid: transparent surfaces and currentColor borders.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: center alert dialogs on phones instead of sliding them up as drawers
An alert is a short question, not a drawer: on a phone it is now a centered
card, 90% of the screen wide and as tall as its text, scrolling only past the
screen's height. It zooms in as on desktop, and its footer drops the margin
the sheet needed.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: space the working directory drawer for a phone
The drawer reused the popover's compact picker: Browse sat against the search
field, the divider ran past the rows and each row had its own inset. In the
drawer the picker now takes the header's inset, gives Browse room, lines its
rows up with the search results and sizes its text for a phone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tap the haptic engine when a message sends, and once per hold
A phone now confirms a sent message by feel, at the moment the send goes
through (not on every press of the button), and before any await, so it still
runs inside the tap or keypress that iOS needs to play its tick. The hold that
opens a row's menu now taps the engine from the long-press hook itself, once:
the hook vibrated and the row tapped as well, which buzzed twice on Android.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the gauge dial sizes, the chat form's bottom offset and the sidebar fade
Review follow-ups: the context gauge dial reads its grid, radius, stroke and
track from one table per size instead of ternaries; the chat form's distance
from the screen's bottom per platform and the sidebar content's fade duration
are named constants. The viewport debug readout moves out of src, next to the
tests, as it is a debug tool the app only loads in a dev build.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say why the microphone can't record instead of failing silently
A llama-server opened over plain HTTP on the local network or over Tailscale
is not a secure context, where browsers hide the microphone, so a tap on the
mic did nothing but log to the console. The button now shows a crossed-out mic
in the warning tone with the reason in its tooltip, and a tap says it in a
toast, as a phone shows no tooltip. A start that fails names the browser's
reason (access denied, no microphone, in use by another app) the same way,
and the next tap tries again. The recorder passes the browser's error on
instead of replacing it with a generic one.
Assisted-by: Claude (Opus 5.5 Medium)
* model : allow loading external PLE files
Assisted-by: Codex
* convert : add separate PLE exports
Assisted-by: Codex
* tests : fix PLE export test lint
Assisted-by: Codex
* model : make --ple an alias of --shard, allow multiple shard locations
Reuse the multi-shard loading path for external PLE files instead of a
separate option.
* --ple is an alias of --shard: both take a comma separated list and may
be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
common_model_load() uses llama_model_load_from_splits for two or more
paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
memory measurement load every shard
Assisted-by: GPT-6-Astra
* rm -isnm
* make sure to dedup paths
* rm test_convert_ple
* address comments
* allow ple shard to override tensor
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* Windows unbuffered io
* Fix read len calculation
* Add test and model load timing
* chunked fallback read + removing filepointer semantic divergence between linux and windows
* tests : remove unused mmap test arguments
Assisted-by: Codex
* remove timing, set file pointer after chunk read, test refinement
* Update tests/test-llama-mmap.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Update src/llama-mmap.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* test refinement
Assisted-by: Codex
* remove duplicate constexpr
Assisted-by: Codex
* adding label behavior to tests
Assisted-by: Codex
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* server : functionality test to make sure every time when we add new spec dec method the existing ones are not broken.
* server: use compact models for speculative decoding CI
* ci: avoid GHA cache for speculative models
* ci: run speculative tests on DGX Spark
* server: address speculative test review
* server: use consolidated speculative models
* server: refine speculative test calibration
* docs: record speculative Spark validation
* ci : run the server-speculative steps with bash
* server: recalibrate speculative test floors on GB10
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server : accept top-level schema in response_format with type json_schema
The OpenAI-compatible chat completions endpoint only applied the schema
when it was nested inside response_format.json_schema.schema. When the
schema was provided directly as response_format.schema (as documented in
the server README, and used by some clients), it was silently ignored
and generation ran unconstrained.
Also mention json_schema in the error message for invalid response_format
types, document the OpenAI wrapper form in the README, and add a test
case for the top-level schema form.
Fixes#10732
* server : let the json_schema wrapper take precedence by presence
Pick the schema source based on whether response_format has a
json_schema key, instead of falling back when the parsed wrapper schema
is empty. An empty schema {} is valid (any JSON value), so the old check
let a top-level schema override an explicit {} in the wrapper. Use
json_value() for the top-level schema too, like the rest of the
function, and add tests for the precedence.
---------
Co-authored-by: linsen <251731047+linsen458-spec@users.noreply.github.com>
* ggml: implement the backward pass for TRI
GGML_OP_TRI had no case in the backward switch, so a trainable graph containing it aborted with "unsupported ggml op for backward pass: TRI". The op is linear, so its gradient is masked the same way. The incoming gradient is not necessarily contiguous (the backward of a transpose hands over a view) and ggml_tri requires contiguity, so it is made contiguous first.
test_tri gains a post_transpose variant (out = transpose(tri(a))) that aborts as above without this change.
* ggml : simplify the TRI backward pass
Pass the gradient directly to ggml_tri and drop the transpose test. The existing TRI cases cover the backward pass for contiguous F32 gradients.
The dummy-model generator fills every tensor from a plain normal pool.
`blk.N.ffn.act.scales` is a divisor in the mpt graph, so with seed
1224489258 it draws |v| ~ 1e-6, and `ffn_act = gelu(...) / scales` reaches
2.7e5. Metal stages both operands of `kernel_mul_mm_f32_f32` as half, so
those activations overflow to inf, the norm downstream turns the inf into
nan and the logits come out nan, while the CPU reference stays in f32 and
stays finite. Both NMSE values then become nan and the check fails.
Give `ffn.act` the same treatment as `rope_factors` and keep its magnitude
away from zero.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* ui : fold the sidebar's conversation sections
Pinned and Recent conversations fold under their heading. The whole heading
row is the toggle; its chevron follows the title, shows on hover and stays
while the section is folded. Folded sections are remembered across reloads.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : show a sidebar section's first six chats
Pinned and Recent list their first six chats and end with a Show more
row that opens the whole section; Show less, sticking to the bottom of
the sidebar while the section scrolls under it, closes it back. The
expanded sections last for the session.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : rearrange the sidebar's sections and chats by dragging
Pinned and Recent can be dragged by their title. The real section follows
the pointer, tinted like the sidebar's resize handle and edged with a
faint border, the other sliding aside as it passes; let go, it glides from
there into its new slot. The order is remembered across reloads, and a
section never moved keeps its default place. All sections now scroll
together.
A chat dragged onto Pinned is pinned, onto Recent unpinned. A lifted copy
of the row follows the pointer while the row slides out of its section.
The section under the pointer takes the same tint and border; a section
ignores its own chats. Let go, the row slides in where its section sorts
it, the rows below making room, and the copy glides into it; anywhere
else, it slides back into its own section.
While dragging, the rows and headings passed over take no hover. Dragging
needs a mouse or a pen, so a finger still scrolls the sidebar; the sidebar
scrolls near its edges, and Escape cancels.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : set the sidebar's sections apart from their rows
A section heading now reads above its chats: larger, semibold and in the
full foreground colour. Chat titles drop to a regular weight in a slightly
softened colour, full again when the chat is open or hovered, and "Show
more" drops to a regular weight in the muted colour. Chats inside a
section sit closer together, sections further apart, and the sidebar's
actions get more room above and below.
Assisted-by: Claude (Opus 5.5 Medium)
* fix(rocm,ci): Update ci workflow to use devlab ci runners for gfx1151
* ci : move gpu-rocm job to Strix Halo devlab runners
Assisted-by: Claude
* ci : ROCm 10 wheels with per-GPU device packages for gpu-rocm; add gfx1201
The runners' system ROCm 7.2.1 cannot compile the current HIP code for
gfx1151 ("Invalid dpp_ctrl value" in mmf-instance-ncols_*.cu). Go back
to the TheRock ROCm 10 wheels with the device-<gfx> extra, so rocBLAS
has the target's kernels, and keep ci/run.sh's results and models under
$RUNNER_TEMP instead of the runners' RAM-backed home. Add a gfx1201
(Radeon AI PRO R9700) row with HIP graphs disabled.
Assisted-by: Claude
---------
Co-authored-by: David Hekimian <dhekimian@users.noreply.github.com>
* metal: make ggml_tri select instead of scaling by the comparison
The kernel multiplied by the 0/1 comparison, which turns a +-inf outside the triangle into NaN, while the CPU kernel writes a plain zero there. Select, like the CPU does.
Adds test_tri_special, which puts +-inf into the cells that have to be discarded, for all four triangle types. It fails against the previous kernel and passes with this change.
* Just trim down the repetitive comments.
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3
Assisted-by: Claude Opus 5.5
* feat: add temp parameter in llama-tts
* temp: fix default hardcoded values
* fix: remove too narrow test
* fix: addressing PR comments
The rotation angle of a position p is rounded in f32 to about
p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach
it through different rounding sequences (cumulative product versus
pow). With positions up to 8192 this puts the NMSE floor between
backends near 8e-8, so the default 1e-7 threshold fails at random on
Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for
n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
* ggml : fix for the radix TOP_K thread timing bug
* make the key an ordered 64-bit pair
* a rewrite without shift-skip logic
* fix casts
* warp-contiguous layout
* reduce the scope removing vulkan+metal changes
* turn it off for Metal/ROCm, note about cub::DeviceTopK
* fix the hip build
* skip Vulkan as well
* popc lambda from fattn
* do block_cols math in int64_t
* make it allowlist bc SYCL looks non deterministic too
* ggml : make Metal radix Top-K tied selections deterministic
The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.
Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.
This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : enable deterministic top-k check on Metal, add radix-path tie cases
Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.
Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : use coalesced accesses in the Metal radix Top-K compaction
The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.
Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : remove the extra top-k tie and perf cases
The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal: scan the radix bins and warp totals with a simdgroup
The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.
* tests: enable the deterministic top-k check on ROCm
HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.
* cuda: remove the redundant radix top-k counter reset
Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.
Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).
Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.
Assisted-by: Claude Sonnet 5
Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
* ggml : switch to F32 src1 in the CPU backend
Change the default computation for GGML_TYPE_F16 to use F32 for src1
data in matrix multiplications: vec_dot_type(F16) is now F32 and the dot
product goes through the new ggml_vec_dot_f16_f32 (x = F16 src0, y = F32
src1), so src1 is no longer quantized to F16 and worst-case inputs that
only fit in the F32 FP range are handled correctly.
- conv_2d/conv_3d: store im2col patches (the inner mul_mat src1) as F32
and pass separate src0/src1 types to the inner mul_mat; the previous
F16 patches made the inner mul_mat widen src1 into the conv work
buffer, corrupting the patches and crashing on large inputs
- conv_transpose_1d/2d (F16 kernel): keep src1 in F32 in the work buffer
and use ggml_vec_dot_f16_f32; update work buffer planning accordingly
- supports_op: restore MUL_MAT support for F16 src1, which is now
widened to F32
- ggml_vec_dot_f16_f32: add GGML_SIMD support by converting chunks of x
to F32 and reusing the F32 SIMD dot
- mul_mat/mul_mat_id: add TODOs for a more efficient F16 x F16 path that
can be used if ggml_prec_get_src(dst, 1) allows converting src1 to F16
MUL_MAT f16 x f32 perf on M5 Ultra (GFLOPS):
- m=2048, n=512, k=16 : 120.9 -> 101.7
- m=2048, n=512, k=64 : 150.5 -> 344.2
- m=4096, k=14336 GEMV: 63 -> 343-415
- m=4096, n=512, k=14336: 65.6 -> 770.2
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : add some F16 x F32 vectorized paths
* cont : enable the fused F16 x F32 dot on AVX without FMA
The 256-bit fused path of ggml_vec_dot_f16_f32 required AVX2 and FMA,
so CPUs with AVX and F16C only fell back to the chunked conversion and
F16 decode was 4x slower than before. Enable it on __AVX__ && __F16C__
and accumulate with GGML_F32x8_FMA, which does mul + add without FMA.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls
Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.
* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends
Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.
* tests: cover GGML_PREC_BF16 in the flash attention tests
The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.
* llama: request GGML_PREC_BF16 for the KQ multiplication
* cont : add TODO in clip.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* opencl: add kernel_gemv_noshuffle_q8_0_f32_mc
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for q5_1, q4_k
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: adreno kernel k floor 256 for x2
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for more quants
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: rename function
* opencl: disable bin FA for small verify batch
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* hexagon: use absolute skel path
* hexagon: support non-ascii dir and long path in skel-abs path
* hex-libdir: add support for FASTRPC_MANAGE_DSP_LIBRARY_PATH and GGML_HEXAGON_LIBHTP_DIR on all platforms
* hex-libdir: keep extra fwd slash in abs path on Android and Linux
* hex-libpath: always start with absolute path then fallback to session ctl then env
* hex-libdir: add CDSP1,2,3 to the fallback, needed for the IQ10
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* hex-nhvx: apply nhvx updates to the latest master
* hex-nhvx: set N_HVX and N_THREADS based on the arch to avoid overallocation
* hex-build: add support for enabling HTP_DEBUG via build.py
* hex-trace: move tracing state/helpers into new htp-trace.h header
* hex-dcache: cleanup dcache #defines
* hex-hnvx: minimize the diff by reflowing the macros
* hex-ci: fix python type/linter warnings
* hex-nhvx: consitent logging
* mtmd: support MageVL-Cowork model
* mtmd: keep magevl rope_freqs at model level and check its shape
The rope factors are shared by every layer, so they live in clip_model
instead of a pointer copied into each clip_layer. The loader now checks
that v.rope_freqs is F32 with d_head elements and that d_head splits
into the 4:6:6 pairs the graph expects.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* moe-cache : compute part of the misses on the CPU
--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.
- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
selected experts, so the CPU part does not wait for the device, and its
result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
simulator of the cache
* scripts : add moe-cache-tune.py to pick the MoE cache arguments
The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.
docs/moe-cache.md describes the MoE cache arguments and the script.
* add am17an as codeowner for MoE cache
* moe-cache : upload all the misses until the cache is full
With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.
* add backend sync for async transfer queue
* moe-cache : let the scheduler copy the results of the CPU part
* Fix MSVC compilation
* remove sync change
* llama-graph : extract the CPU share condition into can_cpu_share
* llama-graph : make llm_graph_params::moe_cache const
The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* scripts : replay the ClassLRU policy in moe-cache-tune.py
* moe-cache : pick CPU misses by CRF instead of random (#31853)
* moe-cache : remove the backend logic of the CPU part from the graph
The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.
ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.
* llama-graph : return the MoE cache slots and CPU ids in a struct
The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.
---------
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
The kernel_group_norm_f32 kernel and its host dispatch ignored the
batch dimension (ne03) of the source tensor: only ngrp threadgroups
were dispatched and each normalized the flat range
[tgpig*gs, (tgpig+1)*gs) with gs = ne00*ne01*ceil(ne02/ngrp). For
tensors with ne03 > 1, all but the first ne00*ne01*ne02 elements were
left unwritten (exposed by the new GROUP_NORM test with
ne=[3,1,1,65537], num_groups=1).
Dispatch one threadgroup per (group, i03) pair (ngrp in x, ne03 in z,
same work distribution as the Vulkan backend) and compute per-batch
statistics, matching the CPU reference. Also clamp the group element
count for the last group when ne02 is not divisible by ngrp and use
int64_t indexing.
The kernel now only requires ggml_is_contiguous_rows(op->src[0])
instead of full contiguity: it addresses src0 and dst with their own
strides (nb03 and the dst strides are added to the kernel args - dst
can be a strided view of src0 with ggml_group_norm_inplace) and falls
back from the flat fast path to a strided path that decodes the linear
element index when rows or channels are padded. The flat/strided
selection is done on the host in
ggml_metal_library_get_pipeline_group_norm and compiled into one of
two specialized pipeline variants via a FC_group_norm_flat function
constant, so the dead path is eliminated at pipeline creation.
GROUP_NORM support is also gated on F32 src and dst in supports_op -
the kernel only implements f32 and an f16 input would silently be read
as f32.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* server: visit parent slots before child slots
A parent slot wakes its waiting child slots once it finishes the shared
prompt prefix, while slots are visited in id order. When a child task
lands on a lower slot id than its parent, the child is visited before
the parent wakes it and waits an extra batch to process its suffix.
Non-child slots are now visited before active child slots, so every
child woken by its parent joins the same batch.
* server: check systemone test events through test tags
The systemone tests matched free-form debug messages, including the
formatted f_keep value of the prompt cache. The server now emits
__TEST_TAG_WAIT_PARENT_SLOT__ when a child slot waits for its parent,
__TEST_TAG_PROMPT_CACHE_RESTORE__ on every prompt cache restore and
__TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when the whole cached prompt is
a prefix of the new one, compared on token counts.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* Only run webgpu CI on my fork
* Add webgpu only workflow
* Add WebGPU group_norm kernel
* clean up
* resolve merge conflicts
* parameterize data type; clean up cmake changes
* add nullptr check
* clean up webkit experiments
* more cleanup
* more cleanup
* clang format
* Remove unused feature check for WebGPU
* clean up supports_op condition and update documentation
* add perf tests for group_norm
* clean up doc with script and undo change in test-backend-ops
* Do 2d dispatch of workgroups
* Do 2d dispatch of workgroups
* Unstage unrelated changes
* undo docs churn
* add new test case
* undo small change
* undo small change
* ggml-backend-meta : minimum number of compute views per buffer
Assisted-by: Claude Opus 5.5
* tests : revert the deepseek41 test model to 6 layers
Assisted-by: Claude Opus 5.5
* hexagon: fuse post-norm residual add into RMS_NORM_MUL
* hex-fusion: refactor RMS_NORM fusion and add dedicated RMS_NORM_MUL_ADD opcode
Improves performance by keeping things in HVX registers.
Also simplifies overall fusion logic and logging.
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* model: add Kimi-K3.1-Pro text model
K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.
Indexer:
- the selection is an additive mask over the whole cache, not a gather of
the top-k rows, so it reuses the existing sparse flash attention path;
off that path the mask stays correct but still costs a kq over the cache
- the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
because dsa_config declares no eps of its own
Graph and cache:
- new arch with four indexer tensors per DSA layer, and a hybrid memory
that keeps a K-only MLA cache for those layers, recurrent state for the
KDA layers, and the indexer key cache
- the graph node budget shares the K3 entry
Conversion:
- KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
- dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
plus the indexer LayerNorm eps; the layer list is 1-indexed like
linear_attn_config's, and the converter asserts that it covers exactly
the full-attention layers
- the routed experts stay MXFP4 and are repacked as in K3, not dequantized
- the vision tower and mm_projector are skipped: text only for now
- the arch key follows the checkpoint's own model_type ("kimi_k31") and the
loader registers KimiK31ForConditionalGeneration; the release name carries
a -Pro suffix that nothing in the config or the tensor names uses
Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.
Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.
* converter: avoid clutter in the k3 base
* narrow comment to match model
* update metal fusion expectations
* restore
* sync with PR 31733 refactoring
* hexagon: Generalization of CONV_2D implementation for CONV_3D
- New op HTP_OP_CONV_3D is wired through htp-ops.h, main.c (shares op_conv_2d) and the op remap in htp-kparams.cpp.
- conv_3d runs on the existing HMX tiled (implicit GEMM) conv2d path. KD and KH are folded into one virtual kernel row (KD*KH), and output rows are counted as N*OD*OH.
- htp_kparams_solve_conv3d checks the conv_3d layout and calls the tiled solver htp_kparams_conv2d_tiled, which takes an is3d flag.
- The tiled kernel takes the sizes and strides that differ between conv_2d and conv_3d from a struct htp_conv_geom. Only the input stage unfolds depth.
- 256-entry DMA queue can no longer overflow
Assisted-by: Claude Code
* hexagon: modify checking to eliminate divide-by-zero hazards
* hexagon: conv2d tiled: compute the conv_3d unfold divisors on the host
The KH and OH fastdiv values used to unfold conv_3d kh into (kd, kh) and
oh into (od, oh) were built per op on the device, each with a 64-bit
divide. Pass them in the kernel params instead; they fit in the space the
window variant already reserves, so the struct size does not change.
Assisted-by: Claude Code
* hexagon : preserve fold scheduling in conv2d for small size
hexagon : support arbitrary positive conv2d strides
Support positive horizontal strides for WHCN and CWHN inputs. Preserve
load/shuffle paths for strides 1/2 and use gather for larger strides.
hexagon : improve conv2d tile size selection
Prefer the largest feasible output-channel block and derive spatial
chunk capacity from VTCM usage. Compare capacity and thread-aligned
candidates using per-thread work and chunk overhead.
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* hex-conv: fix compiler warning
---------
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* tests : qualify isnan and signbit with std::
The unqualified calls rely on the C math.h names also being present in the
global namespace, which is not the case for the NetBSD 11.99.9/x86_64 gcc
14.4.0 build reported in the issue. Prefix the calls with std:: to select
the standard C++ overloads declared by <cmath>.
ref: https://github.com/ggml-org/ggml/issues/1681
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : qualify C math calls with std::
Follow-up to the isnan / signbit fix: make the remaining math calls in
test-quantize-fns.cpp consistent with the std:: isfinite / max / numeric_limits
already used in the same file.
The float variants are overloads in C++, not distinct names, so the mapping is
fabsf -> std::fabs, copysignf -> std::copysign, nextafterf -> std::nextafter -
not std::fabsf & friends, which do not exist in libstdc++ and would break the
gcc build. NAN and INFINITY are macros and are left unqualified.
Only behavioural difference: array_rmse() fed a double accumulator to sqrtf(),
now std::sqrt() keeps the computation in double. The result is more accurate
and differs from the previous value by at most 1.4e-7 relative, three orders of
magnitude below the tightest tolerance in this test.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The self-hosted CPU runners are containers with 8 CPUs, but
std::thread::hardware_concurrency reports the cores of the host. The
table mode of test-llama-context sizes its pool from that count, so it
runs 4 models x 8 threads on an 8-CPU job and hangs.
Add TEST_ARG_HARDWARE_CONCURRENCY and TEST_ARG_PARALLEL_WORKERS as the
defaults for the auto-selected counts of test-llama-context and of
test-backend-ops, and set them to 8 and 4 for cpu-x64-high-perf.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* moe-cache : LRFU/MQ-inspired hybrid cache
* make the clock advance by N tokens: use the real token count (ne[1] is always 1, ids have n_expert_used entries per token)
* sel is a RESHAPE node, the orig input is in src[0], so sel->src[0]->ne[1] is the count
* c for the class index, last_use for last use time
* explain the roles of these
* compute the key once
* use the key directly instead of re-reading
* show how the score is computed from the age
* remove the branch and clarify exp2 and log2 relationship
* reword the header comment and elaborate on unused slots and sorting
* not a copy, not just skip
* std::array maybe
* Revert "std::array maybe"
This reverts commit 7649b1f172db4941a854ad115bec0c7e062b65b9.
* half life it is
* add key and iteration helpers
* extract decayed_crf
* bump classes to 8, document the formula, display the cache size in token worth count
* assert the computed number of classes matches the half life
* opencl: use the multi-column q6_K vocab head GEMV for 2..4 tokens on X2E
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q6K_MC3 to GGML_OPENCL_Q6_K_MC3
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q40/Q41/Q4K_MC3 to GGML_OPENCL_Q4_0/Q4_1/Q4_K_MC3
GGML_OPENCL_Q4K_MC3_DQ and GGML_OPENCL_Q4K_MC3_LDS become GGML_OPENCL_Q4_K_MC3_DQ
and GGML_OPENCL_Q4_K_MC3_LDS.
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: remove unnecessary debug probe
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: clean up
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hex-repack: move repacking/tiling into graph-optimize
* hex-tiling: simplify tiling rules and remove obsolete helpers
* hex-ops: use a hash for de-duplicating tensor descriptors
* hex-bufs: improve tensor hash
* hex-ops: accept empty (zero-elements) ops
* hex-bufs: enable hostbuf by default and reject tensors that do not fit into max_vmem
* hex-bufs: improve error message and update readme
* hex-bufs: make sure to sync flush opbatch in all sync set_tensor paths
* feat : added ricursiveai ripple support
* fix : mandatiry SWA, scale through helper, comments back
* fix : metal
* model : make ricursive RoPE/NoPE explicit and validate the layer pattern
Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.
Assisted-by: Claude Opus 5.5
* model : make ricursive swiglu_clamp per-layer
Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.
Assisted-by: Claude Opus 5.5
* model : use the ricursive per-tensor scales and RoPE parameters
Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.
Assisted-by: Claude Opus 5.5
* convert : drop the ricursive chat_template.jinja override
The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.
Assisted-by: Claude Opus 5.5
* fix : remove NVFP4 checkpoint quantization block
* match #31733 refactor
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* RDNA3/4 Vulkan Large-Tile Split-K Optimization
* Add single-operator performance test cases
* Update the policy to use driver-reported LDS usage
* Add comments
* Use the simpler split_k rule from review
Assisted-by: GitHub Copilot
* tests : drop the split_k test cases
* vulkan : simplify the RDNA3/4 split_k condition
* metal : support scale-bias routing norm in the TOPK_MOE fusion
The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sqrt(softplus) MoE routing with a selection bias
Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sigmoid MoE routing with a selection bias
The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.
- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
tail; the selection-bias ADD is required, same as the sqrt(softplus)
chains
- generalize the gating function constant from a sqrt(softplus) bool
to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
deepseek2/32, kimi, minimax, ...) now fuse their routing, and
muse-spark prefill drops one MOE_REDUCE fire because the reordered
graph allocates the routing weights inside the reduce output range,
which the dst/input overlap check correctly declines
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : remove low-signal comments
* ocd : valign ggml_metal_fusions
* CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a)
* Addressing reviewers comments, mma.cuh and physical warp size
* CUDA/HIP: single mma.cuh matrix-core lightning indexer ::CDNA::
* Support RDNA3
* CUDA/HIP: stream heads through the mma lightning indexer kernel
Process 16 heads at a time with double-buffered Q staging instead of keeping
all heads resident in shared memory, and drop the GGML_INDEXER_NO_MFMA env.
* enable for CUDA
* CUDA: Remove wmma lightning-indexer, also run head 32 through mma kernel
* CUDA: Nv mma dosent transpose accumulator
---------
Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* vulkan: add F16 input to the FWHT
The Vulkan FWHT accepts F32 input only. This adds an F16 source variant of the
shader, for both the subgroup and the shared-memory paths, so the kernel reads
an F16 source directly instead of requiring a converted copy.
The F16 shader needs shader-float16, so the pipelines are only created when the
device reports it. A null pipeline makes ggml_vk_can_use_fwht fall back, which
is the same path an unsupported width already takes.
supports_op and the dispatch both already route through ggml_vk_can_use_fwht,
so there is a single place asking whether an op qualifies; nothing to
consolidate here the way the CUDA/Metal ports needed.
test-backend-ops on an M5 Pro (MoltenVK, fp16: 1): MUL_MAT 1128/1128,
including all 24 Hadamard cases (18 existing F32, 6 new F16).
Assisted-by: Codex
* vulkan: FWHT kernels for block widths above 512
The Vulkan FWHT covers block widths 64 to 512; wider blocks fall back to the
dense matmul. Extend the width list to 8192 and pick the shader per width: the
subgroup shader keeps n/subgroup_size values per invocation, so widths above
512 (or a subgroup narrow enough to exceed 64 values per invocation) use the
shared-memory shader with one row per workgroup and a 256-wide block. The rows
per workgroup become a specialization constant so the dispatch matches.
Pipelines for a width are only created when the device's shared-memory and
workgroup-invocation limits allow it; otherwise that width keeps the fallback.
This is the Vulkan counterpart to #29100 (CUDA). Stacked on #29101; the last
commit is the one to review.
Assisted-by: Codex
* checkpoints: reduce overhead
Assisted-by: Codex
* rebase
Adapt the slot save/restore checkpoint appendix from #26004 to the
shared_ptr checkpoint list and the common_state_data buffers.
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: cwriter <cwriter@localhost>
* ggml : support `truncate=false` for YaRN
YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.
Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.
* support CUDA backend
* add tests
* move RoPE helpers to internal header
* format
* conversion : resolve quantized_layers from source tensor names
This commit changes how the per layer NVFP4 quant_algo are resolved from
hf_quant_config.json quantized_layers.
The keys in quantized_layers are module paths in the source checkpoint,
and these can be either be modules like 'lm_head' or parent modules
like 'layers.0.ffn.experts'. Currently these keys are passed through
filter_tensors, but they are not tensor names. So a filter_tensors that
keeps an explicit list of tensor names, like Nemotron's with --mtp
would drop them. That means there was no `_nvfp4_layer_algo entry`, and
`_tag_prec_a4` fell back to `_nvfp4_global_algo`.
This commit records the mapping from source tensor name to filtered
tensor name in index_tensors. Then in prepare_tensors each source
tensor is resolved to its closest quantized_layers parent, and
the algo is stored under the filtered tensor name, which is what
_tag_prec_a4 later looks up.
Converting Nemotron 3.5 Lightning NVFP4 with --mtp now produces:
```console
general.tensor_extra.name = ['output.weight']
general.tensor_extra.prec_a4 = [False]
```
whereas on master no general.tensor_extra fields are written. For the
non-MTP conversion, the general.tensor_extra fields are identical to
master.
Also manually verified (using pdb) that this works for
DeepSeek-V4.1-Flash-1017-NVFP4.
Refs: https://github.com/ggml-org/llama.cpp/pull/31602#issuecomment-6603337403
* conversion : shorten comment
* conversion : also remove '.bias' suffixes
Prevent malformed realtime session updates from throwing an uncaught
exception and terminating the server
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* fit : account for skipped NextN layers when setting n_gpu_layers
Without MTP, fit counts n_layer + 1 layers while the model loader counts
n_layer_all + 1 for n_gpu_layers, including the NextN layers which hold no
weights. A fit result that offloads all layers then leaves layer 0 on the CPU
("offloaded 65/66 layers" for Qwen3.8-27B on two GPUs).
Add the skipped NextN layers to n_gpu_layers and to the tensor split of the
device that holds the output layer so the loader assigns the layers that
fit measured.
Fixes#31531
* fit : pass hp_ngl_skip by reference like the other hparams
* fit : add hp_ngl_skip to common_get_device_memory_data
* graph: store the conv state of every rollback arch in one set_rows
The rollback slots of the causal conv history were written with one
copy per slot in qwen4exp, glm5-next, kimi-k3, bailingmoe3, lfm and
mamba2, while delta_net already wrote them with a single set_rows.
build_conv_state_store now does this for all of them: one copy without
rollback, otherwise one set_rows of the overlapping windows into the
min(n, n_rs_seq + 1) slots a rollback can read.
The slot order {n_rs_seq, ..., 0} is computed once per graph and each
conv layer takes a view of its tail. Its size does not depend on the
ubatch, so the reserved graph covers every ubatch shape, and the graph
input that held it is gone.
A model that is not causal never rolls back, so the context clamps
n_rs_seq to 0 for it, like for an arch without rollback support.
* nits: compute the conv slot order in the graph and keep the n_head name of bailingmoe3
* graph: build the conv slot order in each conv layer
The slot order {K-1, ..., 0} of the conv state store was one tensor
shared by every conv layer of the graph, built on the device of the
first one, so each layer on another device copied the view of its
tail. Each conv layer now builds its own order next to its set_rows,
and llm_graph_input_rs no longer holds it.
* tests: fix CI flakes of TOPK_MOE caused by near ties
With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.
Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.
* tests: build the TOPK_MOE logits by inverting the gating function
Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.
The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
The first command buffer takes the first 10% of the graph nodes and the
rest is split evenly between the extra command buffers. A fusion cannot
span two command buffers, so a boundary that falls inside a fusable
group runs that group unfused, and the output then depends on the node
count of the graph.
Move each boundary to the end of the fusable group it falls into, so
the encoders fuse the same nodes as with a single command buffer.
This commit adds nvfp4 scales to qwen2's build_ffn call which are
currently passed as NULL.
The motivation for this is that a model converted to NVFP4 will
currently crash because the scales are loaded but never applied in
the FFN:
```console
llama-completion: llama.cpp/src/llama-sampler.cpp:1218:
void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*):
Assertion `found' failed.
```
The attention already applies its scales but ffn is missing them.
* opencl: use the small-N f32 GEMV by default on X2E
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: use the q5_K multi-column verify GEMV by default on X2E
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* cuda: fuse NVFP4 gate/up and GLU in MMQ
* cuda: honor the requested src1 precision in fused gate/up MMQ
* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels
* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming
* cuda: check gate_scale before loading the fused NVFP4 gate scale
* llama : fix on-device state copy sizes for quantized KV caches
The on-device sequence state copies (LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) computed
their element count as size / ggml_element_size(). For a quantized type such as
q8_0 or q4_0 that is the number of blocks, not elements, so only part of each
range was copied: the restored state was wrong, or the copy crashed.
Compute the count as size / ggml_type_size() * ggml_blck_size() for the save,
the restore and the copy between differently chunked ranges, and assert that
each range covers whole blocks.
* llama : drop the one-line comments
* llama : drop the redundant cast in the state copy size
* server : realtime: send an error event instead of crashing on exceptions
* server : route websocket handlers through ex_wrapper, remove the realtime test
* server : realtime: keep the previous tools when the chat template rejects new ones
* server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapper
* server : realtime: clarify the tool schema check comment
* opencl: use more workgroups for q4_K gate up + GLU
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: handle FA for GQA 2 and 6 with head size 256
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: handle FA for GQA 16 with head size of 128, 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: handle FA for GQA 2 with head size 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* ui : share directory resolution and the cwd row
Move the lookup of a browser-picked folder to ToolsService.resolveDirectoryPath, let the working directory picker run without its chip, and split the 'Set working directory to' row into its own component, so other views can pick and show a folder the same way.
Assisted by: Claude (Opus 5.5 Medium)
* ui : open an item's actions on right-click
Sidebar conversations and models manager rows open the same actions as
their "more actions" menu on right-click. Both menus render one shared
list of actions, so they cannot drift apart. Phones keep the browser's
long-press; the dropdown's drawer already holds the actions there.
An action can render a flyout submenu in place of its row, through the
kit of the menu it is in, and close the menu after a pick.
Adds the shadcn-svelte context-menu component, mirroring the
dropdown-menu one.
Assisted by: Claude (Opus 5.5 Medium)
* ui : close conversation tabs from a right-click menu
A tab's right-click menu closes it, the other tabs or the tabs to its
right, and stops a response the tab is generating. Closing tabs that
include the active one keeps the tab the menu was opened on.
Assisted by: Claude (Opus 5.5 Medium)
* ui : open a message's actions on right-click
User and assistant messages open the same actions as their action icons
on right-click. Selected text, links, media, code and fields keep the
browser's own menu, so copying them works as before. The fork dialog
opens from the actions context, like the delete one, so both the icon
and the menu can open it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : scope right-click menu shortcuts and selection to their item
Show the conversation shortcuts only on the active row, as they act on
the active conversation, and keep the browser menu only for a selection
inside the message. Name the close-tab shortcut, and drop the unused
chip toggle, menu submenus and shared cwd row.
Assisted-by: Claude (Opus 5.5 Medium)
* fix(ui): show extracted text in PDF attachment preview
Assisted-by: Claude (Opus 5.5 Medium)
* feat(ui): show page thumbnails for PDF attachments sent as images
Assisted-by: Claude (Opus 5.5 Medium)
* ui: stop retrying PDF preview conversion after it fails
A failed PDF-to-image conversion or text extraction reset its loading
flag, which re-ran the effect and retried forever. Remember the failed
item for each view and only retry when the item changes, so the error
state stays on screen.
Add a PdfConversionError story that feeds a broken PDF.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the text of PDFs sent as images and tidy the PDF preview
Store the extracted text of a PDF sent as images in its content, so the
Text view reads it instead of parsing the PDF again; only messages stored
with the old placeholder extract it. The model still receives the images
only. Guard retries on the error state, as the item remounts per id, skip
an empty base64 payload, and share the status markup between views.
Assisted-by: Claude (Opus 5.5 Medium)
* gguf-py : validate every tokenizer merge pair, not only the first
The transformers 4.45 pair format was detected by checking merges[0],
then every element was converted unchecked: a non-pair element raised
a bare TypeError, and pairs with the wrong arity or non-string parts
were silently mangled into invalid merges.
Validate each element (2-string list) up front and raise the existing
"Unknown tokenizer merges format" error for anything else.
Assisted-by: Devin
* gguf-py : remove test_vocab.py per review
The RISC-V vector intrinsic specification requires __riscv_v_intrinsic
to be defined when the compiler supports the API, even when the vector
extension is not selected. Clang therefore defines it for rv64gc,
causing ggml to enter RVV branches where vector types are rejected
because the required ISA extensions are absent.
Use __riscv_v for optional RVV implementations; this macro represents
the selected target ISA. Retain __riscv_v_intrinsic checks in SpaceMIT
sources, where they validate mandatory compiler API support rather than
select optional vector code.
Closes: https://github.com/ggml-org/ggml/issues/1535
* ggml: bound views by the bytes their strides reach
A view was checked against its source with the size of a contiguous
tensor of the same shape, computed before its strides were set. Views
whose rows overlap failed that check although they stay inside their
source, and views with wider strides passed it while reaching past
the end. The view functions now pass their strides to the tensor
constructor, which checks the bytes they actually reach, like
ggml_nbytes. The conv windows of delta_net are created with all their
rows at once instead of being widened after the check.
* ggml: assert that a strided view starts from contiguous elements
The view functions set nb[0] to the type size, so the source must have
contiguous elements along dim 0 for the view to read them.
The gguf writer staged each tensor into a temporary vector before
writing it. For host-buffer tensors the data pointer is directly
writable, so skip the staging copy and write it straight to the file.
This avoids the per-tensor allocation, zero-init and memcpy.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* common : fix -hf tag matching inside longer quant names
":F16" picked a BF16 file when the repo had both, since "BF16"
contains "F16". The same applied to ":Q4_XS" vs "IQ4_XS". The tag
now has to start at a token boundary in the path.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* common : accept underscore as -hf tag boundary
Some repos separate the quant with an underscore, e.g.
stepfun-ai/Step-3.5-Flash-GGUF-Q4_K_S ships
step3p5_flash_Q4_K_S-00001-of-00012.gguf, which :Q4_K_S no longer
matched. Add _ to the token boundary and pin the tag and default
lookup cases.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* hexagon: initial support for LIGHTNING_INDEXER op
* hex-lght-idxr: hmx for prefill
* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute
* hex-lght-idxr: pipelined hmx path for decode and prefill
* hex-lid: add support for K (f32)
* hex-lid: add support for K (f32)
* hex-lid: add support for q4_0
* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills
* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel
* hex-lit: update dma pipeline to use canonical dbl buffering in all paths
* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths
* hex-lit: add support for dims collapse
* hex-lit: make kparams constistent with other ops
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* Optimizing Gemma4 graph for prompt processing
TODO:
- Currently complete microbatch needs to contain tokens which does not require the logits; need to optimize this further
- There are a few more operations that are performed after the kv cache is calculated, need to safely optimize them.
* Not filtering the rows when they are unmasked
* Cropping out the attention mask too.
* gemma-4: improved pruning, mask cropping and PLE scaling
Prune before the first KV-shared layer with the single-stream, causal, non-embedding safety gates.
Use cropped masks only during shared-layer attention.
Preserve requested layer-input embeddings in the KV prefix.
Remove the PLE contiguous materialization and fold equivalent PLE scaling into RMSNorm parameters/weights.
* spec-mtp: Enable masked h_nextn extraction only when the draft context shares the target KV cache
This lets Gemma 4 prune unused prompt rows before its shared layers so that we keep the full perf P.P gains with MTP.
Avoid extracting target hidden states for output-free prompt chunks in that shared-MTP path.
When a batch has a selected final token, extract only that row unless the whole verification batch is selected.
Keep ordinary MTP and classic speculative decoding behavior unchanged: they still use unmasked h_nextn because the draft
context must catch up on all target rows.
speculative-simple needs an update to select the final prompt row only for shared-KV MTP,
so the optimization works there without changing other speculative modes.
And it looks like the server already selects its final prompt row for first-token sampling.
* spec-mtp: make sure to not drop first draft round
* gemma4: enable pruning based on static cparams.kv_unified/n_seq_max=1
* gemma4-mtp: simplify prunning logic and avoid graph branches/re-allocs
* gemma4: update pruning gates to use only n_seq_max==1, kv_unified can now be used for multi-seq
* spec-simple: revert spec-simple changes for now
* gemma4: remove causal_attn from pruning gate (last source of reallocs)
* cont : adapt to llm_attn_mask changes
* cont : add TODO
---------
Co-authored-by: Meet Amin <meetamin378@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* hexagon: faster TOP_K for the backend sampler (skip tiles, then block scan)
Skip the bitonic sort for tiles whose max cannot reach the running k-th
best value, then replace the tile sort with top_k_scan(): reject 256-float
blocks with one HVX max-reduce and insert only surviving elements into a
sorted k-entry list. Used for k in 2..64 and rows wider than 1024, in both
the single-row and multi-row paths.
Ties keep the first-seen index. NaN is treated as -INF.
* hex-top-k: eliminate scalar reads from vtcm
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Playwright's bundled Chromium lags behind Chrome security releases
(1.56.1 ships 141, and even 1.64.1 ships 156.0.8078.4, before the
156.0.8078.73 fixes), and Playwright launches it with --no-sandbox by
default. Outside CI, use the installed, auto-updated Chrome with the
sandbox enabled. CI keeps the bundled Chromium, so the self-hosted
runner's preinstalled 1.56.1 browsers still work.
Assisted-by: Claude (Opus 5.5 Medium)
* hexagon: add support for DSV4_HC_PRE
* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization
* hex-dsv4-hc-pre: double-buffered DMA for prefill
* hex-dsv4-hc-pre: refactor DMA block setup
* hex-dsv4-hc-pre: precompute kernel params
* hex-dsv4-hc-pre: support for n_hc != 4
* hexagon: add support for DSV4_HC_POST
* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block
* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks
* hex-run: add support for test-llama-archs tool
* hex-dsv4: refactoring and optimizations
Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.
Tested with test-llama-archs -a qwen4ext and test-backend-ops
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* speculative : save and restore MTP hidden state
Assisted-by: Codex
* tests : remove MTP state test
Assisted-by: Codex
* speculative : simplify MTP checkpoint state
Save only the boundary hidden row; verification rows are rebuilt on the supported restore paths.
Assisted-by: GPT-6
---------
Co-authored-by: zouyongxing <zouyongxing@bytedance.com>
* mtmd : add Nemotron 3.5 Transcribe support
Assisted-by: Claude
* convert : fix vocab of draft models for Nemotron-H targets
Assisted-by: Claude
* convert : reject non-BF16 Transcribe checkpoints
Assisted-by: Claude
* mtmd : make speaker FIFO predictions local
Assisted-by: Claude
* mtmd : normalize the mel in encode_windows
Assisted-by: Claude
* convert : map the generated speaker kernel
Assisted-by: Claude
* mtmd : validate Nemotron Transcribe mmproj shapes
Assisted-by: Claude
build_conv_state_at makes a contiguous copy of the conv input tail
before writing it into the recurrent state, so every GDN layer issues
two device copies per generated token where one is enough. On
Qwen3.8-Flash-Next (36 GDN layers) that is 72 copies per token, now 36.
Speculative decoding adds one more per rollback slot. ggml_cpy reads
the strided view directly, as the shared build_conv_state already does,
so the extra copy is dropped.
* ui: add Playwright harness for recording demo videos
Scripted walkthroughs under tools/ui/demo are recorded with Playwright and
exported as H.264 mp4 for pull requests (demo/pr) and docs (demo/docs).
Fixture MCP servers and seeded settings keep recordings deterministic, a
drawn cursor and captions make the flow readable, and DEMO_UI_DIR points
the recording at another checkout such as a PR worktree.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: record demo scripts from PR descriptions and a configurable storage
Demo scripts no longer live in the repo. `npm run demo:pr -- <number>`
records the `ts demo` block from a PR description against the PR head,
uploads the mp4, a GIF preview and the script to DEMO_STORAGE, and links
the preview in a `## Demo` section of the PR description.
`npm run demo:docs` records the docs scripts kept in DEMO_STORAGE.
DEMO_STORAGE (hf:// buckets or file://) and the public URL are read
from tools/ui/.env.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: import generated conversations into demo recordings
Demos can now start with existing chats: `test.use({ traces })` imports
llama.app JSONL traces through Settings > Import/Export before the video
starts. Built-in traces are generated deterministically, other names are
downloaded from DEMO_TRACES_DATASET, and `demo:traces` pushes the built-in
ones there. Adds `demo.scroll` for wheel scrolling.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep demo captions clear of the chat form and isolate concurrent runs
Captions move to the top of the frame, since at the bottom they covered
the chat form a demo is often about. Each recording run gets its own
Playwright output folder, so a run starting elsewhere no longer clears
another run's video before it is saved.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: record demos at 2x and 60 fps with human-like timing
Recording goes through Chromium's screencast instead of Playwright's
video, which encoded VP8 at about 1 Mbit/s at 1x. Frames are captured at
the device pixel ratio (DEMO_SCALE, default 2) and encoded into a 60 fps
H.264 mp4, starting once the page has settled.
The cursor now glides faster along a slight arc with Fitts-like
durations, settles briefly before each click, and typing and pauses vary
a little. The randomness is seeded per demo, so re-recording gives the
same motion.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: document the demo harness prerequisites
The demo README now lists the tools the harness needs and what each is
for, and walks through creating the Hugging Face bucket and the optional
traces dataset, with the matching .env settings.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: version linked demo URLs so re-recorded previews refresh
GitHub's image proxy caches a preview by URL, so a demo re-recorded at
the same PR head kept showing the old GIF. The Demo section now links
the files with a content hash in the query string.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: label PRs that feature a video demo
Adds a `video demo` label and a workflow that applies it when a PR
description has a demo written by the UI demo pipeline (the managed
demo:start/demo:end markers, or a preview linking to a demo video),
and removes it when that section is gone.
Assisted-by: Claude (Opus 5.5 Medium)
* ci: label video demos with gh instead of github-script
The label workflow now runs the gh CLI in a shell step, like
pr-draft-label.yml, so it needs no action outside the repository's
allowed list. The video link check only counts videos uploaded for the
PR itself, so a description linking other PRs' demos isn't labeled.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: zoom demos in on details with a two-finger pinch
demo.zoom pinches in on a control or a point, holds, and pinches back out
to 1x, so a detail too small to read in the frame can be shown up close.
It drives Chromium's own pinch gesture, so the page's fixed bars stay put,
and works at desktop and phone sizes alike.
The pinch is evened out where Chromium misses: one anchored near the top
does not zoom at all, so it starts lower, and one near the bottom
overshoots its ratio, so a small pinch from the middle corrects it. The
way out reads the real scale until the page is back at 1x, which the next
gesture needs.
The overlay also draws a fingertip per touch, so a pinch on a touch
device shows as two fingers.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: wait out a cold dev server before a demo starts
A fresh checkout's dev server optimizes its dependencies on the first
load and reloads the page, which can take well past an action's timeout,
so a demo failed before it started. The demo now waits up to 90 seconds
for the app; the recording only starts once it is ready.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: record a video per test in a PR demo, shown in a table
A large PR can split its demo into several shorter videos: each test()
in the ts demo block is recorded as its own video (<sha>-1.mp4, ...),
and the Demo section lays their previews out in a table, each column
headed by its test's title.
GIF previews now keep their longer side at DEMO_PREVIEW_WIDTH and shrink
until they are under 5 MB, since GitHub's image proxy refuses anything
larger and shows a broken image instead.
Importing traces works on a phone viewport too, where Settings sits in
the sidebar under the page until the menu key slides it aside.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fix demo trace imports on desktop and lay landscape demos out two to a row
Importing traces failed on a desktop whose collapsed sidebar sits under
the page: it now expands the sidebar from the page's menu key on a phone
or from the sidebar's logo on a desktop, whichever makes Settings
reachable. Afterwards it forgets the expanded sidebar, so a demo starts
with it collapsed, as on a first visit.
A PR demo with several videos shows up to four portrait (phone) videos
to a row, or two landscape ones, so each preview stays readable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: only record other people's PR demos at a reviewed head commit
demo:pr runs the PR's code and demo script on this machine, with the
gh and hf logins, so a PR by someone else now needs --trust <sha>
naming the head commit that was read, and is refused again after new
pushes. Recordings also launch Chromium with its sandbox on, which
Playwright turns off by default.
Assisted-by: Claude (Opus 5.5 Medium)
ggml_roll validated each shift with abs(shift), but abs(INT_MIN) is signed
integer overflow UB evaluated before the bounds check runs. Compare with a
signed-safe test instead.
This commit updates the _flush_nvfp4_experts to take a new parameter
which is a dictionary mapping the key to the original model tensor name.
The motivation for this is that I noticed that the tensor names being
looked up were not the original model tensor names, for example:
```console
model.layers.1.mlp.experts.up_proj.weight
```
But _nvfp4_layer_algo uses the original per-expert names:
```console
(Pdb) p self._nvfp4_layer_algo
{
'backbone.layers.1.mixer.experts.0.up_proj': 'W4A16_NVFP4',
...
```
This means that the lookup will never match for experts and fall back to
_nvfp4_global_algo. If the global algo is W4A16_NVFP4 then this would
go unnoticed, but for a model like
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 which is
MIXED_PRECISION, the expert tensors will not get prec_a4=false so they
would use the native W4A4 path on Blackwell even though the original
model tensors declare them as W4A16_NVFP4.
* cuda: gather batched tensor reads into mapped host memory
Assisted-by: Codex
* add comments
Assisted-by: Codex
* simplify the implementation based on checks that exist earlier in the flow
* simplify batched tensor gather indexing
* add conditional dispatch
* address comments - make host registration raii, better test mindful of ci time, assert instead of return false
* cuda : gather small reads and use pinned DMA for large reads
* cuda : use std::vector for host gather descriptors
* cuda : add 8-byte gather copies, generalize tensor test and log registration failures
* common buffer for descriptors
* simplify stream sync + increase to 128k descriptor pinned buf
* Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available.
* Drop ggml_backend_comm_allreduce_rank_t and reuse ggml_backend_comm_allreduce_tensor for per-rank communicators
* reuse ggml_backend_cuda_comm_allreduce_nccl for per-rank communicators. Remove the helper function backend_comm_proc_address
* Change ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t to more generic multi-process backend communicator interfaces
* set maxCTAs in ncclConfig_t to 8 for dgx spark in ggml_backend_cuda_comm_init_rank
* meta: support GLM-5.3 next sm tensor
* glm5-next : cont q_absorbed for -sm tensor with non-unified KV
* meta : keep GGML_OP_ARANGE in ggml.h order
* glm5-next : split MLA heads in pairs for -sm tensor
The MLA cache has a single KV head, so the heads on a device are its GQA
ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3
devices the 16 heads of the test model were split 5/5/6 and it aborted.
* meta : keep the split for views inside one slab of the split axis
With a non-unified KV cache, build_attn_mha splits q into streams with a
view. For a permuted q, e.g. the absorbed query of glm5-next, that view
aborted with "view of permuted tensor not implemented". The split is now
kept if every other dim of the source and the view stays inside one slab
of the split axis. The per-device strides are only rescaled above it.
From tetherto/qvac-fabric-llm.cpp#353.
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
* glm5-next : drop the q_absorbed cont
The meta backend can now split the per-stream view of the permuted q, so
the cont is not needed for -sm tensor with a non-unified KV cache. With
the cont, the q of flash attention was not contiguous, and when FA ran on
the CPU (WebGPU) the meta get_tensor aborted.
---------
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
kernel_cpy_t_t indexed the dst innermost dim with an implicit element-size
stride instead of args.nb0, so cpy into a permuted view (nb0 != type size)
wrote the wrong addresses. caught by the new test-backend-ops cpy case with
permute_dst = {1, 0, 2, 3} from #31446. addressing is unchanged when dst is
contiguous in dim 0.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* vulkan: skip maxStorageBufferRange pre-check for non-compute tensors
Pre-allocated leaf tensors and views (GGML_OP_NONE / GGML_OP_VIEW) are never
executed by the backend, but were still rejected by the descriptor-size
pre-check in ggml_backend_vk_device_supports_op when larger than
maxStorageBufferRange on devices without shader_64b_indexing. The scheduler
then aborts with 'pre-allocated tensor (...) in a buffer (Vulkan0) that cannot
run the operation (NONE)' - e.g. the MoE expert cache banks (llama-moe-cache)
of a 256-expert model with --moe-cache-mib on GCN GPUs, where a whole bank
(2.4 GB) exceeds the limit while the consuming ops only access small
per-step views.
Skip the size pre-check for non-compute ops (NONE and VIEW); the buffer type
allocation limit still bounds the actual allocation, and compute ops keep
validating their sources.
Tested on AMD Radeon Pro VII (gfx906, no shader_64b_indexing):
- before: GGML_ABORT 'pre-allocated tensor (moe_cache.0.blk.0.ffn_down_exps.weight) ... cannot run the operation (NONE)' with llama-cli -ngl 99 --cpu-moe --moe-cache-mib 6144 (Cyber-Tiel-Coder-35B-A3B, qwen35moe, 256 experts per layer)
- after: loads and generates at 8.1 t/s (vs 0.6 t/s without the cache)
- regression: -ncmoe 16 path unchanged (19.2 t/s), qwen3moe/qwen35moe without cache unchanged
Fixes#30697
* vulkan: use ggml_op_is_empty for the non-compute size-check skip
---------
Co-authored-by: FamousMonsterr <FamousMonsterr@users.noreply.github.com>
* delta_net: access conv state via views
* fix sm tensor
* delta_net: fill the conv slot index on the host
The rollback slot index of the conv windows is a constant per graph
shape. It is now an I32 input of the recurrent memory filled on the
host, instead of an arange and an argsort computed on the device, and
the meta backend no longer needs a mirrored split for ARANGE.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup
On the non-OCP path, ue4m3_fp32_lut has 128 entries and is indexed with the raw scale byte. ggml_validate_row_data accepts every byte and ggml_ue4m3_to_fp32 ignores bit 7, so a byte >= 0x80 reads past the shared array. Vulkan robustness does not cover shared memory.
Size the table, both init loops, and the two host shared-memory budgets to 256 entries. ue4m3_to_fp32_build() already masks the exponent and treats only 0 and 0x7F as special, so entries 128-255 match the CPU. A 256-entry table keeps 0xFF distinct from 0x7F: the CPU decodes 0xFF as a non-zero value and 0x7F as zero.
Each NVFP4 pipeline uses 512 more bytes of shared memory. ggml_fp32_to_ue4m3 never sets bit 7, so models quantized by llama.cpp are unaffected.
Assisted-by: Claude
Assisted-by: Codex
* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup
Assisted-by: Codex
---------
Co-authored-by: Dongsheng Luo <7137760+flyingdoog@users.noreply.github.com>
The dspark block mask is now derived at input-fill time: llm_attn_mask::block_uniform
fills every token of a sequence with the mask row of its first token, so the
graph-level reshape/repeat in dflash is gone and cnv always aliases raw.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cuda : prevent premature eviction of graphs about to be used
* add last_graph_use_time for entire cuda graph map and testcase for cuda graph
Assisted-by: Codex
* delete the test
A cache that shares the cells of another one (f.ex. the gemma4 assistant) can build a graph before its sequences have any cells, f.ex. at the server warmup. get_n_kv_rows() then returned 0, which means that kv_rows are not used, so the graph fell back to the dense KQ mask and the scheduler re-planned for it. Later graphs with larger masks then aborted with GGML_SCHED_DEBUG_REALLOC=1.
* hexagon: add tiled HVX transpose path for permuted CONT/CPY
Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.
Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.
Host:
- drop extent-1 dims and merge dims that stay adjacent in src into a
canonical view, stored in the kernel params. There are no minimum
size limits.
- the copy is treated as a 2D transpose between dst dim 0 and the
contiguous src dim, batched over the remaining dims.
- tiles are sized to 64 KB per buffer and halved until they fit in
session VTCM. blkk is capped so the halfword vgather offsets fit in
16 bits.
Kernel, per tile:
- 2D DMA of blk0 strided src rows (blkk contiguous elements each)
into VTCM
- vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
- 2D DMA of blkk contiguous rows out to dst
Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.
Small dims:
- packed mode: when dim k follows dim 0 and dim 0 divides the HVX
lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
whole dst rows, and each tile goes out as one contiguous DMA.
- when the contiguous src dim fits in one vector, it is staged exactly
and the tile is made long along dim 0.
All other copies select the same kernels as before.
Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.
* hexagon: cpy transpose cleanups
- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)
Assisted-by: Claude Opus 5.5
* hexagon: use cpy transpose path for ne[0] == 1 permutes
- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests
Assisted-by: Claude Opus 5.5
* hex-cpy: further improve cpy
Bring in improvements from aizip branch/PR.
Cleanup kparams and replace divs with fastdiv.
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* hex-cpy: avoid vtcm bank conflicts
* hex-cpy: trim down the comments
* cpy-test: trim down new tests that cover permute scenarios
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* feat: support PocketTTS voices with no voice cloning
The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.
Assisted-by: Claude Fable 5.1
* convert : pocket-tts voices never fail the conversion, log the drift of the state
The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.
Assisted-by: Claude Fable 5.1
* mtmd : list the predefined voices, expose them in GET /props
mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.
Assisted-by: Claude Fable 5.1
* server : an unknown voice is a bad request
A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.
Assisted-by: Claude Fable 5.1
* refactor api
* fix conversion
* small renaming
* fix lint
* tts : a model with predefined voices speaks with the first one by default
Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.
Assisted-by: Claude Fable 5.1
* mtmd : one comment per voice accessor, converter states the rule only
The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.
Assisted-by: Claude Fable 5.1
* clean up redundant code and comments
* mtmd: match the LFM3-Audio voice name case-insensitively
The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* llama : use llm_attn_mask and kv_rows for unified attention
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont : optimize `get_seq_cells`
* llama : build the kv_rows lists from per-sequence cell bitmaps
Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.
Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).
* cont : make get_kv_rows_p_min private
* cont : minor metal
* llama : use kv_rows for every ubatch of a unified cache
A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.
Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.
* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The note under 'Run the inference' still said that the CANN backend only
supports FP16/Q4_0/Q8_0 models, while the DataType table in the same file,
and ggml-cann.cpp (BF16 is enabled for MUL_MAT, GET_ROWS, CPY and CONT for
every non-310P device), say otherwise. Spell out the per-device support and
mark BF16 as unsupported on 310P instead of leaving the cell empty.
GGML_CANN_NO_PINNED is read by ggml-cann.cpp when allocating host buffers
and when advertising the host_buffer device capability, but it was not
documented, so the escape hatch for a failing pinned host allocation was
not discoverable.
Signed-off-by: li-lizhe <147392333@qq.com>
* vulkan: add tensor-parallel all-reduce communication
* vulkan: require shared semaphores for tensor communication
Assisted-by: Codex
* vulkan: reduce directly from shared host buffers
Assisted-by: Codex
* vulkan: fix ring all-reduce races with 3+ devices
Two hazards in the tensor-parallel ring all-reduce made -sm tensor output
non-deterministic with three GPUs:
- With n = 3, step t's host-slot region starts at t * cels * 2 bytes, which
is not a multiple of 32, so consecutive steps' transfer copies share a
32-byte sector of the imported host memory. On NVIDIA the later copy's
leading bytes (up to 10 F16 values at the start of every chunk) end up
stale in host memory, even when the copies are ordered by a semaphore.
Give every step its own 256-byte-aligned region. n = 2 and n = 4 already
had aligned regions for the usual sizes, so their layout is unchanged.
- A single-shot all-reduce leaves every device's host slot being read by all
peers, but the next ring round only ordered its first transfer after nextd,
and only from round 2 on. With n >= 3 a device could overwrite its slot
while prevd was still summing it. Wait for all peers on the first ring
round after a single-shot all-reduce.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vulkan: fold the tensor-parallel all-reduce into the compute submissions
With -sm tensor the meta backend drives every device from one thread and calls
graph_compute once per subgraph (two per layer), so decode was bound by the CPU
cost of submitting, not by the GPUs: ~1500 vkQueueSubmit per token for a 64-layer
model on two GPUs, with the GPUs idle ~40% of the time.
- Don't split subgraphs by flops while a communicator is active. The heuristic
was tuned for whole graphs and turned each small subgraph into ~5 submits.
- Fold the small (single-shot) all-reduce into the compute submissions: the
last command buffer of a subgraph stays open, the all-reduce appends the
gather copy to it and submits it, then opens the next command buffer with the
peer waits and the reduce, which the next subgraph is recorded into. Each
subgraph now costs one submit per device. The reserved progress value is
attached by whichever path submits the open context.
- Alternate between two host slots, placed outside the ring's regions, so the
gather no longer needs a WAR wait on the peers' previous reduce.
- On synchronize, submit the deferred work of every device in the group first,
so the tail of the graph runs on all devices in parallel.
- Wait for the transfer queue as well as the compute queue before freeing or
reallocating communication buffers, and don't leave a stale cap behind when a
reallocation fails.
qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 32.2 -> 49.7 t/s, 3 GPUs 20.9 ->
40.0, 4 GPUs 15.0 -> 27.6; pp512 on 2 GPUs 2138 -> 2320. Greedy output is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* - Two-phase all-reduce over shared host memory: write own slot + signal, wait peers + add from their slots
- One exported timeline per device, no backend hooks; comm work recorded in the backend compute ctx
- F16 wire above 1 MiB, F32 below; small reductions ride with the next graph submit
- Double-buffered, grow-only host slots; zero-size splits zeroed
- Remove ring, transfer-queue leg, device staging, shrink, Windows shims
- comm_init returns nullptr when unsupported; add GGML_VK_COMM_DISABLE
- No flops-based submit splitting for graphs below 1 GFLOP
[rebased onto the ring all-reduce fix and the folded all-reduce, which this design replaces; their backend hooks are removed here]
* add minimal submit split size
* replace opaque_fd with sync_fd for spec compliance
* add precision check
* add semaphore pool
* rename env var, add logging if requirements not met
* reduce comments
* code improvements
* vulkan: use posix_memalign for the tensor-parallel host buffers
std::aligned_alloc is unavailable on Android bionic below API 28, which
upstream switched away from in #30828; do the same for the communicator's
host slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vulkan: synchronize the tensor-parallel all-reduce through the host
Exchanging SYNC_FD semaphores costs a kernel wake-up on every all-reduce:
on NVIDIA the waiting GPU starts ~35 us after its peer has finished, which
is most of the idle time in a decode step. OPAQUE_FD semaphores avoid that
but must not be imported into another physical device.
Use only device-local timeline semaphores instead. Each device signals
done[i] once its partial sum is in host memory (after a barrier that makes
it available to the host), and a bridge thread signals go[i] from the host
once all peers of i are done. The waiting side makes the host memory
visible with a host-write barrier, since its submit predates the data.
The thread spins for up to 200 us on each wait (GGML_VK_ALLREDUCE_SPIN_US)
and then blocks, so it is idle when there is no work. This also takes the
per-round fd export, dup and import calls off the main thread and drops the
VK_KHR_external_semaphore_fd requirement and the binary semaphore pool.
Graphs below 1 GFLOP, such as tensor-parallel decode slices, are no longer
split into several submits.
qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 37.2 -> 45.4 t/s, 3 GPUs
24.1 -> 35.5, 4 GPUs 16.5 -> 27.1; pp512 unchanged. Greedy output is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* use sync_fd on mesa drivers
* refactor for readability
* move sync_fd semaphore handling to thread, trigger mesa submit threads
* move allreduce functions to new cpp file
* clean comments
* smaller fixes, code improvements
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* Update ggml-sycl.cpp
Increases -sm tensor prompt eval t/s by +19% with zero PPL change.
* Update ggml-sycl.cpp
Replace first pair of wait() calls with ext_oneapi_submit_barrier(). The wait() calls was discovered to cause MTP degradation.
* Update ggml-sycl.cpp
Adding comm_exchange function pointer to switch between 2 and 4 buffer ggml_sycl_comm_exchange versions @ initialization time (versus hot path) controlled by SYCL_ENABLE_DEFAULT_CONTEXTS=1 setting.
* Update ggml-sycl.cpp
adding back 2nd half of 4-buffer free.
* Update ggml-sycl.cpp
Removing extra whitespace.
* Update ggml-sycl.cpp
Excluding Windows until it is validated on Windows.
The router only removes models it took from the Hugging Face cache, but
the models manager offered "Delete from disk" for every model, so a model
from --models-preset or --models-dir failed with a "not removable" toast.
Read the source and can_remove fields of /models and show the delete
action disabled for those models, with a tooltip (a hint line in the
mobile drawer) saying how to remove them instead.
Assisted-by: Claude (Opus 5.5 Medium)
Use the popover background, softer border, rounded-xl corners and
shadow-md like dropdowns, hover cards and the settings dialog, and
match the dialog title line height.
Assisted-by: Claude (Opus 5.5 Medium)
A message queued while a conversation was busy was re-sent through
sendMessage, which targets the active conversation, so switching chats
before the turn ended delivered it into the wrong one. Re-send only when
the owning conversation is active; otherwise keep it queued there and
flush it when that conversation is opened again.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show interrupted state for tool calls that never returned
The chat-level streaming flag was applied to every assistant message, so a
tool call left without a result in an earlier message kept spinning while a
newer reply streamed. Scope streaming to the last assistant message and mark
finished calls without a result as interrupted.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: mark a shell command stopped before any output as interrupted
exec_shell_command creates its result message before running, so stopping
it mid-run leaves an empty result instead of none.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: don't reload a loading model on select, prefer it in pre-selection
Assisted-by: Claude (Opus 5.5 Medium)
* ui: skip hidden models in automatic model selection
Assisted-by: Claude (Opus 5.5 Medium)
* ui: don't count a model being unloaded as loading
Assisted-by: Claude (Opus 5.5 Medium)
* webgpu: keep scratch alloc sizes monotonic in the batch length
The flash attention vec path and the MMVQ path append a scratch area
after dst, but get_alloc_size only accounted for it when the current
batch takes that path. The size was therefore not monotonic in the
batch length: a graph reserved at n_ubatch (tiled path, no scratch)
or at one token did not cover batches of 2 to 19 query rows, and
GGML_SCHED_NO_REALLOC builds aborted on an unexpected reallocation.
Pass the row count to both path predicates and size the scratch with
the row count clamped to the longest batch the short path serves.
* nit: align the mmvq pipeline key with clang-format 22
* webgpu: address review @yomaytk, name the mat-vec row threshold
* hexagon : fix DMA transfers with strides > 24 bits
On v75+ the 2D DMA descriptor stores row_size and strides in 24 bits.
dma_queue_push sent oversized strides to fallback_2d, which only splits
nrows, so strides of 16 MB or more were truncated - wrong addresses.
Fix will fall back to one transfer per row in that case.
Also added two CONT tests with source strides over 16 MB.
* hex-dma: refactor the fallback paths to remove duplication
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Limit the shared prefix to the first required output position of the parent and each child task.
This keeps decision output embeddings available in the final batch when questions share option tokens.
Fixes#30482
Signed-off-by: aiqubits <aiqubits@hotmail.com>
* chat : accept the think block in the qwen3-coder response format with reasoning_format none
Assisted-by: Claude Opus 5.5
* chat : define both qwen3-coder think block variants up front
This commit extracts all apple jobs except for macos-latest-arm64 into a
new daily workflow.
The motivation for this is to reducd the load on the macos runners.
The duplicated output looks up token_embd in its buffer context, so
token_embd must exist first. With the old order a load where input and
output share a buffer type (CPU-only, -ngl 0) held the table twice:
gemma-4-E2B-it Q4_K_M, no mmap, CPU model buffer 3163.73 -> 2947.73 MiB.
* memory: refuse rollbacks past the snapshots of the last ubatch
The recurrent memory and the DSV4 cache accepted any rollback up to
n_rs_seq, whether or not the last ubatch wrote that snapshot. Each now
records the deepest rollback its last ubatch guarantees and seq_rm
refuses anything past it:
- recurrent: n - 1 for a ubatch of n <= n_rs_seq + 1 tokens, 0 for a
longer one, which only guarantees the final state
- DSV4: min(n, n_rs_seq), the snapshots are rebuilt from the state
before the ubatch
The dead rollback block of the rollback test becomes the assertion it
asked for, and rollback_bound checks the exact bound of each memory
after ubatches of 1, 2, n_rs_seq + 1 and n_rs_seq + 2 tokens.
* tests: address review from ggerganov
Detect architectures with llama_arch_from_string in comp_padding and
arch_reserves_single_seq, as in rollback_bound.
gfx117X is more like RDNA4, with 128b WMMA. But they're not identical. Adjust the defines accordingly to have a new RDNA4_M and use it where it makes sense.
Assisted-by: Codex
Assisted-by: GitHub Copilot
Co-authored-by: Mario Limonciello <mario.limonciello@amd.com>
* WIP OCP FP8 E4M3 support
* Fix python type-checker
* Materialize input_scales also for FP8 when converting
* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4
1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
weights are reshaped
* Allow remaining FP8 tensors to be dequantized during conversion
* Allow tensor-strategy for FP8-only compressed-tensor ckpts
* Fix and test gguf-py behavior for FP8_E4M3
* Disallow QKV-fusion for FP8 temporarily
* Fix conversion of fused qkvz & FP8 for Qwen3-Next
* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws
* Emit warnings when not fusing FP8 during conversion
* More FP8 conversion fixes
* Add supported for strided copies to/from FP8 in CPU
* remove unneeded scratch buffer from FP8 cpu cpy/dup path
* add explanatory comments and TODOs
* Add dedicated encoder and decoder tests for FP8
* Whitelist ggml_op_get_rows in metal instead of blacklisting
required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list
https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512
* Fix more conversion bugs
* Defer weight materialization during conversion
* Blacklist fp8 in metal backend instead of white-list
* Adjust indents
* Fix lint + execute tests
* Add fast CPU kernel for F8_E4M3
* Conversion fixes
* Fix remaining gaps
* Enable tied weights with shared scales
* Make mandatory-F32 rules apply to FP8 during conversion
* Fix python type checks
https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512
* Fix kaleidai unit-test failures
https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150
* Relax numeric precision constraints for pre-check
* Leave fast-FP8 CPU on RISC-V for follow-up
We already exclude arm from the fast-path
* Allow output scale tensors to be located in any gguf shard
* Move test_conversion_fp8.py test execution to behavior-check.yaml
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it
* Install all requirements also on Windows for behavior-check
Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512
* Unmap lazy tensors before clean-up in Windows
Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512
* LF line endings
https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512
* Extract FP8 tests from llama-arch to its own test-fixture
See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221
* Update gguf-py/gguf/scripts/gguf_convert_endian.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ui: land at the bottom of a conversation every time it opens
Opening a long conversation often stopped short of its end:
- the press grace counted from page load (pressedAt = 0), so a chat that
landed within 400 ms of a refresh did not follow the rows realizing
after the jump
- the click that opened a chat (sidebar row, tab) started the same grace,
so navigating by click had the same problem
- any scroll event away from the bottom stopped following, including the
ones the browser fires itself while the chat settles
- the chat screen outlives its routes, so going back to the last landed
conversation through the new-chat screen skipped the landing entirely
scrollToBottom() now clears the grace, only an upward scroll away from the
bottom stops following, and every navigation lands anew. Navigations into
a chat also disable SvelteKit's scroll reset, which moved the previous
conversation to its top for a few frames.
Where overflow-anchor is supported, a 1px pin after the messages is the
only scroll anchor while following, so the browser keeps the bottom in
place itself; WebKit keeps the resize observer.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let nested agentic content scrollers chain to the chat
Reasoning, shell output and code blocks used overscroll-behavior:
contain, so the wheel stopped dead at their edges instead of
scrolling the conversation.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: stop following the stream as soon as the reader scrolls up
A trackpad or smooth wheel scrolls in steps smaller than the at-bottom
threshold, so each step up was re-stuck and snapped back on the next
growth while a response streamed. Any upward move that is not content
shrinking now releases at once.
The attach effect also read the hook's own state, so every flip
re-attached and re-measured, which undid such a release.
Assisted-by: Claude (Opus 5.5 Medium)
ChatFormActions gated the mic and stop buttons on a `canSubmit` prop that
ChatForm never passed, so it was always false and typing text did not swap
them for the submit button. Use `canSend` instead and also hide the
skip-reasoning button while there is content to send.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: reveal the collapsed sidebar from its logo and toggle it there
The collapsed sidebar now shows only the logo; hovering it reveals the
icon strip until the pointer leaves. The expanded sidebar drops its
top-right collapse button and the logo collapses it on hover instead.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fade the peeked sidebar in at full width and keep the toggle slide
Hovering the logo of the collapsed sidebar now fades in the fully open
sidebar, floating over the page at its full width, instead of resizing
it. Opening or closing with the toggle still resizes and slides: the
sidebar docks to strip width while it slides, then floats out of sight,
and it stays hidden under the pointer after a toggle close. The unused
icon-only strip is removed from the sidebar actions.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: stack the sidebar above the chat tabs on desktop
The peeked sidebar floats over the page, but the chat tab bar shared its
z-index and came later in the page, so the tabs were drawn on top of it.
Assisted-by: Claude (Opus 5.5 Medium)
MCP server groups in the settings Tools tab now get a group checkbox like the
built-in, server and browser groups, toggling the server-scoped tools key.
Indeterminate checkboxes now use the checked state colors.
Assisted-by: Claude (Opus 5.5 Medium)
Ask the browser to keep IndexedDB and localStorage data under storage
pressure, so conversations and settings are less likely to be evicted.
Assisted-by: Claude (Opus 5.5 Medium)
* gguf-py : add Limite arch
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* convert : add Limite support
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* llama : add Limite support
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
---------
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* metal : faster MXFP4 mat-vec kernel
Decode the fp4 values with the fp16 bit trick instead of a threadgroup lookup table, and load the packed nibbles as aligned 32-bit words. The kernel no longer uses threadgroup memory.
* metal : launch contiguous unary and binary ops with full threadgroups
The contiguous unary path and the row-broadcast binary path used one thread per threadgroup. Use up to 256 threads per threadgroup and bound-check the tail.
* metal : split GLU rows across threadgroups
A GLU row ran on one threadgroup, so single-row decode used a single threadgroup for the whole op. Split each row over threadgroups of up to 256 threads.
Requested in the #31125 review. The cases duplicate coverage that is
already in the file and add CI time. Test-only, nothing changes in the
kernel, the docs or the CMake option.
Assisted-by: Claude
* hex-upscale: basic upscale ops (not fully vectorized)
* hex-upscale: optimize nearest 2x, 4x and bilinear 4x
* hex-upscale: add support for non-contig rows and dims collapsing
* hex-upscale: use vgather instead of scalar staging
* hex-upscale: move all kernel select logic to the host, review comments and cleanups
* tests: add permuted UPSCALE test to cover qwen3-vl image->text scenarios
* sycl : add DPAS flash attention kernel for quantized KV cache
Decode and small batches (1 to 31 query tokens) now run flash attention
straight on q4_0 and q8_0 KV blocks, instead of scalar dequant or
converting the whole cache to f16 per call. Q.K uses s8 x s4 and s8 x s8
DPAS on XMX, V.P stays f16. dpas.s4.s8 is inline vISA since joint_matrix
has no s4.
Covers head sizes 64 to 512, GQA up to 32 and attention sinks. Prompts
stay on the existing kernels. GGML_SYCL_FA_DPAS=0 turns it off.
B580, Llama 3.1 8B Q4_K_M, q4_0 KV, tg128: 31.3 to 54.2 t/s at 16K
context. FLASH_ATTN_EXT 4582/4584, same 2 failures as master.
Assisted-by: Claude
* sycl : 192 head dropped, Wildcat Lake added to docs
Assisted-by: Claude
* sycl : stricter checks from review
Assisted-by: Claude
* [SYCL] : rename XMX macro, add GGML_SYCL_SUPPORT_XMX option, document DPAS
Assisted-by: Claude
* sycl : widen reordered q4_0 dequantization stores
The reordered-layout Q4_0 conversion kernel processed one qs byte per
work-item with two scalar stores, strided 16 elements apart. Give it
the same treatment the Q8_0 kernel got in #30775: a flat
sycl::range<1> launch with four qs bytes (8 outputs) per work-item
inside one 32-element block and two vec<dst_t,4> stores.
Same float math and rounding as the scalar kernel; output is
bit-identical. The existing k % QK4_0 == 0 assert guarantees whole
32-element blocks, so the k/8 launch covers every row exactly. The
now-unreferenced scalar reorder kernel is removed.
Arc Pro B70, primed prefill (prefill after decode on a
reorder-installed tensor): +6.4% to +7.3% pp512/pp2048 on Q4_0
models (tested up to 27B dense); fresh-layout prefill and decode
unchanged; PPL within run noise.
* sycl : vectorize reordered q4_K dequantization stores
The reordered-layout Q4_K conversion kernel already loads its
quantized inputs through sycl::vec but writes the dequantized
results with scalar stores. Build the results in a vec<dst_t,n>
register and emit two vector stores instead. Same per-element
math; output is bit-identical.
Arc Pro B70, primed prefill (prefill after decode on a
reorder-installed tensor): +14.7% to +15.3% pp512/pp2048 on
Q4_K-dominant models (tested up to 27B dense); fresh-layout
prefill, decode and PPL unchanged.
* sycl : remap reordered q5_K dequantization to vec4 loads/stores
The reordered-layout Q5_K conversion kernel gave each thread only
four outputs fed by scalar ql/qh byte loads plus a local-memory
scale broadcast, so the stores were never its bottleneck: widening
the stores alone measured within noise on mixed-quant models and
+0.5% on a q5_K-dominant model. Re-decompose to 32 threads per
block, each consuming four consecutive ql and qh bytes and emitting
two aligned vec<dst_t,4> stores; a 4-element run shares one scale
pair, so the per-element math and the output are identical.
Arc Pro B70, primed prefill on a q5_K-dominant 27B model: +3.6%
pp512/pp2048 over the scalar kernel; fresh-layout prefill, decode
and PPL unchanged. The GET_ROWS reorder path moves to the same
32-thread decomposition.
* sycl : remap reordered q6_K dequantization to vec4 stores
The reordered-layout Q6_K conversion kernel wrote four scalars per
thread, strided 32 elements apart, each pair of neighbors sharing
neither instruction stream nor store transaction. Re-decompose the
same 64-thread work-group so each thread emits one aligned
vec<dst_t,4>: a 4-element run shares one 16-element scale group,
one ql nibble source and one qh bit pair, so the per-element math
and the output are identical to the scalar kernel.
Arc Pro B70, primed prefill: +28.7% pp512/pp2048 on a Q6_K 27B
model; fresh-layout prefill, decode and PPL unchanged.
* sycl : vectorize reordered q3_K dequantization stores
Each thread of the reordered-layout Q3_K conversion kernel already
computes four consecutive outputs; build them in a vec<dst_t,4>
register and emit one vector store instead of four scalars. Same
per-element math; output is bit-identical.
Arc Pro B70, primed prefill: +16.8% pp512/pp2048 on a Q3_K-dominant
27B model; fresh-layout prefill, decode and PPL unchanged.
* sycl : vectorize reordered iq3_xxs/iq3_s dequantization stores
Both reordered-layout IQ3 conversion kernels write eight contiguous
outputs per thread as scalar stores (four from each grid lookup).
Build two vec<dst_t,4> registers and emit two vector stores. Same
per-element math; output is bit-identical.
Arc Pro B70, primed prefill: +20.3% to +21.1% pp512/pp2048 on
IQ3_XXS- and IQ3_S-dominant 27B models; fresh-layout prefill,
decode and PPL unchanged.
* sycl : remap reordered q2_K dequantization to vec4 stores
Re-decompose the reordered-layout Q2_K conversion kernel the same
way as Q6_K: one aligned vec<dst_t,4> per thread instead of four
scalars strided 32 apart. A 4-element run shares one scale byte and
one 2-bit lane, so the per-element math and the output are
identical to the scalar kernel.
Arc Pro B70, primed prefill: +13.1% pp512/pp2048 on a Q2_K-dominant
27B model; fresh-layout prefill, decode and PPL unchanged.
* sycl : widen reordered mxfp4 dequantization stores
Give the reordered-layout MXFP4 conversion kernel the same flat
sycl::range<1> launch as Q4_0: four qs bytes (8 outputs) per
work-item inside one 32-element block and two vec<dst_t,4> stores.
Same LUT lookups and per-block scale; output is bit-identical. The
k % QK_MXFP4 == 0 assert guarantees whole blocks, so the k/8 launch
covers every row exactly. The old scalar kernel is removed.
The reordered layout for MXFP4 is installed on the MoE expert path
(reorder_qw_mxfp4_moe); dense MXFP4 tensors keep the normal layout.
Arc Pro B70, MXFP4 MoE 20B model: +57% pp2048 / +61% pp512 prefill
(primed and fresh alike), decode unchanged, PPL within the model's
run-to-run noise.
SCALE now accepts a src0 view with arbitrary row strides. translate_scale
used the raw input, and on the dynamic path a VIEW translates to its whole
source tensor, so a strided view input scaled the entire source. The model
output then had the source's size, and copying it back overran the result
buffer, which crashes test-backend-ops on the SCALE view cases.
Slice the input with process_view_input_new, as the other translators do.
Reject an in-place SCALE on a non-contiguous view: its result is copied
back densely and would land in the wrong elements.
* hexagon: IM2COL over multiple threads with output-col blocks
Assisted-by: Claude Opus 5.5
* hex-im2col: fix padding handling
* hex-im2col: loop unroll common cases
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
test_cancel_request enables context shift so the generation only ends
when the request is cancelled, then polls /slots with a deadline.
Without context shift the generation fills the context and frees the
slot on its own on fast machines, so the test passes even with
cancellation disabled.
test_server_sleep uses the existing wait_for_sleep helper instead of a
fixed sleep.
* common: bound control vector layer indices by the model layer count
A direction tensor layer index is parsed from the tensor name with no
upper bound, and the buffer size and write offset are computed in int.
A large index overflows both, so the accumulation writes before the
start of the buffer.
Pass the model layer count to the loader, reject indices above it like
the existing zero index check, and compute the sizes in size_t.
* mtmd: fill M-RoPE positions by whole merge tiles
The position loops of the Qwen-VL, GLM, PaddleOCR and Qwen2.5-VL
encoders hardcode a 2x2 merge inside a stride of n_merge, so a patch
grid that is not a multiple of the merge writes past the positions
buffer. Iterate over whole merge tiles with n_merge inner steps, which
bounds the writes by the patch count and gives the same positions on
aligned grids.
PaddleOCR also requires a merge aligned image at graph build, as the
Qwen-VL family already does.
* opencl: run the tiled q4_K lm_head through a tiled GEMM at batch > 1
* opencl: split tiled dp4a q4_k gemm into its own file and test for dp4a
Assisted-by: Codex
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hexagon: add conv2d op
* hex-conv2d: add double-buffered staging
* hex-conv2d: full 64-wide activation block transpose straight from staged buffer
* hex-conv2d: implicit gemm
* hex-conv2d: remove scalar path and kernel
* hex-conv2d: unify conv2d kernel params for all paths
* hex-conv2d: scale HMX result before f16 storage and then restore during unpacking to prevent overflow
* hex-conv2d: fix f16 overflow in HMX paths
* hex-conv2d: refactor to rebase to new op fusion machinery
* hex-conv2d: restructure DMA pipelines and trace scopes
* hex-conv2d: removed the HVX_WIDTH and HVX_TILED paths
* hex-conv2d: remove in-loop vector spills
* hex-conv2d: refactor kparams, vtcm-layout and dma-pipeline to match the other ops
* hex-conv2d: add basic support for CONV_2D_DW
Co-Authored-By: Pranav Dhinakar <pdhinaka@qti.qualcomm.com>
* hex-conv_2d_dw: use DMA in all paths
* hex-conv_2d_dw: support for chunking larger inputs
* hex-conv_2d_dw: remove scalar/hvx access for w tensor (replaced with sync dma copy)
* hex-conv-2d: fix warnings
* hex-conv2d-dw: vectorize all paths
* hex-conv-2d-dw: tree-based reductions and unrolling to optimize
* hex-conv-2d-dw: uninline compute loops from dma loops
* hex-conv-2d-dw: move all kernel selection into the host solver and simplify/specialize kernel dispatch
* hex-softmax: update obsolete gates to enable SOFT_MAX for mmproj
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Pranav Dhinakar <pdhinaka@qti.qualcomm.com>
* mxfp4 sve 256 impl completed
* working with pretty good perf improvement
* review comments addressed
* review comments for SVE 256 check
* trailing white spaces removed
---------
Co-authored-by: Anubhav Srivastava <anubhav.srivastava@fujitsu.com>
* llama : fix DSV4 compressor padding for simple batches
Count actual tokens per sequence when padding compressor writes. Add a graph reuse regression test for odd-sized DeepSeek V4 and V4.1 batches.
Assisted-by: Codex
* tests : cover multiple sequences in DSV4 padding regression
Exercise one and two sequences at different positions while keeping a fixed ubatch shape.
Assisted-by: Codex
* llama : remove simple batch splitting for DSV4
Use the existing sequence splitting to keep n_seq_tokens consistent with
the compressor padding calculation. Extend the graph reuse regression to
6-token batches.
Assisted-by: Codex
* codeowners : refresh teams and maintainer PRs link
Update the CODEOWNERS team legend with the current `ggml-org` team
memberships, and extend the maintainer PRs link in the README to cover
all current members of the `maintainers` team.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : list remaining backend sub-teams
Add the `ggml-*` sub-teams that were missing from the team legend at the
top of the file: base, cpu, et, hip, openvino and riscv. The legend now
covers every sub-team of the `ggml` and `llama` maintainer groups.
No ownership rules change - only the comment block is extended.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : use ggml-hip team for HIP backend
Route `/ggml/src/ggml-hip/` to `ggml-org/ggml-hip` instead of a single
collaborator, so review requests reach the whole team.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : add llama-examples team
List the newly created `llama-examples` team in the team legend at the
top of the file.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : use llama-examples team for examples
Collapse the per-directory rules that only listed a single collaborator
into a single `/examples/` rule owned by `ggml-org/llama-examples`.
Directories owned by other collaborators keep dedicated rules, since the
last matching pattern takes precedence in CODEOWNERS.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* cont : update zdnn
* codeowners : use llama-base team for src
Route `/src/` to `ggml-org/llama-base` and list the new team in the
legend at the top of the file. Rules for individual `src` files are kept,
since the last matching pattern takes precedence; `/src/models/` drops
its individual owner, which is already covered by `llama-models`.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* mtmd : add LFM3-VL and LFM3-Omni video input
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* mtmd : reserve the soundtrack command to avoid a GCC 11 warning
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui: build the reasoning menu from the template effort levels
When the chat template reports effort levels, the menu offers Default,
Off and those levels, and a pick is sent as reasoning_effort. A
template that reports none keeps the token budget menu.
Picking a model other than the one that produced the last answer resets
the conversation effort to Default, and an effort the current menu
lacks resolves to Default.
* ui: wait for a cold router model before building the request
A router serves the props of a loaded model only, so a request sent to
a cold model was built without its template effort levels and fell
back to a token budget. The request now waits for the model to load,
joining a load already in flight, and a stop during the wait sends
nothing.
Status waiters hold a list per model so a load and the requests that
wait on it each get theirs, and a props request in flight is shared
with a second caller instead of answering null.
* fix: clamp the A prefetch to end_k in the Vulkan coopmat1 int8 matmul, like the B prefetch
* fix: zero the A scale for k steps past end_k in the Vulkan coopmat1 int8 matmul instead of relying on the clamped block
* chat : strip trailing whitespace from parsed reasoning and content
Assisted-by: Claude Fable 5.1
* chat : stop qwen3-coder reasoning and content before template separators
Assisted-by: Claude
* tests : remove the Qwen parser case from the chat session tests
Assisted-by: Claude
* common : handle partial prefix until match at end of generation
---------
Co-authored-by: Zeshawn Zahid <zzshawnn@gmail.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
* ui : show save and reset settings buttons only when relevant
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move reset settings button next to reload app, pad settings caption
Assisted-by: Claude (Opus 5.5 Medium)
* ui: support OGG audio attachments by converting them to WAV
Fixes#24164
Assisted-by: Claude (Opus 5.5 Medium)
* ui: convert all non WAV/MP3 audio uploads to WAV
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep new chat draft when adding a system prompt
Fixes#24338
Assisted-by: Claude (Opus 5.5 Medium)
* ui: remove unused initialMessage from chat form
Assisted-by: Claude (Opus 5.5 Medium)
* ui : replace the sidebar search mode with a search dialog
Search opens a spotlight dialog (Cmd+K or the Search entry under New chat in the sidebar) with keyboard navigation over recent chats and sidebar actions. Phones keep the search page.
Sidebar actions show their shortcuts, with new ones for Settings (Cmd+,) and Models (Shift+Cmd+M).
Assisted-by: Claude (Opus 5.5 Medium)
* ui : ignore search dialog keys while an IME is composing
Enter confirms an IME candidate and the arrow keys move through the candidates, so the search field leaves them to the IME instead of opening the highlighted result or moving the highlight.
Assisted-by: Claude (Opus 5.5 Medium)
* server : add realtime conversation and transcription on /v1/realtime
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* examples : default the realtime client to the server port 9931
Assisted-by: Claude Opus 5.5
* server : sample realtime responses right after their own decode
Slots respond in turns on one context: sampling at the start of the next step read the logits of whichever slot decoded last, so concurrent sessions mixed their outputs.
Assisted-by: Claude Opus 5.5
* move more stuff to server-realtime, making it self-contained
* examples : fix the type check of the realtime client
Assisted-by: Claude Opus 5.5
* examples : use the model card prompts in the realtime client
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B
Assisted-by: Claude Opus 5.5
* convert : keep the speech task templates out of the d1 GGUFs
Assisted-by: Claude Opus 5.5
* small refactoring: no_batching
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: expose the reasoning effort levels of the chat template
The capability probe renders a user turn for each candidate level
(minimal, low, medium, high, xhigh, max) and for an unknown level. A
level is honored when it renders without raising and differently from
the unknown level, which shows the template fallback. Levels sharing a
render are aliases and the highest one is kept.
The caps serialize to JSON, and chat_template_caps in /props reports
the list as reasoning_effort_levels, from lowest to highest.
* jinja: keep the caps free of the json dependency
The caps keep reporting their flags as a map, and the effort levels
stay a plain member. common_chat_templates_get_caps() builds the JSON
the server reports from both, so jinja depends on json only for the
probe inputs.
In --models mode the thread count per model is computed and printed,
but never set in the params, so every context runs on the common
default, the number of physical cores. With several models in
parallel this oversubscribes the CPU, and the busy-wait barriers of
the CPU backend make it collapse.
Set the count in cpuparams and cpuparams_batch.
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* convert : read the LFM3-Audio config.json as released
A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).
Assisted-by: Claude Opus 5.5
* mtmd : disable flash attention for the LFM3-Audio depthformer
Assisted-by: Claude Opus 5.5
* tts : add a voice option for LFM3-Audio and LFM3-Omni
The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.
Assisted-by: Claude Opus 5.5
* clean up template handling
* convert : detect LFM3-Audio and LFM3-Omni by model_type
Assisted-by: Claude Opus 5.5
* move banned tokens --> logits bias --> mtmd_helper
* leave mtmd-cli unchanged
* no more template_asr_system
* left over
* llama : keep the LFM3-Audio encoder convs and codec unquantized
Assisted-by: Claude Opus 5.5
* common, mtmd : shorten comments
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* common : support deferred tools and tool_reference content parts
Assisted-by: Claude Opus 5.5
* common : credit the deferred tools co-author
Co-authored-by: Aldehir Rojas <765227+aldehir@users.noreply.github.com>
Assisted-by: Claude Opus 5.5
* tests : fix the type check of the deferred tools test
Assisted-by: Claude Opus 5.5
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* tests : set the SWA rope base of the LFM3 test model
Assisted-by: Claude Opus 5.5
* llama : enable recurrent state rollback for LFM3
Assisted-by: Claude Opus 5.5
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>
* gguf-py : align the lfm3vl mapping comments
Assisted-by: Claude Opus 5.5
* llama : fix CRLF line endings from the review suggestions
Assisted-by: Claude Opus 5.5
* tests : add the LFM3 Metal fusion baseline
Assisted-by: Claude Opus 5.5
* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* llama : build the k-pool sparse selection independently of the kq_mask shape
qwen4exp (QSA) and glm5-next pick cache cells by index and no longer read the kq_mask values, but they still shaped the selection like the dense kq_mask. Build the selection as [n_kv, n_tokens] over the cache cells instead, and split it into the streams of the K view right before the attention.
This lets the dense mask change its layout (e.g. per-sequence row lists) without touching the sparse layers.
* improve comment
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : drop the unused build_fused_top_k_mask args, review nits
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* tests : speed up tensor data generation and model saving
Refactor save_models to create only the model, not a llama_context,
which is not needed for saving. Replace the slow std::normal_distribution
draw with a splitmix64 + box-muller normal generator.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : use a shared pool of normal draws for tensor data
Generate a pool of normal draws once with std::mt19937 and
std::normal_distribution, then have every tensor copy a cyclic slice of
it instead of running a per-tensor box-muller. The name-keyed slice
selection keeps the draw deterministic.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : log the size of saved models
Report the produced file size in bytes and MB after each save.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : simplify the model save log
Put the model name, target and size on one line per saved model.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* vulkan: allow dim 3 broadcast in MUL_MAT
supports_op rejected every MUL_MAT whose src0 and src1 differ in ne3,
although mul_mm, mul_mmq and mul_mat_vec already broadcast src0 over
dim 3 through broadcast3. Such a matmul then ran on the CPU with a
copy of the weight, and the split layout changed with the number of
streams in the ubatch.
Keep the strict check for MUL_MAT_ID only, and route the broadcast
cases away from mul_mat_vec_nc, the one path that requires equal ne3.
* tests : harden MUL_MAT with permuted dim 3 broadcast cases
Harden the MUL_MAT coverage with a weight broadcast over dim 3 against
a permuted per-stream activation, the shape of an MLA output
projection with a non unified KV cache, for a single token, several
heads and several tokens per stream.
The expand button wrapping the system message card defaulted to
inline-block. WebKit takes an inline-block's baseline from its last
line even when that line is clipped by the card's max-height, so the
row grew to the height of the whole prompt and left a large empty gap
below the card. Make the button a block so the row matches the card.
Assisted-by: Claude (Opus 5.5 Medium)
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold
estimate with to-be-loaded model size included
use no_alloc to get memory requirements for model load
only set model memory_mb if not previously calculated
use memory margin instead of total size limit, apply to each device separately
add server memory debug logging
move llama_context_device_memory function to llama-ext.h
fix model count exceeded check
improve memory_per_device map naming
improve variable naming, fix style
also strip models memory margin from child processes
cont : clean-up
replace device memory map with buft memory map. Use llama_get_memory_breakdown
extract duplicated check into helper function
move model memory estimation to subprocess
precompute name->buft map, map GPU host types to CPU buft
cleanup unused variable
remove duplicated init calls
* adapt to upstream changes
* wip
* common_memory_reports internface
* SERVER_ROLE_MEASURE
* server-models-sched
* fix build
* server : fix memory accounting of measure children and claim spin
A measure child runs no_alloc, so the free memory it reports does not
include its estimate: subtracting self there erased the memory held by
other processes, down to zero with --fit on, and the next load could
overshoot the margin. Subtract self only for instance reports.
A queued load only checked --models-max before claiming, so when the
memory margin blocked it behind a busy model, load() threw and the
waiter retried in a tight loop. Refuse the claim while the limits are
exceeded and no idle model can be unloaded, so the waiter sleeps on
the condition variable until one goes idle.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* model : add MiniCPM-OCR-Jev support
The model is a Qwen3.5-VL trunk with a value head read at the last token: it
scores an OCR text against an image rendered from it, it does not generate.
Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone.
The class scores are the pooled output of the sequence, read with last pooling,
so the conversion turns score.weight into cls.output and sets
embedding_length_out. The checkpoint has no LM head, so text generation is
rejected instead of using the output weights duplicated from the token
embeddings.
The task is fixed, so its prompt is entirely in the systemone template, the
same way h2o-lightning keeps its default there. The processor of the checkpoint
allows up to 16384 merged tokens per image, more than the Qwen-VL default of
mtmd, so the mmproj keeps the image budget of the preprocessor config. The
checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as
other models do.
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* fix code
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* use gguf meta classifier_output_labels
* remove redudant code changes
---------
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* metal : add GGML_OP_TOP_K_MASK
Port the fused top-k mask from the CUDA backend: a grid-stride kernel fills the output with -inf, then one
threadgroup per row copies the mask values of the kept cells over it. Function constants drop the score,
blocks and extra paths that a given graph does not use.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* metal : clean up top-k mask kernel formatting
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : relax scale to non-contiguous views
SCALE requires src0 to be 1D-contiguous. Relax this to dim-0
contiguity with arbitrary row strides (ggml_is_contiguous_rows),
same as the unary ops in the Metal backend.
- ggml: relax the assert in ggml_scale_impl
- cpu: address src0 rows through their byte strides (f32, bf16)
- cuda: stride-aware kernel, fastdiv row decomposition
- metal: support bf16 scale (guarded unary instantiations)
- sycl: keep the gate at contiguous, the kernel is flat f32
Add view variants to test_scale. Verified on CPU, Metal and CUDA.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* cuda : skip the softcap fusion on non-contiguous views
The fused softcap kernel reads src and writes dst as dense arrays,
so a view reaching SCALE -> TANH -> SCALE now that SCALE accepts
non-contiguous rows would be read wrongly. Keep such graphs on the
strided kernels until the fusion becomes stride aware.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* tests : fix multi-seq split replay against trained models
The rollback snapshot is captured by the graph that decoded the rolled-back
tail, while the reference context decodes the same prefix with different
batch shapes. The two states differ in the last bits and trained models
amplify the difference with every replayed token. Compare only the first
replayed token of each seq, then restore the rollback state into the
reference context before the seq-1-only tail.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : require bit-exact seq-1 isolation in split replay
The seq state save is not always bit-lossless - SWA-masked cells are not
saved - so reload the restored state back into the rollback context. That
way both contexts enter the seq-1-only tail with byte-identical memory,
including the cell layout, and the tail logits must match bit-exact. Also
assert that the restored state round-trips bit-identical.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : drop the n_swa partial-state check from ps
The check keys off llama_model_n_swa(), which returns 0 for dsv4 even
though its raw KV cache has a 128-cell sliding window. The partial state
grows with the tokens there by design, so the flatness check is wrong.
Keep the partial <= full invariant.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Every caller passes n_pad = 1 and the only two users clamp it to 256.
Drop the parameter, the members and the trivially true kv_size assert.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The org-less group renders without a header, so when it followed a named
org its models looked like they belonged to that org. Put it ahead of the
named org groups.
Fixes#25227
Assisted-by: Claude (Opus 5.5 Medium)
On desktop, the reasoning effort toggle now gets a muted background on
hover and while its menu is open, matches the model selector's 24px
height, uses a 13px icon and sits 5px from the selector.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add conversation compaction
Add a compaction action that asks the model to summarize the
conversation so far and stores the summary in a new compaction node
appended to the branch. Later requests slice the branch at the last
compaction node, keeping the system prefix and continuing from the
summary instead of the full history, which frees up context while the
complete history stays visible and navigable in the UI.
The slice is applied once at the request funnel, covering the plain
send, the agentic flow and the KV cache pre-encode, plus the continue
path that bypasses it. The summary request is a one-shot outside the
SSE replay machinery, streamed live into the node with a full rollback
on abort, error or empty result.
The action is exposed as a /compact slash command and a Compact button
in the context gauge popup, with an overridable summarization prompt
in the settings.
* ui: pre-fill KV cache and refresh context gauge after compaction
Once the summary is stored, pre-encode the compacted branch and stamp
the returned prompt timings on the compaction node: re-encoding the
reduced prefix is part of the compaction itself, so the server KV
cache and the context gauge reflect the summary right away.
The summary stream is registered under a prefixed identity
(compact::convId), so the visibility kick and byte-offset resume
protect it through tab switches while the reload probe never
reattaches it as a chat turn. The stamped node becomes the shared
context truth for the gauge and the processing state restore, and the
stale live state that kept the gauge full is cleared.
* chore: format
* ui: allow a dedicated model for conversation compaction
An opt-in setting reveals a model selector whose options come from the
live router model list. The chosen model writes the summary and is
loaded on demand when it is not resident, while the default path keeps
resolving to the conversation's own model.
* ui: align the compaction model field with project conventions
* ui: fix the model option helpers import path
The compaction model selector pulls filterModelOptions and
groupModelOptions from the models component folder, where they no
longer live, so the build fails to resolve the module. Point the
import at the navigation folder, which now owns them.
Missed during the rebase: the helpers moved in a commit that does
not touch any compaction file, so nothing conflicted.
* ui: import the model option helpers from models/utils
* ui: follow the moved models selector in the compaction model field
The models manager moved ModelsSelectorList under ModelsSelector and
the model option helpers to /utils, where groupModelOptions takes
the options alone. Drop the stale copy the rebase kept and read the
helpers from their new home.
* ui: drop the dedicated compaction model
A summary written by another model loses the prefix the conversation
model cached, so the whole history is prefilled again, and the model
then resumes from text it did not write. The conversation model
always writes its own summary.
* ui: keep the last steps of a compacted conversation
A compaction now folds the steps the model reads but the last ones,
a step being a user message or an assistant message with its tool
results, and stores how many it kept on the compaction node. The
model then reads the system prefix, the summary, the kept steps and
what follows.
The summary request is the request the chat sends, with its options,
tools and message shaping, cut before the preserved steps: the server
reuses the prefix it cached and only the instruction is new. The
number of preserved steps is a setting, 10 by default.
* ui: run every compaction summary through one path
Three callers needed the same thing - the automatic pre-encode pass, the
action in the context gauge, and re-running a summary from the block - and
each of them had grown its own copy of the request, the streaming and the
storing.
There is one path now: a summarization owns a node, streams into it, and
reports itself through a ledger the rest of the app reads (`isCompacting`,
which node belongs to which conversation). Re-running a summary is the same
call on an existing node, so the two entry points cannot drift apart again.
* ui: give the compaction row its own summary UI
The summary was a line of text in a tool-call-shaped collapsible, which is
how a compaction reads as a side note rather than as the thing the rest of
the conversation now stands on.
The row is its own block: a divider that names the boundary, and the summary
inside it with the model that wrote it, its statistics, and the same actions
an assistant turn has - copy, edit, fork, regenerate and delete all apply to
the summary, and it collapses to the divider once a summary is in, because
the shrunk context is the headline and the prose is the detail.
The divider is a variant of the shared collapsible block, which also needed
the toggle and the title split into snippets so a caller can lay them out
without copying the markup. The action icons no longer take a role: they
render what a message offers, and a compaction offers the same as any other
stored message.
* ui: report compaction progress on the row
The row owns the run, so the row says how it is going: the prompt progress
in the header while the context is being read, the phase once the summary
starts streaming, and the token count once it is in. Before the first
summary token there is nothing to collapse, so the header stands on its own
instead of offering an empty block.
* ui: keep the summarized turns quiet while a compaction runs
The turns being summarized used to keep their own "Processing..." indicator
and live reading stats while the summary streamed, so the conversation said
the same thing twice in two places. The row reports the run, so the turns it
replaces stay silent until it is done.
* ui: hand the compact action to the context gauge
Compacting is a decision about the context, so the action belongs with the
numbers that show the context filling up rather than inside the form: it sits
at the bottom of both the hover card and the drawer, under the breakdown it
acts on, quiet enough to stay a footnote and disabled - with the row's own
spinner - while a summary is already running.
* ui: follow the bottom, otherwise leave the view alone
The screen scrolled itself with a controller of observers, intervals,
direction heuristics and landing timers, and still managed to move the view
while the reader was reading: any mutation that grew the branch was treated
as a reason to keep following.
The page is the scroller, so it follows when the reader sits at the bottom,
and stays where it is otherwise. A resize observer plus native scroll
anchoring does the rest, and `pointerdown` buys the reader a moment of grace
to be caught mid-drag.
* ui: expand collapsible bodies with a height transition
Expanding a block snapped it open: the body was mounted the moment `open`
flipped, so a tall block moved the conversation around it in one frame.
The body now goes through the same region the models table uses. The height
interpolates and the visibility follows it, so a body is unmounted once the
collapse has finished rather than the moment it starts - a closed block
still costs nothing while tokens stream past it.
* ui: open a compaction summary at the top of the screen
Toggling a summary left the reader wherever they were, so a conversation that
had just compacted showed the divider at the bottom of the screen and read
down from there.
A toggle now scrolls the page once and lands the divider a breath under the
tabs, both ways: the summary is read from its beginning, and closing it returns
to the same boundary instead of wherever the page happens to come to rest. The
scroll waits for the body to reach its height, because before that the page is
too short to scroll the divider up there and the call would clamp.
That wait is the shared collapse duration rather than a transition event: an
engine without `interpolate-size` animates the region's grid row instead of its
height, so a listener on `height` would never fire there and the page would not
move at all.
* ui: spin the compaction icon while the summary runs
The divider reported the phase in its title while the icon still showed a
finished block, so a run in progress and a summary that had landed looked the
same in the one place the eye reads first. The icon spins while the summary is
running and folds back once it is in.
* ui: let the compaction summary speak without the reasoning trimmings
The summary is the thing the conversation now stands on, not a note beside
it, and the conversation model always writes it, so the body drops the
reasoning block's rule and indent (`flushBody` keeps them for the blocks
that borrow them) and the model selector under the summary is gone - there
is nothing left to swap.
* ui: drop the model override from the compaction path
The summary row no longer offers a model selector, so nothing hands
regenerateCompaction a model anymore. Remove the parameter from the
whole path, with the on-demand load of another model it carried: the
summary always comes from the model the request funnel resolves.
* ui: let the compaction summary think as the chat does
The summary request forced thinking off, overriding the reasoning
effort the user chose. The summary is the text the conversation
resumes from, the task where reasoning helps the most, so the request
now carries the very options of the chat.
* ui: pre-encode the compacted branch as the next turn reads it
The post-compaction pre-encode sent the branch without tools, while
the tools render ahead of the messages, so it warmed a prefix the next
turn never reads and that turn prefilled the whole compacted context
again. The pre-encode now sends the tools, message shaping and
reasoning handling of the chat, so the next turn only prefills its new
message.
* ui: pre-encode the branch the next turn actually reads
The per-turn pre-encode sent the messages of the request plus one
assistant message holding the streamed text, without tools. With
tools on, it replaced the slot the next turn extends with a prefix
nobody reads again, and after an agentic turn its history did not
match the stored calls and results. Without agentic, it never ran.
Both pre-encodes now read the stored branch up to its leaf, cut at the
last compaction, and send it with the tools, message shaping and
reasoning handling of the chat. The plain completion path triggers it
too.
* ui: refuse a compaction summary the output limit cut
The stream handler read every chunk but dropped the finish reason, so
a summary cut by max_tokens folded the conversation into half a text.
The handlers now hand the finish reason to onComplete, and a summary
that ends on length rolls the compaction back with an error.
The summary was framed through a replacement string, so $& or 32560 in
it changed the text the model read; it is framed as written now.
* ui: wait for the collapse transition in the tool call tests
The collapsible body now unmounts once its collapse animation ends rather
than as soon as the block closes, so the "folds back" assertions ran before
the body was gone. Wait out COLLAPSIBLE_TRANSITION_MS first.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* ui: open model details in a global drawer without leaving the page
The model info actions in the selector used to navigate to the models page
to open its details pane, and a phone opened a separate dialog instead. The
drawer now lives outside the models table, driven by uiStore, so the info
icon opens it over whatever page is showing.
The models dropdown and the phone picker stay open under the drawer's
backdrop, and keep focus while the drawer opens and closes, so the trigger's
tooltip no longer pops over the drawer.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let the model details drawer reopen after a drag to dismiss
Dragging the drawer down wrote vaul's own open state, which the one-way open prop then kept closed, so the info action could not open it again until a refresh. Bind open both ways through the store instead.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: close the models selector from the details drawer chat actions
Assisted-by: Claude (Opus 5.5 Medium)
Extract the keyboard-shortcut delete confirmation into DialogConversationDelete
and reuse it in the sidebar and mobile search page instead of the native
window.confirm prompt.
Assisted-by: Claude (Opus 5.5 Medium)
* hexagon: add GGML_HEXAGON_VTCM_MB and GGML_HEXAGON_NTHREADS caps
Change-Id: Idffe1613c8db6f96aa842d43d1b2efaed45462a8
* hexagon: rename VTCM cap to GGML_HEXAGON_VTCM_SIZE, drop NTHREADS cap
Change-Id: If4c5b9ce9fd6b725c3561421c2295d40599d012d
* hexagon: make GGML_HEXAGON_VTCM_SIZE an override, accept K/M/G suffixes
Change-Id: Ib8eea84c22f24e6d986b37f215e504b046478c2c
The hyper-connection and DSA graph builders wrote their extra outputs
(post / comb gates, carried top-k selections, inject gates) through
ggml_tensor** out-params. Return them in a small struct next to the
primary output instead.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Every dst row comes from src0 or src1, or for dim 0 from the row of
src0 followed by the row of src1, so a single kernel covers every
type: a dense source row is one memcpy, quantized types included,
and a strided source is copied element by element with the element
size known at compile time.
Rows are split across threads instead of i2 alone, which is often 1,
and are copied in tiles of 32: for a transposed source the elements
are walked in the outer loop, so the rows of a tile read neighboring
addresses instead of jumping a full row stride per element.
Add transposed b cases to test_concat, the layout of the delta-net
conv input, and a perf case for it.
* sycl: add set_tensor_2d for faster tensor-split loading
Pack host rows into chunks of up to 64 MiB when destination rows are
contiguous, avoiding a separate upload call for each row. Keep the
existing per-row copy path for non-contiguous destinations.
On two Arc Pro B70s with gpt-oss-120b UD-Q4_K_XL, total llama-perplexity
runtime drops from 263-268 s to 114-115 s. A full-load readback check
verified 26,784,256 rows with no mismatches.
* sycl: batch 2D tensor uploads through shared staging slots
Implement set_tensor_2d to speed up tensor-split model loading by
packing rows into the existing pinned staging slots and uploading
each batch with a single device copy.
Share staging initialization with set_tensor and wait only when
reusing a slot, allowing CPU packing and device copies to overlap.
Fall back to per-row uploads for noncontiguous device rows,
oversized rows, or unavailable staging.
---------
Co-authored-by: Jon Staples <250234834+Captain-Tripps@users.noreply.github.com>
* tests: pin model resolution on real repo listings
Extend test-model-resolution with 63 repo listings captured verbatim
from the HF API, chosen to cover every file naming pattern found across
1237 GGUF repos of the major publishers and pruned to whole quant groups.
Each repo pins its resolved plans and its cache listing.
The expectations follow what each file really is per its GGUF header,
not what the current resolver picks, so a mismatch is a resolution bug.
Mismatches are collected and reported together before the test fails.
* tests: leave the listing unchecked on paths the filesystem refuses
Some real split names (DevQuasar's Model.Q2_K.gguf-00001-of-00012.gguf
under a quant folder) exceed MAX_PATH once materialized in the temporary
cache on Windows, so the listing of such a repo is skipped with a note
while its plans stay checked.
* tests: pin real listings to master on GGUF convention names only
Replace the real repo table with 171 HF listings whose file names follow
the GGUF naming convention (lowercase sidecar prefix, - delimited
components, quant last), one per layout pattern found across 1174 GGUF
repos and pruned to whole quant groups. Repos outside the convention are
dropped. Each repo pins the plans and the cache listing master gives
today, so the test is a pure harness: any change in resolution shows up
here, and mismatches are reported together.
test-llama-archs included src/llama-arch.h and src/llama-model-saver.h,
which prevented building on Windows with shared libraries since the
used symbols are not exported from the DLL.
- declare llama_arch_* wrappers around the internal llm_ arch query
functions in the llama-ext.h staging header, together with
llama_arch_kv_name() and llama_arch_supports_save()
- slim llama_model_saver down to the private implementation of
llama_model_save_to_file(): the fixture-writer half used by the test
moves into tests/test-llama-archs.cpp as a local kv_writer
- replace the roundtrip saver usage with the public
llama_model_save_to_file() writing to a temp path (also removes the
tmpfile() fd leak and its Windows workaround)
- move test-llama-archs, test-generate-models, test-llama-context and
test-fusion out of the NOT WIN32 OR NOT BUILD_SHARED_LIBS guard
Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
* Metal : stop overflow from happening in large matrix output offsets
Assisted-by: Codex
* metal : widen batch offsets and MUL_MAT_ID token indices
Assisted-by: Codex
---------
Co-authored-by: feni6 <270037094+feni6@users.noreply.github.com>
* ui : land opened conversations at their bottom before the first paint
Pin the page to its bottom with a ResizeObserver until it stops resizing or the reader interacts, and mount the last rows eagerly so the landing frame shows real content.
Assisted by: Claude (Opus 5.5 Medium)
* ui : make conversation tabs browser-style
A plain navigation shows the chat in the current tab instead of adding one, and the new-chat tab turns into the conversation it starts. New tabs come only from explicit actions: the + button, Cmd/Ctrl-click or middle-click on a sidebar chat, and its Open in new tab menu item, which open in the background. Closing a tab moves to its neighbour among the tabs the view shows.
Assisted by: Claude (Opus 5.5 Medium)
* ui : share the tab surface and tab bar fade styles
Move the tab item surface (shape, resting and active looks) to TAB_SURFACE_* class constants and the fade under the tab bar to a global chat-top-fade class, so other elements can match the tabs.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fit the chat column below the tab bar
The tab bar reported 1.25rem of its height while it takes 3.3125rem (pt-3.25 + h-10), so a short conversation still left a few pixels to scroll. Use the bar's full height and drop the extra 1rem from the column's minimum height, so it fills the viewport exactly; the centered new-chat form moves up by the same 1rem to keep its place.
Assisted by: Claude (Opus 5.5 Medium)
* ui : stop the page from bouncing past its ends
Elastic overscroll on the root scroller dragged the sticky and fixed bars (tab bar, chat form) along at the top and bottom of a chat, and they jumped back on release. Set overscroll-behavior: none on html and body.
Assisted by: Claude (Opus 5.5 Medium)
* ui : stop scrolled dialogs from bouncing past their ends
Dialogs scroll on their own (the settings panel, the search results), so the page rule did not reach them and their sticky navigation still jumped on overscroll. Turn off overscroll on dialog panels and on the scroll containers inside them. Only scroll containers: Firefox counts an overflow-hidden box such as truncated text as one, and `none` there kept the wheel from scrolling the list around it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fit a first exchange opened by the working directory row
The last answer's minimum height leaves room for the last question only, so the synthetic row that a new chat starts with (Set working directory to ...) made a short first exchange overflow by that row. When everything above the last question is synthetic, count those rows as part of it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : keep following a streaming response until the reader scrolls up
Auto-scroll stopped whenever the page moved up, including moves the
reader did not make: scroll anchoring and blocks re-rendering shorter
while a response streams. Only the reader's wheel, touch, scroll keys
or scrollbar drag now count as scrolling up.
The send scroll also looked up the new user bubble by position, which
picked an older message when the answer row had not rendered yet.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fade the chat form between its centered and bottom positions
The form no longer slides across the column between the new-chat center
and the bottom. It fades out drifting toward where it goes and fades
back in where it lands, quicker when opening a chat than when going back
to a new one. Its wrapper keeps the same height in both positions, so
the messages never shift while it moves.
The new-chat greeting fades in as the form settles, also on a refresh or
a first visit. Reduced motion keeps only the fades.
Assisted by: Claude (Opus 5.5 Medium)
* ui : let the expanded sidebar be resized by dragging its edge
Bring back the desktop sidebar resize handle that was dropped with the
shadcn sidebar primitive. The width is clamped between 18rem and 32rem,
persists across reloads, and resets on a double-click of the handle. The
tab bar follows the same width so a wider sidebar never covers it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : rubber-band the sidebar edge past its bounds and drag it shut
Past the max width the edge resists harder the further it is pulled, and
the same happens below the min width until the pull passes 9rem, where it
snaps to the pointer and letting go hides the sidebar. A release outside
the bounds springs the width back to the nearest one.
Assisted by: Claude (Opus 5.5 Medium)
* ui : show the sidebar edge strip faintly on hover and solid while dragging
The strip is clipped to the sidebar's rounded corners through its own
overlay, so the sidebar itself is never clipped.
Assisted by: Claude (Opus 5.5 Medium)
* ui : remember whether the desktop sidebar is open across reloads
A phone opens the sidebar as an overlay, so it always starts closed there
and its toggling never overwrites the desktop preference.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fade the sidebar labels and conversation list with its width
The fade follows the live width, so it tracks both the toggle's width
transition and a drag below the minimum width.
Assisted by: Claude (Opus 5.5 Medium)
* ui : persist the sidebar width on release and clamp a restored one
A drag now updates only the live width and writes it to localStorage once
when released. A width restored from localStorage is clamped to the
current bounds, so a value saved under older bounds does not stick.
Assisted-by: Claude (Opus 5.5 Medium)
* kv-cache : use ggml_backend_tensor_copy to refill the DSV4 K mirrors
* llama : move the Engram pad id to llama_hparams and the token map to llama_model
The saver no longer needs the deepseek41 model class.
* deepseek41 : drop the F32 src1 request on the HC mixes matmul
* ggml : add softplus TODOs in the backends that still use log(1 + exp(x))
* convert : make the DSV4/DSV4.1 modify_tensors generators, drop the redundant vision filter
* deepseek41 : clean up comments
* deepseek41 : drop the F32 src1 requests on the router and indexer matmuls
* convert : use format_tensor_name directly in DeepseekV4Model
* s/comp_kv_src/comp_il_src/g
* cont : fix TODOs for log1p
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cuda : use std::string instead of VLA for archName
Modernize the code (old C style) and fix the following warnings:
```
ggml/src/ggml-cuda/ggml-cuda.cu:180:19: warning: variable length arrays in C++ are a Clang extension [-Wvla-cxx-extension]
180 | char archName[archLen + 1];
| ^~~~~~~~~~~
ggml/src/ggml-cuda/ggml-cuda.cu:180:19: note: read of non-const variable 'archLen' is not allowed in a constant expression
ggml/src/ggml-cuda/ggml-cuda.cu:179:9: note: declared here
179 | int archLen = strlen(devName);
| ^
```
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Add a comment about archName
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
* common, grammar: add recursive depth limit, make it a requirement
* catch more recursive functions
* common : tidy up peg parser changes
* grammar : reject candidates in a loop, without a depth limit
The depth limit in llama_grammar_reject_candidates_for_stack counted
the code points of a vocab token, so any token of 256 chars or more
was always rejected under a grammar, such as the 512-space token of
GLM-4.7-Flash or the long space tokens of Laguna-S-2.1.
The rejection now walks the code points with its own stack of levels
instead of recursing: there is no limit and no stack risk, whatever
the length of a token from the GGUF, and the masks match master.
* grammar : track rejected candidates as indices into the input
At a given code point a candidate only differs from its input by the
position of its code point pointer, so the levels now carry indices
into the input candidates instead of copies. The first level needs no
special case and the rejects map back to the input directly.
Grammar sampling gets faster than master on GLM-4.7, Laguna, Qwen3.6
and gemma-4 with the json, list and spaces grammars, same masks.
* nit : shorten the comment of llama_grammar_reject_step
* common : add depth_guard for schema walks
* grammar : move GRAMMAR_RECURSIVE_LIMIT into the .cpp
* common : reduce recursion limit for peg parser
* improve
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
Co-authored-by: Pascal <admin@serveurperso.com>
* llama-quant : fix quantizing 4D tensors
nrows_total was ne[1]*ne[2], so for a 4D tensor only the first ne[2]
slices were quantized and the rest of the rows were dropped. Use
ggml_nrows(tensor) to count all rows, including ne[3].
Fixes#29916
* llama-quant : detect expert tensors by name and fix imatrix size for 4D tensors
The MXFP4_MOE path used ne[2] > 1 to detect MoE expert tensors. This is
wrong for non-expert tensors with 3 or more dims, e.g. MLA projections
(BailingMoE3) and 4D conv kernels. Use the tensor name instead: all
tensors used by GGML_OP_MUL_MAT_ID are named *_exps or *_chexps. This
also removes the BailingMoE3 special case.
The imatrix holds one vector of ne[0] values per 2D slice, so the
expected size is ne[0]*ne[2]*ne[3], not ne[0]*ne[2]. This matches how
the imatrix is indexed per row for 4D tensors in llama_tensor_quantize_impl.
Assisted-by: Claude Opus 5.5
* cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows
For prompt processing, run the gate and up matmuls of a dense FFN as one
mul_mat_q launch and apply the GLU in the write-back.
- The loaders take the gate weights as a second pointer and fill the weight
tile with up and gate rows interleaved in chunks of 8, so each thread holds
an up value and the matching gate value (rows r and r + 8 of an MMA tile).
- The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[]
per thread. The fused write-back computes the GLU from registers and writes
half the rows.
- The fused path uses the regular tiling (no stream-k) with the column tiles
innermost so that neighbouring blocks share weight tiles.
- Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.
* cuda: assert the fused MMQ gate argument and trim redundant comments
* cuda: move the fused MMQ hint check to should_fuse and add TODOs
* cuda: pass prec_src1 to the fused MMQ write-back
* cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back
* cuda: require has_fusion explicitly, drop the fusion-only src1 precision check
* cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile
* cuda: inline the MMQ tiling-efficiency check
* tests: one fused gate/up MMQ case per type and GLU op
* cuda: always launch the fused MMQ kernel with one block per tile
* tests: shorten the fused gate/up MMQ test comment
* cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs
* cuda: run the fused MMQ kernel on the plain tiling path
* cuda: put the column tiles innermost in the MMQ plain tiling path
* vulkan: Add a larger tile size for conv3d (coopmat2 only)
This adds a larger tile size, in practice it is only for coopmat2 where
we don't need shared memory to store the output (would use more than 64KB).
Add some additional representative backend tests. At least, this is what codex
came up with from looking at sd.cpp Wan VAE.
* only apply large tile size for newer driver versions
WM of the small tiles was 32, or 8 for 8-wide subgroups, whatever the
tile's BLOCK_SIZE and WARP. With 16-wide subgroups (Mali) the second
subgroup of a workgroup wrote into the next tile's columns. With 8-wide
subgroups some _id tiles left rows uncomputed. Derive WM so there is one
warp tile per subgroup.
Tested on my Pixel 9 Android phone. Before the change there were 20
MUL_MAT and 6 MUL_MAT_ID failures. After the change they all pass.
Fixes#28637
Assisted-by: Claude Opus 5.5
* hex-tools: support for --compare option in the profile.py
* hex-unary: improve collapse logic to also do splits if nrows is small
* hex-colapse: handle all colapsing at the host and make it generic
* hex-colapse: remove leftover unary helper
* chat : reject JSON nested more than 128 levels before walking it
The schema builder, the jinja engine and the decision code walk request
JSON with recursion. A request nested deep enough overflows the stack of
the HTTP thread and llama-server dies (SIGBUS on macOS, SIGSEGV on Linux):
for a tool schema from a few hundred levels on macOS and about 10000 on
Linux.
Check the depth first and answer 400 past 128 levels: the chat inputs in
common_chat_templates_apply (tools, tool-call arguments,
chat_template_kwargs, json_schema), the json_schema of /completion, and the
whole /v1/systemone body. A gemma4 tool result that is too deep stays text.
* fix the root, not the call side
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add DeepSeek-V4.1-Flash (deepseek41)
* deepseek41 : remove MXFP8, convert the Engram table to BF16
* deepseek41 : compute the compressed-KV read layers once in the cache
* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le
* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather
* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer
* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle
* deepseek41 : keep il last in build_hc_mixes
* deepseek41 : sort the top-k rows before the sparse gather
* tests : add deepseek41 to the Metal fusion baseline
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* hexagon: KV-head team split for HMX flash attention in multicore
* hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag
0 = off (token-split only); 1 = head-split, plus team-split when
n_kv_heads < N_cores so stranded cores still participate (e.g. gemma-4
with 2 KV heads on 4 cores). Where heads divide evenly across cores,
plain head-split is used.
@@ -16,6 +16,8 @@ The project differentiates between 3 levels of contributors:
>
> Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file.
### Requirements
If AI is used to generate any portion of the code, contributors must adhere to the following requirements:
1. Explicitly disclose the manner in which AI was employed.
@@ -24,6 +26,16 @@ If AI is used to generate any portion of the code, contributors must adhere to t
4. Be prepared to explain every line of code you submit when asked about it by a maintainer.
5. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
### Checklist
> [!IMPORTANT]
>
> **If you use AI, do all of the following:**
>
> 1. **Make sure [AGENTS.md](AGENTS.md) is loaded into your agent's context.** Skipping this step is one of the most common reasons PRs get rejected.
> 2. **Search for related discussions, PRs and issues** to get the broader context before writing any code.
> 3. **Run a local review pass before pushing anything**, using the [`code-review`](skills/code-review/SKILL.md) skill, once the change is implemented. Fix what it finds.
For more info, please refer to the [AGENTS.md](AGENTS.md) file.
`/root/DEPLOY-ROLLBACK-commit-12.txt` = `2fc9f9c35`, the previous production; next free pointer 13),
with the disk prompt cache ON (F11). `2fc9f9c35` was validated with the full `fork-validate.sh` set
(37 PASS, 0 FAIL, runbook section 6). Any later commit that touches compiled code needs the same
validation before a redeploy.
-The disk prompt cache is ON in production (F11). `2fc9f9c35` was validated with the full
`fork-validate.sh` set (37 PASS, 0 FAIL). Any later commit that touches compiled code needs the
same validation before a redeploy.
- Branch `fable5/disk-prompt-cache` added F11 (disk prompt cache) and a CPU_REPACK release fix (F4).
A deploy of cb48bf224 was rolled back on 2026-10-25 (mmproj abort, F11); the fix (adc2aa3ec) passed
the 35B run with the production command line on 2026-11-21 and is deployed.
@@ -44,6 +53,9 @@ Status at the time of writing (2026-11-21):
- F9 Benchmark harness and validation script
- F10 Page-cache drop after a copying load
- F11 Disk prompt cache (save slots on shutdown, restore after a restart)
- F12 Temporary upstream carries (open upstream PRs, drop when merged)
- F13 Per-pass views of host weights in looped layers (inkling2_loop)
- F14 User-message batch-break merge (one ubatch per short turn)
4. Dead ends
5. Configuration that matters
6. Measurement methodology and traps
@@ -125,6 +137,10 @@ Oldest first. "Feature" points at the section below.
| dd851fd0c | llama: read a released KV/recurrent state from its host shadow | F4, F11 |
| 2874dcb10 | server: disk prompt cache, save slots on shutdown and restore after a restart | F11 |
| 2c6c517ba | scripts: fork-validate.sh --diskcache, the disk prompt cache | F9, F11 |
| d2396e4e3 | upstream-carry #32735: vulkan : support non-contiguous src in SCALE (open upstream PR, temporary) | F12 |
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1083,6 +1099,440 @@ first start. Verified in production: a 35B 4.6k-token chat was saved on eviction
only 32 tokens; a gemma image request with the flags on works and is not cached. `/root` had 11-13 GB free on 2026-10-25, and the 1 GiB free-space floor still
applies. Models only pick up the flags when they restart.
### F12. Temporary upstream carries (open upstream PRs, drop when merged)
**What and why.** Open upstream PRs carried unchanged until upstream merges them. Each carry is one
commit whose subject starts with `upstream-carry #<PR>:`; the author is the PR author, and the body
names the PR URL and the PR head that was carried. List them with
`git log --oneline --grep '^upstream-carry' origin/master..HEAD`. They are not fork features: never
edit them in place, re-fetch the PR instead.
Carried since 2026-11-22 (both on upstream `e6d624b9a`, for Inkling-2-Mini, section 5):
- #32735 `vulkan : support non-contiguous src in SCALE` (head `0ea4ffd00`): the Vulkan backend runs
SCALE on a non-contiguous source instead of rejecting it.
- #32740 `ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan` (head
The main goal of `llama.cpp` is to enable LLM (and MLLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
The `llama.cpp` project is built on top of the [ggml](https://github.com/ggml-org/ggml) library.
## This fork - RX 580 (Polaris / GCN) serving of large MoE models
A private fork maintained for one machine: an AMD RX 580 8 GB (RADV Vulkan) serving Qwen3.6-35B-A3B,
# SPIRV-Headers walks up three directories and appends /include, yielding /usr/local/lib/include; the extra vulkan/ level likely breaks its prefix calculation.
string_format("share of the MoE cache misses that the CPU computes from the host experts instead of uploading them to the GPU once the cache is full, from 0 to 1 (default: %.2f)",(double)params.moe_cache_cpu_share),
[](common_params¶ms,conststd::string&value){
constfloatshare=std::stof(value);
if(share<0.0f||share>1.0f){
throwstd::invalid_argument("invalid value");
}
params.moe_cache_cpu_share=share;
}
).set_env("LLAMA_ARG_MOE_CACHE_CPU_SHARE"));
add_opt(common_arg(
{"-ncffn","--n-cpu-ffn"},"N",
"keep the dense FFN weights of the first N layers in the CPU\n"
"experimental: whether to enable MCP CORS proxy - do not enable in untrusted environments (default: disabled)",
"experimental: whether to enable MCP CORS proxy, for Llama Chat to reach public MCP servers without CORS headers; local and private addresses are refused unless --ui-mcp-proxy-allow-private-urls is set - do not enable in untrusted environments (default: disabled)\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
"allow the MCP CORS proxy to reach local or private addresses (e.g. MCP servers on localhost or LAN) - anyone who can reach the proxy can then reach your local network (default: disabled, only public addresses)\n"
string_format("for router server, MiB of device memory to keep free per device, unload LRU models until a new one fits (default: %d, 0 = disabled)",params.models_memory_margin),
Loaded 100 of 2806 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.