Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last
user message (and of earlier ones on a fresh prompt) so that a context
checkpoint lands there. On a hybrid/recurrent model with MoE experts on the
host, that turns a typical chat turn (previous answer + new user message, under
one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the
host experts again.
When the rest of the prompt from the current batch start fits in one ubatch,
skip the user-start break. The near-end rule still checkpoints the batch start
and the end-4 rule the prompt end, so a later prompt that diverges inside the
last user message re-processes less than one ubatch more, in the same ubatch.
Prompts longer than one ubatch, and batches after a media chunk, keep the
upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
What and why (#32755 and the 6.5 GiB of shared expert copies), the knob,
F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the
pod validation plan (reference with the loop experts on the GPU, views on,
views off).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
inkling2_loop runs the core layers (6..17 on Inkling-2-Mini) three times, and
post_arch_tensors makes every cache slot share the tensors of its physical
layer. With the experts in host memory (--n-cpu-moe, -ot exps=CPU) the
scheduler makes one device copy per (host tensor, backend) per graph, at the
first split that reads it, and keeps it until the last reader. So:
- the expert copies of all 12 loop layers are alive at once (about 6.5 GiB on
UD-Q4_K_XL), which is most of the ~7 GiB Vulkan0 compute buffer;
- sched_copy_experts runs only for the first reader. When it reads the ids back
(ubatches of 32-63 tokens: MUL_MAT_ID is offloaded from 32 tokens, the F1
skip starts at 4*n_expert ids), it uploads only the experts that pass 0
selected, and passes 1 and 2 read stale VRAM for the others (upstream issue
#32755, "GGGG" output).
Add a post_load_tensors hook to llama_model (called after the weights are
loaded, and on the no_alloc path after the dummy buffers are set). In
inkling2_loop it replaces, for each slot of a later pass, every tensor in a
host buffer with a view of it (name "<name>.visit<pass>", buffer and data of
the original). The scheduler hashes tensors by pointer, so each pass gets its
own copy, freed after that pass, and the copy callback sees the ids of that
pass. Weights in device buffers and CPU extra buffers (CPU_REPACK, not host)
stay shared. The pointer-keyed weight maps (w_scales, prec_policy, hdmd.rot)
get entries for the views. The views live in a model-owned context outside
ctxs_bufs, so the F4 release/restore, tensor counts and the memory breakdown
do not see them; they point only at host buffers, which F4 never releases.
LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors (A/B knob).
Not built or run yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32740 at head 808fd1c4e,
its four commits (1fa93f902, d29f9d489, ff91e4f84, 808fd1c4e) squashed.
Inkling's per-query bias table is applied inside flash attention
instead of a dense [n_kv, n_tokens, n_head] bias tensor, which cuts
Inkling's compute buffers and long-context prefill time on the RX 580.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").
One conflict, ggml-vulkan.cpp use_mask_opt: kept the fork's GCN
`HSK >= 256 || HSV >= 256` clause and its comment (F2) and upstream's
`!use_sparse`, added the PR's `!bias &&`.
Original commit trailer: Assisted-by: Codex
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32735 (head 0ea4ffd00),
cherry-picked unchanged. The Vulkan backend runs SCALE on a
non-contiguous source instead of rejecting it (CPU fallback). Carried
together with #32740 and evaluated as a pair.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").
Original commit trailer: Assisted-By: Opus 5.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
HF_UI_VERSION must use the tag after the '|' in .ui-stamp; bundle path
for agents told not to push; build backup and copy-based rollback;
status, pointers and last-run record for 3bf6b55b3; q35.py build path
note; disk-cache restore test wording; open questions.
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
Upstream #30591 added its own tiled-transpose concat pipelines, selected
for any dim-0 concat whose src1 is transposed. Behind the fork's fast path
(which already takes ne0 >= 32 and honours GGML_VK_CONCAT_FASTPATH) that
selector still caught the decode case (ne0 < 32), bringing back the
decode regression the TILE_DIM gate fixed (FORK.md F3), and
GGML_VK_CONCAT_FASTPATH=0 no longer gave the generic kernel.
Apply the same gate and the knob to upstream's selector.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort
with the production mmproj), the fix, the new mmproj cases (15 PASS), and
that the earlier 35B numbers were measured with --no-mmproj. The 35B run with
production's exact command line (diskcache-window3.sh) is ready, not run, and
a precondition for deploying --cache-disk-path.
Runbook:
- tests must use production's exact model command line, mmproj included;
- the validated build has all three targets;
- the deploy mtime list includes libllama-server-impl.so and
libllama-common.so (llama-server is a stub);
- deploy and rollback steps run as one setsid-wrapped script on the pod,
never as a backgrounded cat && sh -n && setsid chain from kubectl exec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production runs the 35B and gemma with an mmproj, and the first deploy of the
disk prompt cache aborted on that path. Every case had used --no-mmproj. Now
--no-mmproj is in DC_MM, separate from DC_ARGS, and two cases run with the
0.8B's mmproj (--no-mmproj-offload, arbiter env, LLAMA_MMPROJ_ONDEMAND=1),
against their own reference:
- restore-mmproj: turn 1 is saved on SIGTERM and restored after the restart,
prompt_n equal to the reference, output identical;
- mmproj-image: an image request (no restore, answer "red"), then the text
chat (restored, identical), then an image again, then SIGTERM with media in
the slot (no state copied, exit 0, the file count unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deploy of cb48bf224 with --cache-disk-path was rolled back: the first
request to the production 35B aborted in disk_cache_load().
server_tokens::get_tokens() has GGML_ASSERT(!has_mtmd), and has_mtmd is true
whenever an mmproj is loaded, even for a text-only prompt. Production's 35B
and gemma-26B always load one (-hf pulls it). disk_cache_collect() and
server_prompt_disk_cache::save() made the same call, so the shutdown save
would have aborted too. Every test had used --no-mmproj, which hid it.
disk_cache_text() takes the list from get_text_tokens(). A prompt is
text-only when that list is as long as size(), because media positions hold
LLAMA_TOKEN_NULL. So a text-only prompt is saved and restored with an mmproj
loaded, which is the main case (OpenWebUI chats on the 35B and gemma), and a
prompt with media is never saved or restored. When the slot itself holds
media, its reuse estimate is server_tokens::get_common_prefix(). These four
calls were the only get_tokens()/mtmd-unsafe accesses in the feature.
fork-validate --diskcache with the new mmproj cases, beside production:
15 PASS, 0 FAIL, GPU legs included. With the 0.8B's mmproj loaded, a text
chat is restored: prompt_n 58 of 596, output identical to the reference.
An image request is answered "red" with no restore, the text chat after it
is restored, and a SIGTERM with media in the slot copies no state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.
Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.
Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F11 in the per-feature format: design, invariants, the measured costs (CPU
only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at
30 s). Also: the recommended config (not applied), the commit map entries, the
F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache,
the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run
yet). Runbook: --diskcache in phase 1, the pod window script, the deploy
notes (every deploy empties the cache, /root free space, a slower stop), a
post-deploy check, and the cache dir.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs beside production. The model is the small hybrid (Qwen3.5-0.8B), CPU only
(-dev none -ngl 0, no VRAM), with chat requests and greedy decoding. A
reference server that never restarts runs turn 1 and turn 2. The candidate
saves turn 1 on SIGTERM and restarts. Turn 2 must then be restored from disk
with the same prompt_n as the reference (only the suffix processed) and the
same answer. Cases:
- restore plain, with the arbiter env warm, and forced cold by a fake waiter
on want.lock (the save from a released state). Each case records the exit
time and SIGTERM-to-token-free, also once without the cache;
- no restore for another conversation, or with another n_ctx, KV type or
model path (other key);
- a truncated file is removed at startup and a byte-flipped one at load,
never used;
- the size limit evicts the oldest file;
- a write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored: EFBIG) is logged, the
exit stays 0 with no file left, and the next start serves;
- the same restore on the GPU (-ngl 99, arbiter env, warm and cold) when
>= 1.5 GiB of VRAM is free, else SKIP.
start_server gains SRV_WRAP, a command prefix that execs the server.
First run (2026-10-25, beside production): 11 PASS, 0 FAIL; the GPU leg
SKIPped (1219 MiB free). The first run found the CPU_REPACK release crash
(exit 139 in the cold case).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama-swap SIGTERMs a model when the owner switches models (only one large
model fits in RAM), and switching back re-prefills the whole conversation:
about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR,
each slot with a prompt is saved to DIR on shutdown. After the restart, a
request whose prompt starts with a saved prompt is restored from the file,
so only the suffix is processed. Off by default.
Save, in server_context::start_loop() after the loop returns:
- disk_cache_collect() copies the state of each slot with >= 256 text tokens
to host memory (llama_state_seq_get_data_ext, flags NONE) while the device
buffers still exist. A cold server reads its released state through
llama_context_restore_memory_host(), with no device use and no extra copy.
- vram_share_exit() then frees the VRAM and releases the GPU token as before,
so only the D2H copy delays the token release, never the disk IO.
- disk_cache_flush() writes the files. A slot is skipped if its KV does not
cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not
decoded) or if the same token list is already on disk.
File (server_prompt_disk_cache, server-task.cpp):
- A 48-byte header (magic, version, key hash, sizes, payload hash), then a
payload of 8-byte padded sections: tokens, checkpoint table, state, the
checkpoint states. The hash is FNV-1a over the 64-bit words of the payload.
- Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read
is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked
the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file.
- The index is built at startup from the headers. A bad magic, a size that
does not match the header, or an inconsistent table means the file is
removed. A hash mismatch at load also removes it, and nothing is restored.
Other keys and other versions are kept. A .tmp older than 1 h is removed.
- --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the
least recently used *.lpcd files of every model in DIR are removed until the
new file fits; a restore updates the file mtime. Entries the new prompt
extends are dropped after the write and do not count against the limit.
A write also needs its size + 1 GiB free on the file system. A write error
or a full disk is logged; the file is not saved and the exit stays clean.
Key: the build commit, the model path + size + mtime, LoRA and control
vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the
K/V types, the flash-attn type, and the RoPE/YaRN parameters.
Restore, in get_available_slot() after the RAM cache:
- Only while warm (holding the GPU token), with cache_prompt on and no media
tokens. The file must reuse at least 256 more tokens than the slot holds.
- Reusable means one of two things. Either the saved token list is an exact
prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint
at or below the common prefix can be used. A plain-KV model can be cut back
to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the
template rewrites the previous assistant turn (the reasoning is dropped), so
the saved list is almost never a prefix of the next turn. The checkpoints
are exact snapshots, and update_slots() rolls back to one exactly as it does
for the RAM cache. They are restored with id_task -1, so one that fails to
load means a re-prefill, not an abort.
- The slot's current prompt goes to the RAM cache first. A set_data failure
clears the slot and removes the file.
Not supported with speculative decoding (warning, disabled): the draft state
is not saved. Only the slots are saved, not the RAM cache entries.
Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention):
- 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms,
exit 314 ms (125 ms without the cache). After the restart: read 166 ms,
restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full
prefill. The answer is identical to a server that never restarted.
- 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS
+162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms,
turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill.
- The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node
overlay) stalled once: 5.8 s for 64.5 MiB.
llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a process that is still there after 30 s. So the exit
save adds to the model switch time: about 1 s per GB on the PVC.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After llama_context_release_device(ctx, true) the KV cache and the recurrent
state live only in host shadows, and their tensors have no buffer, so
llama_state_seq_get_data cannot read them. The disk prompt cache must save
a cold server's slots at exit, and a shutting-down server must not restore
to the device (it may not hold the GPU token, F7).
llama_context_restore_memory_host() makes the state readable without the
device. restore_device_buffers(to_host = true) wraps each shadow as a
CPU_Mapped buffer (ggml_backend_cpu_buffer_from_ptr) and points the tensors
into it: there is no copy and no allocation. The first version copied into a
CPU buffer instead, which needs the full KV allocation a second time, about
2.7 GB for the 35B at ctx 131072; the node has had an OOM-controller kill
before. The buffer base is rounded down to the 32-byte alignment that
cpu_buffer_from_ptr asserts (a large std::vector is only 16-byte aligned);
those leading bytes are never accessed. With to_host the shadows are kept,
because the buffers point into them.
It sets device_discarded, so restore_device() aborts afterwards. Exit only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A model with weights on the CPU device (-dev none / -ngl 0) segfaulted in
llama_model::release_device_weights() the first time the VRAM arbiter put
it to sleep: CPU_REPACK buffers are not host buffers, so load_tensors marked
them releasable, and capturing the shadow called their NULL get_tensor.
Reproduced with the production build 2fc9f9c35 (Qwen3.5-0.8B, -dev none
-ngl 0, LLAMA_SLEEP_VRAM_ONLY=1 LLAMA_SLEEP_EVICT_KV=1, force a yield with a
waiter on want.lock): exit 139 at the release, before any SIGTERM. The
faulting call (rip 0) returned into release_device_weights.
Only buffers on a non-CPU device are releasable now. These buffers are in RAM
already, so there is nothing to free. Production is not affected: every
arbiter model there has its weights on the GPU, and a releasable CPU_REPACK
buffer would have crashed it. Found by the --diskcache check, which runs
its arbiter cases CPU only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Feedback from the 2026-10-25 deploy, which used only these docs:
- status: production is 2fc9f9c35; F10 deployed and verified on a 35B
GPU load; --moe-exact and median --perf have run on the GPU; last
full fork-validate run (37 PASS) and its timings; the first-candidate
-arm VRAM spike seen twice is an open question.
- runbook: a no-rebase redeploy path; --arbiter beside production only
if 1536 MiB VRAM is free, else in the swapdown window; full --moecache
model path; page-cache eviction before and after phase 2; deploy
steps reordered so the memory check follows stopping llama-swap;
in-place rebuild and system_fingerprint check; mtime check through the
versioned symlinks; rollback pointer numbering and one format; disk
space; nothink alias and a PPM image helper for post-deploy requests;
power1_cap via sysfs; restart and OOM checks; harmless UI warning.
- fork-validate.sh header: note the --arbiter VRAM requirement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With -lm none the GGUF is read() into our own buffers, but the kernel keeps
the file in page cache afterwards, charged to the pod's cgroup: about 20 GB
after a 35B load. On 2026-10-24 the Talos OOM controller killed the pod right
after a load under exactly that pressure. Those pages are pure waste once the
data is copied.
llama_file::drop_cache() wraps posix_fadvise(POSIX_FADV_DONTNEED) (Linux only,
errors logged at debug). load_all_data() drops each copied tensor's range right
after reading it, so the cache never grows past about one tensor, and drops the
whole file at the final cleanup to catch the metadata and partial pages at
tensor edges. Both only when the loader holds no mapping of the files
(mappings.empty(): not mmap, not mmap+mlock, no lazy tensors), so nothing that
is mapped is ever advised away. Split GGUFs are covered (every file in files).
clip.cpp always copies the mmproj via ifstream, so it drops the whole file
after loading it.
Measured on the pod, gemma-4-E2B UD-Q4_K_XL (2.5 GiB) + mmproj BF16 (0.9 GiB),
CPU only, -lm none, cold cache, same binary with the drops disabled as the
control (4 rounds, rotated order):
- file pages resident after load (mincore): 2431-2444 / 940 MiB -> 0 / 0
- peak model-file residency during load: 2.5 GiB -> 1.2-1.5 GiB (largest tensor)
- pod cgroup file after load: +3.37 GiB -> +0; host MemFree after load
0.6 GB -> 2.5-2.8 GB (MemAvailable counts clean cache, so it cannot show this)
- load time: 5.87 s -> 6.82 s median (+0.95 s); a post-load-only drop is
6.56 s (+0.68 s). The cost is synchronous page freeing that reclaim would
otherwise do later; -lm dio (direct IO) costs the same (+0.9 s) and does not
cover metadata or the mmproj.
- greedy output byte-identical; -lm mmap keeps the model file fully resident
and mapped (2498 MiB, no refaults); -lzm on skips the drop (file mapped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 1 now covers the VRAM release/restore check and the handoff latency
A/B against --ref (which must be the production build); phase 2 adds
--moe-exact on the 35B and keeps its scratch dir. Explains median gating and
--perf-report, that a pod restart empties /tmp, that the manual checks are
now automated, and that gdb is not in the pod image (lost in the 2026-10-24
restart). States the commit trailer convention and what llama-swap does with
a child that exits vs one that wedges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README fork section still said "the two code changes below are auto-on"
(the fork has three prefill changes plus the VRAM sharing and server fixes)
and recommended --n-cpu-moe 28 on the 35B, which FORK.md section 4 records as
worse at depth and unable to hold a long context. Replace it with a short
accurate summary that points to FORK.md, the runbook and the scripts. The
Laguna time profile, the iSWA KV note and Laguna's --parallel 1 saving were
only in the README and move to FORK.md sections 4 and 5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- status: no hardcoded fork-commit count (the list includes docs and
scripts; count git log origin/master..HEAD); what is after the production
commit; the commit-trailer convention of this private fork vs upstream's
AGENTS.md rules
- commit map: the commits after the production commit
- F1: validated by the new --moe-exact check and the LLAMA_MOE_IDS_SKIP knob
- F3: every graph that builds concat(state, transpose(x), 0) (bailingmoe3,
clef, kimi-linear, kimi-k3, glm5-next, mamba-base, plamo2, lfm-base,
qwen4exp besides delta-net); only the delta-net path is measured
- F4/F6: the new v-vram-* and a-handoff-latency checks with first results
- F5: encode_mtmd_chunk was split out of process_mtmd_chunk, not renamed
- F6: the vram_cold ordering rule restated from the code (the measured
failure was under the doorbell design; with the inline probe should_sleep
cannot run inside the restore; keep it, and A/B handoff latency before any
change) and why vram_acquire_for_load sets the flag early
- F7: what llama-swap does when a child dies, verified with a scratch
llama-swap: "upstream exited unexpectedly", restart on the next request
- F9: new flags, phases, median gating, 2026-10-24 reference values, the
/tmp and gdb loss on the pod restart, new grep targets and gaps
- section 7: stale comments fixed, what is not yet run on the GPU
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F1 (the routing-ids readback skip in sched_copy_experts) had no automated
check. --moe-exact MODEL reads expert_count / expert_used_count from the GGUF
header, computes the skip threshold ceil(4 * n_expert / n_expert_used) tokens
per ubatch (128 on the 35B), and runs one greedy request with a prompt of
about 2x that in a single ubatch, every expert on the host, so the prefill
takes the skip and each decode step takes the readback. The candidate's 64
tokens must equal the reference's byte for byte; with the LLAMA_MOE_IDS_SKIP
knob in the candidate, also the candidate with the skip forced off (=0), which
proves the skip exact even when the reference carries it, and =log must show
skips, all at >= 4 * n_expert ids. Without the knob skip-engaged SKIPs with
the batch-size argument. It needs the GPU to itself (same guard as --perf,
now shared as gpu_exclusive); FV_TEST_SHARED_GPU=1 bypasses the guard for
script tests against stub binaries only.
Also unset inherited LLAMA_MOE* variables.
Not run on the GPU yet (needs a swapdown window). Dry-run on the pod
against stub servers and the real 35B GGUF header: threshold 128, all-PASS
when outputs match, FAIL on a differing reference, SKIP without the knob,
FAIL for a dense model and while llama-swap runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing checked that a release actually frees VRAM. --arbiter now takes a
baseline of mem_info_vram_used before any scratch server starts, then with A
warm and alone on the scratch arena makes it release through a fake waiter
on want.lock. v-vram-release requires the card-wide delta to drop by at
least 70% of what A added warm; v-vram-restore requires it back (and the
output equal to the reference) after the next request. Production's own
share is re-read at every point from the DRM fdinfo of every process the
script did not start (deduplicated by drm-client-id); if it moved by more
than 64 MiB the check SKIPs instead of blaming the candidate.
Run beside production on the pod (reb1024 vs itself): warm +546 MiB, cold
+4 MiB, restored +546 MiB, production steady at 5053 MiB; 16 PASS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--arbiter only reported the candidate's handoff latency, while the fork's
process rule (two production deploys lost) is to A/B the ordinary handoff,
not only the pathological cases. After the arbiter cases, a-handoff-latency
starts fresh A/B server pairs of each build on the scratch arena and runs
blocks ref, cand, cand, ref of 16 alternating handoffs each. It FAILs if the
candidate median exceeds the reference median by more than max(25%, 200 ms)
and reports both medians, maxima and per-block medians.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--perf averaged the two per-arm llama-bench means of each build, so one
catastrophic sample (CPU contention from the ~60 other pods on the node) moved
the result, against the fork's own rule of medians, never means. llama-bench
now writes -o jsonl (every repetition sample) next to the markdown table, and
the gates use the median of all samples of a build pooled over its two arms
(4 per prefill cell, 6 for decode); the table shows min-max and n.
--perf-report TSV re-aggregates a saved <scratch>/perf.tsv without a GPU.
Checked offline on the pod: the 2026-09 A/B samples in results.txt give
pp2048 reference 250.24 (old mean) vs 247.76 (new median), candidate 280.15
vs 276.93; a synthetic 2026-10-24-like set with one 180 t/s candidate sample
read -9.3% (FAIL) the old way and -0.2% (PASS) with medians.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The llama-bench sweep measured decode as tg32 at the default depth and the
README called it the continuity metric, while tg32 at n_ctx 32 is launch /
barrier bound on this card and is not a valid decode metric (it once read
-36% between builds at parity at tg256). lbsweep.sh now runs prefill
(-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as
two invocations and labels depth cells (tg256@d4096). summarize.py marks
old depth-less tg rows as invalid-metric history. The README examples and
the reference values are updated to the 2026-10-24 production numbers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LLAMA_MOE_IDS_SKIP=0 forces the ids readback in sched_copy_experts (the
upstream path) and =log reports each skip on stderr, so validation can show
that the skip engages at prefill and that output is byte-identical with and
without it in one build. Read once (function-local statics); unset changes
nothing. Not set in production.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments only, plus one log string:
- the arbiter block comment said the KV cache is never evicted; with
LLAMA_SLEEP_EVICT_KV it is (and the scheduler and scratch with it)
- vram_go_cold said vram_ensure_warm() rebuilds the encoder; only
encode_mtmd_chunk() restores it, and only for an encode
- vram_ensure_warm and the mtmd init block named process_mtmd_chunk; the
restore lives in encode_mtmd_chunk
- handle_sleeping_state said waking happens lazily before decode; it
restores eagerly on the wake transition, before update_slots()
- the vram_cold ordering note now says why it still holds with the inline
want-lock probe
- the sleep log line said "keeping KV cache" even when LLAMA_SLEEP_EVICT_KV
evicts it; it now names what actually happens (the prefix
"server entering sleeping state" that fork-validate.sh counts is kept)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FORK.md documents every fork feature (what and why with measured numbers,
where, how it works, invariants, past failures, rebase hazards, validation),
the dead ends measured on this hardware, the production configuration and
the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation,
deploy, rollback and GPU-recovery procedure plus an operations reference.
The README fork section links both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checks the op kernels, the concat fast path, the VRAM arbiter incl. every
shutdown path that has caused a production incident, and prefill/decode
against a reference build. Prints PASS/FAIL per check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GGML_VK_CONCAT_FASTPATH=0 forces the generic path and =log reports each
fast-path dispatch, so validation can prove the path fires at prefill and
not at decode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:
- SIGTERM with a request in flight: terminate() stopped only the task queue.
The HTTP handler kept waiting for results the loop would never send, the main
thread blocked joining it, and the token was released only in the destructor
after that join - i.e. never, until the client went away. Terminate the
result queue too (readers now get "server is shutting down": a 503, or an SSE
error event), and let wait_until_no_sleep give up once stopping.
- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
inside the wake callback, holding mutex_tasks, so terminate() waited for the
current holder to finish and then for a pointless restore of our weights.
Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
stays cold, skips the restore, and the loop exits before update_slots.
Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.
Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A SIGTERM that lands while the start_loop() thread is inside a sleeping
callback deadlocks the server. The callbacks run under
server_queue::mutex_tasks, and the kernel delivers the signal to that same
thread; the handler calls terminate(), which locks mutex_tasks again, and the
thread waits on itself forever.
With the VRAM arbiter that window is wide - a whole device release, or a
blocking wait for the GPU token - and llama-swap sends SIGTERM to evict
models, so it hit production: an evicted model wedged mid-release, never
released token.lock, and every other model queued behind it. SIGTERM could
not stop it either; only SIGKILL did.
Make the handler only write the signal number to a self-pipe, which is
async-signal-safe, and run shutdown_handler on a dedicated thread. terminate()
there simply waits until the loop thread drops mutex_tasks, after which the
loop exits normally. The thread is joined before the objects the handler uses
are destroyed. If pipe() fails the old direct call is kept.
Reproduced 6/6 off production (SIGTERM during release and during the token
wait); with the fix 8/8 clean exits, and handoffs, image requests and greedy
output are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream #28334 removed --mmap/--no-mmap/--mlock and llama-bench's -mmp, so the
harness flags no longer parse: the server would refuse to start and llama-bench
would silently produce no rows. Switch both harnesses to -lm none, the same
load mode, and note --moe-cache-mib as a server-only option worth measuring.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The MoE cache (--moe-cache-mib, #29887) keeps the most recently used host
experts in device banks. The on-demand VRAM path (release_device /
restore_device, used by the server VRAM arbiter and the on-demand mmproj path)
did not know about it, so its banks were never freed on handoff. Restoring them
by reallocation alone would also be wrong: the banks come back zeroed while the
slot maps still mark experts as cached, so the model would silently read zero
weights.
Free the banks on release and empty the cache instead of shadowing it: re-init
every LRU, re-point its slot-map pointers at the host slot maps and clear those
to -1. The banks are reallocated (which also re-inits their views) and cleared
on restore, before the scheduler is rebuilt, and the cache refills from the
host experts as before. Release runs in both modes, weights-only and evict_kv.
device_resident() now covers the cache too, for the early returns in
release/restore, the auto-restore in decode() and
llama_context_weights_resident().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
release_device(evict_kv) drops the scheduler, so a context evicted by the VRAM
arbiter and then destroyed reaches the destructor with a null sched. The upstream
"compute buffer size matches expectation" loop calls
ggml_backend_sched_get_buffer_size() unconditionally, whose GGML_ASSERT(sched)
then aborts: the server exits 134 instead of 0 on every shutdown taken while
cold, which under llama-swap makes an ordinary stop look like a crashed child and
leaves a ggml backtrace in the log each time. It also skipped the rest of the
destructor, so the cold teardown path had never actually run to completion.
Guard at the call site rather than relaxing the assert - the assert is right, and
every other caller reserves the scheduler first. Same shape as the null guards in
synchronize(), memory_breakdown() and the released-buffer iteration.
The loop is a diagnostic size comparison, already reported by sched_reserve() at
load, so skipping it when cold loses nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the
holder's warden turned the inotify event into a queue flag. A ring carried no
notion of "still wanted", so it was lost whenever the holder was not already warm
and listening - during its model load (the watch does not exist yet, and the
kernel does not queue events for a watch that is not there) or inside
restore_device - and any ring that survived past the point its sender had been
satisfied caused a spurious release. Lost rings wedged the waiter permanently,
because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder
had no other reason to release. The wait blocks under mutex_tasks, so the whole
server stopped answering while /health still returned 200.
Express the request as kernel state instead. A waiter holds <arena>/want.lock
shared while it waits and drops it once it owns the token; the sleep decision
probes that lock non-blocking and releases the GPU while anyone is waiting. The
probe needs its own fd - flock treats two open file descriptions of one file
independently, so probing on the waiter's fd would convert our own lock rather
than conflict with it. Nothing can be missed, nothing goes stale, and a waiter
that dies is cleaned up by the kernel.
Probe from should_sleep() on the loop thread rather than from a warden thread.
Routing it through a flag is what made the first attempts fail: start_loop()
holds mutex_tasks from should_sleep() through the callbacks to the wait, so a
warden's request_yield() blocks on that mutex and is admitted only after the flag
has been consumed, latching a release for the next wake. Reading live state where
the decision is made has no edge to latch, and drops the warden's poll latency.
Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but
the loop clears req_stop_sleeping on the way in, so a loop that slept again
before the waiter ran stranded it forever - re-ask on every wake. And a task
queued after the waiter saw us awake could not wake us by itself, so sleep now
also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the
request that woke us is not queued yet, and yielding at once only sends it round
again.
Measured on the RX 580 pod, two servers contending, 60 alternating handoffs:
0 stranded, median 0.309 s against the doorbell's 0.314 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a
loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of
40 layers) build_conv_state emits a transposed concat per recurrent layer, so
the fast path runs 30 times per decoded token.
At one token the transposed source is a single column, so there is no
uncoalesced stride left to fix, but the path still costs two dispatches and an
unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes
the command stream. Measured -17% tg256 on RX 580; the barrier, not the
half-empty transpose dispatch, is nearly all of it.
Require the transposed source to be at least one TILE_DIM wide. That source is
qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the
fast path, decode and small speculative drafts take the generic one.
pp8192 +11% retained, tg256 back to parity, generation byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.
When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.
Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.
Assisted-by: Claude
Add clip_release_device/clip_restore_device (and mtmd_release_device/
mtmd_restore_device wrappers over the vision + audio contexts) that free
the multimodal encoder's device weight buffer to a read-only host shadow
and rebuild it on demand, using the same shadow/free/reallocate pattern
as llama_model weights. No-op for a CPU-backed encoder. This lets the
server drop the ~hundreds-of-MiB vision encoder from VRAM when it is not
encoding an image.
Assisted-by: Claude
On device release with KV eviction, also call ggml_backend_free_scratch()
on each backend so a cold model drops its Vulkan compute preallocations
(reallocated lazily on the next compute via restore).
Also guard llama_context::memory_breakdown() against a freed scheduler:
evict_kv release resets sched to null, so a cold model that is then torn
down (e.g. terminated by the process manager) hit GGML_ASSERT(sched) in
ggml_backend_sched_get_buffer_type. Skip the compute-buffer accounting
when sched is null.
Assisted-by: Claude
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.
Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).
Assisted-by: Claude
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left
resident when a model's device buffers were released for on-demand VRAM
sharing - llama_memory_recurrent::release_device_buffers() was a no-op
default. Implement it (and restore_device_buffers) with the same
capture-host-shadow / free / reallocate pattern as llama_kv_cache, so
llama_memory_hybrid now evicts both its attention KV and its recurrent
state. The state is read-write, so its shadow is recaptured on every
release.
Assisted-by: Claude
Under LLAMA_SLEEP_EVICT_KV a cold model still held the scheduler's worst-case
compute buffer (hundreds of MiB, e.g. ~700 MiB for gemma-26B) plus the resident
experts. release_device(evict_kv) now also frees the sched (sched.reset() +
sched_need_reserve), and restore_device() rebuilds it via sched_reserve(), so a
fully-evicted model holds essentially no VRAM (gemma cold: 7147 -> 51 MiB).
Trades a heavier re-warm (weights+KV H2D + sched reserve) for the freed VRAM;
weights-only mode is unchanged (fast switch, keeps KV+compute).
Assisted-by: Claude
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.
Assisted-by: Claude
release_device_weights()/release_device_buffers() leave a null buffer in
ctxs_bufs while the device memory is released for on-demand VRAM sharing.
The memory-breakdown / total_size / clear paths iterated these and called
ggml_backend_buffer_get_size()/get_type()/clear() on the null buffer, tripping
GGML_ASSERT(buffer) and aborting on shutdown of a cold model. Skip null buffers
in llama_model::memory_breakdown and llama_kv_cache::{memory_breakdown,
total_size,clear}.
Assisted-by: Claude
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.
- add vram_arena_open() (idempotent flock/doorbell setup) and
vram_acquire_for_load(), called from load_model() before
common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
request without a re-warm (drop the init-time go_cold cold-start).
Assisted-by: Claude
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.
Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.
Assisted-by: Claude
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.
- llama_memory_i: add release_device_buffers()/restore_device_buffers()
(default no-op). Implemented in llama_kv_cache (D2H shadow of the live
ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
buffers when requested; restore_device() rebuilds them. Public API
llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
since the KV shadow adds a D2H/H2D copy of the live cache each cycle.
Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.
Assisted-by: Claude
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.
- llama-model: release_device_weights()/restore_device_weights() capture a
compact host shadow (stable iteration order, view-skipping) and free then
realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
(not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
enforces "resident iff holds token"; an inotify doorbell forces the holder
to release on contention. The warden thread only touches the task queue, so
releases run on the loop thread and never race a decode.
Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).
Assisted-by: Claude
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).
Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).
Assisted-by: opencode
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.
Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.
Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.
Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.
Assisted-by: Claude
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.
Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.
Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.
The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.
Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.
Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
Used angelX to optimize SYCL/drama 3 coordination, results are as follows
Each output reads only inputs with i*s0 <= idx < i*s0 + K, in the same ascending order as before.
Arc Pro B70, oneAPI 2026.1, test-backend-ops perf, base 9b56ccf00 vs patch, median of 3 same-boot A/B/A rounds, A/A 1.000-1.001:
| input [ne0, ne1] | kernel [K, OC, IC] | s0 | base us | patch us | speedup |
|---|---|---|---|---|---|
| [16, 1536] | [16, 768, 1536] | 8 | 3910.57 | 1348.59 | 2.90x |
| [128, 768] | [16, 384, 768] | 8 | 37735.00 | 1665.26 | 22.66x |
| [1024, 384] | [8, 192, 384] | 4 | 270371.75 | 1471.74 | 183.71x |
| [4096, 192] | [4, 96, 192] | 2 | 534614.63 | 752.74 | 710.23x |
Shapes are the drama 3 codec decoder upsampling stack, added as perf cases. Chained 4-layer stack at 1/4/24 input frames: base 66.3/847.4/28638.5 ms, patch 2.3/6.7/30.7 ms. Kernel numbers, not end-to-end TTS latency.
Correctness: test-backend-ops -o CONV_TRANSPOSE_1D 116/116 on base and patch, and 116/116 rebased on 3f03b1772. Seeded output dump (7.46 MB) byte-identical to base.
Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857). With only those cases excluded: 20145/20173 on base and patch, and every patch-run failure also fails on base.
Co-authored-by: Titaniumtown <titaniumtown@proton.me>
* mtmd: support hibiki realtime duplex
* fix and clean up
* fix long session quality
* fix ci
* tests : add the moshi fusion baseline for Metal
The Metal fusion check found 5 RMS_NORM+MUL fusions in the new moshi
arch and no baseline for them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ui: open the context gauge panel on click as a popover
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the context gauge trigger the ghost button hover
Assisted-by: Claude (Opus 5.5 Medium)
The loop slots copied the physical layers at the end of load_arch_tensors,
before the generic pass attaches the optional .scale tensors, so the later
passes through the core ran NVFP4 experts without their per-expert scales.
Add an optional post_arch_tensors hook that runs after the scale pass and
share the slots there.
Fixes https://github.com/ggml-org/llama.cpp/issues/31680
The matmul_tiled() implementation in tinyBLAS_Q0_PPC requires nc to be a multiple of 8 to perform matrix multiplication correctly.
The previous alignment logic contained a special case for n == 4, which allowed n_aligned to be 4. Consequently, the tiled path could be selected with nc = 4, violating the kernel's tile-width requirement.
This caused incorrect matrix multiplication results, observed as garbled inference output when running Flint Q4_K_M and Gemma 3 27B Instruct.
The fused lightning indexer is always used now, so drop the cparams flags, the auto probe and the non-fused fallbacks.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* Conversion
* Separate model_type
* Add gate_rank to allow building from metadata while testing
* Base model
* Loop graph
* Support the new is_varloop attribute
* Adopt naming scheme from https://github.com/ggml-org/llama.cpp/pull/32588
* Rename to varying
* hip: add RDNA1 MMVQ parameter table (1 warp per block for ncols=1)
* hip: write the RDNA1 MMVQ values explicitly instead of recursing into the generic table
* hip: list the RDNA1 MMVQ table before RDNA2 in the enum
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ci : run the OpenVINO GPU job only for OpenVINO changes
The self-hosted OpenVINO GPU job ran for most PRs and every master push
on a shared runner, so it queued for hours.
- ci-self-hosted-openvino.yml (GPU): runs only for changes to
ggml/src/ggml-openvino.
- build-openvino.yml: keeps its triggers (OpenVINO changes on PRs, every
source change pushed to master). The Linux job runs on a GitHub-hosted
runner and tests the OpenVINO CPU device only.
* address review comments
* ci : limit the OpenVINO PR CMake trigger to ggml
* ci : drop the OpenVINO Ubuntu job
* ci : trigger OpenVINO PR runs on tests/CMakeLists.txt
* convert: qwen3-asr: put "language ..." output into thinking block
* convert: qwen3-asr: read the chat template from models/templates
The converter embedded its own copy of Qwen3-ASR.jinja. Read the
bundled file instead, as the Kimi-K3 and Kokoro converters do, so
the GGUF and test-chat share a single template.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Llama Chat compiles @ggml-org/llama-voice from ui/lib/voice/src, but the
CMake stage and the Docker UI stages copied only ui/lib/ui, so their npm
build could not resolve the voice imports. Stage ui/lib/voice/src next to
Llama UI and track it as an input of the CMake build.
Assisted-by: Claude (Opus 5.5 Medium)
* mtmd: drama3 voice cloning
The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3:
- conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2
- fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj,
builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2)
- drama3_gen tower: teacher forcing when the caller gives the codes of the next frame
- drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the
reference frames go through the backbone as codec rows and through the tower with their codes
- mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state
Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX
* fix correctness
* clean up
* rm redundant code
* fish_s2: rm unused import
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui: add a voice engine for capture, peaks and transcription
Voice input needs a live waveform, a clip that can be sent and drawn,
and a transcript, whether the model streams text while the user speaks
or not. This adds the engine for it in src/lib/voice, with no imports
of app stores or components, so it can move into a package later.
- VoiceCapture: one microphone start gives live levels (RMS on a dB
curve, fixed history, newest last), PCM16 mono chunks at 16 kHz from
an inline AudioWorklet (ScriptProcessor fallback), and on stop a
16-bit mono WAV clip with its duration and peaks. Tracks, nodes and
the context are released on stop, cancel, a failed start and page hide.
- computePeaks / computePeaksFromAudio: normalised peaks of samples, a
buffer, or a stored base64 or Blob file.
- transcribeClip: POST /v1/audio/transcriptions, with the server's own
error message on failure.
- RealtimeTranscriber: the /v1/realtime?intent=transcription socket,
queueing audio until session.created, a final commit, and a typed
RealtimeUnsupportedError when the model cannot stream or the socket
is refused (a browser websocket cannot send the API key header).
- startVoiceSession and VoiceRecorder: try realtime, fall back to one
HTTP transcription of the clip, and expose levels and live text.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add voice waveform and transcript components to Llama UI
Voice input needs three visuals that the app can feed with its own
recording and transcription logic: the live waveform while recording,
a playable waveform of a recorded clip, and a clip with its transcript
that opens on hover, keyboard focus or a tap.
VoiceWaveformLive draws input levels as many bars as fit the width,
newest on the right, and holds still with reduced motion.
VoiceWaveform plays a clip through its own audio element, fills the
played part in the primary colour, seeks by press, drag and arrow keys
(a slider) and pauses any other clip that was playing.
VoiceTranscriptReveal keeps the transcript in the DOM for screen
readers while it is closed, and shows transcribing and error states.
The waveform math (resampling peaks, bar capacity, seek and time
formatting) is in src/lib/voice with unit tests. The components are
exported from the package root and from @ggml-org/llama-ui/voice, which
Llama Chat resolves to the sources like the other entry points.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add stories and tests for the voice components
Storybook gets UI/Blocks/Voice: the live waveform fed by a fake
microphone meter in a chat form row, the clip waveform in its sizes and
states, and the transcript reveal in each state and on touch. The demo
audio is a WAV synthesised in the story, so no binary file is added.
The play functions check seeking by keys and by press, the reveal on
hover, keyboard focus and a tap, and a clip without audio. Pressing play
needs a real click before a browser plays audio, so a client test covers
play and pause and one clip pausing another.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: store voice messages with their transcript and send it to models that cannot hear
A voice message recorded in the chat form is an audio attachment like an
uploaded file, plus what the app needs to draw and read it. The audio
attachment gets four optional fields: voice (recorded in the app),
transcript, peaks and durationMs. Stored rows, export and import carry
them as they are, so no migration is needed and older messages keep
working.
The transcript is for reading. A model with audio input still gets the
audio alone; a model without it (the user switched models later) gets the
transcript as a labelled text part in place of the audio. When the
model's modalities are not known yet, or there is no transcript, the
audio goes as before.
A transcript often comes back after the message was sent. The new
voiceTranscriptsStore follows each pending transcript by the clip's name,
hands it to a send that happens after it arrived, and otherwise writes it
into the stored attachment of the message it went out with, in the open
chat or, if the user left it, in the database.
Copying a message now includes its voice transcript.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add Voice settings and resolve what the microphone does per model
The user should not have to think about which kind of voice input is in
use, so the app decides it from the conversation's model and two
settings. resolveVoiceMode is a small pure function: with Voice input on
Automatic and a chat model that has audio input, a recording is a voice
message, transcribed by that same model so a router swaps nothing.
Otherwise it is dictation, with the chat model when it can hear, else,
in router mode, with the chosen transcription model. When no model can
transcribe, the microphone is unavailable and says why.
Settings get a Voice section of their own rather than more rows in
General, which is already long: these four belong together and one of
them lists the server's models.
- Voice input: Automatic (default) or Always dictate.
- Transcription model: Same as chat model (default) or, in router mode,
any model with audio input; on a single-model server it only says
whether that model can transcribe. It notes that dictating with
another model loads it beside the chat model.
- Transcribe voice messages: on by default.
- Transcription language: empty for automatic.
The keys live in the settings registry like the others, so defaults,
storage, reset and export need nothing new. The model picker is drawn by
SettingsChatVoiceTab, the way Extensions and Tools draw their sections.
Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: record voice messages and dictation from the chat form
The microphone now does what the conversation's model allows, without
the user choosing a mode: its label says "Send a voice message" or
"Dictate", and when no model can take audio it is disabled and says why
(shown in router mode only when some model there has audio input, so the
reason can be acted on; a single-model server without audio keeps the
send button as before). The microphone errors keep their warning state
and toast, now also for the voice engine's own capture errors.
Recording goes through the voice engine's VoiceRecorder instead of the
MediaRecorder flow. While it records, the actions row becomes a
recording row: cancel, the live waveform, the elapsed time and stop.
Escape or cancel drops the recording. The form keeps its two rows while
recording, so the message stays visible on desktop and the row fits the
phone layouts and the expanded sheet.
- Dictation writes the live transcript into the message at the caret
as it streams. DictationSpan replaces only the dictated stretch, moves
it when the user types before it, keeps what is typed after it, and
hands text the user edits over to them. When nothing streams, the row
shows Transcribing... after stop and the text goes in once it is back.
Cancel takes the dictated text out again. No audio is attached.
- A voice message streams its live text, if any, under the waveform,
not into the message. On stop the clip is attached as a voice file
with its peaks and length, and its transcript is followed by the voice
transcripts store (unless Transcribe voice messages is off).
Realtime transcription is skipped for servers reached through a tunnel
or with an API key, which a browser websocket cannot carry. The first
dictation with a separate router model shows a one-time note that it
loads beside the chat model.
The form takes an optional voiceRecorder, so tests and stories can run
it without a microphone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw audio attachments as playable waveforms with their transcript
A voice message should read as a clip, not as a file chip. Audio
attachments now render with Llama UI's VoiceWaveform inside
VoiceTranscriptReveal, through a direct branch in the core attachment
item (voice is core, so it does not go through the extension renderers):
the removable sm chip in the chat form, and the md clip in a sent
message, on the user bubble's surface.
- The audio plays from an object URL made from the stored base64 (or the
uploaded file's data URL), revoked when the clip leaves the screen.
- A clip stored without peaks, an older voice message or an uploaded
audio file, gets them computed from its audio once per session, in
memory only; uploaded files keep their name under the waveform.
- The transcript opens under the clip on hover, keyboard focus or a tap,
and shows Transcribing... while the voice transcripts store waits for
it, Transcription unavailable when it failed, and nothing when none
was asked for.
- The full-screen preview keeps its player and shows the transcript.
Tests run the chat form with a fake recorder: the microphone per mode,
the recording row, dictation around typed text, the Transcribing...
state, cancel, and a voice message chip whose transcript arrives later
and goes with the send. Stories show the form recording each kind, a
recorded chip, and sent voice messages with the transcript, pending,
and computed peaks.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the old audio recorder and proxy realtime transcription in dev
The voice engine replaced AudioRecorder, so it goes. The dev server now
proxies WebSockets on /v1, so realtime transcription works under vite dev.
The VoiceCapture test waits for the tone to be recorded instead of a fixed
700 ms, which a loaded test run could miss.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: dictate with a model that is already loaded when the chat model cannot hear
The Transcription model setting's Automatic choice now falls back to a
router model with audio input that is already loaded, so dictating with a
text-only chat model works without picking a model and loads nothing.
A picked model still wins, and the empty value is a named constant.
Idea from upstream PR 27567, by NickM-27.
Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: fill the recording waveform, and open voice message transcripts
- The live waveform fills the row from the start: a recording shorter
than the row is stretched over all its bars, a longer one scrolls with
the newest level on the right. No dotted placeholder.
- A voice message's transcript is open under its clip, in the chat form
while it comes in and in the sent message, with the toggle beside it
to fold it.
- Settings > Voice > Collapse transcripts (off by default) keeps them
folded instead, opening on hover, keyboard focus or a tap.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 2a93f2f948ed976e40f693ce5d3acf1168284b05)
* ui: record voice without ducking other audio, from the first word
- Voice input asks for the raw microphone signal: no echo cancellation,
noise suppression or auto gain. Echo cancellation made the system treat
a recording as a call, which lowers or interrupts other playback, and
auto gain faded the first words in.
- One AudioContext serves every recording, with the capture worklet
loaded once, suspended in between and, where the browser allows it,
without an output device, so recording never touches the speakers.
- The chat form prepares it when it gets focus (no microphone, so no
recording indicator), and a mouse starts recording on press.
- A recording starts counting once audio flows: until then the bar says
Starting... instead of running the timer. Development builds log how
long each step of a start took.
- Settings > Voice > Microphone: Automatic (the system default, or the
built-in microphone when the default is a Bluetooth headset, which
would switch to its low-quality call mode), System default, or a
listed microphone.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 76643e7210d21617d734fdfde677e84af7273153)
* ui: show voice text live in the message box, and follow transcripts as spoken
- A model that hears but has no live ASR (e.g. LFM3-Omni) now gives text
while you record: the realtime client commits the audio piece by piece,
at the pauses the levels show, one piece at a time, and the final
commit takes only what is left.
- A voice message's live text shows as the empty message box's
placeholder. Clicking into the box makes it text to edit there, and
what is said next follows it. On stop the text leaves the box and goes
with the clip; when it was edited, the edit is the clip's transcript.
- Llama UI gains VoiceSpokenText and spokenLength(): spoken words in the
full foreground colour, which they keep, the rest muted. A clip's
transcript follows its playback (estimated over the parts with speech,
as there are no word timings), and live text brightens as it arrives.
- The recording bar has a send button that stops and sends in one go.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit ce773017f902f8a5690ab9ecca368cecb6427422)
* ui: let the capture test start with silence
A capture starts with its first audio block, which can come before a
stream's first sound, so the steady tone is checked from where it begins.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit a41ee5337b83fef685a5cbaff176925f1647ce9d)
* ui: show live voice text sooner, and keep silence from becoming "yeah"
- A model that transcribes per commit (LFM3-Omni) gets shorter pieces:
cut at a 0.35 s pause after speech, or at the next quiet moment once a
piece reaches 3 s, so the text follows the speaker within a few
seconds in dictation, the message box and voice mode.
- When the recording stops, the audio after the last piece is committed
only if speech was heard in it. LFM3-Omni transcribes a commit of pure
silence as "Yeah." (or "はい。"), which ended up at the end of the text.
A recording whose only piece is that rest still commits it.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 43e52cbab0fba9d8e84b9491890a406e3c6c90f3)
* ui: transcribe a recording cut at pauses once more as a whole
Live text comes piece by piece, cut at the speaker's short pauses, and a
model reads each piece as a sentence of its own: "I went. To the store."
When the recording ends, one pass over the whole clip gives the final
text, punctuated by meaning, in dictation and in voice mode alike. The
pieces' text stays the live text until then, and is kept when that pass
fails or the recording is longer than two minutes.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 359dd9632e6c88000a1d23c2575bd1e6fcdd9ec1)
* ui: give live text over HTTP for models without a realtime session
A model that hears but cannot hold a realtime session, such as
Qwen3-ASR, was transcribed only after the recording, so dictation
showed no text while speaking. The recording is now cut at the
speaker's pauses as with a realtime session, and each piece sent to
/v1/audio/transcriptions on its own, one at a time, its text joined
live. The audio is kept from the start, so what is said while the
socket answers goes with the first piece. The whole clip is still
transcribed once more at the end, and the pieces' text is kept if
that fails. A server a websocket cannot reach (an API key, a remote
server) gets live text the same way.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say when the browser processes the microphone despite the raw request
The browser keeps one processing setting per microphone: while a call,
another tab or voice mode has it open with echo cancellation, a
dictation that asked for the raw signal is processed too, sounds
distant, and the system may lower other sound. The capture now warns
when it gets processing it did not ask for, and its development log
names the microphone, its rate and what processing is on.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the microphone beside the stop and send buttons
While a reply streams, or once the message has text, the microphone no longer
gives way to the main button: it stays beside it as a quieter, secondary
button, so more can be dictated without clearing the message or waiting for
the reply.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show the voice components' stories with Llama UI
VoiceWaveform and VoiceTranscriptReveal are Llama UI components, so their
stories move to the library's Storybook, under Llama UI/Voice, with the demo
audio they play. Llama Chat's Storybook shows them there too, and its own
voice stories take the demo audio from the library. VoiceWaveformLive stays
with Llama Chat, as its demo records through the chat form's microphone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a noise suppression setting for voice input
Recordings stay raw by default. Settings > Voice can now switch the
browser's noise suppression on; echo cancellation and automatic gain stay
off either way, as they interrupt other sound and fade the first words in.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep live voice text coming when the room is never quiet
Pieces were cut only at a quiet moment, so with music or talk in the room
none was ever cut and no text showed until the recording ended. A piece half
as long again as its longest length is now cut where it is.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add echo cancellation and automatic gain settings for voice input
Settings > Voice now has a switch for each kind of microphone processing the
browser offers. All three stay off by default, so recordings are raw; each
one says what switching it on costs.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: put the microphone after the stop button, filled like it
While a reply streams, the microphone is the last button of the row and
looks like the stop and skip buttons beside it, instead of a bare icon
before them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say when dictated text is being refined, and send it refined
After a recording with live text, the whole clip is transcribed once more
so pauses do not end sentences. The form now says so while that pass is
out, and a send asked for meanwhile waits for it, so the message goes out
with the refined text rather than the pieces.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the send button in place and put the microphone beside it
The send button no longer gives way to the microphone on an empty
message. The microphone stands before it as a quieter, tertiary button,
with the same tooltip, warning and recording states as before.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let a recording be sent while a reply streams, once it has words
While a reply streams, the recording row now shows its send button as
soon as the first transcribed words are in the message, so the
recording no longer has to be stopped first. The microphone also stays
beside the send button once a voice message is attached.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: raise quiet speech for transcription, and say when it is too quiet
A quiet microphone gave the model near silence to transcribe. A new
Voice setting, Boost quiet speech (on by default), raises speech to a
level speech models hear well, up to eight times louder: the audio
streamed for live text as it goes, and the whole recording in one pass
when it stops, so its first words get the same gain as the rest. Only
speech is measured, never the silence around it, loud speech is left
as it is, and a soft limiter keeps raised peaks from clipping. When
speech stays quiet even at the highest gain, the recording row says so.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the voice engine free of app code
The engine in lib/voice imported the microphone messages from the app's
utils and the WAV type from its enums, the only two things keeping it
from moving into a package of its own. The microphone checks and
messages move into the engine (voice/support.ts), next to the capture
that needs them; the uploaded-file helpers stay in the app's utils.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: start voice sessions from one voice service
How a recording reaches the server (routes, API key, whether the
realtime socket can be used) and how the Voice settings become capture
and transcribe options were built inside the chat form's voice hook,
and copied into every other feature that records. VoiceService builds
them in one place: a feature asks it for its session options and adds
only its own callbacks.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tell speech from the room in one place
The pause detector, the speech gain and (in voice mode) the listener for
speech over a reply each followed a noise floor with their own numbers.
A SpeechDetector now does it for all of them, in dB, with one set of
settings in VOICE_SPEECH.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let any text field take dictation
Writing dictated text into a field (at the caret, around the user's
typing, replaced by the whole recording's text when it comes) moves out
of the chat form's hook into a plain createDictation(). The hook keeps
what is the chat form's own: voice messages, toasts and the hint.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move the recording row into Llama UI
VoiceRecordingBar draws the row shown while recording (cancel, waveform
and time, send, and a caption or warning under it) from data alone, so
voice mode or a notes page can show the same row. The chat form passes
its labels and its microphone button as the stop button.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the voice engine's Svelte recorder behind its own entry
VoiceRecorder, the one part of lib/voice written with runes, moves to
lib/voice/svelte, so the rest of the engine is plain TypeScript any app
can use without Svelte.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move voice input into its own package, Llama Voice
The voice engine (capture, the speech signal, transcription over the
realtime socket or HTTP) and dictation into a text field move from the
app to ui/lib/voice as @ggml-org/llama-voice, so another app, such as a
notes app, can use them. The Svelte recorder is its ./svelte entry.
Llama Chat compiles the sources, as it does Llama UI's, and keeps its
server and settings wiring. The package has its own type check, tests
and build, and a CI job for them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show which model does what in voice input
The model picker and the Models manager mark each model that hears:
Dictation on the one that transcribes dictation, Speech to text on
transcription models such as Qwen3-ASR, and Voice on chat models that
take voice messages. A loaded model kept only for dictation sits apart
from the chat models, a model can be made the dictation model from its
row, and Settings > Voice groups the transcription models apart from
the chat models with audio input. A voice store now resolves what the
microphone does, for the chat form and the model lists alike.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: warn of a quiet microphone inside the recording row
The too-quiet line under the waveform made the form grow and jump. The
warning now stays inside the row: the waveform turns the warning colour,
an icon sits by the time, and the advice opens above the row in a
tooltip for a few seconds, then on hover or focus. Screen readers still
hear it when it appears.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the microphone while the pointer is on its button
The first seconds of a recording could come out quiet while the
microphone settled, and the too-quiet warning showed for them. With the
pointer or the focus on the microphone button, the microphone now opens
ahead of the press (only when the page may already use it, so nothing
asks), and the recording takes it at once; it is let go shortly after
the pointer leaves without a press. The warning also waits until the
speech has stayed too quiet for two seconds.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tidy voice input after review
Fix a microphone left open when a recording is cancelled while it starts
or the pointer leaves while it opens, a capture that is still letting go
silencing the next one, and live text stopping after a realtime error.
Share the dictation model choice, the transcript lookup, base64 decoding,
PCM joining and stopping a stream instead of repeating them, and export
only Llama Voice's public API.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin Llama Voice's dev dependencies to Llama UI's versions
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let demos speak into a synthetic microphone
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the microphone for voice input, off the audio files entry
The attach menu's Audio Files entry used a microphone, which now sits
beside the form's own microphone button. The microphone stands for
recording and dictation only; a model's audio modality is headphones, and
the audio files entry is a file with headphones.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: set voice input on the slimmed design tokens
Voice input's components use the type roles, the one corner scale and
shadcn's utility names for the neutral colours, as the rest of Llama UI
and Llama Chat do since the design tokens were slimmed down.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name voice states, platform strings, patterns and tunings
Addresses the review on the voice PR:
- Llama Voice: VoiceCaptureState, VoiceRecorderStatus, RealtimeState and
RealtimeUnsupportedReason are enums (src/enums.ts, exported); DOMException
names, AudioContext states, transcription form fields, socket protocols,
device kinds, worklet messages, error messages, regexes and tuning values
are named constants.
- Llama UI: VoiceTranscriptStatus, VoiceWaveformSize and
VoiceRecordingBarStatus are enums; KeyboardKey, MouseButton and PointerType
join its UI enums; the voice components' sizes, labels, timings, selectors
and patterns move to constants/voice.constants.ts. The voice unit tests
move from src to tests/unit.
- App: ChatFormVoiceState is an enum, the recording bar's status comes from a
map, and the voice code uses the enums above, BooleanString, HEADERS and
new voice constants in place of inline strings, regexes and numbers.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep all of Llama UI's tests under tests
The Model UI formatting spec moves to tests/unit and the button spec to
tests/client, next to the voice specs; the test projects now read only
tests/unit and tests/client.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a transcript's words without the model's own format
Qwen3-ASR names the language before its text (language French<asr_text>...) and
VibeVoice-ASR answers with timed segments as JSON, which the server passes on as they
are. The voice engine now takes both off, over HTTP and over the realtime socket, also
while segments still stream, so the message box and the clip show the words alone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: dictate when a speech-to-text model is picked for the conversation
A speech-to-text model has audio input, so on Automatic the microphone recorded a voice
message for it, which such a model has nothing to answer with. It now dictates, as the
model's badge already says it is for.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say why a live-only model cannot transcribe behind an API key
A browser websocket cannot send the API key, so the realtime socket is left out on a
server that needs one, and a model that only transcribes over it (Voxtral Realtime)
answers the HTTP route with a 501. The clip and the dictation toast now say so instead of
only that transcription is unavailable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pick the dictation model with a microphone on its row
The model that transcribes dictation was marked with a filled Dictation badge. It is now
marked by a red microphone among the row's actions in the model picker, beside the heart,
and every other model that hears shows the same microphone on hover: a click makes it the
transcription model, and a click on the chosen one hands the choice back to Automatic. The
badge keeps saying what kind of hearing model a row is, and the Models manager marks the
dictation model with a small microphone beside its name.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: search message content and Discover models from the search dialog
The search dialog matched chat titles only. It now also scans what was
said in each chat (content, reasoning, tool calls and results, attachment
names and text), newest chat first, stopping at the first match per chat
and after 20 chats, and shows a snippet around the match. Ids, timestamps
and base64 payloads are not searched.
With Discover on, it also lists matching catalog models and, once typing
settles, Hugging Face repos; picking one opens Discover on that repo.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the search dialog one width whatever it lists
The shared dialog sizes itself to its content from md up, so the search
was narrow while empty and widened as results came in. It now takes the
full max-w-2xl there, less a margin on narrower windows.
Assisted-by: Claude (Opus 5.5 Medium)
The unscoped llama-ui on npm is a 0.0.1 placeholder. After the scoped
publish, the workflow renames the same build and publishes it again at
the same version, with a readme note pointing at the scoped package.
Assisted-by: Claude (Opus 5.5 Medium)
llama-tts built the mtmd context from mtmd_context_params_default(), which hardcodes 4 threads, and never copied n_threads over, so -t never reach the mtmd side.
Copy params.cpuparams.n_threads into mtmd_params.n_threads, as mtmd-cli and the server do; with no -t this now follows the resolved default.
Verified: -t 1 now runs single-threaded, -t 4 is unchanged, and the produced WAV is identical.
Assisted-by: Claude
* fix(server-bench): prevent float bin-count TypeError in histogram generation
Symptom: benchmark crashes while generating gen_rate histogram.
Root cause: np.ceil(...) returns numpy.float64, and that value was used where an integer count/shape argument is required.
Fix: cast ceil result to int before +1, ensuring bin count is an integer.
Result: histogram path completes and gen_rate.png is generated reliably.
* Update scripts/server-bench.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ui: load highlight.js on first use
highlight.js and every lowlight language (~900 KB raw) moved out of
Llama Chat's first load into a chunk that loads with the first code block.
- @ggml-org/llama-ui/code: highlightCode stays synchronous. Until
highlight.js has loaded it returns the code escaped as plain text and
starts the load; a $derived or $effect that called it runs again once
it has loaded. New loadHighlighter() and isHighlighterLoaded().
- Markdown: rehype-highlight runs through a lazy rehype step, like
KaTeX's: the first message with a code block waits for it, every
other one renders in the same pass.
First-load JS of the chat page: 3.47 -> 2.56 MB raw, 1.05 -> 0.75 MB
gzip, 0.84 -> 0.60 MB brotli.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load each highlight.js language on first use
highlight.js now loads one language at a time instead of all ~190 at
once with the first code block: every language is its own chunk, and
the core loads with the first one.
- @ggml-org/llama-ui/code: highlightCode looks a language or alias up
in a generated table (scripts/generate-highlight-languages.mjs), loads
it with the languages it embeds (CSS and JavaScript inside HTML, say)
and returns plain text until then; a $derived or $effect that called
it runs again once it has loaded. Auto-detection picks among lowlight's
37 common languages, loaded the first time something auto-detects.
New loadHighlightLanguages(), isHighlightLanguageLoaded(),
resolveHighlightLanguage() and getLoadedHighlightLanguages() replace
loadHighlighter() and isHighlighterLoaded().
- Markdown: rehypeLazyHighlight highlights with lowlight's core and the
languages a message uses, waiting only for one it is the first to use.
rehype-highlight is no longer in the pipeline, since it bundles
lowlight's common languages.
First-load JS of the chat page stays at 2.58 MB raw, 0.76 MB gzip,
0.61 MB brotli: no highlight.js code, only the 14 KB language table.
The first code block now loads its language (1-12 KB for most) and the
20 KB core instead of 900 KB.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the code-block class names and shell language as constants
Assisted-by: Claude (Opus 5.5 Medium)
A printable key pressed on the chat screen while no field, editor,
dialog or menu has focus now moves focus to the chat form, and the
character lands there.
Assisted-by: Claude (Opus 5.5 Medium)
* model: load `ffn_gate_up_exps` scales
The generic scale pass loads per-expert scales for ffn_gate_exps,
ffn_up_exps and ffn_down_exps but not for the merged ffn_gate_up_exps.
Load ~ffn_gate_up_exps_s there~ and expose it as ffn_up_exps_s ~when the layer
has no separate up tensor,~ since build_moe_ffn applies up_exps_s to the
merged gate_up output. Inkling 2 now passes the expert scales to
build_moe_ffn like the other MoE models.
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-model.h
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama : move the indexer key rotation choice into the model classes
The KV cache forced a Hadamard rotation for the lightning indexer keys
based on a list of architectures. The virtual llama_model::indexer_k_rot()
now returns true for the same models. DeepSeek4.1 inherits from DeepSeek4
but was not in the list, so it overrides it back to false.
* llama : rename indexer_k_rot() to lid_k_hdmd()
* ui: drop component tokens that only alias a semantic token
A component token now exists only for what a component decides itself:
a value no semantic token has, or one that only this component changes
between themes. Components that take a semantic token as it is read it
directly, so 88 of the 118 component tokens go, with no change to what
is drawn.
Also drops semantic tokens that repeat another one (status-* is fill-*,
elevation-tooltip is elevation-surface), the unused info lamp tokens and
14 primitives nothing reads, and adds fg-soft for the label of an option
that is not the picked one (tabs, toggle group).
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a short summary on top of DESIGN.md
The rules most changes need, in one screen: components, colour, type,
shape, sizes, surfaces, tokens and no arbitrary values. The sections
below keep the detail.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: derive design tokens by rule and drop the alias layers
Semantic tokens are either a pick (a primitive per theme) or a derivation,
mixed from another semantic token by a fixed rule and declared once: hover
fills, status surfaces and lines, dark shadows (--elevation-strength), the
middle of a matte edge, a lamp's rim and glyph.
Primitives keep colours only. shadcn's colour names stay as Tailwind
utilities and no longer exist as variables; neutral semantic utilities that
duplicated them are gone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: set all text through type roles and move off-scale values onto the scale
Tailwind's text sizes and bracketed sizes become the type roles, with a new
text-caption-sm (10px) for tags and counts. Body and caption roles no longer
set a weight, and line heights are written as Tailwind writes its own, so the
swap from text-sm and text-xs changes no pixel. A unit test keeps old sizes
from coming back.
Arbitrary values with a scale equivalent use it (z-20, max-w-4/5, min-w-32),
and the two easing curves repeated across the app are ease-out-quint and
ease-sheet. DESIGN.md is updated and regenerated.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: bump @ggml-org/llama-ui to 0.5.0
Removed and renamed tokens are breaking for an app that reads them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a display type role for the biggest headline
The greeting on a new chat keeps its 30px from md up, now as text-display
instead of a Tailwind size.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: one corner scale, one set of shadows and three more derived tokens
Six corner roles named values Tailwind's rounded-md to rounded-4xl
already have, and components used the Tailwind names nine times out of
ten. The roles are gone; the whole scale is computed from --radius.
Tailwind's shadow sizes were restated next to the elevation tokens and
did not follow the theme, so they were five times fainter in dark. The
five uses take elevation tokens and a unit test keeps both from coming
back.
A button group's seam is the button's own text colour at 25%, the dark
status text is the dark status fill, and the pressed shadow scales with
--elevation-strength like the inset one.
Assisted-by: Claude (Opus 5.5 Medium)
The model modality icons match the ones the Llama apps use: glasses for
vision, headphones for audio. The attach menu's list of what the model
takes reads the same icons instead of its own microphone and picture.
Assisted-by: Claude (Opus 5.5 Medium)
graph_max_nodes() picked the nodes-per-tensor factor from lists of
architectures. Each model class now gives it through a new virtual,
llama_model::graph_nodes_per_tensor(), which defaults to 8. The
architectures that need 32 or 64 override it in models.h, and DFlash
keeps its hparams-based rule in dflash.cpp.
LoRA adapter nodes are now added for every architecture, not only for
the ones on the default factor.
* ui: move the reasoning effort into the model picker
The chat form's row no longer holds the reasoning button: the model
picker ends with a Reasoning row, in place of Manage models, that shows
the level in use and opens the levels under it. A phone's model drawer
has the same row. A desktop server with a single model, whose selector
opens the manager instead of a picker, keeps the button beside it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a reasoning effort that is not the default on the model selector
Next to the model on the closed selector, a set level shows as its word,
and reasoning that is off as a small crossed-out lightbulb. At the
default, which leaves it to the model, nothing more shows.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the model picker's footer in place while it opens in Safari
WebKit pins a bottom-sticky element against the scaled box of a parent
that zooms in, so the Reasoning row rode up over the model list until
the opening animation ended. The desktop picker now scrolls its search
and list in a box of their own, with the footer under it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep Pele diagrams inert
Pele turns a node with `click ... href` into a live link, where Mermaid's
strict security level left diagrams without links. Unwrap every link in
Pele's output, keeping its children, so the inline block, the live-preview
editor, the preview dialog and the downloaded svg stay inert. The Diagrams
story's flowchart now carries click directives and its play test checks
that no link or href is left.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open diagrams full screen, zoom around what is on screen
The diagram preview dialog fills the window on every device. The preview
places the diagram from its top left corner, so a zoom keeps a screen point
fixed: the wheel and a trackpad pinch zoom around the pointer, the buttons
and +/- keys around the middle of the screen, wherever the diagram has been
panned to. Two-finger pinch, double-click and arrow keys work too. The
diagram opens fitted and keeps fitting until panned or zoomed; Reset view
fits it again.
Adds MermaidPreview and DialogMermaidPreview stories with play tests and a
unit test for the zoom math.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a diagrams trace to the demo fixtures
Assisted-by: Claude (Opus 5.5 Medium)
The MoE cache adds a slot lookup to every MoE layer, and with
--moe-cache-cpu-share a second expert path, but graph_max_nodes() did
not account for it. Models on the default 8*n_tensors budget with few
tensors per layer (cohere2moe: 443 tensors, 49 layers) overflow the
graph context when the cache is combined with a quantized KV cache
(K/V rotation), aborting in ggml_new_tensor with GGML_ASSERT(obj_new).
Fixes#32584
Assisted-by: Claude Opus 5.5
* tests: add CUDA-only MUL_MAT/MUL_MAT_ID cases with results above the fp16 range
IQ1_M weights have no MMQ kernel, so once the batch is bigger than the MMVQ
limit the path is dequantize + cuBLAS. These cases produce results around
1.3e5, past the fp16 max of 65504, so an fp16 result buffer turns them into inf
while the CPU reference stays finite. They are only added for the CUDA backend.
Vulkan deliberately accumulates quantized matmuls in fp16 at default precision
and would fail them by design.
* CUDA: use an fp32 result for quantized src0 in the cuBLAS fallback
On the F16 compute path, quantized weights went through cuBLAS with fp16
accumulation and an fp16 result buffer that was converted to fp32 afterwards.
Anything above 65504 became inf and the next layer turned into NaN (see #30951:
gpt-oss-120b with IQ1_M experts at -ub 512). Set prefer_f32_output for
quantized src0 so the GEMM still uses fp16 operands but accumulates and stores
in fp32, the same mode already used on Volta, RDNA4 and CDNA. Every other
quantized path already accumulates in fp32.
Diagnosis and fix by denizyuret in #30951.
* CUDA: keep the MUL_MAT_ID precision in the per-expert fallback
The per-expert dst_slice kept op = GGML_OP_NONE, so ggml_prec_get_acc returned UNDEFINED and a requested precision was ignored. Set the op to GGML_OP_MUL_MAT. Also revert the fp32 result change and remove the tests.
---------
Co-authored-by: JonSnow1807 <JonSnow1807@users.noreply.github.com>
On Adreno, subgroupAdd() after divergent code returns wrong sums. The
fused ADD+RMS_NORM partial-sum path hits this, so the decode-step norm
is garbage and models emit stray tokens after long prompts.
Clear subgroup_arithmetic for the Qualcomm proprietary driver, which
also turns off add_rms_fusion and the other subgroupAdd paths.
test-backend-ops on Adreno 850 and X2-90: ADD_RMS_NORM 20/25 -> 25/25,
RMS_NORM_MUL_ADD 33/45 -> 45/45. llama 3B Q4_0 on Adreno 850: pp512
and tg128 unchanged.
load_arch_hparams multiplies n_hrm_layers_per_stack, n_hrm_h_cycles and
(n_hrm_l_cycles + 1) into a uint32_t, then asserts the product equals
block_count.
Any factors whose true product is block_count + k*2^32 still pass the
assert; 1073741825 * 2 * 2 = 2^32 + 4 compares equal to block_count = 4.
load_arch_tensors iterates the same three values in int and writes past
the end of the block_count-sized layers vector.
* opencl: support int8 KQ for DK = 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
* opencl: allow query chunk up to 96 for int8 KQ
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
* opencl: extend `build_program_from_source` to allow non-fatal failure
Assisted-by: Codex
* opencl: add int8 KQ for decomposed FA with DK <= 256
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA)
* cuda : SET_ROWS from BF16 into quantized types and F8_E4M3
* cuda : fuse BF16 rope into a BF16 cache
* tests : trim the BF16 rope cases to one per code path
- Allow cls.output.weight to be absent from the mmproj. It is an optional
Qwen3 reranking head that the YuE2 NAR decoder never uses.
- Without this fix, mtmd_init_from_file() fails with
"tensor cls.output.weight is missing from the mmproj", preventing
audio generation from starting.
* ggml-openvino: round the GPU KV length up to a power of two
Review notes (follow-ups, not blocking):
- convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
copies it into the ov::Tensor. Writing the padded rows straight into
padded.data() avoids the second allocation and copy (up to 16 MB per
call at a 16K context).
- The padding is skipped when the mask is not contiguous, and the KV
inputs are then sliced to more rows than the mask has columns. Masks
are always contiguous today; a GGML_ASSERT would make that explicit.
- compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
is_stateful_enabled() is local to ggml-openvino.cpp.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Optimize model recompiles of prefill stage
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
* ggml-openvino: Remove solution C and keep A as default
* ggml-openvino: match the output count before reusing a compiled NPU prefill
The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.
* ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding
* ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix
- compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
ran before the new one and still exported result_norm, so the
result_norm exclusion had no effect. Keep only the new block.
- compute_node_dynamic_dims checked is_activation_input twice in a row.
Remove the second copy.
- process_view_input_new created the dynamic Split on split_dim. With
stateful execution the input drops the leading size-1 axis, so use
split_axis, which already accounts for the rank difference.
* ggml-openvino: hash the output flags into the model-cache fingerprint
The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.
* ggml-openvino: support f16 SCALE and f16/bf16 TRI
test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.
SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.
* ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
layer
The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.
* enable mamba based models on OV CPU-GPU
* revert changes in tests
* docs : add Falcon-H1 to the OpenVINO validated models
Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.
* ggml-openvino: simplify the SSM_SCAN build check and clean up
- Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
always provides the op. Drop the no-op LANGUAGE CXX source property.
- Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
with -Wextra-semi, which fails builds with fatal warnings.
- Add ggml_openvino_is_static() for the repeated NPU or
GGML_OPENVINO_FORCE_STATIC check.
- Share the f16 and f32 KQ mask padding code.
- Drop the redundant inline on in-class static functions.
---------
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
* cuda: add scatter for batched tensor writes
* update test case to mirror get_n test case
* save restore timing logs
* refactor code into batch-copy.cu file
* Adds initial scheduler stress tests
* Adds new sleep op to simplify scheduler testing
* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321
* Adds experimental vulkan GGML_OP_SLEEP
* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.
* Cleanup: Removes CUDA specific comments and nvtx-markers
* Fix: forgot to add sleep specific test case
* Fix oversights identified by CI
* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior
* Fix: Early exit in CPU-only settings (cause for graviton CI fails)
* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds
* Exclude ACCEL devices from new tests due to incompatibility
* Fixes vulkan test failure in llvmpipe
* Adds asserts, improves comments and some checks
* Fixes nitpicks in white-space and variable naming
* Adds comments that certain vk drivers could time OP_SLEEP incorrectly
* Drops limitiation to CPU/GPU in test-sleep
* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)
* Implements feedback for OP_SLEEP to work in place (no copy)
* Apply non-code suggestions from JohannesGaesslers code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail
* Make clang-tidy happy
* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)
* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name
* Drops unnecessary shader_clock.comp test and guards
* Replaces sched_ prefixes with helper_ prefix
* Moves test suite to new RAII types for better memory management
* Adds sleep option to test_chain_all_backends
* Adds sleep option to first-run arm of the y-shaped test
* Fix switchcase order for legibility in stress_test_dag
* Simplfies stress_test_linked_list via double for-loop
* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)
* Fix RPC Proto patch due to GGML_OPS bump
* Adds ggml-cpp.h treatment to test-sleep.cpp
* Adds TODO pointers for subsequent additional tests.
* Apply suggestion to ggml.h
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Apply suggestion to tests/test-backend-sched.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)
* Apply batched suggestions from gg code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Completes callsite change of gg-suggestion
* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : give Inkling2 its own relative-position attention input
The Inkling2 graph built its flash attention mask from the hybrid iswa
input and passed it to build_attn through a new kq_mask argument. That
argument let a caller pair its own dense mask with the kv_rows of
llm_graph_input_attn_kv_iswa, whose mask follows the per-sequence cell
lists whenever kv_rows is enabled.
Add llm_graph_input_attn_rel_pos instead. It owns dense KQ masks and the
query-to-cell distances, fills both from the same cells, and its
build_attn overload adds the per-head bias to its own mask and never
passes kv_rows. The iswa build_attn is back to its previous signature;
its body moves to build_attn_iswa_impl, which both overloads call.
Inkling2 now folds the per-query sink into the bias with and without
flash attention, so build_attn_mha no longer needs the per-query sink
path.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* inkling2 : make the expert weight scale and final logit softcap optional
Load both keys as optional and skip them when they are zero, as other
models do, instead of rejecting zero at load.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : build kv_rows masks for the hybrid iswa memory
llm_graph_input_mem_hybrid_iswa checked its masks with
can_reuse_attn_mask, which expects kv_rows whenever they are enabled,
but built them dense. With a unified cache and flash attention the
check always failed, and hybrid iswa models (LFM2, LFM3, Inkling2)
never reused a graph. Build the masks with build_attn_inp_attn_mask,
like the plain iswa input.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : make the relative-position input a hybrid iswa input
Address review: llm_graph_input_mem_hybrid_iswa_rel derives from
llm_graph_input_mem_hybrid_iswa and owns the attention and recurrent
inputs, so Inkling2 builds one memory input instead of two. Its
attention input is built with dense masks, can_reuse checks them as
dense, and its build_attn asserts that the mask has no kv_rows (TODO:
a kv_rows variant). build_attn_iswa_impl is now a static helper.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : compose the hybrid iswa rel input, format inkling2
Address review: llm_graph_input_mem_hybrid_iswa_rel owns its attention
and recurrent inputs like the other composite inputs instead of deriving
from llm_graph_input_mem_hybrid_iswa.
Format src/models/inkling2.cpp like the other model sources: grouped
and aligned key and tensor loading, descriptive per-layer names, blank
lines and section comments between the steps, and cb() names for the
intermediate tensors. The graph is unchanged.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
llm_graph_params no longer carries hparams, prec_policy and hdmd, since the model reference already provides them. llm_graph_context sources them from the model and keeps prec_policy as a reference.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal: add a tensor API conv_2d kernel and split long reductions
The conv_2d matmul kernel runs on simdgroup matrices only, and gives
each threadgroup the whole reduction of its output tile. On M5 this
leaves the tensor cores unused, and convolutions with few output
positions and a long reduction (IC * KW * KH in the thousands) run a
handful of threadgroups on an otherwise idle GPU.
Add kernel_conv_2d_mm_tensor for devices with the tensor API: like
kernel_mul_mm, it gathers the patches of 64 output positions into
threadgroup memory as the left operand and reads the kernel rows
from device memory as the right operand. Devices without the tensor
API keep the simdgroup kernel.
Both kernels can now split the reduction until there are enough
threadgroups: each split writes its partial sums to a buffer placed
after the output, as flash attention does, and a small kernel adds
them in a fixed order, so the result stays deterministic. CONV_2D is
marked as an op whose allocation may exceed its output size.
test-backend-ops -o CONV_2D passes, with and without
GGML_METAL_TENSOR_DISABLE.
CONV_2D, F16 kernel, us/run on an M5 Max:
[19,19,256,16] k [4,4,256,4096] 19008 -> 5235
[16,16,128,8] k [3,3,128,512] 333 -> 124
[1536,1536,32,1] k [3,3,32,64] 12086 -> 6512
[33,1,1280,1] k [3,1,1280,1280] s2 659 -> 33
[2054,1,768,1] k [7,1,768,768] 3161 -> 699
* metal: walk the conv_2d patches incrementally, size splits by output
The patch gather of both conv_2d matmul kernels spent most of its
time on address math: 64-bit multiplies, divisions and four bound
checks per value. Walk the reduction index with 32-bit byte offsets
inside the image, updated by additions, with one unsigned compare per
bound; the host checks that an image fits. The tensor kernel no
longer gathers the reduction indices past the end of its split.
The number of splits now targets about 1M partial sums, with at least
3 reduction steps per split, instead of counting the tiles of one
kernel.
CONV_2D, F16 kernel, us/run on an M5 Max:
[19,19,256,16] k [4,4,256,4096] 4998 -> 4512
[1536,1536,32,1] k [3,3,32,64] 6721 -> 5354
[4096,1,1024,1] k [7,1,1024,1024] 2701 -> 2236
[2054,1,768,1] k [7,1,768,768] 1050 -> 871
with GGML_METAL_TENSOR_DISABLE:
[4096,1,1024,1] k [7,1,1024,1024] 9047 -> 8283
* nit: share the CONV_2D simdgroup count between host and kernel
kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per
threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel
constant and res.nsg, so the dispatch and the kernel cannot drift apart.
* metal: fall back to the generic conv_2d kernel for images past 4 GiB
The matmul conv_2d kernels walk the patches with 32-bit byte offsets
inside an image. Instead of asserting on nb13, compute the largest
offset actually read from ne/nb, which also covers strided views of
the input, and route images that do not fit to the generic kernel.
* cont : comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ui: import Lucide icons by path in modules that run in Node
Assisted-by: Claude (Opus 5.5 Medium)
* ui: register highlight.js languages once, from lowlight's modules
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load the pdf.js worker and nerdamer as files of their own
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw Mermaid diagrams with Pele
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin dependency versions and use one Svelte across the UI packages
Assisted-by: Claude (Opus 5.5 Medium)
* ui: build Llama Chat as split chunks
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load KaTeX and pdf.js on first use
Assisted-by: Claude (Opus 5.5 Medium)
* llama : fall back to buffered I/O on EIO
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Preserve bytes already read, resumes at the current file offset
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* vulkan: Implement stream-k MUL_MAT
Stream-k tries to split chunks of 256 K elements evenly across all SMs.
Workgroups that finish a tile write to dst, others write to scratch, and
there's a resolve shader similar to split-k.
The shader code is implemented for scalar+cm1+cm2, but only currently enabled
for cm2. The heuristic for enabling on coopmat2 is pretty simple due to the
large tile size (one workgroup per SM), but would be more complex for the tile
sizes being used for cm1, and it doesn't make sense for me to tune that on
NVIDIA devices.
Assisted-by: Codex
* address review feedback
The MMQ kernels computed the dst tile offset (jt*J*stride_col_dst) in
32-bit int. For a dst with more than 2^31 elements, column tiles that
start past INT_MAX/nrows wrap around and write out of bounds.
This happens with an all-logits output head, e.g. llama-perplexity
-c 32768 -b 32768 -ub 16384 on a 248320-vocab model: the q8_0 output
matmul has a 248320 x 16384 dst (4.07e9 elements) and aborts with an
illegal memory access.
Widen the offsets in mul_mat_q and the stream-k fixup to int64_t, and
add a (disabled by default, ~9 GB) test-backend-ops case.
Assisted-by: Claude Opus 5.5
n_mudd_taps comes from limite.mudd.tap_count, a scalar read separately
from the length-checked tap_indices array. mudd_tap_idx is a fixed
std::array<int32_t, 512> indexed at [i*n_taps] in both the tensor load
and the graph build, so a crafted GGUF with n_layer * n_mudd_taps > 512
reads past the array.
Reject that at hparams load; the one check covers both index sites.
* llama-bench : size the batch to the prompt for models without memory
Assisted-by: Claude Opus 5.5
* server : read up to 16 lfm3-d1 questions in one pass
Assisted-by: Claude Opus 5.5
* cuda: select cuBLAS precision by GPU SKU
Assisted-by: Codex
* cuda: fix remaining GB10 macro reference
* cuda: review comment addressed, store accumulation preference in device info
* Apply batched suggestions from code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* addressed Oliver's comment regarding initialize NVML once and install headers in Windows CI
* move precision selection helper into init
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* model:add Zaya1 8B (Zyphra)
- Added the support for Zaya.
- Fixed the SSM_CONV case, so the model runs on cuda without abort.
Please enter the commit message for your changes. Lines starting
* Made the following changes based on the bot reviewer feedback:
1) Added runtime asserts when loading wrong KV head size, unsupported conv kernel size.
2) Removed the static cast usage.
3) Fixed double load of tokenizer.
4) Updated res_scale_in to be input tensor.
* - Moved the res_scale_in back to layer 0 to fix the cuda bug
* i) Removed the EOS addition.Team agreed that updating Tokenizer is the right move.
ii) Fixed formatting
* Fixed the comment to make it clear
---------
Co-authored-by: varun <varun_subramanian@zyphra.com>
* ui: install @ggml-org/llama-brand from npm
Both packages take llama-brand 0.1.0 from the registry instead of a
codeload tarball, so npm 12 installs them without allow-remote, which
leaves Llama Chat's .npmrc.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin @ggml-org/llama-brand to 0.1.0
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat publish waits for the registry, bin paths without ./
The first publish of @ggml-org/llama-chat took about four minutes to show
up on the registry, so the 60 seconds the check after publishing waited
failed a release that had gone through. It now waits up to ten minutes.
npm normalizes "./bin/cli.js" to "bin/cli.js" on publish and warns that
the bin was "invalid and removed", which reads as if the command was
dropped. Both package.json files now give the paths as npm writes them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the registry check after llama-chat publishes
npm publish already fails when the registry refuses a package, and the
check after it only failed a release that had gone through, while npm
was still processing it. Nothing waits on the registry any more.
Assisted-by: Claude (Opus 5.5 Medium)
The tool permission card took the fields' border (white 30% in dark), which
outweighed everything else on the page; it now has its own token, white 18%
in dark, the focused chat form's edge. The split-button divider had the
keys' own rim colour (white 12% over a white 9% fill) and vanished between
them; it now has its own token, white 20% in dark. Light mode is unchanged.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide Models and MCP behind settings, off by default
Two new settings in General, "Show Models" and "Show MCP", both off by default, like "Enable Discover Models". The WebUI is often the first thing people new to local models see, and most of them don't need a models manager or know what MCP is.
"Show MCP" only hides the way to manage MCP; servers already added keep working. Turning MCP off for real would mean gating every place that reads the server list (health checks, tool groups, prompts, avatars). Anyone who already has servers saved gets MCP shown, so they don't lose the way to manage them.
With Models hidden, every entry point to the manager goes: the sidebar, search, the shortcut and "Manage models" in the selector -- it opens the same full manager. The /models page still opens by its URL, which apps link to. On a single-model server the selector chip used to open the manager; with it hidden, the chip is a plain label.
* ui: keep Models shown while Discover Models is on
Discover opens only from the Models page (its header button and the manager's row actions), so with Models hidden by default, checking Discover showed nothing. Checking Discover now shows Models too and locks its checkbox, the same way it already locks the Hugging Face Hub API one; unchecking Discover leaves Models as it is. Saved configs with Discover on get Models turned on when they load, so people who use Discover don't lose it.
* ui: run Llama Chat with npx @ggml-org/llama-chat
Makes @ggml-org/llama-chat publishable: drops private, adds bin,
files, exports, repository and a prepack build. Its CLI serves the
prebuilt dist on localhost (3000 by default) and forwards every other
request to a llama-server (--server, default http://127.0.0.1:8080),
with the headers llama-server sends for the UI. A thin unscoped
llama-chat package in alias/ forwards to it, so npx llama-chat works.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat revalidates service-worker.js and asks for a build before packing
SvelteKit 3 adds service-worker.js next to sw.js, and llama-server
revalidates it, so the CLI does too. The folder's .npmrc sets
ignore-scripts, so prepack never ran: it goes, and the publishing steps
build dist explicitly and check the packed file list.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat keeps llama-server's DNS rebinding and cross-origin guards
The CLI called llama-server with its own Host and dropped every Origin,
so a rebound host name or another page got past the checks llama-server
makes when --cors-origins is not *. It now drops only its own page's
Origin, passes other Origins and Sec-Fetch-Site through, and, when it
listens on loopback, answers only to localhost and loopback addresses.
`start` on Windows gets its empty title verbatim.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat build writes the app to a folder for any static server
`npx @ggml-org/llama-chat build [--out <dir>] [--force]` copies the
prebuilt app, without source maps, to ./llama-chat or the given folder,
so a plain file server can host it next to the CLI serving it. It
refuses a folder that is not empty unless forced, and the serving flags
and build flags refuse each other's command. The README shows serving
the folder with http-server forwarding the API to llama-server.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: publish Llama Chat to npm from CI when its version is bumped
ui-chat-publish.yml publishes @ggml-org/llama-chat and its llama-chat
alias the way ui-lib-publish.yml publishes @ggml-org/llama-ui: the
version is set by hand, and a commit that reaches master with a version
npm does not have builds dist, checks the packed files and publishes both
with trusted publishing. PRs that change the packages get a dry run.
npm-version-check.mjs takes the package folder, so both workflows share
it. ui/README.md says how releases work, and Chat's AGENTS.md says not
to bump versions unless asked to release.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: call the app Llama Chat in its README
The README's title and a few lines still called the app llama-ui or
Llama UI, the name of the library it is built on. They now say Llama
Chat, and the intro points to Llama UI in ui/lib/ui.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat connects to llama-server's default port, 9931
The CLI's --server default and its examples used 8080, which is not
where llama-server listens. They now use 9931, its default port, so
npx llama-chat reaches a llama-server started without --port.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: a shorter Llama Chat README that matches the code
The README grew to nearly 900 lines, and its tables of components,
hooks, stores and services had drifted from the code. It now keeps what
a reader needs: running it with npx and self-hosting, a short feature
list, development commands, the architecture (the line between Llama UI
and the app, a table of the layers and where they live, a flowchart and
one sequence diagram for sending a message), a few patterns worth
knowing, cross-origin isolation, how the build is embedded in
llama-server, the project tree and publishing. The folders now carry the
detail the tables used to. ASCII replaces the unicode arrows.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: llama-chat serves only static files, without the API proxy
The CLI forwarded every request it had no file for to llama-server, from
loopback. With --host 0.0.0.0 that let any LAN client reach a llama-server
started on 127.0.0.1 as a local request, tools included, which llama-server
on its own refuses without an API key. The browser now talks to each
llama-server directly, so llama-server's own rules apply. --server is gone,
and anything that is not a file in dist is a 404, or a 405 for other methods.
Assisted-by: Claude (Opus 5.5 Medium)
The deploy failed on master with a 403: HF_TOKEN_UI_STATIC_OUTPUT is
scoped to the UI bucket and cannot write ggml-org/llama-ui-storybook, so
hf upload tried to create the Space. The upload now uses
HF_TOKEN_UI_STORYBOOK when that secret exists, and the job runs on
hf-jobs-cpu-upgrade instead of a GitHub-hosted runner. That image gets
python3-venv from apt for the hf CLI, as the other HF Jobs workflows do.
Assisted-by: Claude (Opus 5.5 Medium)
This commit adds a lambda to check if a specified byte is not a valid
continuation byte in a multibyte UTF8 sequence.
The motivation for this is to improve readability and avoid some code
duplication.
* vocab : extract bpe_rank loading into function
This commit extracts the BPE rank loading logic into a separate function
named `load_bpe_ranks`.
The motivation for this is that the came code is currently used by both
gpt2/hybriddna/whitespace tokenizer models and gemma4 and this avoids
the duplication of code to improve readability and maintainability.
* change signature of load_bpe_ranks
* vulkan: expand bf16 support for unary and binary ops and rms_norm
* cpu: add bf16 support for scale, clamp, leaky_relu, swiglu_oai, swiglu_clamp, tri and diag
* fix sycl fill supports_op condition
* restore CUDA op order
* ui: generate DESIGN.md's token sections from the CSS
Add scripts/design-md.js in ui/apps/chat (npm run design:tokens, and npm
run check:design in CI), which writes the colors, typography and rounded
sections of DESIGN.md's front matter from llama-brand's colors.css and
Llama UI's primitives, semantic tokens and font stacks. The front matter
had drifted: it gave Matte's surfaces as the light base and system-ui for
every text role. Twins are now written wherever a token resolves
differently in dark mode (-dark) or in Matte (-matte).
Describe the floating panel's rim as the CSS draws it, on the border,
and say which parts of DESIGN.md are generated and which are kept by hand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say the Storybooks show accessibility findings without enforcing them
Both Storybooks keep a11y test 'todo': under 'error' 98 of the stories
fail, most on colour contrast of the muted foreground and on nested
interactive controls, which need design changes. Reword the Uses tile
and record the reason next to the setting.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list every Llama Chat group in the Introduction's Sections figure
The figure said it was in sidebar order but left out Forms, Actions,
Badges, Misc, Servers and PWA. Its list moves to sections.ts, and a unit
test checks it against storySort in .storybook/preview.ts, which has to
stay inline there because Storybook parses it as written.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: load the package's Llama Mono in both Storybooks
The library's Storybook declared Llama Mono a second time from
tests/stories/brand/fonts, a 72276-byte file that differs from the
76896-byte one the package ships, so its pages showed a Llama Mono that
does not ship. Drop both Storybooks' copies: theme.css already loads the
package's font.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: take Llama Chat's shared Storybook helpers from Llama UI
Llama Chat kept copies of the Editorial, Chapters, StateGallery,
StateGalleryItem, svx.d.ts and docs.css that live with the library's
stories. Its stories now import them as $llama-ui-stories/..., an alias
its Storybook config and tsconfig.json map onto ui/lib/ui/tests/stories,
the way it already compiles the library from source; docs.css and
svx.d.ts are referenced by path. The library's Editorial takes Llama
Chat's newer list numbering. An empty svelte.config.js next to the
stories keeps Llama Chat's svelte-check from loading the library's
vite.config.ts, as src/svelte.config.js does for the sources.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the library's Storybook to Llama Chat's versions
ui/lib/ui used ^10.6.2 with @storybook/addon-mcp at 10.6.2, while
ui/apps/chat pins 10.6.6, so one tree had two Storybooks. Pin the
library's Storybook packages exactly to Llama Chat's: 10.6.6,
addon-svelte-csf 5.1.6 and @chromatic-com/storybook 5.5.1.
Assisted-by: Claude (Opus 5.5 Medium)
* ggml-openvino: fix SWA mask detection
When the SWA cache size matches the full cache size, identify SWA
layers by the number of layers using each mask instead of cache size;
the larger group is SWA, and a tie leaves the layers unclassified.
* ggml-openvino: fix SWA KV cache write indices
Give SWA KV cache index tensors unique input names (a _swa suffix, like
the masks) to prevent them from being merged with the full-attention
cache indices.
* ggml-openvino: classify SWA layers only when the mask shows the window
Two attention masks do not imply SWA: DeepSeek sparse attention uses two
full-attention masks with the same name. When cache extents match,
classify the larger mask group as SWA only once its mask has fewer keys
per row than the other mask. Keep the classification per KV cache buffer
so new sequences reuse the same graph, and include swa_layers when
checking graph reuse.
* ggml-openvino: classify SWA layers from the first graph
Waiting for the window to appear in the mask caused a mid-sequence graph
recompile (~9 s for Gemma-4 at -c 1024 after position 512).
SWA classification already gives each mask group its own mask, KV write
indices, and attention size, which is correct whether or not the mask is
a sliding window. Stateful mask rebuild recovers the window from the
mask, or the causal mask for full attention. Classify the larger of two
mask groups from the first graph and drop the per-KV-buffer bookkeeping.
* ggml-openvino: mark SWA fallback as a model-level heuristic
Add a TODO above the equal-extent fallback to document that the larger
mask group is currently assumed to be the SWA group. Since stateless
graphs only require the two groups to remain distinct, this heuristic
may only be necessary for stateful execution and could potentially be
restricted to that path.
* ggml-openvino: derive attention sizes from the KQ mask shape
Dynamic CPU/GPU graphs previously passed n_seq_active and
attention_size(_swa) as separate inputs. Derive these values directly
from the KQ mask shape using ShapeOf + Gather inside the model. This
allows the GPU plugin to infer the Q reshape and KV slice shapes from
shape inference rather than runtime input values. Q's head size also
becomes static, enabling the fused SDPA kernel.
GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous
separate-input behavior.
The fused SDPA path also requires two supporting changes:
- Clamp the GPU mask to -30000. A key block that is entirely -inf for
a row produces NaNs in the fused kernel.
- Restate the head count and head size after the KV slice. The plugin
loses all slice dimensions when the end index is dynamic, causing
SDPA to fall back to the reference kernel.
Additional graph and quantization improvements:
- Skip the identity reshape of rows in SET_ROWS.
- Share a single Split for one-element view slices in dynamic GPU
graphs (e.g. Gemma-4 per-layer embeddings), replacing one
StridedSlice per layer.
- Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve
the file's native quantization for token_embd/output instead of
requantizing to Q8_0_C.
- Include both environment settings in the graph and model cache keys
and document them.
* ggml-openvino: only reuse Split outputs for exact chunk-aligned views
The dynamic GPU path reused a Split output for any single-element view
by deriving the chunk index from offset / stride. This produced
incorrect results for views that were not chunk-aligned or whose
remaining dimensions did not preserve the source strides, as seen in
test-backend-ops CONT with use_view_slice.
Restrict Split reuse to cases where the view exactly matches a split
chunk, meaning the offset is aligned to the split stride and the
remaining non-unit dimensions preserve the original source strides.
Gemma-4 per-layer embedding slices continue to use the shared Split
path.
* ggml-openvino: initialize ModelExtraInputInfo shape fields
Initialize the newly added `shape_source` and `shape_axis` fields in
the `ModelExtraInputInfo` brace initializer used by
`add_extra_inputs()`.
Without initializers for these fields, GCC emits
`-Wmissing-field-initializers`, causing CI to fail with
`LLAMA_FATAL_WARNINGS`.
* ggml-openvino: remove GGML_OPENVINO_REQUANT_EMBD
The option identifies embedding and output weights by their tensor
names, which are defined by core llama.cpp. Remove it from this PR;
preserving the file quantization of these weights will be addressed in
a separate change that identifies them by their roles in the graph.
* ggml-openvino: keep manual GQA attention 4-D
The manual GQA attention path reshaped Q, K, and V to 5-D so that
query heads sharing a KV head could broadcast against it. The GPU
plugin cannot fuse this graph: it materializes the attention scores
and copies the transposed K and V caches for every layer.
Instead, stack the query heads for each KV head along the sequence
axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask
accordingly. This keeps the graph 4-D from the MatMuls through the
Softmax, allowing the GPU plugin to fuse the attention into its SDPA
kernel and fold the K and V transposes into the fused operation.
Clamp the mask to -30000, matching the SDPA path, because the fused
kernel can return NaN when an entire block is masked with -inf.
Models where the K or V head size differs from the Q head size now
use the SDPA path, since the manual path assumes one head size.
* ggml-openvino: use a 4-D stateless KV cache on GPU
In stateless GPU mode, bind the entire KV cache as a
`[1, ctx, n_heads_kv, head_size]` tensor, with one token per row,
instead of binding the first `n_kv` rows as a
`[1, 1, ctx, n_heads_kv * head_size]` tensor.
Write cache updates with `ScatterUpdate` along the token axis, and
read the cache through `VariadicSplit` and `Transpose`. This allows
the GPU plugin's `StatelessKVFusion` to replace cache reads and writes
with in-place `StatelessKV` nodes, avoiding full-cache copies on every
decode step and repeated memory pool allocations as the bound cache
size grows.
Set `GGML_OPENVINO_DISABLE_KV_4D=1` to retain the previous layout.
* make embedding to have dynamic shape for OV GPU
* ggml-openvino: make the 4-D stateless KV cache opt-in
* add support for Q5_0 data type in OV GPU backend
* resolve CI test error
* ggml-openvino: fix -fa off and optimize grouped-query attention
With -fa off, llama.cpp stores the V cache transposed, with value
elements as rows and tokens along each row. The KV slicing code assumes
one token per row and binds the wrong data when n_kv < kv_size. Detect
transposed caches with GgmlOvDecoder::is_kvcache_token_rows() and skip
slicing for them.
For grouped-query attention, avoid repeating K and V for each query
head. Stack the query heads for each KV head along the row dimension
and let SOFT_MAX apply the scale, mask, and softmax so the GPU plugin
can fuse the operation with the MatMuls.
Read only the current layer's V-cache rows when layers use different
cache sizes. Reject -fa off in stateful mode with a clear error and
document that it requires stateless mode.
* ggml-openvino: fall back to stateless execution when -fa off
With `-fa off`, the graph writes the V cache in transposed form, which is
not supported by the stateful KV cache. Detect this pattern in
`compute_llm_params()` as a `SET_ROWS` into a KV cache through a `[1, n]`
reshape.
When stateful execution is enabled, log a warning and fall back to
stateless execution instead of failing graph conversion.
Update the `-fa off` note in `OPENVINO.md` to document this behavior.
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
* ggml-openvino: preserve flat KV cache layout when -fa is off
With -fa off, the 4-D KV cache layout introduced in #344 breaks MUL_MAT
attention for Phi-3.5 mini and Phi-4 mini, causing oneDNN to fail with
"could not construct a memory descriptor using strides".
Preserve the [1, 1, ctx, n_heads_kv * head_size] layout when the graph
writes a transposed V cache.
* ggml-openvino: extract Q5_0 weights when requantization is off
Q5_0 weights are requantized to Q4_0, but test-backend-ops builds naive
graphs, where requantization is skipped. Q5_0 then reached the normal
weight extraction, which had no Q5_0 case, so every Q5_0 MUL_MAT and
GET_ROWS op test failed with "Unsupported quantized type: q5_0".
Extract Q5_0 as symmetric i8 weights (q - 16) with one f16 scale per 32
weights, the same layout as Q8_0. Models are unchanged: they still
requantize Q5_0 to Q4_0.
* ggml-openvino: Optimize GDN output pre-scale for the NPU static path
The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero.
* Refine the code as review comments
1. Add envs desc into doc
2. Refine the code with removing duplicated env
* ggml-openvino: reuse graph bindings between decode calls
Reduce host overhead outside `infer()` from 0.5–1.1 ms to 0.14–0.38 ms
per token on Panther Lake.
- Reuse KV cache tensors and skip redundant port bindings.
- Use persistent USM host tensors for small GPU inputs and outputs.
- Defer graph-wide consumer lookup until after shape checks.
- Return model and compute params by const reference.
- Preserve tensor extras when memory, shape and type are unchanged.
* ggml-openvino: Add npuw dump envs for profiling
* Add the comments of new npu dump envs into doc
* ggml-openvino : ensure all leaves have unique names, remove input_srcs from graph key
* openvino: slice NPU prefill KV cache by active length
Add GGML_OPENVINO_NPU_KV_SLICE as an opt-in NPU prefill optimization. When enabled, static prefill keeps the graph's attention dimension at the active prompt/KV length instead of expanding it to the full context capacity, and the KV-cache input/output tensors are bound with a shortened context axis over the same backing storage.
This reduces the amount of KV cache and attention-mask data imported by the NPU plugin and lets the prefill graph plan smaller attention buffers for early prompt chunks. For example, with a 4096-token context and a 256-token prefill chunk, the first chunk can expose KV/mask dimensions at 256 instead of 4096, avoiding work over padded cache rows.
The trade-off is graph stability: NPU static shapes now depend on the aligned active attention length for prefill, so a cached prefill model is reused only when the attention sizes match. Decode keeps a fixed full-context graph to avoid recompiling on every generated token, and SWA/ring-cache cases still fall back to full-size binding when the live prefix is not a simple leading slice.
* openvino: use L0 host tensors for NPU buffers
Add GGML_OPENVINO_NPU_L0_HOST_TENSORS as an opt-in NPU buffer allocation mode. When enabled, non-remote OpenVINO backend buffers on NPU are allocated through the NPU default context with create_host_tensor(), and ggml tensor data points directly at that OpenVINO-owned Level Zero host allocation.
The optimization targets the data path rather than the graph math: KV-cache and compute tensors can be passed back to the NPU plugin as OpenVINO tensors backed by importable Level Zero host memory. That avoids an extra ordinary-host-memory-to-plugin-staging copy on each infer boundary, which is especially useful for prefill where large KV/cache buffers dominate input and output traffic.
If Level Zero host allocation fails, the backend logs a warning and falls back to the existing allocation path. The destructor tracks OpenVINO-owned storage so it does not free memory owned by the host tensor. The trade-off is that this is NPU-specific, consumes Level Zero host allocation resources, and improves transfer/import overhead without changing attention compute complexity or tensor shapes.
* openvino: fill NPU prefill masks directly
Add GGML_OPENVINO_NPU_FAST_MASK as an opt-in NPU static prefill input-packing optimization. When enabled, the prefill attention mask is built directly in the OpenVINO input tensor instead of first materializing a padded std::vector and then copying that whole buffer into the tensor.
The new fill_prefill_mask() helper writes each row once: it copies the valid mask prefix, fills padding columns with -inf, handles padded rows, and restores the causal diagonal to zero. This keeps the final mask numerically identical to the existing pad_input() plus set_zero_diagonal() path while removing one intermediate allocation and one chunk_size-by-context_size host copy per prefill chunk.
The optimization targets CPU-side staging overhead, not attention math. It composes with GGML_OPENVINO_NPU_KV_SLICE: KV slice reduces the mask dimensions, while FAST_MASK reduces the work needed to construct whatever mask shape remains. The trade-off is a second mask construction path to maintain, with limited benefit for very small prompts or contexts.
* openvino: make NPU requant layout configurable
Add GGML_OPENVINO_NPU_REQUANT_POLICY as the NPU-side selector for the default weight requantization layout. The existing NPU behavior remains the default: unset or group-128 maps quantized weights to Q4_0_128, while channel-wise selects the experimental Q4_0_C layout. Unknown policy values fail fast during model loading instead of silently compiling a graph with unintended weight packing.
The optimization target is NPU weight bandwidth and compressed-weight code generation. Q4_0_128 keeps weights in a 4-bit symmetric layout with one scale per 128 values, reducing both payload bytes and scale metadata versus smaller groups while preserving a more conservative quantization granularity than channel-wise. Q4_0_C uses one scale over the innermost dimension, which can further shrink metadata and simplify dequant broadcast, but may cost accuracy because the quantization block is much larger.
Include the selected NPU requant policy in the OpenVINO model cache discriminator. The same cgraph compiled with group-128 and channel-wise has different weight constants and dequant shapes, so reusing a cached decoder across policies would be incorrect. This keeps cache reuse stable within a policy while forcing recompilation when the layout policy changes.
Trade-off: policy selection gives a performance/accuracy knob for NPU experiments, but all options still pay the load/compile-time requantization cost. The default remains group-128 for a safer balance of bandwidth, metadata, and accuracy; channel-wise is intentionally documented as experimental.
* openvino: centralize NPU plugin config overrides
Add GGML_OPENVINO_NPU_CONFIG as the recommended catch-all override for OpenVINO NPU plugin and compiler properties. The value is parsed as comma-separated KEY=VALUE pairs and applied after the built-in defaults and named NPU aliases, so advanced users can override any exposed property without adding a new llama.cpp environment variable for every plugin tuning knob.
Keep the common knobs as named aliases for discoverability and compatibility: GGML_OPENVINO_NPU_COMPILER_TYPE maps to NPU_COMPILER_TYPE, GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL maps to NPUW_FUNCALL_FOR_ALL, GGML_OPENVINO_NPUW_UNFOLD_IREQS maps to NPUW_UNFOLD_IREQS, and GGML_OPENVINO_COMPILATION_NUM_THREADS maps to COMPILATION_NUM_THREADS. GGML_OPENVINO_NPU_COMPILE_CONFIG is retained as a legacy alias for NPU_COMPILATION_MODE_PARAMS, while the documentation now recommends expressing it through GGML_OPENVINO_NPU_CONFIG.
This keeps the public tuning surface compact while preserving high-signal aliases for settings with known performance or stability implications. NPU_COMPILER_TYPE can select the driver compiler, which currently generates faster prefill kernels on the tested NPU stack. NPUW_FUNCALL_FOR_ALL and NPUW_UNFOLD_IREQS expose NPUW graph lowering trade-offs around dispatch overhead, memory use, and long-context stability. COMPILATION_NUM_THREADS gives a way to cap compiler worker parallelism to reduce peak host memory during large graph compilation.
Trade-off: a free-form KEY=VALUE string is less type-safe than one env var per option and invalid plugin properties will only be diagnosed by the OpenVINO stack. Applying it last is intentional: it gives one escape hatch for bisecting driver/compiler behavior and for testing new plugin options without rebuilding llama.cpp, while the named aliases continue to document the most important performance knobs.
* openvino: add scatter-elements KV update option
Add GGML_OPENVINO_KV_SCATTER_ELEMENTS as an opt-in lowering for non-stateful single-row KV cache writes. When the flag is enabled and the set_rows indices are the simple decode case, translate_set_rows uses ScatterElementsUpdate with broadcasted rank-4 indices instead of ScatterUpdate.
The motivation is NPU decode performance. The default ScatterUpdate path updates one KV row but the NPU plugin does not perform that update in place; it effectively copies the full destination tensor to write a small slice. As context length grows, that full-tensor movement dominates the per-token KV update cost. ScatterElementsUpdate expresses the same single-row write at element granularity and is measurably cheaper for this shape.
Keep the optimization guarded. Stateful execution keeps its existing concat/update path, and multidimensional indices already use the broader ScatterElementsUpdate lowering. The new flag only changes the common non-stateful decode write, so the default behavior and graph shape remain unchanged unless the user opts in.
Trade-off: ScatterElementsUpdate introduces extra ShapeOf, Reshape, and Broadcast nodes to build indices matching the update tensor. That overhead is worthwhile only when it avoids the NPU plugin's full-destination ScatterUpdate copy, so the path is controlled by an environment variable rather than becoming the unconditional lowering.
* openvino: gather token embeddings from compressed rows
Add NPU token-embedding controls for models whose token_embd.weight would otherwise be widened from Q6_K to FP16. GGML_OPENVINO_TOKEN_EMBD_I8 keeps the embedding table on the normal int8 compressed-weight path, while GGML_OPENVINO_TOKEN_EMBD_I4 requantizes it to the group-128 int4 layout used by the NPU weight path.
The performance target is tied embeddings and large vocabulary tables. When token_embd.weight is also consumed by the lm_head matmul, a normal Gather from the dequantized table gives the dequantization subgraph a second consumer and can force the full table to be materialized. The new gather_compressed_rows helper recognizes the dequantization chain emitted for int4/int8 constants, gathers only the selected packed rows from the leaf constants, and rebuilds the convert/subtract/multiply/reshape chain on those rows.
This reduces memory pressure and avoids dequantizing or materializing a full embedding table when inference only needs the token rows for the current input. The helper is deliberately conservative: it only rewrites static 2D tables above a size threshold, walks only the known compressed-weight node patterns, limits recursion depth, and falls back to the original Gather when the subgraph is not recognized.
Trade-off: int8 and especially int4 token embeddings can change accuracy and may not be faster for small untied embedding tables, because the rewrite adds extra Gather/Reshape nodes and only pays off when it avoids a large full-table dequantization. The default behavior is unchanged; the NPU-specific I8/I4 paths are opt-in through environment variables.
* openvino: stream u4 requantization chunks
Extend the GGML_OPENVINO_REDUCE_COMPILE_MEM streaming requantization path to Q4_0/u4 targets. The previous streaming path intentionally skipped u4 because Q4_0 packs two weights per byte and the zero-point tensor stores two 4-bit values per byte, so the original quantize_q4_0 helper wrote as if every call started at block zero.
Teach quantize_q4_0 to accept a destination block offset and use the absolute block index for weights, scales, and zero-point writes. Streaming requantization emits whole-row chunks in increasing order, and the target block size is required to divide the row width, so no target block straddles a chunk boundary. That keeps packed weight bytes aligned and preserves the even-block assignment before the odd-block zero-point OR for each shared zero-point byte.
With that indexing in place, reduce-compile-memory mode can dequantize a chunk of rows into the scratch buffer and immediately quantize it into the final u4 buffers instead of materializing the entire tensor as temporary F32. This extends the peak-memory reduction already used by Q8/F16 targets to the NPU group-128/channel u4 layouts used for compressed weights.
Trade-off: the u4 path is more sensitive to block alignment than Q8/F16 because of nibble packing, so streaming remains guarded by the same row-divisibility check and only runs when GGML_OPENVINO_REDUCE_COMPILE_MEM is enabled. When the flag is off, the full-materialization path and quantized output layout remain unchanged.
* openvino: fail fast on static recompile after weight release
Extend the GGML_OPENVINO_RELEASE_WEIGHTS safety check to the static graph cache-miss path. Once host weight buffers have been released, any later compile would read invalid/zeroed host pages and bake the wrong constants into a new OpenVINO model.
The dynamic path already rejected this situation. Static execution can still miss the decoder cache when graph shapes or model parameters change, so it needs the same guard before erasing cached infer requests and rebuilding prefill/decode models.
This is a correctness and diagnosability change rather than a throughput optimization. It preserves the memory-saving release-weights mode for stable graph shapes, but turns unsupported recompilation into an explicit abort instead of silently producing a corrupted compiled model.
* openvino: add experimental NPU Q4_0 keep option
Add GGML_OPENVINO_NPU_KEEP_Q4_0 as an opt-in escape hatch for NPU weight handling. When enabled for native Q4_0 tensors, the NPU requant path returns nullopt and keeps the original Q4_0 extraction instead of regrouping the tensor through the selected NPU requant policy.
The motivation is to avoid unnecessary work for weights that are already stored as 4-bit values. The default NPU policy maps quantized weights to layouts such as Q4_0_128, which can reduce scale metadata but requires round-tripping Q4_0 through the requantization path. Keeping Q4_0 native can save that conversion work and preserve the original group-32 packing.
Keep the option experimental and disabled by default. Current NPU driver/plugin stacks can compile the resulting group-32 graph, but inference may hang or fail with ZE_RESULT_ERROR_DEVICE_LOST. The default behavior therefore continues to use the safer NPU requant policy; this flag is mainly useful for debugging driver behavior, measuring the true cost of regrouping, and testing future NPU stacks.
Trade-off: native Q4_0 avoids regrouping overhead and keeps the original quantization blocks, but it gives up the more regular group-128 layout expected by the default NPU compressed-weight path and may be unstable on current hardware/software combinations.
* openvino: unfold NPUW only for decode graph
Compile the static prefill and decode models with separate OpenVINO config maps so the decode graph can opt into NPUW_UNFOLD_IREQS without forcing the same lowering on prefill. The build_static_model helper now receives the config map explicitly, and the decode path gets a copy of the base config with NPUW_UNFOLD_IREQS=YES when the user has not already set it.
The performance target is NPU token generation. Decode executes a long stream of single-token infers, where folded NPUW function-call dispatch overhead becomes visible in every token. Unfolding the NPUW calls into separate infer requests removes that repeated dispatch cost and measured about +20% token-generation throughput on the tested stack, matching the behavior seen in OpenVINO GenAI.
Prefill keeps the original folded config. Prompt processing is a batched workload dominated by larger matmuls, and the folded form is faster there while also using less memory. Splitting the config lets prefill and decode use the lowering that matches their workload shape instead of treating NPUW_UNFOLD_IREQS as a single global compromise.
Trade-off: unfolding can increase memory and request-management overhead, so the automatic default is limited to the NPU decode model. If a user explicitly provides NPUW_UNFOLD_IREQS through the named env alias or GGML_OPENVINO_NPU_CONFIG, that value is respected because the decode override is only applied when the property is absent.
* openvino: tidy NPU tuning helper ordering
Collect the remaining small cleanups from the NPU tuning series. This reorders the NPU KV-slice documentation and environment-variable cache entry next to the other NPU opt-in toggles, adds a short header comment for the centralized NPU helper declarations, and makes the static prefill attention-size assignment in the decoder a little more explicit.
There is no intended behavior change in this commit. The purpose is to make the final source layout match the split commits that introduced the NPU tuning knobs, so future readers can find the related flags and helpers in one place without mixing that mechanical cleanup into the functional performance commits.
* openvino: normalize NPU compile env overrides
Move the default NPU_COMPILER_TYPE=DRIVER setting into the NPU compile_config initializer so the default NPU plugin options are declared in one place.
Route GGML_OPENVINO_NPU_COMPILE_CONFIG and GGML_OPENVINO_NPU_COMPILER_TYPE through the same set_compile_option_from_env helper used by the other named NPU compile option aliases. This keeps the legacy NPU_COMPILE_CONFIG mapping to NPU_COMPILATION_MODE_PARAMS intact while making all optional env-to-compile_config overrides follow the same non-empty-value rule.
GGML_OPENVINO_NPU_CONFIG is still parsed last, so explicit comma-separated KEY=VALUE overrides continue to have the highest precedence over both defaults and named aliases.
* openvino: stream NPU weight requant by default to cut compile peak
Default the streaming requantization path on for NPU so the compile-time F32 dequant transient (~1-2 GB for token_embd) is capped without an env flag. Output is byte-identical to full materialization; GGML_OPENVINO_REDUCE_COMPILE_MEM / MEMORY_OPTIMIZE still override (set to 0 to force off). Kept the g_nonov_weight_cache opt-in to avoid holding token_embd resident (which regressed decode steady memory). pp1024_d0: PrefillPeakPrivWS 5974->4664 MiB (-22%), decode steady and pp/tg throughput unchanged.
* openvino: add self-contained NPU model cache
Add an opt-in embedded-weight cache mode for the static NPU backend. GGML_OPENVINO_SELF_CONTAINED_BLOB=1 exports separate prefill and decode compiled models with ENABLE_WEIGHTLESS=false, uses isolated NPUW weight banks, validates raw GGUF weight manifests, binds imported ports by deterministic frontend order, and serializes the large NPUW compile/export/import operations.
Preserve raw model bytes during cold cache construction by requantizing into separately owned OV tensors instead of rewriting the backend model buffer. This keeps later graph fingerprints and manifests stable when llama-bench creates independent prompt and generation contexts.
Add strict GGML_OPENVINO_SELF_CONTAINED_MMAP=1 warm-import mode. The backend exposes a non-owning buffer_from_host_ptr wrapper over the GGUF mmap, uses raw tensor allocation sizes, avoids allocating/copying the private OpenVINO model-weight buffer, and stages the few mmap-backed runtime constants into small importable NPU host tensors. Invalid or missing cache entries fail clearly instead of trying to requantize into read-only mapped storage.
The normal path and existing weightless MODEL_PTR cache remain unchanged when the new flags are unset. Verified with phi-4-mini: chunk512 pp3072/tg512 cold 1123.10/27.60 t/s and warm mmap 1124.72/27.16 t/s; warm mmap removes roughly 3.9 GiB peak commit in the measured prefill workload. Self-contained blobs consume about 8.34 GiB for the separate pp and tg contexts.
* openvino: add lazy import for self-contained mmap cache
Add GGML_OPENVINO_SELF_CONTAINED_LAZY_IMPORT to make the low-working-set strict mmap path opt in instead of changing the default warm mmap behavior.
When the flag is set with SELF_CONTAINED_BLOB=1 and SELF_CONTAINED_MMAP=1, the static NPU path imports only the current phase blob: prefill imports the prefill blob, decode imports the decode blob later. This avoids the pair-shaped shared compiled-model cache for the lazy path and caches only the current phase infer request.
The regular warm mmap path remains eager and continues to prepare both prefill and decode compiled models. Non-mmap self-contained builds remain serial, and the regular non-self-contained path keeps the existing parallel build behavior.
Also register the new environment variable, document the memory/latency tradeoff, and add profiling-only import timing around model-cache imports.
Validated with ReleaseOV build: cmake --build build/ReleaseOV --target llama-bench llama-simple --config Release. Lazy warm mmap run measured PrefillPeakWS around 5759 MiB versus the eager warm mmap path around 7897 MiB.
* openvino: release GGUF mmap resident pages before blob import
Add GGML_OPENVINO_SELF_CONTAINED_RELEASE_MMAP_PAGES as an explicit opt-in for self-contained compiled-model cache hits.
Before importing a valid self-contained blob, recursively walk the graph to find unique external GGUF mmap-backed weight buffers and drop their resident pages while preserving the mapping address and tensor pointers.
On Linux use page-aligned madvise(MADV_DONTNEED). On Windows, where there is no reliable per-range mapped-file equivalent, call VirtualUnlock for the identified range and trim the current process working set with SetProcessWorkingSetSize(-1, -1). Pages required later can fault back on demand.
Keep the behavior disabled by default and document the physical-memory versus commit-memory tradeoff. Profiling reports the submitted GGUF range and model import duration.
Validated with ReleaseOV llama-bench/llama-simple build and pp3072+tg512 warm mmap lazy import. With page release enabled, PrefillPeakWS dropped from about 5758 MiB to 3053 MiB while throughput remained comparable; PeakCommit stayed about 3331 MiB.
* openvino: add GGML_OPENVINO_NPU_SHARED_BANK to share prefill/decode weights bank
When set, prefill and decode self-contained blobs use one shared NPUW_WEIGHTS_BANK name (ggml-sc-shared-<pfp>-<dfp>) instead of separate per-phase banks. Since both decoders are built from the same model_weights map, NPUW dedups them into a single in-process bank; on warm import the identical (uid,tensor) pairs are content-verified and reused (patched NPUW weights_bank). Cuts resident compiled-weight footprint ~2GB (pp3072 warm: In-Use 7954->5871 MiB, WS 7891->5765) with prefill throughput unchanged.
* openvino: drop SELF_CONTAINED_LAZY_IMPORT and RELEASE_MMAP_PAGES
The shared weights bank (GGML_OPENVINO_NPU_SHARED_BANK) supersedes both: it deduplicates the prefill/decode compiled-weight copy directly, so lazy per-phase import and GGUF-page release no longer reduce steady memory. Remove the two env vars, the self-contained lazy-import branches (defaulting to the eager path that imports both blobs), the release_external_weight_pages helper, and the now-unused ggml_openvino_buffer_is_external / _release_external_pages buffer APIs.
* openvino: demand-page large NPU compute buffer in strict warm-mmap mode
In strict self-contained warm-mmap mode weights come via buffer_from_host_ptr,
so every alloc_buffer is graph scratch/KV. The large prefill compute buffer is
never touched on the host by the on-device fused OV model, yet the NPU L0 host
tensor (create_host_tensor + memset) makes all ~2.4 GiB resident. For buffers
>= 1 GiB in this mode, reserve demand-zero pages (no L0, no memset) so they stay
non-resident; KV/small buffers keep the L0 host tensor for zero-copy import.
pp3072 warm mmap + shared bank: prefill peak WS 5765 -> 3314 MiB (-43%),
In-Use delta ~5940 -> ~3232 MiB; prefill 1256.7 -> 1252.9 t/s and decode
28.27 -> 27.96 t/s (both flat). Commit charge unchanged (demand-paged).
* perf improvement success
* openvino: size static NPU sliding-window masks to the window cache
The static graph built the sliding-window KQ mask at the full context length,
while the windowed layers bind their (smaller) window KV cache. With gemma-4 at
-c 4096 -ub 1024 the window cache is 1536 cells, so SDPA shape inference failed
([1,1,1024,4096] mask vs [1,8,1536,256] K/V) and every prompt errored out.
With GGML_OPENVINO_NPU_KV_SLICE the prefill mask was meant to use
attention_size_swa, but the check matched "swa" in the ggml tensor name, and
llama.cpp names both masks "attn_inp_kq_mask"; the windowed layers got the
full-attention sliced length instead ([1,1,1024,256] vs [1,8,1024,256]).
Identify the sliding-window mask by tensor identity (is_swa_mask) and size it to
ctx_per_seq_swa, both in the graph input shape and in the runtime mask padding.
* ggml-openvino: create the model cache dir under a Windows drive root
make_dirs() mkdir'ed every path prefix, including the drive root "D:", which
fails with EACCES (not EEXIST) on Windows. Any absolute cache path therefore
logged "could not create model cache dir ... caching disabled" and silently
ran without GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. Skip the drive root.
* ggml-openvino: split device-specific passes and NPU config out of shared code
Mechanical refactor, no behavior change (step 1 of the NPU/MoE architecture plan):
* openvino/pipelines/: TranslateSession::apply_transformations() now registers
common passes, then the device's passes, then the execution-mode passes
(stateful KV, static shapes), selected by a per-graph GraphTraits. The GPU MoE
fusions and the GPU eltwise-rank workaround move to pipelines/gpu.cpp; CPU and
NPU pipelines are empty placeholders for device lowerings. Pass order is
unchanged.
* npu/npu_config.*: the NPU (NPUW) compile config built at backend init, the
NPUW strip for cacheless graphs and the per-graph static prefill/decode
variants (HFA prefill, decode NPUW_UNFOLD_IREQS) move out of
ggml-openvino-extra.cpp and utils.cpp.
* npu/npu_quant_policy.*: the NPU requantization rules (requant policy,
token_embd Q6_K, keep-Q4_0) move out of ggml_openvino_get_requant_type().
Verified identical output before/after on OV-CPU, GPU and NPU (gemma-4 E4B).
* ggml-openvino: one KV-slice switch for dynamic and static graphs
KV-cache slicing (bind the KV cache, and size the KQ masks, at the attended
length instead of the full context) existed twice: on by default for dynamic
graphs (GGML_OPENVINO_DISABLE_KV_SLICE to turn it off) and opt-in for static
prefill (GGML_OPENVINO_NPU_KV_SLICE). The static mode is not NPU-specific; it
applies to any static graph, including CPU/GPU with GGML_OPENVINO_FORCE_STATIC.
Describe it as one mechanism with two modes behind GGML_OPENVINO_KV_SLICE
(unset = each mode's default, 0 = off, 1 = on) and two predicates,
ggml_openvino_{dynamic,static}_kv_slice_enabled(), used by the KV binding,
the static graph shapes, the mask padding and the cache keys. The two old
variables remain as deprecated aliases.
* ggml-openvino: initialize all ModelExtraInputInfo fields for the HFA inputs
dev_backend_openvino added shape_source/shape_axis to ModelExtraInputInfo; the HFA past-KV and mask
initializers still listed four fields, which GCC rejects under -Werror=missing-field-initializers.
* ggml-openvino: make new NPU defaults opt-in and fix static model conversion
- Convert the static prefill and decode models one after the other, both
before either is compiled. The decoders share the weight nodes;
converting on two threads crashed intermittently inside libopenvino,
and converting decode after the prefill compile gave wrong output on
Bonsai-8B.
- Compile prefill and decode in parallel only with
GGML_OPENVINO_NPU_PARALLEL_COMPILE=1, since it raises the host memory
peak.
- Unfold NPUW function calls on the decode graph only with
GGML_OPENVINO_NPU_DECODE_UNFOLD=1. The unfold default gave wrong output
on gemma-4-E4B and slowed decode on several small models.
- Stream NPU weight requantization only with
GGML_OPENVINO_REDUCE_COMPILE_MEM or GGML_OPENVINO_MEMORY_OPTIMIZE, as on
the other devices.
- Fix a -Werror=missing-field-initializers build error in utils.cpp.
- Declare ggml_openvino_get_npu_requant_type() only in
npu/npu_quant_policy.h.
- Docs: document the new switches, deprecate stateful execution, show
stateless-only results in the validated models table, and update the
validation system.
* ggml-openvino: honour the -1 skip id in MUL_MAT_ID
Upstream made an id of -1 in MUL_MAT_ID skip the slot and zero its row of
the result. The translator passed -1 straight to GatherMatmul or Gather,
so 169 of the 743 MUL_MAT_ID op tests on CPU (all skip=1) failed.
A skipped slot now looks up a valid expert, and the result is multiplied
by a 0/1 mask of the kept slots. For the packed MXFP4 path (Gather and
MatMul), clamping the id to expert 0 is enough. GatherMatmul computes a
wrong row when a token names the same expert twice, so the general path
gives each skipped slot an expert that no other slot of the token uses.
TopK indices are never -1, and model graphs take the ids from the
router's TopK. Those graphs skip the handling and stay unchanged, so the
GPU MoE fusion still matches them.
* ggml-openvino: Gather rows before dequantizing a quantized embedding table
1. Optimize gather to fetch row before dequant
2. add NPU token_embd quantization option
* ggml-openvino: select the NPU token_embd type with GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT
GET_ROWS had two rewrites that gather the rows of a quantized table before
dequantizing them: gather_compressed_rows, which runs on every device, and
gather_before_dequant, enabled on NPU by GGML_OPENVINO_GATHER_BEFORE_DEQUANT.
Both match the int4 and int8 chains built by make_int4_weights() and
make_int8_weights(). Keep gather_compressed_rows and drop the second rewrite
and its switch.
The NPU type of a Q6_K token_embd.weight was set by two booleans,
GGML_OPENVINO_TOKEN_EMBD_I8 and GGML_OPENVINO_TOKEN_EMBD_I4, and by
GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT. Keep only the string variable, with the
values f16 (default), q8_0_c, q4_0_128 and q4_0_c, so the int8 and group-128
int4 settings stay available. An unknown value aborts, as for
GGML_OPENVINO_NPU_REQUANT_POLICY, instead of silently using f16. The variable
replaces the two booleans in the NPU compiled-model cache key.
* align PAD pads to input rank in stateful mode
* Gate the changes under is_stateful check to address review comment
* ggml-openvino: lift Q tensors for non-RoPE layers to rank 4 in stateful mode
* ggml-openvino: unify KV slicing under GGML_OPENVINO_KV_SLICE, on by default
Replace ggml_openvino_dynamic_kv_slice_enabled() and ggml_openvino_static_kv_slice_enabled() with a single ggml_openvino_kv_slice_enabled(). KV slicing binds the KV cache, and sizes the KQ masks, at the attended length instead of the full context.
GGML_OPENVINO_KV_SLICE now defaults to 1 on CPU, GPU and NPU; set it to 0 to disable. Dynamic graphs (CPU/GPU) bind a view over the cache without recompiling. Static graphs (NPU or GGML_OPENVINO_FORCE_STATIC) compile prefill at the aligned attended length and recompile when it changes; decode keeps its full-context graph.
Remove the deprecated aliases GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_NPU_KV_SLICE from the env var list, the model-cache settings, docs/backend/OPENVINO.md and the backend README.
Behavior change: static KV slicing is now on by default on NPU, so prefill compiles one graph per distinct n_kv (multiples of 256).
* ggml-openvino: keep stateful execution with -fa off
Keep the V cache token-major in stateful mode instead of falling back to
stateless execution. SET_ROWS appends to the state, and PERMUTE reads it
back in the layout attention needs, so MUL_MAT does not transpose it.
Dropping a reshape in grouped attention lets OpenVINO fuse it into SDPA.
For padded V caches, slice after the transpose so the cache is read in
place, without a copy that is recompiled for every KV length.
This removes the stateless fallback for -fa off.
* ggml-openvino: Add the stateful check to make the gate safer
* ggml-openvino: support gemma-2 soft-capped attention
Gemma-2 caps its attention scores as softcap * tanh(scores / softcap)
before the softmax. FLASH_ATTN_EXT now applies the cap on the manual
attention path, as SDPA has no soft-capping, and the decoder looks
through the SCALE, TANH, SCALE chain of -fa off when it matches
attention.
One helper finds the K cache of an attention for both the
sliding-window pre-pass and the main pass. The pre-pass copy did not
see through the chain, so the sliding-window layers used the
full-attention sizes once the context exceeded n_swa + n_ubatch.
Soft-capped attention keeps the flat KV cache layout: with the 4-D
layout the GPU plugin fails in its oneDNN path.
* ggml-openvino: enable NPU Level Zero host tensors by default
Back NPU buffers with Level Zero host memory unless GGML_OPENVINO_NPU_L0_HOST_TENSORS=0.
The plugin imports these buffers without a host-to-device copy each infer; on phi-4-mini
Q4_0 (pp3072+tg512) decode goes from 7.2 to 10.6-10.8 t/s. Allocation failures still fall
back to plain host memory.
* ggml-openvino: enable NPU fast prefill mask by default
GGML_OPENVINO_NPU_FAST_MASK now defaults to 1; set it to 0 to fall back to pad_input() + set_zero_diagonal().
The fast path builds each static prefill mask chunk directly in the OpenVINO input tensor (row memcpy of the valid prefix, -inf padding, zero diagonal), avoiding a temporary chunk_size x context_size vector and a full-tensor copy per chunk. The result is bit-identical to the fallback path.
* ggml-openvino: add opt-in exact zero points for asymmetric weights
Asymmetric quantized weights (Q4_K, Q4_1, Q5_1, and non-requantized
Q5_K) represent each group as `w = d * q - m`. The backend currently
converts these to u4/u8 weights with an f16 scale and an integer zero
point `round(m / d)`, which can shift all weights in a group by up to
half a quantization step.
Add `GGML_OPENVINO_EXACT_ZERO_POINT=1` to preserve the zero point in f16
instead, using the same path already used for MoE expert weights.
This is disabled by default because, with current OpenVINO releases, an
f16 zero point moves many GPU fully-connected layers off oneDNN,
resulting in slower prompt processing.
* ggml-openvino: rename npu/ to device/ and openvino/pipelines/ to openvino/pass_registry/
npu/ holds device-specific backend behaviour (NPU compile config and
requant policy), and pipelines/ only registers passes per device; the new
names describe what the folders contain. No functional change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs: document NPU self-contained model cache and shared weights bank
Add GGML_OPENVINO_SELF_CONTAINED_BLOB, GGML_OPENVINO_SELF_CONTAINED_MMAP and GGML_OPENVINO_NPU_SHARED_BANK to the OpenVINO environment variable table, and a note describing the export/import workflow and the requirement to keep NPU settings identical between runs.
* ggml-openvino: remove experimental GGML_OPENVINO_NPU_KEEP_Q4_0
Keeping native Q4_0 (group-32) on NPU compiles but hangs the device at inference (ZE_RESULT_ERROR_DEVICE_LOST), so the option was never usable. NPU Q4_0 weights now always go through GGML_OPENVINO_NPU_REQUANT_POLICY. Drop the env var, the requant bypass, its self-contained blob fingerprint term and the docs entry.
* ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares
All changes of https://github.com/ravi9/llama.cpp/pull/358 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.
Commits of the PR:
- Update fix to just maintain fp32 precision instead of modifying the graph
- Limit the fp32 precision spreading in the graph by fencing it with Convert. Prevents perf degrade.
- Update OPENVINO.md
- Update flash_attn_ext.cpp
- Update utils.cpp
* enable flash_attn with sink input
* ggml-openvino: fix FLASH_ATTN_EXT input detection and sink scale
translate_flash_attn_ext read get_input_names()[3] and [4] on every
FLASH_ATTN_EXT node, also on nodes with only 3 or 4 inputs. The
out-of-bounds read crashed test-thread-safety and
test-speculative-rejection-sampling. The inputs are q, k, v, then the
optional mask and sinks, and supports_op rejects sinks without a mask,
so the input count tells which inputs are present.
With sinks, the scale was replaced with 1.0. llama.cpp passes the
attention scale in op_params[0] and adds the sinks separately, so keep
op_params[0]. The "scale" input branch is removed: ggml's src[4] is
only ever the sinks.
Only SDPA applies the sinks. The manual GQA path and the NPU HFA
prefill path drop them, so attention with sinks now always takes the
SDPA path. SDPA has no logit soft-capping, so supports_op rejects sinks
together with soft-capping.
* ggml-openvino: accept flash attention on layers without RoPE
On layers without RoPE (SmolLM3: every 4th layer), Q reaches
FLASH_ATTN_EXT through PERMUTE -> VIEW -> RESHAPE. The support check
rejected this pattern, causing these layers to fall back to CPU and the
split graph to fail with -fa on/auto.
* ggml-openvino: keep only weight-divisor DIVs on CPU
The GPU DIV guard for per-channel scale vectors (`MPT`'s `act_scales`) currently checks only the divisor's shape. During single-token Qwen2-MoE decode, the shared-expert gate `silu(x) / x` has a `[1,1,1,1]` divisor that matches this shape, causing its DIVs to be kept on the CPU.
As a result, every decode step incurred CPU execution and Qwen1.5-MoE decoded at only ~2 tok/s.
Apply the guard only when the divisor is a model weight, so runtime tensors such as the Qwen2-MoE gate divisor can remain on the GPU.
* ggml-openvino: Qwen MTP and multi-seq optimization with new OV ops
All changes of https://github.com/ravi9/llama.cpp/pull/370 as one commit. The PR branch had
dev_backend_openvino merged into it, so its commits could not be replayed one by one when
dev_backend_openvino was rebased onto upstream master.
Commits of the PR:
- openvino : support Qwen MTP decoding
- openvino: Optimize TopK
- openvino: remove single-slot recurrent cache shortcuts
- openvino: fuse generic recurrent caches and remove shape waits
- openvino: include dense Q8 weights in all INT4 requantization
- openvino: use two GDN outputs for snapshots
- docs: explain native OpenVINO recurrent cache contracts
- docs: clarify Qwen3.5 purpose of native OpenVINO cache ops
- llama: skip unallocated hybrid graph inputs
- quantize: preserve Qwen hybrid layer metadata when pruning
- openvino: emit fused recurrent computation and rotate device caches
- openvino: limit KV slicing to attention caches and share compiled blobs
- tools: add natural-text server benchmark and OpenVINO profile summary
- docs: record fused-cache branch split and UD validation
- server-bench: preserve native token counts and save request timings
- openvino: document MTP regression and host USM alias recovery
- openvino: record large-context allocation diagnosis and default-batch validation
- server-bench: count buffered native tokens and reject stream errors
- openvino: document UD split and unified server benchmarks
- openvino: record Vulkan long-context baseline and GPU-fit controls
- openvino: finalize UD benchmark report and MTP comparison
- openvino: use stable in-place recurrent caches and optional separate convolution
- openvino: accept matching empty axes in DIV repeat translation
- openvino: remove the obsolete cache rotation environment setting
- openvino: validate imported KV results in their four-dimensional layout
- docs: record the bounded in-place KV and convolution benchmark rerun
- docs: complete default-batch OV unified-KV 32K benchmarks
- openvino: remove the separate recurrent convolution path
- openvino: handle zero-logits batches and changing sampler sequences
- docs: compare default and large Qwen3.6 benchmark batches
- docs: use the historical Vulkan default-batch reference consistently
- server-bench: use an integer histogram bin count
- benchmarks: summarize current OV unified MMLU timings
- docs: record cache-only unified KV MMLU results
- openvino: preserve the TopK index port in GET_ROWS
- openvino: reuse the MTP sampling model across active slots
- openvino: expose selection flags and offsets in graph dumps
- docs: consolidate Qwen OpenVINO defaults and performance evidence
- docs: retain continuous-request MTP benchmark results
- docs: compare stable MTP with the non-MTP serving baseline
- openvino: isolate GDN snapshots and skip naive cache fusion
- ggml: document Qwen3.6 GenAI benchmarks
- ggml-openvino: keep PR changes inside the backend folder
* ggml-openvino: drop the extra semicolons after OPENVINO_OP
OPENVINO_OP expands to member function definitions that end in a closing
brace, so a semicolon after it is an empty declaration inside the class.
GCC 15.2 reports it with -Wextra-semi, which ggml enables for GCC 8.1 and
newer, so a build with LLAMA_FATAL_WARNINGS=ON (the default in ci/run.sh)
fails in gated_delta_net.hpp. The CI compilers do not report it.
recurrent_cache_update.hpp has the same pattern twice. It is compiled only
when the OpenVINO package provides the recurrent cache-update ops.
* ggml-openvino: make q8_0_c the default NPU token_embd requant type
Default of `GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT` changed from `f16` to `q8_0_c`.
Applies only to NPU and only to a Q6_K `token_embd.weight`; other devices and
quantization types are unchanged.
* ggml-openvino: do not treat the recurrent state gather as out_ids
The output_len scan now runs for every model and takes the first GET_ROWS
whose index is a graph input. On Qwen3.5 that is the recurrent state gather
of build_rs, which indexes the inp->s_copy leaf directly, so output_len
became 1. In static (NPU) mode the prompt model then returned one logits
row per chunk, and llama-perplexity read stale rows (PPL about 240000 on
Qwen3.5-2B). The same predicate also gave the s_copy input the out_ids
shape and data in static mode.
is_output_idx now skips GET_ROWS that read a recurrent state cache. The old
predicate stays as is_get_rows_input_idx for the dynamic-dim bookkeeping,
so the s_copy leaf keeps its dynamic length there.
* ggml-openvino: prefer the size-preserving dim when a RESHAPE repeats a stride
The RESHAPE dynamic-dim rule matches the source's dynamic stride against
the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4
E2B has one KV head, so [head_size, n_tokens] -> [head_size, 1, n_tokens]
has the same stride on dims 1 and 2. The first match picked the size-1 head
dim, so the K/V views lost their dynamic token dim and became static
reshapes with the first graph's token count. Every later graph with another
token count failed.
Prefer the stride match that keeps the dynamic size, and fall back to the
first stride match for reshapes that change it.
* ggml-openvino: correct the RESHAPE dynamic-dim comment for size-1 dims
The comment above the RESHAPE stride match in compute_node_dynamic_dims()
still said the match is unambiguous because result strides are strictly
monotone. A size-1 dim has the same stride as the dim after it
([head_dim, T] -> [head_dim, 1, T] with one KV head), so it is not.
Explain the size-1 case in the block comment, drop the duplicate comment
inside the loop, and name the source dim's size once.
* examples: add llama-chat-replay
Replays JSONL prompts like OpenVINO GenAI chat.py (-pf, -i chat history, greedy, -o, -pc metrics). The first prompt is prefilled plus one decode step during load (Warmup time) and the memory is cleared, so compile/import is counted in Load time as in GenAI.
* ggml-openvino: make the NPU lazy host threshold configurable, default 2 GiB
With a large context the KV cache (~1 GiB at ctx 8K) crossed the 1 GiB threshold, lost its Level Zero host tensor in strict warm-mmap mode and made decode copy the KV every token. Default to 2 GiB so only the larger prefill compute buffer stays demand-zero, and allow overriding it with GGML_OPENVINO_LAZY_HOST_THRESHOLD_MB.
* ggml-openvino: fix NPU HFA prefill with ordinal-suffixed KV input names
KV cache inputs are now named with a graph-ordinal suffix (cache_k_l0#l12), but flash_attn_ext looks the HFA past input up as ggml_hfa_past_cache_k_l<layer>, so GGML_OPENVINO_NPU_HFA_PREFILL failed with 'ggml_hfa_past_cache_k_l0 not found in tensor map'. Name the past inputs after the bare cache name and resolve the source cache by that name when binding.
* ggml-openvino: keep the NPU prefill model on cache hits after decode
With KV slicing, decode calls overwrite the shared decoder's attention size with the full context, so the next prefill compared that against its sliced size, missed the cache and rebuilt the decoder and both infer requests (plus a cold first chunk). Remember the attention sizes the prefill model was built with and compare against those.
* ggml-openvino: bind NPU HFA past KV zero-copy in the cache layout
The HFA prefill past K/V inputs were [1, heads, past, head_size] copies of the cache, relaid out per chunk on the host and allocated per decoder (256 MiB on phi-4-mini at 3k). Declare them in the ggml cache layout [1, past, heads, head_size], bind the first past rows of the cache directly and transpose in the graph; NPUW HFA reads the tiles from the BSHD past (needs the matching NPUW HFA support). HFA input ABI v4.
* ggml-openvino: size the NPU decode KV window by GGML_OPENVINO_NPU_DECODE_BUCKET
Static decode attended the whole context, so every token read and wrote the full KV cache. With a bucket base set, decode attends the smallest base*2^k rows covering the next position (clamped to ctx): the KV inputs and masks are sliced to the window, the compiled pair's decode graph is built at the window the prefill reaches, and other windows are built or imported from the self-contained cache on first use with the pair's settings and bank. With self-contained blobs, keep the token_embd weight node for the later window models so they reuse the pair's bank entry instead of embedding a second vocab copy. The static build/convert lambdas become functions shared with the window path.
* ggml-openvino: allocate the NPU KV cache on demand with GGML_OPENVINO_NPU_KV_ON_DEMAND
The NPU KV buffer was committed for the whole context up front. With the option set, the KV buffer is swapped for demand-zero memory at its first KV tensor and every KV tensor gets its own storage in a registry, grown in base*2^k rows (base = decode bucket or 1024) as the static graphs reach further. Host accessors, KV inputs and outputs, sliced KV views and the HFA past bind that storage; when it moves, idle requests bound to the old storage are recreated so it is released.
With decode buckets and KV slicing the storage also takes the layout GenAI uses on NPU: K is [heads][rows][head_size] and V is [heads][head_size][rows], each bound whole to prefill and decode. SET_ROWS scatters new K rows on axis 2 and V rows on axis 3, the attention permute only cuts the window, the HFA past is the storage itself, and attention multiplies the probabilities by V^T with transpose_b instead of SDPA. A chunk-sized scatter on the innermost axis is slow on NPU, so the HFA prefill returns each chunk's V^T rows and the host copies them into the cache. Host reads, writes and fills go through a layout-aware copy, storage growth re-packs each head, and the layout is part of the model cache fingerprint.
* ggml-openvino: NPU weightless blob cache with one shared weights file
GGML_OPENVINO_NPU_WEIGHTLESS_BANK=1 (with GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR) exports weightless prefill/decode NPUW blobs plus one <prefill>-<decode>.weights.bin. Weight Constants are laid out once by data pointer and their WeightlessCacheAttribute is retagged with the real offsets. Warm import uses ov::weights_path, and NPUW dedups both models into one bank through the LazyTensor hashes, so the OpenVINO weights_bank duplicate-uid patches are not needed. Decode KV-window models resolve the shared weights through the pair's file, or get their own file when the pair was imported. Also drop the old blob and manifest before publishing a re-export: std::rename does not replace an existing file on Windows.
Based on anikulk/llama.cpp 4ab270aa9.
Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com>
* ggml-openvino: prebuild all NPU decode KV windows with GGML_OPENVINO_NPU_DECODE_PREBUILD
Decode windows were built lazily when the sequence first grew into them, which stalled generation for a compile without a cache and made a strict-mmap import fail for windows the export run never reached. GGML_OPENVINO_NPU_DECODE_PREBUILD replaces GGML_OPENVINO_NPU_DECODE_BUCKET: 0 (default) keeps decode over the whole context; 1 sizes decode by 1024*2^k KV windows and the call that builds or imports the prefill/decode pair also builds or imports the decode graphs of every other window up to the context, from the pair's graph and compute params. Requests are still created on first use. Window building is split out of the request lookup so both paths share it, and window ports are checked against the pair's decode ports.
* Revert "examples: add llama-chat-replay"
This reverts commit 4eded2c52c2df9fa1251673a523f8c7cded20d7c.
* ggml-openvino: initialize the new compiled-graph and build-args fields
ov_compiled_graph and static_build_args gained members, but three brace
initializations still listed only the old ones. GCC reports
-Wmissing-field-initializers there, and CI builds with warnings as errors.
Set the fields by name, and pass {} for weights_path, which the window
build sets afterwards.
* ggml-openvino: fix gemma-4 when only the output rows reach the KV-shared layers
With one sequence, gemma-4 now drops the rows that are not output before
its first KV-shared layer. Those layers take their positions through
GET_ROWS(inp_pos, out_ids), and the backend mishandled three things:
- The positions are reshaped from [n] to [1, n]. That matched RESHAPE
case 1, which marks the wrong dim dynamic. Use case 3, which keeps
the source dim as dim 1.
- ROPE shared one sin/cos table per frequency setting, so the KV-shared
layers reused the table built from the full inp_pos. Add the position
input to the cache key.
- compute_llm_params took input_len from the last attention op. That is
now a KV-shared layer with only the output rows, so the static (NPU)
path padded prefill to the wrong length. Take the largest query length.
Stateful execution still fails on the per-layer embedding view and is
not covered here.
* ggml-openvino: keep the inp_pos layout for gemma-4's gathered positions
A batch can have no output rows, for example a prompt chunk that
llama-cli processes before its last tokens. gemma-4 then reshapes an
empty position gather for the ROPEs of its KV-shared layers. The size
checks of the RESHAPE cases cannot place a dim of size 0, so the
positions got the wrong layout and the ROPE multiply failed on CPU and
GPU. A RESHAPE that feeds the ROPE positions now always takes the
inp_pos layout [1, 1, 1, n].
* ggml-openvino: enable stateless GPU execution for MLA models
The per-head MLA weights (`attn_k_b` / `attn_v_b`) are 3D quantized tensors. Their `MUL_MAT` operations were falling back to the CPU, splitting the graph at every layer and causing GPU execution to fail.
Accept these tensors in the GPU support check and address the following issues exposed by enabling GPU execution:
* Quantization: Use exact zero points only for MoE expert weights. U4 weights with F16 zero points fail in batched GPU MatMul.
* Dynamic VIEW dimensions: Select the correct dimension when a size-1 dimension shares its stride (e.g. `k_pe`), and match KV cache views by stride.
* KV cache V view: Translate the V view of the K cache as a reshape while preserving the dynamic cell axis.
* Flash Attention: Accept the MLA Q (`CONCAT`) pattern and V views derived from another view.
Limitations: The GPU plugin fails on batched compressed weights with static shapes. Keep these unsupported in op tests and `GGML_OPENVINO_FORCE_STATIC`.
* openvino : use writable recurrent cache inputs
Assisted-by: Codex
* ggml-openvino: fix decode crash for Qwen3.5 4B
When the dynamic size is 1, the code now keeps the innermost match. When the size is greater than 1, it still uses the MLA rule, so the MLA k_pe fix still works for multi-token graphs.
* ggml-openvino: keep the GDN state in the OpenVINO layout
Adds an opt-in switch, GGML_OPENVINO_GDN_NATIVE_STATE. When it's on, the GATED_DELTA_NET recurrent state stays in the layout the OpenVINO GatedDeltaNet op uses, [B, H_v, key_dim, value_dim], instead of the ggml layout [B, H_v, value_dim, key_dim]. That removes the two state transposes the translator adds on every call.
* ggml-openvino: support Ling-mini-2.0 group-routed MoE
Ling-mini-2.0 selects experts by group, but the group-selection graph
used ops that were not supported by the backend. This left 370 nodes on
the CPU and reduced decode to ~1.5 tok/s.
* Translate the batched 4D `GET_ROWS` gather used for group scores.
* Support `SET_ROWS` for compute tensors, not just KV caches.
* Preserve the dynamic token dimension in `FILL`.
* Skip the `DIV` repeat when dimensions already match.
* Use the tracked token dimension for `RESHAPE` / `GET_ROWS` op cases,
so a graph compiled during one-row warmup also works for ubatches
with more output rows.
* ggml-openvino: disable output staging
The backend stages small outputs (up to 4 MB) in a USM host buffer
and binds larger outputs directly to the ggml buffer. When the number
of outputs changes between calls to the same graph, as in
llama-perplexity, an output port can switch from USM host memory to
plain host memory.
The OpenVINO GPU plugin mishandles this switch for some dynamic
outputs, causing out-of-bounds writes on subsequent calls. This
results in `CL_OUT_OF_RESOURCES`. For example,
`llama-perplexity -c 2048 -ub 512` fails on the fourth ubatch with
Gemma-2, Gemma-4, Phi-3.5, Phi-4-mini, and Qwen3.5.
Disable output staging until the plugin issue is fixed, so each output
is always bound the same way. Input staging remains enabled.
* ggml-openvino: match attention masks by identity
Gemma-4 prunes non-output rows before its KV-shared layers. These
layers select the corresponding mask rows with GET_ROWS(mask, out_ids).
Mask parameters were named by their consumers, but GET_ROWS is not
recognized as a mask consumer. As a result, the sliding-window layers
selected their rows from the full-attention mask instead of the
sliding-window mask. This either caused the mask add to fail
broadcasting beyond the sliding-window cache (d1024 and above with
-ub 512) or allowed attention beyond the sliding window at shorter
contexts.
Track graph-input attention masks by identity and assign them the
same parameter name for every consumer. The SWA pre-pass now keeps
each layer's own attention mask, as the KV-shared layers read the
same cache with mask rows derived from it.
* ggml-openvino: preserve input mask names and window sizes
KV-shared layers appear at the end of the graph, and their derived
masks are not graph inputs. Processing these masks overwrote the
names and sliding-window sizes recorded from the original input
masks with empty values.
Preserve the names and window sizes of graph-input masks so the
attention dimensions continue to be derived from the original
mask shapes.
* ggml-openvino: accept empty tensors in stride checks
In Gemma-4 batches without output rows, the attention tensors of
KV-shared layers are empty. Their strides beyond the empty
dimension are zero, causing has_strides_on_element_grid() to reject
them.
This sent SOFT_MAX, the KQV MUL_MAT, and CONT to the CPU with
-fa off. It also caused the per-layer input crossing the graph
split to be bound with the wrong layout.
Accept empty tensors in the stride check. Their contents are never
read, so the remaining strides do not need to satisfy the element
grid constraints.
* ggml-openvino: classify transposed V caches using v_trans
With -fa off, the V cache is transposed. When the attended length
is the whole cache, as with a prompt that fits in one ubatch, its V
view is contiguous and was mistaken for the K cache layout.
The K layout path does not keep only the rows of the layer's own V
size. Gemma-4 allocates 512 V rows in every layer, the size of its
widest layer, and its sliding-window layers use the first 256. These
layers then attended with all 512 rows, and prompts fitting in a
single ubatch failed with a broadcast error.
Use v_trans to identify the V cache layout instead of relying on
contiguity.
* ggml-openvino: use per-layer Split in stateful execution
Stateful execution was excluded from the per-layer Split path
used for Gemma-4's per-layer embedding slices. The generic
re-slicing path therefore cut the token axis instead of the layer
axis.
Allow stateful execution to use the per-layer Split and adjust
its axis by the rank dropped during stateful graph construction.
* ggml-openvino: centralize stateful KQ mask lookup
Stateful execution rebuilds the KQ masks from positions. The
existing replacement logic applied to every mask input of
FLASH_ATTN_EXT and SOFT_MAX, including masks derived through
GET_ROWS.
For Gemma-4's KV-shared layers, this replaced the row-selected mask
with the full rebuilt mask, losing the output-row selection. As a
derived mask's name does not name the sliding window, these layers
also got the full-attention mask instead of the sliding-window one.
Add get_kq_mask_input() to replace only the actual mask parameters.
Use it for FLASH_ATTN_EXT, SOFT_MAX, and GET_ROWS, so the derived
masks select their rows from the rebuilt masks, and the lookup is no
longer duplicated.
* ggml-openvino: Removed duplicated if clause.
* fix flash_attn with sink conversion to OV SDPA
* ggml-openvino: drop the unused sink-without-mask path in FLASH_ATTN_EXT
supports_op already rejects FLASH_ATTN_EXT with sinks but no mask, so the
translator never sees a 4-input node whose fourth input is the sink; the
name check could only reach the assert after it. Restore the input-count
detection, fetch the sink where it is used, and format the sink reshape.
* ggml-openvino: fix dimensions bug in stateful recurrent-state remainder writeback
* ggml-openvino: move scalar scales before weight MatMuls on GPU and NPU
GPU and NPU run the graph in f16. Granite multiplies the attention and FFN
outputs by residual_scale right after their output projections. For some
tokens the ffn_down output is far above the f16 maximum (65504), so it
becomes inf and every later layer turns NaN: granite-4.0-350m and
granite-4.0-1b answer with repeated tokens on GPU, and llama-perplexity
reports NaN.
Add MoveScaleBeforeMatMul. It rewrites Multiply(MatMul(x, W), s), with a
scalar constant |s| < 1 and a constant W, as MatMul(Multiply(x, s), W). The
result is the same up to rounding and the MatMul output stays in range.
Register it for GPU and NPU; CPU runs in f32.
* ggml-openvino: translate convolution rollback snapshots
Assisted-by: Codex
* ggml-openvino: run GDN rollback snapshots in OpenVINO on CPU
Speculative decoding that rolls back recurrent state (MTP, EAGLE3, DFlash)
builds GATED_DELTA_NET with K = n_rs_seq + 1 state snapshots and copies them
into the recurrent cache. supports_op accepted the snapshots only on GPU, so
on CPU the GDN ran in ggml while the copy, whose destination is the OpenVINO
cache buffer, ran in OpenVINO. The copy read the packed ggml GDN output
wrongly and stored wrong states without an error: test-llama-context
rollback failed and MTP accepted no drafts.
translate_gdn_snapshots builds plain GatedDeltaNet ops on token prefixes and
does not depend on the device. Accept the snapshots on every device with
dynamic shapes and keep rejecting them only for static execution (NPU and
GGML_OPENVINO_FORCE_STATIC). Update the two doc sentences that said the
snapshots need the GPU.
* ggml-openvino: abort when GDN snapshots would run in ggml
The snapshot copy writes into the recurrent cache in the OpenVINO buffer, so
it always runs in OpenVINO. When OpenVINO rejects the GATED_DELTA_NET that
produces the snapshots (per-key gating as in Ling-3.0, a permuted v, static
execution), the GDN runs in ggml and the OpenVINO side reads its packed
output wrongly. Depending on the shapes, this silently corrupts the cache or
fails inference with a reshape error. Abort when the graph is scheduled,
with the reason.
* ggml-openvino: share one weights file and NPUW bank across all NPU prefill/decode pairs
With GGML_OPENVINO_NPU_WEIGHTLESS_BANK, every prefill/decode pair (one per KV window, e.g. llama-server's startup probe and a long prompt) wrote its own weights file and used its own NPUW bank, so import kept the weights resident once per pair (+2.1 GiB for phi-4-mini). Pairs now use a model-level <wfp>-<wfp>.weights.bin and bank; the weights file records a layout/content hash and is reused when identical, with a per-pair file and bank as fallback. Old caches are invalidated.
* ggml-openvino: export and preload NPU prefill blobs for every KV window
With head-major KV the prefill also runs at a KV window picked by prompt length, so an import-only run failed for prompts whose window was never exported. After a pair is exported, also export the prefill model of every other 1024*2^k window (weights resolved through the pair's shared weights file), and on import preload them into a process-wide cache of imported blobs that pair and decode-window builds reuse. Cache the source fingerprint of weights outside OpenVINO buffers, which made each model fingerprint cost ~9 ms. A window switch now costs ~60 ms instead of ~250 ms.
Rename GGML_OPENVINO_NPU_DECODE_PREBUILD to GGML_OPENVINO_NPU_BLOB_PREBUILD since it now covers prefill as well as decode windows.
* ggml-openvino: drop invalid prefill blobs and clear blob caches on backend free
The prefill preload imported each KV window's cached blob without a try, so
one blob that failed to import failed the graph compute, and the blob stayed
in the cache for every later run. Remove the blob and its manifest and log a
warning, as the pair import does; the window is built when a prompt first
needs it.
The imported-blob map and the fingerprints of weights outside OpenVINO
buffers lived until the process exited. A process that loads models one
after another kept every compiled model resident, and a later model whose
weight landed at a freed weight's address reused that weight's fingerprint.
Clear both with the shared compiled-model cache when the last backend is
freed.
Also apply clang-format to the changed lines.
* ggml-openvino: enable NPU KV windows by default
With GGML_OPENVINO_NPU_BLOB_PREBUILD unset, every NPU decode step ran a
static graph that attends the whole context, so decode was slow for any
sequence much shorter than -c. On a Lunar Lake NPU, Llama-3.2-1B decodes at
0.79 t/s with -c 4096 and 0.20 t/s with -c 16384, against 11.1 and 12.0 t/s
with KV windows. Each window adds one decode graph to compile at load: with
-c 4096, load goes from 11.1 s to 14.6 s for Llama-3.2-1B and from 61.2 s to
81.6 s for Llama-3.1-8B, whose decode goes from 0.20 to 0.82 t/s.
Default GGML_OPENVINO_NPU_BLOB_PREBUILD to 1; 0 keeps the full-context
decode graph. Update the env table and the Boolean-flag note, which said an
unset flag is always disabled.
* ggml-openvino: add POOL_1D and allow F32 EXP on GPU
* skip empty tensors
* ggml-openvino: detect the gated LFM2 conv input as a recurrent sequence
A reshape whose output reaches SSM_CONV through concat(conv_state,
transpose(x)) gets op_case 6, which keeps the active sequence count apart
from the token count, as the conv state does. The LFM2 and LFM3 short
convolutions gate x first, so the transposed tensor is
mul(view(x), view(x)) and the check missed it. The reshape got op_case 5
instead, and:
- in stateful mode, which drops the leading dim for op_case 5, TRANSPOSE
got a rank-3 input with a 4-axis order and graph conversion failed;
- with more than one sequence, the concat with the conv state failed.
Accept the gated input too. The output for a single stateless sequence
does not change.
* ggml-openvino: always stream weight requantization
Requantizing Q6_K/Q5_K weights to 8-bit (or F16 for some tensors) previously converted the entire tensor to f32 first. For large output or embedding matrices, this required 2–2.5 GB of temporary memory.
The chunked conversion path already processes 256 rows at a time and produces identical output. It was previously enabled by `GGML_OPENVINO_REDUCE_COMPILE_MEM` or `GGML_OPENVINO_MEMORY_OPTIMIZE`; use it by default whenever the target block size divides a row. Q4_1_64, which has no chunked quantizer, still converts the whole tensor.
The variable retains its other effects.
* ggml-openvino: allocate host buffers as untouched OS memory
Host buffers (weights, KV cache and compute) were allocated with `ggml_aligned_malloc` and explicitly zeroed with `memset`. This made the entire weight buffer resident before any tensors were loaded.
Allocate host buffers using `mmap(MAP_ANONYMOUS)` on Linux and `VirtualAlloc` on Windows instead. Both provide zero-initialized memory without touching the pages, allowing the OS to bring them into memory as tensors are written.
This preserves the zero-initialization required by ggml-org/llama.cpp#20857 while avoiding unnecessary memory residency during model loading.
Remove the cache-only allocation branches, the corresponding `ggml_aligned_free` path from the destructor and the `data_owned_by_ov` member it used, since they are no longer needed.
* ggml-openvino: release GGUF file pages once a weight is converted
Once `set_tensor` converts a weight into the backend buffer, its source pages in the mapped GGUF file are no longer needed for that buffer. Release these clean pages using `madvise(MADV_DONTNEED)` on Linux and `VirtualUnlock` on Windows. The mapping itself remains intact, and pages can be loaded again from the page cache or file if needed.
Only release whole pages within tensor ranges belonging to read-only file mappings. Skip buffers that wrap the mapping itself.
Keep locked pages resident. On Linux, `MADV_DONTNEED` fails on mlocked memory. On Windows, check page lock status with `QueryWorkingSetEx` and skip ranges containing locked pages, since `VirtualUnlock` would otherwise unlock them.
* ggml-openvino: key weight constants by name for the NPU model cache
The NPU compiled-model cache (GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR without the
weightless bank) imports a blob with the freshly converted model as ov::hint::model,
and NPUW finds each constant by its WeightlessCacheAttribute bin_offset. The keys came
from a per-model counter, and a constant that already had a key kept it. Weight nodes
are shared between models: between the NPU prefill and decode models, and across the
process through the tensor extras. So a weight kept the key from whichever model
converted it first.
The decode model for a KV window is built both as the decode half of the
prefill/decode pair and as a standalone window model, under one fingerprint, and the
two builds got different keys. Importing the blob of the other build failed with
"Couldn't find Constant in cache!", and the decode model was recompiled and exported
again on every run. With GGML_OPENVINO_NPU_BLOB_PREBUILD on by default, this happens
whenever a prompt reaches a larger KV window.
Key weight constants by a hash of the weight name and their position in the weight's
subgraph, above the range of the counter. Other constants are created per model and
keep the counter.
* ggml-openvino: drop CACHE_MODE=OPTIMIZE_SIZE from the NPU plugin cache
With GGML_OPENVINO_CACHE_DIR set, the NPU compile config added
CACHE_MODE=OPTIMIZE_SIZE next to NPUW_CACHE_DIR. With it, the NPU plugin
compiles the NPUW submodels with weights separation, which fails for them:
with NPUW folding, "No WeightlessCacheAttribute has been found in any of the
model's Constant nodes"; without folding, a driver compile error. So every
NPU run with GGML_OPENVINO_CACHE_DIR failed on its first compile.
Keep the default cache mode. NPUW_CACHE_DIR still caches the compiled
submodels.
* ggml-openvino: load each self-contained NPU blob once per process
A self-contained blob (GGML_OPENVINO_SELF_CONTAINED_BLOB) embeds its NPUW
weights bank, and NPUW cannot load a blob into a bank that already holds
those weights: Bank::read_and_add_tensor asserts. Two cases did this on
every run that imported the cache:
- Pairs that differ only in the prefill model share the decode model and
its bank. A long prompt builds such pairs, and each one imported the
decode blob again. The import failed, so the decode model was compiled
and exported again for every pair, until the run ran out of memory.
- The decode model of each KV window uses the bank of the pair's decode
model, so the import of a window blob failed and stopped the run.
Keep each self-contained model in the process once it is imported or
compiled, and reuse it for every pair with the same fingerprint. Turn the
KV windows off for self-contained blobs; the weightless bank keeps them.
* ggml-openvino: update the Docker image GPU drivers to 26.40.40196.8
The OpenVINO image pins the Intel GPU drivers at compute-runtime
26.35.39758.10. With that driver, gemma-4-E4B hangs on the Lunar Lake
GPU. With 26.40.40196.8 it runs.
Take the GPU driver pins from the compute-runtime 26.40.40196.8 release:
IGC v2.42.6, compute-runtime 26.40.40196.8 and gmmlib 22.10.2.
* ggml-openvino: fix long NPU prompts failing with KV slicing
With GGML_OPENVINO_KV_SLICE, on by default, a static prefill is compiled
at the attended KV length that llama reports for the ubatch. That length
grows with every ubatch of a long prompt, and each new length compiled
another prefill/decode pair, with GGML_OPENVINO_NPU_BLOB_PREBUILD (on by
default) also its decode KV windows. The shared compiled cache keeps
every pair until the backend is freed.
Each loaded prefill model keeps NPU memory for its inferences. On a Lunar
Lake NPU, a prefill model loaded next to two or three others cannot get
it: DRM_IOCTL_IVPU_BO_CREATE fails with "Buffer size is too big", and
pfnAppendGraphExecute returns ZE_RESULT_ERROR_UNINITIALIZED. With -c 4096
and the default -ub 512, a 2200-token prompt failed on smollm2-1.7B,
Phi-4-mini, Ministral-3b and gemma-3-4b.
- Round the attended length of a sliced static prefill up to a power of
two, clamped to the context. A growing sequence now compiles a prefill
model when its attended length doubles, not at every ubatch. The added
rows are masked, so the output does not change.
- When only the prefill attention size changes, the new pair takes over
the decode models (base and KV windows) of the entry's current pair,
because they do not depend on that size, and the current pair is
dropped before the new prefill model compiles. A pair that another
backend may use stays cached. Self-contained pairs are exported as one
blob pair, so they keep their own decode models.
- The taken-over KV window that covers the next decode position becomes
the base decode model, as in a pair built at that length, so the first
decode step does not have to create an infer request for it.
* ggml-openvino: pre-scale the GDN q input by default on the NPU
The NPU GatedDeltaNet kernel writes its output in FP16 and flushes
values below 6.1e-5 to zero, and the RMS norm after it amplifies the
loss. GGML_OPENVINO_GDN_Q_SCALE lifts the output above that floor, but
it was off by default. On a Panther Lake NPU with the compiler bundled
in OpenVINO, Qwen3.5-2B scores a perplexity of 14.77 without it and
13.00 with it (12.93 for the same weights on CPU), and its greedy chat
no longer loops while thinking.
- Enable the pre-scale by default with the factor 64.
GGML_OPENVINO_GDN_Q_SCALE=0 turns it off.
- Scale the eps of the norm only when the GDN that feeds it is
pre-scaled. A norm that reads the GDN output directly got the scaled
eps on every device, while q is scaled only on the NPU. With the
variable set, Qwen3.5 gave wrong output on CPU and GPU.
- Add the factor to the model cache fingerprint of NPU graphs with a
GATED_DELTA_NET, so a cached model is not reused across settings.
- docs: update the GGML_OPENVINO_GDN_Q_SCALE row and note.
* openvino-ggml: Decompose the GDN for LNL
1. Automatically decompose LNL GDN prefill into chunked MatMuls; retain the native op for decode.
2. Isolate lowering selection, fix positive-gate handling, and add overflow guards.
3. Update cache identity and add chunk-boundary regression cases.
* ggml-openvino: fix GDN decomposition in decode and on long prompts
Two problems remained with the GATED_DELTA_NET decomposition on a Lunar
Lake NPU.
The decomposition covered only multi-token (prefill) graphs, and decode
kept the GatedDeltaNet op. Under Linux the decode kernel loses the
device as well: after a successful prefill, the first generated token
failed with ZE_RESULT_ERROR_DEVICE_LOST in llama-completion, llama-bench
and llama-server.
The chunk masks were new constants in every converted model. The NPUW
weights bank is shared by all compiled models (NPUW_WEIGHTS_BANK=shared)
and identifies a constant by its data pointer, also after the model
that owned it is gone. When a longer sequence needs a second prefill
model, its masks can get the addresses of freed masks of the first one
and are then read as those. The Qwen3.5-2B perplexity over 512 tokens
changed between 12.99 and 56.94 from run to run, and a 1900-token
prompt gave garbage.
- Lower a single-token GATED_DELTA_NET to one recurrence step when the
decomposition is on: the chunk formulas for a chunk of one token,
where P = 1 and the decay is 1.
- Keep one instance of each chunk mask for the whole process.
- test-backend-ops: add a single-token case with two sequences, repeated
heads and a positive gate.
- docs: GGML_OPENVINO_GDN_DECOMPOSE covers single-token graphs.
* openvino-ggml: Enhance the safe env parsing
* ggml-openvino: keep the GDN q pre-scaling on when its variable is unset
ggml_openvino_gdn_prescale_enabled() returned false when
GGML_OPENVINO_GDN_Q_SCALE is unset, which switched the default-on
pre-scaling off. Qwen3.5-2B on a Panther Lake NPU went from a perplexity
of 13.00 back to 14.77.
- Treat an unset variable as enabled. Zero, negative and non-numeric
values still turn the pre-scaling off.
- Comments: ggml_openvino_getenv_int() no longer goes through std::atoi.
- docs: a boolean flag with a non-integer value is no longer always
disabled; a flag whose default is 1 stays enabled.
* ggml-openvino: use the PLUGIN compiler for GDN graphs on the NPU
The NPU driver compiler, the default, gives wrong output for graphs
with GATED_DELTA_NET (Qwen3.5). Qwen3.5-2B on a Panther Lake NPU answers
with unrelated text and has a perplexity of 16.77, against 13.00 with
the compiler bundled with OpenVINO. Other models stay on the driver
compiler, which builds faster prefill kernels.
- Compile static graphs with GATED_DELTA_NET with
NPU_COMPILER_TYPE=PLUGIN, unless GGML_OPENVINO_NPU_COMPILER_TYPE or
NPU_COMPILER_TYPE in GGML_OPENVINO_NPU_CONFIG sets the compiler type.
- Add the choice to the model cache key of these graphs, so blobs from
the driver compiler are not imported.
- docs: describe it in the GGML_OPENVINO_NPU_COMPILER_TYPE row.
* ggml-openvino: fix gemma-4 E2B on the Lunar Lake NPU at small contexts
The NPU 4000 compiler computes ScaledDotProductAttention wrongly when K
has one KV head and is square, that is when the attended rows equal the
head size. gemma-4 E2B has one KV head, 512-wide full-attention heads
and 256-wide sliding-window heads, so a context of 512 tokens or less,
and a prefill that attends 256 or 512 rows, gave wrong output.
For that shape on NPU 4000, build the attention from MatMul, Softmax and
MatMul, as the transposed-V path already does. Other shapes and devices
keep the op.
* ggml-openvino: support multiple GPUs with -sm layer
* ggml-openvino: check GGML_OPENVINO_* variables and reorganize the docs
The backend ignored a misspelled GGML_OPENVINO_* name and a value that
was not valid, and the docs listed every variable in one table.
Code:
- One table holds the known variables and their types. A value that is
not valid stops the backend with a message that names the variable and
the accepted values. An unknown GGML_OPENVINO_* name gives a warning.
- With log verbosity 4, one line shows the variables in effect.
- GGML_OPENVINO_GPU_CONFIG and GGML_OPENVINO_CPU_CONFIG pass plugin
properties to the GPU and the CPU, as GGML_OPENVINO_NPU_CONFIG does
for the NPU. If the plugin rejects a property, the error names the
variable.
- GGML_OPENVINO_STATEFUL_EXECUTION gives a deprecation warning.
- The key of the compiled model cache covers the two new variables,
GGML_OPENVINO_KV_SCATTER_ELEMENTS and
GGML_OPENVINO_DISABLE_ELTWISE_RANK_ALIGN.
- GGML_OPENVINO_ENABLE_CACHE, which no code read, is no longer
registered.
Docs:
- docs/backend/OPENVINO.md: Quick Start, collapsible blocks, runtime
configuration in groups (basic, plugin properties, advanced tuning,
compiled model cache, deprecated), updated validated models and known
limitations, corrected stale entries.
- dev_doc/OpenVINO-ENV-VARS.md: the variables for development and the
NPU properties that the backend sets.
* docs : add LFM2.5 and Ling-mini-2.0 to the OpenVINO validated models
* ggml-openvino: rebuild converted weights in get_tensor
set_tensor converts a quantized weight in place into the extracted or
requantized layout, so get_tensor returned bytes that are not in the ggml
format. A reader such as the CPU reference in test-backend-ops then
computed on garbage and produced NaN (MUL_MAT cases with weights=1).
get_tensor now rebuilds the ggml bytes for such a weight: it evaluates the
weight node on the host and quantizes the values back to the tensor type.
The result is close to the original data, not bit-exact. Other reads are
unchanged, and the inference path does not call get_tensor on weights.
* ggml-openvino: drop the GDN test cases from test-backend-ops
Restore tests/test-backend-ops.cpp to the upstream version.
* docs : improve the OpenVINO backend docs
* docs : restore the OpenVINO intro and the development note
* ggml-openvino: fix slow performance of MoE models with requantize and workaround to use fused MOE Compressed
* ggml-openvino: document the fused GPU MoE op limits
Converting K-quant MoE experts to Q4_1_64 on GPU lets the plugin use its
fused MoE op, which runs in f16 only. granitemoe models give wrong output
with it, and an f32 inference precision hint fails to compile. Document
GGML_OPENVINO_MOE_OP=0 as the way to turn the op off, and that
llama-perplexity with MoE models needs -b equal to -c.
Also split the long TODO comment over two lines.
* ggml-openvino : FILL for in-place clear of recurrent cache, sticky NaN fix
* restore this part
* ggml-openvino: bump the model-cache version for the recurrent-cache reset change
Recurrent-cache resets now clear slots with Select instead of Multiply,
so a NaN in a cleared slot no longer survives. Compiled models cached
before this change keep the old graph, so bump the fingerprint version
to rebuild them. Also note how SCALE and FILL store their parameters.
* ggml-openvino: read rope_factors_short/long as weight constants
create_weight_nodes() registers the ROPE frequency factors as
"rope_freqs.weight", but set_input_output() and compute_model_inputs()
kept the tensor's own name. For models whose factors are named
rope_factors_short or rope_factors_long (Phi-3.5, Phi-4, MiniCPM with
LongRoPE) the factors became a Parameter fed from the host weight buffer.
That buffer is empty in cache-only mode and after the host weights are
released, so these models gave garbage output with
GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY, GGML_OPENVINO_RELEASE_WEIGHTS
and GGML_OPENVINO_MEMORY_OPTIMIZE.
Use the same name in all three places, so ROPE always reads the weight
Constant. The graph changes, so bump the model-cache version to 7.
* ggml-openvino: import cached models from a memory-mapped tensor
A stream import reads the whole compiled blob into a heap buffer while
the plugin builds its own copy of the weights. On CPU this held the blob
twice at the peak. Import from ov::read_tensor_data() instead, which maps
the file. This lowers the CPU cache-hit and cache-only peak memory; GPU
memory is unchanged.
* ggml-openvino: do not write CPU model-cache blobs for MoE models
The CPU plugin exports its MoE op (GatherMatmulCompressed) but cannot
import it again with OpenVINO 2026.4 and 2026.5 ("unsupported opset").
Every later run then failed the import, compiled again and rewrote the
blob. Skip the blob for such models on CPU and print a warning.
In cache-only mode no blob exists for them, so the generic advice to run
once without cache-only cannot help. Print an error that says cache-only
does not work for MoE models on CPU instead.
* ggml-openvino: fix GGML_OPENVINO_DEBUG_INPUT on GPU
GPU inputs can be device (USM) tensors, whose data() throws, so every
GPU compute failed with GGML_OPENVINO_DEBUG_INPUT=1. Print a host copy
of remote tensors instead.
* docs : add OpenVINO memory optimization tips and MiniCPM-V-4.5
Add a "Tips for Memory Optimization" section to OpenVINO-ENV-VARS.md
with examples for the spill directory, the compiled-model cache, host
weight release on GPU, CPU MoE compile memory and the NPU weightless
cache, and link it from OPENVINO.md. Add MiniCPM-V-4.5 to the validated
models.
* ggml-openvino: run MoE models with F16 and BF16 weights on GPU
On GPU, supports_op rejected MUL_MAT_ID with non-quantized expert weights
and GET_ROWS with BF16 data, so the scheduler ran those ops on the CPU.
For F16 MoE models the CPU MUL_MAT_ID then read the strided top-k view as
expert ids and crashed. For BF16 models the token embedding came from the
CPU and reached the OpenVINO graph as a static-shaped input, which failed
once the token count changed. Both ops translate correctly on GPU, so
remove the two rejections.
* ggml-openvino: fix BERT embedding models on CPU and GPU
BERT models (bge-m3, nomic-embed-text-v1.5) failed on every device:
- A VIEW of a weight with a GGML_OP_NONE source was marked as having a
dynamic token dimension, a rule meant for KV cache views. The token type
row view of BERT was then never sliced, and the whole two-row table was
added to the activations. Skip the rule for model weights.
- A row range view of a 2D weight now becomes a Slice of the weight rows,
as GET_ROWS already does.
- Attention without a KV cache pads K and V up to the mask width, which
depends on the token count. PAD baked the amount from the first graph,
so later token counts gave mismatched shapes. Compute the end padding
from the mask width at run time.
The NPU still fails for these models.
* docs : add bge-m3 and nomic-embed-text-v1.5 to the OpenVINO validated models
---------
Co-authored-by: Sabnis, Omkar <omkar.sabnis@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
Co-authored-by: Anisha Kulkarni <anisha.dattatraya.kulkarni@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yu, Zijun <zijun.yu@intel.com>
Co-authored-by: Proshunin, Nikolay <nikolay.proshunin@intel.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
* ui: Llama UI's own Storybook in ui/lib/ui
The library gets its Storybook (`npm run storybook`, `npm run
build-storybook`) with a story for everything it holds, under Llama UI:
the foundations, the 29 base components (as actions, forms, display,
overlays and navigation), the app components (ActionIcon, SearchInput,
SyntaxHighlightedCode), the Model UI and the native app mockups. The
stories import the package by its published names, aliased onto src/lib,
run as tests under `npm test` (the storybook project in vite.config.ts),
and are type-checked with the rest of the package. `npm run dev` starts
Storybook next to the SvelteKit check page; the toolbar switches the
color mode and the surface theme, and pages that are mostly prose are
.svx, through mdsvex.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: a Storybook for Llama Chat
Llama Chat's Storybook gets an introduction, Agents › Design guidelines
(DESIGN.md), Llama UI's stories as the library has them, and Llama Chat:
its own foundations (safe areas, the splash screens), then one section per
src/lib/components/app/<area> (Chat with the chat screen, form, messages,
attachments and tabs; Content, Models, MCP, Navigation, Settings, Dialogs,
Forms, Actions, Badges, Misc, Servers and PWA), each accessibility story
next to its component's. It replaces the flat pages it had (chat message,
chat form, markdown content, the PWA refresh alert and their fixtures,
and the ChatForm single-row exploration).
The stories run on the new tokens and surfaces with the color mode and
surface theme in the toolbar (by mode-watcher's key, the ColorMode and
SurfaceTheme enums and applySurfaceTheme), and pages that are mostly
prose are .svx in an editorial frame (AGENTS.md says how to write them
and where a story goes). The story tests run Llama Chat's stories and
leave the library's to the library. The brand and communications pages
are in ggml-org/llama-brand; the logo files come from @ggml-org/llama-brand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let the ChatScreen story store its reply before the next one starts
The ContinueConversation story ended while its reply was still being stored,
and the next story reset the stores under it. It now waits for the reply to
finish.
Assisted-by: Claude (Fable 5.2)
* ui: publish the Storybook to its Hugging Face Space on every UI change
A new UI Storybook workflow builds ui/apps/chat's Storybook whenever
ui/apps/chat or ui/lib/ui changes on master (or on demand) and uploads it
to the static Space ggml-org/Llama-UI-storybook, with a Space card naming
the commit it was built from. Files the new build no longer has are
removed; the Space's .gitattributes stays. It runs only in
ggml-org/llama.cpp, takes the UI's Hugging Face token
(HF_TOKEN_UI_STATIC_OUTPUT), and the Space can be changed with the
HF_SPACE_UI_STORYBOOK variable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: run the library's tests, its stories among them, in CI
The Library checks job type-checked and packed ui/lib/ui but never ran its
tests, so the library's stories, which now hold every library component's,
went untested in CI. It installs Chromium and runs `npm test`.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw a floating panel's rim on its border, so its edge is one line
Menus, popovers, dialogs and the other matte-float panels carry a 1px border that the
surface made transparent, and drew their inner highlight as an inset shadow. An inset
shadow sits inside the border, so the 1px of fill between it and the outer ring showed
as a second edge. The border now draws the highlight, right inside the ring.
Assisted-by: Claude (Opus 5.5 Medium)
The implicit GEMM conv2d kernel recovers the channel and the kernel
position of each reduction index with three integer divisions by
runtime divisors, once per loaded row of every tile. Outside the 3x3
and 1x1 specializations this integer division bounds the kernel:
removing it halves the time on long reductions.
Divide with the precomputed magic numbers of fastdiv instead. The 3x3
and 1x1 specializations keep their compile time divisors.
test-backend-ops -o CONV_2D: 2006/2006 on CUDA0.
CONV_2D, F16 kernel, us/run on an RTX PRO 6000:
[2054,1,768] k [7,1,768,768] 221 -> 138
[16390,1,384] k [7,1,384,384] 433 -> 266
[65542,1,192] k [7,1,192,192] 420 -> 271
[8240,1,256] k [11,1,256,256] 165 -> 108
[19,19,256,16] k [4,4,256,4096] 1672 -> 1076
* server: fix lora and recurrent
* tests: run the hybrid SWA checkpoint test with one token per ubatch
With and without the cache, the prompt was not split into the same
ubatches, and the generated tokens are decoded one at a time while the
uncached run processes them in a batch. On CUDA this moves the top
logprobs of the tiny model by up to 0.03 and swaps near ties, so the
test failed while the cache behaved as expected.
With one token per ubatch both runs go through the same kernels and
the output is compared exactly.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ui: publish @ggml-org/llama-ui from CI when its version moves
ui/lib/ui reaches npm only when someone remembers to publish it by hand, which let 0.3.0
sit a day behind master. The version in its package.json becomes the release: a commit
landing on master with a version npm does not have gets built, linted and published with a
provenance attestation, through OIDC rather than a stored token, as release-publish.yml
already does for GitHub releases. A library change carrying no bump publishes nothing and
says so, on the PR and on the run.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: bump @ggml-org/llama-ui to 0.4.0
The library on npm, published before the design system PR merged, holds no entry points for
the Model UI, the app components or the shared constants and enums, and no dependency on
@ggml-org/llama-brand, whose colours the primitives now build on. Consumers import the new
subpaths and get ERR_PACKAGE_PATH_NOT_EXPORTED.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: name no environment for the Llama UI publish job
Creating a repository environment needs admin rights the UI maintainers do not hold, so a job
that names one cannot be set up by them. The job works without it: npm's trusted publisher
matches on the repository and the workflow file, and its environment field stays blank to
match. Nothing else in the job depends on an environment.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: address review on the Llama UI publish job
- the advisory version check leaves ui.yml for the publish workflow, on a pull_request trigger
whose path filter keeps it off PRs that touch only Llama Chat: a chat-only PR was told to
bump a library it did not change
- publish and verify run on master only. The trusted publisher matches on the repository and
the workflow file, not the branch, so a dispatch from a PR branch - anything with write
access can start one - could publish code no one reviewed. With no environment, this was the
only branch protection in place.
- the registry read after a publish retries six times rather than sleeping once, so a release
the registry is slow to serve is not reported as a failure
- a build metadata suffix (1.2.3+build) is not a prerelease, and versions order by semver's
prerelease rules, so 0.4.0-beta.1 behind a released 0.4.0 is refused instead of shipped to next
- the summary step passes its values through env like the others, actions: read is gone, and the
concurrency comment claims only what the setting does
- npm older than 11.5.1 is refused up front, as it cannot publish with an OIDC token
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* ui: note the two trusted-publishing traps in the Llama UI README
A publisher names a workflow file that must already be in .github/workflows, so it is added
after this workflow lands rather than before, and one created now publishes only through
staged publishing until npm publish is permitted for it. Provenance comes free with either,
and only on a GitHub-hosted runner.
Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)
* Update .github/workflows/ui-lib-publish.yml
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* jinja: impl scope-chain key lookup
* add test for call_statement lambda
* chat: add tts_dict for kokoro G2P template
* more testing
* value_float_t range check
* no json-in-meta, add llama_model_meta_arr_str/_n
* mtmd: add kokoro tts support
* detailed guide about G2P and text norm port
* refine G2P
* clean up
* migrate batch api
* nits
The fast CONV_2D path (implicit GEMM with MMA, or cuBLAS for
pointwise convolutions) required an F16 kernel. F32 kernels fell
back to the naive direct convolution, which is much slower than
im2col + mul_mat on large channel counts.
Convert F32 kernels to F16 in a pool buffer, as is already done for
the input, and take the same path with F32 accumulation. The Metal
implicit GEMM kernel already stages both operands as F16.
* Added support for Qwen3-TTS ICL/reference mode
Assisted-By: Claude Opus 5.5
* fix correctness
* attach transcription to bitmap
* nits
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* sampling : fix crash when both ignore_eos and grammar are on
* server : rework, throw 400 when grammar and ignore_eos are on
* sampling : move the check into the sampler (review suggestion)
* tests : remove redundant tests
As discussed in #31955, the live preview is now opt-in. With it on, the editor renders blocks as widgets, and moving the cursor down from the first line of a prompt with a few blocks (a heading, a list, a code block) jumps to the last line; with it off, the cursor moves one line at a time. Users with a saved config keep their current value, since the full config is stored in localStorage.
The message input also gets spellcheck, autocorrect and autocapitalize back. CodeMirror turns them off by default, so they were off with the preview on or off; the former textarea and contenteditable inputs had the browser's defaults.
* llama : keep the k-pool layout prefix on sequence edits
Assisted-by: Claude Code
* llama : address review of the k-pool layout update
* Append and edit share one branch, using stale > sq.pos_min.
* LLAMA_KPOOL_VERIFY=1 checks every update against a full rebuild. The scan moved into kpool_scan() so the check can reuse it.
* The test turns the check on for its rollback context, skips -sm tensor, and drops kimi-k31.
* Two comments that were no longer true are fixed.
Assisted-by: Claude Code
* llama : check the k-pool layout update in debug builds instead of LLAMA_KPOOL_VERIFY
* llama : remove the k-pool layout verification
* llama : inline the k-pool scan again
* remove dangling new line
* do not search twice
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
* ui: bring @ggml-org/llama-ui into ui/lib/ui
The design system package (tokens, fonts, surfaces, components and Model UI), so far in the temporary ggml-org/llama-ui
repository at edaf518, moves to its home in ui/lib/ui, built with @sveltejs/package and published as @ggml-org/llama-ui
from here. ui/lib holds the packages, one folder each. Its references to tools/ui now name ui/apps/chat, its
package.json points at this repository, and the UI workflow type-checks and packs it. ui/README.md describes the
layout.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: Llama UI design system: tokens, surfaces and brand
The design system from allozaur/docs/storybook-overhaul: three tiers of
tokens, matte glass surfaces, Llama Sans and Llama Mono, and the base
components on the new tokens (Tabs and GlassKey among them). It also
restyles the stack's features with it, replacing the plain stand-ins they
used until now. DESIGN.md is generated from the tokens. The Storybook that
documents it is the next PR.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: have Llama Chat use @ggml-org/llama-ui for its base components, tokens and fonts
Llama Chat kept its own copy of the design system: 29 base components, the
three tiers of tokens, the surfaces, the theme and the fonts, next to the 12
components ui/lib/ui held. The copy goes, and the library gets what it lacked.
ui/lib/ui
- the 17 base components the app had and the library did not (alert,
alert-dialog, button-group, checkbox, collapsible, context-menu, dialog,
drawer, dropdown-menu, empty, hover-card, radio-group, scroll-area,
skeleton, table, tabs, textarea), with vaul-svelte for the drawer
- the inset-group and selected-row tokens and surfaces
- sources import each other by relative path instead of $lib, which in an
app points at the app's own src/lib
- theme.css no longer imports tw-animate-css: it is a peer dependency that
the consumer imports before the theme, as it does Tailwind
- 0.3.0
ui/apps/chat
- imports the base components as @ggml-org/llama-ui/ui/<name> and cn/tv as
@ggml-org/llama-ui/utils; app.css imports the library's theme.css
- compiles the library from source, with no install or build of it:
scripts/vite-plugin-llama-ui.ts resolves its entry points and what it
imports, and svelte.config.js gives TypeScript the same paths
- src/lib/components/ui, styles/tokens, styles/surfaces.css and assets/fonts
are removed; components.json moves to the library
Builds
- CMake stages ui/lib/ui next to the app, keeping the layout of ui/, and
rebuilds when a library source changes
- the Dockerfiles copy the library's sources; the Nix derivation takes ui/
as its source
The built CSS has the same rules in the same order, apart from one
zero-specificity rule and the @font-face rules moving, plus ten utilities
and two tokens that only the library's own components use.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: AGENTS.md places Llama UI in ui/lib/ui
Llama Chat's AGENTS.md says where each part lives: Llama UI is
ui/lib/ui, and Chat imports its tokens, fonts and base components
from there.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: Llama Sans and Llama Mono from llama-brand's latest, and shell commands marked in bash code
The fonts in ui/lib/ui come from llama-brand's latest, with optical sizes
from 10 to 36. Bash code blocks mark shell commands and flags.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open submenus outside the menu they come from
A submenu was rendered inside its parent menu, so the menu's scroll box
clipped it: the glass menus' backdrop filter, and the filter and transform
of the open animation, make the menu the containing block even for the
fixed-position submenu. Dropdown and context submenus now render in a
portal, like the menus themselves.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw submenus like the menu they open from
Submenus had smaller corners than their menus, and context menus had none
of the dropdown's glass. Context-menu parts now match the dropdown's, and
both submenus take the menu's corners and height cap, with their first row
level with the row that opens them, 4px past the menu's edge. Sub-triggers
take the items' radius and pointer.
The submenus' height cap and zoom origin named the dropdown's CSS variables,
which bits-ui does not set on submenus; they now use its --bits-menu-* ones.
The chat group submenu takes the actions menu's z-[999999] layer, as it no
longer renders inside that menu.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a late current-node write off a closed or switched conversation
updateCurrentNode checked the active conversation, awaited the database write, then wrote currNode to whatever was active after it. When the conversation closed during the write it threw, and the reply ended in a streaming error dialog. When another conversation opened, that one got the wrong node.
Assisted-by: Claude (Fable 5.2)
* ui: have Llama Chat use the library's Model UI and app components
Llama Chat still kept its own copies of what the library holds above the
base components. It now takes them from @ggml-org/llama-ui too, and its
copies go.
ui/lib/ui
- ModelId gains what Chat's had: the model's own draft kind, draft
sidecars with their quant (ModelDraftSidecars, new), aliases, hideName
and a tooltip, in the same badge order
- entry points for them, like ui/<name>: @ggml-org/llama-ui/app/<group>
(actions, content, forms, misc) and @ggml-org/llama-ui/models
ui/apps/chat
- ActionIcon, SearchInput, SyntaxHighlightedCode, Logo,
ModelCapabilityIcons, ModelDownloadProgressBar and ModelDraftSidecars
come from the library; Chat's copies and the two constants only they
used are removed
- ModelId and ModelOrgAvatar stay as thin wrappers: ModelId parses the raw
id and applies the user's display settings, ModelOrgAvatar hides the
avatar while Hub metadata is off
- the svelte/* TypeScript path falls back to Svelte's types/index.d.ts:
svelte/reactivity has no file of its own, so a library source importing
it got its types from ui/lib/ui's own install when there was one
Assisted-by: Claude (Opus 5.5 Medium)
* ui: Llama UI takes the brand's colours and logo from llama-brand
@ggml-org/llama-brand is now a dependency of the library, by the same GitHub
tarball Llama Chat already installs, which now names it @ggml-org/llama-brand
too.
- primitives.css drops the 25 tokens it held copies of (the neutral
greys, sand, the skies and cinder) and builds on llama-brand's
css/colors.css instead; the skies' gradients point at llama-brand's.
Apps import that file before theme.css, as they do Tailwind and
tw-animate-css: the library's stylesheets import no package, so an app
that compiles them from source resolves it from its own node_modules
- Logo draws llama-brand's icon, in currentColor, and assets/logo.svg goes
- Llama Chat's TypeScript paths resolve llama-brand from its own
node_modules, like the library's other dependencies
The fonts stay as the library's WOFF2 copies until llama-brand ships
WOFF2 builds of them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: have Llama Chat take the library's constants, enums and helpers
Llama Chat redefined most of what the library exports next to its
components. Where the two were the same, Chat now re-exports the library's
from the files that held them, so its own imports stay as they are.
- the library gets entry points for them: @ggml-org/llama-ui/constants,
@ggml-org/llama-ui/enums and @ggml-org/llama-ui/code, and
@ggml-org/llama-ui/models carries the model data with the Model UI
- from the library now: the shared CSS classes and code-block constants,
the byte units, the memory-fit constants and minMemoryTierGb, the bit
depth labels, the capability and modality icons, labels and order,
ColorMode, TooltipSide, BooleanString, the model enums, ModelModalities,
formatParameters, highlightCode and trimCodePadding
- the library's modality order follows Chat's: vision, video, audio
- Chat keeps its own where they differ: UI_DATA_ATTRS and
ModelDownloadProgress are supersets of the library's (the latter extends
it now), and formatFileSize counts in 1024s
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the chat form's add key and model selector a tighter shadow
In light, the two keys inside the chat form, white on its white glass, took
the control elevation, whose soft spread blended into the form. They take a
new elevation-key instead: a 0.5px hairline all round and a 1px drop. In
dark they keep the control elevation. Chosen from the options in
Scrapbook › UI › Composer keys.
- semantic tokens: --elevation-key (light: the hairline and the drop; dark:
the control elevation)
- component token: --chat-form-key-shadow, for the add key and every model
selector trigger
- DESIGN.md names it next to the other elevations
Assisted-by: Claude (Opus 5.5 Medium)
Browsers already have tabs, so a second tab bar inside the page adds a layer most people don't need. The setting stays, so anyone who wants tabs can turn them on. Users with a saved config keep their current value, since the full config is stored in localStorage.
* ui: replace deprecated packages and fix npm audit findings
Swap the mdast stub for @types/mdast, move the katex override to 0.19.0,
and override glob, source-map and html-encoding-sniffer under
workbox-build and http-server so installs no longer pull deprecated
packages. Apply the non-breaking npm audit fixes to the lockfile.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: migrate Llama Chat to SvelteKit 3 and its built-in service worker
Move to SvelteKit 3, Vite 8, TypeScript 6 and Storybook 10.6. The
SvelteKit options move from svelte.config.js into sveltekit() in
vite.config.ts, $lib imports become #lib subpath imports from
package.json, and $app/environment becomes $app/env. Modules named
server.* or living under a server/ folder are server-only in SvelteKit 3,
so the llama-server components, store and enums move to llama-server
names.
@vite-pwa/sveltekit and Workbox give way to SvelteKit's own service
worker in src/service-worker, registered by usePwa(), which precaches the
app files and leaves API requests to the network. A small plugin emits
manifest.webmanifest. static/sw.js removes the worker of older builds, and
llama-server serves a self-removing service-worker.js as well as sw.js
when the built-in UI is off or replaced.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: use SvelteKit 3's update detection and Svelte 5's current patterns
A page the service worker doesn't control now learns of a new build
from SvelteKit itself: `updated` from $app/state checks
_app/version.json on navigation, on focus and every hour. It replaces
the version Llama Chat kept in localStorage and compared by hand.
Reload on the update prompt pulls in the new worker and activates it
before reloading, so one click lands on the new build, whether or not
a worker controlled the page.
Following Svelte's current guidance:
- usePwa keeps its state in $state rather than a store, and no effect
checks `browser`
- nearViewport, autoResizeTextarea and the custom CSS injection are
attachments ({@attach}) rather than actions (use:)
- class lists use arrays rather than class: directives
- the chat contexts come from createContext, typed, in place of
setContext/getContext with string keys
- ActionIconCopyToClipboard takes $props() rather than `export let`
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move to ESLint 10
ESLint 9 is out of support. Every plugin the app uses supports 10:
typescript-eslint, eslint-plugin-svelte, perfectionist,
simple-import-sort, eslint-plugin-storybook and eslint-config-prettier.
@eslint/js and @eslint/compat follow.
ESLint 10's recommended rules ask for two things the code now does:
errors rethrown from a catch keep the caught error as their `cause`,
and no variable starts with a value that every path overwrites.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: override minimatch to clear npm audit's last finding
ESLint 10.14, @eslint/config-array 0.23.5 and typescript-eslint 8.72
depend on minimatch 10, which GHSA-g3hv-75wg-m249 (denial of service
via globparts amplification) affects below 11.0.1. Their fixed releases
move to minimatch 11 but are all younger than the repo's 7-day
min-release-age, so an override pins minimatch 11.0.5, the newest the
rule allows. The three only call new Minimatch(), match() and
minimatch(), which minimatch 11 keeps. npm audit now finds nothing.
Assisted-by: Claude (Opus 5.5 Medium)
This commit removes a call of the CUDA_SET_SHARED_MEMORY_LIMIT macro as
the fallback decision was moved out of launch_mul_mat_q.
launch_mul_mat_q now has the fallback template parameter which was added
in Commit 6eddde06a4 ("CUDA: refactor MMQ
kernel configuration (#24127)").
This code was then refactored in Commit
c49edf0963 ("CUDA: fuse FFN gate/up
matmuls and GLU in MMQ with interleaved gate/up rows (#29948)") which
added the has_fusion template parameter but it kept both macro calls.
launch_mul_mat_q only launches one type of kernel now (this is now
decided earlier in the templated call chain) and each specialization
will set the limit itself.
The motivation for this is to simplify the code and improve readabilty.
* fix the SYCL fa with mkl error of a mask broadcast over heads or batches; add argument cache-ram to start-svr.sh
* add argument cache-ram to win-start-svr.bat
* fix format issue, remove --cache-ram in script: move to another PR
* Only run webgpu CI on my fork
* Add webgpu only workflow
* resolve merge conflicts
* save state
* save kernel
* add kernel back in
* fix merge issues with kernels
* updates
* update tests
* change invocation count and workgroup to be u32
* clang format
* clang format
* update docs
* format
* remove extra file
Replace the per-element modulo of i0 with fast_div_modulo in the ravel and
unravel bin-bcast kernels, drop the now-unused ne10/ne20 kernel arguments,
and reject divisors that do not fit in 32 bits when the magic values are
computed, as the CUDA implementation does.
Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
* mtmd: support YuE2-3B
* improve speed
* add CFG
* fix streaming
* add cfg control from api
* check ctx size
* fix n_predict
* add flow_steps config
* nits
* fix batch api
* nits
* adapt
* mtmd: yue2 samples from its own saved logits
The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.
* mtmd: let the yue2 snake chain fuse
The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* opencl: release the WMM flash-attention mask fallback image
The fallback image is created on every call of the bin flash-attention
kernel over a sub-buffer of the K cache and was never released, so each
call kept a K-cache sub-buffer alive. Every call with more than one query
row takes this path, including each speculative-decoding verify step, so
the objects pile up and the server slows with every request.
* opencl: keep q8_0 K/V at head size 512 off the WMM flash-attention kernel
With q8_0 K/V at DK=512 the bin flash-attention kernel gives wrong results
whenever V holds different values from K (test-backend-ops FLASH_ATTN_EXT
hsk=512 q8_0, max error 2.04 on X2-90); f16 K/V at DK=512 is correct. The
backend's own flash-attention kernels already decline q8_0 at DK=512, so the
shape is now handled the same way with or without the kernel library.
* opencl: decline WMM flash-attention prefill beyond the device's 3D image limits
The WMM path repacks Q/K/V into 3D images; V is n_kv wide, so past the device's
maximum 3D image width (about 16k tokens of context on X2-90) clCreateImage
failed and the call asserted. Such shapes now take the regular flash-attention
path.
* ui: move tools/ui to ui/apps/chat
The chat app moves to its own root folder, ui/apps/chat, as the first step of splitting the UI into one library
(ui/lib) and the apps built on it (ui/apps). The code is unchanged; every path that pointed at tools/ui now points at
ui/apps/chat: CMake adds it from tools/, and the Dockerfiles, the Nix package, the workflows, the labeler, CODEOWNERS,
.editorconfig, the git hooks, the dev and demo scripts and the docs follow it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name Llama Chat's package @ggml-org/llama-chat
It was still called llama-ui, the name of the library that now sits beside it
in ui/lib (@ggml-org/llama-ui). The package stays private; the app's display name
and the llama-ui.zip build artifact are unchanged.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: call the built-in web UI Llama Chat in docs, help text and comments
The app in tools/ui is Llama Chat; Llama UI is the component library it is
built from. Prose that said "Web UI", "web UI" or "WebUI" for the app now
names it: the READMEs, SECURITY.md, the --ui help text, the CMake option
description, one server log line, code comments and the Storybook intro.
Names that others depend on are unchanged: the deprecated --webui flag
aliases, the LLAMA_*_WEBUI CMake and environment names, the webui_settings
field, the storage migration keys and the Nix attributes.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name tools/ui as a former path of Llama Chat in the code-review skill
Changes under the old tools/ui path, as well as tools/server/webui, still
get the Llama Chat checklist.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name Llama Chat's path in the security policy's exclusions
Llama Chat no longer lives under tools/server/, so the exclusion listed
under it now names ui/apps/chat. What is covered stays the same.
Assisted-by: Claude (Opus 5.5 Medium)
The flash attention shaders indexed the mask without its head
dimension, so masks with ne[2] > 1 gave wrong results. The mask offset
and the vec block states now include the head, broadcast as
head % mask->ne[2] like the other backends. The params buffer grows to
256 bytes to fit the two new mask fields.
The shared memory estimate dropped kv_shmem when only one of K and V
is read directly, while the shaders keep it unless both are, which made
the K f16 / V q8_0 cases exceed the workgroup storage limit. The vec kv
tile now takes both flags: the memory estimate drops kv_shmem only when
both are direct, and the tile still divides the KV padding when either
is. get_alloc_size and supports_op use the same rule as the pipelines,
and supports_op accounts for the register tile fallback.
* ci : use vX.Y.Z tags as docker tags
When the Docker workflow is manually dispatched from a vX.Y.Z tag, use that tag verbatim for the image tags instead of appending the build number and commit hash.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ci : require a tag for manual docker runs
Manual dispatches of the Docker workflow must now use a tag ref, so the image tags are derived from the selected tag. The check can be skipped with the new skip_ref_check input. Scheduled nightly builds are unchanged.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* vulkan: make the radix top-k tie selection deterministic
The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.
Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.
* optimize shader for AMD, Intel and Nvidia
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.
Fixes#30576
Assisted-by: GitHub Copilot
Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.
Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.
Also trim the output_config.effort test to a single level.
* ggml-cpu: use VNNI for Q2_0 dot products
Keep the upstream AVX2 unpacking, accumulation, and fallback. Use VNNI for the integer dot product on supported x86 CPUs.
Assisted-by: Codex
* ggml-cpu: use VBMI for Q2_0 unpacking
Use multishift to unpack Q2_0 weights on CPUs with AVX-512 VBMI and VL. Retain the existing shift-and-mask unpacking on other CPUs.
Assisted-by: Codex
* Update ggml/src/ggml-cpu/arch/x86/quants.c
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* fixed support for DFlash2 for Ripple 65B
* spec: Add NoPE handling to DFlash
(cherry picked from commit 89ea8e3e77e3359d17902a2db136022d5d419d88)
* spec: Fix NoPE not being picked up by DFlash
(cherry picked from commit 43393b95fda2ce7e7d243e2650bf7b1cadf1c732)
---------
Co-authored-by: James Pickering <james_pic@hotmail.com>
* Fail tests on missing or unsupported type
* Fix output
* Remove whitespace
* Fail if no tests ran regardless of backend support
* Revert unnecessary cosmetic change
* lfm3 : put the normed tensor first in residual adds to enable norm fusion
Assisted-by: Claude Opus 5.5
* tests : update Metal fusion baseline for lfm3
Assisted-by: Claude Opus 5.5
* convert : add Inkling2 text GGUF conversion
Assisted-by: Codex
* llama : implement Inkling2 text inference
Assisted-by: Codex
* tests : cover Inkling2 text support
Assisted-by: Codex
* conversion : dequantize packed Inkling2 experts in row chunks
Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.
Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.
Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.
Assisted-by: Codex
* convert : add Inkling2 vision mmproj support
Assisted-by: Codex
* mtmd : implement Inkling2 still-image encoder
Assisted-by: Codex
* inkling2 : scope image loading
Assisted-by: Codex
* inkling2 : enable flash attention with folded bias and sinks
Assisted-by: Codex
* cuda : route per-head flash attention masks through vec
Assisted-by: Codex
* mtmd : support Inkling2 temporal frame means
Assisted-by: Codex
* style : restore shared image comment
Assisted-by: Codex
* tests : enlarge Inkling2 context fixture
Assisted-by: Codex
* inkling2 : load per-layer expert feed-forward widths
Assisted-by: Codex
* tests : add Inkling2 Metal fusion baselines
Assisted-by: Codex
* inkling2 : group and align registry entries
Assisted-by: Fable
* convert : regenerate the Inkling2 tokenizer hash with the update script
Assisted-by: Fable
* conversion : revert the print_registered_models change
Assisted-by: Fable
* conversion : sort the Inkling2 model map entries
Assisted-by: Fable
* convert : map Inkling2 vision checkpoint names in the tensor table
Assisted-by: Fable
* convert : map Inkling2 text checkpoint names in the tensor table
Assisted-by: Fable
* tests : document the Inkling2 packed expert conversion test
Assisted-by: Fable
* tests : cover Inkling2 checkpoint and transformers tensor names
Assisted-by: Fable
* llama : place Inkling2 fields and list entries by file convention
Assisted-by: Fable
* gguf-py : align Inkling2 tensor mapping comments
Assisted-by: Fable
* mtmd : place Inkling2 entries by file convention
Assisted-by: Fable
* tests : place Inkling2 cases after the existing ones
Assisted-by: Fable
* cuda : index per-head flash attention masks in the tile and mma kernels
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : project the relative bias before the gather, drop arch branches
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* convert : require Inkling2 hparams, add inkling2 pre-tokenizer
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* mtmd : cap Inkling2 image size, drop video frame averaging
Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q
* inkling2 : fix CI build, Meta device assert and Windows conversion test
Assisted-by: Claude
* tests : use the default sliding window for the Inkling2 test model
Assisted-by: Claude
* simplify mtmd part
* give _v arch for vision
* remove alpha channel handling, it should be a follow up
* explicit rule about audio + vision models
* add audio support
* rm audio
* rm gguf mtmd rope base
* address review comment
* nits
* do not normalize embd from mtmd
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* common : reject out-of-range grammar_triggers type instead of aborting (#30835)
* tests : remove the invalid grammar trigger type regression test (#30835)
* vulkan: Explicitly include SPIRV-Headers
Distros that don't put the headers in a system include path need
to specify it to the compiler. So we use the appropriate CMake
target to include it.
Assisted-by: Codex
* vulkan: Add explicit headers location to workaround OSX CI runner conf
* vulkan: optimize Flash Attention cm1 for RDNA3 and RDNA4
* reduce shmem use through buffer reuse
* clean up and fixes
* limit q_in_regs to prefill
* vulkan: add flipped FA path, each subgroup owns 16 Q rows
* cleanup
* move flip shader into own file
* replace kvsep with direct load mode
* fix nvidia hang, tune AMD
* only store pv in sfsh if necessary
* fix skip condition
* nvidia tuning
* reduce occupancy on RDNA4 for large attention head sizes
* fix sparse FA fp16 underflow bug
Template soft_max_f32_parallelize_cols_single_row to keep column
elements in registers when they fit, reducing memory traffic from
5 global memory passes to 2 passes.
Assisted-by: Gemini 3.8 Flash
* vulkan: optimize matrix multiplication for narrow N
Add new "narrow" tile configs for coopmat1/coopmat2 that are tuned for N<=32.
* Attempt to handle wave64
* add n=16 tile for cm1
* apply RDNA split_k heuristic to narrow_n generally
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* chat: allow tool calls inside reasoning blocks for Ministral-3 and Qwen3
Fixes#29248Fixes#20837
* chat: address review feedback for Ministral-3 reasoning end
- Remove <function= from Qwen3 (intentionally not used outside outdated coder models)
- Ministral-3: use choice between [/THINK] and peek([TOOL_CALLS]) instead of optional
* chat: add [TOOL_CALLS] to Ministral-3 thinking_end_tags
* chat: simplify Ministral-3 reasoning end choice
* chat: restore peek lookahead for Ministral-3 tool call start
* chat: keep [TOOL_CALLS] only in parser, not in thinking_end_tags
* cont : revert tool call tag removal and update tests
---------
Co-authored-by: Alen Požun <kukirin-ui@users.noreply.github.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
The thought channel is normally closed with <channel|>, but the model
may open it and go straight to a tool call without closing it. Stop the
reasoning at <channel|> OR <|tool_call>, so an unclosed thought does
not swallow the following tool call.
Fixes: #32231
Assisted-by: Qwen 3.8 27b
The qwen2vl graph passed NULL for the ffn_up/gate/down scales, so the
per-tensor NVFP4 scales loaded by the generic pass were never applied
and the model produced garbage (PPL ~1e6 on CUDA, NaN on CPU).
Pass the scales to build_ffn like qwen2 does.
Fixes#32101
* mtmd : allow custom video decoder
* also adapt to wllama requirements
* clean up
* add test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* dflash : move the DSpark markov head and backbone graphs to dspark.cpp
Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and
graph_mistral_large4 move verbatim. build_inp_target_features and
build_dspark_markov_head become static members of llama_model_dflash so
both files can use them.
* dflash : detect the head and the backbone once, split the loaders per file
load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2,
DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4).
The head, backbone and base layer loaders become member helpers in
dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph
dispatches on the backbone.
The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through
a helper in dflash2.cpp, right after their head tensors instead of in
the base layer loop; the other tensors keep their creation order. The
DFlash2 loader drops its copy of the conv group size check, which master
never reached.
* dflash : share the injection input, token embeddings and lm head lookups
build_inject_inp (target features + fc + encoder norm), get_tok_embd and
get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral
Large 4 graphs. build_inp_target_features is file-static again.
* dflash : split the DFlash decoder into a graph per head
graph_base creates the shared inputs and provides build_inject,
build_inp_tokens, build_layers and build_logits. graph_dflash
(dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and
graph_lilicorr (lilicorr.cpp) assemble them and add their head; the
selector, scorer and markov head builders move to the file of their
graph, and the DFlash2 local conv run by build_layers to dflash2.cpp.
build_arch_graph dispatches on the backbone, then on the head.
The op order and the tensor names are unchanged.
* speculative : split the DFlash driver into a class per head
common_speculative_impl_draft_dflash stays the DFlash implementation and
becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its
protected constructor only reads the draft metadata; init() then sets up
the contexts with the options of the draft (draft length, backend
sampling, device features, masked h_nextn). Each derived class overrides
draft() with the path of its head from the former draft(). DFlash2 also
overrides begin() to reseed its generator per request, and LiliCorr
overrides process() to store the context features of each injected
chunk, which its noise block carries as its state.
The factory keeps the current routing: draft-dspark picks DSpark,
draft-dflash picks DFlash2 or LiliCorr from the draft metadata and
DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs
the DSpark reader instead of a mix of both. sample_from_anchor is logged
by DSpark only.
* speculative : add the draft-dflash2 and draft-lilicorr types
draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts,
which draft-dflash keeps running too: it picks them from the draft
metadata. draft-dflash2 and draft-lilicorr must match the draft model,
draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a
DFlash draft is rejected, and a run uses at most one DFlash family type;
these errors leave speculative decoding uninitialized.
The type detected from the draft GGUF is now the concrete one, and
draft-dflash2 and draft-lilicorr download the dflash sidecar.
* docs : describe the DFlash2 and LiliCorr drafts
Add the DFlash2 and LiliCorr sections and the new types, and explain
that draft-dflash also runs DFlash2 and LiliCorr drafts.
* speculative : drop the unused DFlash is_mrope
Nothing reads it since the migration to common_batch (#29385), which
picks the number of M-RoPE position rows from the rope type of the draft.
* doc: update spec dec
* address review comments
* requirements : support Python 3.15
torch 2.11.0, numpy 2.4, matplotlib 3.10, pandas 2.3, pillow 11.3 and
cffi 1.16 have no Python 3.15 wheels, so the CI runner on 3.15 cannot
install the requirements.
Python 3.15 gets the first releases that ship 3.15 wheels through
version markers, and every version below 3.15 stays as is.
pyproject.toml now allows 3.15 and picks torch 2.15.0 there.
* requirements : apply review suggestions
numpy 2.5.4 from Python 3.14, matplotlib 3.11.2 and pandas 3.0.6 from
3.11, cffi 2.1.1 and pillow 12.3.0 for every version. cffi 1.16.0 has
no 3.14 wheel and fails to build there, so this also fixes the
tokenizer test requirements on 3.14.
* hex-bufs: combine queue and fence buffers
* hex-bufs: use n_threads for queue sizing
* hex-mmap: optimize buffer stacking at the host to avoid iova fragmentation
* hex-mmap: add simply retry for spurios mmap failures
* hex-mmap: use eager mode for extended mmap retries
* vulkan: add TOP_K_MASK
Assisted-by: Claude Opus 5.5
* vulkan: fix the TOP_K_MASK barrier and use the common offset helper
Assisted-by: Claude Opus 5.5
* vulkan: declare ggml_vk_top_k_mask in the common header
Assisted-by: Claude Opus 5.5
* spec : skip p_min for sampled MTP and DFlash2 drafts
Under rejection sampling the acceptance at a position is 1 - TV(p, q),
which the draft's top probability does not tell, so the p_min cutoff
dropped draft tokens the target would have accepted. Skip it when
MTP or DFlash2 samples the draft. draft-simple keeps it: each extra
draft token costs a full draft model pass.
* spec : replace the sampled-draft p_min skip with a warning
p_min behaves as on master again for every drafter and sampling mode.
Instead, log a warning when --spec-draft-p-min > 0 is used with
--spec-draft-sampling probabilistic, suggesting p_min 0: under
rejection sampling the draft's top probability does not tell how much
of the draft the target accepts. The cut is not skipped because with
MTP at long drafts it can still pay off.
* spec : warn about p_min only for drafters that use it
The n-gram drafters ignore p_min, and DSpark uses it as the cutoff of its confidence head, so the warning is skipped when only these are configured.
---------
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* grammar : limit the number of alternatives
* grammar : introduce an AST for grammar parsing
* grammar : revert AST refactor and alternatives limit
This reverts commits:
- 84f3b57c4d6c482e7187a30735d16b02de3784fa
- 393a707ebb7e6f798abdb8af34cb1bc102f4aaa6
* grammar: optimize big grammar matching by using left-factorization
* grammar : preserve character class boundaries during left factoring
* grammar : prevent infinite expansion during left factoring
* grammar : use ASCII characters in comments
* grammar : test empty suffixes after left factoring
* grammar : deduplicate left factoring logic
* grammar : add comments
* grammar : use number parameters and clarify RSS output
* grammar : clarify character class test names
* grammar : remove performance benchmark
* grammar : share immutable rules
---------
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
The kernel launches one block per row and 256-wide tile, so the
{2, 16777217} test case asks for 16777217 blocks of 256 threads. HIP
refuses a grid dimension above 2^32 threads and the ROCm jobs fail
with "invalid configuration argument".
The host now launches the grid in chunks of at most UINT32_MAX / 256
blocks and passes the chunk offset to the kernel. Real sizes fit in a
single chunk and keep their timings.
This commit removes the llama version test-cmake and instead uses the
version that is supplied by CMAKE_PREFIX_PATH.
The motivation for this is to avoid issues when bumping major versions.
The D=256 vec flash attention kernel with F8_E4M3 K and V uses 257
VGPRs (255 + 2 spilled) on gfx908, one over the limit of the check.
The spill is a perf issue only, like the Q4_0 D=256 vec kernel
already in the ignore list.
Both variants of the kernel join the ignore list.
[no ci]
* server: read a small temp file in the read_file tool test
test_tools_builtin_read_file read its own file, which grows with every
added test. On Windows the CRLF checkout puts it at 16664 bytes, over
the 16384 byte read limit, and the tool refuses it.
The test now writes a small file of its own and checks the exact
content read back.
* server: use a temp cwd in the x-tool-cwd tool test
test_tools_builtin_cwd_header read test_tools_builtin.py through the
x-tool-cwd header, which hits the same 16384 byte read limit on the
Windows CRLF checkout, and wrote its shell marker into the source tree.
The test now uses a temp directory as cwd, reads a small file from it
and checks the shell marker lands there.
The vision tests download their images from huggingface.co, which
resolves to a private address on the runners inside the HF network.
The media URL check refuses it with "host is not a public address"
and the server-cuda job fails.
The vision fixture enables media_allow_private_urls. The private
address check keeps its own coverage in
test_vision_image_url_private_address.
The same-origin tests over the LAN address block forever on the Metal
runners: the request to the host's own LAN address never returns and
has no timeout, so the job runs until the 6h limit.
get_lan_ip() now probes /health on that address with a short timeout
and returns None when the host cannot reach it, which skips the test.
The test requests themselves use the default request timeout so a
hang fails the test instead of stalling the job.
* fix(hexagon): make host build work with clang-cl on Windows ARM64
Two CMake fixes for building ggml-hexagon natively on Windows-on-ARM64
with x64-hosted clang-cl (validated on Snapdragon X Elite, discussion in
qualcomm/GenieX#1390):
- Some clang drivers (e.g. x64-hosted toolchains used on ARM64) don't
predefine the arch macro, which makes winnt.h fail with "No Target
Architecture". Define _M_ARM64/_ARM64_ explicitly for the host TUs.
- The Hexagon SDK's incs/stddef include (AEEStdDef.h and friends) poisons
the MSVC header chain when it leaks onto dependent TUs' include paths
via htp_iface's PUBLIC includes. Keep it PRIVATE to the interface stub
and system-level for the backend's own TUs (which include <AEEStdErr.h>
directly).
Originally validated 2026-08-26 as reported in the issue; applies
cleanly to 15759e0.
* hex-cmake: trim down excessive comments
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* llama : remove deprecated `llama_set_warmup`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : remove remaining deprecated API
Drop the APIs marked DEPRECATED in llama.h, the deprecated
llama_context_params::defrag_thold field and the --defrag-thold option.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* fit-params : remove stale `llama-ext.h` include
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : document `llama_model_dflash_selector_top_k`
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : promote the NextN / speculative API from `llama-ext.h`
Move the NextN, layer input, embd row and state row APIs to `llama.h`
and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with
the internal `llama_context::set_state_src` following the same name.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : polish the NextN API promotion
Move the NextN draft model queries next to the Adapters section, rename
the `src` parameters of `llama_set_ctx_state_src` and
`llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant
`llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : prune deprecated API
* cont : bump RPC version
* cont : minor
* ggml : remove outdated documentation from ggml.h
The 172-line comment block at the top of the header has not been updated in a
substantive way since the backend v2 split (Nov 2023) and now documents APIs
that no longer exist:
- `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed
- `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist
- `ggml_set_param()` no longer takes a context argument
- `ggml_graph_compute_with_ctx()` moved to the CPU backend header
- the optimization API now lives in `ggml-opt.h`
The block also ended with 5 empty `TODO` sections.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (#31946)
* rpc : bump RPC protocol major version
The tensor op enum is sent over the wire, so removing GGML_OP_ADD1,
GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every
op that follows them and peers built before and after this change would
misinterpret each other's ops.
The handshake compares the major version for exact equality and ignores the
patch version, so bumping the major version is what rejects incompatible
peers instead of silently misbehaving.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: losewayy <1134285612@qq.com>
* CUDA: few-row MMQ for 2-24 src1 columns
New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.
* nit fixes
* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat
* few-row quantize: DS4 sum from the quantized values, as in #31247
* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build
* few-row MMQ: dispatch order MMVQ, few-row, MMQ
With --sleep-idle-seconds, a handler passes wait_until_no_sleep()
while the server is awake, then keeps using the server state, for
example to parse the body and tokenize. If the idle timer expires in
that window, the main loop enters sleep and the callbacks destroy the
model: the handler reads freed vocab memory, and a task posted after
that stays in the queue with nobody awake to process it.
Count the requests that passed wait_until_no_sleep() and are not done
yet, and do not enter sleep while the count is positive. The response
generator releases its count in its destructor, so every return path
of a handler gives it back.
Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
HunyuanVL resamples its position embedding on the host into a float
buffer sized for F32, then reads ggml_nbytes() of the tensor into it.
The graph never uses the tensor, so any type loads, and an F64 tensor
overflows the buffer at the first image request.
Require the position embedding to be present and F32 when loading the
HunyuanVL tensors.
The MiniCPM-V resampler ends with build_mm(mm_model_proj), whose output
width is ne[1], while n_mmproj_embd() returns ne[0]. Both are equal in
real models, the projection is square. A non square projection passes
the text model n_embd check and only aborts at the encode, on the
output buffer size check.
Return ne[1], so the width matches the graph output.
* mtmd: require the MLP projector and patch embedding tensors
The loader marks the patch embedding and the LLaVA projector tensors
as optional, while the graphs and n_mmproj_embd() dereference them
unconditionally. A GGUF without them crashes on a null tensor during
the warmup.
Load the patch embedding as required for the vision encoders that
start with it, which is all of them except GEMMA3NV, DEEPSEEKOCR and
DEEPSEEKOCR2. Load mm.0 and mm.2 as required for MLP, and mm.0, mm.1,
mm.3 and mm.4 for the Yi-type MLP_NORM variant, selected by mm.3.
The graph no longer tests mm.2 and the yasa2 assert on the patch
embedding is covered by the loader.
* mtmd: keep the Yi-type llava TODO
The Yi-type variant is still detected by the presence of mm.3, the
TODO on that detection stays.
The Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Ling3-VL graphs build their
input with a temporal merge that assumes a 2x2 patch grid, and the
Qwen2-VL merger reshapes by 4. Any other spatial_merge_size from the
GGUF reaches a GGML_ASSERT at the first image request and aborts the
server.
Reject a spatial_merge_size other than 2 when loading the hparams.
The SpacemiT compute case was commented out, so the op stayed on the scalar CPU path. Type dispatch stays with the other ops, and use_ref keeps the reference path on the generic CPU kernel.
Assisted-by: Codex
Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
* vulkan : Intel customized FA shader for prefill usage
* vulkan : adding check of max_bias/logit_softcap for Intel FA kernel path
* vulkan : add m16n16k16/SIMD32 support for Xe FA prefill shader
* vulkan : Intel FA shader and host refinement, with accuracy issues fixes
* vulkan : Limit the SLM usage to 48 KiB for Intel prefill FA shader
* vulkan : Update Intel FA shaders to handle cases with large head dimension K, unaligned Q head stride and multi head mask cases
* vulkan : Intel FA shader and host refinement to align with existing design, extreme cases handling, shape check relaxing.
* vulkan : Intel FA shader and host update to support q as tensor view, unaligned batch stride q. Add env variable for enforcing XE FA path for other vendor GPUs with 16x16x16 CM1 and SIMD32 support
* vulkan : Inte FA shader and host update to fix the potential SLM over 48 KiB issue, enhance the buffer access robust and add asymmetric qk/pv ration support.
* vulkan : Inte FA shader and host update to add SLM 48 KiB size check, remove unnecessary extensions and finetune the unroll/dont_unroll loops
* vulkan : Intel FA shader and host update to add SLM size check function, use real device max SLM size in shader specialization constants.
* vulkan : Intel FA shader and host update to consolidate host helper function, remove redundant buffer sync, bypass unused pipeline and buffer, fix subgroup shmem conflict and add negative case handling.
* vulkan : Intel FA host update to correct get_xe_fa_tuning_params() function log message, and add workgroupMemoryExplicitLayout/workgroupMemoryExplicitLayout16BitAccess prerequisite feature check.
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back
* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ
* ggml-cuda: Tests for the fused scale
* ggml-cuda: reuse existing graph to test fusion in mmq epiloque
* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency
* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions
* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
* ggml-backend : prefetch only inputs computed by earlier splits
The copy callback of a split runs after the prefetch and can write tensors that a later split reads, e.g. the MoE cache uploads missed experts into its bank. A prefetch of such a tensor reads it before the write.
* llama : skip the MoE cache when its device cannot run the cached mul_mat_id
Otherwise the scheduler runs the matmul on another backend and copies the whole bank every ubatch.
The 32x256 pack already keeps a separate uint8 zero point. Complete the M1 path and use the same HP kernel as Q4_0.
Assisted-by: Codex
Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
* ui: add search_chats tool
* add from_timestamp / to_timestamp
* ui: search_chats review fixes
- keep search_chats disabled on import, copy tool policy on fork
- stream messages without attachment payloads, one transaction for time ranges
- keep the output within budget, cut whole messages from the older end
- reject impossible date-times, quote titles and attachment names
- typed readOnly tables, move localIsoString to formatters as formatLocalIso
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
npm 11.20 refuses git dependencies by default (EALLOWGIT), so npm ci
failed on fresh installs. Pin llama-brand to the same commit through
an HTTPS tarball, which also adds an integrity hash to the lockfile.
Assisted-by: Claude (Opus 5.5 Medium)
Each graph result reserves metadata for max_nodes, but many graphs use
only a fraction of that capacity. The vector resize initializes the
whole reserve and makes otherwise unused pages resident. Allocate the
reserve with new uint8_t[] instead.
* fix aot for >4g garbage output issue; add --aot in build script
* correct the aot flag
* support no-xmx build, update the code for no-xmx building, rename nossl to no-ssl
ggml_sycl_set_main_device() runs at the start of every graph compute. Its
early-return check called dpct::get_current_device_id(), which locks the
device manager mutex and looks up the thread-to-device map. Use the
thread-local cached id that the rest of the backend already uses.
Assisted-by: Claude Code
Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
* CUDA: skip empty MoE tiles in MMQ stream-k fixup
mul_mat_q_stream_k_fixup loads ids_dst[col_low + j] for all j < J before
checking j_sup. mul_mat_q returns early on col_low == col_high, the fixup
kernel did not. mm_ids_helper emits one extra tile per expert that may be
empty; for trailing experts col_low == ncols_dst, so the load reads one
element past the J_best-1 padding added in #29953. The kernel returns at
j >= j_sup before storing anything, so results are unaffected.
Return early when j_sup <= 0, like the main kernel. The return is
block-uniform (tsj is loaded once per block) and precedes __syncthreads().
Add test_mul_mat_id_unused_tail to test-backend-ops: only the first
n_mats/2 experts are selected, so the trailing experts have zero rows and
their tiles are empty; m=33, k=512 keeps tile efficiency low enough that
stream-k splits along K and launches the fixup kernel.
* CUDA: drop the extra MUL_MAT_ID test case and the explanatory comment (review)
test_router_restricted_cors_same_origin_reaches_child still named
ggml-org/tinygemma3-GGUF, which the router no longer finds since the test
models moved to ggml-org-dev, so the Server test job fails on master.
Assisted-by: Claude (Opus 5.5 Medium)
* opencl: add decomposed FA
Assisted-by: Claude Opus 5.5
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: build FA MQ kernel on first FA call
* so allows warmup to early build neccessary kernels
Assisted-by: Claude Opus 5.5
* opencl: handle inf for fully masked row
* opencl: refine condition for using fa decomp
* opencl: support dp4a for PV mm
Assisted-by: Codex
* opencl: add direct PV for FA decomp to avoid V transpose for small batch
Assisted-by: Codex
* opencl: enable direct pv by default
* opencl: expand X2E decomposed FA to smaller heads and batches
- Lower the default head dimension threshold from 256 to 128 on X2E.
- Allow DK64 with P8, no attention sinks, GQA >= 2 and n_kv >= max(1024, 2*n_q).
- Allow P8 with V transpose for 2-31 queries when direct PV is unavailable, DK >= 128 and n_kv >= 1024.
Assisted-by: Codex
* opencl: adjust format
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* ui : keep downloads in Discover sidebar during search, sort base-org repos first
- Show the sticky Downloading section regardless of an active search query
- Order search results so repos published by their base model's org lead
Assisted-by: Claude (Opus 5.5 Medium)
* server : throttle download progress per file
Files of a download run in parallel threads, but progress updates were
throttled with one shared timer. A small draft sidecar reports far more
often than the main weights, so it took nearly every slot and the main
file's updates, including the first one carrying its total size, were
dropped until the draft finished. The UI then showed the draft's progress
first and restarted from 0% for the model.
Throttle each file on its own and always pass a file's first update.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list a downloading repo once and match base orgs case-insensitively
A search hit that is downloading already shows under Downloading, so the
results leave it out. Base model orgs compare without case, as the hub
treats them.
Assisted-by: Claude (Opus 5.5 Medium)
* server : tell clients a download stopped on request
When a client paused or cancelled a download, every other client following
/models/sse got a bare download_failed and could not tell it from an error,
so the web UI showed "Download failed" for a pause made in the Llama app.
A stop asked for through unload or delete now fails with {"stopped": true}.
Deleting a paused download, which has no instance left, now sends
model_remove as deleting a running one does, so other clients drop its row.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : follow downloads other clients start, pause or cancel
A download another client started (e.g. the Llama app) already showed up
from the feed; it now also picks up the sidecars the list reports for it.
A stop another client asked for reads as a pause instead of a failure, and
a removal clears the download row even when the list does not carry it yet.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : take the logo and favicons from llama-brand
The web UI's logo mark and favicons now come from llama-brand, added as a
dev dependency from GitHub and pinned to a commit, so the UI draws the
same icon as the README cover instead of its own copy of an older one.
Logo.svelte inlines the brand's icon and fills it with currentColor, and
the PWA asset generator starts from the brand's web favicons, which carry
their own margin and optical centring, so the UI's colouring and padding
script, its test and its logo.svg go.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : follow llama-brand's master instead of a pinned commit
Assisted-by: Claude (Opus 5.5 Medium)
* server: fix --cors-origins localhost handling
* server: validate Host on loopback connections against DNS rebinding
A page whose own name resolves to 127.0.0.1 reaches the server with
that name in Host and a matching Origin, so the same-origin shortcut
let it through. On a loopback connection, Host must now name loopback
or the host of an allowed --cors-origins entry. The LAN tests reach
the server through a real interface instead of a forged Host.
* fix python type check
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Allocate gf_res_reserve and both gf_res_prev entries once in the llama_context constructor. Add llm_graph_result::set_max_nodes() and reset the existing instances in sched_reserve() instead of recreating them. get_gf_res_prev() now only selects the arena for the current batch.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
- remove build_delta_net_chunking and build_delta_net_autoregressive
- always use the fused gated delta net
- make graph_max_nodes independent of the ubatch size
- drop the fused GDN probes and fused node tags
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
ggml-sycl.dll built with oneAPI 2026.2 imports mkl_intel_thread.3.dll and libimalloc.dll, which are not part of the release package. LoadLibrary fails and the SYCL backend is silently skipped, llama.cpp falls back to CPU.
Adding both DLLs fixes the load: measured on Arc B570 with LFM2.5-8B-A1B Q4_K_M, pp4096 goes from 404 t/s (CPU fallback) to 2232 t/s (SYCL) with the extracted b12244 package.
Assisted-by: GLM (z-ai/glm-5.3-flash)
Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com>
* CUDA: re-instantiate graphs when the cluster dims of a kernel change
* CUDA: keep the cluster dims with the graph instance, check them only on Hopper+
Assisted-by: Claude Opus 5.5
* CUDA: use CUDA_CHECK and a helper struct for the graph cluster dims
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CUDA: add a TODO for the graph cluster dims test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call
Assisted-by: Claude Opus 5
* ROCm/HIP: corrected by only setting the workspace on ROCm 7.0 or newer
* cpu: accumulate flash attention V in F32 in the one-chunk kernel
The one-chunk kernel accumulated an F16 V cache in an F16 running sum
that is only normalized at the end, which overflows past 65504 on long
contexts: inf, then NaN on every following token.
Every V type now accumulates in F32, like the tiled kernel. F16 rows go
through ggml_vec_mad_f16_f32, which converts and multiplies in the same
register (F16C/AVX512, NEON, RVV) with no round trip through a temporary
F32 row, so decode keeps master speed on x86 and loses about 4% at 4k
context on ARM, where master used native F16 FMA.
* cpu: widen the F16 V rows like ggml_vec_dot_f16_f32
ggml_vec_mad_f16_f32 required F16C and FMA, so AVX builds without FMA
fell back to a scalar loop and ran 3x slower than master. It now
follows the guards of ggml_vec_dot_f16_f32: AVX512, then AVX with F16C
through GGML_F32x8_FMA, then NEON and RVV, then the scalar loop.
* additional FA test cases
Assisted-by: Codex
* vulkan: allow dequantized KV optimization with CM2
Enable the KV dequantization optimization with coopmat2, guarded by a heuristic
based on query tokens*GQA ratio.
While doing this we saw very large speedups for certain head sizes, and it was
because those head sizes were incorrectly treated as unaligned. So also fix
those to be treated as aligned, which should help for when the dequant
optimization doesn't apply.
* gqa uses nbq2 for the stride, so we need to check that value for alignment
* tests : use VARS_TO_STR18 for flash attention
Assisted-by: Codex
CodeMirror takes text input through EditContext on Chrome for Android
only, and there the chat editor never brings up the virtual keyboard.
The editor now always takes its input through contenteditable, as on
desktop and iOS.
A press in the input box outside the text also focuses the editor, in
the collapsed form too, so the whole box is a tap target again as the
textarea was.
MMF tiles 32 rows per block, so a BF16 matrix with few rows (e.g. the
qwen4exp hyper-connection projections, 320 rows with K = 10240) runs only
a handful of blocks and leaves most SMs idle at batch sizes 2..8. MMVF
runs a block per row and wins there when K is long enough.
MMVF's cost grows with rows x batch while MMF's stays roughly flat in that
range, so use MMVF only while rows * ne11 * 256 <= nsm * min(K, max_k) and
K >= 1024. The budget scales with the SM count, which keeps GPUs with few
SMs (e.g. the A16) on MMF. Blackwell allows up to 8 columns and
max_k = 6144; older GPUs stop at 5 columns and max_k = 4096.
Add eval cases at 128x2560, 512x2560 and 320x10240 for n = 2, 3, 4, 5 and 8.
* CUDA : add sm70 FlashAttention config case for the 256x256 head
* CUDA : tune sm70 FlashAttention for 256x256 ncols=32
Assisted-by: Codex
---------
Co-authored-by: Jirka Svítil <mistrjirka@users.noreply.github.com>
* cuda : add MMA FlashAttention support for head sizes 40 and 72
Head sizes 40 and 72 (used e.g. in vision towers of Gemma 3/4 and Qwen3-VL) were excluded from every tensor-core dispatch branch and fell back to the tile kernel.
The fp16 MMA fragment covers 16 halfs in the head direction, so the in-kernel geometry is padded to the next multiple of 16 (40 -> 48, 72 -> 80). Register arrays, shared memory tile strides and MMA loop bounds use the padded size, while global memory loads and stores the real one.
Enabled on NVIDIA Turing and newer while Volta and AMD keep the tile kernel. The RDNA/CDNA config rows and the mma_tile_sizes specialisations are needed because the explicit instantiations are compiled for every target of a translation unit.
* tests : add the head size = 40 encoder perf shape
* cuda : address review comments on the head size 40 and 72 kernels
* Initial CUDA WIP version
* Use fp8 intrinsics
* Do W8A16 for FP8 in GEMV case
(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB
| model | size | params | backend | ngl | n_ubatch | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 | 2781.53 ± 8.48 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 | 37.56 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 @ d32000 | 1616.31 ± 2.61 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 @ d32000 | 34.11 ± 0.03 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 | 2542.00 ± 0.51 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 | 37.36 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 @ d32000 | 1762.86 ± 1.33 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 @ d32000 | 34.09 ± 0.02 |
* Optimize GEMV kernel for FP8
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2
Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099
* Remove trailing whitespace
* Add supported for strided copies to/from FP8 in CUDA
* Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh
* Fix odd-rows for convert/getrows, possible with FP8
* Fix FP8 fallback in cuBlas
* Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8
* Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte
* Add todo about accumulator/precision honoring
This is beyond the scope of the stacked PR
* Avoid races in fp8_fallback
* Allow Hopper and other BW SKUs for FP8
* Use intrinsic over explicit manual conversions
* Resolve TODO
As per spec, we don't have to preserve signs
* Add commend on cublaslt workspaces
* Use convert_unary_cont_cuda over block dequant in convert.cu
* Use get_rows_cuda_float for fp8 kv cache
* Make FP8 copies work on all IHV's
* More cleanup
* Simplify by falling back to shared cuBLAS helper
* Fuse weight_scale multiplication for FP8 also
* More code-cleanup
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4
Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix
* Use F32 activations in fallback mmvf path
* Explicitly link against cublaslt
* Move FP8 conversion intrinsics into ggml_cuda_cast
Cleaner this way
* WIP src1/prec for fp8 in MMVF
* Remove comment
* Keep only f8 x f8 KV cache for now
* Just do FP8 x FP32 x FP32 in MMVF as its fast enough
* Update ggml/src/ggml-cuda/ggml-cuda.cu
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
* Add batched FP8 GEMMS in cublas as suggested by code review
https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644
* Extract fp8x2 -> float2 load into helper
* Searapte copy checks to/from fp8
* Compiler massaging to avoid repeated address offsets
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Modify `ggml_sycl_op_clamp` to use the unary helper function
`ggml_sycl_detail::ggml_sycl_op_unary` to handle non-contigous or
strided views. The fix uses a pattern that matches other stride-safe
unary ops, and is a simpler execution path.
Deleted dead template `unary_op_clamp_kernel`, the flat `clamp` kernel
launcher from element_wise.cpp, and deleted (now) unused
`SYCL_CLAMP_BLOCK_SIZE` from presets.hpp.
The hidden desktop strip relied on inherited `visibility: hidden` to keep
its content from catching clicks. Since the sidebar's sections fold
(#31936), an open section's CollapsibleRegion sets `visibility: visible`
again, and its rows' context-menu triggers carry an inline
`pointer-events: auto`, so they caught clicks meant for the page.
Make the hidden strip's content inert, which no descendant can undo, and
add a client test that hit-tests the hidden sidebar's area.
Fixes#31971
Assisted-by: Claude (Opus 5.5 Medium)
* media : remove the legacy llama0 and llama1 brand assets
The brand assets live in ggml-org/llama.brand; none of these is referenced
in the repository.
Assisted by: Claude (Opus 5.5 Medium)
* readme : use the new llama.cpp cover from llama.brand
Assisted by: Claude (Opus 5.5 Medium)
* ui: show audio in Discover for models whose mmproj has an audio encoder
Discover took audio only from the pipeline tag, so omni models tagged
any-to-any, such as Gemma 4 and LFM 3 Omni, showed vision but never audio,
though their mmproj carries an audio encoder.
Read the encoders from the mmproj's GGUF metadata instead: a ranged read of
its first 64 KB, once per repo, for clip.has_vision_encoder and
clip.has_audio_encoder. The Discover list and details now show audio when
either the pipeline tag or the mmproj says so.
Assisted by: Claude (Opus 5.5 Medium)
* ui: take Discover modalities from the mmproj's encoders in one place
Review fixes for the mmproj header read:
- drop vision for audio-only mmprojs: the converter writes
clip.has_vision_encoder only when true, so a header read in full without
it has no vision encoder; a header cut short leaves vision as it was
- accept only a 206 to the ranged read, so a server that ignores Range
never streams the whole mmproj
- encode the mmproj path in the resolve URL
- derive the list and details modalities with one HuggingFaceService
helper, so both views agree
- point the key comment at tools/mtmd/mtmd-impl.h, where the keys live now
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a gated repo's mmproj from being read again on every mount
A gated, private or missing mmproj answers 401, 403 or 404 to every read,
so caching nothing made each mount of its Discover row request it again and
log an error. Cache it as unknown instead: no audio, and vision from the
mmproj being there, as before the header was read.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move Models, MCP and Settings to a sidebar footer
The sidebar's actions split in two: New chat, Search and Projects stay at
its top, and Models, MCP and Settings move to a footer at its bottom, under
a thin rule, in the expanded sidebar and its collapsed strip alike. On a
phone the footer keeps clear of the home indicator, and Models stays
desktop-only as before. The search dialog still lists every action.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : group the sidebar footer in an inset surface
The design system gains inset-group-surface, for rows grouped inside a glass
panel: white on the light glass and a faint tint of the text colour on the
dark one, with the sidebar's own ambient edge, no inner highlight, a faint
wide shade under it and the card radius. It is matte-glass pointed at its
own component tokens, like the sidebar and chat form surfaces, and reads as
part of the panel: no tight contact shadow or solid border like a button,
no inset shade like a field.
The sidebar footer uses it in place of the rule above Models, MCP and
Settings. DESIGN.md and the Surfaces page document it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : give the sidebar's logo button the add button's key look
Whenever the sidebar shows, open or peeking, its logo button takes the same
key as the chat form's add button (matte-key, white in light mode, a light
tint in dark), in place of its own muted fill. While the sidebar is hidden
the logo has no fill and no shadow, and hover and focus never change the
fill. It keeps its round shape and its own size.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : put search in a round key at the top of the phone sidebar
On a phone the sidebar's Search row moves to a round key at the right end
of its top row, across from the logo, with the same key look as the logo
and the chat form's add button. It opens search as the row did. Desktop
keeps the Search row.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : size the phone sidebar for a thumb and list its footer with the actions
On a phone the sidebar's actions and rows are 48px tall at 17px, flush one under
the other with 16px corners, its section headings 15px and grey, and its groups
28px apart. MCP and Settings join the top actions there: the footer is a desktop
pattern.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : mark the selected sidebar row with a tinted inset surface
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the phone sidebar's round keys white on the grey page
Assisted-by: Claude (Opus 5.5 Medium)
* ui : open Settings and MCP as pages on a phone, dialogs elsewhere
On a phone the sidebar opens them at #/settings and #/mcp-servers, with the menu key and a
title like any phone page. Pinned under the status bar, Settings' section tabs fill the
screen above them in the page's colour.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : open a phone sidebar's pages behind it, settled before they slide back
A tap in the phone sidebar now opens its page while the current one is still aside and slides
back once it has rendered, instead of swapping pages mid-slide. The slid page's fixed bars are
pinned again as the page under them changes, the lock ends with the slide itself, and a
phone's new chat shows its form and greeting at once, without the move and entry they have
on a desktop.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style the sidebar footer group and the selected row with master's classes until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add GlassKey with master's styles for the phone sidebar's round keys
Assisted-by: Claude (Opus 5.5 Medium)
* ui: center the top edge of a new chat's form on the screen
The form rested 42px below the middle up to 2xl and 6px above it past
that; one drop, from the resting card's height, now puts its top edge on
the middle at every width from md up.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: round the sidebar's action rows concentric with the footer group
The sidebar stacks four surfaces: the page, the sidebar, the footer group
inset in it and a row in the group. app.css now names the group's radius
and padding, and derives the row radius from them (16px less 4px), so the
footer's rows run parallel to its corners; the actions at the top of the
sidebar share that radius. The footer moves to its own component, the
README documents the pattern, and a client test holds the rule on the
rendered footer.
A selected action row was bg-accent, the same grey as a hovered row in
light and darker than the footer's fill in dark, so it read as hovered or
not at all; it is now raised and edged, while a hover only tints.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: import demo chats through the Settings page on a phone
A phone opens Settings as a page now, not a dialog, so the demo harness
waited for a dialog that never came and every phone demo with traces
failed; it now imports through whichever of the two opens.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: count the sidebar's rows from their items in the surface test
The test expected three footer rows and two top actions, which the
stacked PRs change as they add Projects, Library and Servers; it now
expects as many rows as the sidebar has items.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a line on Meta+Enter off macOS too
The chat form takes Meta+Enter as Ctrl+Enter and leaves the newline to
the editor, but the editor bound only Mod-Enter, which is Ctrl off
macOS, so on Linux Meta+Enter neither sent nor added a line. This failed
the client test on CI's Linux runners, on master too.
Assisted-by: Claude (Opus 5.5 Medium)
This commit removes the unused ids and type_src0 parameters from
quantize_row_q8_1_cuda.
The motivation for this is that these parameters were required in the
past because this function was used in combination with the function
pointer quantize_cuda_t, which has been now been removed, and this
function no longer needs to adher to the function pointer signature.
Refs: https://github.com/ggml-org/llama.cpp/pull/31884.
* Add FP8 KV-cache support
* Add tests for scaled, FP8 KV-cache to test-backend-ops
* Fix get_*_scale mapping and dequnatize FP8 also before rope shift
* Normalizes cache-scale names independently of projection layout.
While fused KQV is currently impractival for non-shared scales, this is
an improvement for future use regardless
* Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs
* Update comment wording
* Move E2E FP8 KV cache tests to test-llama-archs
They belong there as they are not just a test for ggml's backend ops
* Increase support surface, remove seed
* make scaled kv generic
* Fix CI failings for test-kv-cache
https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392
* CI: execute kv-cache-scale tests
* exclude fp8-kv tests from OpenVINO
* Fix linting errors of test_kv_cache_scales
* Disable kv-ffload=false + multi-GPU tests for FP8
See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413
* Adjust test label for test-kv-cache.cpp
* Fix conflict-resolution mistakes in test-kv-cache.cpp
https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392
* Move KV tests to test-fp8.cpp
https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4410518158
* Remove tmp file for test-kv-cache-scales.py on Windows as well
* Fix fp8 kv cache test by population kv_source_impl
* Exclude models that consume KV cache directly
* Remove obsolete comment
* Allow BF16 scales as well for KV cache
* Rename *_cache_scale to *_cache_s
https://github.com/ggml-org/llama.cpp/pull/28392#discussion_r4517194999
* Move `is_type_scaled()` to `llama-kv-cache.cpp`
* ui: let a menu action open a submenu
Group and project pickers open as flyouts from a chat's menus. Master's menu
list has no submenus, so bring them back next to its tooltips, with a close
callback for picks made inside them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : store groups and let conversations join them
Groups live in their own IndexedDB table (schema version 2) with a name.
groupsStore creates, renames and deletes them; deleting a group keeps its
chats, out of any group. A conversation records the group it is sorted
into, and a group picked before the first message is kept pending and
applied when the conversation is created.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : show each group as a sidebar section
Every group is a sidebar section next to Pinned and Recent, folding,
showing its first six chats and dragged into place like them. A chat
dropped on a group joins it and leaves Pinned; dropped on Recent, it
leaves its group. Recent is called Ungrouped once a group exists, and its
menus create a group in a small naming dialog; so does a right-click
anywhere in the list that no row or heading takes, down to the empty space
under the last section. A group's heading starts a new chat in it, and its
menus rename or delete it; deleting keeps its chats, back in Ungrouped.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move chats into groups from their menus
A chat's menus, in the sidebar and on right-click, move it into a group
or take it out of its group. Desktop menus show a searchable group flyout
whose New group entry names a group in the group dialog and moves the chat
into it; a phone gets a picker dialog instead, which creates a group from
what is typed.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the group picker from a phone's hold menu
The hold menu is anchored to the row, so it showed the desktop flyout,
which ran off the left edge of the screen and whose New group closed
without naming one. On a phone, Move to group now opens the picker
dialog, as intended.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open a new chat in a tab of its own
With conversation tabs on, New chat from the sidebar, a group's heading
or the shortcut replaced the open chat's tab. It now adds a tab, as the
tab bar's "+" does, and the open chat keeps its own.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list a phone's groups under the picker's search
On a phone the picker fills the screen, and its grid spread the rows
over the height, so the list sat in the middle. It is now a column: the
list starts under the search and scrolls in the rest of the screen.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fix gaps in chat groups found in review
- A phone shows a group heading's New chat and menu, and the Ungrouped
heading's menu, which were reachable only by hover or right-click.
- A pinned chat moved into a group from its menu leaves Pinned, as a
drag onto the group already did, so the move shows.
- Only the sidebar's New chat, a group's New chat and the new-chat
shortcut open a new tab; stepping past the last chat, search and the
models drawer show the new-chat screen in the current tab again.
- An export carries each chat's group, and an import sorts the chat
into the group of that id or name, or creates it; a group the file
does not carry is dropped.
- The group picked with a group's New chat survives leaving the
new-chat tab and coming back.
- A phone's menus skip any submenu in the shared menu list, not only
the group one.
- Deleting a group forgets its section's place and fold.
- Groups load newest first by creation, so a rename no longer moves one
after a reload.
- groupsStore looks groups up in a map and answers which live group a
chat is in, for the three places that checked it by hand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a forked chat in its source chat's group
A fork copied the working directory but not the group, so it landed in
Ungrouped.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move a chat with its whole fork family between groups
Moving a chat into a group, or out of one, takes its parents and its
children along, so a fork tree never splits across sidebar sections. A
pinned member that joins a group leaves Pinned with the rest. Group
sections render the tree with its depth, as Ungrouped does.
* ui: show the group name prompt as a small centered card on phones
The new and rename group prompt opened full screen on a phone, like a
large dialog. It is now built on the alert dialog, as the chat rename
prompt is, so a phone shows it as a card in the middle of the screen.
Its Cancel is a plain button, so Enter in the name field saves.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: space a phone's group sections like Ungrouped
On a phone a group or Pinned section put its first chat right under the
heading, where Ungrouped leaves a row gap. They now leave the same gap,
and the sections sit a little further apart than the rows inside them.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Pascal <admin@serveurperso.com>
This commit removes the unused quantize_cuda_t from quantize.cuh.
The motivation for this is that the usage of this typedef as a function
parameter was removed in Commit 74976e1aef
("CUDA: remove -sm row, refactor cuBLAS (#24216)") and there is no longer
any usages of it.
The sampling chains are read on the host after the graph compute, and the
graph addresses are reused across builds with different selected rows: the
token-generation reserve graph has one output row per sequence, while the
actual block draft has n_max rows per sequence. The reserve allocates the
unselected chains without keeping them, so their buffers alias, and the
actual graph then reuses those addresses by node index.
set_outputs only marked the tensors stored for the selected rows, so the
unselected per-(sequence, row) chains were freed and reused. Collect every
chain tensor in t_sampling_all and mark all of them as outputs there.
Reproduces with -np 2 (non-unified KV) and 2 concurrent requests for
draft-dspark, draft-dflash and draft-eagle3; with different prompts one
sequence's acceptance collapses (eagle3 0.15 vs 0.40). After the fix:
dspark 0.71/0.78, dflash 0.56/0.44, eagle3 0.35/0.42.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ui: expand the chat form into a large editor for longer messages
The chat form expands into a large editor once a message has markdown
beyond plain paragraphs or reaches four lines, and a Maximize button in its
top-right corner toggles it by hand. The expanded form starts around half the
screen height and grows with the text up to the space below the tabs bar,
animating its height by mirroring the editor's content height.
On a new chat the expanded form keeps the collapsed form's top edge and the
greeting only fades out, waiting for a collapse to settle before it fades
back in. In a conversation the messages follow the form as it grows, so the
latest one stays in view above it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: make the expanded chat form a full-screen sheet on phones
In the expanded editor Enter adds a line and Cmd/Ctrl+Enter sends.
On a phone the expanded form is a full-screen sheet with no corners or
border. The page around it and Safari's bars take its colour, and the
sidebar's logo button and the working-directory chip step aside while it is
open. It moves by relative offsets and a clip on the main thread, so its
edges, text, actions and chip stay in step in every engine, and its content
keeps the collapsed card's places and widths, so nothing reflows when it
settles. Expanding does not focus the editor on a phone.
On a new chat on desktop the form is lifted by its sticky bottom offset
instead of a translate, and the expanded size is built on the collapsed
card's real height, so its top edge holds still through any sequence of
quick toggles.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the phone menu toggle and the sheet's focus edge while expanded
On a phone the menu toggle rides on the page, above the chat form: it now
fades out while the expanded form covers the screen, and comes back when
it collapses. The expanded sheet also drops the glass rim and inner ring,
so focusing the editor no longer outlines the whole screen.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the expanded phone sheet clear of the status bar
The expanded chat form fills the screen on a phone, under the status bar
an installed app draws over the page. Its text now starts below the bar
(pt-safe) and its collapse button sits below it too (top-safe-2.5).
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the phone's context gauge while the chat form is expanded
The gauge pinned to the phone's top right sat over the expanded sheet's
collapse key. Like the menu toggle across from it, it now fades out while
the expanded form covers the screen and comes back when it collapses.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: lift the expanded phone sheet's actions above the iOS keyboard
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move the chat form with a short, quick drift, without sliding after it lands
Its position transition came back on before the browser had laid out the new position,
so it slid the form there after the fade; the drift is now 10px each way and quicker.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style the expanded chat form with master's surfaces until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tighten the expanding chat form after review
- Expand on markdown only in finished lines: a lone `-`, `#` or `1.` on the
line being typed no longer opens the large editor.
- Show "Cmd/Ctrl+Enter to send" by the send button while expanded.
- Move the expansion state, the phone sheet and the held collapsed height
into useChatFormExpansion, and the sheet's geometry into pure functions.
- Drop the working-directory chip's phone animation, which never ran since
the phone chip moved to the + menu.
- Clearing or sending mid-move cancels the sheet's animation.
- A click only continues at the end of the text when it lands below it.
- Reuse ActionIcon for the toggle and INPUT_CLASSES for the card, derive the
centered form's offset from its drop, name the shared easing and fade.
- The greeting owns its fade behind the expanded form.
- MarkdownEditor's view is reactive, so following its content starts once it
exists.
Assisted-by: Claude (Fable 5.2)
* ui: keep the expanded chat form to the message being written
- The form expands on its own only for what is written or pasted, and
another conversation starts collapsed: a restored draft no longer opens
the large editor, or the phone sheet, on arriving at a chat.
- In the expanded editor the pickers open by the line being written, above
it or, short of room, below, instead of above the form's top edge.
- The sidebar's swipe is off while the phone sheet covers the screen.
- The send hint shows only beside the send button, where the row has room.
- The inner area keeps master's focus shadow, except on the phone sheet.
- The card and the form's position honour reduced motion.
- The phone greeting's place accounts for the wrapper it is counted from.
- The sheet's animation tolerates the form going away mid-move.
Assisted-by: Claude (Fable 5.2)
* ui: turn the typed pickers off on phones, and scroll long messages
- On a phone, typing `/` or `@` opens no picker for now: each mobile
browser moves the page its own way as the keyboard opens, so a list by
the form can not stay clear of the text being written. A note marks it
for later.
- The editor scrolls inside the form again: MarkdownEditor's own box sat
between the input and the editor without taking its height, so a message
longer than the form ran under the actions instead of scrolling.
Assisted-by: Claude (Fable 5.2)
* ui: settle the expanding chat form's review findings
- A phone expands the form only by hand: the sheet switching in mid-typing
moves the text under the keyboard.
- The typed pickers are off wherever a finger is the main pointer, not
below a viewport width, so a narrow desktop window keeps them and an
iPad loses them. The prompt picker opened from the + menu floats by the
sheet's actions on a phone.
- A message put back after a failed send expands the form again.
- Only the left button moves the caret from a click below the text.
- The sheet's fill is one token, used by the sheet, the page and Safari's bars.
- The collapsed-height hold is its own module and no longer finds the
editor's elements by class name.
- Shorter comments.
Assisted-by: Claude (Sonnet 5.5)
* ui: keep a one-line message on the chat form's actions row
On a desktop the main chat form is a single pill while its message is one
line: the add button, the text, then the model selector and send. A second
line, or a line that wraps beside the actions, brings back the two rows,
and they stay until the text is shorter than where it wrapped, so the
wider row can not flip it straight back. The editor stays mounted: the
actions join the text's row by CSS alone.
The single row's corners are `--chat-form-single-row-radius` in app.css, a
pill by default; the "Scrapbook / ChatForm single row" story shows the real
form with other radii.
Assisted-by: Claude (Opus 5.5)
* ui: give the one-line chat form 1.25rem corners
Assisted-by: Claude (Opus 5.5)
* ui: pick the working directory from the + menu on desktop too
The chip under the chat form is gone: the + menu offers the working
directory everywhere but a phone's chat, whose header shows it. The menu
item names the directory once one is set, and the picker gains a row to
clear it, which the chip's X used to do.
Assisted-by: Claude (Opus 5.5)
* ui: put the single-row chat form behind a Display setting
"Single-row chat form for short messages", on by default.
Assisted-by: Claude (Opus 5.5)
* ui: send the chat form on Enter, and add a line with Shift or Ctrl/Cmd
Enter sends wherever the caret is, the expanded editor and code blocks
included; Shift+Enter and Ctrl/Cmd+Enter add a line, continuing lists and
quotes. With "Send message on Enter" off the two swap as before. The
expanded editor's send hint and the code-block rule are gone with it.
Assisted-by: Claude (Opus 5.5)
* ui: start the chat form on the row layout it will settle on
Assisted-by: Claude (Opus 5.5)
* ui: show a set working directory under the chat form again
On desktop the chip under the form is back once a directory is set, and
the + menu offers the directory only until then.
Assisted-by: Claude (Opus 5.5)
* ui: add a line on Enter with an on-screen keyboard
A phone's Return has no Shift, so on a touch screen Enter adds a line and
the send button sends, as "Send message on Enter" turned off does; on a
desktop Enter still sends.
Assisted-by: Claude (Opus 5.5)
* cuda : Lowering VMM pool reservation size to 8 GB on GPUs with combined memory due to CUDA out of memory errors
Assisted-by: Gemini Argon
* cuda : Halve VMM pool reservation size and retry on failure
Assisted-by: Gemini Argon
* cuda : only shrink VMM pool on CUDA_ERROR_OUT_OF_MEMORY
Assisted-by: Gemini Argon
* cuda : log warning when VMM pool reserves less than max size
Assisted-by: Gemini Argon
ggml_cuda_graph, ggml_cuda_concurrent_event and ggml_cuda_stream_context
are only used by ggml-cuda.cu, but keeping them in common.cuh meant any
change to them rebuilt every CUDA translation unit. Move them to graph.cuh
and leave forward declarations in common.cuh. The context constructor,
cuda_graph() and any_cuda_graph_*() are now defined in ggml-cuda.cu, and
the stream context is held through a unique_ptr.
Also define ggml_cuda_kernel_can_use_pdl once in ggml-cuda.cu instead of
as a static function in the header, so there is one PDL cache instead of
one per translation unit, and fold the GGML_CUDA_PDL env check into it.
* ui: use a live-preview markdown editor for the chat form input
Replace the contenteditable rich input with a CodeMirror editor that shows
the markdown source of the block being edited and renders every other block
through the MarkdownContent pipeline and stylesheet, so they match chat
messages. Plain text types exactly as before, the value stays the exact
markdown source, and mention links show as atomic badges.
The editor lives in a shared MarkdownEditor component, and the highlight.js
theme loader is shared between MarkdownContent and SyntaxHighlightedCode.
The old rich input, its tokenizer, source history and their tests go away.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the production build more memory, note what mdsvex is for
The single-bundle build now includes CodeMirror and runs past Node's default
heap limit on CI runners. Raise it in the build script, so local and CI builds
behave the same on every platform.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the live preview's blocks while only the cursor moves
Moving the cursor re-walked the syntax tree and rebuilt every block's
decoration. Keep the blocks with the decorations and rebuild only when the
text changes or the cursor enters another block.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: point the chat form story at the editor and drop the engraved placeholder until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the markdown editor for screen readers
CodeMirror's content is a textbox with no name of its own, so the chat form
input was announced as an unnamed text box, and could not be found by its name
as the old input could. It takes an aria-label, its placeholder by default,
which makes the chat form's "Type a message..." again.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: format the chat form input's classes
Assisted-by: Claude (Opus 5.5 Medium)
* ui: share the markdown styles and diagram drawing between messages and the editor
The editor and MarkdownContent each @imported markdown-content.css into their own
<style>, so the bundle carried two scoped copies of it. Both now render into a
MarkdownSurface, the one element that carries the stylesheet; the few rules for
MarkdownContent's own template elements are global under .markdown-content.
Drawing mermaid and svg blocks and the actions of their header buttons move to
markdown-diagrams.ts, so the editor's rendered blocks can draw diagrams the same
way messages do.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: harden the live preview editor after review
- onChange skips only the editor's own syncs from `value`, marked by an
annotation, instead of reporting only edits that carry a user event
- Enter and the mention badges ask the syntax tree whether they are in code, so
~~~ fences and indented code count, and a closing fence ends a block
- a picked mention is one undoable edit, through replaceText
- rendered blocks draw mermaid and svg diagrams with their header buttons, a $$
math block with blank lines renders as one block, and a link reference
definition stays raw instead of rendering to nothing
- rendered blocks render again when the mention path settings change
- a drag in a rendered block selects its text and copy takes it, across blocks;
a click still opens the block
- the block margins are the ones a chat message renders (each heading is a first
child there), checked by a test; the blank lines after a raw block are its gap
- the tree comes from syntaxTree, without forcing a full parse on every edit
- placeholder, label, extensions, mention widgets and render key follow prop
changes after mount, through compartments
- the editor uses the markdown language alone, which drops the html, css and
javascript parsers markdown() pulls in for html tag completion (-70 KB gzip)
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the CodeMirror packages and set the build heap in .npmrc
The new dependencies are pinned like the rest of package.json. The build heap
moves from a node call on vite's bin path to npm's node-options, so the build
script is plain `vite build` again and the setting reaches it on every platform.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: update the live preview block by block and estimate block heights
An edit used to build every block's decorations and widgets again, which the
editor then compared one by one. The previous decorations, and the mention badges
of raw blocks, now move through the edit, and only the blocks whose range, text,
raw or rendered state, or gap changed get new ones; the texts of the blocks an
edit left alone are reused instead of sliced again. A test drives random edits,
cursor moves and focus changes and checks each result against a full build.
Rendered blocks report the height they were last measured at, or a guess from
their text, so CodeMirror's height map for blocks off screen in a long document
stays close and scrolling stays steady.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: skip the editor's redundant syncs, reconfigures and theme reloads
- syncing `value` compares it with the text the editor last reported or synced, so
a keystroke coming back through onChange no longer serializes the document again
- the prop effects reconfigure the editor only when a prop changed, not once more
right after it is created (which rebuilt the whole preview)
- loadHighlightTheme leaves the theme's style element alone while the mode stays,
so a message or an editor mounting no longer restyles the whole page
Tests check that typing in one block leaves the other rendered blocks without a
single DOM mutation, and that mounting reconfigures nothing.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: simplify the live preview to a plain rebuild in three small modules
Rebuilding the decorations of every block on an edit measures the same as
patching only the changed ones (0.09 ms against 0.11 ms per keystroke at
200 blocks, 0.9 ms against 1.1 ms at 2000), and equal widgets keep their
DOM either way, so the incremental update and its test go.
- markdown-blocks.ts collects the document's blocks, markdown-block-widget.ts
renders one, markdown-live-preview.ts decides which ones show raw text
- one bounded cache holds a block's HTML and its measured height
- the render key is a plain option: a new one reconfigures the preview,
which keeps the editor's focus
- MarkdownEditor follows its props with one helper instead of five effects
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep setext headings raw in the live preview editor
`test` over `---` is a setext heading, so the preview rendered it as an h2
and the underline vanished once the cursor left the block. The editor now
shows a setext heading's text as typed at all times, like a link
reference definition.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a setting to turn the live markdown preview off
"Live markdown preview in the message input" (General) is on by default.
Off, the input shows the plain markdown text, with file mention badges
still in place; switching it takes effect without a reload.
Assisted-by: Claude (Opus 5.5 Medium)
normalize_quotes_to_json copies raw control characters verbatim while
converting pythonic tool arguments to JSON, so calls like
python(code='a<LF>b') produce invalid JSON in the arguments returned to
clients; replaying that history then fails hard. Escape C0 control
characters inside quoted strings via escape_json_string_inner.
* sycl: Add device-side MoE scheduler
Assisted-by: Codex
* Add width-heuristic override bit and Q2_K and Q3_K types
Assisted-by: Codex
* tests: cover skipped MUL_MAT_ID slots on the device MoE schedule
Assisted-by: Claude Opus 5.5
* sycl: keep graphs on for lazily reordered MUL_MAT and reject BN=64 combos on the device MoE schedule
The graph check disabled SYCL graphs until every MUL_MAT weight with a
reorder extra was reordered, which never happens for some (dense MXFP4,
the fused GLU plain path), so graphs stayed off and the check reran on
every compute. Upstream already reorders those during recording with
the async memory extension, so only MUL_MAT_ID weights are reordered
ahead of recording now.
ggml_sycl_grouped_dequant_gemm_device_supported() accepted any XMX
combo, but the device schedule only runs with BN == GGML_SYCL_FG_BN, so
a 32x64 combo (GGML_SYCL_XMX_GATHER_SHAPES=4 or 8) made the graph check
pass and the compute fall back to the host sync path while recording.
Assisted-by: Claude Opus 5.5
* sycl: keep the ggml_sycl_mul_mat_id_needs_sync name for the graph check
Assisted-by: Claude Opus 5.5
* fix: stop erroneously reordering host-offloaded tensors
With -ncmoe, prompt processing offloads the CPU-resident expert matmuls
to the GPU: the scheduler uploads the experts into a copy in the compute
buffer before every run. init_tensor gave that copy a reorder record, so
the MUL_MAT_ID gather path reordered every offloaded expert tensor on
every ubatch, and the next upload overwrote the result.
Qwen3-30B-A3B Q4_K_M, 1x B60, llama-perplexity -c 4096 -ub 2048
-ncmoe 48, 8 chunks, on master ac8e1694:
master 116 s, PPL 7.739
master XMX_GATHER_TYPES=0 67 s, PPL 7.738
with this change 67 s, PPL 7.729
Refs ggml-org/llama.cpp#31262
Assisted-by: Claude Opus 5.5
* Remove SYCL-specific test comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sycl : name the MUL_MAT_ID path predicates after what they decide
ggml_sycl_mul_mat_id_mmvq_ok -> ggml_sycl_mul_mat_id_should_use_mmvq
ggml_sycl_mul_mat_id_mmvq_shape_ok -> ggml_sycl_mul_mat_id_mmvq_eligible
ggml_sycl_mul_mat_id_grouped_shape_ok -> ggml_sycl_mul_mat_id_device_sched_eligible
node_is_device_local -> graph_node_is_in_local_device
ggml_sycl_gg_rows -> ggml_sycl_grouped_gemm_rows
ggml_sycl_gg_tile -> ggml_sycl_grouped_gemm_tile (from #29245, renamed to match)
"eligible" is a necessary condition; the type and width checks follow
elsewhere. "should_use" means the path runs. No functional change.
Assisted-by: Claude Opus 5.5
* sycl : reorder Q3_K MoE experts and load them as vectors in the grouped GEMM
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: cwriter <cwriter@localhost>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vulkan: tiled transpose for dim-0 concat with a transposed source
The delta-net conv-state path does ggml_transpose() straight into a dim-0
ggml_concat(), so the generic concat kernel reads src1 with the transposed
stride, one element per thread, and neighbouring lanes land on different cache
lines. Stage a 32x32 tile through shared memory instead, which keeps both the
load and the store coalesced. 33 columns pads away the bank conflicts.
Gated narrowly: dim 0, 2D, unquantized 4-byte elements, contiguous src0/dst
rows, and src1 actually transposed. Everything else falls through to the
existing kernel.
Qwen3.6-35B-A3B on gfx1151/RADV, 30 gated-delta-net layers: the CONCAT op goes
11877 us -> 957 us, and pp2048 at ub2048 goes 1224 -> 1628 t/s. Generated text
is unchanged.
The approach follows the tiled transposed concat Gaetan Puleo wrote for the HIP
backend in his Strix Halo fork.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* enable by default, add smaller shader sizes
* clean up comments
* improve shader
* Add non-contiguous concat tests
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
* handle x or y axis overflow
---------
Co-authored-by: Nathan Wilson <nath.flagman151@passmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com>
* ggml-rpc: fail gracefully on unreachable endpoints and failed handshakes
rpc_dispatcher::start() aborted the whole process (GGML_ABORT -> SIGABRT) when
the endpoint could not be parsed, the transport failed to initialise, the socket
could not connect, or the HELLO handshake failed. A typo in --rpc or a
temporarily down node killed the host application, and the reason went to a log
stream the host may never see.
Make the connect-time failure reportable instead of fatal: start() returns bool
(LOG_ERROR + false; the dispatcher thread only starts after a successful
handshake), negotiate_hello() returns false on a half-open socket,
get_dispatcher() returns nullptr and does not cache failed attempts, and every
consumer treats nullptr as "endpoint did not answer" with the conservative
behaviour of its domain (allocation -> nullptr, alloc-size -> lower bound,
device memory -> 0/0, events -> no-op, device count -> 0, and add_server()
already maps 0 devices to nullptr).
Mid-stream failures keep the existing RPC_STATUS_ASSERT by design (fail fast
against silent corruption) and are intentionally out of scope.
Field motivation: on Android (llama.rn + phone RPC cluster) an unreachable
worker crashed the app; the crash buffer showed
abort <- ggml_abort <- rpc_dispatcher::start <- ggml_backend_rpc_add_server.
* tests : add a regression test for unreachable rpc endpoints
Calls ggml_backend_rpc_add_server with a socket that nothing listens on and with a
malformed endpoint, and expects nullptr from both. Without the fix in this branch
the first call aborts the process, so the test cannot even fail cleanly.
* ggml-rpc remove low-value comments
* tests : drop the internal device-count check after the rpc registry refactor
---------
Co-authored-by: figkcoci-debug <figkcoci-debug@users.noreply.github.com>
With head sizes above 128 and KV >= 1024, scalar flash attention uses
16 rows per workgroup. On the Qualcomm Adreno driver this variant
spills registers to scratch memory (264 bytes per invocation on the
X1-85, none at 8 rows) and runs about 40x slower. The largest
test-backend-ops case then becomes a single dispatch of several
seconds, past the Windows GPU timeout, and the test never completes on
the X1 Elite.
Use at most 8 rows per workgroup on Qualcomm when hsk or hsv > 128.
Smaller heads keep 16 rows, which is faster for them on the X2 Elite.
Adreno X1-85, FLASH_ATTN_EXT hsk=256 kv=4096: nb=64 711 ms -> 17 ms,
nb=512 no result -> 125 ms; test-backend-ops completes. Adreno X2-90:
Gemma 4 E4B pp512 at depth 4096 with flash attention 81-90 -> 117 t/s.
On a device without mmap support, --lazy-mode auto resolved to off even
with --load-mode mmap, so llama_prefetch_rows never ran and the rows of
the large tensors faulted in one page at a time. Resolve it to off only
when mmap is off; the default iGPU load path is unchanged.
Fixes#31943
* server: add oai-compat /v1/audio/speech endpoint
* add e2e pytest
* server: share the s16 rounding, type speech form fields
The WAV writer of the audio helper truncates while pcm_f32_to_s16
rounds, so wav and pcm differ by one LSB. Both now round.
pcm_f32_to_s16 takes a pointer and a count and converts the TTS
bytes without an intermediate copy.
/tts and /v1/audio/speech parse their body with parse_tts_body, so
multipart fields such as speed get their type before the speech
conversion checks them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* mtmd: support gemma 4 input audio stream
* use 480ms step
* server: enable realtime embeddings only for audio generation
The realtime slot forced embeddings on, which marks every row of a
batch as an output. Only a model that generates audio reserves
n_batch outputs, so a chat model with an audio stream (Gemma 4)
aborted on its first batch of audio rows. The hidden state is only
read by the audio generation pipeline.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
A cold dev server reloads the page once it has optimized its
dependencies, and that reload can land right after the harness opened
the phone sidebar, closing it again before Settings is clicked. Repeat
the pass over the menu keys up to three times while Settings is not
reachable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: reveal the mobile sidebar by sliding the page aside
On a phone the sidebar now stays put under the page, and the page slides
right to uncover it, as a rounded card with the card border. The menu
button rides on the page; a swipe from the left edge opens the sidebar,
and a swipe back or a tap on the page closes it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: polish the mobile sidebar reveal and add a hold menu to rows
Holding a conversation row on a phone opens its action menu anchored to
the row, in place of the hidden three-dot trigger. The page no longer
widens the document when it slides aside, so iOS stops panning it on the
first open.
The page and sidebar backgrounds start in the page color and blend into
the sidebar tint as the sidebar opens, following a drag frame by frame,
and a theme-color tag follows them for the browser bars. The sidebar's
contents fade in on a log curve and scale up slightly, and sit at the
same inset as the menu toggle, which now uses a two-line icon in a
filled circle.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the context gauge on phones and lock the slid-aside page
On a phone the context gauge leaves the chat form for the top right of
the chat screen, in a round button across from the menu toggle. Both
take the scroll-down button's frosted look.
While the sidebar is out, and until the page is fully back in place,
the page neither scrolls nor takes taps; a tap or drag on it still
closes the sidebar.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the mobile sidebar with a swipe from anywhere
A rightward swipe from anywhere on the page now opens the phone sidebar
too, beside the edge swipe that competes with the browser's own back and
forward gesture. It starts after a short sideways move, skips text
fields and content that scrolls sideways, and keeps the page from
scrolling under it. Letting go opens or closes the sidebar past 30% of
the way or on a flick.
The top buttons and the chat form now hold their place on screen while
the page is slid aside, instead of jumping to the far ends of a
scrolled conversation.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the model org name on mobile unless a setting keeps it
A phone's model selector is narrow, so it now drops the organization
from model names by default. A new Display setting, "Show organization
name on mobile", brings it back; it applies only while the organization
name is shown at all.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: close the mobile sidebar before following a tap in it
On a phone, picking a conversation, New chat, Search, MCP or Settings
in the open sidebar now slides it shut first and acts 180ms later, so
the page or dialog appears once the sidebar is out of the way. With the
sidebar already closed, or on a desktop, the action runs at once.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: make the app feel native on phones and run it under the status bar
Applies the mobile-native checklist: no tap highlight or landscape text
inflation, instant taps on buttons and links, no label selection or
callout on a held control, hover styles only where a pointer can hover,
and 16px textareas on phones so focusing them does not zoom the page.
The page now runs under the notch (viewport-fit=cover), so the installed
iOS app tints its status bar from the page and the sidebar. The menu
toggle, the context gauge, the sidebar and the top of the chat and
models pages pad themselves clear of it, and the locked, slid page
reaches down by the status bar's height, which iOS leaves out of it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a viewport height readout for debugging on phones
With VITE_DEBUG set, a dev build shows the window, viewport unit and
safe-area heights, and the elements that reach past the window's bottom
and make the page scroll. It is loaded on demand, so production builds
leave it out.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fill the screen from launch in the installed iOS app
iOS 26 gives an installed app with viewport-fit=cover and a translucent
status bar a viewport short of that bar (WebKit bug 301108), clipping
the bottom of the page and lifting the chat form, but only while the
page is shorter than the screen. In the installed app the page now
reaches the screen plus the status bar, capped at the screen, so it
gets the full screen from the start without ever scrolling.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: raise the phone's corner controls in the installed app
The installed app's status bar already sets the menu toggle, the context
gauge and the open sidebar's logo apart from the screen's edge, so they
sit just below it rather than a full 1rem lower. They share one value,
which stays at 1rem below the inset in a browser tab.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add safe-area utilities for what is pinned to a screen edge
The page runs under the status bar and over the home indicator, which an
installed iPhone app always shows. New pt-safe, pb-safe, top-safe and
bottom-safe utilities, each with a spacing step (pt-safe-12 is 3rem plus
the top inset), keep pinned content clear of them instead of hand-written
calc()s. A --safe-area-bottom joins --safe-area-top, and the chat screen and
Models page use pt-safe-12.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open search in a full-height drawer on phones
A phone used to leave for the /search page. It now opens the same search
as the desktop dialog, in a bottom drawer that fills the screen below the
status bar. Keyboard shortcuts, the Enter hint and the keyboard highlight
stay on desktop, and so does the models manager, as in the phone sidebar.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show every dialog full screen on phones, clear of the status bar
A phone now shows every dialog full screen, whatever size it asks for on a
desktop, instead of each full-screen dialog repeating its own classes. Its
top and bottom borders take the status bar's and the home indicator's
height, transparent, so an installed app's dialogs keep their content clear
of both with no change to their padding. The attachment preview, which has
its own full-screen content, pads itself and lowers its close button the
same way.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fade the page out under the status bar of the installed app on phones
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show search in the full-screen dialog on phones instead of a drawer
Assisted-by: Claude (Opus 5.5 Medium)
* ui: start a phone page's title a row below the corner controls
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the slid page's bottom bars against the screen in the installed iOS app
Assisted-by: Claude (Opus 5.5 Medium)
* ui: cast a soft shadow from the slid page onto the phone sidebar
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the scroll-down key at the real bottom on iOS and show it only well above
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fade the status-bar fade out as the page slides aside for the sidebar
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw the slid page's shadow above its rounded frame, so it follows the corners
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a phone chat's title and working directory at the top, and pick it in a drawer
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give iOS Safari a solid top and fade the bottom of the screen on phones
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let iOS Safari find the top strip, which it skips when it takes no taps
Assisted-by: Claude (Opus 5.5 Medium)
* ui: lengthen the top fade and lift the chat form a little in phone browsers
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give iOS Safari's bottom toolbar the page's colour, not the chat form's
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw the phone's context gauge key like the menu key across from it
Assisted-by: Claude (Opus 5.5 Medium)
* ui: confirm opening the phone sidebar, its hold menu and its rows with a haptic tap
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the screen-edge fades on the page as it slides aside for the sidebar
Fading them out with the slide made the page flicker; the rounded frame over the
slid page already clips them at its corners.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the working directory chip optional for hosts with their own trigger
The phone layout opens the picker from the chat's header, so ChatForm hides the
chip there. Master's version of the chip has no such switch; bring it back.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style layer 5's additions with master's tokens until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the design system's stylesheets and fonts from app.css until it lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep master's font stacks until the design system brings its fonts
Assisted-by: Claude (Opus 5.5 Medium)
* ui: restore master's colour palette in app.css until the design system lands
Resolving the safe-area commit took the design system's app.css, which drops
the palette for its token files and aliases the shadcn names to semantic
tokens. With the tokens gone those aliases point at themselves, so every
colour went invalid: transparent surfaces and currentColor borders.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: center alert dialogs on phones instead of sliding them up as drawers
An alert is a short question, not a drawer: on a phone it is now a centered
card, 90% of the screen wide and as tall as its text, scrolling only past the
screen's height. It zooms in as on desktop, and its footer drops the margin
the sheet needed.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: space the working directory drawer for a phone
The drawer reused the popover's compact picker: Browse sat against the search
field, the divider ran past the rows and each row had its own inset. In the
drawer the picker now takes the header's inset, gives Browse room, lines its
rows up with the search results and sizes its text for a phone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tap the haptic engine when a message sends, and once per hold
A phone now confirms a sent message by feel, at the moment the send goes
through (not on every press of the button), and before any await, so it still
runs inside the tap or keypress that iOS needs to play its tick. The hold that
opens a row's menu now taps the engine from the long-press hook itself, once:
the hook vibrated and the row tapped as well, which buzzed twice on Android.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the gauge dial sizes, the chat form's bottom offset and the sidebar fade
Review follow-ups: the context gauge dial reads its grid, radius, stroke and
track from one table per size instead of ternaries; the chat form's distance
from the screen's bottom per platform and the sidebar content's fade duration
are named constants. The viewport debug readout moves out of src, next to the
tests, as it is a debug tool the app only loads in a dev build.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say why the microphone can't record instead of failing silently
A llama-server opened over plain HTTP on the local network or over Tailscale
is not a secure context, where browsers hide the microphone, so a tap on the
mic did nothing but log to the console. The button now shows a crossed-out mic
in the warning tone with the reason in its tooltip, and a tap says it in a
toast, as a phone shows no tooltip. A start that fails names the browser's
reason (access denied, no microphone, in use by another app) the same way,
and the next tap tries again. The recorder passes the browser's error on
instead of replacing it with a generic one.
Assisted-by: Claude (Opus 5.5 Medium)
* model : allow loading external PLE files
Assisted-by: Codex
* convert : add separate PLE exports
Assisted-by: Codex
* tests : fix PLE export test lint
Assisted-by: Codex
* model : make --ple an alias of --shard, allow multiple shard locations
Reuse the multi-shard loading path for external PLE files instead of a
separate option.
* --ple is an alias of --shard: both take a comma separated list and may
be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
common_model_load() uses llama_model_load_from_splits for two or more
paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
memory measurement load every shard
Assisted-by: GPT-6-Astra
* rm -isnm
* make sure to dedup paths
* rm test_convert_ple
* address comments
* allow ple shard to override tensor
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* Windows unbuffered io
* Fix read len calculation
* Add test and model load timing
* chunked fallback read + removing filepointer semantic divergence between linux and windows
* tests : remove unused mmap test arguments
Assisted-by: Codex
* remove timing, set file pointer after chunk read, test refinement
* Update tests/test-llama-mmap.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Update src/llama-mmap.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* test refinement
Assisted-by: Codex
* remove duplicate constexpr
Assisted-by: Codex
* adding label behavior to tests
Assisted-by: Codex
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* server : functionality test to make sure every time when we add new spec dec method the existing ones are not broken.
* server: use compact models for speculative decoding CI
* ci: avoid GHA cache for speculative models
* ci: run speculative tests on DGX Spark
* server: address speculative test review
* server: use consolidated speculative models
* server: refine speculative test calibration
* docs: record speculative Spark validation
* ci : run the server-speculative steps with bash
* server: recalibrate speculative test floors on GB10
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server : accept top-level schema in response_format with type json_schema
The OpenAI-compatible chat completions endpoint only applied the schema
when it was nested inside response_format.json_schema.schema. When the
schema was provided directly as response_format.schema (as documented in
the server README, and used by some clients), it was silently ignored
and generation ran unconstrained.
Also mention json_schema in the error message for invalid response_format
types, document the OpenAI wrapper form in the README, and add a test
case for the top-level schema form.
Fixes#10732
* server : let the json_schema wrapper take precedence by presence
Pick the schema source based on whether response_format has a
json_schema key, instead of falling back when the parsed wrapper schema
is empty. An empty schema {} is valid (any JSON value), so the old check
let a top-level schema override an explicit {} in the wrapper. Use
json_value() for the top-level schema too, like the rest of the
function, and add tests for the precedence.
---------
Co-authored-by: linsen <251731047+linsen458-spec@users.noreply.github.com>
* ggml: implement the backward pass for TRI
GGML_OP_TRI had no case in the backward switch, so a trainable graph containing it aborted with "unsupported ggml op for backward pass: TRI". The op is linear, so its gradient is masked the same way. The incoming gradient is not necessarily contiguous (the backward of a transpose hands over a view) and ggml_tri requires contiguity, so it is made contiguous first.
test_tri gains a post_transpose variant (out = transpose(tri(a))) that aborts as above without this change.
* ggml : simplify the TRI backward pass
Pass the gradient directly to ggml_tri and drop the transpose test. The existing TRI cases cover the backward pass for contiguous F32 gradients.
The dummy-model generator fills every tensor from a plain normal pool.
`blk.N.ffn.act.scales` is a divisor in the mpt graph, so with seed
1224489258 it draws |v| ~ 1e-6, and `ffn_act = gelu(...) / scales` reaches
2.7e5. Metal stages both operands of `kernel_mul_mm_f32_f32` as half, so
those activations overflow to inf, the norm downstream turns the inf into
nan and the logits come out nan, while the CPU reference stays in f32 and
stays finite. Both NMSE values then become nan and the check fails.
Give `ffn.act` the same treatment as `rope_factors` and keep its magnitude
away from zero.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* ui : fold the sidebar's conversation sections
Pinned and Recent conversations fold under their heading. The whole heading
row is the toggle; its chevron follows the title, shows on hover and stays
while the section is folded. Folded sections are remembered across reloads.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : show a sidebar section's first six chats
Pinned and Recent list their first six chats and end with a Show more
row that opens the whole section; Show less, sticking to the bottom of
the sidebar while the section scrolls under it, closes it back. The
expanded sections last for the session.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : rearrange the sidebar's sections and chats by dragging
Pinned and Recent can be dragged by their title. The real section follows
the pointer, tinted like the sidebar's resize handle and edged with a
faint border, the other sliding aside as it passes; let go, it glides from
there into its new slot. The order is remembered across reloads, and a
section never moved keeps its default place. All sections now scroll
together.
A chat dragged onto Pinned is pinned, onto Recent unpinned. A lifted copy
of the row follows the pointer while the row slides out of its section.
The section under the pointer takes the same tint and border; a section
ignores its own chats. Let go, the row slides in where its section sorts
it, the rows below making room, and the copy glides into it; anywhere
else, it slides back into its own section.
While dragging, the rows and headings passed over take no hover. Dragging
needs a mouse or a pen, so a finger still scrolls the sidebar; the sidebar
scrolls near its edges, and Escape cancels.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : set the sidebar's sections apart from their rows
A section heading now reads above its chats: larger, semibold and in the
full foreground colour. Chat titles drop to a regular weight in a slightly
softened colour, full again when the chat is open or hovered, and "Show
more" drops to a regular weight in the muted colour. Chats inside a
section sit closer together, sections further apart, and the sidebar's
actions get more room above and below.
Assisted-by: Claude (Opus 5.5 Medium)
* fix(rocm,ci): Update ci workflow to use devlab ci runners for gfx1151
* ci : move gpu-rocm job to Strix Halo devlab runners
Assisted-by: Claude
* ci : ROCm 10 wheels with per-GPU device packages for gpu-rocm; add gfx1201
The runners' system ROCm 7.2.1 cannot compile the current HIP code for
gfx1151 ("Invalid dpp_ctrl value" in mmf-instance-ncols_*.cu). Go back
to the TheRock ROCm 10 wheels with the device-<gfx> extra, so rocBLAS
has the target's kernels, and keep ci/run.sh's results and models under
$RUNNER_TEMP instead of the runners' RAM-backed home. Add a gfx1201
(Radeon AI PRO R9700) row with HIP graphs disabled.
Assisted-by: Claude
---------
Co-authored-by: David Hekimian <dhekimian@users.noreply.github.com>
* metal: make ggml_tri select instead of scaling by the comparison
The kernel multiplied by the 0/1 comparison, which turns a +-inf outside the triangle into NaN, while the CPU kernel writes a plain zero there. Select, like the CPU does.
Adds test_tri_special, which puts +-inf into the cells that have to be discarded, for all four triangle types. It fails against the previous kernel and passes with this change.
* Just trim down the repetitive comments.
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3
Assisted-by: Claude Opus 5.5
* feat: add temp parameter in llama-tts
* temp: fix default hardcoded values
* fix: remove too narrow test
* fix: addressing PR comments
The rotation angle of a position p is rounded in f32 to about
p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach
it through different rounding sequences (cumulative product versus
pow). With positions up to 8192 this puts the NMSE floor between
backends near 8e-8, so the default 1e-7 threshold fails at random on
Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for
n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
* ggml : fix for the radix TOP_K thread timing bug
* make the key an ordered 64-bit pair
* a rewrite without shift-skip logic
* fix casts
* warp-contiguous layout
* reduce the scope removing vulkan+metal changes
* turn it off for Metal/ROCm, note about cub::DeviceTopK
* fix the hip build
* skip Vulkan as well
* popc lambda from fattn
* do block_cols math in int64_t
* make it allowlist bc SYCL looks non deterministic too
* ggml : make Metal radix Top-K tied selections deterministic
The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.
Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.
This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : enable deterministic top-k check on Metal, add radix-path tie cases
Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.
Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : use coalesced accesses in the Metal radix Top-K compaction
The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.
Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : remove the extra top-k tie and perf cases
The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal: scan the radix bins and warp totals with a simdgroup
The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.
* tests: enable the deterministic top-k check on ROCm
HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.
* cuda: remove the redundant radix top-k counter reset
Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.
Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).
Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.
Assisted-by: Claude Sonnet 5
Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
* ggml : switch to F32 src1 in the CPU backend
Change the default computation for GGML_TYPE_F16 to use F32 for src1
data in matrix multiplications: vec_dot_type(F16) is now F32 and the dot
product goes through the new ggml_vec_dot_f16_f32 (x = F16 src0, y = F32
src1), so src1 is no longer quantized to F16 and worst-case inputs that
only fit in the F32 FP range are handled correctly.
- conv_2d/conv_3d: store im2col patches (the inner mul_mat src1) as F32
and pass separate src0/src1 types to the inner mul_mat; the previous
F16 patches made the inner mul_mat widen src1 into the conv work
buffer, corrupting the patches and crashing on large inputs
- conv_transpose_1d/2d (F16 kernel): keep src1 in F32 in the work buffer
and use ggml_vec_dot_f16_f32; update work buffer planning accordingly
- supports_op: restore MUL_MAT support for F16 src1, which is now
widened to F32
- ggml_vec_dot_f16_f32: add GGML_SIMD support by converting chunks of x
to F32 and reusing the F32 SIMD dot
- mul_mat/mul_mat_id: add TODOs for a more efficient F16 x F16 path that
can be used if ggml_prec_get_src(dst, 1) allows converting src1 to F16
MUL_MAT f16 x f32 perf on M5 Ultra (GFLOPS):
- m=2048, n=512, k=16 : 120.9 -> 101.7
- m=2048, n=512, k=64 : 150.5 -> 344.2
- m=4096, k=14336 GEMV: 63 -> 343-415
- m=4096, n=512, k=14336: 65.6 -> 770.2
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : add some F16 x F32 vectorized paths
* cont : enable the fused F16 x F32 dot on AVX without FMA
The 256-bit fused path of ggml_vec_dot_f16_f32 required AVX2 and FMA,
so CPUs with AVX and F16C only fell back to the chunked conversion and
F16 decode was 4x slower than before. Enable it on __AVX__ && __F16C__
and accumulate with GGML_F32x8_FMA, which does mul + add without FMA.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls
Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.
* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends
Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.
* tests: cover GGML_PREC_BF16 in the flash attention tests
The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.
* llama: request GGML_PREC_BF16 for the KQ multiplication
* cont : add TODO in clip.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* opencl: add kernel_gemv_noshuffle_q8_0_f32_mc
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for q5_1, q4_k
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: adreno kernel k floor 256 for x2
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for more quants
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: rename function
* opencl: disable bin FA for small verify batch
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* hexagon: use absolute skel path
* hexagon: support non-ascii dir and long path in skel-abs path
* hex-libdir: add support for FASTRPC_MANAGE_DSP_LIBRARY_PATH and GGML_HEXAGON_LIBHTP_DIR on all platforms
* hex-libdir: keep extra fwd slash in abs path on Android and Linux
* hex-libpath: always start with absolute path then fallback to session ctl then env
* hex-libdir: add CDSP1,2,3 to the fallback, needed for the IQ10
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* hex-nhvx: apply nhvx updates to the latest master
* hex-nhvx: set N_HVX and N_THREADS based on the arch to avoid overallocation
* hex-build: add support for enabling HTP_DEBUG via build.py
* hex-trace: move tracing state/helpers into new htp-trace.h header
* hex-dcache: cleanup dcache #defines
* hex-hnvx: minimize the diff by reflowing the macros
* hex-ci: fix python type/linter warnings
* hex-nhvx: consitent logging
* mtmd: support MageVL-Cowork model
* mtmd: keep magevl rope_freqs at model level and check its shape
The rope factors are shared by every layer, so they live in clip_model
instead of a pointer copied into each clip_layer. The loader now checks
that v.rope_freqs is F32 with d_head elements and that d_head splits
into the 4:6:6 pairs the graph expects.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* moe-cache : compute part of the misses on the CPU
--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.
- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
selected experts, so the CPU part does not wait for the device, and its
result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
simulator of the cache
* scripts : add moe-cache-tune.py to pick the MoE cache arguments
The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.
docs/moe-cache.md describes the MoE cache arguments and the script.
* add am17an as codeowner for MoE cache
* moe-cache : upload all the misses until the cache is full
With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.
* add backend sync for async transfer queue
* moe-cache : let the scheduler copy the results of the CPU part
* Fix MSVC compilation
* remove sync change
* llama-graph : extract the CPU share condition into can_cpu_share
* llama-graph : make llm_graph_params::moe_cache const
The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* scripts : replay the ClassLRU policy in moe-cache-tune.py
* moe-cache : pick CPU misses by CRF instead of random (#31853)
* moe-cache : remove the backend logic of the CPU part from the graph
The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.
ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.
* llama-graph : return the MoE cache slots and CPU ids in a struct
The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.
---------
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
The kernel_group_norm_f32 kernel and its host dispatch ignored the
batch dimension (ne03) of the source tensor: only ngrp threadgroups
were dispatched and each normalized the flat range
[tgpig*gs, (tgpig+1)*gs) with gs = ne00*ne01*ceil(ne02/ngrp). For
tensors with ne03 > 1, all but the first ne00*ne01*ne02 elements were
left unwritten (exposed by the new GROUP_NORM test with
ne=[3,1,1,65537], num_groups=1).
Dispatch one threadgroup per (group, i03) pair (ngrp in x, ne03 in z,
same work distribution as the Vulkan backend) and compute per-batch
statistics, matching the CPU reference. Also clamp the group element
count for the last group when ne02 is not divisible by ngrp and use
int64_t indexing.
The kernel now only requires ggml_is_contiguous_rows(op->src[0])
instead of full contiguity: it addresses src0 and dst with their own
strides (nb03 and the dst strides are added to the kernel args - dst
can be a strided view of src0 with ggml_group_norm_inplace) and falls
back from the flat fast path to a strided path that decodes the linear
element index when rows or channels are padded. The flat/strided
selection is done on the host in
ggml_metal_library_get_pipeline_group_norm and compiled into one of
two specialized pipeline variants via a FC_group_norm_flat function
constant, so the dead path is eliminated at pipeline creation.
GROUP_NORM support is also gated on F32 src and dst in supports_op -
the kernel only implements f32 and an f16 input would silently be read
as f32.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* server: visit parent slots before child slots
A parent slot wakes its waiting child slots once it finishes the shared
prompt prefix, while slots are visited in id order. When a child task
lands on a lower slot id than its parent, the child is visited before
the parent wakes it and waits an extra batch to process its suffix.
Non-child slots are now visited before active child slots, so every
child woken by its parent joins the same batch.
* server: check systemone test events through test tags
The systemone tests matched free-form debug messages, including the
formatted f_keep value of the prompt cache. The server now emits
__TEST_TAG_WAIT_PARENT_SLOT__ when a child slot waits for its parent,
__TEST_TAG_PROMPT_CACHE_RESTORE__ on every prompt cache restore and
__TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when the whole cached prompt is
a prefix of the new one, compared on token counts.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* Only run webgpu CI on my fork
* Add webgpu only workflow
* Add WebGPU group_norm kernel
* clean up
* resolve merge conflicts
* parameterize data type; clean up cmake changes
* add nullptr check
* clean up webkit experiments
* more cleanup
* more cleanup
* clang format
* Remove unused feature check for WebGPU
* clean up supports_op condition and update documentation
* add perf tests for group_norm
* clean up doc with script and undo change in test-backend-ops
* Do 2d dispatch of workgroups
* Do 2d dispatch of workgroups
* Unstage unrelated changes
* undo docs churn
* add new test case
* undo small change
* undo small change
* ggml-backend-meta : minimum number of compute views per buffer
Assisted-by: Claude Opus 5.5
* tests : revert the deepseek41 test model to 6 layers
Assisted-by: Claude Opus 5.5
* hexagon: fuse post-norm residual add into RMS_NORM_MUL
* hex-fusion: refactor RMS_NORM fusion and add dedicated RMS_NORM_MUL_ADD opcode
Improves performance by keeping things in HVX registers.
Also simplifies overall fusion logic and logging.
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* model: add Kimi-K3.1-Pro text model
K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.
Indexer:
- the selection is an additive mask over the whole cache, not a gather of
the top-k rows, so it reuses the existing sparse flash attention path;
off that path the mask stays correct but still costs a kq over the cache
- the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
because dsa_config declares no eps of its own
Graph and cache:
- new arch with four indexer tensors per DSA layer, and a hybrid memory
that keeps a K-only MLA cache for those layers, recurrent state for the
KDA layers, and the indexer key cache
- the graph node budget shares the K3 entry
Conversion:
- KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
- dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
plus the indexer LayerNorm eps; the layer list is 1-indexed like
linear_attn_config's, and the converter asserts that it covers exactly
the full-attention layers
- the routed experts stay MXFP4 and are repacked as in K3, not dequantized
- the vision tower and mm_projector are skipped: text only for now
- the arch key follows the checkpoint's own model_type ("kimi_k31") and the
loader registers KimiK31ForConditionalGeneration; the release name carries
a -Pro suffix that nothing in the config or the tensor names uses
Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.
Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.
* converter: avoid clutter in the k3 base
* narrow comment to match model
* update metal fusion expectations
* restore
* sync with PR 31733 refactoring
* hexagon: Generalization of CONV_2D implementation for CONV_3D
- New op HTP_OP_CONV_3D is wired through htp-ops.h, main.c (shares op_conv_2d) and the op remap in htp-kparams.cpp.
- conv_3d runs on the existing HMX tiled (implicit GEMM) conv2d path. KD and KH are folded into one virtual kernel row (KD*KH), and output rows are counted as N*OD*OH.
- htp_kparams_solve_conv3d checks the conv_3d layout and calls the tiled solver htp_kparams_conv2d_tiled, which takes an is3d flag.
- The tiled kernel takes the sizes and strides that differ between conv_2d and conv_3d from a struct htp_conv_geom. Only the input stage unfolds depth.
- 256-entry DMA queue can no longer overflow
Assisted-by: Claude Code
* hexagon: modify checking to eliminate divide-by-zero hazards
* hexagon: conv2d tiled: compute the conv_3d unfold divisors on the host
The KH and OH fastdiv values used to unfold conv_3d kh into (kd, kh) and
oh into (od, oh) were built per op on the device, each with a 64-bit
divide. Pass them in the kernel params instead; they fit in the space the
window variant already reserves, so the struct size does not change.
Assisted-by: Claude Code
* hexagon : preserve fold scheduling in conv2d for small size
hexagon : support arbitrary positive conv2d strides
Support positive horizontal strides for WHCN and CWHN inputs. Preserve
load/shuffle paths for strides 1/2 and use gather for larger strides.
hexagon : improve conv2d tile size selection
Prefer the largest feasible output-channel block and derive spatial
chunk capacity from VTCM usage. Compare capacity and thread-aligned
candidates using per-thread work and chunk overhead.
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* hex-conv: fix compiler warning
---------
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* tests : qualify isnan and signbit with std::
The unqualified calls rely on the C math.h names also being present in the
global namespace, which is not the case for the NetBSD 11.99.9/x86_64 gcc
14.4.0 build reported in the issue. Prefix the calls with std:: to select
the standard C++ overloads declared by <cmath>.
ref: https://github.com/ggml-org/ggml/issues/1681
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : qualify C math calls with std::
Follow-up to the isnan / signbit fix: make the remaining math calls in
test-quantize-fns.cpp consistent with the std:: isfinite / max / numeric_limits
already used in the same file.
The float variants are overloads in C++, not distinct names, so the mapping is
fabsf -> std::fabs, copysignf -> std::copysign, nextafterf -> std::nextafter -
not std::fabsf & friends, which do not exist in libstdc++ and would break the
gcc build. NAN and INFINITY are macros and are left unqualified.
Only behavioural difference: array_rmse() fed a double accumulator to sqrtf(),
now std::sqrt() keeps the computation in double. The result is more accurate
and differs from the previous value by at most 1.4e-7 relative, three orders of
magnitude below the tightest tolerance in this test.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The self-hosted CPU runners are containers with 8 CPUs, but
std::thread::hardware_concurrency reports the cores of the host. The
table mode of test-llama-context sizes its pool from that count, so it
runs 4 models x 8 threads on an 8-CPU job and hangs.
Add TEST_ARG_HARDWARE_CONCURRENCY and TEST_ARG_PARALLEL_WORKERS as the
defaults for the auto-selected counts of test-llama-context and of
test-backend-ops, and set them to 8 and 4 for cpu-x64-high-perf.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* moe-cache : LRFU/MQ-inspired hybrid cache
* make the clock advance by N tokens: use the real token count (ne[1] is always 1, ids have n_expert_used entries per token)
* sel is a RESHAPE node, the orig input is in src[0], so sel->src[0]->ne[1] is the count
* c for the class index, last_use for last use time
* explain the roles of these
* compute the key once
* use the key directly instead of re-reading
* show how the score is computed from the age
* remove the branch and clarify exp2 and log2 relationship
* reword the header comment and elaborate on unused slots and sorting
* not a copy, not just skip
* std::array maybe
* Revert "std::array maybe"
This reverts commit 7649b1f172db4941a854ad115bec0c7e062b65b9.
* half life it is
* add key and iteration helpers
* extract decayed_crf
* bump classes to 8, document the formula, display the cache size in token worth count
* assert the computed number of classes matches the half life
* opencl: use the multi-column q6_K vocab head GEMV for 2..4 tokens on X2E
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q6K_MC3 to GGML_OPENCL_Q6_K_MC3
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q40/Q41/Q4K_MC3 to GGML_OPENCL_Q4_0/Q4_1/Q4_K_MC3
GGML_OPENCL_Q4K_MC3_DQ and GGML_OPENCL_Q4K_MC3_LDS become GGML_OPENCL_Q4_K_MC3_DQ
and GGML_OPENCL_Q4_K_MC3_LDS.
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: remove unnecessary debug probe
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: clean up
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hex-repack: move repacking/tiling into graph-optimize
* hex-tiling: simplify tiling rules and remove obsolete helpers
* hex-ops: use a hash for de-duplicating tensor descriptors
* hex-bufs: improve tensor hash
* hex-ops: accept empty (zero-elements) ops
* hex-bufs: enable hostbuf by default and reject tensors that do not fit into max_vmem
* hex-bufs: improve error message and update readme
* hex-bufs: make sure to sync flush opbatch in all sync set_tensor paths
* feat : added ricursiveai ripple support
* fix : mandatiry SWA, scale through helper, comments back
* fix : metal
* model : make ricursive RoPE/NoPE explicit and validate the layer pattern
Write attention.rope_pattern and read it as optional, falling back to the
sliding window pattern so existing GGUFs keep loading. Refuse a
sliding_window_pattern that does not cover every layer, a model with no
sliding layer, and a gate or value residual present in only some layers.
The converter derives layer_types when the config omits it, ignores a stale
hybrid_layer_pattern and drops tensors of features the config disables.
Assisted-by: Claude Opus 5.5
* model : make ricursive swiglu_clamp per-layer
Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the
scalar in existing GGUFs still applies to every layer, and refuse a negative
or non-finite value. Add a test-llama-archs fixture with mixed per-layer
clamps and no rope_pattern, so the saver roundtrip checks both.
Assisted-by: Claude Opus 5.5
* model : use the ricursive per-tensor scales and RoPE parameters
Pass the attn_gate and output weight scales to build_lora_mm, take
ext_factor and attn_factor from the graph instead of hardcoding them, and
copy rope_freq_scale_train to the SWA layers instead of pinning it to 1.
Cite the reference lines for the value residual and the NoPE layers.
Assisted-by: Claude Opus 5.5
* convert : drop the ricursive chat_template.jinja override
The precedence between tokenizer_config.json and chat_template.jinja is to
be changed in SpecialVocab for all models instead. Until then a new
conversion stores the short tokenizer_config.json template; GGUFs that
already carry the tool-aware template are not affected.
Assisted-by: Claude Opus 5.5
* fix : remove NVFP4 checkpoint quantization block
* match #31733 refactor
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* RDNA3/4 Vulkan Large-Tile Split-K Optimization
* Add single-operator performance test cases
* Update the policy to use driver-reported LDS usage
* Add comments
* Use the simpler split_k rule from review
Assisted-by: GitHub Copilot
* tests : drop the split_k test cases
* vulkan : simplify the RDNA3/4 split_k condition
* metal : support scale-bias routing norm in the TOPK_MOE fusion
The DeepSeek-V4.1 graph normalizes the routing weights with a scale+bias
(sum + 1e-20) instead of the lower clamp used by the other models. Add
norm_bias/norm_scale_bias fusion chains that accept a SCALE node with a
unit scale factor at the clamp position, pass an additive bias in the
kernel args, and apply it as max(wt_sum, clamp) + bias so both variants
share one kernel. Cover the new chains in test-backend-ops.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sqrt(softplus) MoE routing with a selection bias
Add TOPK_MOE chains anchored at the softplus node (UNARY+SQRT+ADD+
ARGSORT+GET_ROWS plus optional norm/scale) so the DeepSeek-V4/V4.1
routing fuses on Metal. The kernel scores with sqrt(softplus) and ranks
selection-biased keys while the gathered weights and the norm stay
unbiased, matching the reference graph. Dispatch hooks at the softplus
node, cover the norm-bias combination in test-backend-ops and record the
new deepseek4/deepseek41 fusion counters.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal : fuse sigmoid MoE routing with a selection bias
The sigmoid-gated MoE archs (GLM, DeepSeek-V2/V3.2, ...) rank
sigmoid(logits) + exp_probs_b with an optional weights-norm, but only
the softmax and sqrt(softplus) routing chains were fusable.
- add 6 sigmoid-anchored TOPK_MOE chains: sigmoid + ADD + ARGSORT +
GET_ROWS with optional SCALE and SUM_ROWS + CLAMP/SCALE + DIV norm
tail; the selection-bias ADD is required, same as the sqrt(softplus)
chains
- generalize the gating function constant from a sqrt(softplus) bool
to an int (0 = softmax, 1 = sqrt(softplus), 2 = sigmoid); sigmoid
scores are 1 / (1 + exp(-logits)), matching the CPU, Metal unary and
CUDA kernels
- anchor the unary dispatch hook at sigmoid nodes too
- test-backend-ops: 4 sigmoid routing cases with and without a
weights scale
- re-record tests/fusion/MTL.csv: the sigmoid archs (glm5-next,
deepseek2/32, kimi, minimax, ...) now fuse their routing, and
muse-spark prefill drops one MOE_REDUCE fire because the reordered
graph allocates the routing weights inside the reduce output range,
which the dst/input overlap check correctly declines
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : remove low-signal comments
* ocd : valign ggml_metal_fusions
* CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a)
* Addressing reviewers comments, mma.cuh and physical warp size
* CUDA/HIP: single mma.cuh matrix-core lightning indexer ::CDNA::
* Support RDNA3
* CUDA/HIP: stream heads through the mma lightning indexer kernel
Process 16 heads at a time with double-buffered Q staging instead of keeping
all heads resident in shared memory, and drop the GGML_INDEXER_NO_MFMA env.
* enable for CUDA
* CUDA: Remove wmma lightning-indexer, also run head 32 through mma kernel
* CUDA: Nv mma dosent transpose accumulator
---------
Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* vulkan: add F16 input to the FWHT
The Vulkan FWHT accepts F32 input only. This adds an F16 source variant of the
shader, for both the subgroup and the shared-memory paths, so the kernel reads
an F16 source directly instead of requiring a converted copy.
The F16 shader needs shader-float16, so the pipelines are only created when the
device reports it. A null pipeline makes ggml_vk_can_use_fwht fall back, which
is the same path an unsupported width already takes.
supports_op and the dispatch both already route through ggml_vk_can_use_fwht,
so there is a single place asking whether an op qualifies; nothing to
consolidate here the way the CUDA/Metal ports needed.
test-backend-ops on an M5 Pro (MoltenVK, fp16: 1): MUL_MAT 1128/1128,
including all 24 Hadamard cases (18 existing F32, 6 new F16).
Assisted-by: Codex
* vulkan: FWHT kernels for block widths above 512
The Vulkan FWHT covers block widths 64 to 512; wider blocks fall back to the
dense matmul. Extend the width list to 8192 and pick the shader per width: the
subgroup shader keeps n/subgroup_size values per invocation, so widths above
512 (or a subgroup narrow enough to exceed 64 values per invocation) use the
shared-memory shader with one row per workgroup and a 256-wide block. The rows
per workgroup become a specialization constant so the dispatch matches.
Pipelines for a width are only created when the device's shared-memory and
workgroup-invocation limits allow it; otherwise that width keeps the fallback.
This is the Vulkan counterpart to #29100 (CUDA). Stacked on #29101; the last
commit is the one to review.
Assisted-by: Codex
* checkpoints: reduce overhead
Assisted-by: Codex
* rebase
Adapt the slot save/restore checkpoint appendix from #26004 to the
shared_ptr checkpoint list and the common_state_data buffers.
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: cwriter <cwriter@localhost>
* ggml : support `truncate=false` for YaRN
YaRN calculates correction intervals to switch frequency interpolation
methods, but the setting to not round the boundaries of these intervals
to integers (`truncate=false`) was not supported.
Models that use `truncate=false` include gpt-oss and some PLaMo-3
models; to support them correctly, the GGML operations and CPU backend
are modified to handle calculations without rounding. The default
behavior is `truncate=true`, and existing calls not explicitly
configured will maintain their previous behavior.
* support CUDA backend
* add tests
* move RoPE helpers to internal header
* format
* conversion : resolve quantized_layers from source tensor names
This commit changes how the per layer NVFP4 quant_algo are resolved from
hf_quant_config.json quantized_layers.
The keys in quantized_layers are module paths in the source checkpoint,
and these can be either be modules like 'lm_head' or parent modules
like 'layers.0.ffn.experts'. Currently these keys are passed through
filter_tensors, but they are not tensor names. So a filter_tensors that
keeps an explicit list of tensor names, like Nemotron's with --mtp
would drop them. That means there was no `_nvfp4_layer_algo entry`, and
`_tag_prec_a4` fell back to `_nvfp4_global_algo`.
This commit records the mapping from source tensor name to filtered
tensor name in index_tensors. Then in prepare_tensors each source
tensor is resolved to its closest quantized_layers parent, and
the algo is stored under the filtered tensor name, which is what
_tag_prec_a4 later looks up.
Converting Nemotron 3.5 Lightning NVFP4 with --mtp now produces:
```console
general.tensor_extra.name = ['output.weight']
general.tensor_extra.prec_a4 = [False]
```
whereas on master no general.tensor_extra fields are written. For the
non-MTP conversion, the general.tensor_extra fields are identical to
master.
Also manually verified (using pdb) that this works for
DeepSeek-V4.1-Flash-1017-NVFP4.
Refs: https://github.com/ggml-org/llama.cpp/pull/31602#issuecomment-6603337403
* conversion : shorten comment
* conversion : also remove '.bias' suffixes
Prevent malformed realtime session updates from throwing an uncaught
exception and terminating the server
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* fit : account for skipped NextN layers when setting n_gpu_layers
Without MTP, fit counts n_layer + 1 layers while the model loader counts
n_layer_all + 1 for n_gpu_layers, including the NextN layers which hold no
weights. A fit result that offloads all layers then leaves layer 0 on the CPU
("offloaded 65/66 layers" for Qwen3.8-27B on two GPUs).
Add the skipped NextN layers to n_gpu_layers and to the tensor split of the
device that holds the output layer so the loader assigns the layers that
fit measured.
Fixes#31531
* fit : pass hp_ngl_skip by reference like the other hparams
* fit : add hp_ngl_skip to common_get_device_memory_data
* graph: store the conv state of every rollback arch in one set_rows
The rollback slots of the causal conv history were written with one
copy per slot in qwen4exp, glm5-next, kimi-k3, bailingmoe3, lfm and
mamba2, while delta_net already wrote them with a single set_rows.
build_conv_state_store now does this for all of them: one copy without
rollback, otherwise one set_rows of the overlapping windows into the
min(n, n_rs_seq + 1) slots a rollback can read.
The slot order {n_rs_seq, ..., 0} is computed once per graph and each
conv layer takes a view of its tail. Its size does not depend on the
ubatch, so the reserved graph covers every ubatch shape, and the graph
input that held it is gone.
A model that is not causal never rolls back, so the context clamps
n_rs_seq to 0 for it, like for an arch without rollback support.
* nits: compute the conv slot order in the graph and keep the n_head name of bailingmoe3
* graph: build the conv slot order in each conv layer
The slot order {K-1, ..., 0} of the conv state store was one tensor
shared by every conv layer of the graph, built on the device of the
first one, so each layer on another device copied the view of its
tail. Each conv layer now builds its own order next to its set_rows,
and llm_graph_input_rs no longer holds it.
* tests: fix CI flakes of TOPK_MOE caused by near ties
With uniform random logits, two experts sometimes land within one f32
ulp of each other at a routing boundary, mostly with sqrt_softplus
gating whose output range is narrow. A backend whose gating differs by
a few ulps from the CPU then picks the other expert and the test fails.
Draw each row of logits again until the group selection and the final
top-k both have a relative margin of 1e-4, computed on the host in
double precision.
* tests: build the TOPK_MOE logits by inverting the gating function
Draw the selection scores (probs plus bias) of each row as a random
permutation of a grid, move the experts of the best groups up by half a
step, and set the logits to the inverse of the gating function at these
scores. Every expert and group decision is then one grid step apart on
all backends, and the host no longer recomputes the routing.
The softmax probs are built to sum to one so that the softmax keeps the
scores, and the views of the logits are not initialized again after the
logits are set.
The first command buffer takes the first 10% of the graph nodes and the
rest is split evenly between the extra command buffers. A fusion cannot
span two command buffers, so a boundary that falls inside a fusable
group runs that group unfused, and the output then depends on the node
count of the graph.
Move each boundary to the end of the fusable group it falls into, so
the encoders fuse the same nodes as with a single command buffer.
This commit adds nvfp4 scales to qwen2's build_ffn call which are
currently passed as NULL.
The motivation for this is that a model converted to NVFP4 will
currently crash because the scales are loaded but never applied in
the FFN:
```console
llama-completion: llama.cpp/src/llama-sampler.cpp:1218:
void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*):
Assertion `found' failed.
```
The attention already applies its scales but ffn is missing them.
* opencl: use the small-N f32 GEMV by default on X2E
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: use the q5_K multi-column verify GEMV by default on X2E
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* cuda: fuse NVFP4 gate/up and GLU in MMQ
* cuda: honor the requested src1 precision in fused gate/up MMQ
* cuda: share NVFP4 scale checks, skip unused fused FP4 kernels
* cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming
* cuda: check gate_scale before loading the fused NVFP4 gate scale
* llama : fix on-device state copy sizes for quantized KV caches
The on-device sequence state copies (LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) computed
their element count as size / ggml_element_size(). For a quantized type such as
q8_0 or q4_0 that is the number of blocks, not elements, so only part of each
range was copied: the restored state was wrong, or the copy crashed.
Compute the count as size / ggml_type_size() * ggml_blck_size() for the save,
the restore and the copy between differently chunked ranges, and assert that
each range covers whole blocks.
* llama : drop the one-line comments
* llama : drop the redundant cast in the state copy size
* server : realtime: send an error event instead of crashing on exceptions
* server : route websocket handlers through ex_wrapper, remove the realtime test
* server : realtime: keep the previous tools when the chat template rejects new ones
* server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapper
* server : realtime: clarify the tool schema check comment
* opencl: use more workgroups for q4_K gate up + GLU
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: handle FA for GQA 2 and 6 with head size 256
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: handle FA for GQA 16 with head size of 128, 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
* opencl: handle FA for GQA 2 with head size 512
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* ui : share directory resolution and the cwd row
Move the lookup of a browser-picked folder to ToolsService.resolveDirectoryPath, let the working directory picker run without its chip, and split the 'Set working directory to' row into its own component, so other views can pick and show a folder the same way.
Assisted by: Claude (Opus 5.5 Medium)
* ui : open an item's actions on right-click
Sidebar conversations and models manager rows open the same actions as
their "more actions" menu on right-click. Both menus render one shared
list of actions, so they cannot drift apart. Phones keep the browser's
long-press; the dropdown's drawer already holds the actions there.
An action can render a flyout submenu in place of its row, through the
kit of the menu it is in, and close the menu after a pick.
Adds the shadcn-svelte context-menu component, mirroring the
dropdown-menu one.
Assisted by: Claude (Opus 5.5 Medium)
* ui : close conversation tabs from a right-click menu
A tab's right-click menu closes it, the other tabs or the tabs to its
right, and stops a response the tab is generating. Closing tabs that
include the active one keeps the tab the menu was opened on.
Assisted by: Claude (Opus 5.5 Medium)
* ui : open a message's actions on right-click
User and assistant messages open the same actions as their action icons
on right-click. Selected text, links, media, code and fields keep the
browser's own menu, so copying them works as before. The fork dialog
opens from the actions context, like the delete one, so both the icon
and the menu can open it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : scope right-click menu shortcuts and selection to their item
Show the conversation shortcuts only on the active row, as they act on
the active conversation, and keep the browser menu only for a selection
inside the message. Name the close-tab shortcut, and drop the unused
chip toggle, menu submenus and shared cwd row.
Assisted-by: Claude (Opus 5.5 Medium)
* fix(ui): show extracted text in PDF attachment preview
Assisted-by: Claude (Opus 5.5 Medium)
* feat(ui): show page thumbnails for PDF attachments sent as images
Assisted-by: Claude (Opus 5.5 Medium)
* ui: stop retrying PDF preview conversion after it fails
A failed PDF-to-image conversion or text extraction reset its loading
flag, which re-ran the effect and retried forever. Remember the failed
item for each view and only retry when the item changes, so the error
state stays on screen.
Add a PdfConversionError story that feeds a broken PDF.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the text of PDFs sent as images and tidy the PDF preview
Store the extracted text of a PDF sent as images in its content, so the
Text view reads it instead of parsing the PDF again; only messages stored
with the old placeholder extract it. The model still receives the images
only. Guard retries on the error state, as the item remounts per id, skip
an empty base64 payload, and share the status markup between views.
Assisted-by: Claude (Opus 5.5 Medium)
* gguf-py : validate every tokenizer merge pair, not only the first
The transformers 4.45 pair format was detected by checking merges[0],
then every element was converted unchecked: a non-pair element raised
a bare TypeError, and pairs with the wrong arity or non-string parts
were silently mangled into invalid merges.
Validate each element (2-string list) up front and raise the existing
"Unknown tokenizer merges format" error for anything else.
Assisted-by: Devin
* gguf-py : remove test_vocab.py per review
The RISC-V vector intrinsic specification requires __riscv_v_intrinsic
to be defined when the compiler supports the API, even when the vector
extension is not selected. Clang therefore defines it for rv64gc,
causing ggml to enter RVV branches where vector types are rejected
because the required ISA extensions are absent.
Use __riscv_v for optional RVV implementations; this macro represents
the selected target ISA. Retain __riscv_v_intrinsic checks in SpaceMIT
sources, where they validate mandatory compiler API support rather than
select optional vector code.
Closes: https://github.com/ggml-org/ggml/issues/1535
* ggml: bound views by the bytes their strides reach
A view was checked against its source with the size of a contiguous
tensor of the same shape, computed before its strides were set. Views
whose rows overlap failed that check although they stay inside their
source, and views with wider strides passed it while reaching past
the end. The view functions now pass their strides to the tensor
constructor, which checks the bytes they actually reach, like
ggml_nbytes. The conv windows of delta_net are created with all their
rows at once instead of being widened after the check.
* ggml: assert that a strided view starts from contiguous elements
The view functions set nb[0] to the type size, so the source must have
contiguous elements along dim 0 for the view to read them.
The gguf writer staged each tensor into a temporary vector before
writing it. For host-buffer tensors the data pointer is directly
writable, so skip the staging copy and write it straight to the file.
This avoids the per-tensor allocation, zero-init and memcpy.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* common : fix -hf tag matching inside longer quant names
":F16" picked a BF16 file when the repo had both, since "BF16"
contains "F16". The same applied to ":Q4_XS" vs "IQ4_XS". The tag
now has to start at a token boundary in the path.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* common : accept underscore as -hf tag boundary
Some repos separate the quant with an underscore, e.g.
stepfun-ai/Step-3.5-Flash-GGUF-Q4_K_S ships
step3p5_flash_Q4_K_S-00001-of-00012.gguf, which :Q4_K_S no longer
matched. Add _ to the token boundary and pin the tag and default
lookup cases.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* hexagon: initial support for LIGHTNING_INDEXER op
* hex-lght-idxr: hmx for prefill
* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute
* hex-lght-idxr: pipelined hmx path for decode and prefill
* hex-lid: add support for K (f32)
* hex-lid: add support for K (f32)
* hex-lid: add support for q4_0
* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills
* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel
* hex-lit: update dma pipeline to use canonical dbl buffering in all paths
* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths
* hex-lit: add support for dims collapse
* hex-lit: make kparams constistent with other ops
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* Optimizing Gemma4 graph for prompt processing
TODO:
- Currently complete microbatch needs to contain tokens which does not require the logits; need to optimize this further
- There are a few more operations that are performed after the kv cache is calculated, need to safely optimize them.
* Not filtering the rows when they are unmasked
* Cropping out the attention mask too.
* gemma-4: improved pruning, mask cropping and PLE scaling
Prune before the first KV-shared layer with the single-stream, causal, non-embedding safety gates.
Use cropped masks only during shared-layer attention.
Preserve requested layer-input embeddings in the KV prefix.
Remove the PLE contiguous materialization and fold equivalent PLE scaling into RMSNorm parameters/weights.
* spec-mtp: Enable masked h_nextn extraction only when the draft context shares the target KV cache
This lets Gemma 4 prune unused prompt rows before its shared layers so that we keep the full perf P.P gains with MTP.
Avoid extracting target hidden states for output-free prompt chunks in that shared-MTP path.
When a batch has a selected final token, extract only that row unless the whole verification batch is selected.
Keep ordinary MTP and classic speculative decoding behavior unchanged: they still use unmasked h_nextn because the draft
context must catch up on all target rows.
speculative-simple needs an update to select the final prompt row only for shared-KV MTP,
so the optimization works there without changing other speculative modes.
And it looks like the server already selects its final prompt row for first-token sampling.
* spec-mtp: make sure to not drop first draft round
* gemma4: enable pruning based on static cparams.kv_unified/n_seq_max=1
* gemma4-mtp: simplify prunning logic and avoid graph branches/re-allocs
* gemma4: update pruning gates to use only n_seq_max==1, kv_unified can now be used for multi-seq
* spec-simple: revert spec-simple changes for now
* gemma4: remove causal_attn from pruning gate (last source of reallocs)
* cont : adapt to llm_attn_mask changes
* cont : add TODO
---------
Co-authored-by: Meet Amin <meetamin378@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* hexagon: faster TOP_K for the backend sampler (skip tiles, then block scan)
Skip the bitonic sort for tiles whose max cannot reach the running k-th
best value, then replace the tile sort with top_k_scan(): reject 256-float
blocks with one HVX max-reduce and insert only surviving elements into a
sorted k-entry list. Used for k in 2..64 and rows wider than 1024, in both
the single-row and multi-row paths.
Ties keep the first-seen index. NaN is treated as -INF.
* hex-top-k: eliminate scalar reads from vtcm
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Playwright's bundled Chromium lags behind Chrome security releases
(1.56.1 ships 141, and even 1.64.1 ships 156.0.8078.4, before the
156.0.8078.73 fixes), and Playwright launches it with --no-sandbox by
default. Outside CI, use the installed, auto-updated Chrome with the
sandbox enabled. CI keeps the bundled Chromium, so the self-hosted
runner's preinstalled 1.56.1 browsers still work.
Assisted-by: Claude (Opus 5.5 Medium)
* hexagon: add support for DSV4_HC_PRE
* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization
* hex-dsv4-hc-pre: double-buffered DMA for prefill
* hex-dsv4-hc-pre: refactor DMA block setup
* hex-dsv4-hc-pre: precompute kernel params
* hex-dsv4-hc-pre: support for n_hc != 4
* hexagon: add support for DSV4_HC_POST
* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block
* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks
* hex-run: add support for test-llama-archs tool
* hex-dsv4: refactoring and optimizations
Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.
Tested with test-llama-archs -a qwen4ext and test-backend-ops
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* speculative : save and restore MTP hidden state
Assisted-by: Codex
* tests : remove MTP state test
Assisted-by: Codex
* speculative : simplify MTP checkpoint state
Save only the boundary hidden row; verification rows are rebuilt on the supported restore paths.
Assisted-by: GPT-6
---------
Co-authored-by: zouyongxing <zouyongxing@bytedance.com>
* mtmd : add Nemotron 3.5 Transcribe support
Assisted-by: Claude
* convert : fix vocab of draft models for Nemotron-H targets
Assisted-by: Claude
* convert : reject non-BF16 Transcribe checkpoints
Assisted-by: Claude
* mtmd : make speaker FIFO predictions local
Assisted-by: Claude
* mtmd : normalize the mel in encode_windows
Assisted-by: Claude
* convert : map the generated speaker kernel
Assisted-by: Claude
* mtmd : validate Nemotron Transcribe mmproj shapes
Assisted-by: Claude
build_conv_state_at makes a contiguous copy of the conv input tail
before writing it into the recurrent state, so every GDN layer issues
two device copies per generated token where one is enough. On
Qwen3.8-Flash-Next (36 GDN layers) that is 72 copies per token, now 36.
Speculative decoding adds one more per rollback slot. ggml_cpy reads
the strided view directly, as the shared build_conv_state already does,
so the extra copy is dropped.
* ui: add Playwright harness for recording demo videos
Scripted walkthroughs under tools/ui/demo are recorded with Playwright and
exported as H.264 mp4 for pull requests (demo/pr) and docs (demo/docs).
Fixture MCP servers and seeded settings keep recordings deterministic, a
drawn cursor and captions make the flow readable, and DEMO_UI_DIR points
the recording at another checkout such as a PR worktree.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: record demo scripts from PR descriptions and a configurable storage
Demo scripts no longer live in the repo. `npm run demo:pr -- <number>`
records the `ts demo` block from a PR description against the PR head,
uploads the mp4, a GIF preview and the script to DEMO_STORAGE, and links
the preview in a `## Demo` section of the PR description.
`npm run demo:docs` records the docs scripts kept in DEMO_STORAGE.
DEMO_STORAGE (hf:// buckets or file://) and the public URL are read
from tools/ui/.env.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: import generated conversations into demo recordings
Demos can now start with existing chats: `test.use({ traces })` imports
llama.app JSONL traces through Settings > Import/Export before the video
starts. Built-in traces are generated deterministically, other names are
downloaded from DEMO_TRACES_DATASET, and `demo:traces` pushes the built-in
ones there. Adds `demo.scroll` for wheel scrolling.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep demo captions clear of the chat form and isolate concurrent runs
Captions move to the top of the frame, since at the bottom they covered
the chat form a demo is often about. Each recording run gets its own
Playwright output folder, so a run starting elsewhere no longer clears
another run's video before it is saved.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: record demos at 2x and 60 fps with human-like timing
Recording goes through Chromium's screencast instead of Playwright's
video, which encoded VP8 at about 1 Mbit/s at 1x. Frames are captured at
the device pixel ratio (DEMO_SCALE, default 2) and encoded into a 60 fps
H.264 mp4, starting once the page has settled.
The cursor now glides faster along a slight arc with Fitts-like
durations, settles briefly before each click, and typing and pauses vary
a little. The randomness is seeded per demo, so re-recording gives the
same motion.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: document the demo harness prerequisites
The demo README now lists the tools the harness needs and what each is
for, and walks through creating the Hugging Face bucket and the optional
traces dataset, with the matching .env settings.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: version linked demo URLs so re-recorded previews refresh
GitHub's image proxy caches a preview by URL, so a demo re-recorded at
the same PR head kept showing the old GIF. The Demo section now links
the files with a content hash in the query string.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: label PRs that feature a video demo
Adds a `video demo` label and a workflow that applies it when a PR
description has a demo written by the UI demo pipeline (the managed
demo:start/demo:end markers, or a preview linking to a demo video),
and removes it when that section is gone.
Assisted-by: Claude (Opus 5.5 Medium)
* ci: label video demos with gh instead of github-script
The label workflow now runs the gh CLI in a shell step, like
pr-draft-label.yml, so it needs no action outside the repository's
allowed list. The video link check only counts videos uploaded for the
PR itself, so a description linking other PRs' demos isn't labeled.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: zoom demos in on details with a two-finger pinch
demo.zoom pinches in on a control or a point, holds, and pinches back out
to 1x, so a detail too small to read in the frame can be shown up close.
It drives Chromium's own pinch gesture, so the page's fixed bars stay put,
and works at desktop and phone sizes alike.
The pinch is evened out where Chromium misses: one anchored near the top
does not zoom at all, so it starts lower, and one near the bottom
overshoots its ratio, so a small pinch from the middle corrects it. The
way out reads the real scale until the page is back at 1x, which the next
gesture needs.
The overlay also draws a fingertip per touch, so a pinch on a touch
device shows as two fingers.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: wait out a cold dev server before a demo starts
A fresh checkout's dev server optimizes its dependencies on the first
load and reloads the page, which can take well past an action's timeout,
so a demo failed before it started. The demo now waits up to 90 seconds
for the app; the recording only starts once it is ready.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: record a video per test in a PR demo, shown in a table
A large PR can split its demo into several shorter videos: each test()
in the ts demo block is recorded as its own video (<sha>-1.mp4, ...),
and the Demo section lays their previews out in a table, each column
headed by its test's title.
GIF previews now keep their longer side at DEMO_PREVIEW_WIDTH and shrink
until they are under 5 MB, since GitHub's image proxy refuses anything
larger and shows a broken image instead.
Importing traces works on a phone viewport too, where Settings sits in
the sidebar under the page until the menu key slides it aside.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fix demo trace imports on desktop and lay landscape demos out two to a row
Importing traces failed on a desktop whose collapsed sidebar sits under
the page: it now expands the sidebar from the page's menu key on a phone
or from the sidebar's logo on a desktop, whichever makes Settings
reachable. Afterwards it forgets the expanded sidebar, so a demo starts
with it collapsed, as on a first visit.
A PR demo with several videos shows up to four portrait (phone) videos
to a row, or two landscape ones, so each preview stays readable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: only record other people's PR demos at a reviewed head commit
demo:pr runs the PR's code and demo script on this machine, with the
gh and hf logins, so a PR by someone else now needs --trust <sha>
naming the head commit that was read, and is refused again after new
pushes. Recordings also launch Chromium with its sandbox on, which
Playwright turns off by default.
Assisted-by: Claude (Opus 5.5 Medium)
ggml_roll validated each shift with abs(shift), but abs(INT_MIN) is signed
integer overflow UB evaluated before the bounds check runs. Compare with a
signed-safe test instead.
This commit updates the _flush_nvfp4_experts to take a new parameter
which is a dictionary mapping the key to the original model tensor name.
The motivation for this is that I noticed that the tensor names being
looked up were not the original model tensor names, for example:
```console
model.layers.1.mlp.experts.up_proj.weight
```
But _nvfp4_layer_algo uses the original per-expert names:
```console
(Pdb) p self._nvfp4_layer_algo
{
'backbone.layers.1.mixer.experts.0.up_proj': 'W4A16_NVFP4',
...
```
This means that the lookup will never match for experts and fall back to
_nvfp4_global_algo. If the global algo is W4A16_NVFP4 then this would
go unnoticed, but for a model like
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 which is
MIXED_PRECISION, the expert tensors will not get prec_a4=false so they
would use the native W4A4 path on Blackwell even though the original
model tensors declare them as W4A16_NVFP4.
* cuda: gather batched tensor reads into mapped host memory
Assisted-by: Codex
* add comments
Assisted-by: Codex
* simplify the implementation based on checks that exist earlier in the flow
* simplify batched tensor gather indexing
* add conditional dispatch
* address comments - make host registration raii, better test mindful of ci time, assert instead of return false
* cuda : gather small reads and use pinned DMA for large reads
* cuda : use std::vector for host gather descriptors
* cuda : add 8-byte gather copies, generalize tensor test and log registration failures
* common buffer for descriptors
* simplify stream sync + increase to 128k descriptor pinned buf
* Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available.
* Drop ggml_backend_comm_allreduce_rank_t and reuse ggml_backend_comm_allreduce_tensor for per-rank communicators
* reuse ggml_backend_cuda_comm_allreduce_nccl for per-rank communicators. Remove the helper function backend_comm_proc_address
* Change ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t to more generic multi-process backend communicator interfaces
* set maxCTAs in ncclConfig_t to 8 for dgx spark in ggml_backend_cuda_comm_init_rank
* meta: support GLM-5.3 next sm tensor
* glm5-next : cont q_absorbed for -sm tensor with non-unified KV
* meta : keep GGML_OP_ARANGE in ggml.h order
* glm5-next : split MLA heads in pairs for -sm tensor
The MLA cache has a single KV head, so the heads on a device are its GQA
ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3
devices the 16 heads of the test model were split 5/5/6 and it aborted.
* meta : keep the split for views inside one slab of the split axis
With a non-unified KV cache, build_attn_mha splits q into streams with a
view. For a permuted q, e.g. the absorbed query of glm5-next, that view
aborted with "view of permuted tensor not implemented". The split is now
kept if every other dim of the source and the view stays inside one slab
of the split axis. The per-device strides are only rescaled above it.
From tetherto/qvac-fabric-llm.cpp#353.
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
* glm5-next : drop the q_absorbed cont
The meta backend can now split the per-stream view of the permuted q, so
the cont is not needed for -sm tensor with a non-unified KV cache. With
the cont, the q of flash attention was not contiguous, and when FA ran on
the CPU (WebGPU) the meta get_tensor aborted.
---------
Co-authored-by: Pratik Narola <pratik.narola@tether.io>
kernel_cpy_t_t indexed the dst innermost dim with an implicit element-size
stride instead of args.nb0, so cpy into a permuted view (nb0 != type size)
wrote the wrong addresses. caught by the new test-backend-ops cpy case with
permute_dst = {1, 0, 2, 3} from #31446. addressing is unchanged when dst is
contiguous in dim 0.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* vulkan: skip maxStorageBufferRange pre-check for non-compute tensors
Pre-allocated leaf tensors and views (GGML_OP_NONE / GGML_OP_VIEW) are never
executed by the backend, but were still rejected by the descriptor-size
pre-check in ggml_backend_vk_device_supports_op when larger than
maxStorageBufferRange on devices without shader_64b_indexing. The scheduler
then aborts with 'pre-allocated tensor (...) in a buffer (Vulkan0) that cannot
run the operation (NONE)' - e.g. the MoE expert cache banks (llama-moe-cache)
of a 256-expert model with --moe-cache-mib on GCN GPUs, where a whole bank
(2.4 GB) exceeds the limit while the consuming ops only access small
per-step views.
Skip the size pre-check for non-compute ops (NONE and VIEW); the buffer type
allocation limit still bounds the actual allocation, and compute ops keep
validating their sources.
Tested on AMD Radeon Pro VII (gfx906, no shader_64b_indexing):
- before: GGML_ABORT 'pre-allocated tensor (moe_cache.0.blk.0.ffn_down_exps.weight) ... cannot run the operation (NONE)' with llama-cli -ngl 99 --cpu-moe --moe-cache-mib 6144 (Cyber-Tiel-Coder-35B-A3B, qwen35moe, 256 experts per layer)
- after: loads and generates at 8.1 t/s (vs 0.6 t/s without the cache)
- regression: -ncmoe 16 path unchanged (19.2 t/s), qwen3moe/qwen35moe without cache unchanged
Fixes#30697
* vulkan: use ggml_op_is_empty for the non-compute size-check skip
---------
Co-authored-by: FamousMonsterr <FamousMonsterr@users.noreply.github.com>
* delta_net: access conv state via views
* fix sm tensor
* delta_net: fill the conv slot index on the host
The rollback slot index of the conv windows is a constant per graph
shape. It is now an I32 input of the recurrent memory filled on the
host, instead of an arange and an argsort computed on the device, and
the meta backend no longer needs a mirrored split for ARANGE.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup
On the non-OCP path, ue4m3_fp32_lut has 128 entries and is indexed with the raw scale byte. ggml_validate_row_data accepts every byte and ggml_ue4m3_to_fp32 ignores bit 7, so a byte >= 0x80 reads past the shared array. Vulkan robustness does not cover shared memory.
Size the table, both init loops, and the two host shared-memory budgets to 256 entries. ue4m3_to_fp32_build() already masks the exponent and treats only 0 and 0x7F as special, so entries 128-255 match the CPU. A 256-entry table keeps 0xFF distinct from 0x7F: the CPU decodes 0xFF as a non-zero value and 0x7F as zero.
Each NVFP4 pipeline uses 512 more bytes of shared memory. ggml_fp32_to_ue4m3 never sets bit 7, so models quantized by llama.cpp are unaffected.
Assisted-by: Claude
Assisted-by: Codex
* vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup
Assisted-by: Codex
---------
Co-authored-by: Dongsheng Luo <7137760+flyingdoog@users.noreply.github.com>
The dspark block mask is now derived at input-fill time: llm_attn_mask::block_uniform
fills every token of a sequence with the mask row of its first token, so the
graph-level reshape/repeat in dflash is gone and cnv always aliases raw.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cuda : prevent premature eviction of graphs about to be used
* add last_graph_use_time for entire cuda graph map and testcase for cuda graph
Assisted-by: Codex
* delete the test
A cache that shares the cells of another one (f.ex. the gemma4 assistant) can build a graph before its sequences have any cells, f.ex. at the server warmup. get_n_kv_rows() then returned 0, which means that kv_rows are not used, so the graph fell back to the dense KQ mask and the scheduler re-planned for it. Later graphs with larger masks then aborted with GGML_SCHED_DEBUG_REALLOC=1.
* hexagon: add tiled HVX transpose path for permuted CONT/CPY
Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.
Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.
Host:
- drop extent-1 dims and merge dims that stay adjacent in src into a
canonical view, stored in the kernel params. There are no minimum
size limits.
- the copy is treated as a 2D transpose between dst dim 0 and the
contiguous src dim, batched over the remaining dims.
- tiles are sized to 64 KB per buffer and halved until they fit in
session VTCM. blkk is capped so the halfword vgather offsets fit in
16 bits.
Kernel, per tile:
- 2D DMA of blk0 strided src rows (blkk contiguous elements each)
into VTCM
- vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
- 2D DMA of blkk contiguous rows out to dst
Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.
Small dims:
- packed mode: when dim k follows dim 0 and dim 0 divides the HVX
lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
whole dst rows, and each tile goes out as one contiguous DMA.
- when the contiguous src dim fits in one vector, it is staged exactly
and the tile is made long along dim 0.
All other copies select the same kernels as before.
Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.
* hexagon: cpy transpose cleanups
- fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
- set packed directly in the packed case
- add dim_k == 3 CONT tests (full reversal)
Assisted-by: Claude Opus 5.5
* hexagon: use cpy transpose path for ne[0] == 1 permutes
- drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
- try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
- add ne[0] == 1 permute CPY tests
Assisted-by: Claude Opus 5.5
* hex-cpy: further improve cpy
Bring in improvements from aizip branch/PR.
Cleanup kparams and replace divs with fastdiv.
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* hex-cpy: avoid vtcm bank conflicts
* hex-cpy: trim down the comments
* cpy-test: trim down new tests that cover permute scenarios
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* feat: support PocketTTS voices with no voice cloning
The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.
Assisted-by: Claude Fable 5.1
* convert : pocket-tts voices never fail the conversion, log the drift of the state
The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.
Assisted-by: Claude Fable 5.1
* mtmd : list the predefined voices, expose them in GET /props
mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.
Assisted-by: Claude Fable 5.1
* server : an unknown voice is a bad request
A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.
Assisted-by: Claude Fable 5.1
* refactor api
* fix conversion
* small renaming
* fix lint
* tts : a model with predefined voices speaks with the first one by default
Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.
Assisted-by: Claude Fable 5.1
* mtmd : one comment per voice accessor, converter states the rule only
The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.
Assisted-by: Claude Fable 5.1
* clean up redundant code and comments
* mtmd: match the LFM3-Audio voice name case-insensitively
The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* llama : use llm_attn_mask and kv_rows for unified attention
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cont : optimize `get_seq_cells`
* llama : build the kv_rows lists from per-sequence cell bitmaps
Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed.
Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA).
* cont : make get_kv_rows_p_min private
* cont : minor metal
* llama : use kv_rows for every ubatch of a unified cache
A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices.
Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.
* llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The note under 'Run the inference' still said that the CANN backend only
supports FP16/Q4_0/Q8_0 models, while the DataType table in the same file,
and ggml-cann.cpp (BF16 is enabled for MUL_MAT, GET_ROWS, CPY and CONT for
every non-310P device), say otherwise. Spell out the per-device support and
mark BF16 as unsupported on 310P instead of leaving the cell empty.
GGML_CANN_NO_PINNED is read by ggml-cann.cpp when allocating host buffers
and when advertising the host_buffer device capability, but it was not
documented, so the escape hatch for a failing pinned host allocation was
not discoverable.
Signed-off-by: li-lizhe <147392333@qq.com>
* vulkan: add tensor-parallel all-reduce communication
* vulkan: require shared semaphores for tensor communication
Assisted-by: Codex
* vulkan: reduce directly from shared host buffers
Assisted-by: Codex
* vulkan: fix ring all-reduce races with 3+ devices
Two hazards in the tensor-parallel ring all-reduce made -sm tensor output
non-deterministic with three GPUs:
- With n = 3, step t's host-slot region starts at t * cels * 2 bytes, which
is not a multiple of 32, so consecutive steps' transfer copies share a
32-byte sector of the imported host memory. On NVIDIA the later copy's
leading bytes (up to 10 F16 values at the start of every chunk) end up
stale in host memory, even when the copies are ordered by a semaphore.
Give every step its own 256-byte-aligned region. n = 2 and n = 4 already
had aligned regions for the usual sizes, so their layout is unchanged.
- A single-shot all-reduce leaves every device's host slot being read by all
peers, but the next ring round only ordered its first transfer after nextd,
and only from round 2 on. With n >= 3 a device could overwrite its slot
while prevd was still summing it. Wait for all peers on the first ring
round after a single-shot all-reduce.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vulkan: fold the tensor-parallel all-reduce into the compute submissions
With -sm tensor the meta backend drives every device from one thread and calls
graph_compute once per subgraph (two per layer), so decode was bound by the CPU
cost of submitting, not by the GPUs: ~1500 vkQueueSubmit per token for a 64-layer
model on two GPUs, with the GPUs idle ~40% of the time.
- Don't split subgraphs by flops while a communicator is active. The heuristic
was tuned for whole graphs and turned each small subgraph into ~5 submits.
- Fold the small (single-shot) all-reduce into the compute submissions: the
last command buffer of a subgraph stays open, the all-reduce appends the
gather copy to it and submits it, then opens the next command buffer with the
peer waits and the reduce, which the next subgraph is recorded into. Each
subgraph now costs one submit per device. The reserved progress value is
attached by whichever path submits the open context.
- Alternate between two host slots, placed outside the ring's regions, so the
gather no longer needs a WAR wait on the peers' previous reduce.
- On synchronize, submit the deferred work of every device in the group first,
so the tail of the graph runs on all devices in parallel.
- Wait for the transfer queue as well as the compute queue before freeing or
reallocating communication buffers, and don't leave a stale cap behind when a
reallocation fails.
qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 32.2 -> 49.7 t/s, 3 GPUs 20.9 ->
40.0, 4 GPUs 15.0 -> 27.6; pp512 on 2 GPUs 2138 -> 2320. Greedy output is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* - Two-phase all-reduce over shared host memory: write own slot + signal, wait peers + add from their slots
- One exported timeline per device, no backend hooks; comm work recorded in the backend compute ctx
- F16 wire above 1 MiB, F32 below; small reductions ride with the next graph submit
- Double-buffered, grow-only host slots; zero-size splits zeroed
- Remove ring, transfer-queue leg, device staging, shrink, Windows shims
- comm_init returns nullptr when unsupported; add GGML_VK_COMM_DISABLE
- No flops-based submit splitting for graphs below 1 GFLOP
[rebased onto the ring all-reduce fix and the folded all-reduce, which this design replaces; their backend hooks are removed here]
* add minimal submit split size
* replace opaque_fd with sync_fd for spec compliance
* add precision check
* add semaphore pool
* rename env var, add logging if requirements not met
* reduce comments
* code improvements
* vulkan: use posix_memalign for the tensor-parallel host buffers
std::aligned_alloc is unavailable on Android bionic below API 28, which
upstream switched away from in #30828; do the same for the communicator's
host slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vulkan: synchronize the tensor-parallel all-reduce through the host
Exchanging SYNC_FD semaphores costs a kernel wake-up on every all-reduce:
on NVIDIA the waiting GPU starts ~35 us after its peer has finished, which
is most of the idle time in a decode step. OPAQUE_FD semaphores avoid that
but must not be imported into another physical device.
Use only device-local timeline semaphores instead. Each device signals
done[i] once its partial sum is in host memory (after a barrier that makes
it available to the host), and a bridge thread signals go[i] from the host
once all peers of i are done. The waiting side makes the host memory
visible with a host-write barrier, since its submit predates the data.
The thread spins for up to 200 us on each wait (GGML_VK_ALLREDUCE_SPIN_US)
and then blocks, so it is idle when there is no work. This also takes the
per-round fd export, dup and import calls off the main thread and drops the
VK_KHR_external_semaphore_fd requirement and the binary semaphore pool.
Graphs below 1 GFLOP, such as tensor-parallel decode slices, are no longer
split into several submits.
qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 37.2 -> 45.4 t/s, 3 GPUs
24.1 -> 35.5, 4 GPUs 16.5 -> 27.1; pp512 unchanged. Greedy output is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* use sync_fd on mesa drivers
* refactor for readability
* move sync_fd semaphore handling to thread, trigger mesa submit threads
* move allreduce functions to new cpp file
* clean comments
* smaller fixes, code improvements
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* Update ggml-sycl.cpp
Increases -sm tensor prompt eval t/s by +19% with zero PPL change.
* Update ggml-sycl.cpp
Replace first pair of wait() calls with ext_oneapi_submit_barrier(). The wait() calls was discovered to cause MTP degradation.
* Update ggml-sycl.cpp
Adding comm_exchange function pointer to switch between 2 and 4 buffer ggml_sycl_comm_exchange versions @ initialization time (versus hot path) controlled by SYCL_ENABLE_DEFAULT_CONTEXTS=1 setting.
* Update ggml-sycl.cpp
adding back 2nd half of 4-buffer free.
* Update ggml-sycl.cpp
Removing extra whitespace.
* Update ggml-sycl.cpp
Excluding Windows until it is validated on Windows.
The router only removes models it took from the Hugging Face cache, but
the models manager offered "Delete from disk" for every model, so a model
from --models-preset or --models-dir failed with a "not removable" toast.
Read the source and can_remove fields of /models and show the delete
action disabled for those models, with a tooltip (a hint line in the
mobile drawer) saying how to remove them instead.
Assisted-by: Claude (Opus 5.5 Medium)
Use the popover background, softer border, rounded-xl corners and
shadow-md like dropdowns, hover cards and the settings dialog, and
match the dialog title line height.
Assisted-by: Claude (Opus 5.5 Medium)
A message queued while a conversation was busy was re-sent through
sendMessage, which targets the active conversation, so switching chats
before the turn ended delivered it into the wrong one. Re-send only when
the owning conversation is active; otherwise keep it queued there and
flush it when that conversation is opened again.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show interrupted state for tool calls that never returned
The chat-level streaming flag was applied to every assistant message, so a
tool call left without a result in an earlier message kept spinning while a
newer reply streamed. Scope streaming to the last assistant message and mark
finished calls without a result as interrupted.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: mark a shell command stopped before any output as interrupted
exec_shell_command creates its result message before running, so stopping
it mid-run leaves an empty result instead of none.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: don't reload a loading model on select, prefer it in pre-selection
Assisted-by: Claude (Opus 5.5 Medium)
* ui: skip hidden models in automatic model selection
Assisted-by: Claude (Opus 5.5 Medium)
* ui: don't count a model being unloaded as loading
Assisted-by: Claude (Opus 5.5 Medium)
* webgpu: keep scratch alloc sizes monotonic in the batch length
The flash attention vec path and the MMVQ path append a scratch area
after dst, but get_alloc_size only accounted for it when the current
batch takes that path. The size was therefore not monotonic in the
batch length: a graph reserved at n_ubatch (tiled path, no scratch)
or at one token did not cover batches of 2 to 19 query rows, and
GGML_SCHED_NO_REALLOC builds aborted on an unexpected reallocation.
Pass the row count to both path predicates and size the scratch with
the row count clamped to the longest batch the short path serves.
* nit: align the mmvq pipeline key with clang-format 22
* webgpu: address review @yomaytk, name the mat-vec row threshold
* hexagon : fix DMA transfers with strides > 24 bits
On v75+ the 2D DMA descriptor stores row_size and strides in 24 bits.
dma_queue_push sent oversized strides to fallback_2d, which only splits
nrows, so strides of 16 MB or more were truncated - wrong addresses.
Fix will fall back to one transfer per row in that case.
Also added two CONT tests with source strides over 16 MB.
* hex-dma: refactor the fallback paths to remove duplication
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Limit the shared prefix to the first required output position of the parent and each child task.
This keeps decision output embeddings available in the final batch when questions share option tokens.
Fixes#30482
Signed-off-by: aiqubits <aiqubits@hotmail.com>
* chat : accept the think block in the qwen3-coder response format with reasoning_format none
Assisted-by: Claude Opus 5.5
* chat : define both qwen3-coder think block variants up front
This commit extracts all apple jobs except for macos-latest-arm64 into a
new daily workflow.
The motivation for this is to reducd the load on the macos runners.
The duplicated output looks up token_embd in its buffer context, so
token_embd must exist first. With the old order a load where input and
output share a buffer type (CPU-only, -ngl 0) held the table twice:
gemma-4-E2B-it Q4_K_M, no mmap, CPU model buffer 3163.73 -> 2947.73 MiB.
* memory: refuse rollbacks past the snapshots of the last ubatch
The recurrent memory and the DSV4 cache accepted any rollback up to
n_rs_seq, whether or not the last ubatch wrote that snapshot. Each now
records the deepest rollback its last ubatch guarantees and seq_rm
refuses anything past it:
- recurrent: n - 1 for a ubatch of n <= n_rs_seq + 1 tokens, 0 for a
longer one, which only guarantees the final state
- DSV4: min(n, n_rs_seq), the snapshots are rebuilt from the state
before the ubatch
The dead rollback block of the rollback test becomes the assertion it
asked for, and rollback_bound checks the exact bound of each memory
after ubatches of 1, 2, n_rs_seq + 1 and n_rs_seq + 2 tokens.
* tests: address review from ggerganov
Detect architectures with llama_arch_from_string in comp_padding and
arch_reserves_single_seq, as in rollback_bound.
gfx117X is more like RDNA4, with 128b WMMA. But they're not identical. Adjust the defines accordingly to have a new RDNA4_M and use it where it makes sense.
Assisted-by: Codex
Assisted-by: GitHub Copilot
Co-authored-by: Mario Limonciello <mario.limonciello@amd.com>
* WIP OCP FP8 E4M3 support
* Fix python type-checker
* Materialize input_scales also for FP8 when converting
* Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4
1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on
dims alone. Also check on dtype for FP8
2. Need to reshape FP8 QKV projections scales in the same way that
weights are reshaped
* Allow remaining FP8 tensors to be dequantized during conversion
* Allow tensor-strategy for FP8-only compressed-tensor ckpts
* Fix and test gguf-py behavior for FP8_E4M3
* Disallow QKV-fusion for FP8 temporarily
* Fix conversion of fused qkvz & FP8 for Qwen3-Next
* Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws
* Emit warnings when not fusing FP8 during conversion
* More FP8 conversion fixes
* Add supported for strided copies to/from FP8 in CPU
* remove unneeded scratch buffer from FP8 cpu cpy/dup path
* add explanatory comments and TODOs
* Add dedicated encoder and decoder tests for FP8
* Whitelist ggml_op_get_rows in metal instead of blacklisting
required to fix failing CI, and white-listing should be more
future-proof than adding FP8 to the black-list
https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512
* Fix more conversion bugs
* Defer weight materialization during conversion
* Blacklist fp8 in metal backend instead of white-list
* Adjust indents
* Fix lint + execute tests
* Add fast CPU kernel for F8_E4M3
* Conversion fixes
* Fix remaining gaps
* Enable tied weights with shared scales
* Make mandatory-F32 rules apply to FP8 during conversion
* Fix python type checks
https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512
* Fix kaleidai unit-test failures
https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150
* Relax numeric precision constraints for pre-check
* Leave fast-FP8 CPU on RISC-V for follow-up
We already exclude arm from the fast-path
* Allow output scale tensors to be located in any gguf shard
* Move test_conversion_fp8.py test execution to behavior-check.yaml
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it
* Install all requirements also on Windows for behavior-check
Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512
* Unmap lazy tensors before clean-up in Windows
Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512
* LF line endings
https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512
* Extract FP8 tests from llama-arch to its own test-fixture
See https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4400628221
* Update gguf-py/gguf/scripts/gguf_convert_endian.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ui: land at the bottom of a conversation every time it opens
Opening a long conversation often stopped short of its end:
- the press grace counted from page load (pressedAt = 0), so a chat that
landed within 400 ms of a refresh did not follow the rows realizing
after the jump
- the click that opened a chat (sidebar row, tab) started the same grace,
so navigating by click had the same problem
- any scroll event away from the bottom stopped following, including the
ones the browser fires itself while the chat settles
- the chat screen outlives its routes, so going back to the last landed
conversation through the new-chat screen skipped the landing entirely
scrollToBottom() now clears the grace, only an upward scroll away from the
bottom stops following, and every navigation lands anew. Navigations into
a chat also disable SvelteKit's scroll reset, which moved the previous
conversation to its top for a few frames.
Where overflow-anchor is supported, a 1px pin after the messages is the
only scroll anchor while following, so the browser keeps the bottom in
place itself; WebKit keeps the resize observer.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let nested agentic content scrollers chain to the chat
Reasoning, shell output and code blocks used overscroll-behavior:
contain, so the wheel stopped dead at their edges instead of
scrolling the conversation.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: stop following the stream as soon as the reader scrolls up
A trackpad or smooth wheel scrolls in steps smaller than the at-bottom
threshold, so each step up was re-stuck and snapped back on the next
growth while a response streamed. Any upward move that is not content
shrinking now releases at once.
The attach effect also read the hook's own state, so every flip
re-attached and re-measured, which undid such a release.
Assisted-by: Claude (Opus 5.5 Medium)
ChatFormActions gated the mic and stop buttons on a `canSubmit` prop that
ChatForm never passed, so it was always false and typing text did not swap
them for the submit button. Use `canSend` instead and also hide the
skip-reasoning button while there is content to send.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: reveal the collapsed sidebar from its logo and toggle it there
The collapsed sidebar now shows only the logo; hovering it reveals the
icon strip until the pointer leaves. The expanded sidebar drops its
top-right collapse button and the logo collapses it on hover instead.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fade the peeked sidebar in at full width and keep the toggle slide
Hovering the logo of the collapsed sidebar now fades in the fully open
sidebar, floating over the page at its full width, instead of resizing
it. Opening or closing with the toggle still resizes and slides: the
sidebar docks to strip width while it slides, then floats out of sight,
and it stays hidden under the pointer after a toggle close. The unused
icon-only strip is removed from the sidebar actions.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: stack the sidebar above the chat tabs on desktop
The peeked sidebar floats over the page, but the chat tab bar shared its
z-index and came later in the page, so the tabs were drawn on top of it.
Assisted-by: Claude (Opus 5.5 Medium)
MCP server groups in the settings Tools tab now get a group checkbox like the
built-in, server and browser groups, toggling the server-scoped tools key.
Indeterminate checkboxes now use the checked state colors.
Assisted-by: Claude (Opus 5.5 Medium)
Ask the browser to keep IndexedDB and localStorage data under storage
pressure, so conversations and settings are less likely to be evicted.
Assisted-by: Claude (Opus 5.5 Medium)
* gguf-py : add Limite arch
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* convert : add Limite support
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* llama : add Limite support
Assisted-by: Claude Opus 5.5
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
---------
Signed-off-by: Giuseppe Scrivano <giuseppe@scrivano.org>
* metal : faster MXFP4 mat-vec kernel
Decode the fp4 values with the fp16 bit trick instead of a threadgroup lookup table, and load the packed nibbles as aligned 32-bit words. The kernel no longer uses threadgroup memory.
* metal : launch contiguous unary and binary ops with full threadgroups
The contiguous unary path and the row-broadcast binary path used one thread per threadgroup. Use up to 256 threads per threadgroup and bound-check the tail.
* metal : split GLU rows across threadgroups
A GLU row ran on one threadgroup, so single-row decode used a single threadgroup for the whole op. Split each row over threadgroups of up to 256 threads.
Requested in the #31125 review. The cases duplicate coverage that is
already in the file and add CI time. Test-only, nothing changes in the
kernel, the docs or the CMake option.
Assisted-by: Claude
* hex-upscale: basic upscale ops (not fully vectorized)
* hex-upscale: optimize nearest 2x, 4x and bilinear 4x
* hex-upscale: add support for non-contig rows and dims collapsing
* hex-upscale: use vgather instead of scalar staging
* hex-upscale: move all kernel select logic to the host, review comments and cleanups
* tests: add permuted UPSCALE test to cover qwen3-vl image->text scenarios
* sycl : add DPAS flash attention kernel for quantized KV cache
Decode and small batches (1 to 31 query tokens) now run flash attention
straight on q4_0 and q8_0 KV blocks, instead of scalar dequant or
converting the whole cache to f16 per call. Q.K uses s8 x s4 and s8 x s8
DPAS on XMX, V.P stays f16. dpas.s4.s8 is inline vISA since joint_matrix
has no s4.
Covers head sizes 64 to 512, GQA up to 32 and attention sinks. Prompts
stay on the existing kernels. GGML_SYCL_FA_DPAS=0 turns it off.
B580, Llama 3.1 8B Q4_K_M, q4_0 KV, tg128: 31.3 to 54.2 t/s at 16K
context. FLASH_ATTN_EXT 4582/4584, same 2 failures as master.
Assisted-by: Claude
* sycl : 192 head dropped, Wildcat Lake added to docs
Assisted-by: Claude
* sycl : stricter checks from review
Assisted-by: Claude
* [SYCL] : rename XMX macro, add GGML_SYCL_SUPPORT_XMX option, document DPAS
Assisted-by: Claude
* sycl : widen reordered q4_0 dequantization stores
The reordered-layout Q4_0 conversion kernel processed one qs byte per
work-item with two scalar stores, strided 16 elements apart. Give it
the same treatment the Q8_0 kernel got in #30775: a flat
sycl::range<1> launch with four qs bytes (8 outputs) per work-item
inside one 32-element block and two vec<dst_t,4> stores.
Same float math and rounding as the scalar kernel; output is
bit-identical. The existing k % QK4_0 == 0 assert guarantees whole
32-element blocks, so the k/8 launch covers every row exactly. The
now-unreferenced scalar reorder kernel is removed.
Arc Pro B70, primed prefill (prefill after decode on a
reorder-installed tensor): +6.4% to +7.3% pp512/pp2048 on Q4_0
models (tested up to 27B dense); fresh-layout prefill and decode
unchanged; PPL within run noise.
* sycl : vectorize reordered q4_K dequantization stores
The reordered-layout Q4_K conversion kernel already loads its
quantized inputs through sycl::vec but writes the dequantized
results with scalar stores. Build the results in a vec<dst_t,n>
register and emit two vector stores instead. Same per-element
math; output is bit-identical.
Arc Pro B70, primed prefill (prefill after decode on a
reorder-installed tensor): +14.7% to +15.3% pp512/pp2048 on
Q4_K-dominant models (tested up to 27B dense); fresh-layout
prefill, decode and PPL unchanged.
* sycl : remap reordered q5_K dequantization to vec4 loads/stores
The reordered-layout Q5_K conversion kernel gave each thread only
four outputs fed by scalar ql/qh byte loads plus a local-memory
scale broadcast, so the stores were never its bottleneck: widening
the stores alone measured within noise on mixed-quant models and
+0.5% on a q5_K-dominant model. Re-decompose to 32 threads per
block, each consuming four consecutive ql and qh bytes and emitting
two aligned vec<dst_t,4> stores; a 4-element run shares one scale
pair, so the per-element math and the output are identical.
Arc Pro B70, primed prefill on a q5_K-dominant 27B model: +3.6%
pp512/pp2048 over the scalar kernel; fresh-layout prefill, decode
and PPL unchanged. The GET_ROWS reorder path moves to the same
32-thread decomposition.
* sycl : remap reordered q6_K dequantization to vec4 stores
The reordered-layout Q6_K conversion kernel wrote four scalars per
thread, strided 32 elements apart, each pair of neighbors sharing
neither instruction stream nor store transaction. Re-decompose the
same 64-thread work-group so each thread emits one aligned
vec<dst_t,4>: a 4-element run shares one 16-element scale group,
one ql nibble source and one qh bit pair, so the per-element math
and the output are identical to the scalar kernel.
Arc Pro B70, primed prefill: +28.7% pp512/pp2048 on a Q6_K 27B
model; fresh-layout prefill, decode and PPL unchanged.
* sycl : vectorize reordered q3_K dequantization stores
Each thread of the reordered-layout Q3_K conversion kernel already
computes four consecutive outputs; build them in a vec<dst_t,4>
register and emit one vector store instead of four scalars. Same
per-element math; output is bit-identical.
Arc Pro B70, primed prefill: +16.8% pp512/pp2048 on a Q3_K-dominant
27B model; fresh-layout prefill, decode and PPL unchanged.
* sycl : vectorize reordered iq3_xxs/iq3_s dequantization stores
Both reordered-layout IQ3 conversion kernels write eight contiguous
outputs per thread as scalar stores (four from each grid lookup).
Build two vec<dst_t,4> registers and emit two vector stores. Same
per-element math; output is bit-identical.
Arc Pro B70, primed prefill: +20.3% to +21.1% pp512/pp2048 on
IQ3_XXS- and IQ3_S-dominant 27B models; fresh-layout prefill,
decode and PPL unchanged.
* sycl : remap reordered q2_K dequantization to vec4 stores
Re-decompose the reordered-layout Q2_K conversion kernel the same
way as Q6_K: one aligned vec<dst_t,4> per thread instead of four
scalars strided 32 apart. A 4-element run shares one scale byte and
one 2-bit lane, so the per-element math and the output are
identical to the scalar kernel.
Arc Pro B70, primed prefill: +13.1% pp512/pp2048 on a Q2_K-dominant
27B model; fresh-layout prefill, decode and PPL unchanged.
* sycl : widen reordered mxfp4 dequantization stores
Give the reordered-layout MXFP4 conversion kernel the same flat
sycl::range<1> launch as Q4_0: four qs bytes (8 outputs) per
work-item inside one 32-element block and two vec<dst_t,4> stores.
Same LUT lookups and per-block scale; output is bit-identical. The
k % QK_MXFP4 == 0 assert guarantees whole blocks, so the k/8 launch
covers every row exactly. The old scalar kernel is removed.
The reordered layout for MXFP4 is installed on the MoE expert path
(reorder_qw_mxfp4_moe); dense MXFP4 tensors keep the normal layout.
Arc Pro B70, MXFP4 MoE 20B model: +57% pp2048 / +61% pp512 prefill
(primed and fresh alike), decode unchanged, PPL within the model's
run-to-run noise.
SCALE now accepts a src0 view with arbitrary row strides. translate_scale
used the raw input, and on the dynamic path a VIEW translates to its whole
source tensor, so a strided view input scaled the entire source. The model
output then had the source's size, and copying it back overran the result
buffer, which crashes test-backend-ops on the SCALE view cases.
Slice the input with process_view_input_new, as the other translators do.
Reject an in-place SCALE on a non-contiguous view: its result is copied
back densely and would land in the wrong elements.
* hexagon: IM2COL over multiple threads with output-col blocks
Assisted-by: Claude Opus 5.5
* hex-im2col: fix padding handling
* hex-im2col: loop unroll common cases
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
test_cancel_request enables context shift so the generation only ends
when the request is cancelled, then polls /slots with a deadline.
Without context shift the generation fills the context and frees the
slot on its own on fast machines, so the test passes even with
cancellation disabled.
test_server_sleep uses the existing wait_for_sleep helper instead of a
fixed sleep.
* common: bound control vector layer indices by the model layer count
A direction tensor layer index is parsed from the tensor name with no
upper bound, and the buffer size and write offset are computed in int.
A large index overflows both, so the accumulation writes before the
start of the buffer.
Pass the model layer count to the loader, reject indices above it like
the existing zero index check, and compute the sizes in size_t.
* mtmd: fill M-RoPE positions by whole merge tiles
The position loops of the Qwen-VL, GLM, PaddleOCR and Qwen2.5-VL
encoders hardcode a 2x2 merge inside a stride of n_merge, so a patch
grid that is not a multiple of the merge writes past the positions
buffer. Iterate over whole merge tiles with n_merge inner steps, which
bounds the writes by the patch count and gives the same positions on
aligned grids.
PaddleOCR also requires a merge aligned image at graph build, as the
Qwen-VL family already does.
* opencl: run the tiled q4_K lm_head through a tiled GEMM at batch > 1
* opencl: split tiled dp4a q4_k gemm into its own file and test for dp4a
Assisted-by: Codex
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hexagon: add conv2d op
* hex-conv2d: add double-buffered staging
* hex-conv2d: full 64-wide activation block transpose straight from staged buffer
* hex-conv2d: implicit gemm
* hex-conv2d: remove scalar path and kernel
* hex-conv2d: unify conv2d kernel params for all paths
* hex-conv2d: scale HMX result before f16 storage and then restore during unpacking to prevent overflow
* hex-conv2d: fix f16 overflow in HMX paths
* hex-conv2d: refactor to rebase to new op fusion machinery
* hex-conv2d: restructure DMA pipelines and trace scopes
* hex-conv2d: removed the HVX_WIDTH and HVX_TILED paths
* hex-conv2d: remove in-loop vector spills
* hex-conv2d: refactor kparams, vtcm-layout and dma-pipeline to match the other ops
* hex-conv2d: add basic support for CONV_2D_DW
Co-Authored-By: Pranav Dhinakar <pdhinaka@qti.qualcomm.com>
* hex-conv_2d_dw: use DMA in all paths
* hex-conv_2d_dw: support for chunking larger inputs
* hex-conv_2d_dw: remove scalar/hvx access for w tensor (replaced with sync dma copy)
* hex-conv-2d: fix warnings
* hex-conv2d-dw: vectorize all paths
* hex-conv-2d-dw: tree-based reductions and unrolling to optimize
* hex-conv-2d-dw: uninline compute loops from dma loops
* hex-conv-2d-dw: move all kernel selection into the host solver and simplify/specialize kernel dispatch
* hex-softmax: update obsolete gates to enable SOFT_MAX for mmproj
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Pranav Dhinakar <pdhinaka@qti.qualcomm.com>
* mxfp4 sve 256 impl completed
* working with pretty good perf improvement
* review comments addressed
* review comments for SVE 256 check
* trailing white spaces removed
---------
Co-authored-by: Anubhav Srivastava <anubhav.srivastava@fujitsu.com>
* llama : fix DSV4 compressor padding for simple batches
Count actual tokens per sequence when padding compressor writes. Add a graph reuse regression test for odd-sized DeepSeek V4 and V4.1 batches.
Assisted-by: Codex
* tests : cover multiple sequences in DSV4 padding regression
Exercise one and two sequences at different positions while keeping a fixed ubatch shape.
Assisted-by: Codex
* llama : remove simple batch splitting for DSV4
Use the existing sequence splitting to keep n_seq_tokens consistent with
the compressor padding calculation. Extend the graph reuse regression to
6-token batches.
Assisted-by: Codex
* codeowners : refresh teams and maintainer PRs link
Update the CODEOWNERS team legend with the current `ggml-org` team
memberships, and extend the maintainer PRs link in the README to cover
all current members of the `maintainers` team.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : list remaining backend sub-teams
Add the `ggml-*` sub-teams that were missing from the team legend at the
top of the file: base, cpu, et, hip, openvino and riscv. The legend now
covers every sub-team of the `ggml` and `llama` maintainer groups.
No ownership rules change - only the comment block is extended.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : use ggml-hip team for HIP backend
Route `/ggml/src/ggml-hip/` to `ggml-org/ggml-hip` instead of a single
collaborator, so review requests reach the whole team.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : add llama-examples team
List the newly created `llama-examples` team in the team legend at the
top of the file.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* codeowners : use llama-examples team for examples
Collapse the per-directory rules that only listed a single collaborator
into a single `/examples/` rule owned by `ggml-org/llama-examples`.
Directories owned by other collaborators keep dedicated rules, since the
last matching pattern takes precedence in CODEOWNERS.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* cont : update zdnn
* codeowners : use llama-base team for src
Route `/src/` to `ggml-org/llama-base` and list the new team in the
legend at the top of the file. Rules for individual `src` files are kept,
since the last matching pattern takes precedence; `/src/models/` drops
its individual owner, which is already covered by `llama-models`.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* mtmd : add LFM3-VL and LFM3-Omni video input
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* mtmd : reserve the soundtrack command to avoid a GCC 11 warning
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui: build the reasoning menu from the template effort levels
When the chat template reports effort levels, the menu offers Default,
Off and those levels, and a pick is sent as reasoning_effort. A
template that reports none keeps the token budget menu.
Picking a model other than the one that produced the last answer resets
the conversation effort to Default, and an effort the current menu
lacks resolves to Default.
* ui: wait for a cold router model before building the request
A router serves the props of a loaded model only, so a request sent to
a cold model was built without its template effort levels and fell
back to a token budget. The request now waits for the model to load,
joining a load already in flight, and a stop during the wait sends
nothing.
Status waiters hold a list per model so a load and the requests that
wait on it each get theirs, and a props request in flight is shared
with a second caller instead of answering null.
* fix: clamp the A prefetch to end_k in the Vulkan coopmat1 int8 matmul, like the B prefetch
* fix: zero the A scale for k steps past end_k in the Vulkan coopmat1 int8 matmul instead of relying on the clamped block
* chat : strip trailing whitespace from parsed reasoning and content
Assisted-by: Claude Fable 5.1
* chat : stop qwen3-coder reasoning and content before template separators
Assisted-by: Claude
* tests : remove the Qwen parser case from the chat session tests
Assisted-by: Claude
* common : handle partial prefix until match at end of generation
---------
Co-authored-by: Zeshawn Zahid <zzshawnn@gmail.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
* ui : show save and reset settings buttons only when relevant
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move reset settings button next to reload app, pad settings caption
Assisted-by: Claude (Opus 5.5 Medium)
* ui: support OGG audio attachments by converting them to WAV
Fixes#24164
Assisted-by: Claude (Opus 5.5 Medium)
* ui: convert all non WAV/MP3 audio uploads to WAV
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep new chat draft when adding a system prompt
Fixes#24338
Assisted-by: Claude (Opus 5.5 Medium)
* ui: remove unused initialMessage from chat form
Assisted-by: Claude (Opus 5.5 Medium)
* ui : replace the sidebar search mode with a search dialog
Search opens a spotlight dialog (Cmd+K or the Search entry under New chat in the sidebar) with keyboard navigation over recent chats and sidebar actions. Phones keep the search page.
Sidebar actions show their shortcuts, with new ones for Settings (Cmd+,) and Models (Shift+Cmd+M).
Assisted-by: Claude (Opus 5.5 Medium)
* ui : ignore search dialog keys while an IME is composing
Enter confirms an IME candidate and the arrow keys move through the candidates, so the search field leaves them to the IME instead of opening the highlighted result or moving the highlight.
Assisted-by: Claude (Opus 5.5 Medium)
* server : add realtime conversation and transcription on /v1/realtime
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* examples : default the realtime client to the server port 9931
Assisted-by: Claude Opus 5.5
* server : sample realtime responses right after their own decode
Slots respond in turns on one context: sampling at the start of the next step read the logits of whichever slot decoded last, so concurrent sessions mixed their outputs.
Assisted-by: Claude Opus 5.5
* move more stuff to server-realtime, making it self-contained
* examples : fix the type check of the realtime client
Assisted-by: Claude Opus 5.5
* examples : use the model card prompts in the realtime client
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B
Assisted-by: Claude Opus 5.5
* convert : keep the speech task templates out of the d1 GGUFs
Assisted-by: Claude Opus 5.5
* small refactoring: no_batching
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: expose the reasoning effort levels of the chat template
The capability probe renders a user turn for each candidate level
(minimal, low, medium, high, xhigh, max) and for an unknown level. A
level is honored when it renders without raising and differently from
the unknown level, which shows the template fallback. Levels sharing a
render are aliases and the highest one is kept.
The caps serialize to JSON, and chat_template_caps in /props reports
the list as reasoning_effort_levels, from lowest to highest.
* jinja: keep the caps free of the json dependency
The caps keep reporting their flags as a map, and the effort levels
stay a plain member. common_chat_templates_get_caps() builds the JSON
the server reports from both, so jinja depends on json only for the
probe inputs.
In --models mode the thread count per model is computed and printed,
but never set in the params, so every context runs on the common
default, the number of physical cores. With several models in
parallel this oversubscribes the CPU, and the busy-wait barriers of
the CPU backend make it collapse.
Set the count in cpuparams and cpuparams_batch.
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* convert : read the LFM3-Audio config.json as released
A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).
Assisted-by: Claude Opus 5.5
* mtmd : disable flash attention for the LFM3-Audio depthformer
Assisted-by: Claude Opus 5.5
* tts : add a voice option for LFM3-Audio and LFM3-Omni
The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.
Assisted-by: Claude Opus 5.5
* clean up template handling
* convert : detect LFM3-Audio and LFM3-Omni by model_type
Assisted-by: Claude Opus 5.5
* move banned tokens --> logits bias --> mtmd_helper
* leave mtmd-cli unchanged
* no more template_asr_system
* left over
* llama : keep the LFM3-Audio encoder convs and codec unquantized
Assisted-by: Claude Opus 5.5
* common, mtmd : shorten comments
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* common : support deferred tools and tool_reference content parts
Assisted-by: Claude Opus 5.5
* common : credit the deferred tools co-author
Co-authored-by: Aldehir Rojas <765227+aldehir@users.noreply.github.com>
Assisted-by: Claude Opus 5.5
* tests : fix the type check of the deferred tools test
Assisted-by: Claude Opus 5.5
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* tests : set the SWA rope base of the LFM3 test model
Assisted-by: Claude Opus 5.5
* llama : enable recurrent state rollback for LFM3
Assisted-by: Claude Opus 5.5
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>
* gguf-py : align the lfm3vl mapping comments
Assisted-by: Claude Opus 5.5
* llama : fix CRLF line endings from the review suggestions
Assisted-by: Claude Opus 5.5
* tests : add the LFM3 Metal fusion baseline
Assisted-by: Claude Opus 5.5
* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* llama : build the k-pool sparse selection independently of the kq_mask shape
qwen4exp (QSA) and glm5-next pick cache cells by index and no longer read the kq_mask values, but they still shaped the selection like the dense kq_mask. Build the selection as [n_kv, n_tokens] over the cache cells instead, and split it into the streams of the K view right before the attention.
This lets the dense mask change its layout (e.g. per-sequence row lists) without touching the sparse layers.
* improve comment
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : drop the unused build_fused_top_k_mask args, review nits
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* tests : speed up tensor data generation and model saving
Refactor save_models to create only the model, not a llama_context,
which is not needed for saving. Replace the slow std::normal_distribution
draw with a splitmix64 + box-muller normal generator.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : use a shared pool of normal draws for tensor data
Generate a pool of normal draws once with std::mt19937 and
std::normal_distribution, then have every tensor copy a cyclic slice of
it instead of running a per-tensor box-muller. The name-keyed slice
selection keeps the draw deterministic.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : log the size of saved models
Report the produced file size in bytes and MB after each save.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : simplify the model save log
Put the model name, target and size on one line per saved model.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* vulkan: allow dim 3 broadcast in MUL_MAT
supports_op rejected every MUL_MAT whose src0 and src1 differ in ne3,
although mul_mm, mul_mmq and mul_mat_vec already broadcast src0 over
dim 3 through broadcast3. Such a matmul then ran on the CPU with a
copy of the weight, and the split layout changed with the number of
streams in the ubatch.
Keep the strict check for MUL_MAT_ID only, and route the broadcast
cases away from mul_mat_vec_nc, the one path that requires equal ne3.
* tests : harden MUL_MAT with permuted dim 3 broadcast cases
Harden the MUL_MAT coverage with a weight broadcast over dim 3 against
a permuted per-stream activation, the shape of an MLA output
projection with a non unified KV cache, for a single token, several
heads and several tokens per stream.
The expand button wrapping the system message card defaulted to
inline-block. WebKit takes an inline-block's baseline from its last
line even when that line is clipped by the card's max-height, so the
row grew to the height of the whole prompt and left a large empty gap
below the card. Make the button a block so the row matches the card.
Assisted-by: Claude (Opus 5.5 Medium)
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold
estimate with to-be-loaded model size included
use no_alloc to get memory requirements for model load
only set model memory_mb if not previously calculated
use memory margin instead of total size limit, apply to each device separately
add server memory debug logging
move llama_context_device_memory function to llama-ext.h
fix model count exceeded check
improve memory_per_device map naming
improve variable naming, fix style
also strip models memory margin from child processes
cont : clean-up
replace device memory map with buft memory map. Use llama_get_memory_breakdown
extract duplicated check into helper function
move model memory estimation to subprocess
precompute name->buft map, map GPU host types to CPU buft
cleanup unused variable
remove duplicated init calls
* adapt to upstream changes
* wip
* common_memory_reports internface
* SERVER_ROLE_MEASURE
* server-models-sched
* fix build
* server : fix memory accounting of measure children and claim spin
A measure child runs no_alloc, so the free memory it reports does not
include its estimate: subtracting self there erased the memory held by
other processes, down to zero with --fit on, and the next load could
overshoot the margin. Subtract self only for instance reports.
A queued load only checked --models-max before claiming, so when the
memory margin blocked it behind a busy model, load() threw and the
waiter retried in a tight loop. Refuse the claim while the limits are
exceeded and no idle model can be unloaded, so the waiter sleeps on
the condition variable until one goes idle.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* model : add MiniCPM-OCR-Jev support
The model is a Qwen3.5-VL trunk with a value head read at the last token: it
scores an OCR text against an image rendered from it, it does not generate.
Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone.
The class scores are the pooled output of the sequence, read with last pooling,
so the conversion turns score.weight into cls.output and sets
embedding_length_out. The checkpoint has no LM head, so text generation is
rejected instead of using the output weights duplicated from the token
embeddings.
The task is fixed, so its prompt is entirely in the systemone template, the
same way h2o-lightning keeps its default there. The processor of the checkpoint
allows up to 16384 merged tokens per image, more than the Qwen-VL default of
mtmd, so the mmproj keeps the image budget of the preprocessor config. The
checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as
other models do.
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* fix code
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* use gguf meta classifier_output_labels
* remove redudant code changes
---------
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* metal : add GGML_OP_TOP_K_MASK
Port the fused top-k mask from the CUDA backend: a grid-stride kernel fills the output with -inf, then one
threadgroup per row copies the mask values of the kept cells over it. Function constants drop the score,
blocks and extra paths that a given graph does not use.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* metal : clean up top-k mask kernel formatting
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : relax scale to non-contiguous views
SCALE requires src0 to be 1D-contiguous. Relax this to dim-0
contiguity with arbitrary row strides (ggml_is_contiguous_rows),
same as the unary ops in the Metal backend.
- ggml: relax the assert in ggml_scale_impl
- cpu: address src0 rows through their byte strides (f32, bf16)
- cuda: stride-aware kernel, fastdiv row decomposition
- metal: support bf16 scale (guarded unary instantiations)
- sycl: keep the gate at contiguous, the kernel is flat f32
Add view variants to test_scale. Verified on CPU, Metal and CUDA.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* cuda : skip the softcap fusion on non-contiguous views
The fused softcap kernel reads src and writes dst as dense arrays,
so a view reaching SCALE -> TANH -> SCALE now that SCALE accepts
non-contiguous rows would be read wrongly. Keep such graphs on the
strided kernels until the fusion becomes stride aware.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* tests : fix multi-seq split replay against trained models
The rollback snapshot is captured by the graph that decoded the rolled-back
tail, while the reference context decodes the same prefix with different
batch shapes. The two states differ in the last bits and trained models
amplify the difference with every replayed token. Compare only the first
replayed token of each seq, then restore the rollback state into the
reference context before the seq-1-only tail.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : require bit-exact seq-1 isolation in split replay
The seq state save is not always bit-lossless - SWA-masked cells are not
saved - so reload the restored state back into the rollback context. That
way both contexts enter the seq-1-only tail with byte-identical memory,
including the cell layout, and the tail logits must match bit-exact. Also
assert that the restored state round-trips bit-identical.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : drop the n_swa partial-state check from ps
The check keys off llama_model_n_swa(), which returns 0 for dsv4 even
though its raw KV cache has a 128-cell sliding window. The partial state
grows with the tokens there by design, so the flatness check is wrong.
Keep the partial <= full invariant.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Every caller passes n_pad = 1 and the only two users clamp it to 256.
Drop the parameter, the members and the trivially true kv_size assert.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The org-less group renders without a header, so when it followed a named
org its models looked like they belonged to that org. Put it ahead of the
named org groups.
Fixes#25227
Assisted-by: Claude (Opus 5.5 Medium)
On desktop, the reasoning effort toggle now gets a muted background on
hover and while its menu is open, matches the model selector's 24px
height, uses a 13px icon and sits 5px from the selector.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add conversation compaction
Add a compaction action that asks the model to summarize the
conversation so far and stores the summary in a new compaction node
appended to the branch. Later requests slice the branch at the last
compaction node, keeping the system prefix and continuing from the
summary instead of the full history, which frees up context while the
complete history stays visible and navigable in the UI.
The slice is applied once at the request funnel, covering the plain
send, the agentic flow and the KV cache pre-encode, plus the continue
path that bypasses it. The summary request is a one-shot outside the
SSE replay machinery, streamed live into the node with a full rollback
on abort, error or empty result.
The action is exposed as a /compact slash command and a Compact button
in the context gauge popup, with an overridable summarization prompt
in the settings.
* ui: pre-fill KV cache and refresh context gauge after compaction
Once the summary is stored, pre-encode the compacted branch and stamp
the returned prompt timings on the compaction node: re-encoding the
reduced prefix is part of the compaction itself, so the server KV
cache and the context gauge reflect the summary right away.
The summary stream is registered under a prefixed identity
(compact::convId), so the visibility kick and byte-offset resume
protect it through tab switches while the reload probe never
reattaches it as a chat turn. The stamped node becomes the shared
context truth for the gauge and the processing state restore, and the
stale live state that kept the gauge full is cleared.
* chore: format
* ui: allow a dedicated model for conversation compaction
An opt-in setting reveals a model selector whose options come from the
live router model list. The chosen model writes the summary and is
loaded on demand when it is not resident, while the default path keeps
resolving to the conversation's own model.
* ui: align the compaction model field with project conventions
* ui: fix the model option helpers import path
The compaction model selector pulls filterModelOptions and
groupModelOptions from the models component folder, where they no
longer live, so the build fails to resolve the module. Point the
import at the navigation folder, which now owns them.
Missed during the rebase: the helpers moved in a commit that does
not touch any compaction file, so nothing conflicted.
* ui: import the model option helpers from models/utils
* ui: follow the moved models selector in the compaction model field
The models manager moved ModelsSelectorList under ModelsSelector and
the model option helpers to /utils, where groupModelOptions takes
the options alone. Drop the stale copy the rebase kept and read the
helpers from their new home.
* ui: drop the dedicated compaction model
A summary written by another model loses the prefix the conversation
model cached, so the whole history is prefilled again, and the model
then resumes from text it did not write. The conversation model
always writes its own summary.
* ui: keep the last steps of a compacted conversation
A compaction now folds the steps the model reads but the last ones,
a step being a user message or an assistant message with its tool
results, and stores how many it kept on the compaction node. The
model then reads the system prefix, the summary, the kept steps and
what follows.
The summary request is the request the chat sends, with its options,
tools and message shaping, cut before the preserved steps: the server
reuses the prefix it cached and only the instruction is new. The
number of preserved steps is a setting, 10 by default.
* ui: run every compaction summary through one path
Three callers needed the same thing - the automatic pre-encode pass, the
action in the context gauge, and re-running a summary from the block - and
each of them had grown its own copy of the request, the streaming and the
storing.
There is one path now: a summarization owns a node, streams into it, and
reports itself through a ledger the rest of the app reads (`isCompacting`,
which node belongs to which conversation). Re-running a summary is the same
call on an existing node, so the two entry points cannot drift apart again.
* ui: give the compaction row its own summary UI
The summary was a line of text in a tool-call-shaped collapsible, which is
how a compaction reads as a side note rather than as the thing the rest of
the conversation now stands on.
The row is its own block: a divider that names the boundary, and the summary
inside it with the model that wrote it, its statistics, and the same actions
an assistant turn has - copy, edit, fork, regenerate and delete all apply to
the summary, and it collapses to the divider once a summary is in, because
the shrunk context is the headline and the prose is the detail.
The divider is a variant of the shared collapsible block, which also needed
the toggle and the title split into snippets so a caller can lay them out
without copying the markup. The action icons no longer take a role: they
render what a message offers, and a compaction offers the same as any other
stored message.
* ui: report compaction progress on the row
The row owns the run, so the row says how it is going: the prompt progress
in the header while the context is being read, the phase once the summary
starts streaming, and the token count once it is in. Before the first
summary token there is nothing to collapse, so the header stands on its own
instead of offering an empty block.
* ui: keep the summarized turns quiet while a compaction runs
The turns being summarized used to keep their own "Processing..." indicator
and live reading stats while the summary streamed, so the conversation said
the same thing twice in two places. The row reports the run, so the turns it
replaces stay silent until it is done.
* ui: hand the compact action to the context gauge
Compacting is a decision about the context, so the action belongs with the
numbers that show the context filling up rather than inside the form: it sits
at the bottom of both the hover card and the drawer, under the breakdown it
acts on, quiet enough to stay a footnote and disabled - with the row's own
spinner - while a summary is already running.
* ui: follow the bottom, otherwise leave the view alone
The screen scrolled itself with a controller of observers, intervals,
direction heuristics and landing timers, and still managed to move the view
while the reader was reading: any mutation that grew the branch was treated
as a reason to keep following.
The page is the scroller, so it follows when the reader sits at the bottom,
and stays where it is otherwise. A resize observer plus native scroll
anchoring does the rest, and `pointerdown` buys the reader a moment of grace
to be caught mid-drag.
* ui: expand collapsible bodies with a height transition
Expanding a block snapped it open: the body was mounted the moment `open`
flipped, so a tall block moved the conversation around it in one frame.
The body now goes through the same region the models table uses. The height
interpolates and the visibility follows it, so a body is unmounted once the
collapse has finished rather than the moment it starts - a closed block
still costs nothing while tokens stream past it.
* ui: open a compaction summary at the top of the screen
Toggling a summary left the reader wherever they were, so a conversation that
had just compacted showed the divider at the bottom of the screen and read
down from there.
A toggle now scrolls the page once and lands the divider a breath under the
tabs, both ways: the summary is read from its beginning, and closing it returns
to the same boundary instead of wherever the page happens to come to rest. The
scroll waits for the body to reach its height, because before that the page is
too short to scroll the divider up there and the call would clamp.
That wait is the shared collapse duration rather than a transition event: an
engine without `interpolate-size` animates the region's grid row instead of its
height, so a listener on `height` would never fire there and the page would not
move at all.
* ui: spin the compaction icon while the summary runs
The divider reported the phase in its title while the icon still showed a
finished block, so a run in progress and a summary that had landed looked the
same in the one place the eye reads first. The icon spins while the summary is
running and folds back once it is in.
* ui: let the compaction summary speak without the reasoning trimmings
The summary is the thing the conversation now stands on, not a note beside
it, and the conversation model always writes it, so the body drops the
reasoning block's rule and indent (`flushBody` keeps them for the blocks
that borrow them) and the model selector under the summary is gone - there
is nothing left to swap.
* ui: drop the model override from the compaction path
The summary row no longer offers a model selector, so nothing hands
regenerateCompaction a model anymore. Remove the parameter from the
whole path, with the on-demand load of another model it carried: the
summary always comes from the model the request funnel resolves.
* ui: let the compaction summary think as the chat does
The summary request forced thinking off, overriding the reasoning
effort the user chose. The summary is the text the conversation
resumes from, the task where reasoning helps the most, so the request
now carries the very options of the chat.
* ui: pre-encode the compacted branch as the next turn reads it
The post-compaction pre-encode sent the branch without tools, while
the tools render ahead of the messages, so it warmed a prefix the next
turn never reads and that turn prefilled the whole compacted context
again. The pre-encode now sends the tools, message shaping and
reasoning handling of the chat, so the next turn only prefills its new
message.
* ui: pre-encode the branch the next turn actually reads
The per-turn pre-encode sent the messages of the request plus one
assistant message holding the streamed text, without tools. With
tools on, it replaced the slot the next turn extends with a prefix
nobody reads again, and after an agentic turn its history did not
match the stored calls and results. Without agentic, it never ran.
Both pre-encodes now read the stored branch up to its leaf, cut at the
last compaction, and send it with the tools, message shaping and
reasoning handling of the chat. The plain completion path triggers it
too.
* ui: refuse a compaction summary the output limit cut
The stream handler read every chunk but dropped the finish reason, so
a summary cut by max_tokens folded the conversation into half a text.
The handlers now hand the finish reason to onComplete, and a summary
that ends on length rolls the compaction back with an error.
The summary was framed through a replacement string, so $& or 32560 in
it changed the text the model read; it is framed as written now.
* ui: wait for the collapse transition in the tool call tests
The collapsible body now unmounts once its collapse animation ends rather
than as soon as the block closes, so the "folds back" assertions ran before
the body was gone. Wait out COLLAPSIBLE_TRANSITION_MS first.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* ui: open model details in a global drawer without leaving the page
The model info actions in the selector used to navigate to the models page
to open its details pane, and a phone opened a separate dialog instead. The
drawer now lives outside the models table, driven by uiStore, so the info
icon opens it over whatever page is showing.
The models dropdown and the phone picker stay open under the drawer's
backdrop, and keep focus while the drawer opens and closes, so the trigger's
tooltip no longer pops over the drawer.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let the model details drawer reopen after a drag to dismiss
Dragging the drawer down wrote vaul's own open state, which the one-way open prop then kept closed, so the info action could not open it again until a refresh. Bind open both ways through the store instead.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: close the models selector from the details drawer chat actions
Assisted-by: Claude (Opus 5.5 Medium)
Extract the keyboard-shortcut delete confirmation into DialogConversationDelete
and reuse it in the sidebar and mobile search page instead of the native
window.confirm prompt.
Assisted-by: Claude (Opus 5.5 Medium)
* hexagon: add GGML_HEXAGON_VTCM_MB and GGML_HEXAGON_NTHREADS caps
Change-Id: Idffe1613c8db6f96aa842d43d1b2efaed45462a8
* hexagon: rename VTCM cap to GGML_HEXAGON_VTCM_SIZE, drop NTHREADS cap
Change-Id: If4c5b9ce9fd6b725c3561421c2295d40599d012d
* hexagon: make GGML_HEXAGON_VTCM_SIZE an override, accept K/M/G suffixes
Change-Id: Ib8eea84c22f24e6d986b37f215e504b046478c2c
The hyper-connection and DSA graph builders wrote their extra outputs
(post / comb gates, carried top-k selections, inject gates) through
ggml_tensor** out-params. Return them in a small struct next to the
primary output instead.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Every dst row comes from src0 or src1, or for dim 0 from the row of
src0 followed by the row of src1, so a single kernel covers every
type: a dense source row is one memcpy, quantized types included,
and a strided source is copied element by element with the element
size known at compile time.
Rows are split across threads instead of i2 alone, which is often 1,
and are copied in tiles of 32: for a transposed source the elements
are walked in the outer loop, so the rows of a tile read neighboring
addresses instead of jumping a full row stride per element.
Add transposed b cases to test_concat, the layout of the delta-net
conv input, and a perf case for it.
* sycl: add set_tensor_2d for faster tensor-split loading
Pack host rows into chunks of up to 64 MiB when destination rows are
contiguous, avoiding a separate upload call for each row. Keep the
existing per-row copy path for non-contiguous destinations.
On two Arc Pro B70s with gpt-oss-120b UD-Q4_K_XL, total llama-perplexity
runtime drops from 263-268 s to 114-115 s. A full-load readback check
verified 26,784,256 rows with no mismatches.
* sycl: batch 2D tensor uploads through shared staging slots
Implement set_tensor_2d to speed up tensor-split model loading by
packing rows into the existing pinned staging slots and uploading
each batch with a single device copy.
Share staging initialization with set_tensor and wait only when
reusing a slot, allowing CPU packing and device copies to overlap.
Fall back to per-row uploads for noncontiguous device rows,
oversized rows, or unavailable staging.
---------
Co-authored-by: Jon Staples <250234834+Captain-Tripps@users.noreply.github.com>
* tests: pin model resolution on real repo listings
Extend test-model-resolution with 63 repo listings captured verbatim
from the HF API, chosen to cover every file naming pattern found across
1237 GGUF repos of the major publishers and pruned to whole quant groups.
Each repo pins its resolved plans and its cache listing.
The expectations follow what each file really is per its GGUF header,
not what the current resolver picks, so a mismatch is a resolution bug.
Mismatches are collected and reported together before the test fails.
* tests: leave the listing unchecked on paths the filesystem refuses
Some real split names (DevQuasar's Model.Q2_K.gguf-00001-of-00012.gguf
under a quant folder) exceed MAX_PATH once materialized in the temporary
cache on Windows, so the listing of such a repo is skipped with a note
while its plans stay checked.
* tests: pin real listings to master on GGUF convention names only
Replace the real repo table with 171 HF listings whose file names follow
the GGUF naming convention (lowercase sidecar prefix, - delimited
components, quant last), one per layout pattern found across 1174 GGUF
repos and pruned to whole quant groups. Repos outside the convention are
dropped. Each repo pins the plans and the cache listing master gives
today, so the test is a pure harness: any change in resolution shows up
here, and mismatches are reported together.
test-llama-archs included src/llama-arch.h and src/llama-model-saver.h,
which prevented building on Windows with shared libraries since the
used symbols are not exported from the DLL.
- declare llama_arch_* wrappers around the internal llm_ arch query
functions in the llama-ext.h staging header, together with
llama_arch_kv_name() and llama_arch_supports_save()
- slim llama_model_saver down to the private implementation of
llama_model_save_to_file(): the fixture-writer half used by the test
moves into tests/test-llama-archs.cpp as a local kv_writer
- replace the roundtrip saver usage with the public
llama_model_save_to_file() writing to a temp path (also removes the
tmpfile() fd leak and its Windows workaround)
- move test-llama-archs, test-generate-models, test-llama-context and
test-fusion out of the NOT WIN32 OR NOT BUILD_SHARED_LIBS guard
Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
* Metal : stop overflow from happening in large matrix output offsets
Assisted-by: Codex
* metal : widen batch offsets and MUL_MAT_ID token indices
Assisted-by: Codex
---------
Co-authored-by: feni6 <270037094+feni6@users.noreply.github.com>
* ui : land opened conversations at their bottom before the first paint
Pin the page to its bottom with a ResizeObserver until it stops resizing or the reader interacts, and mount the last rows eagerly so the landing frame shows real content.
Assisted by: Claude (Opus 5.5 Medium)
* ui : make conversation tabs browser-style
A plain navigation shows the chat in the current tab instead of adding one, and the new-chat tab turns into the conversation it starts. New tabs come only from explicit actions: the + button, Cmd/Ctrl-click or middle-click on a sidebar chat, and its Open in new tab menu item, which open in the background. Closing a tab moves to its neighbour among the tabs the view shows.
Assisted by: Claude (Opus 5.5 Medium)
* ui : share the tab surface and tab bar fade styles
Move the tab item surface (shape, resting and active looks) to TAB_SURFACE_* class constants and the fade under the tab bar to a global chat-top-fade class, so other elements can match the tabs.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fit the chat column below the tab bar
The tab bar reported 1.25rem of its height while it takes 3.3125rem (pt-3.25 + h-10), so a short conversation still left a few pixels to scroll. Use the bar's full height and drop the extra 1rem from the column's minimum height, so it fills the viewport exactly; the centered new-chat form moves up by the same 1rem to keep its place.
Assisted by: Claude (Opus 5.5 Medium)
* ui : stop the page from bouncing past its ends
Elastic overscroll on the root scroller dragged the sticky and fixed bars (tab bar, chat form) along at the top and bottom of a chat, and they jumped back on release. Set overscroll-behavior: none on html and body.
Assisted by: Claude (Opus 5.5 Medium)
* ui : stop scrolled dialogs from bouncing past their ends
Dialogs scroll on their own (the settings panel, the search results), so the page rule did not reach them and their sticky navigation still jumped on overscroll. Turn off overscroll on dialog panels and on the scroll containers inside them. Only scroll containers: Firefox counts an overflow-hidden box such as truncated text as one, and `none` there kept the wheel from scrolling the list around it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fit a first exchange opened by the working directory row
The last answer's minimum height leaves room for the last question only, so the synthetic row that a new chat starts with (Set working directory to ...) made a short first exchange overflow by that row. When everything above the last question is synthetic, count those rows as part of it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : keep following a streaming response until the reader scrolls up
Auto-scroll stopped whenever the page moved up, including moves the
reader did not make: scroll anchoring and blocks re-rendering shorter
while a response streams. Only the reader's wheel, touch, scroll keys
or scrollbar drag now count as scrolling up.
The send scroll also looked up the new user bubble by position, which
picked an older message when the answer row had not rendered yet.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fade the chat form between its centered and bottom positions
The form no longer slides across the column between the new-chat center
and the bottom. It fades out drifting toward where it goes and fades
back in where it lands, quicker when opening a chat than when going back
to a new one. Its wrapper keeps the same height in both positions, so
the messages never shift while it moves.
The new-chat greeting fades in as the form settles, also on a refresh or
a first visit. Reduced motion keeps only the fades.
Assisted by: Claude (Opus 5.5 Medium)
* ui : let the expanded sidebar be resized by dragging its edge
Bring back the desktop sidebar resize handle that was dropped with the
shadcn sidebar primitive. The width is clamped between 18rem and 32rem,
persists across reloads, and resets on a double-click of the handle. The
tab bar follows the same width so a wider sidebar never covers it.
Assisted by: Claude (Opus 5.5 Medium)
* ui : rubber-band the sidebar edge past its bounds and drag it shut
Past the max width the edge resists harder the further it is pulled, and
the same happens below the min width until the pull passes 9rem, where it
snaps to the pointer and letting go hides the sidebar. A release outside
the bounds springs the width back to the nearest one.
Assisted by: Claude (Opus 5.5 Medium)
* ui : show the sidebar edge strip faintly on hover and solid while dragging
The strip is clipped to the sidebar's rounded corners through its own
overlay, so the sidebar itself is never clipped.
Assisted by: Claude (Opus 5.5 Medium)
* ui : remember whether the desktop sidebar is open across reloads
A phone opens the sidebar as an overlay, so it always starts closed there
and its toggling never overwrites the desktop preference.
Assisted by: Claude (Opus 5.5 Medium)
* ui : fade the sidebar labels and conversation list with its width
The fade follows the live width, so it tracks both the toggle's width
transition and a drag below the minimum width.
Assisted by: Claude (Opus 5.5 Medium)
* ui : persist the sidebar width on release and clamp a restored one
A drag now updates only the live width and writes it to localStorage once
when released. A width restored from localStorage is clamped to the
current bounds, so a value saved under older bounds does not stick.
Assisted-by: Claude (Opus 5.5 Medium)
* kv-cache : use ggml_backend_tensor_copy to refill the DSV4 K mirrors
* llama : move the Engram pad id to llama_hparams and the token map to llama_model
The saver no longer needs the deepseek41 model class.
* deepseek41 : drop the F32 src1 request on the HC mixes matmul
* ggml : add softplus TODOs in the backends that still use log(1 + exp(x))
* convert : make the DSV4/DSV4.1 modify_tensors generators, drop the redundant vision filter
* deepseek41 : clean up comments
* deepseek41 : drop the F32 src1 requests on the router and indexer matmuls
* convert : use format_tensor_name directly in DeepseekV4Model
* s/comp_kv_src/comp_il_src/g
* cont : fix TODOs for log1p
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* cuda : use std::string instead of VLA for archName
Modernize the code (old C style) and fix the following warnings:
```
ggml/src/ggml-cuda/ggml-cuda.cu:180:19: warning: variable length arrays in C++ are a Clang extension [-Wvla-cxx-extension]
180 | char archName[archLen + 1];
| ^~~~~~~~~~~
ggml/src/ggml-cuda/ggml-cuda.cu:180:19: note: read of non-const variable 'archLen' is not allowed in a constant expression
ggml/src/ggml-cuda/ggml-cuda.cu:179:9: note: declared here
179 | int archLen = strlen(devName);
| ^
```
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Add a comment about archName
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
* common, grammar: add recursive depth limit, make it a requirement
* catch more recursive functions
* common : tidy up peg parser changes
* grammar : reject candidates in a loop, without a depth limit
The depth limit in llama_grammar_reject_candidates_for_stack counted
the code points of a vocab token, so any token of 256 chars or more
was always rejected under a grammar, such as the 512-space token of
GLM-4.7-Flash or the long space tokens of Laguna-S-2.1.
The rejection now walks the code points with its own stack of levels
instead of recursing: there is no limit and no stack risk, whatever
the length of a token from the GGUF, and the masks match master.
* grammar : track rejected candidates as indices into the input
At a given code point a candidate only differs from its input by the
position of its code point pointer, so the levels now carry indices
into the input candidates instead of copies. The first level needs no
special case and the rejects map back to the input directly.
Grammar sampling gets faster than master on GLM-4.7, Laguna, Qwen3.6
and gemma-4 with the json, list and spaces grammars, same masks.
* nit : shorten the comment of llama_grammar_reject_step
* common : add depth_guard for schema walks
* grammar : move GRAMMAR_RECURSIVE_LIMIT into the .cpp
* common : reduce recursion limit for peg parser
* improve
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
Co-authored-by: Pascal <admin@serveurperso.com>
* llama-quant : fix quantizing 4D tensors
nrows_total was ne[1]*ne[2], so for a 4D tensor only the first ne[2]
slices were quantized and the rest of the rows were dropped. Use
ggml_nrows(tensor) to count all rows, including ne[3].
Fixes#29916
* llama-quant : detect expert tensors by name and fix imatrix size for 4D tensors
The MXFP4_MOE path used ne[2] > 1 to detect MoE expert tensors. This is
wrong for non-expert tensors with 3 or more dims, e.g. MLA projections
(BailingMoE3) and 4D conv kernels. Use the tensor name instead: all
tensors used by GGML_OP_MUL_MAT_ID are named *_exps or *_chexps. This
also removes the BailingMoE3 special case.
The imatrix holds one vector of ne[0] values per 2D slice, so the
expected size is ne[0]*ne[2]*ne[3], not ne[0]*ne[2]. This matches how
the imatrix is indexed per row for 4D tensors in llama_tensor_quantize_impl.
Assisted-by: Claude Opus 5.5
* cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows
For prompt processing, run the gate and up matmuls of a dense FFN as one
mul_mat_q launch and apply the GLU in the write-back.
- The loaders take the gate weights as a second pointer and fill the weight
tile with up and gate rows interleaved in chunks of 8, so each thread holds
an up value and the matching gate value (rows r and r + 8 of an MMA tile).
- The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[]
per thread. The fused write-back computes the GLU from registers and writes
half the rows.
- The fused path uses the regular tiling (no stream-k) with the column tiles
innermost so that neighbouring blocks share weight tiles.
- Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.
* cuda: assert the fused MMQ gate argument and trim redundant comments
* cuda: move the fused MMQ hint check to should_fuse and add TODOs
* cuda: pass prec_src1 to the fused MMQ write-back
* cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back
* cuda: require has_fusion explicitly, drop the fusion-only src1 precision check
* cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile
* cuda: inline the MMQ tiling-efficiency check
* tests: one fused gate/up MMQ case per type and GLU op
* cuda: always launch the fused MMQ kernel with one block per tile
* tests: shorten the fused gate/up MMQ test comment
* cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs
* cuda: run the fused MMQ kernel on the plain tiling path
* cuda: put the column tiles innermost in the MMQ plain tiling path
* vulkan: Add a larger tile size for conv3d (coopmat2 only)
This adds a larger tile size, in practice it is only for coopmat2 where
we don't need shared memory to store the output (would use more than 64KB).
Add some additional representative backend tests. At least, this is what codex
came up with from looking at sd.cpp Wan VAE.
* only apply large tile size for newer driver versions
WM of the small tiles was 32, or 8 for 8-wide subgroups, whatever the
tile's BLOCK_SIZE and WARP. With 16-wide subgroups (Mali) the second
subgroup of a workgroup wrote into the next tile's columns. With 8-wide
subgroups some _id tiles left rows uncomputed. Derive WM so there is one
warp tile per subgroup.
Tested on my Pixel 9 Android phone. Before the change there were 20
MUL_MAT and 6 MUL_MAT_ID failures. After the change they all pass.
Fixes#28637
Assisted-by: Claude Opus 5.5
* hex-tools: support for --compare option in the profile.py
* hex-unary: improve collapse logic to also do splits if nrows is small
* hex-colapse: handle all colapsing at the host and make it generic
* hex-colapse: remove leftover unary helper
* chat : reject JSON nested more than 128 levels before walking it
The schema builder, the jinja engine and the decision code walk request
JSON with recursion. A request nested deep enough overflows the stack of
the HTTP thread and llama-server dies (SIGBUS on macOS, SIGSEGV on Linux):
for a tool schema from a few hundred levels on macOS and about 10000 on
Linux.
Check the depth first and answer 400 past 128 levels: the chat inputs in
common_chat_templates_apply (tools, tool-call arguments,
chat_template_kwargs, json_schema), the json_schema of /completion, and the
whole /v1/systemone body. A gemma4 tool result that is too deep stays text.
* fix the root, not the call side
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add DeepSeek-V4.1-Flash (deepseek41)
* deepseek41 : remove MXFP8, convert the Engram table to BF16
* deepseek41 : compute the compressed-KV read layers once in the cache
* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le
* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather
* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer
* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle
* deepseek41 : keep il last in build_hc_mixes
* deepseek41 : sort the top-k rows before the sparse gather
* tests : add deepseek41 to the Metal fusion baseline
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* hexagon: KV-head team split for HMX flash attention in multicore
* hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag
0 = off (token-split only); 1 = head-split, plus team-split when
n_kv_heads < N_cores so stranded cores still participate (e.g. gemma-4
with 2 KV heads on 4 cores). Where heads divide evenly across cores,
plain head-split is used.
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the
per expert-token amax scaling done in the compute graph
(ref: https://github.com/ggml-org/llama.cpp/pull/30861), which keeps the
MoE down-projection activations within f16 range before the op runs.
Drop the two-stage reduction kernels, their scratch buffer and the
GGML_PREC_F32 dispatch added in https://github.com/ggml-org/llama.cpp/pull/26223,
as well as the test cases that drive MUL_MAT_ID src1 out of f16 range.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ui : move models manager to its own route, keep Discover as a dialog
Models now lives at #/models instead of a dialog. Discover opens as a
standalone dialog over it, and leaving the page returns to the chat it
was opened from.
Assisted by: Claude (Opus 5.5 Medium)
* ui : polish the models page layout and filters
Give the models page a page header with the Discover action, calmer
column labels, tinted section and family headers, and an empty state.
Load the model list on a direct visit, slide the detail pane over the
table below xl, and draw the filter toggles as one box matching the
context select, with room for their shadow.
Assisted by: Claude (Opus 5.5 Medium)
* ui : round the models table family headers
Assisted by: Claude (Opus 5.5 Medium)
* ui : fix models page navigation and Discover gating
Record the models page's way back after every navigation, so Back,
a typed URL or a reload no longer return to a stale chat. Escape
closes the detail pane, then leaves the page. Gate every Discover
entry point on one shared check, recheck the API key on /models,
skip the duplicate router fetch on a direct load, and share the
route load between pages.
Assisted by: Claude (Opus 5.5 Medium)
* ui : show model details in a drawer
Open the models manager's detail pane in a drawer, from the right on
desktop and from the bottom on a phone, instead of an inline side
panel and a full-screen cover. Pad the pane on both sides, drop its
close button on a phone, and assert that filtered rows are gone from
the page rather than merely hidden.
Assisted by: Claude (Opus 5.5 Medium)
* ui : make whole models table rows clickable
A click anywhere on a model or quant row opens it, unless it lands on
one of the row's own controls (load toggle, actions menu). The name
button stays the keyboard path to the same action.
Assisted by: Claude (Opus 5.5 Medium)
Use the dropdown menu surface (border, radius, background, shadow,
padding) for the hover card and the context gauge popup, and fix the
hover card open/close animation to use bits-ui data-state attributes.
Assisted-by: Claude (Opus 5.5 Medium)
The two T4 runners in the ggml-ci group (ggml-6-x86-vulkan-t4,
ggml-8-x86-nvidia-a10) share the same labels and can both pick up the
same jobs, so extend the per-runner results dir from #31046 to them.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The generic scale pass in llama-model.cpp loads the per-expert
NVFP4 scale2 tensors (ffn_{up,gate,down}_exps_s) and the per-tensor
MXFP8 scales for the dense FFN and the shared expert, but
minimax-m3.cpp never forwarded any of them to the graph.
build_moe_ffn() defaults these to nullptr, so build_lora_mm_id()
skipped its w_s branch entirely and the routed-expert weights came
out global_scale times too large, swamping the residual (reported
G ~= 15123 for M3.1 Flash) and producing token soup.
Forward all scale tensors at the three call sites, matching the
pattern used by flint and the other NVFP4-capable MoE models.
Co-authored-by: origin-labs-ai <origin-labs-ai@users.noreply.github.com>
Add test-speculative-rejection-sampling to the ctest -E exclusion list of the macos-latest-arm64 CI job.
Assisted-by: dsh:llama.cpp/ggml-org/Flint-GGUF (RTX 5090 32GB, DDR5 128GB)
The amax POOL_1D reduction spans the whole matmul row, so a device cannot
compute it from an axis-0 split. The Meta split-state case reported
MIRRORED, which gives each device a partial max of its own slice. Drop
that case and deny -sm tensor for the arches which build the amax graph,
see [TAG_MOE_AMAX_SCALE].
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : allow MUL_MAT_ID -1 index to skip the computation
* nits
* update vulkan code
* ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec
(cherry picked from commit d8a4b9efb6e2b9abf3a866e847756b7e6bbc6b69)
* fix cuda
* openvino
* exclude CUDA from this PR
* add TODO tag
---------
Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
* tests : add speculative rejection draft sampling test
This commit adds a test for speculative rejection sampling test that
excercises the speculative rejection sampling mechanism in
common/speculative.cpp.
The motivation for this is have a test that verifies that the
speculative draft models emitts tokens that match the target models
distribution. So when rejection sampling occurs it should still
produce the same probability distribution as if we directly sampled
from the target model.
Running the test:
```console
$ ctest --test-dir build -R "^test-speculative-rejection-sampling$" \
--n_trials 5000 -V
```
This will generate two models, one target and one draft model.
For subsequent runs use the "fixture exclude any" option to avoid
recreating the models:
```console
$ ctest --test-dir build-cpu-debug -R \
"^test-speculative-rejection-sampling$" \
-FA "^speculative-models$" --n_trails 5000 -V
```
* remove draft model
* ci : fix data race between runners sharing the model cache
Two runners per mac mini share HOME, so concurrent jobs corrupt each
other downloads, converted models and venv under the shared MNT cache.
- gg_wget: download in a temp dir, publish with mv, so readers only
ever see complete files (-N cannot be combined with -O)
- convert, quantize and wikitext unzip now produce under a per-runner
scratch dir in OUT, published to MNT via hardlink when missing
- gguf-split and quantize tests get a per-runner TMP_DIR
- python venv moved from MNT to OUT
- metal, vulkan and webgpu mac jobs pass a per-runner OUT path
Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)
* ci : clean up scratch models after tests
Assisted-by: dsh:llama.cpp/agent-qwen4 (RTX 5090 machine)
* ci : fix q4_0 publish when a cached model exists
cp -n skips an existing file with exit code 1, so the ln fallback
aborted the qwen3 job under set -e. Publish through a unique temp
file with an atomic mv: readers never see a partial file and a
stale cached q4_0 gets replaced.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)
* ci : cache-first gg_wget to avoid re-downloads
HF sends no Last-Modified, so wget -N re-downloads the full files on
every run. Skip the download when a non-empty cache file exists: the
HF CI artifacts are static. Downloads still go through a temp dir and
an atomic mv, so concurrent runners never read partial files.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 machine)
* HIP: add byte permute fallback when __builtin_amdgcn_perm is unavailable
amdgcnspirv rejects target-specific builtins unless they are guarded by `__builtin_amdgcn_is_invocable`, so 66 of 145 HIP files failed to build
Wrap the perm calls in a helper that keeps v_perm_b32 when the target has it and falls back to a portable byte select otherwise
* move perm helper to vendors/hip.h
* mtmd : lfm2 bilinear resize matches torchvision
Lfm2VlImageProcessor and d1-omni vision.py resize uint8 images with
torchvision (antialias=True). That is Pillow's algorithm, except that the
weights are int16 with as many fractional bits as the largest weight allows,
where Pillow uses 22 bits. The Pillow port was 1-2 levels off on about 1% of
the values of the d1-omni and LFM2.5-VL crops.
RESIZE_ALGO_BILINEAR_TORCH rounds the weights of the Pillow port to that
precision and scales them back to 22 bits, which gives the output of
torchvision with the same passes, and "bilinear" maps to it for LFM2 and
d1-omni. Other models keep the Pillow algorithms.
* mtmd : lfm2 bicubic resize matches torchvision
Lfm2VlImageProcessor resizes with torchvision for the bicubic checkpoints too
(d1-3B, LFM2-VL), so "bicubic" maps to RESIZE_ALGO_BICUBIC_TORCH for LFM2: the
Pillow bicubic filter (a = -0.5, as torchvision with antialias) with the int16
weights of torchvision. The Pillow port was 1-2 levels off on about 0.25% of
the values of the d1-3B crops.
* use bitfield, add hparams setter
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Add Soprano-80M and Soprano-1.1-80M (https://huggingface.co/ekwek/Soprano-1.1-80M)
as an audio generation model: a Qwen3 backbone plus a Vocos decoder that runs
as a gen-only mmproj.
- convert: detect Soprano checkpoints and export the backbone as qwen3 and the
decoder.pth Vocos weights as the mmproj
- vocab: add the soprano pre-tokenizer (lowercase, whitespace collapse,
per-digit split, [UNK] fallback)
- mtmd: add the Vocos graph and ISTFT, allow loading a gen-only mmproj for
Soprano and include the gen-audio context in mtmd_get_memory_usage
- mtmd-helper: add the Soprano pipeline; the decoder runs over the whole
utterance, so stream mode sends the audio in one chunk after flush()
- tts: document Soprano usage
Assisted-by: Codex
Assisted-by: Claude Opus 5.5
* hexagon : add SGN support
Assisted-by: OpenCode
* hexagon : optimize SGN with sign bit
Assisted-by: OpenCode
* hex-sgn: add f16
* hex-unary: update SGN to use generic unary_colapse and relax colapse checks to apply to more conditions
* hex-unary: improve dims colapsing and run it in the host-side solver
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The __builtin_amdgcn_wmma_f32_16x16x16_bf16_w32 (and _gfx12) builtins
take their A/B operands as vectors of short (_ExtVector<16, short> /
_ExtVector<8, short>), not __bf16. Declaring the fragments as
ext_vector_type(N) __bf16 fails to compile:
error: cannot initialize a parameter of type 'short
__attribute__((ext_vector_type(16)))' with an lvalue of type 'const
bf16x16_t' (vector of 16 '__bf16' values)
short and __bf16 are both 16-bit and the value is only ever passed to
the builtin via reinterpret_cast over raw bytes, so changing the element
type is bit-identical. The operand type has been short since the builtin
was introduced, so this works across LLVM/ROCm versions without a
version gate. Fixes HIP builds for RDNA3 (gfx1150/1151) and RDNA4
(gfx1200/1201).
Signed-off-by: Pablo Cholaky <waltercool@slash.cl>
* CUDA: fix build after chunked CUDA graphs (#30804)
* Apply suggestion from @JohannesGaessler
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Fix newlines in common.cuh
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* qwen4exp : build the QSA selection without the kq_mask
* clean
* lang
* qwen4exp : reshape the fused QSA selection to the kq_mask streams
ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while
build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream].
With several streams the shape assert failed at graph reserve.
* test the QSA optimization
* reword
* update build_fused_top_k_mask instead of handwiring
* simplify
* keep il at the end for consistency
Run npm audit fix, then bump the remaining vulnerable packages within
their current majors: storybook 10.6.2, vitest 4.1.12, vite 7.3.7,
@modelcontextprotocol/sdk 1.32.1, dompurify 3.4.17, mermaid 11.17.2,
@vite-pwa/assets-generator 1.0.4 and @tailwindcss/typography 0.5.20.
Override katex to 0.18.11 and postcss-selector-parser to 7.1.6 since
their dependents still pin vulnerable ranges, bump the sharp override
to 0.35.5, and pin @storybook/svelte so npm does not resolve the 11.0
alpha through the addon-svelte-csf peer range.
Storybook stays on 10.6.2 as 10.6.3 is newer than min-release-age.
Assisted by: Claude (Opus 5.5 Medium)
* hex-fusion: support for GELU+MUL, REPEAT+GET_ROWS and BINARY_NX (back-to-back ADD/MUL/...)
* hex-fusion: factor out fusion logic into separate .h and .cpp and update all related code
* hex-fusion: allow for RMS_NORM+MUL fusions for non-contig tensors
* hex-fusion: streamline fusion eligibility checks and cover missing corner cases
* hex-fusion: streamline fusion flow
* hex-opbatch: make n_tensors limit more scalable
* hex-bufs: adjust the trace buff sizing
* hex-glu: cleanup glu-ops interfaces and add kernel_params
* hex-kparams: clean up kparams handling in unary, ssm-conv, and other ops
* hex-kparams: update the rest of the ops PAD,DIAG,... to use kparams
* hex-unary: add support for collapsing dims
* hex-ops: cleanup redundant runtime checks
* hex-ops: consistent use of kernel_type in all kparams
* hex-fusion: remove redundant check in gdn+cpy
* hex-fusion: further streamlining of fusion rules and code
* hex-fusion: cleanup naming
* hex-opfmt: cleanup op formatting
* hex-fusion: bump fusion lookahead to 64
* hex-fusion: fuse SSM_CONV+SILU
* hex-fusion: fix hvx register spills in fused ssm_conv+silu
* hex-fusion: add more GLU fusions
* hex-fusion: remove duplicate checks and use ggml-common helpers
* hex-kparams: refactor kparams into htp-kparams.h/cpp
* hex-fusion: fix binary-nx fusion regression
* hex-kparams: another pass at cleaning up kparams (removing unused, etc)
* hex-kparams: streamling nameing and move _check_ wrappers into .h
* hex-kparams: templetize htp_kparams_check_<op> to reduce code bloat
* hex-ops: de-duplicate op dispatch switches
* hex-kparams: trim down exposed kparams calls
* hex-kparams: simplify type casting of kparams structs
* hex-fence: de-dup fence tensor construction
* hex-opbatch: simplify tensor shape init/checks
* hex-mm: cleanup leftovers of the original QK defines
* hex-fa: fix stack overflow on older npu firmware
* hex-ctx: move dma_caches into ops_ctx
* server: realtime API: send complete after committed audio drains
* server: realtime API: complete a commit at its end row
Completion is triggered when the stream runs dry, which is a timing
proxy: audio appended while the tail drains leaks into the committed
item, a final commit before the drain merges both items, and back to
back commits collapse into one.
mtmd_audio_stream_flush() records the row count at the end of the
flushed audio (mtmd_audio_stream_get_n_rows_flushed), next to the rows
produced so far (mtmd_audio_stream_get_n_rows). The slot keeps a FIFO
of these end rows and completes each commit once every row up to it is
decoded.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* opencl: add Adreno depthwise conv2d
Assisted-by: OpenCode
* opencl: use local program for depthwise conv2d
Assisted-by: OpenCode
* opencl: use singed type for args
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
qwen4exp published the hc-wide residual [n_embd, hc, n_tokens] as
t_layer_inp, but the drafter reads layer inputs as [n_embd, n_tokens]
rows: llama_context sizes each embd_layer_inp buffer as n_embd * n_batch
and the DFlash encoder input is n_extract * n_embd wide. Evaluating a
DFlash draft over a qwen4exp target therefore aborts on the first request
in llama_context::extract_layer_inputs.
Collapse the streams with a plain mean before exposing them, as deepseek4
already does for its hyper-connection residual, and build the mean only
for the layers a drafter actually requested.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* Multi-column on Intel XMX matrix engines refactored
Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).
* Fixed regression bug
real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4
* Removed GGML_TYPE_Q8_0 activation
* rewritten xmx_mul_mat
64-byte alignment and 16-byte pitch check on the activations
* Implemented fused gate/up kernel
One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.
* Removed the temporary XMX_NOGLU environment switch
* Q2_K min-sum optimization
Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group
* Added dispatch check and GRF sweep
tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.
* Renamed variables
* Added 2 environment variables
GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.
* Update esimd.hpp
1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.
2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.
The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
* nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type
* gguf-py : write Metaspace prepend_scheme as GGUF metadata
* vocab : parse metaspace prepend_scheme into enum, rename key
* vocab : use metaspace prepend_scheme in detokenize, add TODOs for other tokenizers
* vocab : use constant for metaspace escaped space
---------
Co-authored-by: Martin Cimmino <martin.cimmino@domyn.com>
* server : support typed content input for rerank endpoint
Assisted-by: Opencode/Qwen3.8 27B
* server : address rerank multimodal review findings
- reject multimodal rerank input when the model has no rerank chat template (images were silently dropped)
- translate handle_media download/base64 failures to 400 instead of 500
- reject negative top_n with 400 instead of letting resize throw
- fix comment typos, redundant comments, indentation and trailing whitespace
- README: document that TEI input accepts the same shape as Jina; drop stray blank line
Assisted-by: opencode
* Trimmed README
---------
Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
* sycl: fix CONV_2D wrong results for non-contiguous input
The kernel indexes the `input` as a contiguous [W, H, C, N] tensor, but
supports_op claimed the op for any layout. A non-contiguous (e.g. `CWHN`
permuted) input was read with the wrong element offsets, silently
producing wrong results.
Gate the CONV_2D op on `input` and `kernel` contiguity, matching the
CUDA backend contract, and assert input contiguity in the op.
`test-backend-ops -b SYCL0 -o CONV_2D`: goes from 16 failures to 0,
none.
```text
2006/2006 tests passed
Backend SYCL0: OK
Backend 2/2: CPU
Skipping
2/2 backends passed
OK
```
Assisted-by: pi (unsloth/Qwen3.8-27B-GGUF:Q4_K_M via llama.cpp)
* sycl : CONV_2D cwhn=1 support
Make the SYCL backend compute `GGML_OP_CONV_2D` correctly for `cwhn=1` (CWHN
channel-order) inputs and kernels, so the previously failing test-
backend-ops CONV_2D cases pass on SYCL instead of being skipped or
miscomputed.
- Implement CWHN layout support in `ggml/src/ggml-sycl/conv2d.cpp`
(`ggml_sycl_op_conv2d`).
- Update SYCL `supports_op` for `GGML_OP_CONV_2D` in
`ggml/src/ggml-sycl/ggml-sycl.cpp` to accept both layouts.
Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF via llama.cpp)
* sycl docs : update ops for SYCL
* feat(ui): add Show tool call in progress display setting
Add the showToolCallInProgress key and its checkbox to Settings > Display,
right next to its "always show" counterpart. Upstream dropped a setting of
this exact name in 43b5e63589, folding it into alwaysShowToolCallContent
because the old name promised expansion that outlives the call; this brings
back the narrower, in-progress-only meaning.
The entry is registry-driven, so SETTING_CONFIG_DEFAULT, the dialog field and
the ui_settings admin baseline pick it up without extra wiring. It ships
enabled, mirroring Show thought in progress, and the always-show help text is
reworded so the two toggles read as a pair rather than contradicting each
other.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)
* feat(ui): expand tool call blocks while they are in progress
Drive the default expansion of TOOL_CALL_STREAMING and TOOL_CALL_PENDING from
showToolCallInProgress and leave TOOL_CALL, the terminal state, to
alwaysShowToolCallContent. A finished call now folds back on its own once its
result lands, which is exactly how the reasoning block behaves when the
thought completes.
Manual toggling keeps precedence over both defaults in either direction:
isExpanded() only consults the settings while expandedStates holds no entry
for the section.
Client tests pin the truth table for the two toggles, including the automatic
fold-back and both manual overrides.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)
* fix(ui): keep running tool calls expanded
A call the agentic loop reports as running now outranks its section type.
exec_shell_command persists an empty result before reading any output, so a
live call is already TOOL_CALL and folded back, which hid the streaming
output the setting is meant to show.
Move the expansion default to isSectionExpandedByDefault so the running-id
rule is covered by a truth table, and test the pending -> finished transition
instead of only the finished state: a block that only ever got mounted
finished cannot prove it folds back on its own.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)
* feat(ui): stream the exec command from the first character
The exec parser read its args strictly, so while the model was still
typing the call the block title stayed empty and the body claimed
"Running...". Parse the args partially (the read_file / edit_file /
write_file pattern), keep a terminal cursor while the args JSON is
incomplete, and stand the conversation cwd in for the x-tool-cwd that
only arrives with the result message, so the prompt is there from the
first character. The body stays empty while the args stream, shows
"Executing command..." once the call is executing, and the wrapper
drops its padding for empty phases so the title sits flush with the
frame.
Assisted by: Delta (zai-org/GLM-5.3)
* fix(ui): stream file tool bodies even before a path arrives
write_file and edit_file refused to render until their args carried a
path, so a call that streams its body first (or a same-named tool with
other arg names) showed an empty block. Meta now carries optional
fileName/filePath, path alias reading moves to shared helpers, and the
blocks fall back to the raw args JSON when no known content key is
present.
Assisted by: Delta (zai-org/GLM-5.3)
* fix(ui): keep an empty write_file body instead of the raw args
A call carrying {"content":""} fell through to the raw-args fallback, so
a finished write of an empty file showed the args JSON. The parser now
leaves content undefined when the key is missing and the block tests
!== undefined, so an empty write renders as an empty code block with its
byte count.
Assisted by: DeepSeek-V4.1-Flash (deepseek-ai)
Two quants of one repo pulled each other's draft: the badge read every
--model-draft arg of the repo, so each row showed both drafts.
Assisted-by: DeepSeek-V4.1-Flash-1017
* opencl: add F32-to-I32 ARGMAX
Assisted-by: OpenCode
* opencl: init to 0 to be consistent with cpu
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* Add Flint model (FlintForCausalLM). Assisted by Codex
* cleanup
* one more
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : add SenseNova U1 understanding support
Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.
- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
tensors and concatenating temporal and spatial Q/K normalization
weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.
Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.
* conversion : treat one-expert SenseNova U1 configs as dense
* mtmd : add SenseNova U1 vision support
Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.
The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.
* mtmd : fix SenseNova U1 vision token layout
* model : clean up SenseNova U1 RoPE metadata
Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.
Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].
* mtmd : simplify SenseNova U1 projector integration
Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.
Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.
* mtmd : fix SenseNova U1 image positions and attention
* fix spacing
* mtmd : restore generic test input chunks
* mtmd : restore SenseNova U1 integration after upstream merge
* Update conversion/sensenova_u1.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : remove SenseNova U1 BF16 activation casts
Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.
Assisted-by: Codex
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama: create new fused op for QSA scoring
* enroll glm5.3-next too
* tests : initialize the remaining tensors in test_kpool_mask
The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check.
* ggml: generalize the k-pool mask op into ggml_top_k_mask
The k-pool mask op only covered the pooled QSA selection of qwen4exp
and glm5-next, while the DSA models and the DeepSeek V4 CSA build the
same top-k mask by hand with fill, set_rows and add.
ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask
cells of the live picks and the extra cells, and sets the rest to
-inf. Every row tensor is 4D in the layout of top_k, and the mask is
broadcast to those rows, so a selection can also be made per head.
score, blocks and extra are optional: without score every pick is
live, without blocks a pick is a cell, and extra holds the cells a
row keeps without a pick.
llm_graph_context::build_fused_top_k_mask views the row tensors in
the layout of kq_mask and registers the fused node. The k-pool
models, the DSA attention of build_attn and the DeepSeek V4 CSA mask
use it when the probe enables the op, and keep their current path on
backends without it.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ggml-opencl : add IM2COL_3D support
Assisted-by: OpenCode
* opencl: use same type for kernel and host
* opencl: fix for older adreno
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* server : forward llama.cpp sampling params on /v1/messages
* server : pass the request through on /v1/messages instead of copying listed keys
* NIT: Change comment from multiline to single line
* server : accept llama.cpp params under extra_params on /v1/messages
* server : remove test case for trivial functionality
* server : show extra_params in the /v1/messages README example
* ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table
* ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path
* ggml-cpu : add the VXE gelu_quick path
* tests : add test-gelu-quick for the f32 CPU kernels
* tests : drop inf/nan inputs from test-gelu-quick
* llama : scale moe down projection per expert token
Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. Undo the scale on the output with the same exact power-of-two factor. Handle POOL_1D in the Meta split-state logic so a split along axis 0 becomes a replicated per-device partial result, enabling tensor parallelism.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : simplify comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : refactor amax src1 scaling result
Return the rescaled activation and the optional scale tensor through a small struct instead of an output pointer.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : add TODO + clean-up
* mtmd: add audio out stream api
* add /tts endpoint
* add docs
* wire up mtmd_helper_model_can_chat
* upload speaker_ref via form-data
* expose seed to api
* fix header for pcm
* wip
* fix penalty_last_n
* fix server_output_limits
* fix pocket-tts stream
* wip
* add server_schema for tts
* add server routing
* server: coerce seed and stream from multipart form fields
Form fields arrive as strings, so -F seed=42 and -F stream=true were
rejected by the schema. seed joins the integer fields, parsed as a long
long so a full uint32 seed fits, and stream is read as a boolean when
it comes as a string.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server: add realtime transcription API
* drop transcription_sessions
* proxy ws
* add docs
* server: let the API key middleware guard the websocket
httplib runs the pre-routing handler before the websocket upgrade, so
the API key middleware answers 401 at handshake like any other route.
This covers the router proxy, whose children run keyless, and accepts
the same key forms as HTTP (Bearer, raw key, X-Api-Key).
* server: keep realtime slots out of the prompt loop
A realtime slot waiting for audio is GENERATING with nothing to decode,
so it fell through the prompt loop and took slot_batched. A request on
a higher slot then failed can_batch_with on the task type and starved
for the whole session. Realtime slots have no prompt: the loop sets
them up once and always leaves them.
* server: start the realtime session once its slot is live
The audio stream was created in the prompt loop, which skips a slot
that cannot batch with the one generating, and session.created went out
before the task was posted: early appends failed and the reader quit,
leaving the socket open and the slot held until the HTTP timeout.
The slot now sets up its stream at launch and signals it with the begin
result, session.created waits for it, and a reader that stops closes
the socket so the session ends with it.
* server: name the committed item in input_audio_buffer.committed
Per the Realtime spec, item_id is the item made from the committed audio
and previous_item_id the one committed before it, null for the first.
committed now comes before the transcript that completes that item, and
both go out with the item switch under one lock, so a delta cannot land
between them.
* server: use the GA transcription session shape
The session object follows the GA Realtime schema: type transcription,
with the input format, transcription model, turn detection and noise
reduction under audio.input. The rate stays the model sample rate, which
is what the client must send for now.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama : move LFM2 shortconv and FFN blocks into a shared LFM base
* Update src/models/lfm-base.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* return logprobs in spec path
* server: add regression test for logprobs with speculative decoding
Asserts accepted draft tokens report the target model's logprobs,
comparing a draft-enabled run against a no-draft baseline (#24271).
Fails on master (top_logprobs empty, logprob 0.0), passes with the
fix from #27196.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KARBWpPcijcBXzJEvttuwk
---------
Co-authored-by: Jeonghyun Lee <sig.jhl@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* mtmd : lfm2 tiles stretch the image to the grid
Lfm2VlImageProcessor.crop_image_to_patches resizes the image straight to
(512 * rows, 512 * cols) before it cuts the tiles. llama.cpp kept the aspect
ratio and added black bars for LFM2 (only d1-omni stretched), so the tiles of
d1-3B and every LFM2-VL / LFM2.5-VL model differed from transformers unless
the image had the aspect ratio of the grid. With PAD_NONE the tiles are the
same as in transformers.
* mtmd : lfm2 layout with Python rounding in double precision
Lfm2VlImageProcessor.smart_resize rounds with Python's round (half to even)
and computes beta and the tile grid aspect ratio in double. llama.cpp used
std::round and float, so 31926 of the 16.7M sizes in [1,4096]^2 got another
thumbnail size or grid than transformers (and d1-omni vision.py), e.g.
400x600 -> 416x608 instead of 384x608 and 720x1600 -> 2x5 tiles instead of
2x4. The lfm2 smart resize and grid choice now follow the Python code and
match it on every size of that range.
* fix
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add H2O-Lightning-4B decision model
Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b
(Qwen3.5-4B with a merged decision LoRA).
None of the existing types match this model. It answers with " A"
rather than "A", uses up to 255 labels, and was trained on Python
json.dumps text, so floats like 12.0 must render as 12.0.
The converter writes the decision metadata and the systemone template
from the model's serve_config.json and refuses a config that doesn't
match. Tests skip unless the tiny test model is available locally.
Assisted-by: Claude Opus 5.5
* rm redundant code
* put labels to gguf
---------
Co-authored-by: Jonathan C. McKinney <jon.mckinney@h2o.ai>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation
* tests: satisfy router disconnect type check
* server : remove timing-sensitive router cancellation test
Assisted-by: Codex
* server: include output_index in Responses API SSE events
* server: reuse Anthropic streaming code for fix output_index
* server: remove unnecessary generated_tool_call_ids and oai_resp_tool_calls_started
* mtmd : read LFM2 tiling params from GGUF metadata
Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from
processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp
with the previous constants as defaults. do_image_splitting=false maps to
min_tiles=max_tiles=1 (no tiling).
Rebased version of #25524, original implementation by Michael DM (@michael-dm).
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* fix test
---------
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* metal : optimize argmax and pool_1d
Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : add pool_1d perf cases
Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : minor fix
* model : drop the redundant precision overrides in the Kolibri-1 router
Follow-up to the review in #30673: the router matmul works with the
default precision, so both ggml_prec_set_* calls can go away.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* cont : fix comment
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* opencl: fix OOB read in `mul_mv_mxfp4_f32_flat`
* opencl: remove large-N guard that forces CPU weight placement
* this condition claims that large batch produces incorrect results on
adreno and cannot be reproduced
* model loader uses N512 to probe op support, resulting in cpu fallback
* thus this condition does not seem necessary
* opencl: cap num of subgroups by workgroup size for q4_0 mc3
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add small batch variant for q4_k dp4a GEMM
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a narrow variant of mul_mm_f16_f32_l4_lm
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a one subgroup variant for gemv_noshuffle_q1_0_f32 for big M
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
---------
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
The __ARM_NEON branch defines both GGML_CPU_COMPUTE_* macros but selects
only FP16_TO_FP32, so FP32_TO_FP16 falls through to the portable
software conversion while neon_compute_fp32_to_fp16, a single fcvt,
stays defined and referenced by nothing. __POWER9_VECTOR__ and
__riscv_zfhmin select both.
Add an #error pair that refuses a branch defining a COMPUTE macro
without selecting it. That is how this stayed unnoticed: the fallback
silently supplies a correct software implementation, so the only
symptom is a hardware instruction nothing ever reaches. The
FP16_TO_FP32 half carries a !defined(__F16C__) carve-out, because PR
30343 keeps that direction on the lookup table on purpose.
Replace the comment above the fallback: it singled out NEON and POWER9
and pointed at a selection that now happens in the branch itself.
* spec : save the dynamic ngram cache file
* save the cache at the common_speculative_impl_ngram_cache destruction
* remove save_static/save_dynamic flags as well as the relevant TODO
* if the dynamic cache file does not exist, start with an empty cache
that is written on exit. if it exists but cannot be read, abort
instead of overwriting it.
* spec : fix draft_one not updating the ngram cache history correctly
* common_ngram_cache_update received only the new tokens with none of
the tokens before them causing ngrams ending at the new token to not
being formed, leading the context cache to only ever hold the prompt
* fix: pass the full history to common_ngram_cache_update with nnew set
to the number of new tokens the way llama-lookup does and also reset
cache_size to 0 on each new request
---------
Co-authored-by: Petros Sideris <petros.sideris@nokia.com>
* server : pass id_slot through the Anthropic /v1/messages conversion
server_chat_convert_anthropic_to_oai copies a small whitelist of common
params from the Anthropic request into the internal OAI-style body and
left id_slot out. Every other completion endpoint honours id_slot; a
proxy that pins a session to the slot holding its KV cache needs the
same on /v1/messages, or it silently falls back to automatic slot
selection and, on hybrid/recurrent models, pays a full re-prefill.
Adds a test asserting a request with id_slot set is served by that
slot, observed via GET /slots.
* server : shorten comments
* rm test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* tests : make MoE router selection independent of the backend noise
The router scores are in the range [0, 1), so a random draw can leave the
last expert (or expert group) still selected and the first one dropped tied
within the noise between the backends. They then route a token to different
experts and the logits differ by 4e-3 instead of the usual 5e-7.
Beam is the only arch here with a real top-k (8 experts, 2 used) and a group
top-k (4 groups, 2 used), so it is the only one that hits this: 2 of 20 seeds
failed on an M2 Ultra, the other MoE archs select 2 of 2 and cannot tie.
Fill the selection bias with a ramp of step 2 to give every expert a score
range of its own, which turns the top-k into a function of the expert index
alone. The bias affects only the selection, the weights still come from the
unbiased probs, so the MoE output keeps its magnitude. Beam now matches the
CPU reference within 4e-7 for all of 160 seeds.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : tell the CPU and the GPU of an Apple SoC apart in the device table
Both devices report the same description ("Apple M2 Ultra" here), so the
device column of the test table had two indistinguishable rows and a failing
row did not say whether the Metal device or the host CPU produced it.
Add the device name, which is also what -b takes, unless the description
already is the name.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* hex-build: bump toolchain, enable v85
* hex-build: bump toolchain to v0.8 in the CI and build scripts
* hex-build: add a little python script for swiv needed on some iot devices
* hex-build: fix flake8 warnings
* hex-arch: update version cap to v85
* hex-inspect: flexible TOOLS_ROOT
* hex-build: update to snake_case naming
* hex-build: update leftover old toolchain refs
* hex-target: update build and run scripts to support ubuntu with adb transport
SearchInput already renders its own clear button.
Hide the browser's native search cancel button to avoid showing both controls.
Fixes#28627
Signed-off-by: aiqubits <aiqubits@hotmail.com>
* init
* wip
* wire it up
* make it clear that it's experimental
* add props.connect_code
* nits
* fix windows build
* server: do not advertise a connect code without the tunnel
--connect-code on its own had no effect other than being served in
/props, so the Web UI showed a remote access code that no tunnel was
serving. Clear it when --connect is off, and say so once at startup.
The teardown comment claimed that terminate() closes the child stdout.
It does not: the log thread leaves fgets() only once every writer of
that pipe is gone, so a descendant of llama-connect would block the
join forever. Describe the real contract instead.
Also restore the --reasoning-preserve default in the generated docs.
* Update tools/connect/CMakeLists.txt
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
* server: dial llama-connect on the bound port and a reachable host
Pass ctx_http.port so --port 0 and the GCP override work, pick the
dial host from --host (wildcards map to loopback, IPv6 is bracketed)
and refuse --connect when only unix sockets are bound.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
In the attachment preview, SyntaxHighlightedCode's wrapper (overflow-auto) was not the element that ended up owning horizontal overflow: the inner code box kept its content clipped at the wrapper's width, so a single line longer than the preview pane was cut off with no scrollbar (scrollWidth == clientWidth while the code's own content measured tens of thousands of px).
Give the pre/code boxes their intrinsic width (width: max-content, min-width: 100%) so long-line overflow propagates to the scrolling wrapper. Verified in the running WebUI: with a 6000-char line, wrapper scrollWidth goes from 990 (clipped, unscrollable) to 50449 with a working horizontal scrollbar; short content still fills the container exactly as before (min-width: 100%).
Fixes#30796
The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a
single dispatch, reading experts and weights while writing dst. The graph
allocator computes tensor liveness on the unfused graph, where the inputs are
fully consumed by earlier nodes of the fused group, so it can legally recycle
the input memory as dst. In the fused kernel this becomes a
threadgroup-vs-threadgroup race that produces rare non-deterministic wrong
results (observed with test-llama-archs -a kolibri on the meta backend).
Reject the fusion when dst overlaps either input, same guard as the
mul_mat_add fusion. The unfused MUL + ADD path is always correct.
Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
* cuda: cap Stream-K blocks at the number of k-blocks
Stream-K normally launches one CUDA block per SM. For small matrices,
this can exceed the total number of output-tile/k-block pairs, leaving
blocks with no work and adding launch and fixup overhead. Cap the grid
at that count to remove these excess blocks without changing the useful
rounded K ranges or numerical results.
Assisted-by: Codex
* cuda: cap Stream-K blocks at the number of k iterations
Cap the grid at the number of K_vram iterations across all output tiles, rounding up partial iterations. This removes extra CUDA blocks left by the k-block cap.
Partial iterations can change FP32 summation grouping. MUL_MAT and MUL_MAT_ID backend tests pass on RTX 5090.
* Update ggml/src/ggml-cuda/mmq.cuh
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ui : keep markdown tables inside the chat column on phones
The phone rule made the table wrapper full-bleed with -1rem margins, so the
first and the last column sat under the screen edge. Keep the wrapper inside the
chat column and cap it at its width, so a wide table scrolls in place instead.
Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* test : show a table-only model answer in the chat message story
Adds a story with an answer that is mostly a table, so the table layout can be
checked at a phone width.
Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* ui : indent markdown lists by 2rem
1.5rem put the marker too close to the item text.
Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* The problem here in the repo is gguf-split can silently produce corrupted tensor output when the input GGUF file is truncated. The fix is to add tensor-data bounds checks and short-read checks in gguf-split.cpp and export-lora.cpp, and a regression test that checks truncated input is rejected and valid tensor data is preserved.#
* Bug : GGUF-split can silently produce corrupted tensor output when the input GGUF file is truncated. Fix : add short-read checks in gguf-split.cpp (in copy_file_to_file in split_strategy) and export-lora.cpp (in read_tensor_data of file_input) so that truncated input is rejected.
* tests: add beam fusion rows to MTL baseline
Regenerate tests/fusion/MTL.csv with test-fusion --record. The newly
added beam architecture has no baseline rows, so test-fusion fails with
2 failing rows (MUL+ADD x6, RMS_NORM+MUL x25). All other arch counts are
unchanged.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* skills: add update-fusion-table skill
Add a skill for regenerating a per-device test-fusion baseline. It keeps
the workflow from tests/fusion/README.md and the parts that are easy to
get wrong: diff the recorded file before applying it, the distinct
meanings of a FAIL row, and never recording a partial run.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
.pi/gg/SYSTEM.md repeated the full agent ruleset. Move it to .pi/gg/APPEND_SYSTEM.md and keep only the llama.cpp-specific rules, so it can be appended to a main system prompt with --append-system-prompt.
Drop the .pi/SYSTEM.md symlink setup from wc2wt.sh and pr2wt.sh. Worktrees no longer need it now that the folder holds an addendum instead of a full prompt.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* CUDA: fix KQ write-after-read race in flash_attn_ext_vec
Assisted-by: ChatGPT
* CUDA: apply review suggestions for KQ sync comments
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* hex-fa: support for masked block skipping to reduce DMA fetching
* hex-fa: finetuning the hmx->hvx crossover thresholds (boosting gemma-4)
* hex-fa: update hmx gate to account for mdev.count
Port of the Vulkan HC_POST_GATE fusion (#29520). hc_combine builds the
hc_post weights as 2*sigmoid(inject/hc); dsv4_hc_post_f32 now takes a
gated template flag and computes scale_out*sigmoid(scale_in*p) from the
raw weights, so the chain runs as one kernel instead of four.
* ggml: support non-contiguous SUM (CPU, CUDA)
* tests: add non-contiguous SUM cases
* tests: SUM over non-contiguous views
* vulkan: support non-contiguous SUM
* webgpu: support non-contiguous SUM
* hex-hadamard: add support for hadamard ops
* hex-mm: reduce pkt counts in the q8_1 quantizer and vec_dot_q1
* hex-q1/q2: add support for flat and tiled GET_ROWS for Q1_0 and Q2_0
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* common : resolve <quant>-<sidecar> download tags and list cached sidecars
A Q4_0-mtp style tag now resolves the sidecar file when no model file matches it, so a solo draft or mmproj download actually pulls the file. Cached sidecar files list as their own entries so the state survives a restart, and removing such a tag deletes only the sidecar.
Assisted-by: pi:zai-org/GLM-5.3
* ui : optional sanitized raw HTML in markdown
Add an allowHtml prop to MarkdownContent: raw HTML found in the markdown is
rendered after DOMPurify sanitization instead of being escaped as literal
text. Default stays escaped.
Assisted-by: pi:GLM-5.3-Flash
* ui : add the discover models components
The Discover surface: the curated-catalog list with its search and
skeletons, the details pane with its readme, metadata, Hub stats and
download options, and the standalone download progress bar that replaces
the plain one. The quant download button sizes its select with a new xs
trigger, so the select gains that size and the muted box look. The store
stays incomplete when a repo fails, so the next mount retries it.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : put the discover view in the models manager
The manager dialog gains its second view: the Discover Models call to
action fades the table out, slides the title with its compass mark, and
fades the discover surface in; the arrow button returns to the table.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
# Conflicts:
# tools/ui/src/lib/components/ui/button/button.svelte
* ui : add the Enable Discover Models setting
The flag gates browsing and downloading HuggingFace GGUF models, so it is
declared on the discover layer; the repo-tree lookup in the manager layer
above reads it from here.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : resume model downloads from the pause point
A stopped download keeps its partial file instead of deleting it, so
re-posting the tag resumes from disk. The first progress record now
carries the resume offset, the paused snapshot survives into the
resumed entry, and a pause settles as soon as it is requested.
Assisted-by: pi
* ui : lead the discover list with downloads
The discover sidebar lists running downloads above the suggested models,
and steps aside while a query is active. A paused download stays in the
manager table, where it is resumed or dropped.
Assisted-by: pi
* ui : make resuming a paused download work
Re-posting the tag of a paused download hit the router's "already
exists" check, because a paused download parks its entry as DOWNLOADED
until the next reload: a download entry that is not running is now
replaced and resumed. The UI keeps the paused row when the request still
fails, instead of dropping it and looking finished.
Assisted-by: pi
* ui : remember unfinished model downloads
A partial file is named after its content hash, so nothing on disk says
which model it belongs to once the router restarts. The router now keeps
the tags of the downloads it started, clears a tag when the download
finishes or is deleted, and lists what is left as still downloading
without starting anything. The UI adopts those tags as paused on every
list fetch, so a refresh or a fresh open shows an incomplete download and
can resume it from the files already on disk.
Assisted-by: pi
* ui : show a paused download's state on its chip
A paused quant chip reads as paused at rest, the way a running one shows
its spinner, and offers the resume action on hover. The selector's
download row gets a View in Discover action beside its cancel.
Assisted-by: pi
* ui : collapse the download options by default
The copyable download command starts collapsed so the option rows
lead; expanding it stays one click away.
Assisted-by: pi
* ui : keep decision models out of the hub lists
Decision models answer with calibrated probabilities instead of text,
so llama-ui cannot run them: the hub data layer drops them from the
catalog and every search, keyed by the decision-model tag or a
text-classification pipeline.
Assisted-by: pi
* ui : read download tags as quant and sidecar
A `<repo>:<quant>-<sidecar>` tag now parses into its quant and sidecar
instead of one opaque quant token, and ModelsService.draftSidecarsByRepo
collects the router listing's sidecar-only entries per repo, replacing
the inline copy in the models store.
Assisted-by: pi
* ui : cluster draft downloads under their main model
Tracked downloads group by repo: the main weights lead and the draft
sidecars ride inside their item as compact sub-lines, on the selector,
the discover list and the manager table. The main model's controls
pause, resume and cancel the whole cluster, and the manager's download
rows select into the configuration pane.
Assisted-by: pi
* ui : act on tracked downloads from the table rows
The table passes its clusters' draft sub-lines to the model rows. A
row's actions cascade pause and resume, an in-flight row offers Cancel
download while a paused one keeps Delete from disk, and the confirm
dialog's copy follows the state.
Assisted-by: pi
* ui : give downloads a configuration pane of their own
Selecting a tracked download opens the pane in a download mode: live
progress with pause, resume and cancel, and See in Discover, instead of
metadata and load actions that a model without files cannot offer.
Assisted-by: pi
* ui : make the discover details self-fetching and responsive
ModelsDiscoverDetailsContainer owns the details, tree and readme
fetches so hosts outside the discover view can render the pane, and the
details layout adapts to narrow panes with container queries instead of
viewport breakpoints.
Assisted-by: pi
* ui : download a matching draft with the main model
A main quant download now pulls the quant-matched draft of the most
preferred kind the repo ships (DFlash before MTP), fire-and-forget, so
speculative decoding is ready when the main lands. Manual draft chips
stay available; the options row also collapses to one column in narrow
panes.
Assisted-by: pi
* ui : keep discover off by default and couple it with the hub setting
Discover Models starts unchecked. Enabling it turns the Hugging Face
Hub setting on and disabling it turns it back off, so the checkbox
never lies; with discover off the hub setting stays available on its
own for the display data. Stored and reset state are corrected the same
way on load.
Assisted-by: pi
* ui : open the manage models dialog on the manager view
The dialog now opens on the manager instead of returning to the last
view, discover entries still land on the discover view while the
setting allows it, and the discover surfaces follow the setting's
state.
Assisted-by: pi
* ui : give the discover search room for its focus ring
The sticky search bar insets from the panel edges so the focus ring
stays inside the scroll container, and rows scroll behind an opaque bar
instead of through it.
Assisted-by: pi
* common: settle sidecar names on the listing and drop unfinished partials
An infix or trailing sidecar token is ambiguous: draft heads use it
(Model-MTP-BF16.gguf), so do full models with embedded heads
(Model-Q8D-MTP.gguf). Such a name is a sidecar only next to token-free
main weights of the same family, while mmproj or imatrix anywhere in the
name, or a leading token, decides alone (mmproj_f16.gguf).
A quant tag always carries its bit count, so a sidecar named after no
quant (mtp-Model-it.gguf) gets the bare tag: the listing, the plan and
remove all use it. An imatrix tag resolves into a plan slot of its own,
downloaded but never wired into the params.
remove resolves the tag the way a download does and deletes the partial
files of every planned blob, so cancelling a running download or
dropping one left unfinished by an earlier run leaves nothing on disk.
The router passes its token and releases its lock around the call, and
records the unfinished downloads with an atomic write.
llama download succeeds on a sidecar-only plan and prints its paths.
* ui : move dialog views and draft states into enums and constants
The models dialog's views become the ModelsDialogView enum, the draft
subline's states join ModelRowDownloadState, the view fade and the
copy-feedback reset move to ui constants, and the dialog's compass
marks come from DISCOVER_ICON.
Assisted-by: pi
* ui : drop the unused discover surface
The dead openDiscoverModels twin, the commented-out command divider,
the details pane's unused onClose and its header's close button, and a
redundant type cast leave; the command's copy feedback joins the shared
reset constant.
Assisted-by: pi
* ui : trim the download option comments and highlight the chat template
The comment verbosity the options panel and its chips carried shrinks
to the non-obvious why, and the chat template dialog renders through
SyntaxHighlightedCode as jinja instead of a plain pre block.
Assisted-by: pi
* common: only a .gguf file is a sidecar, and mmproj/imatrix as whole words
A non-GGUF file named like a sidecar (quantization-mmproj-Q8_0.json) is
picked as the mmproj, and a sidecar word inside a longer word of a model
name (Phi-2-OmniMatrix.i1-Q4_K_M.gguf) drops the model.
* ui: dedupe draft sidecar badges per quant
The draft sidecar badges deduped by kind and repo without the quant, so
with a draft cached at two quants only one badge survived, and a delete
of the other main quant found no badge to remove its sidecar with.
Also restores the getDownloadEntries doc lost when the clusters method
landed above it.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: adopt the incomplete download status and drop the paused-ids registry
The store inferred an unfinished download from a downloading entry it
held no local state for, which misread a download running in another
tab as paused and offered a resume the server rejects. The listing
carries the "incomplete" status now, so the adoption keys off it
directly.
The paused ids no longer need to survive in localStorage: the server
remembers them across restarts, and the progress snapshot was stale
after a reload anyway. The stale localStorage key is removed.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: quant-match the pause and cancel cascades
Pause, resume and cancel cascaded to every draft sidecar of the repo
while delete only took the quant-matched ones, so canceling a Q4_0
download also killed a Q8_0 draft that belongs to another quant's
main. Every cascade acts on the quant-matched drafts now, matching
what a delete takes with it.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: collapse the duplicated draft subline markup
The downloaded and in-flight branches repeated the identical connector,
label and badge markup; one block with a conditional tail keeps the two
states in sync.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: scrub positioned styles out of sanitized model card html
The style attribute stays allowed for the KaTeX output, which let a
third party card draw a fixed overlay over the app. The sanitize hook
drops position:fixed and position:absolute values and leaves the KaTeX
layout styles alone.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: run the discover mount fetch once and reuse the shared repoOf helper
fetch() toggles its loading flag synchronously inside the mount effect,
which re-ran the fetch on every toggle and retried failed catalog repos
endlessly; the effect now untracks the call. The local repoOf copy and
the duplicate services import give way to the shared helpers.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: narrow the spec-type map to the draft sidecars
SPEC_TYPE carried empty entries for the aux sidecars it is never
indexed with; the map and the command component now type the badge as
a draft sidecar through a draftBadgeOf helper instead of a cast.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: name the decision-model tags and markdown cache keys
The decision-model hub tag and pipeline tag move into the huggingface
constants next to the other pipeline tags, and the markdown transform
cache prefixes get names instead of bare one-letter tokens.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: read the discover setting through one truthiness check
Three places read the setting with !== false and one with Boolean();
all four read it the same way now.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* ui: compare the models dialog view against its enum
The template compared the view state against the 'manage' literal in
three spots while the script used the enum.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* nit: Remove `+` before draft sidecar badge
* server: list unfinished downloads as incomplete
A download that never finished was reported as "downloading", the same
status a live download carries, so a client could not tell a running
download from a dead one. The tags remembered in unfinished-downloads
now list under a dedicated "incomplete" status, without an instance.
Assisted-by: pi (zai-org/GLM-5.3-Flash)
* server: download a model with its sidecars as -hf does
Resolution goes back to master. POST /models takes the sidecar kinds to
pull with the model (mmproj by default) and the download child gets them
as -hf flags, so the server picks the files. /models lists real models
only, each with the sidecars it has on disk, and incomplete downloads
keep their sidecars across restarts. A stopped download keeps its
partial files for the next attempt, and DELETE purges them through the
same plan the download follows.
* ui: download sidecars as parameters of their model
The download options keep their chips, but a chip no longer posts a
sidecar tag: the store maps it to its model and the sidecar kinds to
pull, the quant chip takes its companion draft in the same download,
and draft badges come from the sidecars each cached model lists.
The server adds the sidecars a cached model lacks when it is posted
again, and DELETE removes a single sidecar of a cached model.
* ui : mirror the server's sidecar lookup in draft badges
The server resolves a sidecar by the case-sensitive `<kind>-` token in the same
directory as the weights. The UI matched the token loosely, so a badge could
offer a draft the server never pulls.
Assisted-by: DeepSeek-V4.1-Flash
* ui : post the model's sidecar kinds with its download
The store keeps one download per model: the chip's kinds ride along with the
model's own sidecars, and a download row no longer expands into one row per
sidecar. Draft badges carry no quant of their own yet, so a sidecar of a cached
model is never labelled with the model's quant.
Assisted-by: DeepSeek-V4.1-Flash
* ui : join a quant's draft into its download chip
One chip per quant, its draft badged inside it and its size covering both
files. The draft is the file the server would pick: the same directory, the
exact quant, else the closest bit depth, so a 4-bit model can pair with a
16-bit head.
Assisted-by: DeepSeek-V4.1-Flash
* ui : drop the draft download rows from the manager and selector
A download is one row per model again: the drafts ride with it, and the rows
read the sidecars on disk off the listing instead of pairing sidecar entries
with their model. Pause, resume and cancel act on the model's own download.
Assisted-by: DeepSeek-V4.1-Flash
* ui : keep the discover download list one row per model
The list leads with one row per downloading model, its sidecars as sub-lines.
A row badges only the draft kinds the server can resolve, so a badge never
outlives the file its lookup would pick.
Assisted-by: DeepSeek-V4.1-Flash
* ui : keep the discover chips readable in dark mode
The metadata chips, the chat template button and the View on Hugging Face link
carried a faint border and no surface over the dialog background, so they were
near invisible. They take the tertiary button surface now, and their icons size
with size-* so the variant's base rule leaves them alone.
Assisted-by: DeepSeek-V4.1-Flash
* ui : carry the draft through the download options
One chip per main quant, its draft badged on it and the chip's downloaded state
covering both files. The drafts sit behind an opt-in checkbox with an inline
type pick, and each row opens its own command: it pins the row's quant and only
names the draft type, so -hfd is gone.
Assisted-by: DeepSeek-V4.1-Flash
* ui : line the details skeleton up with the loaded pane
Same root padding and container context as the loaded view, so the container
queries resolve alike. The download options card pads its blocks instead of
itself, the readme gets its card, and a description line stands in for the
catalog paragraph.
Assisted-by: DeepSeek-V4.1-Flash
* common : drop the partials of a removed sidecar
Removing a sidecar alone left the .downloadInProgress files of the sidecars a
download had planned in blobs/. The removal purges them through the same plan
the download follows, and the test covers a finished draft beside an unfinished
one, whose partial goes while the model keeps its files.
Assisted-by: DeepSeek-V4.1-Flash
* server : name why a download is refused
A model that is already downloading, or that is running, answered "already
exists". It now says which: "is already downloading", or "is running, unload it
first", both before the resumable check. The router test covers the two.
Assisted-by: DeepSeek-V4.1-Flash
* ui : post a download once per click
Two quick clicks on the same chip posted twice, and the second answer was
"already exists". The store tracks the model as pending, returns early while it
is pending or in flight, and reads it as in flight before the server answers, so
the chip shows the download at once. The new client test pins the flows: one
POST per double click, the chip reading downloaded once the list reports the
model, a resume keeping its sidecars, a cancel dropping the whole download, and
a reload restoring the paused state.
Assisted-by: DeepSeek-V4.1-Flash
* ui : keep a plain download row the height of the others
The sub-lines wrapper of a download row was rendered even with nothing in it, so
its padding made the row taller than a plain one and the overlaid progress bar
sat in that empty strip. It renders only with sub-lines now, in the selector and
in the discover list, and the selector row takes the phone padding its
neighbours have.
Assisted-by: DeepSeek-V4.1-Flash
* ui : badge a repo's draft once
The args of a loaded model and the listing of a cached one report the same draft
in different words, one with the file's quant and one with the kind alone, so
the merge kept both and a repo row showed the same kind twice. The badges merge
per kind now, the args one winning as it knows the quant, and the test pins both
sources of a real repo.
Assisted-by: DeepSeek-V4.1-Flash
* server : report the sidecars of a preset or a download on its way
A model the preset also names is registered as a preset, so its drafts were
never reported and no row could badge them. A download on its way reported none
either, though the record of the kinds it pulls is right there: those win over
the plan now, the plan resolves through the preset's hf-repo, and a models dir
stays out of it.
Assisted-by: DeepSeek-V4.1-Flash
* ui : badge a download's drafts and open its model from the row
A draft still coming down badges the model id now, the way a landed one does,
while an on-disk draft keeps its checked sub-line. The row itself opens the
model, the status slot beside it pauses or resumes, and the manager resolves a
focus on a download row so its panel opens.
Assisted-by: DeepSeek-V4.1-Flash
* common : report a removed sidecar that only had a partial
Removing a sidecar alone dropped the result of the partial purge, so a sidecar
that only ever landed as a .downloadInProgress blob answered false and the UI
had no reason to refetch. The purge result is folded into the return now, and
the test covers a partial-only removal.
Assisted-by: DeepSeek-V4.1-Flash
* ui : ignore a pause while the download POST is still on its way
The chip reads as downloading before the server answers, so a quick second click
unloaded a model that had not started and left the store neither in flight nor
paused. A pause returns early while the POST is pending, before it touches the
feed or the paused marker. The client test holds the POST open and pins it.
Assisted-by: DeepSeek-V4.1-Flash
* ui : offer a missing draft as its own chip
A model already on disk without its draft had no way to add it: the quant chip
carried the draft, so it read as a download of both. The quant chip keeps to its
own state now, its joined draft shows only while it rides along or is on disk,
and a downloaded quant missing its draft gets a chip of its own - badge, size,
download - that posts the model with that kind, so the server keeps the files
already there and pulls the draft.
Assisted-by: DeepSeek-V4.1-Flash
* server : list a paused sidecar download by the files on disk
A model on disk whose sidecar download stops keeps its unfinished record,
and the listing reported the kinds that record plans as if they were on
disk: the UI checked a draft that never landed and offered no way to add it.
The planned kinds now stand in only while the download runs.
* chore: Clean up comments
* server : list sidecars only for a model that resolves from a HF repo
A preset on a local file has no HF repo, yet the listing resolved its name
as one: the plan threw on the invalid repo format and GET /models failed
for every model. The sidecars now come from the repo a preset names or a
cached model is listed under, and a local file lists without them.
* common : fix download cancel regression
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* common : preserve model partials when removing sidecars
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* server : one sidecar table and an unfinished downloads class
Address review from @ggerganov.
The sidecar kinds live in one table in common/download, mapping each name to
its download option and plan file; the server validates, builds its options
and lists cached sidecars through it instead of its own lists.
The unfinished downloads move into server_unfinished_downloads, a record with
its own mutex and a struct per entry, and server_models exposes them through
unfinished_downloads_get and unfinished_downloads_remove.
* common: one sidecar table, and removals that keep shared files
The resolution, the cached listing and the removal follow a single
table of sidecar kinds, so a kind is declared once with its file
keyword, whether it is a draft, its option and its plan file. The
server reads the same table for the kinds it accepts, the default
sidecars and the child flags.
Removing a tag keeps the files another cached tag still loads, such as
an mmproj named after the removed quant. A sidecar removed on its own
goes for every tag that resolves it.
An unfinished download records the exact paths of its partials when
it starts. A removal deletes those without asking the hub, and keeps a
partial another recorded download also writes. The record is taken
once the child is spawned, and the files go under the models lock, so
a download of the same tag cannot start while they are removed.
* server: drop stale partials and refuse a tag that resolves nothing
Recording a download again deletes the partials of the earlier record
that the new plan no longer writes, such as those of an older hub
revision or of a sidecar the new request leaves out.
A posted tag that resolves no file of the repo is refused before the
download starts. Removing a model follows a single path, whether the
reload erased its entry during the wait or not.
* server: fix the CI race between the router download tests
The pytest workers run in parallel on one shared cache, so a download
test could find the file another test had just fetched and see no
progress, or start from the unfinished downloads record another test
wrote. Each download test now runs on an empty cache of its own.
* server: refuse a DELETE of an unknown name and decide downloads in load()
A DELETE of a name that is neither a model nor a recorded download
went to common_download_remove, and without a tag it wiped the whole
repo cache, including the files of a loaded quant or of a download on
its way. Such a name is now not found, as it was before.
A POST that downloads over an existing entry checked its state outside
the lock, so a load of the same model in between got its instance
killed. The checks now run in load(), under the lock, right before the
spawn. The fake load time of the tests applies to model loads only,
since a download worker loads none.
* server: add incomplete_sidecars to the /models API
A sidecar added to a cached model and then paused kept its record, but
the listing showed only the complete model, so a fresh UI lost the
pause. The model entry now carries incomplete_sidecars, the kinds its
unfinished download has not written yet; a POST with the kind resumes
one and a DELETE with it drops its partial.
* common : drop a kept partial when a resume gets 416
Now that partials survive between attempts, an old or complete
partial makes every retry fail with the same 416.
Delete it so the next attempt can start over from zero.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* common: drop a kept partial when a resume gets 200
A server that ignores the range answers 200 to a resume, and the kept
partial then fails every attempt the same way a 416 does. Drop it as
well, so the next attempt starts over without a range.
* server: refuse a download that shares a partial with one on its way
Two downloads of quants that share a file, the mmproj or an mtp picked
by the closest bits, appended to the same partial and renamed a file
with interleaved bytes. A download whose plan writes a partial that a
running download also writes is now refused until that one is done.
* common : check the saved etag before resuming a partial download
Before resuming, compare the etag saved with the partial to the one
the server now serves. If it is missing or different, start over,
so a resumed download never mixes bytes from two versions of the file.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
* tests: give the random model generator a per-tensor filler table
Muse Spark makes test-llama-context fail on state_load, cp_h and cp_d on about
six percent of the random seeds, with an NMSE of 1.0e-05 to 1.4e-05 against a
threshold of 1.0e-05. The sliding window is what exposes it: a restored state
keeps the window alone, so the attention of an SWA layer reduces over a
different cell layout and the continuation is not bit exact. Every SWA model
carries that, at 1e-08. Muse Spark carried it at 1e-05.
The generator is what turns 1e-08 into 1e-05. A norm preserving residual gate
holds alpha^2 + beta^2 == 1 and the conversion resolves both weights from one
raw weight, while the generator draws every tensor on its own. The pair then
scaled the stream and the branch independently, the stream lost an arbitrary
factor at each layer, and the model became an amplifier. The weights the test
ran on were not a Muse Spark.
Per-tensor rules lived inside the write path, where a flag picked the values of
a tensor inside each of the two branches that store them, so a new rule meant a
new flag threaded through both. The file asks to be rid of that in a TODO of its
own. Give each rule a filler, pick one from a table keyed on a substring of the
tensor name, and keep a single write path behind it. Mamba moves into the table
and Muse Spark joins it: the gate filler draws the raw weight once per block and
resolves alpha and beta the way the conversion does.
Over a thousand seeds on a CPU build the worst case drops from 1.36e-05 to
2.48e-06 and the median from 4.10e-06 to 5.48e-07, with no seed above the
threshold. The other 131 models come out identical to the bit, since no other
architecture carries these tensors. ctest is green on a CPU build and on a CUDA
build.
* tests: draw every tensor filler from fill_normal
The fillers repeated the generator and the distribution each time. The
normal draw now takes its key and the other fillers transform its
values, so the generator lives in one place. The generated models are
identical to the bit.
* fuse grouped experts
* add tests
* hip fix
* add to graph_optimize
* apply suggestion from am17an
* hip fix
* fix hip fix
* fix fix fix
* gate on hip version instead
* model : add Kolibri-1 (Aleph-Alpha) support
Kolibri-1 is a bilingual German/English MoE reasoning model with 78B
parameters and 3.46B active per token: 50 layers, 384 routed experts plus
one shared expert.
Attention interleaves four sliding layers at a 513 window with one full
layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE.
GQA runs 48 query heads against 4 KV heads at head_dim 128 with a
QK-RMSNorm, and every block carries four RMSNorms, so the attention and the
MLP output are normalized before they rejoin the residual stream. Routing
picks the experts from logits + expert_bias and weights them with the
sigmoid of the unbiased logits, without renorm.
The converter reuses AfmoeModel for the per-expert checkpoint layout and
the expert selection bias, and writes the shared expert and the per layer
sliding window pattern. The released FP8 checkpoint needs no special code,
it goes through the generic weight_scale_inv path. Three tensor names are
added to the mapping (post_attn_norm, post_ffn_norm,
moe.router.expert_bias) and the 128k vocabulary registers the qwen2
pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA
and the sliding window pattern.
Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash)
* Update src/models/kolibri1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Add fusion records for kolibri1
* Update src/models/kolibri1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* feat(convert): Attempt to come up with 256 single-token, semantic option markers
Each switch model needs to identify a collection of single-token option
markers that can be mapped to their respective logits for the output
probability distribution. The simple upper/lower/numbers set only yields 66
unique markers, but the model can support up to 255. The challenge is that
which combinations of characters result in single-token markers is model-
specific, so we do a discovery exercise at conversion time by looking
through the specific model's vocab for single-token sequences of 1+ plain
letter strings (eg `A`, `AA`, `BB`, etc).
Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* docs: Note granite-switch with 256 option markers
Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Remove extra newline
Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(convert): Use the computed labels in the template so c++ side is source of truth
Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Use LEV case for GRANITE_SWITCH w/ computed labels
Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* docs: Consolidate g-s into 255 bucket
Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
softplus_f32 and tile_softplus_f32 ran a scalar logf(1.0f + expf(x)) loop.
Replace both with a shared hvx_softplus_f32_aa() helper built from the
backend's existing hvx_vec_exp_f32 / hvx_vec_log_f32 primitives, so no new
numerical approximation is introduced.
hvx_vec_exp_f32 clamps its input to (-88, 88) before exponentiating, so a
naive log(1 + exp(x)) composition would saturate to ~88.72 for large x
instead of returning x. The Q6_V_vmux_QVV select keeps the scalar
reference's x > 20 guard (ggml_compute_softplus_f32 in ggml-impl.h),
which is what makes the vector path match the CPU result.
Assisted-by: Claude
* feat(convert): Add decision.chat to support decision models that can also chat
Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(convert): Add conversion for Granite Switch as decision model
The template follows from the usage sample:
https://huggingface.co/ibm-granite/granite-switch-4.1-8b#guardian-and-decision-side-by-side
Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(common): Add support for chat+decision models in common
Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(common): Add granite switch decision model enum
Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(server): Plumb through decision_chat bool to support decision+chat models
Branch: GraniteSwitchDecision
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(server): Add Granite Switch decision support
Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* docs: Update server README for granite switch decisions / chat+decisions
Branch: GraniteSwitchDecision
AI-usage: draft (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* Revert "feat(convert): Add decision.chat to support decision models that can also chat"
This reverts commit 952ef567647e53ac9beda7a1f79d6927240d068d.
* Revert "feat(common): Add support for chat+decision models in common"
This reverts commit 93ed782e3a8b2fd5968732c6525c7ad29ad4bea2.
* Revert "feat(server): Plumb through decision_chat bool to support decision+chat models"
This reverts commit 97e9af60ddaccbd82f1eb0a488adff43b16ee095.
* fix: If Granite Switch supports decisions, it always supports text too
Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* docs: Remove reference to decision.chat in server README
Branch: GraniteSwitchDecision
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client
common_http_client() was ignoring standard proxy env vars, causing
downloads via -hf and similar flags to fail silently in environments
behind an HTTP proxy (corporate networks, air-gapped infra, etc.).
Checks HTTPS_PROXY/https_proxy for https:// URLs and HTTP_PROXY/http_proxy
for http:// URLs, in that order. If a proxy URL is set and parseable,
passes it to httplib's set_proxy() and set_proxy_basic_auth() as
appropriate. Malformed proxy URLs are silently ignored so existing
callers aren't affected.
Fixes#21694
* common: drop misleading comment about generic proxy fallback
The comment referenced a 'generic one' that doesn't exist in the code — both
HTTP_PROXY and HTTPS_PROXY are scheme-specific. Reviewer noted the mismatch.
Addresses review feedback from @ServeurpersoCom.
* common : warn when the proxy URL from the environment is malformed
* common : honor NO_PROXY when selecting the http proxy
* ci : retrigger checks (test-thread-safety flake)
* common : bypass the proxy for loopback destinations
llama cli talks to its spawned server on 127.0.0.1 through common_http_client, so with HTTP_PROXY set and no NO_PROXY the health polling went to the corporate proxy, which rejects loopback targets, and the wait loop never exits ("Loading model..." hangs forever).
A remote proxy can never reach the client machine's own loopback, so these destinations are now always contacted directly, before NO_PROXY is consulted: localhost, *.localhost, 127.0.0.0/8, ::1 and 0.0.0.0. Similar to Go's net/http. Proxy handling for downloads and remote --server-base endpoints is unchanged.
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
* docs: move proxy tip from README to server docs
per review: the top level README stays lean; the proxy howto now lives with the server docs under Usage
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
---------
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
Co-authored-by: texasich <texasich@users.noreply.github.com>
Co-authored-by: texasich <texasich@gmail.com>
Co-authored-by: marecabo <23156476+marecabo@users.noreply.github.com>
Use the instance's is_mistral_format flag when DeepseekV2Model reloads
configuration, allowing Mistral MoE checkpoints to read params.json.
Fixes#29899
Signed-off-by: aiqubits <aiqubits@hotmail.com>
* llama-eval : add sandboxed execution for code suites
Adds a lightweight replacement for the Docker containers usually used to run
generated code, so a benchmark can be executed without a daemon, an image or
root.
A cached venv is provisioned per suite and reused; candidate programs run in a
throwaway working directory as their own process group, under the strongest
isolation the host offers:
bwrap read-only root, tmpfs /tmp and $HOME, no network
unshare no network, via --map-current-user
rlimit cpu/memory/file-size/process caps only
Candidate code must not run as uid 0, which is why "unshare -r" is a
last-resort tier that warns: root bypasses file permission bits, so a test
asserting that a write to a chmod 0444 file fails would instead see it succeed.
Two details worth noting for the bwrap tier. Only paths hidden by the $HOME
tmpfs are bound back, because bwrap cannot mount onto a symlink and
version-manager alias directories usually are one. And the venv's base
interpreter tree is resolved by walking the symlink chain, since a
version-managed interpreter is commonly reached through an unversioned alias
that startup resolves through.
Suites can declare provisioning steps, which run once at build time while the
network is still up, so test time never needs it.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : add HumanEval and ClassEval code suites
Adds two execution-graded suites alongside the existing text-graded ones. The
model's reply is parsed for a python code block, combined with the task's own
unit tests, and run in the sandbox.
--dataset humaneval 164 function-level problems, scored pass@k
--dataset classeval 100 class-level tasks, scored at class level (feeding
pass@k) and method level, matching upstream
pass@k reuses the existing chunk mechanism: --n_cases at a multiple of the
dataset size gives that many samples per problem, aggregated with the unbiased
estimator from the Codex paper.
Program construction and result interpretation are dataset-owned hooks, because
the two suites differ: one HumanEval problem is a function judged by a single
assert, whereas one ClassEval task is a class judged by many independent test
methods, and reducing that to pass/fail would throw away partial credit. Suites
also declare their own package set, provisioning steps, interpreter and timeout.
The package set is derived from the dataset's own imports rather than a
requirements file. HumanEval turns out to need no third-party packages at all;
ClassEval needs thirteen, two of which must be declared because no import
mentions them -- lxml is selected by string in ClassEval_44, and scipy is a
gensim requirement.
Notes on ClassEval specifically:
* gensim publishes no wheel past cp313, so 3.13 is preferred; on 3.14+ the two
pure-Python APIs it uses are provided by a small shim instead.
* The three nltk corpora it needs are pre-seeded at venv build time. These are
the modern names -- the dataset asks for the since-renamed punkt and
averaged_perceptron_tagger.
* It defaults to a patched fork, ilintar/ClassEval, because three upstream
tests cannot be passed by anything: a hard-coded 2024 date checked against
datetime.now(), an assertion on the dataset author's own hostname, and an
exact float comparison. Reference solutions go from 96/100 to 100/100 classes
and 2182/2196 to 2196/2196 test methods. No skeleton is touched, so what the
model is asked to write is unchanged; --dataset-source selects the original.
--exec-seed (default 0) seeds the RNG before each test, so suites whose tests
draw random inputs score reproducibly. ClassEval_58 fails its own reference
solution about 30% of the time without it.
The simulator gains both suites, so the harness can be exercised without a
model.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : fix silent isolation downgrade, generalise the sandbox runner
The isolation probe ran the interpreter without binding it back, so whenever
this tool is itself started from a venv under $HOME -- the usual case, since it
needs `datasets` -- bwrap's tmpfs hid the interpreter, the probe failed, and the
sandbox quietly fell back to a tier that does not hide $HOME at all. Nothing in
the output looked wrong when that happened, and the existing suites have been
running that way. The probe now uses the same interpreter and the same binds the
suite will actually get.
Two additions the agentic suite needs, neither of which changes the existing
suites:
* run_argv() runs an arbitrary command in a caller-owned directory that survives
the call. run() is now a thin wrapper over it, so HumanEval and ClassEval take
exactly the same path as before. A multi-step agent needs its edits to persist
across many invocations, and a suite for a language other than Python needs to
drive a linter and a test runner rather than a single source string.
* RLIMIT_AS is now per-sandbox, defaulting to what it was. It caps reserved
address space rather than resident memory, which is the wrong instrument for a
runtime that reserves a large region and commits little of it: node's
TypeScript type-stripping goes through WebAssembly, reserves gigabytes per
instance, and aborts at 4 GiB with an out-of-memory error that reads like a
fault in the code under test.
HumanEval stays at 164/164 and ClassEval at 100/100.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : add SACB, a multi-turn agentic coding suite
`--dataset agentic` is a different shape from the other code suites. Rather than
answering a single prompt, the model is given a small but real repository, a
narrow tool set and a bug report, drives its own multi-turn conversation, and is
graded by running a test suite it was never allowed to see.
The corpus is SACB (Simple Agent Coding Benchmark), 60 tasks over three
hand-written repositories in Python and TypeScript, hosted at ilintar/SACB.
Grading needs no container: a cached venv gives ruff and mypy for Python, and
node's own test runner plus tsc for TypeScript, so the whole dependency set is
four packages.
Three constraints are deliberate:
* No shell, and no way to run the tests. The only feedback channel is `lint`. A
syntax or type error can be driven out mechanically; a logic error has to be
reasoned about. A harness that lets an agent iterate against the test suite
measures something closer to search than to understanding.
* Two edit tools, one line-addressed and one content-addressed. Which one a
model reaches for, and whether it keeps line numbers straight after its own
earlier edit, is itself a signal.
* The tests never touch the working tree. They are held outside it and copied
into a throwaway copy only once the agent has stopped, so they can be neither
read nor edited. Every tool path is resolved and confinement-checked against
the repository root, so a symlink planted inside the tree cannot reach out of
it.
A task counts as resolved when every test that was failing passes and nothing
that already passed broke. Both sets are derived by running the tests against
the defective tree and against the reference fix, never written by hand, so a
task whose defect no test exercises cannot enter the corpus. Many tasks carry
several independent defects, so the fraction of target tests passed is reported
alongside, separating "fixed two of the three faults" from "changed nothing".
An episode that arrives with no tool call and no `finish` is nudged rather than
ended. Models routinely narrate their analysis mid-task, and treating that as
completion ends the episode with no edits and scores a reasoning failure that
never happened; about one episode in four exits that way.
Nothing here is installed unless this suite is selected, and then only for the
language being run: agentic_eval is imported lazily and each toolchain is keyed
per language.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : make the agentic hooks typed rather than discovered
`ty` flagged nine unresolved attributes, of which the seven in main() predate
the agentic suite: `EvalState.dataset` is Optional and was dereferenced without
narrowing.
* `loaded_dataset` turns the Optional into a value in one place, raising rather
than handing back None, so callers do not each have to narrow it.
* `is_agentic`, `prepare()` and `run()` are declared on BaseDataset. They were
reached through getattr with a default at three call sites, which no checker
can see through; `run()` now raises a clear error naming the suite instead of
failing as a missing attribute.
* The sandbox setup is one branch on `is_agentic` rather than two tests of
`grader_type`, which also drops a duplicated condition.
No behaviour change: HumanEval stays at 164/164, and `ty check` on
examples/llama-eval is clean.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* feat: API key
* fix reasoning and response length for agentic runs, add option to save entire transcripts
* llama-eval : fix ty unresolved-attribute on loaded_dataset.ae
Assisted-by: AI coding agent
* llama-eval : trim comments and remove non-ASCII from README
Assisted-by: AI coding agent
* gguf-py : add numpy Q1_0 and Q2_0 de/quantization support
* cont: added q1_0 / q2_0 to the test list removing the small test
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
---------
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
* CUDA: HIP: rms_norm: load float4 at a time and store x in registers
* Add template paramter comments
* use ggml_cuda_is_aligned for src
* nits
* also use ggml_cuda_is_aligned for src in fused variants
* fix nits
* tests : update the Metal fusion baseline
Re-record the MTL fusion baseline with the current build: 276 rows over 131
dummy models.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : fail test-fusion when a baseline row stops matching
The report was built from the counters the run produced, and the baseline was
only ever used as a lookup table, so a fusion that stopped matching produced no
row and was never compared. qwen4exp and mellum lost their fusions this way
while the Fusion / metal job stayed green.
Report a baseline row that the run does not reproduce as a regression. Scope
the pass to the arches that were actually run so --model keeps working.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : never write inline comments unless the user asks
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : keep the mellum dummy model softmax-gated
mellum reads expert_gating_func from the GGUF since #30578, while the fixture
writes sigmoid for every MoE arch, so the Mellum dummy model stopped building
the softmax MoE routing fusion. Write softmax for mellum, matching the arch
default and the GGUFs that conversion/mellum.py produces.
Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : use emoji in the AI stats disclosure
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : list the disclosure model after the hardware
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : use monochrome symbols in the AI stats disclosure
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : add muse-spark to the Metal fusion baseline
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round
Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors
* hex-get-rows: vectorize raw k-quant ukernels
* hex-binary: refactor dma-loop thread functions (no need for macros, etc)
* hex-quant: move/cleanup flat and tiled dequant ukernels
* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights
* hex-get-rows: remove redundant cast
* hex-repack: proper filtering for BF16 that were leaking through during the tests
* hex-repack: streamling tiling rules (broken MM for now)
* hex-repack: fix mmid after tiling updates
* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars
* hex-mm: fix valid_rows calc in various kernels
* hex-mm: remove unused hvx_mm_nx_2d_flat
* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16
* hex-mm: another pass at hvx-tiled f16 matmul
* hex-get-rows: optimize small src1
* hex-get-rows: one more pass at recovering get-rows perf regression
* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py
* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul
* hex-mm: fold tiled_f16 checks into the general flow
* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op
* hex-quant: proper tail handling in new dequant ukernels
* hex-get-rows: remove unused code and cleanup support checks
* hex-mm: cleanup ne1 padding/tiling checks
* hex-mm: cleanup fusion checks for tiled tensors
* hex-repack: add a little can_repack helper to reduce duplication
* hex-repack: never repack host buffers
* hex-repack: factor out repack functions into separate .h and .cpp
* hex-tensor: cleanup tensor ifaces for tiled/padded sizes
* hex-repack: update k-quant tiling comments and cleanup #defines
* hex-dcache: factor out l2/dcache dirty tracking and flushing
* hex-tensor: update docs and remove #defs duplication
* hex-tensor: remove unused/incorrect get_data interface
* hex-get-rows: cleanup vtcm layout build
* tests: disable huge MM tests
* hex-dcache: add missing header
The scheduler uploads each view of a host input on its own, so the per-layer reshape of new_pool_idxs (and of sel_mask in glm5-next) was copied once per layer.
* enable OPENSSL=ON as default in build script, update it in CI and dockerfile
* change to LLAMA_BUILD_BORINGSSL on windows
* refactor
* rm Download UI build
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model: add Muse Spark Open (text, vision and audio)
Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.
Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.
The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Muse Spark: drop the redundant forward expand
* Muse Spark: drop the redundant forward expand and tighten the comments
* Update src/llama-quant.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-graph.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* pi : expand AI disclosure with hardware and stats
Prefer PI_HARDWARE for the hardware field, spell out the disclosure block
template (model link, harness, hardware, stats) and source [STATS] from
~/.pi/stats.sh. Record the hardware in the Assisted-by commit tag.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : drop square brackets from placeholders
Brackets inside a markdown link text break the link on GitHub. Describe the
disclosure bullets in words instead of embedding a bracketed template.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Small helper around the GitHub Actions API: lists the workflow runs that are not
finished yet (in_progress, queued, pending, waiting, requested), grouped by status.
Supports selecting runs by status, by branch (server-side, to reduce API traffic)
or by a regex on the branch name, and can cancel the selected runs via --kill.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
The first evaluated token attends to the positions above
pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the
whole window, but the reuse threshold asked for two more cells.
A saved SWA state keeps only the cells inside its window, so a
state read back from a slot file or from the RAM prompt cache was
always rejected and the slot rolled back to a checkpoint.
The SWA threshold now matches the window, while a state without SWA
keeps the exact bound. The test harness gains override_kv, used to
give the SWA test model a window smaller than the test context.
[no release]
The slot file held the target context and the context checkpoints,
but never the draft context, so a restored slot drafted with an
empty or stale draft KV until the cache refilled.
The file now ends with the live draft state, the full draft context
and the speculative state, in the same appendix style as the
checkpoints. A draft state that does not fit the draft context is
dropped and the draft starts empty, while older files and older
servers keep working.
[no release]
Slot selection takes free slots by id or by last use only, so child
tasks often land on a slot holding a parent state while empty slots
are idle, and overwrite it without saving it to the prompt cache.
Free slots are now ordered empty first, then least recently used, for
both the LRU fallback and the child slots, and a child task saves the
state it overwrites. With 4 grouped decision requests in flight and
12 states in rotation, prompt cache reads go from 5 to 36.
[no release]
The registry hard-coded 10 for AGENTIC_MAX_TURNS while DEFAULT_AGENTIC_CONFIG
said 100. The registry seeds SETTING_CONFIG_DEFAULT, so the loop's
`|| DEFAULT_AGENTIC_CONFIG.maxTurns` fallback never fires and users got 10
turns. Reference the shared constant, as the adjacent MCP timeout does.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* Add Support to maion-coder arch
* iAdd Support to support Draft models
* Address comments
* Remove maion context shift function
---------
Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
tojson wrote nan, inf and -inf for these floats, as values and as dict
keys; json.dumps writes NaN, Infinity and -Infinity. The string and safe
filters of a float keep nan and inf, as Python's str() does, so they no
longer use the tojson function. float_to_json is renamed float_repr, as
it now gives str(); float_json gives the json.dumps text.
Also say in a comment why a last digit 9 is not bumped at a power of 2.
* vulkan : fix concurrent first device init
* tests : load models in parallel in test-thread-safety
Loading them serially never exercised concurrent first backend init, so the crash reported in #29764 was invisible to this test.
* vulkan : create fence before retaining device in buffer context
* replace oneAPI package by DLE+oneDNN package
* support to install oneDNN in windows, in release and docker
* for internal test
* remove test code
* Update intel.Dockerfile
* llama : split llama_prefetch() into merge and prefetch steps
Merge the ranges first, then call the system functions.
This fixes -Wunreachable-code-return on Windows.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Use const_cast
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* vulkan : copy strided f16 KV once so reads spread across memory channels
Assisted-by: Claude (Opus 4.8 / Opus 5)
* tests : cover the f16 KV copy to scratch
Assisted-by: Claude (Opus 4.8 / Opus 5)
* vulkan : gate the f16 KV scratch copy to AMD
Assisted-by: Claude (Opus 4.8 / Opus 5)
* vulkan : reuse the generic copy shader for the f16 KV scratch fill
The f16 clause needs only a strided copy, not a dequant, so drop the
dedicated shader and dispatch the existing copy pipeline instead. The
copy is described in source memory order so the strided access lands on
the writes, which stream, instead of the reads, which would concentrate
on a few memory channels.
Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5)
* vulkan : extend the strided KV scratch copy to bf16
Assisted-by: Claude Opus 5
* vulkan : clean up comments
Assisted-by: Claude Opus 5
* consolidate copy shaders
* disable on RDNA4
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* cpu: add F16 SWIGLU_OAI reference implementation and F16 activation op test coverage
Adds ggml_compute_forward_swiglu_oai_f16() to the CPU backend, mirroring
the existing F32 path, and extends test-backend-ops with F16 test cases
for SWIGLU_OAI so downstream backends(e.g. Hexagon) can be CPU-diffed
for correctness.
* ggml-openvino: fix f16 SWIGLU_OAI type mismatch in alpha/one constants
translate_glu_swiglu_oai creates alpha_const and one as f32 constants
unconditionally, so f16 inputs hit a type mismatch in Swish/Add on GPU
("Swish inputs must have the same type but they are: f16 and f32").
translate_glu_swiglu_clamp already upcasts to f32 and converts back at
the end; apply the same pattern here.
---------
Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
* llama: honor PARTIAL_ONLY in the plain KV cache
A non-SWA llama_kv_cache rolls back with seq_rm, so it holds no
partial state, yet state_write ignored the flag and serialized every
cell. A draft context made of a plain KV cache (MTP head, dense -md
draft) therefore copied its whole KV into each server checkpoint,
growing it by the draft KV size per token.
The cache now writes and reads nothing for PARTIAL_ONLY unless it is
SWA, and the iSWA wrapper no longer gates its base cache since it
filters itself.
* Update src/llama-kv-cache.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama: delegate PARTIAL_ONLY to the caches of every memory wrapper
The hybrid, hybrid-idx and dsa-iswa wrappers skipped their attention
caches on PARTIAL_ONLY, which also dropped an SWA window that is
partial state. Each cache now applies the flag itself, so the
wrappers only delegate. The indexer is marked stale only when a
restore rewrites cells.
test-llama-context gains a case checking that the partial state is
not larger than the full state and, without SWA, does not grow with
the tokens.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
An embedding-only graph has no logits, and build_pooling() returns
early when embeddings are off, so since the models no longer expand
t_embd the graph ends up empty and its inputs are never allocated.
Expand t_embd in that case.
* server: save only restorable slot states to the prompt cache
The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.
Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.
* server: test that decision slots skip the prompt cache
Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.
* server: restore any reusing task from the prompt cache
Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.
Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
* spec : add lilicorr support
* convert : use convention for hparam assignment
* add todo for hparam use in llama-batch.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* typo fix
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The clamp kernel indexed source and destination as one flat contiguous array, so
a strided view as input (the new CLAMP v=1 case) read the wrong elements. Add a
strided kernel_clamp_nc, as the other unary ops have, and take it whenever the
source is not contiguous.
* sycl: persistent IQ3_S/IQ3_XXS weight reorder for SYCL decode and prefill
Add persistent reordered weight layouts for IQ3_S and IQ3_XXS. Weights
are reordered once at load time; the reordered MMVQ dot products then
walk dense super-blocks with dp4a and SWAR sign expansion instead of
gathering bits from the interleaved standard layout. Reorder-aware
dequantization feeds the batched prefill GEMMs so prompt processing
sees the same benefit.
Also harden the oneDNN build check: a missing DNNL package or a
CPU-only DNNL build now fails at configure time instead of silently
producing a degraded backend.
Measured on Intel Arc Pro B70 with Qwen3.8-27B-GSQ-RCO-IQ3_S (MTP),
-ngl 999 -b 4096 -ub 1024 -fa on:
tg256 @ 0 ctx 12.92 -> 23.54 tok/s (+82%)
tg256 @ 9,216 12.33 -> 21.69 tok/s (+76%)
tg256 @ 64,000 9.76 -> 14.91 tok/s (+53%)
pp8192 833.9 -> 833.6 tok/s (flat)
Qwen3.5-9B Q4_K_M / Q3_K_M are unchanged within noise (largest delta
-0.5% pp, tg flat), so standard quants are not affected.
The MMVQ GLU fusion widening and rms_norm+scale fusion are
intentionally not part of this change; master already carries them
via #28931.
* removing wait as per review request
* sycl : fix fusion.cpp line ending
* ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (#30342)
#30270 routed both scalar conversions through F16C. For FP16->FP32, Clang 20.1.8
lowers the intrinsic to a vpinsrw merging into the live accumulator register of
the quantized dot products, which serialises the loop (19 vs 4 cycles/iteration
estimated, 45-65% slower Q8_0/Q4_0 MUL_MAT_ID kernels measured). Keep F16C for
FP32->FP16 as intended and the table for FP16->FP32.
* ggml-cpu : comment out the F16C FP16->FP32 define instead of removing it (review)
* ggml-cpu : size set_rows conversion buffer for all threads
SET_ROWS is planned with n_tasks = 1, but every thread runs it and uses its own row of the F16 -> F32 conversion buffer, so threads > 0 wrote past the work buffer when the graph had another multi-threaded op.
* move set_rows to n_task = n_threads group
* cuda, hip: GDN chunked kernel
Ported-from: https://github.com/tetherto/qvac-fabric-llm.cpp/pull/257
Source-commit: 53f0027d8070715fe9ad2131f3201bb0001d4eed
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
* fix older builds
* review fixes
* cuda: enable GDN MMA on tested Ampere and Ada GPUs
Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback.
Assisted-by: Codex
* cuda : use refactored swizzle API in GDN MMA
* ggml : GDN only guarantees the final state when n_tokens > K
* cuda : add short GDN MMA phase comments
---------
Co-authored-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
* skill : add tools/ui review checklist to code-review skill
Adds a WebUI (tools/ui) section distilled from a year of review
feedback on tools/ui PRs: constants/enums placement, layering
(stores/services/utils/hooks), DRY and dead code, Svelte 5 idioms,
comment hygiene, streaming perf, frontend security, and PR hygiene
(static build, demos, scope discipline). Scoped to tools/ui only.
* skill : expand tools/ui constants/enums ruleset
Replaces the short constants bullet list with a decision ruleset
distilled from review outcomes: when a literal becomes a string
enum vs an enum-keyed map vs a declarative config vs a grouped
as-const object vs a scalar constant, naming and suffix
conventions (_REGEX, _MS, _PX, semantic separators), placement
and composition rules, and what legitimately stays inline.
* skill : drop stale static-build-commit requirement for tools/ui
Static build output is no longer stored in the repo - the UI is
built by CI and bundled at release time. Keep the requirement that
npm run build succeeds and the UI build job is green.
* skill : address copilot review of tools/ui checklist
- scope the video-demo requirement to user-facing interactive/visual
changes instead of every tools/ui PR
- narrow the constants trigger to reused/domain-significant literals,
keeping the inline boundary as the escape hatch
- carve out independent literals for persisted/wire-contract test
assertions so they don't become tautological via shared constants
- allow simple presentation-local $derived in components; only shared
business logic, orchestration and parsing leave the component
- match the existing barrel documentation style instead of requiring
per-export JSDoc everywhere
- keep non-obvious why/rationale comments; move only examples and
edge-case enumerations into tests
- point at the actual interpolate-size/grid-template-rows transition
pattern in CollapsibleRegion.svelte instead of calc-size()
* skill : use spaces for nested list indentation
EditorConfig CI requires space indentation for all files; the
scalar-constants sub-bullets used tabs. Matches the 2-space nested
list convention used in docs/.
* ggml: fix conv_1d result layout for batched input
The GEMM behind conv_1d folds the batch into its rows, so the product
sits in memory as [OL, N, OC] while the reshape claims [OL, OC, N].
The shape is right and the contents are transposed, so nothing fires:
no assert, no abort, the graph just runs and the channels come out
interleaved with the batches. With N == 1 the two readings describe
the same buffer, which is why the three vocoders in the tree never
hit it, they all squeeze to 2d right after the call.
Reshape to the real layout and permute OC and N back. The N == 1 path
keeps its direct reshape, so the current callers build the same graph
and pay for no extra copy.
tests/test-conv-1d.cpp compares one batched convolution against N
single batch convolutions built from the same weight and the same
input slices. Five of the seven geometries fail before this change and
all seven pass after. N == 1 and OC == 1 pass either way, since
swapping two dimensions when one of them is 1 changes nothing.
The batched path matters to downstream audio projects where several
synthesis lanes share a single graph. Nothing in mtmd batches a
vocoder today, each one streams one flow and carries its left context
between calls, but the op is correct now if one of them grows a lane
dimension.
* ggml: build the same conv_1d graph for every batch size
Address review from ggerganov: drop the N == 1 branch of ggml_conv_1d.
The GEMM result is always reshaped to [OC, N, OL] and permuted back,
as ggml_conv_2d does, so every batch size builds the same graph. For
N == 1 this adds a cont on an already ordered tensor.
* common : add --spec-draft-n-ubatch
Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).
The default 0 keeps the current behavior of following --n-ubatch.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
[no ci]
Added that the oneapi toolkit includes the sycl dependencies. This makes it clear for first time users, that the toolkit includes all necessary dependencies, so that the user is guided to install the toolkit if they do not already have the dependencies separately
When every device reports zero free memory the split points are
normalized by a zero sum, the NaN split sends layers to a device index
past the end and loading aborts with vector::_M_range_check.
- add rpc_log_format_size() and print byte counts in KiB/MiB/GiB across the
RPC client, server and transport traces
- add rpc_log_format_tensor() and include tensor name, op, type and shape in
the set/get/memset/copy/init tensor logs
- fix rpc_cmd_name() returning UNKNOWN for SET_TENSOR_2D, GET_TENSOR_2D,
COMM_INIT, COMM_ALLREDUCE and COMM_FREE
- move the log helpers to ggml-rpc/log.{h,cpp} under a common rpc_log_ prefix
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
tojson printed floats through a C++ stream (6 significant digits, no
".0"), so 48.858844 became 48.8588 and 1.0 became 1 in tool definitions,
tool-call arguments and other template output. Write the shortest digits
that read back as the same value, as json.dumps does. The string and safe
filters of a float use the same code.
These graphs only produce embeddings, and build_pooling() already
expands its output for every pooling type, so the expand at the end of
the model graph is redundant.
The RDNA3.5 config table selects a 64-row / 128-thread tile up to J=32 and a
128-row / 256-thread tile above that. On gfx115x the smaller tile leaves enough
LDS free for a second workgroup per CU, while the larger one does not.
Extend the 64-row tile to the (type, J) pairs where it measured faster on
gfx1151, and leave every other entry untouched. J=96 is deliberately excluded
because it did not benefit.
test-backend-ops perf on gfx1151, 100 MUL_MAT shapes over the affected types
(N in 48..512, M both padded and unpadded), 3 rounds: median +2.1%, best
+26.0%, no shape outside run-to-run noise. 5 MUL_MAT_ID shapes: median +8.0%.
Assisted-by: Claude Opus 5
* llama : build the k-pool selection in the causal mask layout
- the glm5-next indexer folded the causal mask in through a [n_kv, n_tokens]
view: one reshape of the mask ahead of the add, and one of the result ahead
of every layer reading the selection, including the shared-indexer layers
- build the selection directly in the mask layout, as qwen4exp already does,
and drop the now-degenerate reshape in its attention as well
- assert the selection width matches the input it is taken from
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
* unicode : add opt-in NFC normalization for Unigram tokenizers
Note: used help from Codex, Sol 6.1 Medium
* Fixing initial bot comments + linter error (using existing out instead of print)
* removed test_nfc_tokenizer
The d1-omni and kev prompts escape "<|name|>" in the request text with
std::regex_replace. libstdc++ recurses for each char of the name, so a
state of "<|" and 30000 letters overflows the stack of the HTTP thread
and llama-server dies with SIGSEGV.
server_decision_escape_special_tokens() writes the same text with a loop.
test-server-component compares it with the regex on every string of up to
6 chars of "<|>a-".
* hexagon: add configurable HTP DCVS/HMX power mode
* hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup)
* hexagon: add --hex-power-mode flag to run.py
Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following
the existing --hex-nhvx pattern. The env var must be set before the
process touches the Hexagon backend registry (see comment in
ggml_hexagon_init), so this is exposed as a runner flag rather than a
common/arg.cpp CLI option.
* hexagon: keep HTP awake for DCVS power profiles
* Apply suggestion from @max-krasnyansky
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The no-cache attention pads K, V and the mask to 256, but T5 also adds
a relative position bias of [n_tokens, n_tokens], so the add with KQ
fails unless n_tokens divides the padded width. Pad the bias with
zeros up to the KV dimension of the mask, like K and V.
* llama : do not map foreign ranges into a backend buffer
A buffer created from a host pointer pins everything that it wraps. When
several contexts share the same file (lazy tensors, MoE tensors kept on the
CPU with -cmoe), the single buffer that spans the first and the last tensor
of a context also wraps the ranges owned by the other contexts, pinning data
that the backend never reads.
For models with per-layer token embeddings this pins the whole lazy tensor
into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the
point of reading it lazily.
Instead, compute the file ranges of each context and subtract the ranges
owned by the other contexts, then create one buffer per range. Weights that
are mapped in more than one context (e.g. token_embd.weight) keep being part
of the range of every context that uses them.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : enable prefetch on macOS
The prefetch hint of the tensors marked as prefetchable is a no-op on macOS,
so their pages are faulted in one by one while the graph runs. madvise
(MADV_WILLNEED) behaves as expected here, so add the platform to the guard.
With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read
from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to
735 / 749 / 788 t/s for pp512 / pp2048 / pp4096.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : use a plain buffer list per context
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* graph: skip the LM head for pooled embeddings
A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.
The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.
* graph: drop the redundant t_embd expand
Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.
* graph: add build_logits() for the LM head output
Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.
* graph: call build_logits() before build_sampling()
Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.
* nit
* tests : consolidate the context state tests into test-llama-context
test-save-load-state, test-recurrent-state-rollback and
test-state-restore-fragmented check the same object from three places:
the KV cache and the recurrent state of a llama_context, saved, loaded,
copied, rolled back. Each had its own main, its own loop over the
models, its own copy of the token and the comparison helpers, and a run
of all three loaded the same models several times.
They are one binary now, and the case is the unit of the test:
tests/llama-context/test-case.h the interface a case implements
tests/llama-context/cases-kv.cpp the cases of the KV cache
tests/llama-context/cases-rs.cpp the cases of the recurrent state
tests/llama-context/test-util.cpp tokens, generation, comparison
tests/llama-context/test-runner.cpp the models of a case, on threads
tests/llama-context/test-log.cpp the log file of a model
A run is a table: a row per model, a column per case, a cell holds the
verdict of that pair. The models load on several threads (-j, --threads,
min(cores, 4) by default), the table is printed when it is complete and
the log of a model that failed is printed after it, in the order of the
rows. A case returns a verdict and the reason for it, and the runner is
the only one who reports - the test does not use tests/testing.h, a run
over 130 models is a matrix of verdicts and not a tree of subtests.
Staging no longer travels through the environment: llama_model_set_attn_rot
replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one
thread used to change for the models the other threads were loading, and
llama_memory_get_buffers is added for the cases that copy the buffers of
a memory. common_log_set_stdo keeps the console quiet while the table is
being built.
ggml_aligned_malloc allocates with posix_memalign when the thread
sanitizer is on. On macOS it allocates with the Mach VM, which the
sanitizer does not intercept, so it never sees a page range change hands
and reports the first write into recycled memory as a race - 30 of them
in this test, and every one of them false.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : run single-threaded by default
* cont : try to build the tests on all platforms
* cont : trim down some body fat
* cont : fix
* cont : fix the fix
* ggml-cuda: gate CUDA graph debug logs behind GGML_CUDA_GRAPH_DEBUG
* ggml-cuda: cache env var lookups in static const instead of re-reading per call
* ggml-cuda: read env vars exactly once via ggml_cuda_env_flag/ggml_cuda_env_int
* ggml-cuda: consolidate env flag reads into single static accessors
* ggml-cuda : drop env helper comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : remove NDEBUG gates for CUDA graph logs
* vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount
* vulkan : fix narrowing in chunked matmul dispatch
* vulkan : zero fill the A side of mul_mmq for k steps past the end
The A side skipped the shared memory write for the k steps past end_k, while the B side was already told the bound and zero filled. The A registers are read for all BK_STEP steps, so the stale scale went into the dot product and produced NaN at k = 32.
Pass the bound to block_a_to_shmem and write zeros when it is false, for every DATA_A type, mirroring the B side.
Also assert that the B and D subbuffer sizes are not VK_WHOLE_SIZE before they are adjusted per chunk, and add two batched cases of the chunked shape.
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* vulkan : remove redundant matmul tile adjustment
---------
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* tests: cover weighted scalar broadcasts
Assisted-by: OpenCode
* ggml : support extended scalar DMA binary ops
Assisted-by: OpenCode
* hex-binary: remove all non-dma paths
* hex-binary: add support for non-contig strides and cleanup dma preable a bit
* hex-binary: cleanup the gating and simplify dma retry
* hex-geglu-erf: add a gate for F32 only, we dont support F16 for this op
* tests: cover weighted repeat broadcast
Assisted-by: OpenCode
* hex-binary: update REPEAT and COMPLEX kernels to use dma as well (missed them last time)
* hex-add-id: add dma support for src2 (idx)
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The no-cache path builds a [n_tokens, n_tokens] mask and passes K and V
unpadded, so FLASH_ATTN_EXT sees K->ne[1] == n_tokens. The CUDA kernels
for head size 512 require a multiple of FATTN_KQ_STRIDE, so the op falls
back to the CPU for most lengths, and the -fa auto probe (1 token)
disables flash attention for such models.
Pad the mask like llama_kv_cache::get_n_kv and pad K and V with zeros up
to the KV dimension of the mask. clef builds its own causal mask and
pads it the same way.
* hexagon: move the converting CPY in blocks of rows per DMA descriptor so short rows are not DMA-latency bound
* hex-cpy: cleanup cpy cursor handling
* hex-cpy-concat: introduce threaded dma/hvx gather-row helper
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
* hex-cpy: remove leftover rejections for extended buffers
* hex-cpy: relax the gate for fast-path
* hex-cpy: remove scalar divs and vtcm access and use explicit gather_sync
* hex-cpy: add proper support for mdev work splitting to fastpath
* hex-cpy: optimize dense gather-row tasks
* hex-cpy: unroll gather-rows loops
* hex-cpy: add fastpath for f16
* hex-cpy-concat: support for colapsing 3/4D dims
* hex-cpy: double-buffer support
* hex-cpy: factor out gather-rows into more of an op by itself
* hex-cpy-concat: dedup kernel-params
* hex-cpy-concat: step-down nr to fit instead of hard fail
* hex-gather: replace div with a mask
* hex-gather-rows: cleanup dma pipeline and remove the stalls
* hex-gather: specialize compute funcs
* hex-gather: reduce scalar reg.spills and func calls
* hex-gather: do not recompute rows-per-thread on the npu
* hex-build: add support for building/installing just hexagon
* hex-cpy-concat: remove redundant barrier in concat and move cpy barrier under fence
* hex-gather: annotations for dma-out
* hex-mdev: make all key vtcm_acquire sync state atomic (fixes random hangs in mdev setups)
---------
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
* vulkan: pack FMA for f16 accumulation, no dot product op
* vulkan: dot_product packs two rows per f16 FMA
* vulkan: rename dot_product to dot_product_v2
Give llama_build() and llama_build_and_test() a LINK keyword (default:
llama-common) so call sites can declare what a test actually needs:
- ggml-only tests now link just ggml: test-opt, test-metal-graph-optimize,
test-backend-ops, test-tiled-mulmat, test-barrier, test-quantize-fns,
test-quantize-perf, test-rope, test-col2im-1d, test-gguf, test-alloc
- llama-only tests drop llama-common: test-unicode, test-sampling,
test-grammar-parser, test-llama-grammar, test-batch-alloc,
test-gbnf-validator, test-quant-type-selection
- both mtmd tests link mtmd only, dropping the LLAMA_TEST_NAME /
target_link_libraries workarounds
Both helpers now set cxx_std_17 explicitly, since the tests no longer
inherit it from llama-common's PUBLIC interface.
tests/testing.h no longer includes common.h; the only symbol it used from
there (string_join) is inlined.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* metal : fix mul_mat_id limits and kernel name
* metal : fix mul_mat_id with duplicate ids
kernel_mul_mm_id_map0 assumed at most one match per token and a fixed
capacity of ne21 entries per expert, which produced wrong or dropped
output rows and could overflow into the slice of the next expert when
the ids contain duplicates.
build a compact per-expert list of the (token, slot) entries instead:
- pass 1 counts the entries of every expert, thread 0 computes the
offsets of the expert slices (tpe now holds [base, count])
- pass 2 writes the entries with one unconditional store per token
(conditional stores of divergent lanes are very slow), the dummy
value of the tokens without a match lands on a padding slot of the
slice that is never read - duplicates are handled by a rare branch
- kernel_mul_mm_id reads the base/count of its expert and loops over
the entry tiles that do not fit into the grid
The ids buffer is sized as ne20*ne21 + ne02 (entries + padding).
883/883 test-backend-ops MUL_MAT_ID tests pass, llama-bench is within
run-to-run variance of the previous behavior.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* metal : make mul_mat_id map0 ne02 a function constant
use FOR_UNROLL for the id loops of kernel_mul_mm_id_map0 and turn ne02
into the function constant FC_mul_mm_id_map0_ne02 (FC_MUL_MM + 7), so
that the threadgroup offsets of cnt/bas/sids and the prefix-sum bound
fold at compile time.
the pipeline name now encodes ne02 (same convention as the ssm_conv
nc/silu constants) and the map0 pipeline is compiled with the constant
value set. ne02 is no longer part of ggml_metal_kargs_mul_mm_id_map0 -
the dispatch grid still comes from the tensor ne[2] on the host.
no measurable perf change: 883/883 test-backend-ops MUL_MAT_ID tests
pass, op-level and llama-bench results are within run variance.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : clean-up + TODO
* metal : dispatch mul_mat_id entry tiles from the prepass
follow the approach of the vulkan implementation (#29998): instead of
launching a grid over experts x ne21 entry tiles and letting most of
the threadgroups early exit, kernel_mul_mm_id_map0 now also emits a
row map - the total tile count plus one (expert, first entry, valid
rows) descriptor per 32-entry tile that has work.
kernel_mul_mm_id is launched with a static upper bound on the number
of tiles (full tiles over all entries plus at most one partial tile
per expert), reads its descriptor and processes a single tile - the
tail early exits. This removes the round loop that was needed to cover
duplicate ids past the grid (register pressure: th_max 768 -> back to
the plain mul_mm level) and duplicates are now handled implicitly,
since they only produce more descriptors within the bound.
The grid shrinks by ~8-16x on typical shapes (e.g. 128 experts,
8 used, 512 tokens: 128 tiles instead of 2048 entry tiles, of which
94% only launched to exit) and ne21 is no longer needed by the mm
kernel kargs.
883/883 test-backend-ops MUL_MAT_ID tests pass. llama-bench
(Qwen3.6-35B-A3B Q8_0) is now faster than master on all pp sizes
(pp2048/4096/8192 +2%, pp512 +0.5%), tg32 unchanged.
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
* cont : add smem assert
* examples : fix draft sampling and rejection handling
This commit fixes two issues in the speculative example which are
described in detail in the linked issue.
Resolves: https://github.com/ggml-org/llama.cpp/issues/29975
* examples : add kv-unified requirement for parallel > 1
Add an optional ggml_backend_sched_realloc_callback that user code registers with
ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a
graph are not sufficient and have to be reserved again. The callback reports whether the
backend assignments and the graph size changed since the previous scheduling.
llama_context registers the callback when creating the scheduler (wrapped into the new
llama_context::sched_new helper) and treats the reallocations that follow a change of the
backend assignments as expected, counting the rest by graph size - unchanged or changed
from the previously scheduled graph. Both counters are reported on one line in the perf
stats.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* ui: fix text attachment preview overflow
Set the max height to 80dvh and the max width to full on screens
below medium and 80vw from medium onward.
* style: run formatter
* hexagon: add IM2COL_3D support
Assisted-by: OpenCode
* hexagon: add DMA paths for IM2COL_3D
Assisted-by: OpenCode
* hexagon: avoid scalar fp16 conversions in IM2COL
Assisted-by: OpenCode
* remove in-loop spills in pre-v81 devices
Assisted-by: OpenCode
* hex-im2col: re-write to use canonical dma dbl-buff pipeline, remove divs and very inefficient fp conversions
* hex-im2col: move kernel_params and vtcm layout precompute into the host
* hex-im2col: remove explicit tracking of number of dma-pushes
* hex-im2col: remove incorrect uint64_t casts
* hex-im2col: no need for macros for dma loops and pass compute func ptr via local context
* hex-im2col: use cursor pattern, tracing and proper n_threads handling
* hex-im2col: support non-contigue views
* hex-im2col: minor reformatting
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* opencl: fallback when Q4_K weight images exceed device limits
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: avoid incomplete subgroup in rms_norm
* opencl: refine conditions for dense bin kernels and fix bcast for q4_0
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
---------
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs
GGML_RPC_DEBUG is now parsed as a number: 0/unset disables debug logs,
1-3 emit increasingly detailed output (events, per-command trace,
transport detail). Non-numeric values fall back to 1.
The duplicated env/macro blocks in ggml-rpc.cpp and transport.cpp are
replaced by a shared log.h, which becomes the single choke point for
all logging of the RPC backend: LOG_ERROR/LOG_WARN/LOG_INFO for
unconditional severity logs and LOG_DBG/LOG_DBG2/LOG_DBG3 for the
verbosity-gated ones. The transport files no longer need ggml-impl.h,
and the server banner now also goes through the ggml logger (stderr).
Missing logs are added on both the client (handshake, buffer ops,
tensor transfers, graph computes, cache decisions) and the server
(per-command dispatch, graph nodes), including the negotiated
transport via the new socket_t::transport_name().
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
* rpc : align log levels with the documented verbosity semantics
The first pass introduced GGML_RPC_DEBUG levels, but many call sites did not
follow the documented semantics and some logs bypassed the macros entirely:
- Route the remaining raw GGML_LOG_* call sites through the local LOG_* macros
so that log.h stays the single choke point for RPC logging
- Emit per-tensor transfers and per-command handler traces at level 2, and
one-shot lifecycle events (backend and buffer type creation, buffer
allocation, graph compute, graph cache eviction) at level 1
- Drop the duplicated enqueue trace in rpc_dispatcher::send, work() already
logs every dispatched command together with its round-trip timing
- Promote degraded-operation events to LOG_WARN: remote allocation failure,
RDMA falling back to TCP, unexpected peer disconnect
- Log graph cache eviction on both sides, since client and server must stay
in sync for the incremental graph update to be valid
- Distinguish Unknown command (opcode out of range) from Unhandled command
(valid opcode with no handler)
- Fix format specifiers (%zu for size_t, %u for device ids, 0x for hex
pointers) and a missing newline in an error log
- Document that llama.cpp applications drop ggml debug records unless the
verbosity threshold is raised with -lv 5
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* vulkan: compute every row of an expert in mul_mm_id when ids repeat
* vulkan: handle mul_mat_id duplicates in the prepass rather than looping
Extend the "hoist row ids" optimization to always be enabled and to emit a
compact list of tile descriptions that need to run, and to emit multiple tiles
when needed. Then we can launch a tighter upper bound on the number of
workgroups and the tail can trivially early exit.
---------
Co-authored-by: Anjielon <[EMAIL_REDACTED]>
A request asking for a busy id_slot still ran the prompt cache update
on that slot before being deferred. When the RAM cache held a better
match, it was loaded into the slot while another request was still
generating there, and that generation continued on the wrong context.
The busy slot is now returned as is and the request waits for it.
* common : read a GGUF's trained context from its metadata
common_get_gguf_n_ctx_train opens only the file's metadata (no_alloc,
like common_get_decision_type) and reads <arch>.context_length, so a
caller can learn the trained context without loading the model. It
accepts both u32 and u64 values and returns 0 when the file is missing,
unreadable, invalid, or reports no context length.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* server : report the trained context in the models listing
update_caps already resolves the model file offline to read its
modalities, so it now reads the trained context from the same GGUF
metadata, and GET /models reports it as context_length when it is
known. A router listing then carries the context without any Hub
request, which lets the UI sort and filter by it offline.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : take the trained context from the models listing
The router now reports context_length per model, so the option mapping
fills contextLength from it and the manager reads it before the Hub
record. The Context column, the context sort and the context filter
then work with the Hugging Face Hub API turned off. A browser suite
guards the sort and the search, the Hub-cache driven context filter and
the re-sort when details arrive after the sort was clicked.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* ui : mark favorite models with a heart
A favorited model shows a rose heart in the selector even before its
row is hovered, and the crossed heart takes its place on hover, so
unfavoriting stays one hover away. The manager table marks its
favorited rows with the same heart after the badges and capabilities.
Assisted-by: pi:zai-org/GLM-5.3-Flash
* fix: UI text nit
* fix: UI nits
* fix: Favorite models grouping in models table
* feat: Remove sorting from Status column in Models Table
* server: read the GGUF metadata once per model
Read the decision type and the trained context in a single GGUF open,
accept only a UINT32 context length like the model loader, and reset
n_ctx_train with the other caps so a failed refresh drops it.
* fix: Post-review fixes
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* CUDA: fuse copy of updated state snapshots into recurrent cache with ssm_scan
* CUDA: remove redundant cuda copies with K==1 (non spec-dec) scenario as well
* opencl: skip kernel_cpy_f32_f32_pack on A6X to avoid shader compiler crash
* The A6x compiler backend found in iot device with a623 (E031.50.31.01)
cannot handle kernels with a large number of arguments. Skip this
kernel for A6x to avoid compiler crash
* opencl: A6X constant-fold workaround for get_local_size in GEMV kernels
* opencl: add Adreno 623 to A6X GPU detection list
* model : use exact GELU for ModernBERT encoders
Assisted-by: Codex
* model : keep tanh GELU aliases on ggml_geglu
Assisted-by: Claude Opus 5.5
* model : map gelu_python to ggml_geglu_erf
Assisted-by: Claude Opus 5.5
The reserve builds n_outputs_max_per_seq sampling chains per sampler,
while a decode built one per output row, so the graph changed its
topology after the reserve and GGML_SCHED_NO_REALLOC builds aborted on
the next same sized graph. Every sampler now builds
n_outputs_max_per_seq chains, the ones without a row of the ubatch on
the padding row and not selected, and graph_max_nodes counts them.
ggml_acc_impl narrowed a size_t offset to int32_t without checking that it
fits, so a large offset could truncate to a negative int32_t. The forward
then sign-extended it to a huge size_t and the bounds assertion wrapped,
allowing an OOB write below the dst buffer. Check the offset before the
narrowing, matching the existing check in ggml_set_impl.
* meta : handle views of tensors allocated on the host
A view shares the memory of its view_src, so ggml-alloc never allocates a view in
the buffer of the split it lands in - the scheduler copies the source into the
split and the ops that use the view read that copy. The view node itself is a noop
and does not need a split of its own, but the meta backend asserted when one was
left inside a meta split:
- ggml_backend_meta_get_split_state() dereferenced tensor->buffer->context
- the graph rebuild mapped every node with ggml_backend_meta_buffer_simple_tensor()
Accept such nodes when they are views of host tensors, which also generalizes the
previous s_copy_main workaround. This fixes the assert hit by KV cache views when
using --split-mode tensor with partial offload.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* archs : re-enable sm tensor for K2 Horizon
* cont : add TODO and reference
The mixed batch path of PR-29622 writes the token rows with set_rows
into a dup of the embeddings. WebGPU did not support DUP, so the dup
ran on the CPU while the set_rows writing into it was scheduled on
WebGPU, which then bound a CPU buffer and crashed. DUP is the same copy
as CPY and CONT and now goes through the same path.
* ui : add the drawer, sheet and grouped list primitives
Add the drawer and sheet overlay components, the toggle and toggle
group, the shared searchable input, the collapsible sections and the
grouped list, and the near viewport helper they position with.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : add the shared model components and data layer
Add the model row pieces shared by the selector and the manager, the
models store with its download status feed, the huggingface and
migration services, and the model utils, enums and constants they
read through.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : rework the models selector around its providers
Group the selector components under their own folder, rework the list
around the provider grouping, add the mobile trigger and the download
item, and derive the selection state in one hook.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : add the models manager
Add the manager table with its repo, quant and status rows, the
filters and the toolbar, the row actions in a drawer, the downloads
section, the manage models dialog and the stories and tests.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : rework the chat form actions and the mobile experience
Move the add actions into a drawer and a sheet, float the model
actions in a bar on a phone, open the context panel in a drawer, and
derive the attachment, reasoning and tools menus in hooks.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : move the mcp servers to the sidebar rail and tidy the dialogs
Move the mcp servers dialog to the sidebar rail with its menu
entries, even out the dialogs on a phone, and group the mcp and
settings stores behind their own modules.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : polish the shell
The secondary button gets its own look back and the chat add button its
own light surface. Pressable elements get a pointer cursor again, the
font rendering smooths on the app shell, and the agent skills stay out
of prettier's way.
The root layout props probe rework that came with this polish reads the
providers api url and stays with the providers change instead.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : wrap the model id classes
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* ui : load the model the new chat CTA picks
Start a new chat selected the model but left it unloaded, so the chat
opened against a server with nothing in memory. Both CTAs share a load
helper now.
Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
* refactor: Post-review fixes
* hexagon: fix IM2COL patch-embed DMA ring overflow
The exact-tiling (stride == kernel, no pad/dilation) IM2COL DMA kernel
issues IC*KH DDR->VTCM descriptors per output row without checking the
return value of dma_queue_push(), and then pops IC*KH times. The per-thread
DMA ring holds 256 entries and a push into a full ring returns false
and drops the transfer, so for IC*KH > 255 the remaining rows of the
VTCM staging buffer were never written and stale data (often NaN/inf)
leaked into the output.
Solution is to retire the oldest descriptor when the ring is full, just as the blocked
kernel in the same file already does, and wait with dma_queue_flush().
For testing, added exact-tiling test cases with IC*KH > 256 (2D 1x1, 2D 2x2 patch
embed and 1D, F16 and F32 dst), which fail on HTP without this fix.
* Apply suggestion from @max-krasnyansky
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
- [PEG parser](docs/development/parsing.md) - alternative to regex that llama.cpp uses to parse model's output
- [Auto parser](docs/autoparser.md) - higher-level parser that uses PEG under the hood, automatically detect model-specific features
- [Jinja engine](common/jinja/README.md)
Testing:
- [Slice model](skills/slice-model/SKILL.md) - build a small slice (few layers/experts) of a big HF model without downloading the full weights, to test conversion and inference
@@ -16,6 +16,8 @@ The project differentiates between 3 levels of contributors:
>
> Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file.
### Requirements
If AI is used to generate any portion of the code, contributors must adhere to the following requirements:
1. Explicitly disclose the manner in which AI was employed.
@@ -24,6 +26,16 @@ If AI is used to generate any portion of the code, contributors must adhere to t
4. Be prepared to explain every line of code you submit when asked about it by a maintainer.
5. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
### Checklist
> [!IMPORTANT]
>
> **If you use AI, do all of the following:**
>
> 1. **Make sure [AGENTS.md](AGENTS.md) is loaded into your agent's context.** Skipping this step is one of the most common reasons PRs get rejected.
> 2. **Search for related discussions, PRs and issues** to get the broader context before writing any code.
> 3. **Run a local review pass before pushing anything**, using the [`code-review`](skills/code-review/SKILL.md) skill, once the change is implemented. Fix what it finds.
For more info, please refer to the [AGENTS.md](AGENTS.md) file.
The main goal of `llama.cpp` is to enable LLM (and MLLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
The `llama.cpp` project is built on top of the [ggml](https://github.com/ggml-org/ggml) library.
## This fork - RX 580 (Polaris / GCN) serving of large MoE models
A private fork maintained for one machine: an AMD RX 580 8 GB (RADV Vulkan) serving Qwen3.6-35B-A3B,
gemma-4-26B-A4B and smaller models under llama-swap, with the MoE experts in host RAM
(`--n-cpu-moe`). **[FORK.md](FORK.md) is the source of truth**: every fork change with its
invariants, rebase hazards and validation, the measured dead ends, the production configuration and
the measurement traps. The rebase and deploy procedure is
[docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md); `scripts/fork-validate.sh` validates a
build against a reference; `scripts/rx580-bench/` is the benchmark harness.
In short, the fork adds:
- three Polaris prefill wins, on without a flag and token-identical to mainline: the MoE routing-ids
readback skip in `sched_copy_experts`, flash-attention `mask_opt` for GCN at head size >= 256, and
a tiled-transpose fast path for the delta-net conv-state concat;
- opt-in VRAM time-sharing between several llama-server processes (`LLAMA_SLEEP_VRAM_ONLY`,
`LLAMA_SLEEP_EVICT_KV`, `LLAMA_MMPROJ_ONDEMAND`, `LLAMA_VRAM_ARENA`): a model releases everything it
holds in VRAM and gets it back without a restart, so its RAM prompt cache survives;
- fixes to the server's sleep, wake and shutdown paths and to cold teardown that the sharing exposed;
- validation-only env knobs (`GGML_VK_CONCAT_FASTPATH`, `LLAMA_MOE_IDS_SKIP`), unset in production.
Production settings for the 35B are in FORK.md section 5 (`-lm none -t 6 -ngl 99 --n-cpu-moe 40
-b 2048 -ub 2048 -fa 1 --parallel 1`, ctx 131072). Keep all experts on the host (`--n-cpu-moe 40`)
for long contexts: lower values are faster only at short context and run out of VRAM as the KV grows
(FORK.md section 4).
## Quick start
A few options to get `llama.cpp` installed on your machine:
# SPIRV-Headers walks up three directories and appends /include, yielding /usr/local/lib/include; the extra vulkan/ level likely breaks its prefix calculation.
(time ./bin/llama-fit-params --model ${model_f16} 2>&1| tee -a $OUT/${ci}-fp-f16.log)
(time ./bin/llama-completion -no-cnv --model ${model_f16} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-f16.log
(time ./bin/llama-completion -no-cnv --model ${model_bf16} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-bf16.log
(time ./bin/llama-completion -no-cnv --model ${model_q8_0} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q8_0.log
(time ./bin/llama-completion -no-cnv --model ${model_q4_0} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q4_0.log
(time ./bin/llama-completion -no-cnv --model ${model_q4_1} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q4_1.log
(time ./bin/llama-completion -no-cnv --model ${model_q5_0} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q5_0.log
(time ./bin/llama-completion -no-cnv --model ${model_q5_1} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q5_1.log
(time ./bin/llama-completion -no-cnv --model ${model_q2_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q2_k.log
(time ./bin/llama-completion -no-cnv --model ${model_q3_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q3_k.log
(time ./bin/llama-completion -no-cnv --model ${model_q4_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q4_k.log
(time ./bin/llama-completion -no-cnv --model ${model_q5_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q5_k.log
(time ./bin/llama-completion -no-cnv --model ${model_q6_k} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-q6_k.log
(time ./bin/llama-perplexity --model ${model_f16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-f16.log
if[ -z ${GG_BUILD_NO_BF16}];then
(time ./bin/llama-perplexity --model ${model_bf16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-bf16.log
fi
(time ./bin/llama-perplexity --model ${model_q8_0} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q8_0.log
(time ./bin/llama-perplexity --model ${model_q4_0} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q4_0.log
(time ./bin/llama-perplexity --model ${model_q4_1} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q4_1.log
(time ./bin/llama-perplexity --model ${model_q5_0} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q5_0.log
(time ./bin/llama-perplexity --model ${model_q5_1} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q5_1.log
(time ./bin/llama-perplexity --model ${model_q2_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q2_k.log
(time ./bin/llama-perplexity --model ${model_q3_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q3_k.log
(time ./bin/llama-perplexity --model ${model_q4_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q4_k.log
(time ./bin/llama-perplexity --model ${model_q5_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q5_k.log
(time ./bin/llama-perplexity --model ${model_q6_k} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-q6_k.log
(time ./bin/llama-imatrix --model ${model_f16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-imatrix.log
(time ./bin/test-save-load-state --model ${model_q4_0} -ngl 10 -c 1024 -fa off --no-op-offload) 2>&1| tee -a $OUT/${ci}-save-load-state.log
(time ./bin/test-save-load-state --model ${model_q4_0} -ngl 10 -c 1024 -fa on --no-op-offload) 2>&1| tee -a $OUT/${ci}-save-load-state.log
(time ./bin/test-save-load-state --model ${model_q4_0} -ngl 99 -c 1024 -fa off ) 2>&1| tee -a $OUT/${ci}-save-load-state.log
(time ./bin/test-save-load-state --model ${model_q4_0} -ngl 99 -c 1024 -fa on ) 2>&1| tee -a $OUT/${ci}-save-load-state.log
(time ./bin/llama-completion -no-cnv --model ${model} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-${qnt}.log
(time ./bin/llama-perplexity --model ${model} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-${qnt}.log
(time ./bin/llama-fit-params --model ${model_f16} 2>&1| tee -a $OUT/${ci}-fp-f16.log)
(time ./bin/llama-completion -no-cnv --model ${model_f16} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-f16.log
(time ./bin/llama-completion -no-cnv --model ${model_bf16} -ngl 99 -c 1024 -s 1234 -n 64 --ignore-eos -p "I believe the meaning of life is") 2>&1| tee -a $OUT/${ci}-tg-bf16.log
(time ./bin/llama-perplexity --model ${model_f16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-f16.log
if[ -z ${GG_BUILD_NO_BF16}];then
(time ./bin/llama-perplexity --model ${model_bf16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-tg-bf16.log
fi
(time ./bin/llama-imatrix --model ${model_f16} -f ${wiki_test} -ngl 99 -c 1024 -b 512 --chunks 2) 2>&1| tee -a $OUT/${ci}-imatrix.log
(time ./bin/test-llama-context --model ${model_q4_0} -ngl 10 -c 1024 -fa off --no-op-offload) 2>&1| tee -a $OUT/${ci}-test-llama-context.log
(time ./bin/test-llama-context --model ${model_q4_0} -ngl 99 -c 1024 -fa off ) 2>&1| tee -a $OUT/${ci}-test-llama-context.log
(time ./bin/test-llama-context --model ${model_q4_0} -ngl 10 -c 1024 -fa on --no-op-offload) 2>&1| tee -a $OUT/${ci}-test-llama-context.log
(time ./bin/test-llama-context --model ${model_q4_0} -ngl 99 -c 1024 -fa on ) 2>&1| tee -a $OUT/${ci}-test-llama-context.log
@@ -521,6 +495,8 @@ function gg_run_embd_bge_small {
(time ./bin/llama-embedding --model ${model_f16} -p "I believe the meaning of life is" -ngl 99 -c 0 --no-op-offload) 2>&1| tee -a $OUT/${ci}-tg-f16.log
(time ./bin/llama-embedding --model ${model_q8_0} -p "I believe the meaning of life is" -ngl 99 -c 0 --no-op-offload) 2>&1| tee -a $OUT/${ci}-tg-q8_0.log
"save the state of each slot with a prompt to this directory on shutdown, and restore it for a prompt that starts with it after a restart (default: disabled)",
string_format("size limit of the --cache-disk-path directory in MiB, the least recently used files are removed (default: %d, 0 - no limit)",params.cache_disk_mib),
[](common_params¶ms,intvalue){
if(value<0){
throwstd::invalid_argument("cache-disk-size must be non-negative");
string_format("share of the MoE cache misses that the CPU computes from the host experts instead of uploading them to the GPU once the cache is full, from 0 to 1 (default: %.2f)",(double)params.moe_cache_cpu_share),
[](common_params¶ms,conststd::string&value){
constfloatshare=std::stof(value);
if(share<0.0f||share>1.0f){
throwstd::invalid_argument("invalid value");
}
params.moe_cache_cpu_share=share;
}
).set_env("LLAMA_ARG_MOE_CACHE_CPU_SHARE"));
add_opt(common_arg(
{"-ncffn","--n-cpu-ffn"},"N",
"keep the dense FFN weights of the first N layers in the CPU\n"
"experimental: whether to enable MCP CORS proxy - do not enable in untrusted environments (default: disabled)",
"experimental: whether to enable MCP CORS proxy, for Llama Chat to reach public MCP servers without CORS headers; local and private addresses are refused unless --ui-mcp-proxy-allow-private-urls is set - do not enable in untrusted environments (default: disabled)\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
"allow the MCP CORS proxy to reach local or private addresses (e.g. MCP servers on localhost or LAN) - anyone who can reach the proxy can then reach your local network (default: disabled, only public addresses)\n"
string_format("restrict to only support embedding use case; use only with dedicated embedding models (default: %s)",params.embedding?"enabled":"disabled"),
string_format("for router server, MiB of device memory to keep free per device, unload LRU models until a new one fits (default: %d, 0 = disabled)",params.models_memory_margin),
string_format("physical maximum batch size of the draft context, use a smaller value to spare VRAM (default: %d - same as --n-ubatch)",params.speculative.draft.n_ubatch),
Loaded 100 of 3105 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.